Thanks for Tyrell for finally locating the source of the elusive Java heap space error, which I came across sporadically during my daily Mashup Server stress testing.  Tyrell describes the fix that you can add to your 1.5.1 installation here.

Can’t help but repeat a few choice bits from a recent article Enterprise Mashups Part II: Why SOA Architects Should Care by Chris Warner and John Crupi.  Too bad I didn’t see this before the Enterprise Capabilities of the WSO2 Mashup Server 1.5 webinar I gave this morning - I could have cribbed a few bits of wisdom! Like these:

Mashups bring a “user” into the SOA mix. … By having the business build mashups upon the foundation of your service-oriented architecture, they essentially become SOA champions without knowing it.

This is a point we tried to hit strongly in the WSO2 Mashup Server, which can help bring individuals, teams, or enterprises together into a community around services, and strengthen the concept of a "user" and help build a diverse set of user experiences tied to services.  Until the SOA platform gets connected to the business users, the promises of SOA (business agility) won’t be fully realized.

In addition to SOA-friendly formats, such as RSS and Atom plus REST and SOAP, mashup creators can publish mashups to spreadsheets, as WSRP-compliant portlets, wiki- and blog-friendly widgets, or even into a mobile phone as a micro-application. Mashups can become the vehicle through which services become part of the everyday tools of the enterprise business user.

Which is why we’ve added on to the RSS, Atom, REST, and SOAP support in 1.0 the ability to interact with spreadsheets and databases (through Data Services), and are starting our foray into the world of portlets, widgets, and gadgets with support for Google gadgets.

Mashups can go well beyond leveraging an SOA by becoming part of that SOA, allowing developers to create customized “service skins” from core services. …

Because mashups can be exposed as REST-, WSDL- and JSON-based services, they look and feel like a real SOA-based service to developers who want to consume them. … The mashup, created by the developer, becomes the tailored service which is directly aligned with their particular need. Major enhancements to core services can be accommodated with a reformulated or updated mashup by the mashup creators themselves.

Yes, we’ve found our users (and ourselves) using the WSO2 Mashup Server for more than just combining services into a new one, but for rapid development of new services, or for customizing services to a particular scenario.  From the outside, these don’t just look like real services - they are real services - for example in the 1.5 release high levels of security can be applied to the services even though they are so easy to write.  A good example of a "skinned" service - the digit2image sample service that comes with the mashup server tailors a very general purpose and complicated service (Flickr) for a particular narrow purpose - to find a random image, with appropriate copyright, for a single numeric digit.  By providing a narrow API, this capability can be exposed more easily and efficiently (some caching is performed by the service to speed up returns).  And it allows innovation without touching the original Flickr service which we don’t have control over.  For instance, I could search beyond Flickr for appropriate images without breaking the service contract or rewriting the Flickr service.

Toward the end, the authors write:

At this point any SOA architect worth his WSDL should see that mashups can greatly enhance an SOA and don’t necessarily ignore or break the principles that made your SOA great. But governance, granularity and scope for mashups do require subtle tweaks and adjustments to your enterprise toolset.

Indeed, governance is important as your SOA grows and chaos theory has a chance to get a toehold.  The WSO2 Mashup Server is taking the first steps into governance by relying on the WSO2 Registry to store mashups in a versioned repository, manage ownership, users and roles, and facilitate communication and the building of trust in an Enterprise SOA community.  But we still have to to a better job of working with the Registry’s features for supporting multiple versions, product lifecycles, and dependency management.  For instance, while the Mashup Server stores all it’s mashups, comments, tags, ratings, etc. in the Registry, when you deploy a new mashup, it doesn’t automatically populate the registry with the WSDL.  We’re looking at much better integration with the Registry as a primary feature of our next Mashup Server release.

Kudos to the authors for clearly presenting not just the case for service mashups in an SOA architecture, but why any SOA architecture that doesn’t expand into the mashup space is missing out on huge opportunities.

As announced on the mooshup blog, the WSO2 Mashup Server team completed a new release, which includes 50 bug fixes.  Most of these are small, but a few are worthy of note:

• Issues with Email host object in v1.5 fixed.
• Private key management features.
• Fixes in the SMTP and JMS transports (Keith is working on an article about how to enable them.)
• WSRequest host object now supports a fire-and-forget message exchange pattern.
• Supporting http proxies more consistently (e.g. in the Feed object).
• Lots of little UI nits and polish.

I’m very pleased with this release, as it is often the case that some of the fit and finish work gets postponed in the last stages of a major release like 1.5.  Open source is about rapid and continual improvement, and this release fits that bill nicely!

Download it today!

As announced here:

Title: Enterprise Capabilities in the WSO2 Mashup Server 1.5
Date: August 12, 2008
Start Time: 9.00 am PST
Duration: 60 min
Presenter: Jonathan Marsh

The WSO2 Mashup Server provides a powerful platform for developing and deploying mashups. In this Webinar, we’ll provide a brief introduction to the Mashup Server architecture and Javascript-based development model, as well as highlight the new features in the 1.5 release that support enterprise deployments. These include support for message level security (authentication, encryption, signing) on mashup services, and the ability to expose a data source such as a relational table, an Excel spreadsheet, or a simple CSV file as a zero-coding Data Service.

Attend this Webinar if you:

• You have services, web pages, or other information sources available but you want smarter ways to use those services.
• You want to expose or compose services in a simple yet powerful way, complete with enterprise-level security.
• You are an existing WSO2 Mashup Server user and would like an overview of the new features in the 1.5 release.

Your presenter, Jonathan March, is the Director of Architecture for Mashup Technologies, and led the design and development of the WSO2 Mashup Server. Jonathan, a product designer and devoted script hacker, has spent over ten years developing and standardizing, XML and Web service technologies, most recently at Microsoft where he contributed to the standardization of XML, XSLT, XPath, WSDL, and other technologies.

Register now!

What is XSS and why is it restricted?

[Note: also published on the Mooshup.com blog.]

Disclaimer: I’m not a security guru, so what follows is my opinion, observation, and experience.  Please feel free to comment and correct!

Modern browsers protect against release of private information to a third (possibly malicious) party by imposing “cross-site scripting” or “XSS” restrictions. The basic attack is that a browser is pointed to a web site that is trusted (to some degree) by the user. In using the web site the user may provide some confidential information to that web site, such as a password, bank account number, or some other information that could be exploited by a malicious entity. Naturally, the user would hesitate to provide this information to a site that he doesn’t trust (the user can be fooled through a set of techniques known as phishing, but that’s a different story.) Here we assume that the user has gone to a site that is authentic.

However, it is possible that even a trustworthy site, through poor construction or through compromised delivery mechanisms, could be “hacked” by a third party. For instance, accessing the site through an open (but malicious) wireless network may allow the page to be subtly changed during transmission. This change might be to insert a bit of script code that records the interactions the user has with the page, including information he enters such as a password, and also information that is provided by the website to the user. The inserted script could collect this private information, and then "phone home" to the attacker.  HTTPS can mitigate such attacks by securing the communication channel, but interactions with plain HTTP sites may still disclose user secrets of various levels of sensitivity.

Attacks can also come, and generally do come, over a trusted internet connection, even possibly through HTTPS. Anytime user-generated content appears in a page (e.g. comments on a blog, etc.) there is a possibility that third party, and thus untrustworthy, content is piggybacked on a trusted site.  Plain text third-party content is benign (what you see is what you get), but if the content can be submitted in html, it is possible that such html can include malicious scripts. For this reason, a trustworthy site that allows user-generated content must scrub any user-generated content provided to it, removing anything that could be executed as script.

From the perspective of browser vendors there are a lot of sites out there and not all of them consider the security implications of user-generated content adequately. To help protect the user, XSS counter-measures in the browser attempt to limit the ability of and scripts within the page to “phone home”. The is accomplished by preventing HTTP POST (the protocol used to submit forms and upload data) access to any web site domain called on by the page other than the one from which the main page originated. For instance, a page from http://wso2.org can access “safe” content from alternate domains like http://wso2.com, including images, stylesheets, even script libraries. The page however won’t be able to post a form containing user input to anywhere but http://wso2.org. The XMLHttp object provides a way to POST from script, and also prevents information from being posted to any domain other than the page domain.

So as a user, having a browser watch out for these types of attack and prevent them seems useful.  But let’s consider situations where they get in the way of useful, trustworthy work.

Why a developer might want to access alternate domains

For a web developer (especially a mashup developer) XSS can be quite a pain, as it limits your ability to write a page that spans domains.  It limits your ability to host AJAX and Web Service interactions (powered by the XMLHttp object) anywhere other than your primary domain.  For instance, you can’t host a Web service on mooshup.com and use it within your own application (at least directly from the browser).  Even though both sites may be trusted by you as the web site author, the browser enforces a blanket restriction on this access. (Each browser has mechanisms that may loosen this in some circumstances, but there isn’t anything with zero-config or cross-browser.)

This restriction limits applications such as gadget pages (e.g. iGoogle.com) that aggregate information from a large number of sources.  The Google Gadget framework, for instance, provides a way to GET information through a proxy on the trusted server, but currently disallows similar capabilities for POSTing.

The Loophole - Script Injection

Don’t start feeling too secure as a user, or too disappointed as a developer trying to do legitimate work – there are some loopholes that can be exploited.

As described above, an HTTP GET operation is assumed to be safe across domains, while HTTP POST is not.  If one could masquerade a POST as a GET, one could circumvent security restrictions.  In particular, script can be fetched regardless of domain.  This powers important functionality, such as third party libraries, an important feature supporting simplified development, analytics, and advertising.  Basically one needs to translate the body of the POST into url parameters on the GET (recognizing there are length and encoding issues to deal with), insert a <SCRIPT> tag dynamically into the page which uses the GET, and the server on the external domain can access the "posted" information.  It can even send a response back in the form of a block of script (essentially a callback).  Of course, you need to insert script into the page initially to get the ball rolling - which can be pretty difficult over a secure connection or for sites that properly sanitize user-generated content.  But if you’re the owner of the original site, it’s not terribly difficult once the technique has been mastered.

Naturally, a user can protect themselves reliably against these attacks by turning off Javascript in the browser.  Or cutting the internet wires.  Or burying the computer in the backyard and raising carrier pigeons.  All quite practical alternatives, don’t you think?

WSO2 Mashup Server 1.5

The new WSRequest.js in the new WSO2 Mashup Server 1.5 release has facilities to “exploit” this loophole when the developer wants to access a mashup from a page within another domain.  If an "access denied" error is returned from the XMLHttp object, a script injection is attempted instead.  This allows you to use the Mashup Server’s convenient stubs within a page or a gadget without encountering the XSS pain.  There are some restrictions in the fine print of course - only asynchronous calls are allowed, message size is limited, and the wire-level messages no longer are conformant to open standards for Web services, but those aren’t unreasonable considering the alternatives.

Conclusion: time to drop XSS restrictions?

So, my question is, if XSS restrictions are so painful, yet circumvented with a modest bit of work (hey I’m no genius at this stuff and I did it) why are the XSS restrictions in place at all?  Instead of trading off convenience for security, you’re imposing convenience without actually making a meaningful contribution to the user’s security.  The additional level of security provided by making cross domain access simply obscure rather than truly prohibited doesn’t seem worth it.  Is it time to dump XSS restrictions?  Or do we need to add a new (and further inconvenient) restriction against inserting <SCRIPT> tags into a page dynamically?  As long as there is any cross-domain access I don’t think I’ll be completely secure.  And that rules out advertisement insertions which I don’t think is going to happen anytime soon!

I’m proud to say that v1.5 of the WSO2 Mashup Server is now available for download!  This release has been a few months in the making, and I’m thrilled to see a major update to the Mashup Server finally out.  News release here. 

Thanks to the efforts of Keith, Tyrell, Channa, Yumani, and many others at WSO2, we’ve been able to put some snazzy new features in this release:

• Integrated award-winning Data Services: quickly and with zero-coding map queries on your databases, Excel spreadsheets, or CSV files into Web services.
• Google Gadgets: power a Google gadget with a mashup service - we generate a try-it, a code template for customizing the gadget, and even a build in Gadget dashboard based on Apache Shindig.
• Security, security, security: Limit access to your mashup service to only authorized users, encrypt and sign your messages, and access services which are secured.  I think our security interface is the easiest way to engage advanced WS-Security features that’s out there!
• OpenID login support.

There’s lots of little improvements too, and I hope to talk about those, as well as the major features above, in upcoming posts.

For now though, a hearty congratulations and a good night’s sleep to the Mashup Server team!

Yumani has published a nice tutorial on Sending Instant Messages via Mashup Server showing how to use the instant messaging capabilities built into the WSO2 Mashup Server.  I think the ability to have instant alerts as part of your mashup is a cool feature, and as Yumani shows in this article, they are pretty trivial to do.

I’m looking forward to adding the next step in a future release - the ability to invoke a mashup using instant messaging.  Then we’ll have a scriptable platform for building agents that can communicate via IM.  Imagine converting your favorite service to an IM conversational interface.  Or making a data source available through an IM conversation.  Or even imagine new games…  Endless possibilities!

Microsoft, Google, and others have just launched the Information Card Foundation, to promote awareness and interoperability around InfoCards.  This is great news - I think InfoCards have the potential to solve some real problems with managing identities and securing them against theft.  Despite the client-side technology being available in Vista for a year plus, we’re still slow to see installations emerge in the marketplace.  The only use I make of InfoCard (and then still pretty rarely) is in the WSO2 Mashup Server, which has both InfoCard and OpenId support (though you can use an old fashioned username/password).

Managing identity and security without complicating the user experience is still a challenge.  I’m hoping the Information Card Foundation can encourage broadly useable solutions as it gets rolling.  And I’m hoping some large properties (PayPal is a member) may actually start putting stronger identity management technologies in place on a broad scale.

It should be no surprise with our products supporting InfoCards and OpenId that WSO2 is now a member of both the Information Card Foundation and the OpenID Foundation.

One of my mashups fell prey to the dreaded scrape rot - a complete overhaul of the target site that invalidated all of my scraping rules.  The pages in question are from globalincidendmap.com, which previously powered my internationalincident mashup (see Sri Lanka Incident Mashup).  The change was catastrophic - queryable content from the site is no longer free, but requires a paid membership and a login process.  One option would be to pay for a membership, but besides the steep price I doubt that the license terms allow republishing of the data.  I could support the mashup only for paid users of the service, collecting credentials and forwarding them on, but that again is both questionably secure (a user would have to trust that I didn’t abuse the credentials temporarily in my possession), and unrealistic since few if any of my audience would spring for the cost of membership.  In effect, my mashup has been totaled.

This illustrates one reason why scraping should be used only as a last resort, when no more stable forms of content are available - feeds or Web services.  When you mix the content and presentation, changes in the presentation are easily confused with changes in the content.  Although the scraping features of the WSO2 Mashup Server are popular, I like to think of them as a stop-gap while publishers find cost-effective ways of serving up presentation-free content, such as delivering simple services using the Mashup Server ;-).  Ideally, more and more publishers will recognize the value of raw content, and the need for Web scraping will diminish.  Gonna take a while though…

Even without scraping, there remains one of the deep problems with mashups and distributed programming, that of services that disappear, are altered, change usage terms, etc., breaking their dependent mashups in the process.  There has been lots written on this, which can be generally summed up as "this is a hard problem."

One thing we plan to do in the future to make sure that service changes don’t harm downstream dependents is use more of the advanced functionality of the WSO2 Registry upon which the WSO2 Mashup Server is built - namely, versioning.  Today, each time a service changes, an old copy is retained in the database, but no longer is "alive" as a service.  Some future version will have a simple interface for continuing to keep the old versions online, and help users to lock into one of these previous versions.  Some cool dependency management features on the drawing board for the Registry will also help find and record dependencies and notify dependents of changes.

But would these help in the case of the internationalincident service?  This is a case where there is a deliberate change which prevents "unauthorized" access.  The solution in this case was to mark the service as obsolete, and go out and find a whole new source of data.  The new srilankanincident service is a result - though the data is slightly different, perhaps a result of focusing narrowly on Colombo, it was a fairly short task to reprogram it, and even improve it, once I had found a new source of data.  The speed of fixing catastrophic failures is my current best hope against scrape rot.

I’ve got another free Webinar coming up - again an Introduction to the WSO2 Mashup Server and to mooshup.com on 13 May from 9-10AM PST.  Join me if you:

Seems the WSO2 crew has been blogging about WSO2 appearing on the "cool 5" companies in a recent Gartner report (paid subscribers only).  What intrigued them about the WSO2 Mashup Server was support for the hitherto paradoxical "lightweight but enterprise-oriented" services.

And here I am a couple of days late.  I guess for breaking news and the real skinny on "cool" you would do well to add the feeds of Paul, Sanjiva, Daniel, Glen, and Keith to your blogroll.

Just noticed this interview I gave at Mashup Camp posted on YouTube.  (Does everyone hate hearing themselves talk as much as I do?)

I’m giving a free Webinar on 15 April 2008 9-10AM PDT about the approach we took to service composition in the WSO2 SOA Platform.  Instead of a declarative approach which my XSLT days showed can be powerful yet also limiting in many ways compared with a full programming language, the WSO2 Mashup Server uses a "scriptable Web Services" metaphor, and supports the ability to consume and produce Web Services using simple Javascript expressions.  Add to that the ability to script non-Web Service materials like Web pages and feeds, and you’ve got a powerful yet accessible platform for creating Web Service mashups.  Register now!

I just posted a finished version of a Mashup designed to help answer the question "is Sri Lanka getting safer or not?"  This is a question we on the global WSO2 team ask each time we arrange travel to that unfortunately troubled country.  Despite a spate of violence early this year, designed to coincide with the formal dissolution of the cease fire that has done little to prevent violence, it seemed to me things were getting a little better.  But I needed some facts to back that up.

The mashup plots bombings and other incidents as a bar chart, measuring the severity of the incident (how many killed & wounded) over the last 18 months.  The idea was to see if there was a clear overall trend in the violence or not, something not readily apparent from a Google map (see right).

The mashup service itself consists of several items, each one a simple task accomplished in half a page of javascript:

• Scrape the search results page at globalincidentmap.com to pull the essential data out of an HTML table for the country of interest, and put it into a simple XML structure.
• Cache the page if it has already been accessed within 24 hours (scraping is expensive, the first access in any 24 hour period will be pretty slow.)
• Parse the headline for patterns such as "eight killed" or "injures 7" and turn that into killed and wounded digits (this isn’t perfect, but we can tolerate a few errors since we’re trying to present an overall sense of the problem rather than perfectly accurate statistics.)  Also filter out as much as possible the killing of LTTE, as that’s more a measure of war than of terrorism.
• Provide a helper method to look up details and get a link to a new story for any item.

Using this service (called internationalindicent) I created a custom HTML UI to format Sri Lanka-specific results into the bar chart and to make it interactive (click on a bar to see more info about the incident.)  Then I used "share this mashup" to upload the service to mooshup.com so others could try it out (or copy the code.)

The rough version took a couple of hours, mostly figuring out the details of scraping the page and coming up with the headline patterns to look for, but then I spent a couple more polishing the HTML UI so I wouldn’t be embarrassed to share it.  In the process I demonstrated some of the powerful aspects of using mashup technologies in your development arsenal:

• Instead of investigating the current situation using Google News for 30-60 minutes each time we plan a trip for the latest information, I can browse this site in a few minutes, see the trend, and get details of any recent incidents I’m interested in.  This will pay for itself in terms of my own personal productivity before long.
• There is also a small user group (really, just the handful of WSO2 employees based outside Sri Lanka) who can also benefit from this micro-application, increasing their productivity as well.
• The WSO2 Mashup Server makes the data available as a service, so others can reuse it too, for alternate displays or to generate displays for other parts of the world.
• And, it’s just fun!

So … is Sri Lanka getting safer?  I’ll have to let you be the judge of that.  Go to http://mooshup.com/services/jonathan/internationalincident/ to see for yourself!

[Update 5/27: This service is no longer available.  See the newer entry for more info.]

Just returned from some travel which included 2 days of Mashup University and a day of Mashup Camp.  A few thoughts:

• Maps are still well represented.  This surprised me a bit, but I recognize I’ve been well immersed in the WSO2 Mashup Server which supports a wide variety of User Interactions (html pages including maps, but also feeds, email, instant messaging.)  I predict non-map mashups will start to eat into the map-based mashup market share dominance over the next year.
• There’s lots of interest in consuming various Web APIs.  Some vendors were promoting their APIs, others tools for consuming those APIs.  I think the WSO2 Mashup Server can tap into an underserved market here, since it’s the easiest way I know of to deliver a comprehensive Web API on top of a bit of Javascript logic, which in turn can front information sources as diverse as databases and scraped web pages.  I think the Mashup Server can become a "design your own API" tool that can have great appeal to the mashup developer.
• A lot of Javascript was in evidence, further validating our choice to use Javascript for mashup logic in the WSO2 Mashup Server.
• For the first time I ran into a few people with serious interest in WADL.  Has its time finally come?  More on that in a subsequent post.
• The "unconference" style was quite interesting and successful, especially if you’re like me and are more interesting in connecting with interesting individuals than in hearing yet another vendor pitch (mine excepted of course!) ;-)  One thing that is sorely lacking is any kind of organization to the conference Wiki, and a surprising lack of ability to record much of the activity there.  I couldn’t even find who won the Best Mashup contest…

I can’t wait till next year!