WSO2 Mashup Server Screencast Series #3: Service Composition

[Also posted to the mooshup blog.]

The third installment of my introduction to the WSO2 Mashup Server is now live. 

This week’s episode augments last week’s episode (on scraping a web page) by composing the scraping service with a simple storage service, and adding periodic execution to generate an ongoing history of the number of downloads of my podcast.

For best results (when you have more bandwidth) view the screen cast in it’s original size and quality here or click below.  [Note: this appears to require an OxygenTank login – a mistake that should be corrected soon.]

image

If you’re bandwidth is constrained and you’re willing to put up with some blurriness, a smaller version has been posted on YouTube here.

WSO2 Mashup Server Screencast Series: #2 Web Scraping

[Also posted on the mooshup blog.  10/29: Updated links.]

Following up on last week’s screencast, the second in the series is also live now.

This episode shows a simple but fairly realistic scenario for the WSO2 Mashup Server, analyzing a web page using Firefox/Firebug, creating a service to scrape information out of the page, and expose it as a service.  Including a little live debugging!  Also looking a little deeper into the type annotations and their effect on the generated WSDL.  Hope you enjoy it!

For best results (when you have more bandwidth) view the screen cast in it’s original size and quality here or click below.

image

If you’re bandwidth is constrained and you’re willing to put up with some blurriness, a smaller version has been posted on YouTube here.

Launching the WSO2 Mashup Server Screencast Series: #1 Hello World

[Also posted on the mooshup.com blog.  10/29 Updated links.]

Last week we launched the first in a series of screencasts I created, showing how to get started on the WSO2 Mashup Server, and then jumping in to some fairly realistic scenarios for building a powerful mashup.

Screencast #1 is the very basics – building a Hello World service using nothing more than the WSO2 Mashup Server and Notepad.

For best results (when you have more bandwidth) view the screen cast in it’s original size and quality here or click below.

image

If you’re bandwidth is constrained and you’re willing to put up with some blurriness, a smaller version has been posted on YouTube here.

Bot wars!

I wondered how Southwest found my auto-checkin mashup so quickly.  Until I did a Google search of “southwest automatic checkin” and found that my corresponding blog entry showed up at number 6.  I have little doubt that Southwest subscribes to such a feed and uses it to notify their legal department of any new potential infractions.

So was my mashup caught by a mashup?  Four out of five conspiracy theorists say yes!  (The fifth doesn’t want to cross his secret CIA paymasters.)

Bots and Mashups taking over the world

This Southwest incident has made me somewhat introspective.  As I started to muse in my previous post:

… it would be nice to have not just Terms and Conditions that allowed the site and its content to be reused freely. …  It’s unfortunate they can’t do this without changing their checkin and boarding procedures in some way.

I really wonder whether in the long term whether Southwest’s procedure can withstand the onslaught of “bots” like mine.  It seems doubtful.  But the consequences of being overrun by “bots” is already causing alarm in some places.

For a very thoughtful look at the risks, the consequences, and maybe some ways to approach solutions, I highly recommend checking out the Long Now Foundation’s excellent podcast featuring Daniel Suarez talking about Bot-Mediated Reality.  Based on that I can’t wait for his book to be rereleased ;-).

Southwest mashup meets Terms and Conditions

That didn’t take long!  My (now deleted) southwestAutoCheckin mashup has generated some controversy.  Within a few days of posting it, WSO2 received the following email from Southwest Airlines (minus headings and signature):

October 9, 2008

VIA EMAIL

Dear Sanjiva Weerawarana Ph. D,

I represent the Brand Protection Team for Southwest Airlines Co. (“Southwest”). Jonathan, Director of Mashup Technologies at WSO2, recently blogged about offering a service in which your company, without Southwest’s permission, offers online check-in services and boarding passes for Southwest flights. The name of this website is Mooshup.com.

The program offered on Mooshup.com uses southwest.com to check Southwest Customers in for a Southwest Airlines flight in an effort to obtain for them an “A” boarding pass. By using the Southwest Airlines website you are bound by the Terms and Conditions and the Use Agreement which all users are subject to in exchange for using southwest.com. Mooshup.com is in direct violation of the Use Agreement including, but not limited to, the following:

Southwest’s web sites and any Company Information is available to you only for your personal use to determine the availability of goods and services offered on the web sites and to transact business with Southwest. Unless you are an authorized Southwest travel agent, you may not use Southwest’s web sites for any commercial use or other purpose.

You may not use Southwest’s web sites for or in connection with offering any third party product or service not authorized or approved by Southwest. For example, online check-in service providers may not use the Southwest web sites to check-in Customers online or attempt to obtain for them a boarding pass in any certain boarding group.

For any use not specifically listed herein, you may not use Southwest’s web sites after Southwest notifies you that your use is not authorized and requests that you cease such use.

As your company is using southwest.com in an unauthorized manner which Southwest believes to be harmful to its business and its Customers, we are asking you to discontinue this activity immediately.

Southwest hopes that this matter can be quickly and amicably resolved. I’m asking that by October 23, 2008 you either sign and return a copy of this letter to indicate your agreement with the terms, or reply back to this e-mail with an indication that you have discontinued offering this service and removed the entry from Mooshup.com homepage and the blog posting from MSN Windows Live (auburnmarshes.spaces.live.com/blog).

Please be advised if we do not hear from you by the specified date, Southwest reserves the right to take whatever action we deem necessary to enforce our rights.

Very truly yours,

I did the mashup to demonstrate a simple workflow including scraping and notifications.  I wasn’t really trying to undermine Southwest’s business model.  Everyone knows Southwest is my favorite airline.  So, where did I go wrong?  Southwest above outlines two complaints: personal use only, no online check-in service providers.

First of all, personal use seems pretty tricky.  A user of my service is actually using it for personal use, to interact with Southwest’s web sites.  I don’t really think providing the user with a better tool, essentially a better “browser” tailored to Southwest’s site, changes this.  At no time does the mashup contact Southwest except under specific instructions from the user.  So I hold the mashup itself blameless.

However, their second complaint seems a bit more concrete - offering the southwestAutoCheckin service online.  I can see that Southwest would consider mooshup.com, or even my user account on it, as an “online check-in service provider”, and that could be argued as a violation of the second item quoted in the terms above, especially since I made it publicly available.

Are these Terms and Conditions reasonable?  I think they fit within the current norms of society, reflecting the fact that an online service should be used in a way consistent with it’s purpose and limiting abusive behavior.  If for instance, a random user had posted the southwestAutoCheckin service, our mooshup.com Terms and Conditions would have allowed us to remove it for the simple reason of avoiding even a semblance of conflict.  If every service provider had to consider countermeasures for every possible type of abuse we wouldn’t have the array of services we do now.

And I can see if my service became popular that it would cause significant upheaval in Southwest’s customer experience (if 50% of a flight used my service and 50% didn’t, the latter would be disadvantaged in their seat selection, with no recourse through Southwest if they thought the procedure unfair.)

However, I think the norms to have some evolution ahead of them, and it would be nice to have not just Terms and Conditions that allowed the site and its content to be reused freely, but also real Web Services to make this valuable information (some of which is owned by the user after all) available in an easier manner.  It’s unfortunate they can’t do this without changing their checkin and boarding procedures in some way.

The moral of the story – carefully read the terms and conditions of any site you scrape, and make sure you stay within those terms.  If you don’t like the terms, find a different provider, or agitate for better terms.  This is a real danger for mashup authors.

In this case, there is no percentage for me in conflict, real or perceived, with Southwest – definitely not for a simple Mashup Server demo.  So the southwestAutoCheckin mashup is permanently offline.  I don’t see anything in the Terms and Conditions that justifies removing any blog post, and I’m confident that updating the previous post and providing context here will satisfy Southwest while remaining fully transparent.

Southwest Airlines Automatic Online Checkin

[Update 9 Oct 2008: this mashup has been pulled off of mooshup.com.  See here for details.]

I wanted to go the head of the line on my recent Southwest Airlines flight.  Although Southwest doesn’t have assigned seats, those who check in first get to board first.  The last end up with yucky middle seats.  Especially useful in playing this system is the 24-hour advance checkin online.  If you can check in early, you get the lowest boarding number and your pick of the choicest seats.  But virtually everyone seems to know this now and there’s a bit of a grab for the best boarding numbers.  Wait two or three hours and half the passengers have checked in.

This time I successfully checked in online within 5 minutes of the opening of online checkin, even though I was actually in the air on another flight at the time!  My new and still under-development southwestAutoCheckin mashup worked brilliantly and gave me one of the lowest boarding numbers I’ve seen.  It works like this:

  1. Use the try-it page to input information about your upcoming flight (confirmation number, passenger name, airport codes and so forth) into the mashup when you make your reservation.
  2. The mashup will alert you that it is tracking your flight for automatic checkin.
  3. 48 hours before the flight, the mashup will remind you it’s planning to check you in automatically.
  4. 24 hours before the flight, the mashup will actually check you in, although it can’t print your boarding pass for you.
  5. 3 hours before the flight, the mashup will alert you of the time of your flight.
  6. Reprint your boarding pass at an airport kiosk.
  7. After the scheduled flight time, the mashup will alert you that it has completed its work and is deleting the flight from its watch queue.

When I say “alert” I mean that the southwestAutoCheckin service uses the alertme service to distribute a notification to you.  You can register with the alertme service to send alerts through any set of email addresses, instant messenger accounts (MSN and Yahoo currently supported), or a Twitter feed.  Pretty cool, no?

Still lots to of improvements that can be made – more IM providers and even SMS for the alertme service, as well as a nice HTML interface; scraping the information out of a GMail message instead of having to input it manually, as well as automatically calculating the GMT time for the flight based on local time and the airport name.  But the basic mashup seems to have worked brilliantly in it’s first real world trial.

So yesterday when my flight started boarding, was I there proudly at the head of the line?  Ironically, no.  Boarding started early, and in the few moments while I was packing up my laptop and charger, all 18 other passengers had boarded.  Yes, despite having the best boarding pass obtained by my personal digital agent, I was the very last to board.  But I still had 119 empty seats to choose from ;-).

Upcoming flight on Southwest?  Let the southwestAutoCheckin mashup give you a helping hand!

Creating a custom Google Gadget for a mashup

Mihoko asks a question about creating a custom Google Gadget for a mashup using the WSO2 Mashup Server, and displaying it in the Dashboard:

I have a question about Dashboard feature.   Is it possible to display images on my own Dashboard as the Try-it Gadget?  … For example, about digit2image Mashup, this gadget is able to get the URL information stored image, but not able to display the image.  Is it possible to display the image on my own Dashboard?

I thought this would be a good opportunity to explain some of the widget/gadget features now available in the WSO2 Mashup Server and walk though building a super-simple Google Gadget.

What is a gadget or widget?

A widget or gadget is a little program, usually with a cute and compact UI, that runs inside a widget engine.  The widget engine generally has the characteristic that a number of widgets can be viewed at once, allowing a user to construct their own digital dashboard of relevant information sources.  There are a number of widget technologies under various names – gadgets, widgets, portlets.  Some examples of widget engine include the Google Desktop, iGoogle, Windows Vista Sidebar, Windows Live widgets, Yahoo! Widgets, Apple’s Macintosh Dashboard, and lots more.

Most widgets or gadgets today seem to be simple user interfaces for online services – tracking weather, the stock market, movie times, quotes-of-the-day, etc.  We heard from a number of users that the WSO2 Mashup Server was a powerful tool for building the on-line part of these applications, and in the 1.5 release we added features to make it easier to create, test, and even host widgets.

WSO2 Mashup Server widget support

Of course we had to start small, so we focused on one widget engine, Google Gadgets, which can be

used in the Google Desktop or iGoogle, and provided some great features:

  • A Google Gadget version of the try-it for a mashup
  • A source code template for building a custom Google Gadget for a mashup
  • A link to the custom Google Gadget for a mashup
  • Support in the Javascript stubs for cross-domain access to a mashup
  • A customizable Dashboard for each user, which can host Google Gadgets from the Mashup Server or from other places.

Tyrell wrote a fairly definitive guide to these gadget features in Converting your WSO2 Mashup Server to a Personalize Dashboard, which I recommend.  I’ll give a brief and rather simplistic summary here for those of us with short attention spans!

Adding a Google Gadget to the Dashboard

You can find your customized dashboard for a local installation of the Mashup Server at https://localhost:7443/dashboard/ or you can get a free account (just supply a valid email address) at http://mooshup.com and access your customizable dashboard at http://mooshup.com/dashboard.  There are a few default 3rd party gadgets installed by default, but you can close any you dislike and replace them with your own.  Just select “Add Gadgets” and either type in the url of the gadget (available from the gadget publisher), or select one of the local mashups for a “try-it” gadget for that service.

Note that not all services can fit into the limited size of the gadgets yet – try out the “version” gadget for one that fits pretty well.  We have some work to do to enable gadgets to be resized, and to try to squeeze more into the limited size available for mashups with (e.g.) too much documentation (as if there were such a thing!)

For a mashup with a custom gadget, you can find the url of the gadget on the main page for that mashup.  For instance, the built-in upgradeChecker service has a custom gadget which you can add to your dashboard by pasting the link http://localhost:7762/services/system/upgradeChecker/gadget.xml into the “Add Gadgets” page.

Creating a custom Google Gadget

Let’s create a sample custom gadget to show how easy it is to provide the cute and customized UI that is the hallmark of successful gadgets.  You could take the gadget try-it code generated by the Mashup Server, save it as {mashupname}.resources/www/gadget.xml, and start to modify it, but it is usually easier to start from a smaller template.

I’m going to show you how to make a simple gadget for the built-in digit2image service, which when given a digit and a size, returns a URL to an appropriately licensed Flickr image of that digit.

Go to the mashup page https://localhost:7443/mashup.jsp?author=system&mashup=digit2image (if you’re logged on as an administrator, otherwise you’ll have to copy the digit2image mashup to your own user space and adjust the above url accordingly.)  You’ll notice a “Source code template for building a Google Gadget for this service" link.  You can download the template and edit it with a text editor, saving it as digit2image.resources/www/gadget.xml.  But today I’ll demonstrate the built-in editor instead.

So, click on the “Edit this Mashup” link in the task pane, and select the tab for “Gadget UI code.”  The editor will indicate that there isn’t a custom gadget yet, but by clicking the “Generate Template” button farther down the page, we can get the basic gadget definition written for us automatically.

The template even creates source code examples for an asynchronous invocation of each operation, so it’s easy to just tweak the code a bit and do new things:  (some comments removed)

// Demonstrates calling an operation of the 'digit2image' Mashup function
init() {
  // Set up a callback and an error handler for the digit2image operation.
  digit2image.digit2image.callback = showPayload;
  digit2image.digit2image.onError = handleError;
  digit2image.digit2image(/* (string) digit */ "0", /* (string) size */ "small");
}

// Sample invocations (unused) for other operations.
function samples() {
}

// Handles and error by displaying the reason in a dialog
function showPayload(payload) {
  log ("result-console", payload);
}

This automatically generated sample code asynchronously invokes the first operation in the service with some sample values, and includes a “samples” function to show how other operations can be invoked.  The results are serialized into the “result-console” part of the gadget page.  While some services won’t work out of the box because of additional constraints on the inputs of an operation (e.g. another operation must be invoked first, that although a parameter has a declared type such as string there are other constraints like that is must be a valid username, and so forth.)  The first thing we usually do is clean up the operations we don’t want, order the ones we do, and provide meaningful input values.

In this case, let’s use the values “9” and “thumbnail”, and display the resulting url as an image tag.  Replace the above code (all three functions), with this:

// Display an image for the digit "9"
function init() {
  digit2image.digit2image.callback = function (payload) {
    var output = document.getElementById("display");
      output.innerHTML = "<img src=" + payload + ">'";
    };
  digit2image.digit2image.onError = handleError;
  digit2image.digit2image("9", "thumbnail");
}

And let’s simplify the body of the mashup to just the image display and a place for errors to be displayed:

<div id="body">
  <div id="display">
    <!-- This div will contain the resulting image -->
  </div>
  <div id="error-console">
    <!-- This div will contain a description of any errors encountered. -->
  </div>
</div>

When you save the gadget and return to the mashup page, you will see a new link "View the custom Google Gadget for this service."  Copy this link and use the “Add Gadget” link on the Dashboard as I described earlier, and you will see the simple custom gadget as something like this:

image

There are naturally many improvements that need to be made to the Dashboard, and to our ability to power gadgets, widgets and portlets.  But I hope you find our experimental support intriguing and give us some feedback on the Mashup Server User Forum, just as Mihoko did!