Had a nice hike partway up Mount Rose today, and snapped a few shots along the way. Enjoy!
|
Sanjiva and I spent a couple of days last weekend knocking around the Five Lakes Basin near Yuba Gap in the northern Sierra Nevada mountains. Photoset here. Primary goal was to test our equipment and techniques for potentially more strenuous adventures. Crooked Lakes Basin is a high-reward area, with most trails having a lakes per mile ratio of greater than one.
We cooled off at the first of the Five Lakes, and continued up the trail towards the granite cliffs hoping to find more of the Five at the base – but it turned out the pristine lake wasn’t one of the Five and was named Glacier Lake. Although there were a few parties camped around the lake, the prospect of watching night fall over the lake and the Black Buttes beyond was too tempting and we also pitched camp there.
The rising sun found us climbing the ridge and scaling the Black Buttes, from which we had some amazing panoramic views, and a look into the basin on the south side, including Beyers Lakes and Baltimore Lake. Looks like another cool place to explore. Here’s a map.
The morning brought a quick hike out and home so Sanjiva could catch his flight back to Sri Lanka. All in all an excellent trip – about 15 miles almost half of which was strenuous cross country work. Within that short span we saw a microcosm of the Sierra high country and discovered that even this remote edge of the Sierra holds many charms.
Another advantage of the Prius – it’s a good car for While I’m at it I don’t think I ever posted my pic of the small bobcat that peeked in through a window at me before enjoying some r&r poolside a couple of months ago. [Photoset here.] Our adventure started as all adventures do in Sri Lanka, with a long drive on progressively smaller roads. Until the roads give out completely. Sights spin by at a The end of the road in this case was the tiny village of Palabathgala, near Ratnapura, and the eleven of us, plus guide, were deposited at the bottom of a staircase into the rain forest, leading up up and up into a misty escarpment. Already the grey skies were dripping on us.
After hours of this we reach a rocky river, spanned by a spiderweb of white threads. Then hours more of stairs, as we climb right up into the clouds. The temperature drops to a pleasantly chilly range, but the humidity remains so high everything remains soaked. At last we reach a few nondescript buildings, vendor shacks deserted now on the off season, and dry out a little. We engage the sole visible resident for some hot water and relish in a hot beverage as only the perpetually wet and cold can.
Now we’ve had our dinner packets - despite being tasty my post-extreme appetite is small and I can’t do much more than sample. With another round of animated chatter we laid mats in our cell and have lined up like sleepy sardines to await the morning.
Well, that night felt like Survivor – with each person having a share of the floor approximately two feet by five, cramping legs, whistling wind and rain, chilly enough to cause shivering, bumping into each other as you turn, occasional snores, and someone scraping through the door to use the bathroom every 45 minutes or so. I listened to music on my iPhone almost all night, but dozed off a few times for a total of maybe two hours of sleep. The whole experience felt like something from a 19th century Himalayan explorer’s tale. The traditional climax of a Sri Pada trip – viewing the sunrise – fell victim to the continued mist and increasing rain. But after some bread and jam for breakfast we visited the temple. My preparation for the elements – a complete poncho over clothes and pack – disintegrated in seconds in the howling wind – but I dutifully rang the bell once representing my first visit, and fled.
If you happen to find yourself travelling half way around the world, and have a few extra hours to burn in the middle, what could be nicer than Hong Kong? The landscape is conveniently compressed into a vertical form that makes it photogenic and easy to get around.
Next was an exhibit of painted scrolls, which although ancient capture an expressionism and spontaneity that the European tradition took centuries to reach. And the juxtaposition of text and image seemed quite trendy! The final exhibition was contemporary paintings – all of similar aesthetic to the ancient scrolls, but with modern subjects, colors, and compositions. Although some came across as fairly flat and cartoony, there were quite a number of amazing pieces from both the point of graphical composition and expressionistic brush artistry. Well worth the visit!
Then back to the train, quickly to the airport by about 3PM to await my final flight leg to Sri Lanka. A report of my adventures there coming soon! Full Flickr set here.
Flickr set here. A couple of weeks ago I ran across a fun little program that evolves to recreate an image out of overlapping transparent algorithms. Although not IMO really genetic, it mutates it’s “DNA” over many generations to reproduce a source image. I’m still having fun with it, trying to figure out if it could be adapted to glass. Here are a couple of samples:
Check out Roger’s gallery too.
It’s been a busy fall – not much time for taking pictures. Or uploading them promptly! Now that we’re in vacation mode, I’m looking through some of this falls photos. Here’s a small set from a hike with family members up to Donner Peak in early September. Some amazing rock formations…
See the photos here. We had a great time exploring Singapore over the last few days. Here are just a few of the highlights:
(This article first appeared a few days ago on the WSO2 Oxygen Tank.) I recently wrote a neat little mashup which demonstrates a little of the power of the WSO2 Mashup Server to flow information from one place to another, and from one format to another. I had a simple set of requirements:
Essentially then the task was to scrape the URLs from the photo of the day, and package them into a feed. The complication comes from the fact that there doesn’t seem to be a list of photos of the day available on the National Geographic web site – just links from a particular photo to the one for the previous (or next) day. Because a feed of 30 photos requires 30 different pages to be scraped, some caching really becomes necessary to improve performance, especially since feed readers can be expected to bombard the service if it proves popular. I initially broke down the task into three parts:
Here’s how I approached each of these tasks. Scraping a photo of the day pageThe first order of business for scraping a page like this is simply to fetch the page, tidy it into XML so we can navigate it using tools like XPath. The WSO2 Mashup Server provides a “Scraper” object that accepts an XML language describing the steps involved in scraping. This configuration language is defined by the Web Harvest component that we use for scraping. I usually start with a scraping mashup using a simple function that configures and performs the scrape, and returns the results:
function scrape_picture_page() {
var config =
<config>
<var-def name='response'>
<html-to-xml>
<http method='get'
url="http://photography.nationalgeographic.com/photography/ ¬
photo-of-the-day" />
</html-to-xml>
</var-def>
</config>;
var scraper = new Scraper(config);
var bodyWithoutXMLDecl =
scraper.response.substring(scraper.response.indexOf('?>') + 2);
var result = new XML(bodyWithoutXMLDecl);
return result;
}
The config language itself is pretty straightforward, once you learn to read it inside out – the <http> element fetches the requested URL, the <html-to-xml> does just what it sounds like and tidies the result, which is put into a variable named “response”. The scrape is performed by initializing a new “Scraper” object with the config, and the result is made available through the “response” property on the result – corresponding to the “response” variable we defined within the config file. One trick though – the result is a stream of XML text, including an XML declaration. The E4X extensions can parse this into XML (new XML()), but can’t handle the XML declaration. We have to strip off the declaration ourselves using string manipulation. By placing the above function in a file named “nationalgeographic.js” in the “scripts” directory of the Mashup Server, a Web service with a scrape_picture_page operation will be deployed. We can get to it through the try-it page (http://localhost:7762/services/jonathan/nationalgeographic?tryit) and see what the tidied HTML looks like for the page. Extracting the data from the page can be a tedious process, involving looking at HTTP request-response pairs and trolling through the HTML source of a page. Fortunately the National Geographic site’s HTML is simple and straightforwardly structured, with a number of well-placed identifiers to help us zero in on the interesting content. I usually end up using Firebug (Firefox debugging extension) to navigate the live HTML of the page and develop some XPath expressions that extract the desired metadata for the page. I’ve also found that, since Web Harvest communicates between components using strings rather than parsed XML, that defining a lot of XPath filters to extract information one element at a time during a scrape can perform poorly. Instead it seems much faster to wrap a series of XPath expressions into a simple XSLT stylesheet so the XML can be parsed once, queried as much as needed, and an XML structure containing the results returned in one action. To do that, I added an XSLT stylesheet to the above configuration: var config =
<config>
<var-def name='response'>
<xslt>
<xml>
<html-to-xml>
<http method='get' url={url} />
</html-to-xml>
</xml>
<stylesheet><![CDATA[
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" omit-xml-declaration="yes"/>
<xsl:template match="/">
<photo>
<xsl:for-each select="//*[@id='content-center-well']">
<date><xsl:value-of select="div[@class='date']"/></date>
<previous>http://photography.nationalgeographic.com ¬
<xsl:value-of
select="div[@class='slide-navigation'][1]/p/a/@href"/>
</previous>
<xsl:for-each select="div[@class='image-viewer clearfix']">
<xsl:for-each select="table/tbody/tr[1]/td/a">
<page>http://photography.nationalgeographic.com/photography/¬
photo-of-the-day/<xsl:value-of
select="substring-before(substring-after(@href,'enlarge/'),
'_pod_image.html')"/>.html</page>
<xsl:variable name="href"
select="concat('http://photography.nationalgeographic.com',
substring-before(img/@src, '-ga.jpg'))"/>
<location type='small'>
<xsl:value-of select="$href"/>-ga.jpg</location>
<location type='medium'>
<xsl:value-of select="$href"/>-sw.jpg</location>
<location type='large'>
<xsl:value-of select="$href"/>-lw.jpg</location>
<location type='wide'>
<xsl:value-of select="$href"/>-xl.jpg</location>
</xsl:for-each>
<xsl:for-each select="div[@class='summary']">
<title><xsl:value-of select="h3"/></title>
<credit><xsl:value-of select="p[@class='credit']"/></credit>
<description>
<xsl:copy-of select="div[@class='description']/node()"/>
</description>
</xsl:for-each>
</xsl:for-each>
</xsl:for-each>
</photo>
</xsl:template>
</xsl:stylesheet>
]]></stylesheet>
</xslt>
</var-def>
</config>;
Again, fairly straightforward – the <xslt> task has two inputs, <xml> and <stylesheet>. The stylesheet unfortunately has to be enclosed in a CDATA section rather than as straight XML. One other nice trick though – when the output is an XSLT template, the “omit-xml-declaration” flag can be used to strip off the XML declaration so we don’t have to do it through text manipulation, simplifying and accelerating our Javascript code. So we’re almost there with this capability. Some minor improvements and adding caching are all we need:
system.include("storexml.stub.js");
var cachePath = "nationalgeographic/cache/";
scrape_picture_page.operationName = "test_scrape_picture_page";
scrape_picture_page.inputTypes = {"url" : "xs:string?"};
scrape_picture_page.outputType = "xml";
function scrape_picture_page(url) {
if (url == null)
url = "http://photography.nationalgeographic.com/photography/photo-of-the-day";
var config =
<config>
<var-def name='response'>
<xslt>
<xml>
<html-to-xml>
<http method='get' url={url} />
</html-to-xml>
</xml>
<stylesheet>
...
</stylesheet>
</xslt>
</var-def>
</config>;
var scraper = new Scraper(config);
var result = new XML(scraper.response);
if (result.hasComplexContent()) {
var date = xsDate(new Date(result.date));
storexml.store(cachePath + date, result);
}
return result;
}
xsDate.visible = false;
function xsDate(d)
{
return d.getUTCFullYear() + "-" +
(d.getUTCMonth() < 9 ? "0": "" ) + (d.getUTCMonth() + 1) + "-" +
(d.getUTCDate() < 10 ? "0": "" ) + d.getUTCDate();
}
As an aside, this shows a couple of my wishes:
Finding a picture for a particular dateNow that we have a function that can scrape a page given a URL, and given that the data returned and cached by that function contains a link to the page for the previous day’s page, we can do some walking around in the cache to find data for a particular date. That’s what this function does. First, we look in the cache for a photo’s metadata. If it’s there, we can simply return it – we’re done. Otherwise we need to find the URL for the page representing that date and call the scrape_picture_page operation. If I can’t find the requested date in the cache, I look for the next earlier date, and so on, until I do find a photo in the cache (or I reach today’s date). That’s the first while loop. Then, using the <previous> page url, I work backward again, incidentally populating the cache as I go, until I’m back to the date I was looking for. The couple of “if” statements look for exceptional conditions: the first one handles the case where I’ve looked all the way forward till today but still haven’t found anything in the cache, and the second makes sure that if a page can’t be scraped for some reason that we give up and return what little we have before we dig ourselves any deeper. picture_for_date.inputTypes = {"date" : "xs:string"};
picture_for_date.outputType = "xml";
function picture_for_date(date) {
try {
return storexml.retrieve(cachePath + date);
} catch (e) {
print("failed to find cached photo for date " + date);
var photo;
var startDate = parseDate(date);
var today = new Date();
// work forwards in the cache until we find something (or hit today)
while (startDate <= today) {
try {
photo = storexml.retrieve(cachePath + xsDate(startDate));
break;
} catch (e) {
startDate.setUTCDate(startDate.getUTCDate() + 1);
}
}
// start with the most current thing in the cache (if any) an work
// backwards to the requested date, filling in the cache as we go...
var targetDate = parseDate(date);
while (startDate > targetDate) {
var previousPageUrl;
if (photo == null) previousPageUrl = null;
else previousPageUrl = photo.previous;
print("fetching photo for " + startDate);
photo = scrape_picture_page(previousPageUrl);
if (!photo.hasComplexContent())
break;
startDate.setUTCDate(startDate.getUTCDate() - 1);
}
return photo;
}
}
Generating the feedNow we have all the pieces in place to aggregate the data and generate a list of some kind as output. The picture_of_the_day operation does that for us. The function has some parameters controlling aspects of the feed – whether to link to the small, medium, large, or wide aspect ratio images, and how many items to include. If no number is specified, we generate a feed of the latest 30 photos – just long enough to enjoy the photo but not so long we get tired of it. The WSO2 Mashup Server has a Feed object to help construct feeds, but because I’m targeting this feed at the Google Photos Screensaver I need to include some feed extensions that aren’t supported in the 0.2 release (though they’ve just been added to the nightly build!). It’s not hard to create an RSS by hand though, so that’s what I chose to do. First I prepopulate the channel with title, links, and description, and then loop through the photos adding an item for each of them. The first time through the loop, I also add in a <pubDate> reflecting the date of today’s photo. Again, this isn’t rocket science – the hardest thing is simply to format the dates appropriately. During the loop I use Javascript Date objects to increment days and tick over at the end of the month. I convert that to an xs:date to access the cache, to an RSS Profile-conformant string for the <pubDate>, and to an xs:dateTime for use in the <atom:published/> element, which seems useful for the subscription page displayed in Internet Explorer 7. picture_of_the_day.inputTypes =
{"size" : "small | medium | large | wide", "numPhotos" : "number?"};
picture_of_the_day.outputType = "#raw";
function picture_of_the_day(size, numPhotos) {
if (numPhotos == null) numPhotos = 30;
var feed =
<rss version="2.0">
<channel>
<title>National Geographic Picture-of-the-day (from WSO2 Mashup Server)</title>
<link>http://mashups.wso2.org/services/nationalgeographic/¬
picture_of_the_day?size={size}</link>
<description>WSO2 Mashup Server mashup acquiring and caching links to the ¬
National Geographic Picture of the Day
(http://photography.nationalgeographic.com/photography/picture-of-the-day),
and exposing them as a feed. Sizes of "small", "medium", "large", and "wide"
are available. A max number of photos can be specified with the "numPhotos"
parameter.</description>
</channel>
</rss>;
var startDate = new Date();
var photo, photoDate, url, urlsmall, entry;
for (var i = 0; i < numPhotos; i++) {
photo = picture_for_date(xsDate(startDate));
if (photo.hasComplexContent()) {
url = photo.location.(@type == size).toString();
urlsmall = photo.location.(@type == 'small').toString();
photoDate = new Date(photo.date.toString());
if (i == 0) {
feed.channel.appendChild(<pubDate>{rssDate(photoDate)}</pubDate>);
}
entry = <item xmlns:media="http://search.yahoo.com/mrss/">
{photo.title}
<description>
<a href='{url}'><img src='{urlsmall}'/></a>
{photo.description.*.toXMLString()}
</description>
<pubDate>{rssDate(photoDate)}</pubDate>
<link>{photo.page.toString()}</link>
<guid isPermaLink='false'>{photo.page.toString()}</guid>
<media:content url={url} type="image/jpeg" />
<media:thumbnail url={urlsmall} />
<atom:published xmlns:atom="http://www.w3.org/2005/Atom"
>{xsDate(photoDate)}T00:00:00Z</atom:published>
</item>;
feed.channel.appendChild(entry);
}
startDate.setUTCDate(startDate.getUTCDate() - 1);
}
return feed;
}
You can access this operation through the try-it page at http://localhost:7762/services/jonathan/nationalgeographic?tryit and see that the operation returns a feed. However, the try-it uses SOAP by default under the covers, which isn’t terribly friendly to feed readers like the Google Photos Screensaver. No problem – the Mashup Server also exposes it’s operation through a REST interface. By accessing the URL http://localhost:7762/services/jonathan/nationalgeographic/photo_of_the_day?size=wide, you can see the feed directly in the browser, point the screen saver at it, subscribe to it, etc. By adjusting the “size” and “numPhotos” parameters you can generate variants of the feed that suit your purpose. Publishing the feedOnce I had the service written, tried it for a day or two to ensure it was stable (and fixed a couple of edge cases as a result), I used the administrative UI in the Mashup Server to publish it to http://mooshup.com, which hosts the service live on the internet for others to use. The publishing process is simple – click the share button, confirm that http://mooshup.com is the destination, and click OK. While we have lots to do to make this site an attractive and useful place for members of the mashup community to hang out, it does give me a stable internet URL for the feed (for example http://mooshup.com/services/jonathan/nationalgeographic/picture_of_the_day?size=wide) so others can enjoy it. You can exercise the try-it page live from there, look at the metadata, or download the service to your local installation of the Mashup Server and run it there. Last WordHopefully this helps you get a feel for the Mashup Server in action. We did some screen scraping, fairly sophisticated caching by invoking an external storexml Web service, formulated an RSS feed, and made it (and intermediate functions) available through a Web service including SOAP 1.2, SOAP 1.1, and HTTP bindings, including an HTTP GET binding amenable to RSS agents. Although we didn’t look at them in detail in this article, the Mashup Server generated a try-it page for debugging and exercising the service, WSDL, Schema, stubs for accessing the service simply from Javascript or E4X environments, even generated some human-readable documentation for the mashup. We ran the service locally, then published it live onto the internet. It also would not be hard to generate a custom HTML interface providing (for example) a slideshow of these photos, but in this case I wanted to show that user interfaces can go beyond just HTML pages by using Google Photos Screensaver as my ultimate user interface. So what’s next for this service? The main improvement I can think of is rewriting the code to use the Feed object when it becomes capable of handling the images. It took me a while to figure out which RSS extensions were necessary and it would be nice not to worry about the representation of dates. Maybe I could even offer an Atom feed in parallel. Another idea related to performance would be to experiment with a different, perhaps additional, caching strategy – which is to cache the entire feed to disk and periodically refresh it using the recurrence capabilities of the mashup server. But those are perhaps good topics for future articles! Until then, enjoy the great photos available from National Geographic! [Updated 6 Feb 2008 - added "jonathan" user to endpoint urls as required by the Mashup Server 1.0 release, and changed the online links to point to http://mooshup.com.] For an end-of-summer blast, an elite selection of Marsh family members returned to the Crooked Lakes Basin for an overnight backpacking trip. Photos are now online here, joining the set from our last visit a couple of years ago. The air was a little smoky from recent California wildfires, but it was fun to revisit a place and to dig up new subjects and try to do better on some shots I’d done before.
[Pictures here. Now geotagged.] We arose early on our final day - we needed to attain Shi Shi Beach and a water source there before we could rehydrate our breakfast. The tide was again low so we had little trouble navigating the final gateway to the smooth wide sands of Shi Shi Beach. As the driftwood at the high-water line was deep in shade at the forest’s edge, and would be for hours, we spread a small tarp on the damp flat sand, made stools of our bear canisters, and tucked into a random array of our remaining food. My policy is never to pack food for your final lunch - by then you’d rather just hold out for the first cheeseburger. I discovered that my camera had enough power to take a shot - but not to save it on the SD card - so I used the miniscule internal memory to take one final shot of Gen and Anna on the beach (sorry, marked friends/family only in flickr as are all my shots of friends/family - yes, you’re missing about 30 pictures out of the set if you’re not on my list.)
Along the length of Shi Shi we passed a number of tent encampments starting breakfast fires tucked up against the forest. Shi Shi wasn’t as deserted as the previous two days had been. Already we were returning to civilization. At the end of the beach one final steep climb up a cliff led to a couple of miles of forest trail, gradually straightening, widening and becoming less muddy, and eventually developing into a quite civilized set of boardwalks and bridges before depositing us in a car day-park a mile of paved road from our car. This trail took us from the wild where time is marked by stride after stride, by tides, shadows, sunsets, and the song of the soul, and returned us again to the precise increments of omnipresent second-counting LCDs, per-minute roaming charges, miles per hour, ferry schedules. We will learn to appreciate these things again, and ease gently into that world of objective time with a dinner reservation for a seafood extravaganza. But part of us will always remain in that narrow strip of land between the flat sea and the towering stone along this extraordinary piece of wilderness.
|
|
|
|
|
It is apparently a custom for new visitors, to weave a thread into the web. All of us newbies partake in this custom (photo courtesy of Udeshika). This is a good spot for lunch and we break out out lunch packets - rice, curry, cal, sambol, all wrapped in plastic and then newspaper - a traditional lunch favorite here.
And as all Sri Lankan adventures ended, so did this. Hours of the most jolting, harrowing driving on narrow, steep, windy roads, leading us inexorably back to Colombo, a shower, dinner, and late that night to the airport, and eventually home.
Posts (RSS)