Coder, Gamer, Runner, Husband.
During the last few weeks, I’ve built a couple of pipes I want to share. (“Pipe”, in this context, means an application built in Yahoo! Pipes.)
Here’s one of them.
As I’ve mentioned once or twice in the past, I’ve become a more or less avid runner during the last year. One central piece of my equipment is my Nike+. I don’t run without it. I am a geek, I love numbers. The Nike+ provides me with numbers. It makes running a game.
I sync my iPod with my iTunes after each run, and the Nike site is taking the raw data, crunching it, giving me graphs and more numbers, and (this is the interesting thing) badges for my website.
Now, I don’t really care about those. But when there are badges, there must be an API which provides the raw data to them. Looking behind the scenes, I quickly found it. It’s not password-protected or secured in any way; when you set your nikeplus.nike.com profile to “public”, the API will return some of your data (run overviews, run details etc. — no personal details).
So, knowing the API URL, I’ve built a pipe which will do a few things:
When writing the pipe, I’ve made a few assumptions, namely that…
Since I am lazy, I’ll only use the data for the most recent run, so the pipe’s results will be exactly that — just a single item, your most recent run.
That being said, I found this sufficient. After adding the pipe’s RSS URL as new “Custom RSS”-type service to Friendfeed, FF will effectively trigger the pipe a few times each day, and your latest run will be added to your stream quickly. Next time you sync your iPod, the Nike site will pick up the new data, the API will return the new data to the pipe, and the new run will be added as new Friendfeed item.
The pipe will post your runs in the following format: “[Nike+ runner name] ran [distance] [km/mi] in [time]”, for example “3R ran 11.3543 km in 1:03’54””. The message will link to the public page of the run — a page like this. (3R is my Nike+ moniker.)
Now, why would I want to add my runs to Friendfeed? Well, why not? For me, running is a nice part of my life. I’m actually enjoying it, I’m keeping it casual, and I am proud of every damn kilometer mark I pass. Plus, as mentioned, I am a geek, and I like to share what I build. :)
If you have questions or suggestions, speak your mind in the comments.
During the last few weeks, I’ve built a couple of pipes I want to share. (“Pipe”, in this context, means an application built in Yahoo! Pipes.)
Here’s one of them.
A few weeks ago, I’ve first tested and then bought the excellent CrossOver Games. It’s an emulator (basically a highly specialized version of WINE) which allows me to play a slate of Windows games, old and new, under OSX. So, that’s how I’ve spent big chunks of my spare time during the last few weeks: playing through the wonderful Portal and the great Half-Life 2 games. (On a related note, I’ve noticed the World of Goo demo is running flawlessly in CXG. Awesome!)
I’ve got the games via Steam, and was both delighted and highly annoyed to learn that newer Steam games offer achievements. You see, I am a sucker for achievements. I love them, even though they aren’t good for anything. I usually spend too much time trying to get this or that achievement. These meaningless little pixel badges are “awarded” for different things you manage to do in different games. You can get achievements in various games, on various platforms. For example on Xbox Live, or, as mentioned, on Steam.
So, being a male gamer in his mid-30s, I naturally like to use these superfluous thingies to brag about my mediocre gaming skills. Meaning, I want them to show up on my Friendfeed profile.
Thus, I wrote a pipe which grabs the achievements from any (public) Steam ID page (here’s mine), spitting them out in an usable format — in Friendfeed’s case, that’d be RSS. (Pipes also returns the data as JSON if you want, or even as a handy HTML badge you can put on your blog or whereever.)
I’ve then added the RSS URL of the finished pipe as new service (type: “Blog”1) to Friendfeed.
So, that’s all there is to it. Maybe I am the only one caring about this type of thing, maybe not. If you have questions or suggestions, sound off in the comments. :)
Friendfeed’s terminology is a bit misleading here… If you want to add an RSS feed, you’ll have to use “Blog” as new service. Eh. ↩
A couple of weeks ago (mid-June) I’ve participated in our internal Yahoo! EU Hack Day. This time around I’ve decided to tinker with Yahoo! SearchMonkey by building a couple of modules (dubbed SearchMonkey apps) during the 24 hours available.
In case you have never heard about SearchMonkey: it’s a technology built into Yahoo! Search that allows developers to present their own or other peoples’ sites in a different way (examples to follow). These search result augmentations are dubbed “SearchMonkey apps”.
The Search users then have the option to use the SearchMonkey apps they like, which enhances their search results by presenting them in a different way; review sites for example might present their articles in a way that includes the rating or verdict of the reviewed item. There is a good number of different apps available at the SearchMonkey Apps Gallery.
Using SearchMonkey doesn’t influence the ranking or the order of the results; it displays the results in the same order, just differently.
Anyways. I ended up writing 6 apps, and most of them I’d like to share. For more information about a particular app, just click the links. They’ll take you to the related Gallery listing, where you’ll have the option to enable the app for your searches.
Whenever a Twitter profile turns up in your search, you’ll get a bit more information about the person—number of followers, number of people followed, name, bio, location, last tweet. like this:
By the way, it doesn’t query the Twitter API; instead it just grabs the HTML page and extracts the publicly available information.
Once you’ve visited the Gallery page and installed it, try a Yahoo! UK example query to see it in action.
(On a related note, there’s apparently another Twitter SearchMonkey app, but I couldn’t find the installation link to give it a try. Bart, the developer, does things in a more complex and probably more fail-safe way; in my defense, I wanted to keep it simple.)
If a Wikipedia article is found, the app will show you its first paragraph and, if available, a thumbnail of the first image. Also the category:
Once you’ve visited the Gallery page and installed it, try a Yahoo! UK example query to see it in action.
Xing is a more Europe-centric LinkedIn contender. It’s big in Germany. If a Xing person is found, the app shows the mugshot, the location, the title and the industry. Unfortunately Xing doesn’t reveal the company on public profiles unless you’re logged in, which my SM app clearly is not. Still:
Once you’ve visited the Gallery page and installed it, try a Yahoo! UK example query to see it in action.
In a desperate attempt to suck up to Anil Dash (I kid) I’ve hacked together a module to show the profile of any Vox blogger when a related page is found:
Once you’ve visited the Gallery page and installed it, try a Yahoo! UK example query to see it in action.
Using cutting edge high tech web 3.0 algorithms straight out of Munich, Germany, this module is able to extract the gist of any given Techcrunch or Valleywag article.
I don’t like these sites very much. At least their Yahoo! coverage (which I am interested in, given my background) is usually hysterical and often times sensationalist. DO. NOT. WANT.
Once you’ve visited the Gallery page and installed it, try a Yahoo! UK example query to see it in action.
Well, that’s it. Maybe you like those, maybe not. :) If you’re interested in writing your own apps for Yahoo! SearchMonkey now, take a look at the official Manual for SearchMonkey Developers and Publishers! And if you’re now wondering what other applications might be out there, the Gallery is a good place to start.
UPDATE/DISCLAIMER: Even though I work for Yahoo! during the day, these are NOT official SearchMonkey apps by Yahoo!. You’re looking at personal stuff. Again, this post, as everything on this site and under this domain, is NOT YAHOO! BUSINESS. Thank you.
Over the last few days I’ve pursued my idea to build a “Twitter spammer detector” of sorts using only client-side technologies. I didn’t feel like setting up server components at all.
What was it supposed to do? It should check the number of followers of everyone on your contact list, the number of people they are following, and the ratio between those. If the person is following more than 1000 people (can be customised), and has a Following-to-Followers ratio higher than 1:1 (can be customised), you’d be informed (by a handy “Block” link).
Thanks to Pipes, the YUI libraries and the Twitter API returning JSON, I was able to finish the first version of Twitter Twerp Scan in a relatively short period of time.
My Javascript-fu is still a bit weak, but getting better. It’s a nice change to only have one local file to work with (excluding CSS, of course).
Over the weekend, the traffic to that page increased quite a bit, mostly because several people liked the idea enough to link the site.
I’ve set up a Twitter account for service updates etc. — twerpscan. The rest of my tweets will go to Carlo as usual.
It’s amazing what 30 minutes of frantic walking can do for your sanity. Tonight, when getting out of the subway and noticing my bus had just left, I’ve popped in a 131bpm Podrunner mix and started walking. A good 30 minutes later I found myself on my doorstep with a clear head.
Check out Podrunner, a free weekly podcast of one hour gapless music mixes for workouts and the likes. The only trick is to find a bpm range that suits you. Then put it on your iPod.
It’s simply that good.
Nerdy pro tip: once you’ve found your bpm range, build a Pipe like I did and subscribe to this feed instead of the original one to only get podcasts which are “suited” for you. Oiy!
I’m done with the redesign. Both my tumblelog and this blog now share the same layout and (the most important parts of the) sidebar, adding some much needed consistency.
Also I’ve got rid of the Google Coop search and put the Technorati search to use instead, it’s good enough in my eyes.
The biggest change in functionality is the switch from the built-in Wordpress comments system to Disqus, on both my blog and tumblelog. Naturally, I’ve added a tiny bit of logic to the Wordpress templates so the existing comments are preserved, displayed, and those entries are still accepting new comments (assuming the Comments Timeout plugin allows it). But for new entries, Disqus is used. I figured if I don’t like it, I can always switch back and cram the exported Disqus comments back into WP.
Yeah.
Now if I could use Markdown for new posts instead of Textile, I’d be a happy camper. The WP Text Control plugin is no help, unless you consider making the server drop the connection every time helping. I don’t.
Repeated tips of the hat to Bill Israel for sharing his wonderful Letterhead Tumblr theme with the crowds (CC licensing FTW!). Due to its beautifully simple semantic structure I was able to transform it into a Wordpress theme in about two hours. Thanks, Bill!
In somewhat related news: the Hendromat has launched a new tumbelog last night—Make Games, Not Love, MGNL for short. Go check it out.
Over the last week I’ve been using RescueTime. It’s a neat little service that’s keeping track of what you are doing all day long on your machine (Mac or PC) by logging which applications are having the focus. On the site you can then tag the different applications with different keywords (for example, my Mail.app times are tagged with “work”, “communications” and “email”).
And, to make it actually useful, you can analyse your time spent, graphs and all! Check the product tour on RescueTime.com for more infos and some screenshots. It’s pretty neat. (Yes, privacy concerns, blah blah blah. It’s not the topic of this article, so please keep it to yourself for the moment. Thanks. :])
So far so good.
My problem, tho, is that I spend quite a bit of time both on the phone and in meetings. Naturally, RT won’t track these times, since these are basically off-screen activities. Uncool.
So after digging around in the RT log files I was amazed to learn they’re just YAML files. Eeeeexcellent. I can work with that!
But I wanted something with a GUI, something a bit “cooler” than just a Ruby script. So I took this as opportunity to get a bit into Applescript. Unfortunately, RescueTime’s Mac doodad doesn’t offer an Applescript API, and I had to improvise.
After two hours of playing around and cursing at Applescript’s syntax (they’re all on crack…), I had something sufficiently cool to show for.
So, what is it? It’s a little application that you run everytime you want to log some off-screen time. Basically, it’s just a dialog that allows you to enter a number (i.e. minutes spent), and has some buttons to either log these minutes as “was on the phone” or “was in a meeting”. If you hit “Cancel”, it won’t do anything. (Surprise.)
Let’s say I’ve just spent 15 minutes on the phone, I’ll run the application, enter “15” and hit the Return key—and the last 15 minutes will be logged as phone call in RT. The app will quit once that’s done. Another meeting, another phone call? Rinse, repeat.
Data-wise, it’ll then create a new file in the right format in RescueTime’s log directory (~/Library/RescueTime/Logs/Pending/). On the next data upload RT will pick up the new file, and it’ll appear in your stats as “manual_phonecall”. Your meetings will appear as “manual_meeting”.
Please note: Neither existing RescueTime files nor the RescueTime application will be touched or tinkered with.
Below you’ll find two downloads. The first one is the actual script in plain text. The second is the script compiled as application bundle. If you’re unsure which to pick, you’ll most likely want the latter. ;)
Have fun.
DISCLAIMER: I’ll take no responsibility for loss of data, hair or life. I am very, very sure there won’t be any losses, but still. It works for me, and I’m sharing. Use at your own risk.
A few months ago I was talking to Hendrik about lifestreams, and in my ongoing struggle for his undying love (as a friend), I’ve whipped up a little somethingsomething using the wonderful Yahoo! Pipes.
“Lifestream?”, you ask. “What the deuce is a lifestream?!” A good question. A lifestream is basically a big bucket where all the updates and update notifications from your blog, your ADD-induced Twitter posts, your Flickr uploads etc come together in one concise way so it’s easier for others to ignore them. ;) Also, you only have one URL to hand out to hot women (or men) in pubs because the stream inadvertedly works as a hub page, too!
Now, while the prototype was quickly hacked together, it felt clunky. Sure, you can pass a dozen URLs or so to the pipe, but what if the URLs would change? Or if you wanted to add a new one or delete one from the list? Then it’d be a lot of tinkering with the script call in your HTML code.
So I had the idea to build a site where you could configure the list of URLs and the layout and everything, and which would give you a HTML badge for your blog or site, a snippet that wouldn’t change.
And today, I proudly open escaloop to the public.
I feel it’s good enough to test the waters, I believe. I’ve played around with it, fiddling with different implementations on different types of pages, and it looks okay. I guess I could try to think of every possibility for every site on earth, but we know how that would turn out.
It certainly looks fine on my own blog. (Considering that I don’t have that many active feeds, that is.)
So with that, I’ll release escaloop into the wild. It’s still a bit rough around the edges, and there might be bugs. That said: Please take a look, play around with it, build yourself a badge or two for your site, blog, MySpace page, whatever. If you have feedback, please let me know in the escaloop Google Group.
Have fun, Carlo
PS: You might ask what took me so long. In my defense, I am a lazy bastard. Also, I wrote the first rough draft in Python, then switched to Ruby. There I’ve wrote the first prototype using the Sinatra DSL, and finally settled to make use of nifty lightweight Ramaze framework. (By the way, Sinatra is nice, but not what I was looking for.)
During the last few months I’ve also had to put a lot of time into Mass Effect, which is a great game. You understand.
So, (almost) everyone is totally crazy for lifestreams these days. In case you managed to get around the whole issue so far: a lifestream is basically a big bucket (i.e. web page) where all the updates and update notifications from your blog, your ADD-induced Twitter posts, your Flickr uploads etc come together in one concise way so it’s easier for others to ignore them. Also, you only have one URL to hand out to hot women in pubs because the stream inadvertedly works as a hub page, too!
Joy.
Anyways, I was talking with Hendrik about the various stream services out there, and I figured it’d be fairly easy to build something server-less using a wee bit of Javascript and Yahoo! Pipes. So I’ve spent around an hour to do just that.
The basic idea is that you make a pipe containing all your feeds, and then access the output using Javascript from your page. Add some styling, and voilá.
It’s a bit rough around the edges, but hey, it’s just a prototype. It’s not life- or game-changing in any way, but I’d like to share anyways.
Update 2007-11-21: It’s mostly broken right now since I am playing around with something. Not to worry.
Over the weekend I finally had some time to continue playing around with Yahoo! Pipes. (Turns out that quitting World of Warcraft makes your days longer. Huh. Who’d have thought…) It really is a neat toy/tool.
Alas, it is not without hitches. If you’ve worked with Pipes before, you know that it has a module named ‘Fetch Data’, but unfortunately this module is only able to deal with XML and JSON data. It’s just that it will outright reject anything else. And most HTML pages fall in this category.
Here’s my story on how I managed to make Pipes process run-of-the-mill HTML pages. I couldn’t find anything on this topic on the net, so even though I don’t think I am the first guy to tackle this problem, I want to share what I found out.
A little background. I wanted to build a comic feed for Explosm.net’s Cyanide & Happiness, a comic strip for the …wicked. C&H has an official RSS feed, but it only carries links to the new comic pages. Which is fine and dandy for most people, but I am a busy guy, you know, what with my hectic web-3.1 lifestyle and all, and I really can’t afford to even lose 20 seconds by clicking through to anywhere. (Read: I’m really lazy.)
So I tried to make Pipes to parse the Explosm.net comic pages and produce a Pipes-powered feed that not only had the link, but also the particular comic image for that day.
I started out with the Fetch Feed module reading the Explosm.net RSS feed. I am usually not interested in anything but the comics, so in a second step, I made use of the Filter module to ditch feed items that doesn’t contain the word “comic”.
So that left me with a list of feed items, each with a link, a description, a title and some other metadata.
That’s where the trouble started. Boy, what was I thinking?! The HTML pages are far from perfect, they sure as hell doesn’t validate as XML. Then again — that’s alright! The site wasn’t made for scraping or parsing, it was made to display -tasteful- weird comic strips, and that’s enough. I like it for that. It’s just that there is no Pipes module to grab a glob of textual data and work with it. (Yet? Please, someone say “Yet”, employing a knowing undertone.)
After a while of trying to beat any of the available ‘Fetch *’ modules into submission, I was close to giving up. But then I remember I had come across a HTML Tidy service a few months or years ago. After a few minutes, I had rediscovered it: W3C’s own Tidy service. It’ll digest any HTML page (you just pass the address), and it will clean it up and return valid a XHTML document, which also constitutes as XML. Huzzah!
Well then, commence plumbing! The third module I used was the wonderful Loop module. It’s special because it will process every item in a given dataset, in this case it’d iterate over all remaining feed items. I’ve told the Loop module to use the String Builder module, there I concatenated the Tidy URL, http://cgi.w3.org/cgi-bin/tidy?forceXML=on&docAddr=, with the item.link attribute, and assigned the resulting string to a new a attribute, item.link_tidy.
Step four consisted of another Loop, in which I would use a Fetch Data module to grab the data from the Tidy’d Explosm.net pages, which had now been transformed into neat and valid XML structures and therefore could be digested by the module. It took me a few minutes to find the exact location of the image element (body.div.table.tr.td.div.0.div.1.table.tr.td.1.div.font.0.content), which I then stored in a new attribute named item.content.
(While we’re at it, in the future I would very much like to have the option to use XPath addressing in ‘Fetch Data’… or any related module, for that matter. Thanks in advance. :) )
Anyways, the next steps are using yet another Loop to concatenate the image HTML with the original item.description, and then another one to rewrite the original image URLs a bit so the images are served through the free and excellent Coral Distribution Network.
(Technically, I might not be stealing from Explosm.net, especially since they allow hotlinking of their comic strips from message boards and pretty much anywhere, but for the sake of this example and story here, and the assumption that a few people might end up cloning the Pipe — yes, I am a dreamer —, I feel a bit better knowing I leave less of a footprint on their image servers.)
Then it was applying a Reverse module to flip the order of the items (a cosmetical issue), and that was it.
So, to sum up this long and rather tedious story: you can process HTML pages with Yahoo! Pipes. It’s just not as straightforward as you would expect.
Well, and if you want to take a look at it: here’s the finished Pipe — it might not look like much since the HTML preview doesn’t show the images, but the RSS output is complete.
Tip o’ the hat to Cyanide & Happiness for mindblowing hijinx, the W3C for the free & open Tidy service, and the CDN for being good people.
Now go and use Yahoo! Pipes. It’s one of the cooler products we (as a company) have released in the last few years, and quite frankly, it deserves praise.
So there.
Quick links: Finished Pipe, RSS Output
Update, 2007-09-19:* I’ve slightly adjusted the element addressing in the Pipe to waterproof it a bit. The general idea and order of the modules didn’t change, tho.
Update, 2007-12-07: My technique has been superceded by the offical new Fetch Page module — enjoy! :)
