Mar 6, 2012

Osprey

[Disclosure: I work for Twitter as a software engineer.]

In 2006 and 2007, I really wanted to go to SXSW but was still in college so it wasn’t very feasible. You’ll recall that 2007 was the year that Twitter exploded at SXSW and the service has grown incredibly since. Between then and now, I’ve watched many friends and colleagues attend — and heard of their tales through 140 characters. The panels that were excellent, the ones that bombed, the parties that were unbelievable, the conversations that were unbeatable, etc. Experiencing the event second-hand became a common frustration of Twitter users—myself included—especially as hashtags and retweets gathered momentum as a way to post live updates, and share others’ thoughts. Caterina Fake coined the term FOMO — the fear of missing out — due to events like this.

This will be my first year at SXSW — and my first time speaking at it. As our panel has prepared for the event, we wanted a way for the community to interact and contribute to our discussion. On a call, we had the idea to create a way for people to submit thoughts and questions to us during the panel in real-time, like Google Moderator, but built on Twitter so anyone could participate. My gut reaction here made my stomach churn a little — if we used the existing mechanisms of Twitter, retweets and hashtags would be the customary signals available for us to monitor through the API. I didn’t want to use those though, and impose our discussion onto those not at the event — often times, the internet needs to be quieter.

There are other mechanisms available through Twitter we could use, but we’d have to build something on top of the service to fully utilize them. My two favorite signal mechanisms are replies and favorites, because they are inherently quiet. Replies are scoped to only those who follow both accounts, and favorites are nearly silent — but we can monitor both through the API, with a little work. This is how Osprey was born.

Implemented for our panel at http://designfromthegut.com, Osprey is an application built on top of Twitter that allows people to submit questions via @-replies, and vote up submissions they like via favorites. We’re opening it up in advance of the panel so those interested in the topic can submit comments, questions, or thoughts they’d like to see answered during the panel. We’ll start it fresh again on Friday before the session for real-time questions.

The flow looks like this: you have a question or comment so you post a new tweet to our panel’s account, @dftg and the application listens for these @-replies via the User Streams API. When a reply to this account comes through the stream, the application parses the text of the tweet to remove ‘@dftg’ at the beginning, and reposts it as a new tweet on behalf of @dftg, with attribution. If I posted:

@dftg What are your thoughts on Apple’s approach to design?

then the application would post a new tweet that looks like:

From @couch: What are your thoughts on Apple’s approach to design?

The application behaves this way for a few reasons, foremost for counting favorites. Twitter does not currently offer a public favorites count API, so I can’t query how many times a tweet has been favorited. The only way to get this count currently is to monitor favorite events that come through on a User Stream. Additionally, favorite events are only pushed for tweets originating from the account established with the User Stream. This means I could not have the application simply retweet the user’s tweet natively, it must to be reposted to the account.

While seemingly more complex, there are several niceties about this setup. When reposting the tweet, I pass the original tweet’s ID along so the new tweet is effectively formed as a reply, binding the two entities through metadata. Additionally, as a participant, one can browse through @dftg's timeline to see all of the questions submitted, the original author (and their original post), and then easily favorite any of them in a client of their choosing. If others want to follow along outside of the event, they can opt-in by following @dftg.

As users mark tweets they like as favorites, Osprey stores in a database the tweets that have been favorited in the original data structure provided through the API, but additionally adds a user array that stores some account data so the application can keep track of who and how many favorited the tweet. If a user removes one of the tweets from their favorites, they’re also removed from the user array. After enough users browse through the submissions, the favorite counts will populate and the application’s interface will bubble up the top audience picks, where our moderator can make a choice about which to direct to the panel. The application additionally recognizes favorite events from our moderator’s account as a sort of “moderator’s pick” to elevate that choice in the interface as well.

When tackling how to build this application, I thought this would be a good opportunity to experiment with node.js and MongoDB as I hadn’t really used either. The application doesn’t have high-stakes — if problems arise during the panel or time to develop wasn’t on my side, there is always a microphone. But we’re in a decently-sized venue at the conference and we though it could be a bit of fun to have a real-time discussion.

So the nitty-gritty: on the backend, the application uses node.js with Matador for some clean MVC architecture and mongoose to interact with MongoDB to store the favorited tweets. ntwitter is used as a simple abstraction of the Twitter API for node. To enable near real-time updates of new submissions and favorites in the interface, the application also employs WebSockets via socket.io.

On the front end, the application uses Bootstrap for the scaffolding and basic responsive layouts, underscore.js for data manipulation, twitter-text to process the text in tweets, hogan.js for template rendering, moment.js for formatting and localizing timestamp strings, and Twitter’s Web Intents for performing actions on Twitter. There’s even a little bit of intelligence for reloading content—should WebSockets fail if the site’s in the background—by way of the new Page Visiblity API proposed by the W3C.

When it came time to deploy, I considered three different node hosting providers: Joyent’s no.de, heroku’s node service, and nodejitsu. I ended up using nodejitsu, who I’ve found to be really fantastic. Due to the use of these open-source libraries and frameworks, many of which are rather new, the application had a considerable number of requirements for its hosting provider: support for node 0.6.0 or greater (for Matador), WebSocket support, and custom domain mapping. Joyent’s no.de doesn’t support custom domains so they were out of the running. Heroku looked promising, and after successfully deploying the application with a custom buidpack to enable node 0.6.0 instead of their Cedar stack’s 0.4.7 default, all looked okay — until I realized they don’t support native WebSockets. I then chose nodejitsu: they supported all of the application’s requirements, include database creation through MongoHQ, and were very responsive and helpful on their IRC channel.

I’m excited to see the app in action, but I was also walking through a dark tunnel with only a flashlight when building it so pardon if it hiccups or if you see some of its wiring hanging around. That’s always the conundrum of development though: if you’re not embarrassed by your first version, you’ve launched too late.

Notes:

About
Ephemera, thoughts, and curiosities from Bill Couch (@couch).
Ask a question, view the archive, or subscribe via RSS.