Joseph Smarr

Thoughts on web development, tech, and life.

Home Sous Vide: What the Books Don’t Tell You

After years of passive enchantment with sous-vide cooking, my wife bought me an Anova immersion circulator for my birthday so I could finally try it at home. Despite reading the book everyone recommends, I was still left with a bunch of newbie questions that didn’t have immediately obvious answers. I forged ahead with a mix of phoning a friend, asking the Internet, and just giving it my best shot, reciting the mantra that worst case the food will just be inedible. In the end, things worked out fine–though there’s much more experimenting to be done!

For the benefit of other first-timers, here’s some collected Q&A based on my experiences so far. I hope this will help remove a roadblock or two and help convince you to “take the plunge” (see what I did there?).

Q: How should I prepare the water bath? How warm should it be?

Turns out it doesn’t really matter, since the circulator can heat it up pretty quickly on its own. I just filled the pot with warm water from the sink, which turned out to be about 110°F. In my case, I was cooking at 142° and the circulator brought it up within several minutes. No need to heat the water on the stove or use a kettle first (remember, you’re almost always cooking at temperatures well below boiling).

Q: How high should I fill the pot with water?

Technically, it just needs to comfortably cover the food you’re cooking, but it’s nice to give it some buffer. The Anova has a min and max line and you need the water to be in between. One thing I hadn’t thought of is how much water can evaporate over a long cooking time (more on this below), so that’s another reason not to cut it too close. The circulator moves the water pretty gently, so there’s no danger of it sloshing over the top of the pot or bubbling over like with boiling water.

Q: If my meat is frozen, do I need to fully thaw it before starting the sous vide process?

Not if you’re doing a longer cook. For instance, I did 24-hour beef short ribs at 142°, and even though the cubes of meat were 2-3″ thick, there’s plenty of heat and time to fully thaw the meat and bring it fully up to temperature. That’s one of the beautiful things about sous vide–since it’s low and slow, you don’t have the same worries about unevenly heating the meat like you would on a grill. And most meat defrosting techniques involve placing it in warm water, which is exactly what you’re doing. I just pre-defrosted the meat in warm water for ~20 mins and then put it in the bags and had no trouble. This is handy since otherwise it can take a long time to defrost meat, which means you need to plan days ahead to cook it.

Q: Many books/sites say you can use Ziploc freezer bags to seal the food. What about Ziploc storage bags?

I had a minor panic attack when I realized that all of our gallon-sized Ziploc bags at home were “storage” not “freezer”. I obviously didn’t want to risk melting the plastic or rupturing the bag during cooking. But according to the Internet, both are plenty well qualified for the task, being made of heat resistant food-grade polyethylene. If the food isn’t sharp and pointy, you’re not going to poke a hole in the bag during cooking, and the water is not going to be anywhere near the melting point of the plastic. In the end, I decided to brave the storage bag and it seemed to work just fine. I’d probably still prefer a freezer bag if I had one handy (just to be paranoid), but I don’t think it’s required. Just remember in either case to get all the air out of the bag using the water displacement method, which you do right in the sous vide pot.

Q: How do you keep the bag from getting sucked into the circulator during cooking?

When I first started the cooking process, I just naively put my ziploc bag into the pot with the circulator clamped onto the side. Nothing in the book or the instruction manual said to do otherwise! But inevitably over time the circulator would suck the bag towards it, ultimately impeding the water circulation and making some weird noises. Definitely didn’t seem like it was working as intended. Turns out most people clip the bag to the side of the pot with a chip clip or clothespin. That fixed the problem. I have no idea why this isn’t specifically mentioned in any of the basic instructional material!



Q: Why does the circulator sometimes make a metallic grinding noise?

The instruction booklet said it means the case isn’t tight enough, but I couldn’t obviously tighten it or loosen it. It definitely sounded wrong though, so I tried a few more times and eventually got the right amount of twisting and pulling to remove the case and then reattach it with a satisfying snap at the end, after which point the sound went away. It didn’t seem to affect the cooking process to have that noise, but it was definitely somewhat worrying and annoying, so I’m glad I finally fixed it.

Q: If you set the cooking timer, will it turn the circulator off when the timer finishes?

No. The Anova timer is just FYI and does not control anything. It’s confusing that they don’t tell you that, but it means you don’t need to worry about the exact cook time, which is part of the point of sous vide (it’s hard to overdo it, since the water is the same temperature you want the food to end up at, vs. an oven or grill, which is much hotter). For instance, I’d planned for my short ribs to be done around the time I would get home from work, but my schedule is not always super predictable, and I didn’t want the circulator to shut off and let the water cool down if I came home later than expected. Fear not, it’ll just keep going until you stop it.

Q: Is it safe to cover the entire top of the pot with cling wrap to reduce water loss from evaporation?

A recipe I read suggested that I do this, but I wasn’t sure if that would cause lack of airflow or pressure to build up or something. Fear not, it didn’t seem to cause any problems, and it significantly reduced water loss from evaporation, which is especially important for longer cooking times. I let it cook over night uncovered and it lost an inch or two of water in the process, so if I’d let it continue all day, it might well have gotten down below the min line by dinner time. The only hard part was keeping the cling wrap from sagging down into the water (don’t want to cook it!), but by using some longer pieces, pulling them tight, and wrapping the extra around the pot handles and edge, leaving room for the circulator on one side and the clip on the other side, it was stable and effective. Definitely recommended.

Q: What happens if the power goes out in the middle of cooking, esp. while you’re away at work?

You’re in trouble. :) This happened to me, and even though it was only a brief outage, the Anova frustratingly does not restart itself after waking back up! So the water just slowly cools, leaving your meat under-cooked and potentially in the danger zone for bacterial growth. This is what I came home to, and at first I thought the meal was ruined. But after some research and talking to some scientists in the family, I felt a bit better. It usually takes at least four hours in the danger zone for bacteria to grow, and the meat had been cooking above that zone for many hours already, so most bacteria should have been killed by then. When the power went out, the water didn’t immediately cool, so it only spent some of the post-outage time in the danger zone. And most preparations call for searing the outside of the meat before serving, which re-kills anything that might have started growing on the surface. In the end, after much soul searching, I decided to crank up the circulator for a bit and then pull out the meat and sear it. The meat was not as tender as I was hoping (possibly because the cooking time had been cut short) but it was definitely edible and, as I write this the next morning, I have no indications that my stomach is any worse for wear. The advice from the Anova folks seems to be to buy a UPS battery backup and plug the circulator into that.

My first home sous vide: 24-hour beef short ribs

Q: Can you reuse the cooking water for a subsequent sous vide run?

The consensus on the Internet seems to be yes, provided the water is not obviously funky or contaminated. It’s possible some bugs could grow in it between cooking cycles, but you’re keeping the food sealed in plastic and turning the water back up to a temperature that will kill anything in there, so you should be fine. Hey, it’s drought-tolerant cooking! After my scare with the power outage, I wanted to “get back on the horse” and do another run for the next dinner. I could have poured out all the water and started over, but it seemed wasteful, provided I wasn’t risking my health. I’ll post an update if this turns out to be a mistake, but I don’t think it will. :)


The promise of sous vide is “set it and forget it”, which as an engineer I love. It’s precision cooking that’s not fussy on technique. And it can produce some pretty stunning, unique results. But the danger is that by cooking food at lower heat for longer time, you’re at greater risk of growing bacteria if you’re not careful. So most of the first-timer questions above ultimately boiled down to how to minimize that risk without giving up entirely. That plus stuff that I’m sure you get used to after you’ve been doing it for a while, but that’s not obvious the first go around, and that the books don’t tell you about one way or the other.  So I hope this addendum finds and helps someone. In the meantime, I’ll keep at it. And if you’re a home sous vide cook and have any tips to share, or if you found any of this useful (or confusing, or provocative), please let me know!

Dreaming big


I’ve recently been working on a side project with my good friend Chris Lamb to scale up Google’s Deep Dream neural net visualization code to operate on giant (multi-hundred megapixel) images without crashing or taking an eternity. We recently got it working, and our digital artist friend (and fellow Plaxo alum!) Dan Ambrosi has since created some stunning work that’s honestly exceeded all of our expectations going in. I thought it would be useful to summarize why we did this and how we managed to get it working.

Even if you don’t care about the technical bits, I hope you’ll enjoy the fruits of our labor. :)

The ‘danorama’ back story

gridDan’s been experimenting for the past several years with computational techniques to create giant 2D-stitched HDR panoramas that, in his words, “better convey the feeling of a place and the way we really see it.” He collects a cubic array of high-resolution photos (multiple views wide by multiple views high by several exposures deep). He then uses three different software packages to produce a single seamless monster image (typically 15-25k pixels wide): Photomatix to blend the multiple exposures, PTGui to stitch together the individual views, and Photoshop to crop and sweeten the final image. The results are (IMO) quite compelling, especially when viewed backlit and “life size” at scales of 8’ wide and beyond (as you can do e.g. in the lobby of the Coastside Adult Day Health Center in Half Moon Bay, CA).

“I’d like to pick your brain about a little something…”

Dan AmbrosiWhen Google first released its deep dream software and corresponding sample images, everyone went crazy. Mostly, the articles focused on how trippy (and often disturbing) the images it produced were, but Dan saw an opportunity to use it as a fourth tool in his existing computational pipeline–one that could potentially create captivating impressionistic details when viewed up close without distorting the overall gestalt of the landscape when viewed at a distance. After trying and failing to use the code (or any of the DIY sites set up to run the code on uploaded images) on his giant panoramas (each image usually around 250MB), he pinged me to ask if I might be able to get it working.

I had no particular familiarity with this code or scaling up graphics code in general, but it sounded like an interesting challenge, and when I asked around inside Google, people on the brain team suggested that, in theory, it should be possible. I asked Chris if he was interested in tackling this challenge with me (both because we’d been looking for a side project to hack on together and because of his expertise in CUDA, which the open source code could take advantage of to run the neural nets on NVIDIA GPUs), and we decided to give it a shot. We picked AWS EC2 as the target platform since it was an easy and inexpensive way to get a linux box with GPUs (sadly, no such instance types are currently offered by Google Compute Engine) that we could hand off to Dan if/when we got it working. Dan provided us with a sample giant panorama image, and off we set.

“We’re gonna need a bigger boat…”

18-KolobCanyon_megoHDR60-1600Sure enough, while we could successfully dream on small images, as soon as we tried anything big, lots of bad things started happening. First, the image was too large to fit in the GPU’s directly attached memory, so it crashed. The neural nets are also trained to work on fixed-size 224×224 pixel images, so they had to downscale the images to fit, resulting in loss of detail. The solution to both problems (as suggested to me by the deep dream authors) was to iteratively select small sub-tiles of the image and dream on them separately before merging them back into the final image. By randomly picking the tile offsets each time and iterating for long enough, the whole image gets affected without obvious seams, yet each individual dreaming run is manageable.

Once we got that working, we thought we were home free, but we still couldn’t use the full size panoramas. The GPUs were fine now, but the computer itself would run out of RAM and crash. We thought this was odd since, as mentioned above, even the largest images were only around 250MB. But of course that’s compressed JPEG, and the standard Python Imaging Library (PIL) that’s used in this code first inflates the image into an uncompressed 2D array where each pixel is represented by 3×32 bits (one per color channel), so that the same image ends up needing 3.5GB (!) of RAM to represent. And then that giant image is copied several more times by the internal code, meaning even our beefiest instances were getting exhausted.

So we set about carefully profiling the memory usage of the code (and the libraries it uses like NumPy) and looking for opportunities to avoid any copying. We found the memory_profiler module especially helpful, as you can annotate any suspicious methods with @profile and then run python -m memory_profiler to get a line-by-line dump of incremental memory allocation. We found lots of places where a bit of rejiggering could save a copy here or there, and eventually got it manageable enough to run reliably on EC2’s g2.8xlarge instances. There’s still more work we could do here (e.g. rewriting numpy.roll to operate in-place instead of copying), but we were satisfied that we could now get the large images to finish dreaming without crashing.

BTW, in case you had any doubts, running this code on NVIDIA GPUs is literally about 10x faster than CPU-only. You have to make sure caffe is compiled to take advantage of GPUs and tell it explicitly to use one during execution, but trust me, it’s well worth it.

Solving the “last mile” problem

new-dreamWith our proof-of-concept in hand, our final task was to package up this code in such a way that Dan could use it on his own. There are lots of tweakable parameters in the deep dream code (including which layer of the deep neural net you use to dream, how many iterations you run, how much you scale the image up and down in the process, and so on), and we knew Dan would have to experiment for a while to figure out what achieved the desired artistic effect. We started by building a simple django web UI to upload images, select one for dreaming, set the parameters, and download the result. The Material Design Lite library made it easy to produce reasonably polished looking UI without spending much time on it. But given how long the full images took to produce (often 8-10 hours, executing a total of 70-90 quadrillion (!) floating point operations in the process), we knew we’d like to include a progress bar and enable Dan to kick off multiple jobs in parallel.

Chris took the lead here and set up celery to queue up dispatching and controlling asynchronous dreaming jobs routed to different GPUs. He also figured out how to multiply together all the various sub-steps of the algorithm to give an overall percentage complete. Once we started up the instance and the various servers, Dan could control the entire process on his own. We weren’t sure how robust it would be, but we handed it off to him and hoped for the best.

“You guys can’t believe the progress i’m making”

dreamingOnce we handed off the running EC2 instance to Dan, we didn’t hear much for a while. But it turned out that was because he was literally spending all day and night playing with the tools and honing his process. He started on a Wednesday night, and by that Saturday night he messaged us to say, “You guys can’t believe the progress I’m making. I can hardly believe it myself. Everything is working beautifully. If things continue the way they are, by Monday morning I’m going to completely amaze you.” Given that we’d barely gotten the system working at all, and that we still really didn’t know whether it could produce truly interesting output or not, this was certainly a pleasant surprise. When we probed a bit further, we could feel how excited and energized he was (his exact words were, “I’m busting, Jerry, I’m busting!”). It was certainly gratifying given the many late nights we’d spent getting to this point. But we still didn’t really know what to expect.

The following Monday, Dan unveiled a brand new gallery featuring a baker’s dozen of his biggest panoramic landscapes redone with our tool using a full range of parameter settings varying from abstract/impressionistic to literal/animalistic. He fittingly titled the collection “Dreamscapes”. For each image, he shows a zoomed-out version of the whole landscape that, at first glance, appears totally normal (keep in mind the actual images are 10-15x larger in each dimension!). But then he shows a series of detail shots that look totally surreal. His idea is that these should be hung like giant paintings 8-16’ on a side. As you walk up to the image, you start noticing the surprising detail, much as you might examine the paint, brush strokes, and fine details on a giant painting. It’s still hard for me to believe that the details can be so wild and yet so invisible at even a modest distance. But as Dan says in his intro to the gallery, “we are all actively participating in a shared waking dream. Science shows us that our limited senses perceive a tiny fraction of the phenomena that comprise our world.” Indeed!

From dream to reality

galleryWhile Dan is still constantly experimenting and tweaking his approach, the next obvious step is to print several of these works at full size to experience their true scale and detail. Since posting his gallery, he’s received interest from companies, conferences, art galleries, and individuals, so I’m hopeful we’ll soon be able to see our work “unleashed” in the physical world. With all the current excitement and anxiety around AI and what it means for society, his work seems to be striking a chord.

Of course the confluence of art and science has always played an important role in helping us come to terms with the world and how we’re affecting it. When I started trying to hack on this project in my (copious!) spare time, I didn’t realize what lay ahead. But I find myself feeling an unusual sense of excitement and gratitude at having helped empower an artistic voice to be part of that conversation. So I guess my take-away here is to encourage you to (1) not be afraid to try and change or evolve open source software to meet a new need it wasn’t originally designed for, and (2) don’t underestimate the value in supporting creative as well as functional advances.

Dream on!

Has Tesla’s personalization crossed the line?

One of the cool things about owning a Tesla is that when you have to take it in for service, they often give you a loaner Tesla to take home. It feels like nothing’s changed–until you realize that all of your personalized settings have effectively been reset. In the past, this wasn’t a big deal, but I just took my car in for 36 hours and had an unexpectedly jarring experience. I thought I’d share it because it surprised and fascinated me how pervasive and important this personalization has become.

It may be tempting to dismiss this all as “first world problems”, and obviously the car still did go from A to B just fine, but I was struck by how much I’d unknowingly come to depend on many forms of personalization Tesla offers. None of the following examples are hypothetical or contrived–they each took me by surprise and made me go “hmm”:

  1. I couldn’t open my garage door when I got home (I don’t normally carry a house key since I come and go by car, but luckily the garage has a keypad backup!)
  2. My seat position, steering mode (stiffness), etc. were different enough to frustrate my muscle memory
  3. The loaner hadn’t enabled “walk away door locks”, so I inadvertently left the car unlocked for several hours
  4. I had to re-pair my phone via bluetooth before I could resume listening to my current audible audiobook (never as smooth an experience as it should be, esp. when adding a duplicate “Tesla Model S” endpoint!)
  5. My home/work/favorite map destinations were missing, meaning I had to re-find and enter my daughter’s preschool address (to get traffic-aware driving directions/time)
  6. I was surprised to find my windshield and windows fogged up in the morning, until I remembered that my car normally remembers when I tend to leave in the morning and automatically warms itself up beforehand (“smart preconditioning”)
  7. Similarly, my car normally notifies me proactively if there’s a faster route to home/work than usual (based on current traffic conditions), but not this time
  8. My phone buzzed from the Tesla app, but it was because they’d stopped charging my car at the service center, not the car I was currently charging at work
  9. I can normally find where I parked my car, see how much charge it has, and even unlock and start it using the Android app, but none of those things work with the loaner car
  10. My daughter loves listening to some of the kids music stations on Slacker radio (each car comes with a pre-registered account), but the loaner car didn’t have any account attached, meaning I couldn’t play her that station (and even if I could, it wouldn’t remember which songs we’d ranked up/down before)

By the time I swapped back, I’d manually corrected many of these differences, but of course that creates its own problem–this loaner car was now full of my personal data (home, work, daughter’s school, paired to my phone, etc.) with no obvious way of wiping the data (I presume they do something like a factory reset before handing out the car to the next customer, but it was never discussed with me).

Now, any loaner car situation would entail some of the problems above, but most cars aren’t as personalized to start with, and the fact that I was still driving “the same car” made it all the more surprising when the car didn’t behave like it used to. More to the point, Tesla’s persistent internet connection and over-the-air software updates put it in a unique position to actually solve many or all of the above problems. I normally think of cars as fairly fungible, but clearly these are not normal cars.

“You can’t go this far and then not go further!” –Homer Simpson

To me, Tesla’s personalization has clearly crossed the line from a minor convenience to an essential part of the ownership experience. The solution then, it seems to me, is to take Homer’s advice and sync all your personalized settings to the cloud, thus enabling you to “log in” to a car and transfer your settings before you start driving. Then you could also “log out” when you’re done and reset the car. Tesla already has all the necessary building blocks to build this–the car is persistently connected to the Internet, each driver has a “My Tesla” account that knows the VIN number(s) of their car(s), and the Tesla mobile app lets you log in with your My Tesla account and view/control the state of your car.

They even dipped their toe in the water with their recent calendar integration, which uses bluetooth and your My Tesla app to show your upcoming calendar appointments on the Tesla console (with one-tap driving directions to the destinations). It’s an effective but oddly orchestrated integration, suggesting that they were aware of the potential to “do it all in the cloud”, but chose to stop short (perhaps for a mix of technical and privacy reasons?)

Tesla also currently has support for multiple “driver profiles” per vehicle, which control not only the seat positions but also favorite radio/streaming stations, preferred steering mode, A/C and temperature, and more. It’s not much of a leap to suggest that you should be able to enter a new Tesla, pair your phone (perhaps with a quick tap on an NFC terminal built into the center console?) and choose to “download your driver profile”, bringing with it all of the personalized settings mentioned above. This would be useful not only for loaner cars but also when upgrading your own car or even borrowing a friend’s car. Since not all cars have identical hardware configurations, this “import” step would allow each car to flexibly pull down the applicable settings while ignoring the rest. Such a model would also be a lot simpler than attempting a full two-way sync.

As a bonus, this might solve a silly but common frustration / feature request I’ve heard from multiple Tesla owners: the ability to select your driver profile from the mobile app before entering the car. Anyone with a significantly shorter spouse often has to embark on a series of contortions to reach the center console when the seat was  left all the way forward. :) I presume this isn’t currently possible because the driver profiles are not synced to the cloud, but once they were…

To end on a personal note, one of the things I love most about using and working on cutting-edge technology is the feeling of how “unfinished” it is. Swimming in uncharted waters reminds me that the future has yet to be written, and that it’s up to all of us to figure out where to head next. I didn’t realize that my car had gotten sufficiently personalized to need a cloud-based auth system, but it now seems obvious in hindsight. What other needs and opportunities will we discover as the lines between digital technology and the physical world continue to blur? Only one way to find out…

Machines Learning About People: What’s Next? (Future in Review 2015)

Interview with Ed Butler (photo by Kris Krüg)

I’m still buzzing with excitement and energy from attending my first Future in Review (FiRe) conference in October. I’ve been to my fair share of tech conferences, but rarely have I experienced such a dense concentration of brilliant and inspiring people from such a diverse set of fields (from cyber security to graphene, from China experts to environmental activists, and more) in such a relaxed and inviting atmosphere (we were all holed up in the Stein Erikson lodge in Deer Valley for the week, so nobody just gave their talk and took off). I see now why The Economist has called it “The best technology conference in the world.”

Another thing that makes FiRe special is how many of the participants are also speakers. True to form, conference organizer Mark Anderson threw me in the deep end with not one but three speaking slots: a 5-minute “view of the future” lightning talk, a 15-minute 1:1 interview with the BBC’s Ed Butler on “Machines Learning About People: What’s Next?”, and co-hosting a breakout session on AI and Robotics with Scout co-founder Brett Horvath. Unbeknownst to me, the first two sessions were recorded, and Mark has allowed me to share the MP3s of those talks here:

  • “Hot Spots: Five Views of the Future” (first 5 minutes are me talking about how ubiquitous sharing and context-aware gadgets will soon make us much smarter about the places we visit and people we meet)
  • “Machines Learning About People: What’s Next?” (Ed and I discuss why machine learning is increasingly used for personalization, why it’s both necessary and exciting, what the technical and privacy challenges are, and so on; featuring a surprise Q&A with Cory Doctorow at the end).

Demoing Google Stories (photo by Kris Krüg)

You can probably tell from my speed of talking in the first one that I was a bit nervous/excited to be speaking in front of such an august crowd. By the second talk (on the following day), I had calmed down a little (but this is still me we’re talking about). l hope they’ll invite me back next year, but either way, I’m even more optimistic for the future having seen firsthand how many super smart and driven people are out there tackling the full spectrum of challenges facing us!

Turns out we still need Plaxo (or something like it)

Despite having helped build a startup 12 (!) years ago focused on keeping your address book up to date, despite the massive intervening adoption of smart phones and social networks, and despite being connected to an unusually tech-savvy cohort of friends and relatives, my address book is still a disastrous mess. Sound familiar? This thought always lingered in my head, but it confronted me full-force this weekend while my wife and I struggled in vain to produce an updated mailing list to announce the birth of our new son. We’d done this for our last child (sadly using a shared Google spreadsheet in absence of any more purpose-built tool), so we were only 2-3 years out-of-date, yet an astonishing 45% of the addresses of our friends and family needed updating, and the task was not easily achieved.

This led me to ponder why this seemingly tractable problem remains so stubbornly unsolved, and what, if any, hope lies ahead.

The false saviors

It’s hard to remember that when we started Plaxo in 2002, there was no iPhone, no Facebook, not even Friendster or Flickr. Our biggest opportunity was writing a Microsoft Outlook plugin. Since then, many people have said that the problem of out-of-date contact info would soon be a relic of the past. Smartphones would keep us seamlessly connected. Social network profiles would always contain our latest info. Number portability, Gmail, and mainstream messaging services would create long-lasting identifiers. The death of snail mail would obviate the need for physical addresses. And so on. We often worried about these trends, and in some sense they have each helped us get more connected, yet they clearly haven’t solved the core problem. Why not? A few reasons…

Contact info still changes frequently (esp. in aggregate). As mentioned above, nearly half of the people we sent birth announcements to last time physically moved in the past 2-3 years. Now in our age range that’s probably more than usual (buying a first house, getting a new job, etc.), but it’s still staggeringly high. And it’s not just physical addresses. I’m constantly wondering “is this the right email address or cell phone number to use for so-and-so, or is it dead / rarely checked these days?” Ditto for “who do I know who still works at [insert company here]?” Even using circles in Google+, I’m often wondering “gee, is my Googlers circle still a safe place for Google-only discussions?” Even when the info hasn’t changed, I’m often unsure if it’s still current. And of course I keep meeting new people, many of whom I haven’t ever collected the relevant info for (even if they wouldn’t mind me having it).

Social networks suck at contact info. As surprising as the staleness of our address list was, I was even more surprised how few of my contacts had their latest address in Google+, Facebook, or elsewhere online. Many of these people used to work at Plaxo, and all are online using social networks daily. Yet the info is either missing, stale, or not shared with me. Why? In theory, social networks subsumed Plaxo-like functionality, but in practice there are deep reasons why they fall short.

  • Contact info is buried. I have to click to someone’s profile, click to their about/contact tab, scroll down, and then hope their info is shared with me. When’s the last time you viewed your own “about” tab (let alone someone else’s contact info section)? It’s out of sight, out of mind. In fact, even my own home address was out of date on Google+ and Facebook until long after I moved. It’s just not something you naturally think about while checking your news feed.
  • You don’t want everyone to have your info. Even though most social networks provide a way for you to share personal contact info, most users don’t want all of their “friends” to have all of their personal details. I recall hearing back in the day as Facebook grew that more people were actually deleting their contact info because they were making more and more loose-tie friends who they didn’t feel comfortable sharing that info with. On Google+ you can (of course) use circles to finely control who sees your home info, and Facebook has since followed suit, but as my wife put it when I asked her today, “eew, I don’t want to put my home address on Facebook”. You have to trust the site itself, your ability to navigate their privacy controls (and keep them up-to-date as your life changes), and the site’s ability to honor your choices before you’ll use social networks to share sensitive info. For most people, that bar has not yet been met.
  • Oh, and that whole “data portability” thing. When I want to see if I have someone’s latest home address, where should I look? My address book? Only if I can pull in the info shared with me from social networks. Not surprisingly, Google+ syncs to Google Contacts, but everything else is still a walled garden. You have to go knock on all the doors. For every person you care about. Every time. Even though they chose to share it with you. Sound broken? I agree. But I guess we weren’t loud enough.

Smartphones aren’t smart about contact info. Your smartphone address book does a good job of following you around from device to device and desktop to mobile. The only problem is that it is syncing garbage data. You’d think that using it to send and receive phone calls, text messages, emails, and looking up driving directions would make your address book fresher and more complete. But you’d be wrong. Chalk it up to app fragmentation and no one really trying hard to solve the problem in the first place, esp. at the OS level. Even at Google, the Gmail Contacts team is separate from the android “People app” team, and most OEMs bundle their own separate address book. Good luck.

No one’s helping you scrounge. Another fascinating if infuriating aspect of my recent Labor Day labor to update our mailing list was how often I could find the addresses by scrounging through my email archives, text messages, and manually entered address book info. I’d even been to many of the homes I lacked addresses for! In the absence of a good contact info syncing solution, most people still solve the problem “out of band” via existing communication channels (“leaving now, btw what’s your address in SF?” “it’s 123 Fake St., see you soon!”). Yet nothing is helping you extract, save, and aggregate that info for the next time you need it. It’s still a painful, manual process that you need to “get good at”. And this is for close ties–not random acquaintances–so it’s surely just an “entropy” problem, not a “stalking” problem.

The wisdom of crowds? Another trend I was sure would take off in the age of the cloud was some kind of solution for pooling contact info between families, friends, and organizations. At Plaxo we used to always have the problem, “Who’s got the cell phone for that new guy we just hired?” and the answer was always “someone in the room” (you just don’t know who in advance, and you have to ask them first). Many families still have a designated aunt who dutifully maintains the conglomerated birthday and holiday card list. And “hey Garret, remind me what Pete’s new address is?” still gets the job done in a pinch without offending. So why does each address book still start from scratch as if it were the only record of knowledge in the universe?

A new hope?

In the years since leaving Plaxo to help start Google+, I’ve talked to nearly every “address book 2.0” startup that’s reared its head. Apparently I’ve got a reputation in the Valley for not being “over” this contact info problem. Many have offered clever twists, but none have fundamentally addressed the challenges above. And perhaps unsurprisingly, as a result, none have saved the world from its frustrating fragmentation. So why am I still eternally optimistic? Because it’s a real, mainstream problem, there’s no good reason people want fragmentation to persist, and increasingly smartphones do participate in the events that collect or verify contact info. Plus there’s still the cloud opportunity for making each other smarter. So how might such a solution emerge?

When trying to solve any complex social problem, one good question to ask is, “What does the ideal solution state look like?” In the case of up-to-date contact info, I’d argue we still don’t know. You could say it’s everyone being on a single social network and using it perfectly, but is that ever going to be realistic? I’d say it’s more likely a mix of assisted sync and scrounge. In other words: help me collect whatever’s been shared with me via social networks or communication tools. And the place to do that is logically a smartphone (backed by a cloud-hosted account). Google or Apple are, in theory, in a great position to make progress on this, but I suspect it will be a startup that gets the job done, since it can be their sole focus and brand identity.

Such a startup would have to embrace the messy reality I’ve outlined above and turn it into a strength. Use all the available APIs and other tricks to help me find the contact info that has been shared with me. Keep track of when it was last updated (don’t make me guess). Parse through all my emails and texts for stuff that looks like contact info. Use my phone’s location history to ask me whose house I just visited when it doesn’t look like a business. Remind me what email or phone number each contact last used, and let me easily ping them back if I need some updated info. Help me build custom lists for things like holiday or birth announcements, and use that as an opportunity to ask for updated info. And partner with sites like TinyPrints not only to send those cards but also to send change-of-address cards when I myself move (something you should also be able to detect using my phone). Once you start gaining traction helping individuals keep their address books up-to-date, add a layer to pool it with family, friends, and colleagues in a privacy-preserving way (e.g. an easy way to see who knows someone’s phone number, but you still have to ask them to share it with you).

Is there enough here to build a successful business around? You be the judge. But is this still a real problem that real people still wish someone would solve? Abso-f*cking-lutely.

Personal Trainer: The Best Investment I’ve Made in Years

Growing up, I was pretty active physically. But after leaving home for college and pursuing a career in computers, my activity level continued to fall while my weight moved in the opposite direction. I tried numerous times to start working out regularly, but I would always fall off the wagon after a matter of weeks when life got extra busy. While I’d resisted the idea for years, last fall I finally decided to start working twice a week with a personal trainer, and the results have been fantastic–I now work out regularly and reliably (both with my trainer and on my own in between), I feel stronger, healthier, and happier, and my career is none the worse for spending the extra time I didn’t think I had.

I’ve told several colleagues this story (not surprisingly, many in Silicon Valley are in the same boat–working long hours and neglecting their health even though they’re aware that it’s a problem), and it’s inspired several of them to follow suit. And apparently I’m not alone. So I thought it would be worth recounting here both why this has worked so well and why it nevertheless took me far too long to convince myself to do it.

What’s so great about a personal trainer?

The biggest advantage by far of working with a personal trainer is that I actually go work out regularly now. That’s because my trainer puts our appointments on my calendar every week, waits for me to show up, and charges me money if I don’t. That may sound glib, but the core problem I’ve had for the past few years is that I couldn’t bring myself to work out regularly on my own. I’d get too busy, or there wouldn’t be a convenient time, or I’d be tired, or whatever, but the bottom line is it wouldn’t get done, and now it does. I like to think of myself as someone with a strong will, a good sense of discipline, and a rational, long-term outlook, but clearly I’m not the only one that finds it hard to “kick my own butt” into exercising regularly. Instead of worrying about exactly what exercises to do and how often, the “high-order bit” in fixing my situation was “just show up a few times a week and do anything“.

In retrospect, the solution is so obvious: success looks like you blocking off an hour on your calendar a few times a week to go to the gym, and doing that consistently for the foreseeable future. Rather than trying to convince myself to do this ad hoc on my own, working with a trainer ensures this outcome by design. It’s a classic form of the well-known “brain hack” where you force yourself into doing something you know is right for you by creating the conditions in advance, when you’re rationally detached, rather than in the moment, when your baser instincts are in firmer control. And since I pay for my sessions in chunks of 10, and my trainer books our appointments every week, it really keeps me “on the treadmill” (both literally and figuratively).

Of course, there’s more to the value of working with a trainer than just the “forced accountability” to actually show up. My trainer knows what I’m capable of, where my weak spots are, and what my goals are (I just want to be in decent shape, not a body builder or top athlete), and she tailors our workout each time to keep me working on the right exercises with the right intensity and variety. She definitely pushes me harder than I would work on my own (“you can do one last set, can’t you? come on…”), but she also keeps me safe by watching my form and adjusting or eliminating exercises if anything is starting to hurt or look weird. This in turn lets me “turn my brain off” by not having to will myself to keep going or worry about what exercise to do next or whether I’m doing it right (all of which I’d have to do if I were working out solo), and as a result I get a more efficient and impactful workout for the time allotted.

What took you so long?

For several years, “not being in great shape”–or, more specifically, “not having regular exercise be a built-in part of my lifestyle”–has been at the top of my list of personal problems I’d like to improve. I’ve been quite fortunate in terms of family and career (my big problem being “I just want to work all the time”), but I just couldn’t seem to get “work out regularly” checked off my life’s to-do list. Reflecting on why it was so hard for me to tackle this problem–even though I cared about fixing it and had the means–I think it boiled down to the following combination of factors, which many people in my situation also face:

  • It’s not a crisis (yet). I was never “dangerously over weight”, just a bit pudgier than I’d like, and with the situation slowly worsening over time. I gained on average about 5 lbs per year since getting married, though my weight often fluctuated that much in the space of a week, so it was the long-term trend that was a problem, but in the short term it was hard to convince myself “you’ve got to drop everything and deal with this now“.
  • I should be able to fix this on my own. As I mentioned above, I tend to think “I’m a smart, capable person, so I don’t need help from other so-called professionals” when it comes to solving my problems. Especially for getting in shape, I know how to do it, and my needs are fairly modest and normal, so I often told myself that getting a personal trainer amounted to “being a wuss” and admitting a lack of will power or discipline or knowledge, which was “unbecoming for someone of my abilities”.
  • It’s expensive. While I’m well-employed enough to reasonably afford to pay a personal trainer, it’s still a lot of money. My trainer is part of Google’s “g-fit” program, and I pay a subsidized rate of $50/hour (it’s often higher at private gyms). Meeting twice a week works out to roughly $5,000/year for this service. That’s a serious chunk of change, and although “it’s hard to put a price on your health and happiness”, it was enough to further fuel my apprehension about “taking the plunge”.
  • I can’t afford the time. Probably the single biggest deterrent to getting in shape was my inability to convince myself that the few hours a week I would need to dedicate would actually be worth the opportunity cost of less time working or less time spent with my family. As a “startup guy”, both at Plaxo and more recently building Google+, I always feel like there’s more to do and it really matters and the impact is real. In other words, I have the “first-world problem” that I love my job, I’m good at it, and I have the chance to really change the world and get well compensated in the process. So it’s always hard to pull myself away from that, even to spend time with my family, whom I adore and already don’t see enough. Thus taking even more time away to work something as “vain and optional” as improving my physique was just a hard thing to convince myself was a worthy use of my limited time.

Now rationally, of course, I realize that none of these arguments actually make much sense. Of course staying healthy is important and worthwhile, of course it’s a reasonable (and ultimately modest) investment of time and money, and of course “Parkinson’s law” will ensure that my career will not go up in flames because I spend a few hours a week working out. I know of plenty of people who are a lot busier and more important than me that work out regularly and swear by it, so clearly this is a solvable problem with plenty of “existence proofs”. But it held me up for years nonetheless, so I bring it up in the hopes that if some of these excuses ring a bell for you, this might help you fight back.

Is it really worth the investment of money and time?

Obviously not everyone can afford a personal trainer, but I would argue that for most busy professionals, it’s both achievable and money well spent. Most people I know in Silicon Valley work really hard and wish they were in better shape, but are actually bounded more by time than by money to do something about it. So once you convince yourself that you can afford the time, the money should actually be “the easy part”. If someone offered to wave a magic wand and make you healthy and active for the cost of $5,000, I bet a lot of people would take it. Yet that’s exactly what you can do by getting a trainer (and even cheaper if you do small group sessions or only meet once a week). It’s particularly convenient at Google because they have on-site trainers and on-site gyms with lockers/showers (so it really is as simple as “going to my 3pm meeting on the first floor”), but many people live or work near gyms and have flexible enough schedules that they can pencil in time with a trainer a couple times a week. And of course, as many people told me before (and I’m telling you now), it does come with a “high ROI” in terms of having more energy, sleeping better, and spending less time worrying that you’re not getting in shape.

In short, I couldn’t be happier with the choice to start working with a personal trainer, and if you’re anything like me, I bet you’d feel the same if you gave it a try. Let me know if you do (or have already done so recently); I’d love to hear from more people who are figuring out how to be healthy and geeky at the same time!

Winning Market Share in the Sharing Market

[Usual disclaimer: I currently work for Google on Google+ (which I think is awesome), but as always the thoughts on this blog are purely my personal opinion as a lifelong technology lover and builder.]

At the core of innovation in social networking is the competition to be a user’s tool of choice when they have something to share. Most people go to social networks because their friends are sharing there, and there is an inherit scarcity to compete for because most people aren’t willing to share something more than once (some may syndicate their content elsewhere, if permitted, but the original content is usually shared on a single service). Thus every time people are moved to share, they must choose which tool to use (if any)–a social network, via email/sms, etc.

Conventional wisdom holds that the main factor determining where a user will share is audience size (i.e. where they have the most friends/followers, which for most people today means Facebook or Twitter), but the successful rise of services like Foursquare, Instagram, Path, Tumblr, and Google+ shows that there’s more to the story. Despite the continued cries of “it’s game-over already for social networking” or “these services are all the same”, I don’t think that’s the case at all, and apparently I’m not alone.

These newer services are all getting people to share on them, even though in most cases those users are sharing to a smaller audience than they could reach on Facebook or Twitter. What’s going on here? It seems to me that these services have all realized that the competition for sharing is about more than audience size, and in particular includes the following additional two axes:

  • Ease of sharing to a more specific/targeted audience
  • Ease of sharing more tailored/beautiful content

Sharing to a more specific/targeted audience sounds like the opposite of “sharing where all of my friends/followers already are”, but not everything we say is meant to be heard by everyone. In fact, when I ask people why they don’t share more online, two of the most frequent responses I get have to do with having too much audience: privacy and relevance. Sometimes you don’t want to share something broadly because it’s too personal or sensitive, and sometimes you just don’t want to bug everyone. In both cases, services that help you share more narrowly and with more precision can “win your business” when you otherwise would have shared reluctantly or not at all.

Foursquare, Path, and Google+ (and email) are all good examples of this. Sharing your current location can be both sensitive and noisy, but on foursquare most people maintain a smaller and more highly trusted list of friends, and everyone there has also chosen to use the service for this purpose. Thus it wins on both privacy and relevance, whereas Facebook Places has largely failed to compete, in part because people have too many friends there. Similarly, Path encourages you to only share to your closest set of friends and family, and thus you feel free to share more, both because you trust who’s seeing your information, and because they’re less likely to feel you’re “spamming them” if they’re close to you. And by letting you share to different circles (or publicly) each time you post, Google+ lets you tune both the privacy and relevance of your audience each time you have something to say. Google+ users routinely post to several distinct audiences, showing that we all have more to say if we can easily control who’s listening.

Sharing more tailored/beautiful content is in theory orthogonal to audience size/control, but it’s another instance in which smaller services often have an advantage, because they can specialize in helping users quickly craft and share content that is richer and more visually delightful than they could easily do on a larger and more generic social network. This is an effective axis for competition in sharing because we all want to express ourselves in a way that’s compelling and likely to evoke delight and reaction from our friends/followers. So services that help you look good can again “win your business” when you otherwise would have shared something generic or not at all.

Instagram and Tumblr jump out from my list above as services that have attracted users because they help you “look good for less”, but actually all five services have succeeded here to some extent. A foursquare check-in has rich metadata about the place you’re visiting, so it’s a more meaningful way to share than simply saying “I’m eating at Cafe Gibraltar”. Instagram lets you apply cool filters to your photos, and more importantly IMO, constrains the sharing experience to only be a single photo with a short caption (thus further encouraging this type of sharing). Path just looks beautiful end-to-end, which entices you to live in their world, and they also add lovely touches like displaying the local weather next to your shared moments or letting you easily share when you went to bed and woke up (and how much sleep you got). Again, these are all things you could share elsewhere, but it just feels better and more natural on these more tailored services. Tumblr is “just another blogging tool” in some ways, but its beautiful templates and its focus on quickly reblogging other content along with a quick message has led to an explosion of people starting up their own “tumblelogs”. And on Google+, one of the reasons a thriving photography community quickly formed is just that photos are displayed in a more compelling and engaging way than elsewhere.

The future of social networking will be largely about making it easier to share richer content with a more relevant audience. Some already decry the current “information overload” on social networks, but I would argue that (a) many/most of the meaningful moments in the lives of people I care about are still going un-shared (mainly because it’s too hard), (b) the feeling of overload largely comes from the low “signal-to-noise ratio” of posts shared today (owing to the lack of easy audience targeting), and (c) the content shared today is largely generic and homogeneous, thus it’s harder to pick out the interesting bits at the times when its most relevant. But as long as entrepreneurs can think up new ways to make it easier to capture and share the moments of your life (and the things you find interesting) in beautiful, high fidelity, and make it easier to share those moments with just the right people at the right time, we’ll soon look back on the current state of social networking with the same wistful bemusement we cast today on brick-sized feature phones and black-and-white personal computers. Let’s get to it!

Fighting for the Future of the Social Web: Selling Out and Opening Up (OSCON 2011)

Fighting for the Future of the Social Web: Selling Out and Opening Up
O’Reilly Open Source Convention (OSCON) 2011
Portland, OR
July 27, 2011

(Note: some of the footer fonts are messed up on slideshare, sorry.)

Download PPT (12.5 MB)

A year and a half after joining Google, and a year after my last talk on the Social Web, I returned to OSCON (one of my favorite conferences, which I’ve been speaking at for over half a decade now!) to reflect on the progress we’ve collectively made (and haven’t made) to open up the social web. I covered the latest developments in OpenID, OAuth, Portable Contacts, and related open web standards, mused about the challenges we’re still facing to adoption and ease-of-use and what to do about them, and considered what changes we should expect going forward now that many of the formerly independent open social web enthusiasts (myself included) now work for larger companies.

Not to spoil the punchline, but if you know me at all it won’t surprise you to learn that I’m still optimistic about the future! 😉

Bridging the islands: Building fluid social experiences across websites (Google I/O 2010)

Bridging the islands: Building fluid social experiences across websites
Google I/O 2010
San Francisco, CA
May 19, 2010

View talk and download slides as PDF

My third year speaking at Google I/O, and my first as a Googler! I teamed up with fellow Googler John Panzer, and together we demonstrated how far open standards have come in allowing developers to build rich cross-site social integrations. From frictionless sign-up using OpenID, OAuth, and webfinger to finding your friends with Portable Contacts and microformats to sharing rich activities and holding real-time distributed conversations with, PubSubHubBub, and salmon, it really is remarkable how much progress we’ve made as a community. And it still feels like we’re just getting started, with the real payoff right around the corner!

We took a literal approach to our concept of “bridging the islands” by telling a story of two imaginary islanders, who meet while on vacation and fall in love. They struggle with all the same problems that users of today’s social web do–the pain of immigrating to a new place, the pain of being able to find your friends once they’ve moved, and the pain of being able to stay in touch with the people you care about, even when you don’t all live in the same place. Besides having fun stretching the metaphor and making pretty slides (special thanks to Chris Messina for his artistic inspiration and elbow grease!), the point is that these are all fundamental problems, and just as we created technology to solve them in the real world, so must be solve them on the Social Web.

Chris’s talk at I/O told this story at a high level and with additional color, while we dove more into the technology that makes it possible. Make sure to check out both talks, and I hope they will both inspire and inform you–whether as a developer or a user–to help us complete this important work as a community!

Implementing PubSubHubbub subscriber support: A step-by-step guide

One of the last things I did before leaving Plaxo was to implement PubSubHubbub (PuSH) subscriber support, so that any blogs which ping a PuSH hub will show up almost instantly in pulse after being published. It’s easy to do (you don’t even need a library!), and it significantly improves the user experience while simultaneously reducing server load on your site and the sites whose feeds you’re crawling. At the time, I couldn’t find any good tutorials for how to implement PuSH subscriber support (add a comment if you know of any), so here’s how I did it. (Note: depending on your needs, you might find it useful instead to use a third-party service like Gnip to do this.)

My assumption here is that you’ve already got a database of feeds you’re subscribing to, but that you’re currently just polling them all periodically to look for new content. This tutorial will help you “gracefully upgrade” to support PuSH-enabled blogs without rewriting your fundamental polling infrastructure. At the end, I’ll suggest a more radical approach that is probably better overall if you can afford a bigger rewrite of your crawling engine.

The steps to add PuSH subscriber support are as follows:

  1. Identify PuSH-enabled blogs extract their hub and topic
  2. Lazily subscribe to PuSH-enabled blogs as you discover them
  3. Verify subscription requests from the hub as you make them
  4. Write an endpoint to receive pings from the hub as new content is published
  5. Get the latest content from updated blogs as you receive pings
  6. Unsubscribe from feeds when they’re deleted from your system

1. Identify PuSH-enabled blogs extract their hub and topic

When crawling a feed normally, you can look for some extra metadata in the XML that tells you this blog is PuSH-enabled. Specifically, you want to look for two links: the “hub” (the URL of the hub that the blog pings every time it has new content, which you in turn communicate with to subscribe and receive pings when new content is published), and the “self” (the canonical URL of the blog you’re subscribing to, which is referred to as the “topic” you’re going to subscribe to from the hub).

A useful test blog to use while building PuSH subscriber support is, since it lets anyone publish new content. If you view source on that page, you’ll notice the standard RSS auto-discovery tag that tells you where to find the blog’s feed:

<link title="PubSubHubbub example app" type="application/atom+xml" rel="alternate" />

And if you view source on, you’ll see the two PuSH links advertised underneath the root feed tag:

<link type="application/atom+xml" title="PubSubHubbub example app" rel="self" />
<link rel="hub" href="" />

You can see that the “self” link is the same as the URL of the feed that you’re already using, and the “hub” link is to the free hub being hosted on AppEngine at In both cases, you want to look for a link tag under the root feed tag, match the appropriate rel-value (keeping in mind that rel-attributes can have multiple, space-separated values, e.g. rel="self somethingelse", so split the rel-value on spaces and then look for the specific matching rel-value), and then extract the corresponding href-value from that link tag. Note that the example above is an ATOM feed; in RSS feeds, you generally have to look for atom:link tags under the channel tag under the root rss tag, but the rest is the same.

Once you have the hub and self links for this blog (assuming the blog is PuSH-enabled), you’ll want to store the self-href (aka the “topic”) with that feed in your database so you’ll know whether you’ve subscribed to it, and, if so, whether the topic has changed since you last subscribed.

2. Lazily subscribe to PuSH-enabled blogs as you discover them

When you’re crawling a feed and you notice it’s PuSH-enabled, check your feed database to see if you’ve got a stored PuSH-topic for that feed, and if so, whether the current topic is the same as your stored value. If you don’t have any stored topic, or if the current topic is different, you’ll want to talk to that blog’s PuSH hub and initiate a subscription so that you can receive real-time updates when new content is published to that blog. By storing the PuSH-topic per-feed, you can effectively “lazily subscribe” to all PuSH-enabled blogs by continuing to regularly poll and crawl them as you currently do, and adding PuSH subscriptions as you find them. This means you don’t have to do any large one-time migration over to PuSH, and you can automatically keep up as more blogs become PuSH-enabled or change their topics over time. (Depending on your crawling infrastructure, you can either initiate subscriptions as soon as you find the relevant tags, or you can insert an asynchronous job to initiate the subscription so that some other part of your system can handle that later without slowing down your crawlers.)

To subscribe to a PuSH-enabled blog, just send an HTTP POST to its hub URL and provide the following POST parameters:

  • hub.callback = [the URL of your endpoint for receiving pings, which we’ll build in step 4]
  • hub.mode = subscribe
  • hub.topic = [the self-link / topic of the feed you’re subscribing to, which you extracted in step 1]
  • hub.verify = async [means the hub will separately call you back to verify this subscription]
  • hub.verify_token = [a hard-to-guess token associated with this feed, which the hub will echo back to you to prove it’s a real subscription verification]

For the hub.callback URL, it’s probably best to include the internal database ID of the feed you’re subscribing to, so it’s easy to look up that feed when you receive future update pings. Depending on your setup, this might be something like or Another advantage of this technique is that it makes it relatively hard to guess what the update URL is for an arbitrary blog, in case an evil site wanted to send you fake updates. If you want even more security, you could put some extra token in the URL that’s different per-feed, or you could use the hub.secret mechanism when subscribing, which will cause the hub to send you a signed verification header with every ping, but that’s beyond the scope of this tutorial.

For the hub.verify_token, the simplest thing would just be to pick a secret word (e.g. “MySekritVerifyToken“) and always use that, but an evil blog could use its own hub and quickly discover that secret. So a better idea is to do something like take the HMAC-SHA1 of the topic URL along with some secret salt you keep internally. This way, the hub.verify_token value is feed-specific, but it’s easy to recompute when you receive the verification.

If your subscription request is successful, the hub will respond with an HTTP 202 “Accepted” code, and will then proceed to send you a verification request for this subscription at your specified callback URL.

3. Verify subscription requests from the hub as you make them

Shortly after you send your subscription request to the hub, it will call you back at the hub.callback URL you specified with an HTTP GET request containing the following query parameters:

  • hub.mode = subscribe
  • hub.topic = [the self-link / topic of the URL you requested a subscription for]
  • hub.challenge = [a random string to verify this verification that you have to echo back in the response to acknowledge verification]
  • hub.verify_token = [the value you sent in hub.verify_token during your subscription request]

Since the endpoint you receive this verification request is the same one you’ll receive future update pings on, your logic has to first look for hub.mode=subscribe, and if so, verify that the hub sent the proper hub.verify_token back to you, and then just dump out the hub.challenge value as the response body of your page (with a standard HTTP 200 response code). Now you’re officially subscribed to this feed, and will receive update pings when the blog publishes new content.

Note that hubs may periodically re-verify that you still want a subscription to this feed. So you should make sure that if the hub makes a similar verification request out-of-the-blue in the future, you respond the same way you did the first time, providing you indeed are still interested in that feed. A good way to do this is just to look up the feed every time you get a verification request (remember, you build the feed’s ID into your callback URL), and if you’ve since deleted or otherwise stopped caring about that feed, return an HTTP 404 response instead so the hub will know to stop pinging you with updates.

4. Write an endpoint to receive pings from the hub as new content is published

Now you’re ready for the pay-out–magically receiving pings from the ether every time the blog you’ve subscribed to has new content! You’ll receive inbound requests to your specified callback URL without any additional query parameters added (i.e. you’ll know it’s a ping and not a verification because there won’t be any hub.mode parameter included). Instead, the new entries of the subscribed feed will be included directly in the POST body of the request, with a request Content-Type of application/atom+xml for ATOM feeds and application/rss+xml for RSS feeds. Depending on your programming language of choice, you’ll need to figure out how to extract the raw POST body contents. For instance, in PHP you would fopen the special filename php://input to read it.

5. Get the latest content from updated blogs as you receive pings

The ping is really telling you two things: 1) this blog has updated content, and 2) here it is. The advantage of providing the content directly in the ping (a so-called “fat ping“) is so that the subscriber doesn’t have to go re-crawl the feed to get the updated content. Not only is this a performance savings (especially when you consider that lots of subscribers may get pings for a new blog post at roughly the same time, and they might otherwise all crawl that blog at the same time for the new contents; the so-called “thundering herd” problem), it’s also a form of robustness since some blogging systems take a little while to update their feeds when a new post is published (especially for large blogging systems that have to propagate changes across multiple data-centers or update caching tiers), so it’s possible you’ll receive a ping before the content is available to crawl directly. For these reasons and more, it’s definitely a best-practice to consume the fat ping directly, rather than just using it as a hint to go crawl the blog again (i.e. treating it as a “light ping”).

That being said, most crawling systems are designed just to poll URLs and look for new data, so it may be easier to start out by taking the “light ping” route. In other words, when you receive a PuSH ping, look up the feed ID from the URL of the request you’re handling, and assuming that feed is still valid, just schedule it to crawl ASAP. That way, you don’t have to change the rest of your crawling infrastructure; you just treat the ping as a hint to crawl now instead of waiting for the next regular polling interval. While sub-optimal, in my experience this works pretty well and is very easy to implement. (It’s certainly a major improvement over just polling with no PuSH support!) If you’re worried about crawling before the new content is in the feed, and you don’t mind giving up a bit of speed, you can schedule your crawler for “in N seconds” instead of ASAP, which in practice will allow a lot of slow-to-update feeds to catch up before you crawl them.

Once you’re ready to handle the fat pings directly, extract the updated feed entries from the POST body of the ping (the payload is essentially an exact version of the full feed you’d normally fetch, except it only contains entries for the new content), and ingest it however you normally ingest new blog content. In fact, you can go even further and make PuSH the default way to ingest blog content–change your polling code to act as a “fake PuSH proxy” and emit PuSH-style updates whenever it finds new entries. Then your core feed-ingesting code can just process all your updated entries in the same way, whether they came from a hub or your polling crawlers.

However you handle the pings, once you find that things are working reliably, you can change the polling interval for PuSH-enabled blogs to be much slower, or even turn it off completely, if you’re not worried about ever missing a ping. In practice, slow polling (e.g. once a day) is probably still a good hedge against the inevitable clogs in the internet’s tubes.

6. Unsubscribe from feeds when they’re deleted from your system

Sometimes users will delete their account on your system or unhook one of their feeds from their account. To be a good citizen, rather than just waiting for the next time the hub sends a subscription verification request to tell it you no longer care about this feed, you should send the hub an unsubscribe request when you know the feed is no longer important to you. The process is identical to subscribing to a feed (as described in steps 2 and 3), except you use “unsubscribe” instead of “subscribe” for the hub.mode values in all cases.

Testing your implementation

Now that you know all the steps needed to implement PuSH subscriber support, it’s time to test your code in the wild. Probably the easiest way is to hook up that feed, since you can easily add content it to it to test pings, and it’s known to have valid hub-discovery metadata. But you can also practice with any blog that is PuSH-enabled (perhaps your shiny new Google Buzz public posts feed?). In any case, schedule it to be crawled normally, and verify that it correctly extracts the hub-link and self-link and adds the self-link to your feed database.

The first time it finds these links, it should trigger a subscription request. (On subsequent crawls, it shouldn’t try to subscribe again, since the topic URL hasn’t changed. ) Verify that you’re sending a request to the hub that includes all the necessary parameters, and verify that it’s sending you back a 202 response. If it’s not working, carefully check that you’re sending all the right parameters.

Next, verify that upon sending a subscription request, you’ll soon get an inbound verification request from the hub. Make sure you detect requests to your callback URL with hub.mode=subscribe, and that you are checking the hub.verify_token value against the value you sent in the subscription request, and then that you’re sending the hub.challenge value as your response body. Unfortunately, it’s usually not easy to inspect the hub directly to confirm that it has properly verified your subscription, but hopefully some hubs will start providing site-specific dashboards to make this process more transparent. In the meantime, the best way to verify that things worked properly is to try making test posts to the blog and looking for incoming pings.

So add a new post on the example blog, or write a real entry on your PuSH-enabled blog of choice, and look in your server logs to make sure a ping came in. Depending on the hub, the ping may come nearly instantaneously or after a few seconds. If you don’t see it after several seconds, something is probably wrong, but try a few posts to make sure you didn’t just miss it. Look at the specific URL that the hub is calling on your site, and verify that it has your feed ID in the URL, and that it does indeed match the feed that just published new content. If you’re using the “light ping” model, check that you scheduled your feed to crawl ASAP. If you’re using the “fat ping” model, check that you correctly ingested the new content that was in the POST body of the ping.

Once everything appears to be working, try un-hooking your test feed (and/or deleting your account) and verify that it triggers you to send an unsubscribe request to the hub, and that you properly handle the subsequent unsubscribe verification request from the hub.

If you’ve gotten this far, congratulations! You are now part of the real-time-web! Your users will thank you for making their content show up more quickly on your site, and the sites that publish those feeds will thank you for not crawling them as often, now that you can just sit back and wait for updates to be PuSH-ed to you. And I and the rest of the community will thank you for supporting open standards for a decentralized social web!

(Thanks to Brett Slatkin for providing feedback on a draft of this post!)

« Older posts

© 2016 Joseph Smarr

Theme by Anders NorenUp ↑