Thursday, July 5, 2012

Moneyball

The first thing they always do is quiz you. When big tech companies look for employee prospects, the first test is always basic knowledge. The Googles of the world have a checklist. "Coverage Areas" is what they call the talents they're checking for in a candidate. The big three are Coding, Algorithms and Analysis, Systems Design. Maybe they will also care about Communication skills, and Cultural Fit. They put you through the paces. Can she code? Does she know her Big-O analysis, her data structures? Can she design a distributed system to mimic Twitter? Tick, tick, tick. Pass enough of the checkboxes, and they'll make an offer. Nice salary, great benefits, the chance to work on big problems with these intimidatingly smart folks. When it comes to the deep pockets and well-oiled recruiting machine of a big tech company, it can be hard for a startup to compete for talent. 

I was thinking about this a lot as I watched Moneyball last night. It got me thinking. I wish there was a sabermetrics for developers. After all, what is a startup but a cash-strapped team trying to win against the Yankees? This is an imperfect analogy, but let me break it down.

First a step back for those of you that haven't watched the movie and don't give a flip about baseball. Sabermetrics is basically a system that tries to predict, through past performance, how well a player will do in the future on certain metrics. In contrast to tests of basic skills (running, throwing, fielding, hitting, power) that talent scouts look for, sabermetrics focuses on the statistics of a player's actual game performance. Instead of going broke on a couple of big-name players, as many teams would, or gambling on hyped up fresh "potential", the Oakland As used this system to draft cheap players that together ended up forming a competitive team against the A-list roster of the New York Yankees.

The thing about baseball that makes it a useful analogy for many startups is that baseball is a team sport where the size of the team is fixed and the potential of the team in aggregate matters. The same is true for a tech team. We see so many stories about how you should only hire "A" players, that they attract other A players, and they out-perform other developers 10-1 (or 50-1 or 100-1 depending on how grandiose you're feeling). But how do you find these A players? The Derek Jeters are probably settled, and cost more than you can afford. Google and its kin have deep enough pockets to hire anyone that ticks off all the "Coverage Areas", know that they will get lucky and find some natural As in that bunch, train up some more, and the rest will be good enough to take on the mountains of other work needed at such a huge company.

The thing that Google intentionally neglects, and the thing you absolutely can't afford to ignore, is past performance in the game. What is the tech equivalent of on-base percentage? Product shipped. This is why everyone drools over open source developers as hiring potential. You can see their work, you can know that they've shipped. Teasing this out of others is a tough process. We don't have visibility into the work that most people have done. Writing code for you proves that they have the tools to deliver, and is an essential part of the interview process, but it's not enough to show that they actually have delivered. All we can do is ask them, as a detailed part of their interview, what they have done. 

I realize that relying on reports of past experience as part of the hiring process is fraught. I would never suggest you neglect technical vetting in favor of experience. It's easy to get burned by someone whose personality you like and who sounds smart, only to find out they don't have the basic skills to succeed. And even with the best interviewers in the world, it's always possible that you will still hire someone that chokes at the plate. In the case where you find yourself with a player that isn't shipping, you have to take to heart the other thing that you can do more easily than Google: get rid of underperformers. It sucks, and no one enjoys letting people go, but a startup cannot afford to keep people that aren't delivering.

Will this result in your team being full of A-list all stars? Maybe. Or you may find yourself with a team where few people are exceptional standouts, but everyone delivers. When everyone in your roster can reliably get on base, you're winning. You're shipping. And that's what the game is about.

1 comment:

  1. Fascinating idea, and one I'll be sure to pursue - when we're hiring again :)

    ReplyDelete