Saturday, June 7, 2014

Accountability and authoritarians

A not-so-secret aspect of my personality is that I have no problem with the policing aspects of leadership. I have a law and order side to me, if I believe in a rule/practice, and I see people breaking it, I have no problem being the one to call them to task.

This habit actually works out ok in a room full of peers; it doesn't exactly endear me to people but it means that, eg, the build doesn't stay broken for long. Unfortunately as I have gone up in the management chain, it has become a problem. Why does having a peer that holds you day-to-day accountable work, but having a boss that does so fail?

First, it seems that having me hold individuals accountable ends up causing some individuals to stop holding each other accountable. It is viewed as the thing I do, the boss' responsibility. If it's important, the boss will care, and she will come down and call us out. This is obviously far from ideal. A high performing team will hold each other accountable; after all, you want feedback to come as quickly and naturally as possible, and that can only happen when individual team members call out each other.

Second, I have come to discover that it is rather demoralizing for team members to be called out be me. I know, I know, why is this a surprise? It is still hard to get used to the idea that I am not just one of the team, and that everything I do is amplified and taken in ways I don't always anticipate. In fact, I think that it is important that most negative/corrective feedback from me go to my direct reports instead of farther down the chain. That doesn't mean that I can't offer suggestions on architectural improvements or process tweaks, but it is demoralizing for me to ask a developer why the build is broken. In fact, it is important that I'm seen as an inspirational figure to my team, someone they look up to and look forward to interacting with, and not vice-versa.

Finally, I'm actually pretty far away from the impact of the rules these days. What might have made sense when I was in a team writing code may not be ideal for the current team, the current environment. The only true best practice out there is to let your team take a concept or process and iterate on it until it takes on a form that works effectively for them. It simply doesn't make sense for me to make and enforce the rules myself; it is my job to provide the high-level goals (such as "creating a Cinderella experience for our customers and ourselves") and to push the team to find the right ways to implement those goals.

So, what is the takeaway here? I'm getting out of the business of being the rule maker and rule enforcer. Instead, I'm setting goals and very high-level guidelines, and giving the power to create policies and practices to the members of my team. I want to work myself out of a job, after all, and the best ideas don't come from me, so why should the policing?

Wednesday, May 28, 2014

Revisiting ideas: Promotion from Within

Ages ago a friend of mine wrote a rather seminal post on "promotion from within". It is interesting to go back and read this post through the lens of hiring and leading a team for a few years. It is a great post, but I think that one major point often gets overlooked, and this point causes many companies who follow its advice to fail. That point is this:
2) Using either the few experienced managers you've been able to internally promote or failing that, outside executive coaches, intensely mentor your more inexperienced managers to develop their skills. Typically, because many of your management candidates were less than fully-qualified, they will demonstrate potential but still be unsure in their new roles. Until they are comfortable and practiced in their roles, both they, their peers, and their teams will exist in a state of some distress. 

This is problem I have seen both at my own company and also observed at others': We promote from within, but provide no mentoring or guidance to those so promoted. Great managers are truly not born. They are made, and usually made through both being sat down and patiently taught the ways to effectively lead projects and people, but also through observing both successes and failures.  They are also made by being called to task on their own personal failures, something that many startups are unwilling or unable to do. The cult of personality around founders and early employees can work against the one thing necessary to make "promote from within" successful: some form of external help.

Most of us in the startup world are working amongst people that have very little experience managing. And we've taken these ideas that Yishan so eloquently voiced, that culture is paramount, and elevated them to high status, while forgetting that there is a lot to learn to be a successful manager. I know that I came into this job two years ago thinking that given my natural willingness to be in charge and my strong technical skills I would be a great manager. Haha! Truthfully I'm only now getting to the point where I have an inkling of all the things I don't know, and a large part of that is thanks to a ton of coaching. I would not be able to lead my team successfully without coaching, and even with my own coach, I need a coach to help the managers that report to me.

So, promote from within. But don't cheap out on the process by forgetting that these new managers and leaders need help, need training, need to be held responsible for both the good and the bad that they will inevitably produce in their first months and years as managers. Otherwise you might as well hire experienced external managers, because my hunch is that the payoff is actually equivalent, risking culture vs risking unguided learning.

Saturday, May 10, 2014

Please stop threatening me with Moore's Law

For as long as I can remember, Moore's Law has been one of the great tech bogeymen. It's going to end, we fret, and the way we write code is going to have to change dramatically! The sky will fall, and we need to prepare ourselves for this end times!
When you hear the same message for well over ten years, its efficacy starts to fade a bit. You start to wonder, when exactly is Moore's Law going to end any more than it already has? When is this going to happen in a way that actually affects me more than it did five years ago? What even IS right around the corner, if the foretold end of performance hasn't much affected me in ten plus years?
The truth of the matter is this: if you care about Moore's Law, you're probably already writing code that combats it, because you care about performance. The trick of Moore's Law, as we all have been beaten over the head about, is parallelism. But why would I wait for the end of Moore's Law to bite me? The times I have cared about performance, I haven't waited. I've parallelized the crap out of code to move performance sensitive code bases from few fast cores to many slow cores. And as soon as I could, I ripped most of it out in favor of a distributed system that was faster and more scalable. And then that was ripped out in favor of smart streaming from SSD. The circle of tech life takes advantage of the latest hotness as needed to get the job done, and I have no reason to believe that Moore's Law is anything more than a factor in that equation.
These days most of us are concerned about a much more complex interaction of performance issues than simple processor speed. We're network sensitive, IO bound, dealing with crazy amounts of data, or simply trying to deal with a ton of simple things at once. We already have systems built to make IO and network calls asynchronous. We're already processing completely separate work independently. Because we can, because even without the terrible end of Moore's Law we care about performance. There's no need to call up the bogeyman to make your case. He's sitting in the next cube, making sure we parallelized all our outgoing requests, and he'd rather you stopped getting so hysterical on his behalf.

Wednesday, February 5, 2014

Getting From Here to There

As part of my work to make myself a better leader, I'm reading the book "What Got You Here Won't Get You There." The phrase itself resonates with me strongly now as I'm in reviews season and writing reviews for a number of senior engineers and engineering managers. One of the standard refrains I see in self-review "areas for improvement" is the desire to improve something very technology specific. A manager might say "I want to jump in more to the code so I can help take problems off my team." A senior engineer looking to grow to the staff engineer level might say something like "I want to get better at understanding distributed systems." Heck, even I am tempted to put "I want to spend more time in the early phases of architecture design so that I can help improve our overall technology stack." We're all falling victim to the problem of "what got us here won't get us there."

Engineers should spend their first several years on the job getting better at technology. I consider that a given and don't love to see that goal in reviews even for junior engineers, unless the technology named is very specific. Of course you're getting better at technology, programming, engineering, that's your job. You are a junior individual contributor, and your contribution is additive. You take tasks off of a team's list, get them done, create value by doing, and work on getting better at doing more and more. You got here because you put this coding time in.

After a certain point, it is more important to focus on what will make you a multiplier on the team. Very very few engineers write code of such volume and complexity that simply by writing code they enhance the entire organization. For most of us, even those in individual contributor roles, the value comes through our work across the team, teaching junior engineers, improving processes, working on the architecture and strategy so that we simply don't write as much code to begin with. There is a certain level of technical expertise that is necessary to get to this multiplier stage. As an individual contributor, a lot of that expertise is in knowing what you don't know. What do you need to research to make this project successful? You don't have to be a distributed systems expert but you should know when you're wading into CAP theorem territory.

It's harder for managers. Every time you switch jobs, you're interviewed with an eye towards the question, "can this person write code?" We don't trust managers that can't code, we worry that they're paper-pushers, out of touch. But when you hire an engineering manager, you often don't want them writing code. Managers that stubbornly hold onto the idea that they must write a lot of code are often either overworked, bottlenecks, or both. The further up you go in the management chain, the less time you will have to write code in your day job. The lesson here is not "managers should carve out lots of time to write code". It is, instead, don't get pushed into management until you've spent enough time coding that it is second nature. If you think that writing more code is the unlock for you to manage your team better, to grow as a better leader, you're going backwards. The way forward into true team leadership is not through writing more code and doing the team's scutwork programming. It is through taking a step back, observing what is working, what is not working, and helping your team fix it from a macro level. You're not going to code your team out of crunch mode by yourself, so spend your time preventing them from getting into crunch mode in the first place.

Coding is how you got here, engineering is what we do. But growing to levels of leadership requires more than just engineering. You've gotta go beyond what got you where you are, and get out of your comfort zone. Work on what makes you a multiplier, and you'll get there.

Wednesday, January 15, 2014

SOA and team structure

This week I sat on a panel to discuss my experiences moving to a service-oriented architecture (SOA) at Rent the Runway. Many of the topics were pretty standard: when to do such a thing, best practices, gotchas. Towards the end there was an interesting question. It was phrased roughly like this:
What do you do when you need to create a new feature, and it crosses all sorts of different services? How do you wrangle all the different teams so that you can easily create new features?
I think this is a great question, and illustrates one of the common misperceptions, and common failure modes, of going to a service-oriented architecture.

Let me be clear: SOA is not designed to separate your developers from each other. In my team, developers may work across many different services in order to accomplish their tasks, and we try to make it clear that the systems are "internal open source". You may not be the expert in that system, but when a feature is needed, it's expected that the team creating the feature will roll up their sleeves and get coding.

At big companies SOA is sometimes done in order to create areas of ownership and development. At my previous company, SOA was (among other things) a better way to expose data in an ad-hoc, but monitored, manner, without having to send messages or allow access to databases. The teams owning the services may be in a totally different area of the company from their clients. To access new data, you needed to coordinate with the owning team to expose new endpoints. It created overhead, but data ownership and quality of the services themselves was an important standard and losing velocity to maintain these standards was an acceptable tradeoff.

That is not the way a small startup should approach SOA, but if you don't anticipate this it may become an unintended outcome. When your SOA is architected as mine is, with a different language powering the services than that which powers the client experience (Java on the backend, Ruby on the frontend), you start to segregate your team into backend and frontend developers. We address this separation by working in business-focused teams that have both backend and frontend developers. Other SOA-based companies approach the challenge of separation by creating microservices that only do enough to support a single feature, so that a new feature doesn't always require touching existing functionality. Some companies do SOA with the same framework (say, Ruby on Rails) that powers their user-facing code, so there's no language or framework barrier to overcome when crossing service boundaries.

SOA is a powerful model for creating scalable software, but many developers are reluctant to adopt it because of this separation myth. There are many ways to approach SOA that work around this downside. Acknowledging the risk here and architecting your teams with an eye to avoiding it is important in successfully adopting this model in an agile environment.

Tuesday, December 31, 2013

2013: The Constant Introspection of Management

I've been a serious manager for a little over a year now, and it has been my biggest challenge of 2013. Previously I had managed small teams but looking back, while I thought that would prepare me to lead multiple teams and manage managers myself, in reality nothing could be further from the truth.

The hardest part for me of going from individual contributor/architect/tech lead to managing in anger has been the lack of certainty. I believe that, for experienced managers, management can have a level of rigor and certainty, but I'm not there yet. I am striving to be a compassionate manager and that requires developing a level of emotional intelligence that I have never before needed. And so as a result I would have to say that the past year has probably been one of the most emotionally draining of my life (and that is not even counting the baby I had in the middle of it).

I do not want to be a dispassionate leader who views people like pawns on a chessboard. But the emotional resilience that is required for management makes me understand how folks with that tendency may find it easier. A successful manager needs to care about her people without taking the things they do personally. Every person who quits feels like an indictment of all the ways you failed them. If only I had given better projects, fought harder for their salary, coached them better, done more to make them successful! If you wonder why it seems sometimes that management roles get taken up by sociopaths, just think that for every interaction you have with a difficult coworker, your manager has probably had to deal with ten of them. It's not a surprise that a certain bloody-mindedness develops, or, more likely, survives.

In addition to the general emotional angst of all those people, there's the general feeling of utter incompetence. As an engineer, I know how to design successful systems. I can look back on a career of successful projects, and I know many of the best practices for building systems and writing code. Right now, if I were to design a system that ultimately failed for a technical reason, I would be able to pinpoint where the mistakes were made. I am a beginner all over again when it comes to the big-league management game, and it's discouraging. I miss doing what I'm good at, building systems, and I'm afraid that I've given it up for something I may never do particularly well.

One of my friends who has faced the same struggle put it best. 
I like the autonomy/mastery/purpose model of drive. This feels like an issue with mastery. Not building means moving away from something where you have mastery to something new. There’s fear of losing mastery.[1]
As a new manager I believe you lose both autonomy and mastery for a time being, and arguably autonomy is lost forever. You are always only as good as your team, and while some decisions may ultimately rest on your shoulders, when you choose to take the "servant leadership" path you do sacrifice a great deal of autonomy. But I think for many engineers the loss of mastery hits hardest. When you've spent ten plus years getting really, really good at designing and developing systems, and you leave that to think about people all day? It's hard, and no, there isn't always time for side projects to fill the gap. In an industry that doesn't always respect the skills of management, this is a tough pill to swallow. After all, I can become the greatest manager in the world, but if I wanted to work in that role at Google they would still give me highly technical interviews.

So why do it? In the end, it has to be about a sense of purpose. I want to have a bigger impact than I will ever be able to have as an architect or developer. I know that leading teams and setting business direction is the way to ultimately scratch the itch I have for big impact, for really making a lasting difference. And I know that a great manager can have a positive impact on many, many people. So here's to growing some management mastery, and making 2014 a year of purpose and impact.

Wednesday, July 31, 2013

Replatforming? The Proof is in the Hackday

It's pretty common for teams, especially startups, to get to a point with their tech stack where the platform they've been working on just isn't cutting it anymore. Maybe it won't scale, maybe it won't scale to the increased development staff, maybe it was all written so poorly you want to burn it to the ground and start fresh. And so the team takes on the heroic effort we know and love: replatforming.

When you're replatforming because the current system can't handle the necessary load, it's pretty easy to see if your effort was successful. Load test, or simply let your current traffic hit it and watch it hum along where you once were serving 10% failures. But if the replatforming is done to help development scaling, how do you know the effort was a success?

I accidentally discovered one answer to this question today. You see, Rent the Runway has been replatforming our systems for almost the last two years. We've moved from a massive, and massively complex Drupal system to a set of Java services fronted by a Ruby thin client. Part of the reason for this was load, although we arguably could've made Drupal handle the load better by modernizing certain aspects of our usage. But a major reason for the replatforming was that we simply weren't able to develop code quickly and cleanly, and the more developers we added the worse this got. We wanted to burn the whole thing to the ground and start fresh.

We didn't do this, of course. We were running a successful business on that old hideous platform. So we started, piece by piece, to hollow out the old Drupal and make it call Java services. Then, with the launch of Our Runway, we began to create our new client layer in Ruby using Sinatra. Soon we moved major pages of our site to be served by Ruby with Java on the backend. Finally, in early July, we moved our entire checkout logic into Java, at which point Drupal is serving only a handful of pages and very little business logic. 

So yesterday we had a hackday, our first since the replatforming. We do periodic hackdays although we rarely push the entire team to participate, and yesterday's hackday was one of the rarer full-team hackdays. Twenty-some tech team members and four analytics engineers participated, with demos this morning. I was blown away with what was accomplished. One project by a team of four enabled people to create virtual events based on hashtags and rent items to those events, pulling in data from other social media sources and providing incentives to the attendees to rent by giving them credits or other goodies as more people rented with that hashtag. This touched everything from reservations to checkout. We had several projects done by solo developers that fixed major nasty outstanding issues with our customer service apps, and some very nice data visualizations for both our funnel and our warehouse. All told we had over ten projects that I would like to see in production, whether added to existing features, as new features, or simply in the data dashboard displayed in our office. 

Compare this to the last full-team hack day and the differences are striking. That hack day had very few fully functional projects. Many were simulations of what we could do if we could access certain data or actions. Most didn't work, and only a handful were truly something we could productionize. The team didn't work any less hard, nor was it any less smart than it is now. The major difference between that hack day and now is that we've replatformed our system. Now adding new pages to the site is simple and doesn't require full knowledge of all the legacy Drupal. Getting at data and actions is a matter of understanding our service APIs, or possibly writing a new endpoint. Even our analytics data quality is better. 

So if you're wondering whether your replatforming has really made a difference in your development velocity, try running a hackday and seeing what comes out. You may be surprised what you learn, and you'll almost certainly be impressed with what your team can accomplish when given creative freedom and a platform that doesn't resist every attempt at creativity.