Why I wrote a Ray Tracer this weekend

“The game is to keep learning, and you gotta like the learning process.  I don’t think people are going to keep learning who don’t like the learning process.  And if you don’t keep learning, other people will pass you by. Temperament alone won’t do it – you need a lot of curiosity for a long, long time. "  - Charlie Munger 
Since my wife joined the law firm of Munger, Tolles and Olsen, I've become infatuated with Charlie Munger.  He's the mostly unsung hero of Berkshire Hathaway and Warren Buffett's long time partner.  Munger has a few great books out there filled with his wisdom, and one of his strongest encouragements is to keep looking for new things to learn.

My excuse is that I don't have enough time.  Between having a family, a full time job, and trying to keep up with the few friends I still can, I don't have the time to try as many new things as I'd like.  But that's an excuse, and with all things that take time, it's never that you don't have enough, it's that you don't make enough. Life is a prioritization exercise, and you have to prioritize what matters.

Having seen Toy Story 3 come out, I started looking into how computer graphics have evolved.  MIT has a great resource having their entire curriculum online with the Open CourseWare initiative, so I started taking 6.837 Computer Graphics.  It's great, they have the lecture notes, and the problems sets, and the problem sets are coding problem sets where over several assignments you write a ray tracer.  It was a fantastic re-introduction to a lot of Linear Algebra that I'd forgotten, and honestly, a bunch of C++ that I'd forgotten.  Given I work at Microsoft now, I figured I'd use Visual Studio to develop it to make sure I'm up to speed on the tool chain that most of the people on my team are using.

I'm about 70% through the class, and here are a couple images I generated.  Hardly rocket science, or works of art, but pretty fun for me.  I also tried to distill what I'm taking away from this effort:

1. Charlie Munger talks about learning the key ideas in a discipline and applying that to problems you see elsewhere.  The more key ideas you have, the better off you are.  For example, evolution is a key idea in Biology, and being able to apply that competition and survival of the fittest to investing strategies makes you a much smarter investor.  In Computer Graphics, the key idea is to reverse the direction of light, and trace the rays from the viewer's eye, back through the scene to the light source.  That touches on a 2nd favorite theme of Munger's, "Invert, always Invert".

2. It's been years since writing code was my full time job, and I find the further I get from it, the harder it gets for me to fully grasp the time required for each phase of the process.  Digging back in and writing code, and having to crawl through the debugger to find a simple math mistake is a refresher that every manager of software developers should take at least once a year.

A couple crappy looking rabbits, but I'm pretty proud of them.  I'd post the source to my ray tracer, but I figure I'll leave the MIT students to copy their homework from somewhere else.  I'm happy to send it to anyone who contacts me.

Joy - the ultimate incentive

My weight loss challenge ended in March, so I've set a new goal for myself, to run the San Francisco half marathon.  I ran the Boston marathon (without qualifying) the year after I graduated college, so I don't have to prove I can run a full one, and quite honestly, I don't enjoy running enough to suffer through weeks of 2-3 hour training runs.  But I know myself, and without a goal I have trouble motivating.

At the same time, I've been reading "Born to Run," a book describing  a tribe in northern Mexico called the Tarahumara.  This tribe has some amazing athletic ultra distance prowess as they regularly have races where they kick a ball from one person to another in a relay over a race that goes over 100 miles and lasts for several days.  Their approach to these races is amazing as instead of resting up, they party all night the night before the race, getting absolutely smashed on their corn based beer, and then shake off the beer an run miles and miles for several days straight.  The book chronicles how some enterprising Americans got them to come to the US to compete in ultra-marathon events like the Leadville 100, a grueling 100 mile race that hits two mountain summits.  The second year they entered, a 52 year old Tarahumara runner won the race, and the year after that a 25 year old Tarahumara set the course record that held up for 11 years.  And just as quickly as they dominated the race, they stopped participating in it.

As amazing as these people are, the book digs into the motivation of them, and in particular notes how much they laugh and smile, even as they finish the last marathon of an almost 4 consecutive marathon 100 mile race. Their spirit is often a crushing blow to their competitors who see the Tarahumara running lightly and with ease after 75+ miles, while the competitor themselves is struggling to remain upright.  The passage in the book that caught me talked about the motivation to run and how we as Americans have lost the joy of running.  We run to lose weight, to achieve a certain distance, to hit a particular time.  Much of the time running is spent wishing it was over.  The Tarahumara, on the other hand, run as a social event, the elders in front leading the way, and the young bucks chasing them down and pushing them faster.  It's more than fun, it's joy.  And when you run for joy and forget about the competition, it's easy and you actually go faster.

Ok, so there's a bunch of new-age hokum in there.  I don't subscribe to how much American's have lost the joy of running, and I also think some of the glorification of the Tarahumara is exaggerated.  But for kicks, on my 9 mile run this weekend, every time I started lagging a bit, I tried smiling and imagining running lighter on my feet.  Running lighter will force you onto your toes and generally quicken the pace.  And the positive effects of smiling are quite well known.  So how'd it go?  Well, 9 miles is still no picnic for me, but it definitely felt easier and at the same pace as some of my previous grinds had been.

This thought comes to me at the same time as I've had these couple of lectures by Dan Pink sent to me.  The first is a Ted Talk describing how using money as a motivation works well for tasks that are primarily physical in nature, but is actually negatively correlated to tasks where even minimal creativity or higher thought is required.  The second talk is an RSA Animate covering the same material.  I preferred the first as the examples are a little better, but the style of RSA Animate is hard to beat for entertainment value.

Dan's point is to get us to rethink our management practices where we rely on what economists have taught us for decades, that if we want to incent certain behavior, we incent it with money.  Dan's point is that the science doesn't back that up, and that instead, we should focus more on intrinsic motivation.  That people want to work on things because they matter, because they're interesting, because they're important.  Dan suggests we focus on three things:

  1. Autonomy - The idea that individuals should control what they work on as opposed to being directed by management.
  2. Mastery - The desire to get better and better at something that matters.
  3. Purpose - The yearning to do what we do in the service of something larger than ourselves.
I think Dan's dead on the money, and he's got the management science to back himself up.  I do see the gap in my own thinking.  I try to direct my teams to have Autonomy, Mastery and Purpose, but what I teach managers who work for me, and what I talk most about is incentives and overcompensating stars and cutting off underperformers.  

I find a strange parallel to the Tarahumara.  They have Autonomy.  They run because they want to and because their social circle wants to.  They have Mastery.  Running is something that matters in their society, and they are some of the best runners in the world.  They have Purpose.  They are serving the society, something larger than themselves.  And doing those three things brings them joy.

Can I possibly run my teams so they find joy in their work?  A tall order, but I'm a believer that it'll produce the best performers.  

As for my running, I still need my goal, and I still long more for the run to be over.  But I'm trying harder to find the joy in the moment.

How to avoid blowing your presentation

You've finally got the opportunity to present your idea to the right VC with connections in your industry and a track record of success.  You rush out to work on your presentation pulling in all the data, industry trends, snappy graphics, and entertaining anecdotes that support your position.  You conclude with the big ask on the last slide having lead everyone inexorably to your stunning conclusion.  You wind up with 30 slides to present in your 15 minute slot.

And predictably, what happens?  You get through through two slides, there's lots of debate around irrelevant topics and you never come close to getting your point made.  In the last 20 seconds, you race to your last slide on a Hail Mary hoping someone will see the brilliance of your plan from your asks alone.  The meeting ends with a couple nice handshakes and we'll get back to you, which of course they never do. It's a wreck.

Obviously, you blew it, but what went wrong?  The problem was you designed a presentation you wanted to present, not one your audience wanted to hear.  I see this mistake made all the time, and not just with VC presentations, with all presentations.  So now, lets stop and think about how we avoid this mistake.  Go back to the beginning before you even start your presentation.

1. What is your goal for this presentation?
If you aren't clear on that, you're dead before you start.  You want to convince someone to invest.  In other settings, you want additional funding for your project, or to reorganize a team a particular way, or to shift to a particular strategy.  There's some reason you've gathered important people and are taking their time.  You want them to do something.  Be crystal clear with yourself about that before you even get started.

2. What does your audience need to hear?
The second biggest mistake I see is that people present what they want to say.  Perhaps they're proud of a slide they put together, or of the way they implemented something.  They want to talk about how good they are at something, but if the audience doesn't need to hear that, then you've just wasted the time.

3. Be Hyper Aware of the time.
In your most important presentations, you've got an opportunity with the attention of the people that matter.  But you don't have much time, and it's probably hard to get more.  Time is your most precious resource, and you don't have much so think through how you want to spend every minute.  If you've got 15 minutes or 3 hours, plan out how you want to spend it.  In a 15 minute presentation, I'd probably think through something like this:
  • Get them interested (3 minutes)
  • Give your pitch (5 minutes)
  • Support it (2 minutes)
  • Questions (5 minutes)
4. Grab their attention.
Most presenters are boring.  Not a little boring, really boring.  Plus the environment of a darkened room with 10-15 people around a table and a 15 minute slot isn't conducive to paying attention.  Right at the start, you've got to demand that people pay attention to you. If you're funny, say something funny, but be careful because it's really got to be funny.  Lame jokes don't get people's attention, they're just lame jokes.  I often like to start way out in left field.  A few times I've opened presentations by saying I want to talk about a "Really Important Topic" and then flash to a slide with the Boston Red Sox logo.  And then I start talking about the Red Sox, and people are hooked.  How am I going to tie that back to my main point?  Do I have a point?  Plus, the story of how the Sox beat the Yankees in the 2004 ALCS can't be told often enough.

5. Get to your point
Now that  you've got a goal, and you're aware of the time pressure you're under, and you've got their attention, now you need to make sure you get to the point.  Think carefully about every slide you put in front of your ask, because that's a potential rat hole that will prevent you from getting to your goal.  It's often a good idea to lead with the ask unsupported and have all of your clever analysis and details follow that up.  At a minimum, it ensures that you'll spend the time talking about what you want to talk about.  I've seen many meetings where you open with your conclusion, and everyone starts attacking it, and their questions often lead right into your supporting slides.  "Where did you come up with that revenue forecast?" "Why, I have that on the next slide..." etc.

6. Use words sparingly.  In general use pictures.   
A presentation should be a multi-media entertainment experience.  The fact that you're talking about the important topic you care about is an engineered coincidence to the observer.  Here's my aforementioned Boston Red Sox presentation (with where I was really leading them cut off).  You probably can't follow it.  That's the point. If I'm writing a presentation for you to read, then I've got to use words, but if it's something for me to present, then I speak all the words.  And there's no excuse for not finding good pictures.  With Bing Image Search, you can find the perfect picture in three clicks.  Can you use it legally?  Probably not...but most of the time you're presenting to 10 people who don't care about that.  If you're a presenter for a living and getting paid, then you better find images you're allowed to use.

There's a whole host of other material on how to give effective presentations out there, and I really consider this the basic "101" version.  If you don't think about these points, it doesn't matter how you use your dynamic range, the graphics, fonts, colors on your slides etc.  Your job in a presentation is to communicate and lead to decision, and you have to engineer your time and your delivery to get to that decision.

Facebook's HipHop not fast enough

Facebook developers announced and released HipHop, a compiler that takes PHP code and outputs C++ code to later be compiled by g++.  I'll disclaim that I haven't seen the source code (still not on github as I'm writing this) and am basing my conclusions only on their blog post, but I think there won't be a "tremendous impact" from this project.  It's basic cost/benefit analysis.

The benefits are nearly great enough.  Only 50% CPU reduction?  Let's walk through the math of what that might mean for Facebook in terms of cost savings.  Their post says they do 400B PHP impressions per month, which works out to about 150k/sec.  That's a phenomenal number and a challenge for any site to perform at.  Facebook describes how PHP is used almost exclusively for the front end, and that the back end services are powered by Erlang, Java, Python, etc.  So if we assume they've got the front and back fairly well separated and can scale the front end based entirely on CPU load, we can make up some numbers to guess their total cost.

In my experience, even a poorly implemented PHP site can run around 20 requests per second on a single core.  Assuming Facebook uses beefy 8 core machines, that would be 160/sec per box, and 1,000 machines to serve their 150k/sec load.  Given that their load probably peaks with US traffic, and that they probably need double capacity to handle a co-location failing, I could see that going up to 4,000 machines.

Facebook has over 30,000 machines, most of which I'm sure are used to handle photos data and their tremendous memcached layer, so 4,000 web serving machines sounds about right.  At Facebook's scale, they can probably get a beefy 8 core machine relatively cheaply, but let's use $5k as a number, that means that saving 2,000 machines saves Facebook $10MM.  That's pretty impressive, but 2,000 machines compared against their total machine load of 30k isn't that impressive.  It's 7%.

Further, if we expect this to catch on and have a "tremendous impact" on the world at large, it would have to be a big benefit for all the small PHP sites out there.  I'd wager that almost all small sites are not bottlenecked on PHP CPU performance, but database performance instead.  Even if it was PHP performance, for most small sites the savings would likely be going from 30 machines to 15, which is almost unnoticeable.  For this system to be really effective, I'd want to see a speedup of 10-100x, not merely 2x.

So I've laid out that the savings aren't that great.  But they aren't nothing, a speedup of 2x which saves Facebook $10MM is pretty impressive.  but what does it cost.  I suspect the costs are pretty high and measured almost exclusively in terms of developer productivity.  Let's set aside the man year of development their lead developer spent developing this technology.  Facebook has to integrate this system into the development process, and there are two obvious ways.

1. Developers add a "compile" step to their development cycle and test using HipHop on their development boxes.  This, to me, would represent a worst case outcome (and from their blog post is almost certainly not what they do).  One of the many development speed benefits of scripted languages like PHP and Ruby comes from the fact that developers don't have a compile cycle.  While that compile cycle seems trivial (how hard is it to interrupt your code/test cycle with a 2 minute compile?) in my experience it's a large amount of the benefit.  Sure, dynamic typing is another major benefit, but I believe that if every Facebook developer now has to add a compile cycle, their overall productivity will plummet.

2. Alternatively (and more likely), they keep developers with an interpreted language on their dev box, and they move the compile step into the release process.  They indicate this in their blog post by talking about developers using HPHPi, their experimental interpreter.  The problem here is that you've now doubled your testing.  A developer writes his code and tests it locally using HPHPi, or even the standard PHP install.  After that, they have to compile it with HipHop and test it again to make sure the compiler didn't break anything.  If the compiler were a mature technology, you could probably skip this step, but with an experimental compiler you simply can't trust that it will work on your code.  Further, when you find an issue in production that you didn't anticipate, you're now not sure if it's the compiler or the code that is broken.  Given the difficulty of debugging live site bugs that often come about due to circumstances that are hard to duplicate in the test environment, this seems like a disaster.

So we're left with a system that doesn't provide enough benefit for the costs that will be associated with it.  A neat toy, but if I were Facebook, I would value the developer productivity way over the $10MM I might be able to save in server costs.  What do you think?

On Mark Pincus's NY Times interview: Responsibility and Accountability

Mark Pincus, founder/CEO of social gaming company Zynga, is interviewed in the NY Times Sunday on the topic of leadership and he touches on a couple of my favorite topics: delegation, responsibility and accountability.

You can manage 50 people through the strength of your personality and lack of sleep. You can touch them all in a week and make sure they’re all pointed in the right direction. By 150, it’s clear that that’s not going to scale, and you’ve got to find some way to keep everybody going in productive directions when you’re not in the room.

I met Mark once, and from what I could tell he operates at a frenetic pace all of the time.  So perhaps at that frenetic pace he can touch 50 people in a week, however the number for most mortals is much lower.  I max out around 20.  So if you plan to scale, you have to delegate effectively, and that's one of the biggest challenges people face when they turn into 2nd level managers.  You need to delegate completely so that you don't have to worry about the details of a particular function, but you also are still responsible for it being done properly.  So how do you know if the person you've delegated to is up to the task?

You have to first make it clear to everyone what they are responsible for and give them all the tools they need (Gallup Q12 #1 and #2).  One of the most important tools is the ability to make their own decisions in that framework (Zod axiom #4).  Once you've made the responsibilities clear, you then have to hold people accountable.  You have to set up the right mechanism to check in on people and make sure they're getting the work done properly so you can catch mistakes before they veer too far off course.  Setting milestones and regular reviews of them are key.  You trust your people to perform well, but you check on them anyhow.

Pincus continues:
One thing I did at my second company was to put white sticky sheets on the wall, and I put everyone’s name on one of the sheets, and I said, “By the end of the week, everybody needs to write what you’re C.E.O. of, and it needs to be something really meaningful.” And that way, everyone knows who’s C.E.O. of what and they know whom to ask instead of me. And it was really effective. People liked it. And there was nowhere to hide.  
Here I like the concept, but not the implementation.  By calling people C.E.O.'s he's making it clear that all of the responsibility and accountability goes to the person.  If you're CEO of selecting a phone system, then you are tasked with gathering the input from everyone, evaluating all the vendors and making the decision.  Important here is that you own the decision, there isn't a committee of deciders among which to share the responsibility.  The credit/blame resides fully with one person, and that provides accountability.

His implementation leaves something to be desired, however.  He's asking his team to sign up for responsibility and then he'll hold them accountable for it.  That will work provided that no two people sign up for the same responsibility, and that all of the responsibility you want to delegate gets assigned.  If two people both agree to be CEO of phone vendor selection, then now you've got a committee and shared decision making, which just doesn't work.  Further, if nobody signs up to select the phone vendor, then either you're left with that responsibility, or (more likely) the work won't get done.  So as a manager, you want to solicit input from your team on what they want to own, but ultimately you own the responsibility for dividing up your responsibility in a way to ensure everything is covered and nothing is left undone.

Maslow's Hierarchy of Employee Needs (with help from Gallup)

I previously talked through Zod's axioms, which form the cornerstone of my management philosophy.  However, they aren't the complete set as I've added a few things to how I think about management.  Consider this the second pillar of my overall management philosophy.

The Gallup organization surveys companies all over the world, and once they had amalgamated data from over 1,000,000 employees, they dug into their data and did some research.  Their goal was to distill management principles that were the most predictive of a successful company.  The result was Q12, a set of 12 questions that best predict company success based on the engagement of their employees.  While Gallup uses them as a measuring tool, it's interesting how many of them form a checklist to management to ensure they're creating the right environment.  I find these questions are something of a Maslow's hierarchy of needs for employees, ranging from the most basic necessities to some of the last few which I don't worry as much about.  The questions are all true/false and phrased as statements from the employee.

1. I know what is expected of me at work.
This is so basic and obvious that managers assume they have explained this already, and often haven't.  If an employee isn't 100% certain how he's being measured and what you expect of them, they have almost no chance of delivering against it.  Especially when someone is joining a new organization it may not be obvious what falls in their domain or not.  Is setting the schedule the engineering manager's job?  or the product manager's?  How about prioritizing the next 3 months? Is the current product critical and has been promised for customers?  Or a research project?

Regular reviews are critical for delivering on this statement.  New employees particularly should have quarterly reviews, and once they're got the rhythm you can probably fall back to semi-annual.  Reviews should cover clear goals of what you expect should be done, and evaluation of how they've done against those goals in the previous period.

2. I have the materials and equipment I need to do my work right.

If the first statement is water, this one is food. Knowing what you need to do and not having the tools to get it done is a recipe for frustration and resentment.  Again, it's so obvious as to be frequently overlooked.  Perhaps your Unix developers need a side windows box to read the Word file requirement documents.  Or Valgrind doesn't work on your platform so you need to find an alternate memory leak detection tool.  Your job as a manager is to remove obstacles for your stars, and providing the right tools is one of the simplest obstacles to overcome.

3. At work, I have the opportunity to do what I do best every day.
This statement echo's Zod's first Axiom that people don't change. Since people don't change, the job of a manager is to put people into a role that plays to their strengths, allowing them to do what they do best every day.  Take the person who is super anal, and put them in charge of a very complicated project management task with lots of details.  Give your creative person an open ended design problem to solve.  Not only will they do it well, but they'll love their work since everyone loves doing something they're good at.

The challenge here is that you have a set of work that needs to be done that usually doesn't match up 100% to the skills you've got on your team.  That's the art of management.  Understanding the principle that you want to assign roles based on what people are great at, you will also have to get people to do work that doesn't match up 100%.  The key to me is "every day" not "all day". There will always be work that doesn't match your interests, but you should have some time every day to work on something you are great at.

4. In the last seven days, I have received recognition or praise for doing good work. 
5. My supervisor, or someone at work, seems to care about me as a person. 
6. There is someone at work who encourages my development.
I lump these three together as the Maslow equivalent of "Love."  Does someone care about you and work to make you better.  #4 is so basic and easy, and one that I struggle with.  I'm just not a cheerleader and I assume that people know that I appreciate the work they're doing.  But it's so cheap and easy to tell someone "great job on that feature!" that there's no excuse not to.  #5 and #6 are reflections of Zod's axiom that you are only as good as your lieutenants. If you hope to develop and take on more responsibility, then you need to be encouraging your lieutenants to develop and care about them as people.

7. At work, my opinions seem to count.
This argues for a transparent decision making process where input is broadly solicited.  Oracle is well known for having a top down decision making process.  The top exec team decides on the priorities and strategies and maps those down through the organization.  Google, on the other hand, is known for a bottom up decision making process.  I've heard managers there complain that they can't even get their own team to work on something.  To ensure that people's opinions count, it's got to be clear where to express them to have an impact on the company.

8. The mission/purpose of my company makes me feel that my job is important.
This simple sentence really needs it's own post.  It's super important for every person on the team to know how what they're working on maps to the short term projects, and how those map to the companies strategy, and how that maps to the overall mission of the company. 

9.  My associates are committed to doing quality work
While not immediately obvious, this is a corollary to Zod's not being a communist in compensation.  Your job as a manager is to build a high performing organization, and it's critical to highly reward the high performers, and to get rid of the low performers.  Jack Welch talk's extensively about 20%/70%/10% and the culture of rewarding high performers and getting rid of the bottom 10% each year.  While it may seem a little harsh, letting the low performer remain in your org will lead to people thinking their peers aren't committed to doing quality work, and thus they don't need to work as hard.  It's a cancer in your organization that will spread if you don't get rid of it.

10. I have a best friend at work

11. In the last six months, someone at work has talked to me about your progress.
12. In the last year I have had opportunities at work to learn and grow
The rest of the list.  11 and 12 are a little redundant with 4-6.  10 is something that comes about when the culture is excellent, but I see it as more akin to self actualization than driving actions for the management team.

I use this list as a checklist, particularly at review time.  I run through the list and try to answer for all of my directs, and for the key people in their orgs.  Any areas that I find I don't have good answers to create a clear list of actions I need to take.

Doing incentives right

I joined a group of my friends in a 90 day body fat challenge starting this January 1st.  We each pay $100 into the pot, and there are monthly winners and the overall 90 day winner takes the bulk of the money.  We're two weeks in and I'm pretty impressed with the group average being 3% weight loss, and 13% body fat loss (that's percentage of a percentage, so 22% body fat down to 20% body fat).

We'll see how it goes, but I am fascinated by this graph.  It's the Google Trend graph of "fitness" with a two week spike coming each January as people make their New Years resolutions.  If that's any predictor, then most of the people in my contest should start dropping out over the next couple weeks.  I suspect, however, that the money component people paid in should change the outcome significantly, as I suspect people will be motivated to try harder.  Further, I think the weekly public spreadsheet we're keeping of everyone's progress will also better motivate people.

It's about setting the incentives properly.  Behavioral economists have long said that if you design the incentives properly, you can get people to do anything.  I even think the incentives in our contest are poorly designed.  I expect in a few weeks a small group of people will be way out in front, leaving the rest of the pack with little to no chance to catch them.  People will then see their money in as a sunk cost, and essentially drop out of the contest.  I proposed that instead of an entry fee and prize, that we go to a tax model.  For each person, if they lose 20% of their body fat, then they pay nothing.  If they lose 15%, then they pay $25, 10% pays $50, 5% pays $75 and 0% pays $100.  That way, even if you were losing late in the game, you still have an incentive to continue dieting and exercising.

Incentive design is a tremendous problem in the workplace.  I'll run through just a couple examples that are fairly well known:

Empire Building
In reviewing promotions within the management ranks, one of the first things that is reviewed is the size of a manager's organization.  HR will regularly say that org size isn't an important criteria in getting promoted, but the truth of the matter is that it is a very quick gut check on how much span of responsibility a person can handle.  Someone who is succeeding with a 200 person team can likely handle more span of responsibility than someone who is succeeding with a 20 person team.  However, as soon as this gets recognized within the ranks of an organization, you can count on the following behaviors:
  • Turf wars as people compete to keep all parts of their organization, regardless if they make sense or not.
  • Low performing orgs.  This isn't as obvious as managing out low performers typically results in the manager getting a backfill to upgrade.  But if you have high performers, you need fewer of them to do the same work, so perversely, you're incenting low performing orgs.
  • Refusal to do anything new without additional headcount.  New projects are typically the only time when new heads are given to teams, so teams routinely cost out a new project, but say that they can't get it done within their own team.
I've seen all these and more.

Management by metrics

Many management guru's have preached the value of metrics with one liners like "Inspect what you Expect!"  I couldn't agree more with the practice as metrics focus you on the areas you really care about, and give you a yardstick by which to measure your progress.  The problem with most metrics is that they are imperfect and can be gamed.  In the early days at Yahoo, one of the most important metrics we reported to wall street was page views.  Gaming this metric is trivial because you can design a flow that has extra steps in it.  Logging into mail takes 4 clicks to get to your inbox instead of 3.  This metric improvement comes at the expense of your user, but if this is the system you set up to reward (either directly, or what wall street sets up), then you can bet the incentives are in place to drive this behavior.  This line of reasoning makes me very suspicious about the current search share war that the press is focused on with Bing and Google.  In search, doing fewer searches is usually indicative of a better product (you found what you were looking for).  Too much focus on metrics can eventually doom them.

Sales targets

Sales people are usually given a quota and given a bonus based on how well they do on a quarterly basis at achieving that quota.  Typically they have a target bonus for hitting 100% of their quota, and then a scale for missing or going over.  Sell 50% of quota, get 0% bonus.  75% of quota, get 50% bonus.  150% of quota, get 300% of bonus, etc.  Since the top and bottom of the incentives are usually capped at 50% and 150%, this can lead to incenting tom strange behavior.  If a salesperson is having a great quarter and has already hit the 150%, then they have no incentive to sell more this quarter, and instead have an incentive to delay sales to next quarter.  Or if a sales person has sold 95% of their quota, they may call their accounts to get them to move next quarter's order into this quarter to get to the 100% bonus.  Or if they're selling terribly, they may decide to take a bath on this quarter and come in below 50% and delay all pending sales to next quarter.

The key lesson to take away from here is that the incentives you set up for your employees are what they will work towards.  Even if they're trustworthy and high integrity people, there have been many many studies showing people will always shade towards their own self interest when incented to do so.  So be very careful designing your incentives.  Going back to the body fat loss challenge, the biggest concern I have is that it's on the honor system, and measuring body fat is an inexact science.  The method we're using is based on measuring your girth in several places.  I suspect that come March, those tape measures start squeezing bellies in a bit more than they did on Jan 1st.

That said, if the $100 I paid continues to motivate me as much as it has for the past two weeks, then it will have been money well spent.

Thoughts on the power of founders

In response to the Google decision on China, Fred Wilson writes about the Founder Factor and how that leads to bolder decisions.  While certainly not on the same scale as Google's decision to potentially pull out of China, I thought I'd relate that to a few of the experiences I had at Mochi Media.

On Monday of this week, Mochi announced that it was being acquired by Shanda for 80MM.  This is a fantastic outcome for the company, and I'm super excited for the team there.  I wanted to review two critical decisions that were made in the past year that led Mochi to this acquisition as opposed to alternative paths that they could have taken.  For the record, I was on the wrong side of both of these decisions but fortunately for the company the founders pushed for and got the right decisions made.

A little background, Mochi Media is a company that provides tools for flash game developers, and in particular, tools to embed ads directly into the flash game.  A game developer would write a game in flash, and then come to Mochi for a couple lines of code to insert into the game to add advertising to it.  Publishers would come to Mochi, where they were able to get tens of thousands of games for free to use as fantastic content on their game portal or other site.   There are many examples of Mochi powered games, and the format of pre-loading video ads works particularly well for advertisers.  I joined in April of 2008 to head up the engineering team there.

1. Picking the right new market
Around December '08/January '09 Mochi was thinking through how it wanted to expand its business and was considering a couple of new areas to enter into.  One possibility would be to create a new virtual currency and make it easy for our game developers to integrate with it.  Virtual currencies were quickly becoming all the rage, and companies like Zynga and Playfish were making tens or even hundreds of millions of dollars from them.  This model would be very aligned with Mochi's traditional flash tools model where we are an infrastructure provider and the developers would still be responsible for creating the content that would be critical to our success.

Several other alternatives were discussed in detail, and a few prototypes were built as well.  I won't go into too much detail on what they were, but one of the key differences was that it didn't play to our strengths as an infrastructure provider nearly as well as Coins did.  But we had started moving down that path, and were preparing to jump in with both feet when Jamseon, one of our founders, started pushing hard for us to drop the alternatives and move into Coins.  There was a lot of internal discussion, and I personally wanted to continue down some of the other options we were exploring since we hadn't seen them to their conclusion and would be making a big bet the company decision and throwing that other work away.  The key questions up for debate were:
  • How much should we focus on playing to strengths we already had (building infrastructure) vs getting into areas that we'd probably need to hire new people for.  Anything from SEO to customer service that we simply didn't have expertise in.
  • How could we protect the ecosystem of developers and publishers and advertisers we'd worked so hard to cultivate..
  • Finally, there was a fundamental debate about the potential upside of coins vs our other options.
We discussed extensively internally with the entire company, and certainly over my objections, we made the choice to pull the plug on our other project and deploy all of our resources into building out Coins.

In hindsight, it was absolutely the right choice.  As soon as we started down that path, interest from potential acquirers heated up because Coins added so much more strategic value, whereas other options added none.   And our developers made many fantastic Coins enabled games.  As for the financial value, all I'll say is that I've heard it's doing quite well.

2. Picking the right acquirer
As I mentioned, after we released Coins, we had interest from a number of strategic acquirers.  We got pretty far down the road with one of them, and some of the outcome of that was covered on Techcrunch.  While the battle that Techcrunch portrays is way overblown, there was a legitimate difference of opinion, and Jameson and Bob under pressure from me to do otherwise made the courageous decision to stay independent.

That's the point where I left Mochi.  I left on good terms and keep in touch with the people there, but having been on the wrong side of two decisions that were very strategic for the company, it became clear to me that I didn't have the same vision for the company that the founders did.  And now several months later, I'm happy to see that Jameson and Bob got to a fantastic outcome, much better than the one discussed earlier.

Without the Founders there to make courageous and bold decisions, bucking the convential "wisdom," Mochi would have wound up in a much different, and in my opinion, much worse position.  So I heartily congratulate Jameson and Bob.  They stuck to their guns and made tough bold decisions as only Founders can, and Mochi is better for it.

Zod's Axioms: 10 Principles of Engineering Management

Years ago when I was just starting to figure out how to manage, Zod Nazem, the CTO of Yahoo, held an engineering management summit and at the end of it he presented his 10 management axioms. Background for those who don't know Zod, he's absolutely brilliant, tough as nails and one of the best leaders I've ever seen or worked for, but he's not the strongest public speaker. This day was the exception, people were blown away with his axioms.

That hour had a profound impact on the way I manage (and I expect much of the Yahoo engineering leadership in attendance that day). I've been meaning to write this blog post for years but instead kept spreading this philosophy byword of mouth.  Finally I've gotten them down and I want to thank Zod for looking over it to make sure I stay true to his original intent.  These are all short and pithy sayings, and all of them seem like common sense, but as I've stated, you need to go beyond knowing your principles and master them.

Without further ado, here are Zod's axioms.

1. People don't change.
2. People can learn to become better "actors"
3. Under stress people revert back to their true self

I put the first three together because I think 2 and 3 are corollaries of #1 and they all refer to the intrinsic nature of people. I had an employee who was a terribly long winded communicator.  It just wasn't possible for him to get to the point in less than 30 minutes (and he always cornered me as I was headed to my car).  I worked with him on being more concise for years, and over time when he focused on it he was able to "act" as someone who was a better communicator.  But when the pressure was on and a big decision confronted us, the old habits came out and all the work we did was out the window.  Fortunately, he was a brilliant coder and concise communication wasn't a large part of his job.  Often times, the brilliant coder who communicates poorly is pushed (or pushes) to move into a management role where communication is a critical skill.  Had I done that with this guy it would have been a disaster.  I want to draw a distinction between intrinsic characteristics and skills.  Any good programmer can learn a new programming language as that's simply a new skill.  But people come pretty hard wired on their underlying intrinsic characteristics, and many a manger has foolishly tried to change people when they need to accept that they can't.  People are intrinsically organized or not, prompt or not, lazy or not, high energy or not.  Needing someone to change on a dimension like that is a recipe for failure. 

4. Make quick decisions knowing you will make mistakes
One of the best ones on the list.  Your job as a manager is to make decisions.  You will never have complete information, and in fact part of the skill of your job is making strong decisions in the face of ambiguous information.  Time estimation, Promotions, Strategy calls, Architecture decisions.  You'll get some wrong, and fix them quickly, but the best managers make decisions quickly.

    5. When it comes to compensation, don't be a communist.
    Your best employee at a given level performs 5-10x better than your worst employee at a given level.  I call them 10xers.  But that best employee is still only paid 10-20% better.  Almost no company has the compensation practices to deal with that (Netflix claims they do (slide 97), and from what I've seen, they might).  So when the company 3% raise pool comes to you, don't be a communist and spread that evenly to all of your people, give almost all the money to your stars and give no raise to your bottom 30-40%.  Will that hurt their morale?  Perhaps, but it's so critical to keep your stars happy, and you really don't have enough cash to give them, so you've got to take it from somewhere.

    6. Spend all your time removing roadblocks from your stars
    So many new managers think they can fix their bad employees.  They spend hours with them coaching, pushing and trying to make them better.  That's a mistake.  First and foremost, I think it's the employee's responsibility to figure out how to excel in the company, the manager is supposed to define his responsibilities, give him the tools and opportunity to be successful and hold him accountable.  Further, at best you'll make  your poor performer a bottom of the barrel acceptable performer, but investing the same effort in your stars you can move them from a 5xer to a 10xer by helping remove the obstacles.  Get approvals for them, software/hardware they need, decisions made, cross org support, etc. etc.  If there's something blocking a star, that's your top priority. 

    7. You are only as good as your lieutenants
    You're probably a fantastic developer/architect and you can probably do the work of your team better than they can.  But you can't scale, you will always only be able to get one person's work done.  It may be 10x as productive as low performers, but you'll never get the work of 100 people done.  So you need to have lieutenants who can take the direction and skill you've got and apply it more broadly than you can.  And if you have great lieutenants, they'll push you up and take over work that you had been doing allowing you to focus on the next bigger problem.  And if they're bad, then they'll pull you down to do their job for them.  And when you're doing the job of your lieutenants, you have no capacity to take on more and scale further, so you're stuck.  Once you really start to embrace this, you'll start hiring people who are better than you, an intimidating but tremendously important step to make.

    8. If you are not failing, you are not trying hard enough.
    I tend to reword this to "You'll do your best work when your uncomfortable with your situation."  I'm a skier, and when I was learning to ski, I fell all the time.  That's the only way to get better, you have to push yourself into a situation that you aren't comfortable handling and do your best with it.  You'll screw up, make mistakes and hopefully learn from them.  So if in your professional life, you don't ever fail, that means you're being too conservative and not putting yourself into the uncomfortable situation.

    9. If you are behind the competition, change the game.
    Back in 1998 Goto.com was an also-ran search engine that was way behind Yahoo, AOL, Altavista etc. (pre Google).  They were overrun with spam and generally a pretty poor service.  They changed their model from trying to have the best search results to ranking results based on how much each bidder paid.  This was the creation of the paid search market which Google now earns billions from each year.  Goto morphed into Overture and got bought by Yahoo.  But instead of trying to beat the established players at algorithmic search, they decided to play a different game that they could position themselves to win in and then everyone was playing catch up to their game.

    10. Don't pass your garbage to your neighbor.
    If you have a low performer on your team, you are not allowed to transfer them to another team.  You must fix your own problems, either by performance managing them until they perform better, or by working them out.  It's so easy to just let a poor performer transfer, and big companies do this all the time, but it's death.  Before 2006, a poor performing teacher in San Francisco could jump to the top of the line above qualified candidates to transfer into a different schools.  So Principal's were left shuffling the "lemon" teachers around until the law was passed.  And we wonder why so few people want to send their kids to San Francisco schools...

    And that's it.  10 pithy rules to manage by.  I've added a few others to my list (another post) but this is a pretty good basic starting point on how I try to operate.

    Blackjack and engineering management

    My 15 minutes of fame are up.  I was on the MIT blackjack team, and they made a book and a movie about us.  So many people I talk to are absolutely fascinated by the blackjack stories and they all want to know how we're so smart that we can remember every card in a 6 deck shoe.

    We can't.  Well, at least I can't.

    Counting cards is really simple.  So simple that I can explain it to you in 5 minutes (see bottom of post if you want to know how).  The math isn't harder than division of fractions.  Once some people realize this, they set off determined to go to the casino and win lots of money.  And that's foolish.  Because even though they know how to count cards, to do it successfully, it has to be so ingrained it's second nature.  I always compare it to driving.  Driving a car is a really complex action, and most of us do it so often, that we don't even think about it.  We can hold a conversation, play with our ipods, send text messages (don't do that) all while performing a really complicated task.

    Jeff Ma, Me and Jim Sturgess

    Counting cards has to be at that level.  We would practice so much that I never had to think about what to do, it was 2nd nature.  Basic strategy was automatic and non-thinking.  I'd hit or stand without even realizing I'd done it.   I'd talk to the dealers about the big fight the next day while calculating ((17 / 3.33) - 1) * 800 = 3300.  And that was critical because the distractions that happen in the pressure of a real casino require counting to be a background task.  I'm 100% focused on the pit person I'm talking to and convincing them that I'm a half drunk partying rich guy, and making sure they're not calling upstairs or saying anything about me.  Oh yeah, and I've also got to count and play blackjack. 

    I find management to be the same as counting cards.  My management principles are so simple as to seem like common sense.  I can certainly explain them to you in 5 minutes (a future post), and many newly minted managers think it's all so obvious that they know exactly what to do.  It's why I find all those management classes companies offer so terrible.  They tend to focus on a lesson like "Make sure your employees know what they're expected to do."  Duh.  Do we really need to take a class on that?

    What we really need is the practice that makes it 2nd nature.  It's hard enough to get a project on track and team members motivated and delivering that you need 100% of your energy focused there.  So the performance management, career planning, making sure you've got the right people in place, making sure people know what's expected of them, etc.  That's all got to be subconscious.  And until we're there, we need to make time every day to review the list and make sure we're on top of all of them.

    It's the difference between having learned something and having mastered it.

    P.S. So how do you count cards? When the shoe starts, start a running count in your head at zero. Every time you see a 2-6 (low card) add 1 to your count. Every time you see a 10-A (high card) subtract 1 from your count. Ignore 7-9.

    At the end of the round, you need to figure out what to bet. First calculate the "True" by normalizing the count by the number of decks remaining. Divide your running count by your estimate of how many decks you have left to play.

    Then bet True - 1 units.

    For example. After playing through 1 deck of a 6 deck shoe, lets say your running count is 15. That means you've seen 15 more low cards than high ones. You divide by the 5 remaining decks and get a True count of 3. That means for each deck reamining, you expect to see 3 extra high cards. Then subtract 1 (since the casino has an advantage off the top). Bet 2 units. If your unit is $100, bet $200.

    This is a decent more detailed explanation.

    How to reduce testing time and still improve quality

    This is the conversation I've had the most frequently since joining Microsoft a few months ago.  The sides of the debate are framed like this:

    People who have had experience in the world of online services will say that release time frames need to be short to remain competitive.  They'll cite agile methods with lightweight planning, and generally very light test plans.  People who have come from the packaged software (windows/office) side say that you end up cutting the corner on quality and by having longer releases can get more work at higher quality done.

    Today I'll focus on the quality angle.  I argue that rapidly released software can be higher quality with a lower QA cost.  You can have your cake and eat it too.  To think through this, we need a measure of software quality.  That's an elusive problem, but for my purposes, we don't actually need to be perfect.  I propose:

    • Software Bugginess = Sum of cost of all bugs (that seems obvious)
    • Cost of a bug = Severity of bug * number of user sessions encountering it.
    • Severity is loosely defined as the amount of pain a bug causes to a user (and the part that is hard to quantify).
    Now in the packaged software world, once the software is burned onto DVDs or pre-loaded on new computers for purchase, it's effectively frozen.  Oh yes, there are patches that get released, but those come with a fairly intrusive cost to users and are never 100% applied to all your customers.  So as a software development manager, to reduce the bugginess of your software you have to focus on getting rid of all high severity bugs.

    Now I have no science to back this up, but I'm pretty sure most QA processes lead to a graph similar to the one at left.  Most of the bugs are ferreted out in the early phase of testing, but to getting rid of all high severity bugs becomes a decreasing return on investment.  So packaged software development managers must insist on a long test cycle to ensure high quality in their software.

    Online services, however, have the beautiful property that you can update the software for all of your users by simply updating the server.  While that isn't always a trivial process, it is completely in the software development manager's control.  So now, instead of focusing on getting rid of every last high severity bug, we can focus on the other side of that equation; reducing the number of user sessions experiencing it.  That is a huge win because we can just do the first part of the QA curve where we find most of the bugs without paying the long price to get near 100% of the bugs out.  Further, there are lots of techniques for reducing the number of user sessions experiencing a bug, from rolling out the release slowly to a subset of servers, to doing daily bugfix releases to fix the bug you created yesterday.

    But here's where most people screw it up.  People hear about agile and moving fast, and decide that they too must move fast to be competitive.  And they get excited about lowering their quality bar since it's easy to fix the bugs the next day.  And they hire only developers since all small companies should require developers to be testers (a subject of a future post).  And the developers rush out code they've barely tested at 7pm on Friday.  Then they go home, and are startled to get the SMS at 4am that their new code has broken.  So they resolve to fix it, and test it more carefully, and getting the fix and the test out takes until next Wednesday.
    I see lots of places where they lower the quality bar, but don't take the next step of fixing problems quickly and generally release crap.  And thus was born the idea that agile development processes tend to be low quality.

    But it doesn't have to be this way.  To release often and have a lower QA bar to release you must do several things:
    1. Have a simple and automated release process.  You plan to release often, but if it's cumbersome or manual, you will fail at this and spend too many resources doing releases.
    2. Have a simple and automated rollback process.  You're making a decision to roll out software that hasn't been rigorously tested.  You're definitely going to make mistakes and you'll need to fix them quickly.  Making sure that rollback works should be a part of your release to the staging environment before you release to production.
    3. Have a segment of your production site that you can release new software too without having to release to everyone.
    4. Have a source control environment that 100% replicates your production code, and is separate from your bleeding edge development environment.
    5. Plan to release regularly, and insist on fixing bugs the same day or next day they're discovered.
    6. Measure the number of days in production you've had for your various severities of bugs.  If you don't measure it, you will never fix it.
    It's a culture and a mentality to do these things, but they're basics and really 101 level fundamentals on running a fast moving development organization.  But doing them, and focusing on the number of days in production for severe bugs, you can actually achieve higher quality systems than packaged software that has undergone a 6 month testing cycle.

    We don't need another blog!

    Rational Self: Really?  You're going to start a blog?  Aren't you 5+ years behind that trend?
    Me: So what?  I've got important things to say.
    Rational Self: Important?  Things that nobody else has already said 100 times before?
    Me: Well, the other people saying them weren't me.
    Yeah, I get it, the last thing the world needs is another technology blogger.  And Lord knows I don't really have the time to keep this up to date.  But I've got some ideas I want to get down on paper.  Ideas I find myself regularly regurgitating to people, and I think it'd be simpler to point them to a good write up of it.  And since "Writing is Thinking" it'll allow me to better coordinate my thoughts.