Promotion Systems

When doing research for my book, I interviewed several Googler old-timers. What surprised me was that two ex-Googlers said the same thing pretty consistently: they felt that the Engineering Ladder was a very bad idea. At the time, I disagreed. I believed that in general, incentives systems worked and it was a good idea to spur everyone along with carrots. Since then, I've come around to agreeing with these two very smart people.

What changed my mind? The first was my reading of the book Drive. To a large extent, software engineering is a field full of ambiguity and complexity, and that makes compensation hard. So incentives tend to fail or backfire. Then I tried to invert the problem. What if I tried to design a promotion system to piss off as many employees as possible? What characteristics would it have?

• No pleasant surprises. In other words, you can only be disappointed if you didn't get a promotion, you can't be pleasantly surprised by a promotion.
  
• Create unhappiness by dependence on scarce resources. In other words, gate promotions based on scarce resources so that even people who would otherwise be qualified could become disgruntled through no fault of their own.
  
• Eliminate accountability from people who make the promotion decisions (e.g., through a committee). That way, promotion decisions can seem arbitrary.
  
• Ensure that promotions are competitive races between all qualified candidates. This ensures that people who manipulate that packet in such a way as to have the best looking packets will win over people who are trying to get feedback and improve, which is supposedly the point behind all these feedback systems.
  

When I looked at Google's promotion system through this lens, I was very impressed. It seemed as though the system was designed to create disgruntled employees out of people who might otherwise be perfectly happy. And note that the ex-Googlers I interviewed were people who benefited from the system, as in: they were very well respected and were hence ranked highly. They were people who couldn't care less what their ranking was, yet they disliked the system anyway, purely because they thought that having a ranking system actually had a deleterious effect on the exchange of ideas. Note that at one time the system did work: when the company was small enough that everyone knew what everyone else was working on (more or less), and hence felt accountable when making promotion decisions. The company also had so few managers that they didn't have time to put people up for promotion anyway, so putting the onus on the employee made sense. (Actually, at one point the promotion committees would consider everyone for a promotion, not just the ones who applied) Furthermore, if you tried to game your packet, chances were, someone on the promotion committee would know, so it was futile. This went away when the company got big, and got so many managers that there was no excuse for your manager not to say good things about you if he wanted to see you promoted. Someone did tell me that he thought that this evaluation of the system was biased: the alternative is the traditional management system, where everyone tries to suck up to the boss. When I discussed this with Reed Hastings, he said that he's still the biggest fan of the traditional management system executed well. When I thought about it, that makes sense, because of the following:

• You have to select good managers anyway. Even in a peer based system, a bad manager can still sink good engineers by selecting bad goals, or blatantly playing favorites.
  
• Managers are typically incentivized through business goals. If they deliberately promote poor engineers, good engineers tend to leave their teams and the business goals will be much harder to achieve. In general, most companies do a very good job of properly aligning managers with the corporate objectives. It's much harder to do so for rank and file employees.
  
• A manager can be held accountable for her promotion decisions. It's much harder to hold a faceless committee accountable.
  

No matter what promotion system is in use, there'll be flaws. Worse, these flaws are insidious. Since superstars are those who get promoted frequently, they see nothing wrong with the system. In fact, they'll usually try to torpedo any changes that don't benefit them, and since they're the superstars, their opinions will carry more weight. Instead, it's the misfits who see the flaws of the system most often. Talented misfits will stick around for as long as compensation is adequate, and leave once it's not. And any manager will tell you that once compensation is the only thing holding employees to a company, you're in a very tenuous and dangerous situation. Silicon Valley has a history replete with companies who were stripped nearly bare of talent by smaller startups that had the ability to offer higher compensation (usually through stock options) to under-valued employees. Obviously, if you're one of the misfits, you should check your compensation package often and leave the moment it's inadequate.

Ultimately, having to have an engineering ladder at all is why I consider startups better places to work. If you've negotiated substantial stock compensation, then it doesn't matter whether or not you're a misfit or superstar under the promotion system. If the company does well, you're going to do very well. If the company does badly, even the superstars aren't going to be adequately compensated. That makes doing the right thing much easier for everyone, and explains why startups tend to have much less office politics than bigger companies.
[Update: Follow up Post]

Selling Stock Post-IPO

Google was unusual in that it had 3 lock up periods: a 30 day lockup, a 90 day lockup, and the 180 day lockup. The idea was to let pre-IPO stock (and option) holders sell 5% of their holdings after 30 days, an additional 10% after 90 days and any amount after 180 days. The other company I heard of which got an exemption from the usual 180 day lockup rule was Netscape. In my book I discussed some of the pitfalls behind early selling, but here I want to talk a bit about selling strategy.

The first thing I observed was that the float at the time was tiny. This meant that it was conceivable for Google employees who are divesting to move the stock. I also observed that employees (especially employees with large stock holdings) were mostly housed in Mountain View. Many of the large stock holders were engineers, and were Owls, rather than Larks. Since I was effectively locked out of the 30 day lockup due to my vesting date, I got to observe market movements.

Indeed, what I saw was that the highest prices of the day tended to come before 10am PST, and the stock would take a consistent drop after that and stay low for the rest of the day, before another morning spike the next day. I saw this repeated day after day, week after week, and that convinced me that my theory was right.

Obviously, for me to take advantage of the idea, I had to ensure that I was the only person who would execute on it. Otherwise, everyone else would pile on and I would lost my chance to capture the morning spikes. So I deliberately unsubscribed myself from the internal financial-planning discussion mailing lists, and deliberately kept myself from blogging during that period, internally or externally. And yes, I did capture the morning spike when I finally did sell (I'm a lark by nature, so getting up at 6:30am PDT/PST to trade wasn't a problem).

Note that this only occurred at the 30 day and 60 day lockups. At the 180 day lockup periods, institutional investors/venture fund holders could sell, and those guys could move huge chunks of stock in a day, and employee movement would be lost in that noise.

In late 2007, I did the inverse. I took an informal poll of relatively naive employees, and asked if they were going to sell at the all-time high of $700+/share. When everyone else told me that they were holding on for more capital gains, I remembered Warren Buffett's adage: "Be fearful when everyone else is being greedy, and be greedy when everyone else is fearful", and sold until my hands shook from pushing the "sell" button. It was one of the hardest things I ever did, but was also incredibly lucrative. My only regret was not selling even more than I did.

I am not normally a market timer: those of you who've read my financial posts over the years have probably noticed that I have no faith in anybody's ability to time the markets, including my own, and I follow a rough asset-allocation model. But as William Bernstein said during his visit to Google on one of my birthdays: Just because you believe in the efficient market does not absolve you of the responsibility to do the math and look at what makes sense.

An investor always has the responsibility to check current conditions against common sense. As the above two examples illustrate, common sense means that when you know the market is wrong, acting on your beliefs quickly and decisive.

Review: Dollhouse

Dollhouse is Eliza Dushku's vehicle for showing off her versatility (lead actress and producer), and it's Joss Whedon's world for exploring the nature of identity.

The show is superficially a science fiction show, but there's very little science. The premise of the show is that you've got an outfit that can imprint personalities into people (called Dolls), and then use those people to fulfill fantasies, or perform more sinister roles. The obvious uses are touched upon, but the show really hits its strides when the Dollhouse's sinister past and secrets come back to revisit it.

The acting is generally good, with Olivia Williams stealing every scene she's in, and lending what would otherwise be a very dull character a surprisingly sympathetic life. The writing is intelligent, and with none of the teen buffyisms you almost come to expect out of a Joss Whedon show. With extremely strong female leads, however, it is very clear that this show plays to his strengths.

The technology, however, requires massive suspension of disbelief, not just of the basic tech piece of "imprinting", but from the giant ass plot-holes in the show. For instance, we're supposed to believe that the brilliant people who build all this amazing technology have never heard off-site backups (despite the Dollhouse being a widely distributed operation). There are basic security issues that are never addressed, and of course, many episodes leave you wondering why the equivalent of the New York Times in this universe hasn't already exploded the story.

All that aside, however, I enjoyed the themes that are explored, and definitely enjoyed the plot twists: they were both surprising and believable at the same time. Ultimately, however, I agree with Grant McCracken (I'd provide a link, but his blog seems to be down) as to why the show failed to connect with a broader audience:
Ours is a transformational world, and to that extent Dollhouse should speak to us.

But there are two kinds of transformation: involuntary and voluntary. We hate the former. We love the latter even more.

Which transformation is Dollhouse about? It's about a woman (Eliza Dushku, pictured) trapped in an endless series of involuntary transformations. A mysterious corporation wipes her soul, inserts a new persona, and hires her out. This is the stuff of our worst realities and dreams.

The trouble is not that we can not identify with "Echo." The problem is we can identify too well. We too have been the captives of forced transformations, and it gives us no comfort to see someone else endure this horrible condition.

All in all, it's a short Season (13 episodes), and I don't think it was a waste of time at all. It's not on par with the best Buffy season, but I consider it a show that's at least as good as Firefly.

I look forward to checking out Season 2 from the library.

Facebook is stealing Adsense Revenue

Just before I left Google, I heard rumors that if you told your manager that you were leaving to join Facebook, you would be escorted out of the building right away. What's interesting to me about this rumor is that it was quite believable, and also despite all that publicity about the Apple-Google rivalry, there was no such rumor circulating about that happening if you were to leave Google to join Apple.

Now, first of all, I will state my anti-Apple bias: I think that Apple would be a horrible place to work if you're a Googler. It is in every sense the anti-Google in outlook. I have friends who worked there (and one who still does), and I know for one thing it would drive me crazy. People tell me that there's so much secrecy that when you have to debug something, getting enough information to even do your work if the bug spans up or down one level of abstraction would be a challenge. And obviously, there's a lot more secrecy than there is at Google, even for internal employees.

On the other hand, while Apple might be able to start establishing ads on mobile devices, Facebook is already stealing ad revenue out of adsense. How? Take this blog for instance. I run Adsense Ads on the side and bottom (unobtrusive, but valuable if Google does a good job of monetization). The blog, however gets syndicated to various places: Buzz, Friendfeed and Facebook. The first is controlled by Google, the second doesn't try to monetize, and in fact helps drive traffic to the blog. The last, however, strips out all Google ads, and replaces it with Facebook ads!

Not only does Facebook see all the revenue that Google otherwise would see, it also grabs all the revenue that I would otherwise see, since Google shares Adsense revenue with the content provider (in this case, me). You can imagine a world in which all content thus gets syndicated through Facebook and Google's content ads network goes to zero.

So if the rumors are true, then indeed Google is terrified of Facebook, and has every reason to be. The potential for disruption to its content ads network is huge. The caveat, of course, is that the high traffic sites which have lots of traffic clearly have a disincentive to syndicate their content through Facebook, so it's only low traffic sites like mine that are affected by this. My guess though, is that Facebook will eventually provide some sort of revenue sharing arrangements with the high traffic sites, at which point it would really start chipping into Google's content ads network.

Then of course, there's that other insidious threat: Facebook is currently in the position Google was 6.5 years ago, being able to offer hot pre-IPO stock to potential employees. This is a cycle in Silicon Valley, however, and the correct response is to properly compensate your employees rather than threaten to escort them out of the buildings when they give notice. I doubt if upper-management at Google is unaware of that, given how recently the shoe was on the other foot.

Laundry

Google has an unusual perk: on-site laundry machines. When I first joined, it was just one laundry machine sitting in building 0. By the time I left, there were no less than 4 laundry rooms with entire banks of washers and dryers. When I first joined, I didn't think much of the perk: one laundry machine that operated at European speeds meant that it was never available when you needed it.

When the first laundry room with a bank of machines showed up, I started using them, because it was more work to dig up quarters for the laundry machines at my apartment building than to just dump the laundry onto my trailer and tow it to work on my bicycle. In California, you can do this most of the year and not even need a waterproof bag for the laundry. I also started keeping a week's worth of clothes at work and then never even bringing those home: this let me do a bike commute sans saddlebag, if I so chose to.

As mentioned in my book, Google had an unusual number of fresh graduates. This led to the kind of "accidents" you would expect in a college dorm. One day, I came in to the laundry room to find one of our administrative assistants in a huff. "Darn Engineers with PhDs shouldn't be allowed to do laundry," she muttered under her breath. I asked her what happened, and she said, "Some idiot PhD went and stuck washing powder into the dryer. Now the whole thing's jammed up and there's powder all over the place." She went out, grabbed a marker and a sheet of paper, and taped up the dryer with an "Out of Order" sign. Then she had a better idea. She went out, grabbed a ton of paper and tape, and wrote on them: "Washing machine. Use detergent here. Do not put dryer sheets here.", and "Dryer. DO NOT using detergent. Use dryer sheets here!" And proceeded to label every machine in the laundry room.

Since then, we have not had a repeat of the "detergent in drying machine" incident. So at least we know that even engineers with PhDs can read.

Referral Programs

Most companies, startups included, have employee referral programs. Nearly every corporation will tell you that referrals are the best source of their employees. If you compare the cost of an employee acquisition through a retained recruiter and through a referral, it's usually an order of magnitude. A retained recruiter usually gets 30% of the recruit's first year salary: for a $100k/year engineer, that's $30k a year. The typical referral bonus paid out to an employee is less than $3000.

One response that companies make when faced with urgent need for hiring is to raise the referral bonus. Strangely enough, this makes very little difference to the rate of recruits unless the referral fee is increased dramatically. In fact, the biggest way you can get more referrals from your employees is to make them happy. Google was the master at this: the campus is filled with fun exhibits, so when you invite your friends over for lunch, you pretty much have to take them all over campus and show them the various cool things. It makes you feel good, and it makes your friends want to work at this great environment. The mind-blowing free food perk is also amazing, and Google did not enforce strict limits on guests: these guests frequently became future full-time employees.

People recruit friends because they want their friends to be happy too, and even new employees are very good at filtering people whom they instinctively know won't fit in the culture. It wasn't uncommon to see a note from an employee referring a friend who wouldn't fit thus: "Hi, this is my friend X. He's a major pain in the ass to work with, and I wouldn't want him here. I don't want to offend him by not referring him, so could you please write him a note saying that there's no position here for him?" For some positions, you might want to ignore such notes, but by and large, if the person with a financial incentive to get someone hired would forgo that incentive so as to never have to work with that person, that's a pretty strong signal.

Finally, if you start to see the number of referrals drop off, this is a signal not that your carrot isn't good enough, but that your employees are no longer as happy at work. Nobody wants to bring their friends into a toxic work environment. Unfortunately, very few companies pay attention to this as a signal.

Review: Garmin Edge 500

Someone I knew decided he didn't like the color of his Garmin Edge 500, so I bought it off him for $200. You can buy the version that's bundled with a HRM and Cadence/Speed sensor, but it's cheaper to buy those pieces separately. Since I didn't care about having a Cadence/Speed sensor, I just bought the Garmin Heart Rate Monitor. [Update: I bought the Cadence/Speed Sensor after all]

First, a note about what the Edge 500 is and is not. It's not a navigation unit like the GPSMap 76CSx or the Edge 705. There's no slot for an SD card to store maps, and it won't even give you route pointers or directions. It is purely a replacement for a traditional bike computer. My Sigma 2006MHR had given up the ghost recently, and I wanted a stoker computer for the bike position anyway, and the Edge 500, while expensive, had the benefit of not needing wheel magnets and sensors, as well as not needing adjustment when I switched it from bike to bike. Having nice looking tracks when you're done is a plus, though a minor one.

The unit came with 2 bike mounts, though you can buy more mounts if you have other bikes. The tie-on is very similar to the Sigma's with 2 rubber bands. The computer slots in and then twists 90 degrees to secure. I found the mounting not as reliable as I expected, since in pushing the buttons I ended up twisting the computer back towards the unlock position a couple of times.

The computer itself is fairly easy to use. There's a power button, and a start-stop button. A mode button switches between 3 different configurable displays, and accessing the menu system to switch settings, bikes, etc., did not require reading the manual. One note that the person who sold it to me said to do is to ensure that you always stop the timer and immediately reset the GPS unit while you still have a GPS signal, as otherwise the unit will lose all the data from your ride. I can definitely see this being very annoying.

Connecting the unit to my Windows 7 PC through a USB cord was straightforward, though I had to use the control panel to force Windows to recognize the unit. Once that was done, tracks uploaded quickly and easily. If you want to see my tracks, here's a hilly ride without the HRM and the next day's recovery ride with a HRM.

GPS speeds are not very accurate. In the first ride, the GPS unit recorded a top speed of 55.2mph. I had a separate, traditional bike computer on that bike (it was a tandem), and it recorded a top speed of 41.5mph. That's a 20% error. In other words, if you went on a Tour of the Alps, and all you had was a GPS track telling me your top speed of 100+kph, I won't believe you. Use a traditional bike computer if you want accurate speed measurements. Or get the speed/cadence sensor. The temperature sensor is also not very accurate, and doesn't react quickly to changes in ambient temperatures. This is of concern because without accurate temperature sensor, elevation changes measured by the barometric altimeter might not be accurate.

One improvement from previous years is that Garmin's Connect software now does appropriate smoothing for elevation gained/lost if you turn on the Elevation Corrections on the web-site. This makes Garmin's elevation gained/lost reports much more believable than in previous years.

Battery life with the Edge 500 is much better than in previous units. I've recorded about 7 hours of elapsed time with the unit on, and it reports 75% battery life. [Update: After two back to back rides (King Ridge, Coleman Valley Road]without a recharge, the unit was down to 30% from 100%. This was with both the HRM and the GSC 10 in use, which means that a 15 hour battery life is a reasonable expectation] The quoted 18 hour battery life seems very achievable. The unit is also light enough that I wouldn't consider leaving it behind on the Tour of the Alps just to save weight. All in all, I'm satisfied with this purchase as a bike computer replacement. It costs about $100 more than a similar non-GPS bike computer, but does quite a bit more. However, if you care about navigation features and absolute minimum weight isn't a big issue, I would still recommend the Edge 705 instead. I'm certainly not leaving behind my 76CSx for the upcoming tour, but would happily use the Garmin Edge 500 on my single and tandem for local rides.

Facebook: Google Redux?

It amuses me that what people say about Facebook today sound substantially the same as what people said about Google six and a half years ago. What were they saying to me back then?

It must be too late to join Google now. There's no way you'd become a millionaire by joining them at such a late stage.

How could they possibly make money? Those little ads? Nobody clicks on those! I don't know any one who clicks on those things, and I myself never click on those!

The former statement obviously might or might not be true, depending on how generous the stock compensation package handed out to employees today are. The latter statement, however, is easily verified: try placing an ad on Facebook. The last time I tried, it was a minimum of $0.60 per click, and they wouldn't take my $1/day budget. For reference: Google will happily take a $1/day ad campaign, and will take bids as low as $0.01.

When I visited Facebook campus a few weeks ago, it reminded me of Google campus 6 years ago. The parking lots were overflowing, and people were clearly working at high intensity. Even the cafeteria looked familiar (well, it should, since it was for all intents and purposes Google's old Cafe-14, complete with the same staff).

Ultimately, two things convinced me of Facebook's value. The first was when the Caja team needed a contact with Facebook 2 years ago. I dug through my Facebook contacts and came up with Jeff Rothschild, who's the VP of Technology at Facebook. I made the connection and they had their meeting. (Jeff by the way is featured in my book) Now, there might not be an easy way to monetize this (though see above about the price per click on Facebook today), but I was convinced of the value of Facebook after that.

The second thing that convinced me of Facebook's value happened when I was in Europe, and again when I was in Australia. Over and over again, whenever I met someone, instead of giving me their e-mail address, they'd tell me to friend them on Facebook. This by the way, is a very bad idea for most Europeans: there are so many name duplicates on Facebook that it's impossible to disambiguate the two hundred "Ben Smiths" for instance. I tell people to friend me instead since my name is much more unique than a typical European/Australian/American.

Thirdly, people I talked to who weren't in Silicon Valley all thought that Facebook must have been a giant corporation with more than 10,000 employees. More than anything else, having small companies have such a huge impact on the rest of the world is the major reason why Silicon Valley is still the place to be, if you're a computer scientist.

Finally, if it is true that Facebook has found a way to monetize social networks, this is huge news. The Google-MySpace deal famously fell short of expectations, so if Facebook really has trounced Google at the social-network monetization game, that would be huge.

AMT and stock options

An e-mail came into my mail box addressed to a group of high net-worth friends. Apparently, a group of individuals were trying to put together a sum of money structured as a loan to an employee of a high profile startup in Silicon Valley. The startup has been noted as "the next hot IPO" in the Valley, and an early employee had recently exercised all his or her stock options, which triggered a large AMT capital gain immediately on the difference between the current "fair market value" price and his option price.

Let's take a look at this from an employee point of view: as a private company employee, he's in a very bad spot, since he can't actually sell enough stock on the open market to pay the AMT due. If there was any way he could sell the stock at all, that's the correct way to go. Alternatively, he could attempt a private placement (i.e., sell it to accredited investors) in order to raise the funds. That would however require expensive investment bankers, etc., in order to structure a deal. So it's understandable that he would try to structure a loan using some stock as collateral.

From an investor's point of view, however, this is a bad deal. If the hot startup collapses instead of going public, then the investor loses all his money anyway, since there would be no recourse other than the aid private company stock, which would be worthless. If the hot startup goes public, the investor would have his upside limited to the agreed upon interest rate. The only way such a deal would be attractive was if the investor got equity as well.

Obviously, the best thing to have done would have been for the employee to have read my book An Engineer's Guide to Silicon Valley Startups several years ago, but since it was only published this year, it's understandable that the advice (which is not at all common knowledge) wasn't followed.

Preparing Employees to be Wealthy

Of all the companies I ever worked for, Google put the most effort into educating its employees about financial planning prior to its IPO. I was impressed by the folks brought on campus to talk to us. These included Burton Malkiel, Bill Sharpe, and Robert Shiller. If after this world-class education in financial planning, investing, and basic economics, you couldn't figure out what to do with your money, it wasn't Google's fault for not providing opportunities to learn!

Google also organized financial fairs, where the big name financial firms, local financial advisors, brokerages, and even Vanguard would show up and talk to you one on one to try to sell you on their services, if that's what you felt you need. I didn't go, since I pretty much have my strategy all laid out, but I know others did make use of those services. Over the months coming up to the IPO, I saw more than one high level engineer/old-timer schedule meetings with these crooks financial professionals.

One of my friends did attend, however, and told me this story:
So I was at the Morgan Stanley booth, and O. came up and started talking to the MS rep. As you know, O. is very financially sophisticated, and started asking all these complicated questions about forward-contracts, asset allocation, options, and hedging strategies. You could see the rep's eyes get bigger and bigger. He clearly decided that O. was a potential big huge customer, and put lots of effort into selling him. Then C. came in (C was a serious old-timer with a huge amount of stock), and asked a couple of naive-sounding questions. You could see the rep getting irritated that this guy who by the looks of him wasn't in the same class as O. did was wasting his time and replied very curtly and tried to get rid of C.

At that point I couldn't help myself and laughed very hard. As with everything else, appearances are deceiving, and the sales rep had no way of knowing who the big fish was. Here in the valley, judging people's net-worths by their appearance or mode of dress is a serious mistake.

Review: Freedom

I found Daemon enjoyable, so I looked forward to reading Freedom when it came out. Unfortunately, it came out during a particularly busy time of my life, so rather than buy it, I waited in a virtual queue at the library.

Freedom continues where Daemon left off. We've got a networked AI of sorts slowly infiltrating the world, while a group of plutocrats try to fight back and take advantage of its capabilities for its own ends. As with the previous book, the characters are nothing to write home about: their motivations are simple, and even the (very mild) romance thread seems mishandled.

What the book does do a good job of, however, is to imagine a world where a MMORPG designer had the tools to reshape the way the world worked according to game design principles. You would have quests, levels, and ability to gather resources based on resources, and on top of that, a reputation based economy. But what would people do with it? Could you really run a community, a country, or the world this way? How would the non-digerati react? Dan Suarez hints at some answers, but is better at demonstrating the immediate consequences that unrolling the full implications of the world he has built.

All in all, if you turn off your brain, and then treat this book as a techno-thriller, you won't be disappointed: loud explosions, death matches, and full on fights are common. You could see a Hollywood movie made from this book. Furthermore, there's actually a conclusion: it's a bit pat, and I find it difficult to imagine someone who would actually execute such a vision to have the humility to build in the kind of checks and balances Suarez has him do, but there are no hanging threads and if there's a sequel, it'll have to involve different characters.

This novel is unusual in that it has a bibliography. It's an unusually well-read bibliography as well, including The Transparent Society, a book all too often ignored in the privacy debates, but is a much better book than most privacy advocates give it credit for. Recommended, but I'm not sure I would buy it at full price.

Infrastructure

If there's anything Google loves to build, it's infrastructure. Google had entire buildings worth of machines, and lots of ways to make use of all of them. There's MapReduce, Bigtable, Blobstore, and all sorts of other distributed infrastructure. So much so that engineers frequently told me that they thought that developing without all that infrastructure would be crippling, and would slow them down too much.

The irony, of course, is that companies like FriendFeed gives the lie to that statement. Friendfeed was launched in weeks! If you're not Google, you don't have to scale to billions of users right away. Existing tools can be made to be extremely scalable. For instance, even MySQL can be made to scale. The truth is, launching products at a big company takes a much longer time than at a startup because of non-coding related reasons. In fact, many of the engineers who made that above statement would find creative ways around missing infrastructure if they were at a startup: context is everything.

I remember attending a talk by YouTube engineers after the acquisition (this was at OSCON, so I know it's unclassified information). What impressed me was how closed they always seemed to be to falling over completely. Yet they never did. Then it occurred to me that a startup should always be running at the ragged edge of what their systems can handle: to do otherwise would mean that you're not using all your resources efficiently. By contrast, Google can afford a few under-utilized machines. In addition, all that generic infrastructure has overhead. Generic cluster management software, for instance, doesn't (and can't) know enough about the overall job structure of your tasks to put compute-intensive tasks on the same machine as network bound tasks. But a startup with a customized software stack can do that (and frequently must do so) because they don't have enough machines to do otherwise.

In short, I think startups have to be very careful about building generic infrastructure just because that's the way Google did things. Google built generic infrastructure because its big problem (search) had to have massive scalability right away. Even with a single user, a search engine still has to search as much of the web as possible. But what applied to Google doesn't apply to all startups. Build only the tools you need as the need arises.

GOOG digger

After about 6 months, I finally had enough time to sit down at Charlie's and have a meal, rather than eating at my desk, or more frequently, just snacking rather than eating meals. So sitting down alone in the cafeteria, I was about to get started when I heard an unfamiliar voice say, "Mind if I join you?" I looked up from my plate and saw a gorgeous Asian woman with a tray of food. Well, any geek will tell you that us dorky guys rarely get pretty women inviting themselves to sit with us when we're alone eating dinner, so I agreed.

She was in [department redacted], and we spent the next 15 minutes exchanging details of our work, what we were working on. Since we were in completely different departments, we had lots to talk about that was new to each other (or so I thought). Then she asked me this question: "So... how long have you been here?" My response, "About 6 months."

Not five minutes later, she looked at her watch and said, "Oh no! I have a meeting to run to!" It was already 7pm, and at that time we didn't have distributed offices to introduce time-zone craziness. Off she went, and I never saw her again. I thought nothing of the event and finished my dinner. When I told my girlfriend Lisa the story, she laughed hard for 10 minutes and said, "Next time a pretty girl in the cafetaria asks how long you've been working there, try saying 6 years!" I then realized what had happened and laughed along with her.

Unicode Names in E-mail headers

If you've corresponded with folks via gmail, you might have noticed that some folks write their English names, and then put their non-English names in parenthesis after that. For instance, in my case, I write: Piaw Na(蓝俊彪).

This use started in my office in 2005. I was complaining to Pengtoh about the pain of using Chinese input methods, and how they weren't actually easy to use on my Linux workstation anyway. Pengtoh pointed me at Mandarin Tools, and I used that to construct my Chinese name in pinyin. We then wondered whether Gmail would handle unicode in the headers fine, so we tested that by sticking our Chinese names in the Gmail settings. (You can find this in the "Accounts and Import" tab of the Gmail settings)

It turns out that Gmail handled Unicode in e-mail headers just fine. We then decided we wanted to see if we could make this convention common. We did this by ourselves, and then decided that we had to recruit non-Asians to make the meme stick as well. So Pengtoh contacted a few more of our non-Asian co-workers, and offered to construct Chinese names for them.

I knew we had succeeded when I started seeing Japanese names in parenthesis, then Hindi over the years. The unfortunate part in the early days was that many other mail readers could not handle unicode characters, either blanking them out, or turning them into gibberish. To solve this problem, I also constructed a non-unicode name for use with those mailing lists and corresponding with people who used these mailers. You can do this by adding more than one entry in the "Send Mail" as part of the Gmail settings. My hope is that all mailers everywhere can handle non-Unicode names in headers eventually.

It's not inconceivable that somewhere else, someone else came up with the same convention. However, I do not recall seeing this convention pre-2005, so at this moment, my best guess is that Unicode names in parenthesis originated in my/Pengtoh/Ovidiu's office in 2005. If you know of an alternate origin, please let me know in the comments!

Corporate Culture

Corporate culture is a nebulous term. People use it to include everything from free food to whether or not you can bring your dog to work. I personally think that far more important is the way we interact with each other, and how things get done.

One winter, I decided that it would be a good thing to run a bicycle repair workshop as a series of winter classes. I didn't (and really couldn't) teach every class myself, so I put up a shared spreadsheet, and listed a couple of sessions that I would teach, and then Mark Logan, Ryan Kauffman, Bob Sutterfield all pitched in and taught classes I couldn't teach. It was typical of Google culture that volunteer-ism was common. You could almost always get help from someone if you went up to them and asked about something. If they didn't know, they'd point you to someone who did.

Unfortunately, corporate culture is one of the first things to get diluted when a company gets big. And indeed, when Google got to 5,000, then 10,000 and finally 20,000 people events like this got less common. An attempt was made to formalize events like this as "Googles Teaching Googlers." Pardo's wheel-building lectures were made available that way. While it worked to a certain extent, the formal version to my eyes, were always a pale reflection of the spirit of community that prevailed the the company was much smaller.

Food Story 2

Growing up in Singapore meant that I really loved food, and always appreciated good food. However, I'm also cheap, and so never went to any fancy restaurants. It wasn't until Google that I actually had a formal, multi-course Western-style dinner.

By 2005, those of us who were on the Sarbanes-Compliance projects were deemed to be "done". At other companies, this might mean a bonus, but Google had something much better: in-house Chefs who could prepare a fancy meal on a budget. Many people think that the in-house Chefs at Google are an ostentatious perk that was an expensive luxury for employees, but in reality, I think the value Google got out of them in terms of extra work from employees and being able to run award-type events really cheaply meant that the culinary staff more than paid for themselves.

We sat down in a room near Charlie's (this room would eventually become the B40 gym), which had been laid out like a restaurant, complete with a special door from which food would arrive. I'd never seen table-cloths so white, nor had I ever had a place-setting with this many implements. I thought, "OMG, I'm in for a treat."

The first entrée arrived. It was a salad with dressing. I had been trained to get hungry by 6:30, and was starving, so I ate it with relish. Then came the Ceviche. That was really tasty too! Then came the sorbet. I was shocked. That's it? That was my fancy dinner? I was still hungry. "Oh well," I thought, "I can still go grab a burger at Charlie's afterwards." When the main dish arrived after the sorbet (it was Filet Mignon), I finally realized that the sorbet was a palate cleanser, not desert. The rest of the meal was fantastic.

Google ran many other "event"-type dinners. One of my favorites was the chocolate-themed dinner that Chris persuaded Charlie and the culinary team to run. That was a $20 dinner, but my goodness, you got $100 worth of food out of it. Another week, Google's culinary team ran the Cafe Crawl: visit all the Cafes in a week: Lea and I used bicycles to visit all the Cafes in one lunch period. The reward: a special meal, and a pass that let you skip to the front of the line for a week! Needless to say, I took advantage of that pass to get a huge amount of sushi.

Google cafeteria reached their height in 2007 --- when I visited Paris in 2008 for a culinary tour, I unfavorably compared some restaurants in Paris to Google's Cafe 5IVE, for instance. I was sad to return from Germany to discover that many of my favorite Chefs had left. Olivia Wu and Scott Giambastiani are still at Google though, and they still turn out meals that could blow your mind if you were used to the typical corporate cafeteria.

There's a talent war brewing in the valley over corporate Chefs, so hopefully, having great food as part of your compensation package will be more common.

P.S. I'm fasting for an annual physical/blood-test, and writing this post while fasting was a mistake!

Motivation

An friend of mine was very upset at work. "I look at my bug list and I just want to cry," she said. Now, this was a person who was single-handedly developing and supporting a program used by millions of users. It was significant, important work, and she knew it. But in the face of this hugely negative feedback, even the most self-confident of us would falter.

I had just gotten my first fan-mail as a result of the book, and asked if she had gotten any? "No! Do you think I would be so pissed if I had fan mail?" Here's the thing: there was nowhere in that program that exposed who the developer was. No about box, no credits page nothing.

Compare this to a movie, where everyone from the Key Grip to the Best Boy gets named. Open Source software at least makes a step in the right direction: the Firefox about box gives credit to everyone involved.

This made me think about all the hype I've heard over the last few years about how few engineers there were in general, and how few women engineers there were in particular. One problem is that we hide away all the people behind the amazing products we use every day. Who was the chip-designer behind the iPod's touch technology? Do you know? I know, because I went to college with her. The main excuse most corporations give is that "if we exposed the engineers behind the products, we would be giving competitors a list of people they should poach from us."

I personally think that's a piss poor excuse. Chrome was promoted by a comic book. The comic listed names of many engineers who worked on that product. As far as I know RIM, who is in dire need of a useful browser on their phone, hasn't called any of those engineers asking if they could build one for them. For the engineers who were named, however, the delight of having their names, and faces enshrined in a comic book drawn by Scott McCloud, however, has got to be at least as good as the Founder's Award they got!

Personally, I think until engineers start demanding that they be credited for important software they contributed to, and corporations start recognizing and treating them as people who are deserving the credit rather than being hidden behind a corporate brand, I think we shouldn't be surprised that kids who grow up with iPods, iPads, Android Phones, and other products chose not to go into engineering. After all, those products weren't made by people (whom they could aspire to be), they were made by faceless corporations. And court orders aside, no kid aspires to grow up to be a corporation.

Hiring Committee Stories

Google's engineering hiring is unique as far as I know. Patterned after faculty hiring by top universities, Google engineering had no hiring managers. None. This meant that all the usual job search advice by any number of books and web-sites didn't apply to Google. Bypassing HR and trying to get to a hiring manager didn't do any good. About the only thing that could have been useful would have been to get a strong employee referral (i.e., get an employee who thought the world of you to say so when he submitted your resume). The interview process went like this: you would interview with a panel of engineers, who would then write feedback to an internal database, which would then go to a hiring committee (also composed of engineers) to evaluate the feedback and provide a go/no-go decision. Since engineers might have to live with the code (and personality) of a bad hire, the hiring committee tended to be conservative on hires. Phone interviews and in person interviews were conducted almost exclusively by engineers, with directors brought in only if the candidate requested a meeting with a manager explicitly, or if the candidate himself was interviewing for a manager/director position.

For reasons explained in my book, I ended up on the Site Reliability hiring committee. At that time, the hiring committee was composed out of relatively senior engineers: Lucas, Ben, Bogdan, and various engineering big-wigs like Bill and Urs. Frequently, when the committee found feedback on hiring to be ambiguous, it would assign another interview to an engineer well-known to be decisive (i.e., someone who would be willing to stick his neck out and say "hire" or "no-hire"). This happened surprisingly frequently because many people dislike rejecting people, and occasionally, someone would write feedback that wasn't really informative enough.

We didn't always have the luxury of a second-interview, however, since some folks had to be flown in from far-away places. Google was truly a global company, and in its pursuit of talent would consider resumes from literally anywhere in the world. Now, I didn't think that Google's interviews were particularly hard, compared to startups and other well-known firms in the industry. In general, quality companies reject a large number of engineers because most people who call themselves programmers can't code.

A few incidents came to mind as being particularly funny:

One day, I came to the hiring committee and started reading feedback from interviewers. One of them turned out to be a candidate I had interviewed earlier in the week. I was doing as much as 5 interviews a week at this time, so I didn't always remember the candidate by the time I got to the committee. Lucas's feedback for the candidate started with, "I spent the first five minutes of my interview calming the candidate down after his interview with Piaw..." When the others got to this part of the feedback there was a lot of laughter. I think that was the moment I realized that Bogdan and I would get along, because he high-fived me across the table. The candidate was a no-hire, but I don't think it was because I was particularly harsh.

At one point, we came across a candidate who had to be flown over from the other side of the world. Since we knew there was to be no chance of re-evaluating this candidate if the feedback was insufficient, we asked the recruiters to make sure that we had decisive, experienced interviewers for this candidate, who seemed pretty senior. She replied, "How about Piaw, Ben, Bogdan, and Lucas?" When he heard this, Bill put his head in his hands and said, "Why don't we just save ourselves and the candidate some time and just send him a rejection letter now?" The room burst into laughter.

Our committee took hiring seriously. We agonized over many hiring (and no-hiring) decisions for many years, learned the idiosyncrasies of many interviewers, and tried to match them up to candidates as well as possible. It was very high intensity work, and on one or two occasions I had to go head-to-head and argue my case in front of VPs because I felt strongly about one candidate or another. I didn't always win, but at every point everyone's opinions were considered. I'm sure we made mistakes, but looking back, I'm not sure I would have found a better process. I for one think that the decisions the committee made were far better than the decisions each of us individually would have made.

Turns out I lied...

I didn't retire from Google, I was handed a pink slip! Thanks, Stephan, Larry, Lea, Pardo, Roberto, Catherine, Mike, and Parisa. With such appreciative colleagues, I must have been a fool to even consider retiring. Thank goodness they set me straight.

(I swear it looked more pink in person)

Orkut Story

As many people know, Google ran an annual ski trip for employees, and by the time I joined, it was getting pretty big, with Google taking up most of Squaw Valley's lodging. I remember checking in, and then heading over to my room to shower.

When I got to the room, I noticed that not only it was a suite with separate party area from the bedrooms, but the suite had a table that was piled with Vodka and alcohol floor to ceiling! I thought to myself, "Wow. Google's incredibly generous. Not only did they book an entire suite just for 2 people, they must have also negotiated some amazing deal to give everyone enough Vodka to keep them drunk for weeks!"

I took my shower and then finally noticed that my roommate had already checked in before me and left his luggage in the room. I took a look at the name tag, and that explained everything. It was Orkut Büyükkökten. The alcohol didn't come with the room, it came with Orkut! I was momentarily horrified, since I actually entertained thoughts of sleeping at night so I could go cross-country skiing the next day, but Orkut had clearly intended an all night party. I then realized that I could easily swap with someone who wanted to party all night!

Mike Samuel came to my rescue and bravely agreed to swap places with me, so I got a good night's sleep, and he had to put up with all that alcohol. Over the years, I learned to bring ear plugs with me to the ski trip, since even if I wasn't sharing a room with Orkut, the guy next door could be a hard partier.

One note about Google parties: the music is universally set too loud. I always felt as though the parties were for people about 25 years younger than I was. I thought I was alone in thinking that, and that I was being a fuddy duddy, but then the 30-year old and 25-year old Googlers told me that too! Until the company split up the departments so each department could have its own party, I never did attend what I considered a good party: one in which you could hear your colleagues talk in a social setting.

Getting confused with Meng

One of the perpetual sources of amusement to others (and occasionally myself) at Google is that for whatever reason, I get frequently confused with Meng by other people. This is very confusing to me, because Meng looks like this:

Yes, that's right. He's always shaking hands with a president.

I look like this:

From Pigeon Point 2010

(And that's my good side)

Clearly, Meng is much better looking than I am, yet people confuse me for him! I got annoyed at this at first, but then decided that since it was mostly white people making this mistake, I could understand. When I first moved to this country, all the white people looked alike to me too. It took me a while to realize that color of hair and eyes could be used as distinguishing features.

Then one day, an Asian woman mistook me for Meng. What's worse, I was even riding a bicycle. Meng doesn't ride a bicycle. About the only thing we share in common is that we both grew up in Singapore, and are friends with Pengtoh.

Then one day, Meng came up to me and said that this guy walked up to him and started talking about cycling. I felt mortified that at least once in a while the mistaken identities went the othe way.

Anyway, now that I'm no longer a googler, there's no excuse for mistaking Meng and I. Just look for the Google badge. If there's one, then it's Meng. If there's none, then it's me. Until Meng retires from Google, that is... I don't know what to do if that happens.

Google food story

When I first joined Google, the menus seemed so exotic to me that I resorted to having to search the web to figure out what was being served. The internal mailing list, food-discuss, also had fairly active discussions of certain menu items.

One particularly memorable exchange came up when the dinner menu included the entrée John Dory. The conversation went like this:

"What's John Dory?"
"It's a fish from New Zealand."
"Oh thank god! I was afraid that some hapless Googler pissed off Charlie and got turned into the dinner entrée!"

Last Day at Google

Today was my last day at Google. It's been 6.5 years since I first started at Google, and let me tell you when I first joined I did not expect to stick around for that long. My previous longest tenure was at Mpath Interactive for 3.5 years. I remember joining the company, and thinking, "Boy, this company is so huge. I'm going to get lost here." I became very pleasantly surprised that after my very first TGIF, I ran into Eric Schmidt on the way back to my cube and he knew my name, had obviously read my resume, and knew that I had "been around the block a few times as far as startups were concerned." Looking back at it and reflecting on what my starter project at Google was, I shouldn't have been so surprised.

Financially, it stopped making sense for me to stay at Google about 2 years ago. The risk-reward ratio had mostly tipped down to much less risk and correspondingly less reward, and while regular refreshers were handed out, they weren't really enough to really make a big difference to my net-worth. However, I still had interesting things to do, including my Munich assignment. Munich was such a small office that I felt like I was at a startup again, which was awesome. I felt like I did a lot there, and it was enough to keep me excited and motivated.

After my Australia trip last year, I became a part-time employee, going to a 4-day work week. Amongst other things, I scanned 10 years worth of slides, bought a house, toured Japan, and wrote a book, but somehow still never quite got caught up to everything I wanted to do. And having found a taste for long-term travel (as I got more time, I discovered that I didn't want to do more trips, but wanted to do longer trips), I wanted to do still more! I still never did find time to visit my friends, or even do any of the other nerdier things I thought I could make time for.

The final straw that made me decide to retire from Google was the realization that as an engineer and a professional, I'm highly optimized for startups. As Google got bigger, the pressure to specialize and stick to a formal role grew to the point where everything I did (and that included the assignment in Munich) came at the cost of professional advancement. I supposed I could have sat back and coasted, or as a friend of mine used to say, "rest and vest", but that's not in my personality. John T. Reed's book, Succeeding, made a very good point, which is that it's very difficult to change your personality, and trying to do so would make you very unhappy. However, it's possible and quite easy to change your context and your environment, and your life should be about finding a context and environment where your personality makes succeeding easy, rather than trying to fit your personality into an increasingly ill-fitting context.

In any case, I don't have any plans to jump right away into another work-place, startup or not. For one thing, I have at least one more book I want to write (it has nothing to do with computers or startups), and Lisa and I have booked a 5 week trip to Europe this summer, along with some Googler friends. I have hopes of doing a photography trip in the fall, and yes, I would like to do another sailing trip. Then there's friends to visit, and maybe for once I should try to attend WorldCon or GenCon, events that I have always thought of attending, but never did it because when you have a limited amount of vacation time, you would never waste it on indoor activities when there's so much left of the world you want to see.

For the immediate future, however, I plan to spend the next few weeks writing up some of my experiences at Google (no, I won't divulge any trade secrets), so if you enjoy that sort of thing, stay tuned.

Geyserville Ride

Geyserville Ride

The Western Wheeler Geyserville weekend ride started out close to Chad's weekend retreat, so he invited Lisa and I to join him and Drea at their country house not more than 5 miles away from the ride start. I had been years since I saw Chad, but he didn't look a day older than the last time I saw him --- the years had been good to him.

Chad had been a fearsome rider in his day, so I expected the pace to be fast, but it was also Drea's first ride of the year. Nevertheless, their S&S coupled Supremo with DuraAce parts and a carbon fork looked intimidatingly fast. Phil and his friend Elliot also joined us from the South Bay. With memories from the past of how this ride kept kicking my ass, we opted for the easier C ride.

The ride was gorgeous, with the first blooms of spring showing up along the road. The Geysers road is a lonely lovely county road with a very few gravel sections, and the initial rolling hills would lull you into a sense of complacency with how beautiful the whole thing was. That is, until you get to the stop sign and turn right. The 3-part climb starts out with a consistent 14% grade for 1.5 miles. On a tandem, this just means getting into your lowest gear and suffering. Add to it hot afternoon sun and no breeze, and we were in a world of hurt. Fortunately, even at 2 miles per hour we eventually made it to the initial, false summit. We drank pretty much all our water, ate a bit of food, and waited for the folks who had to walk to catch up before starting the rest of the climb.

The rest of the ride was uneventful, with glorious, sweeping panoramas of the valley as we swept back down to lunch, after which Chad, Drea, Lisa and I opted for the short cut home, since we were running late, and even with the short cut would still get 58 miles and quite a bit of climbing. Unfortunately, Lisa's camera conked out at this point so we had no more pictures the rest of the weekend. I'm hoping Phil posts his.

Day 2's ride started at Healdsburg, and we rode along the vineyards before tackling Sweetwater Springs road, a gorgeous 3 stage climb that mixed sun, hills, lovely redwoods and a beautiful stream all in good measure --- and a painful 16% grade at the end that nevertheless felt easier than the Geysers because it was shorter and shaded. Chad had a trip elsewhere planned, so he and his family had to drive to San Francisco early in the morning.

Bob & Betty pulled us for the remainder of the ride, at speeds well in excess of 20mph most of the way, which made the ride quite a bit of a workout, but we had slept well the night before and so felt quite good despite desperately just hanging on to their rear wheel. What a fun weekend!

[Update: Phil's Pictures]

Clipless pedals

A lot of people buy clipless pedals and then on their first try, fall over at a stop light because they can't clip out. This is very silly, and it has never happened to me because I did something very few other people did when I bought mine: I practiced. Here's how:

1. Find an empty parking lot that's level, and clear of cars.
   
2. Straddle your bike and clip in on the right.
   
3. Clip out.
   
4. Repeat 2-3 until you can do it repeatably without looking down.
   
5. Raise the right foot to the 2 o'clock position, and push down
   
6. The bike will move, so now that your leg is straight, lift your butt up and over the seat and sit down.
   
7. Pedal and push down on the left foot until your left foot clips in. Do not look down. If you're using Looks you may have to use your toe to flip the pedals up to the correct side, but with SPDs, you can just push straight down.
   
8. Slow down a bit with your brakes, and unclip your left foot.
   
9. As you slow down to a stop and brake, turn the handlebars slightly to the right. This will cause the bike to fall to the left and onto your outstretched left foot.
   
10. Repeat 5-9 until step 7 and step 8-9 become natural and easy.
    
11. Practice emergency stops. From about 10mph, brake hard and unclip and land.
    
12. As you gain more confidence, start from higher speeds and try it with both left and right feet. Once you can do this from about 15mph or so you're safe for the streets, though more aggressive types will want to try it from 20mph.
    

I did all this about 14 years ago when I bought my first clipless pedals. The entire learning process will take about 1-2 hours in a parking lot, though body geniuses can do it in half an hour or less. It sounds involved, but as a result I've never had the experience others may have had, which is that of rolling up to a stop light or stop sign and then falling over because I forgot to get my feet out of the clips. One note about buying pedals: don't buy pedals without wrench flats. In other words, the cheap SPD M520Ls are good, but the expensive XTs are not. The lack of wrench flats make putting on and taking off pedals a major pain in the ass. Needless to say, if you're anything but a racer, buying pedals/shoes/cleats that you can't walk in is stupid.

Review: Primitive

Primitive is billed as a thriller by Mark Nykanen. When I saw that it listed global warming as a theme, I read it (it was one of the Amazon giveaways on the kindle) just to see whether a novelist could actually get climate change right.

The plot revolves around a model who gets kidnapped by an environmental commune for her participation in various consumerist ads. The model tries to escape, and her daughter tries to find her. Most thrillers are simple-minded black-and-white, good-and-evil affairs. This one surprised me. There are no heroes.

The environmental commune commits basic mistakes (including settling on poisoned land), the authorities that's try to chase them down also commits predictable mistakes (unfortunately, no twist there). About the only sympathetic character is Sonya, the kidnapped model, who unfortunately also got a predictable consciousness raising epiphany.

On the other hand, the science isn't all made up, which surprised me. The idea of methane under the arctic ice being a driver of climate swings is well-known. The characters are believable (and believably stupid).

I hesitate to recommend this, but if you do read it, at least the science isn't terribly far off, and the characters aren't as black and white as you would expect from a thriller.

Review: Dear Undercover Economist

Dear Undercover Economist is a collection of columns of that name from the Financial Times newspaper. As you might expect, Tim Harford answers questions like an advice columnist, only from the perspective of an economist.

For instance, when a woman writes in to ask if she should propose rather than waiting for her boyfriend to do so, Harford points to a 1962 paper indicating that a world where men propose and women accept or reject is the very worst for women and the best for men. When another person writes in to ask how many different people she should date before settling down, he points to optimal experimentation theory. He similarly explains why grandparents tend to spoil grandkids (as well as providing a way to keep them from doing so).

The answers are mostly written in a flippant advice-columnist style, so reading more than 2-3 at a stretch taxed my patience. That's the main reason this book took me 5 weeks to work through. All in all, while I enjoyed the book, it's definitely not something you would read a second time. Recommended only for economics geeks.

Review: The No 1 Ladies Detective Agency

The No. 1 Ladies Detective Agency is the inverse of all the traditional English mystery novels. Instead of being set in England or America, it's set in Botswana, in Southern Africa. Instead of having skinny spinsters or fat Frenchmen detectives, we have a fat African woman, Precious Ramotswe, who started a Detective Agency after her father died and left her a herd of cattle.

The mysteries start off being very whimsical, and we get a good feel for Ramotswe's character: her mysteries aren't resolved so much as with brilliant deductions, but rather with a direct approach and smart questioning of subjects. It's quite obvious that this is not a set of mysteries intended to challenge your deductive skills, but a series of character and situation sketches.

We do learn quite a bit about Ramotswe's background before the novel proceeds onto more serious topics. The plot unrolls like a TV series: each episode has a main mystery, while another sub-mystery unfolds in the background, as well as a very unsubtle romance. By the end of the novel, everything's been unraveled, with the ending tied up very neatly, but we don't get the feeling that Ramotswe's done any introspection whatsoever --- none of the feeling of change or bleakness of characters found in Sue Grafton or Raymond Chandler is in evidence here.

At $2.00 for the Kindle edition of the book it was quite a bargain, but I don't think I'd pay full price for this. It will, however, make a fine airplane novel.

Hacker News Dinner

Xianhang Zhang, founder of Bumblebee Labs placed a notice on Hacker News a few weeks ago asking if there was anyone interested in having him visit their house and cooking dinner for them. I was intrigued, and signed up. Another couple of Yahoo employees were interested as well, and since we all lived in Sunnyvale, agreed to batch up our meetings. Of course, a Googler's home is unlikely to have to be better equipped for cooking than almost anybody else's home (it makes sense if you think about it!), so we ended up choosing Jaisen's home for the event.

Lisa and I walked into a kitchen that was already in full swing. What amazed me was that XianHang had already done everything since he got off the train at 5:30pm, including shopping, prep, etc. He drafted us into service to shell pistachos, form the lamb sticks (which I did very badly), and various other duties, but it was quite clear that XianHang was in charge and knew what he was doing. You can look at Jaisen's pictures to see what was involved. Lisa and I clearly didn't have what it would have taken to host this dinner.

After dinner, XianHang gave us his pitch about social interaction design. He had a very important insight, which is that most companies and social software is built as a tool, whereas in reality, what social software should do is to be built as a space (as in a building, meeting room). This is a very important distinction. For instance, he pointed out that mailing list software seems almost designed to facilitate flame wars and endless discussions over minutiae, rather than useful discussion, and you don't have to be a social software expert to realize that. In any case, it's a great presentation and I think anyone involved in social software should consider hiring XianHang for his insights. The unfortunate thing about the internet is that most software platforms are designed by engineers for engineers as a demonstration of technology and tools (my own TinyMuck 2.0 was just one such example amongst many), rather than as a space for useful interaction.

Any way, the dinner was very much time well spent. Afterwards, I gave Xianhang a copy of my book for his long train ride home. I'm glad he seemed to have liked it!

Pigeon Point 2010 Edition

Pigeon Point 2010

This year's Tour of the German Speaking Alps qualifier turned into a much larger group gathering with 13 people participating, including a sag wagon!

The day before the tour, it rained horribly and temperatures dropped to a low of 36 degrees. The morning of the ride, it was cold, but it was clear and beautiful. Mike Samuel, Lea Kissner, Kekoa Proudfoot, Phil Sung, Cynthia Wong, Scotty Allen, Katelyn Mann, Heather Whitney, Chris, Kyle Stickle, and Li Moore all showed up to ride. In addition, Catherine Rondeau and Tiffany Lau showed up with a SAG wagon to carry the gear. This being a qualifier, Cynthia and Kekoa had to do without the SAG and ride it with gear. Lisa and I put panniers on and carried our gear as well out of sympathy.

The route took us up Montebello, which can be a bear when it's hot, but this year was pleasantly cool, which meant we could work as hard as we like and sweat only a little bit. Li elected to do the bonus climb up to Peacock Court, and did it so quickly that he had enough time to catch us before the regroup at the Montebello school. Once we got up to the end of the pavement, the fire road turned out to be well-packed due to the recent rains, making it ideal for riding: our retire tire only spun out once and I had never had such an easy time on the fire road.

On the Black Mountain summit we had a glorious panorama of the area: it was so clear that we could see all the way to San Francisco, a first for the year! Descending the fire road we saw lots of deer, and got to Page Mill Road with no incident.

Coming down West Alpine, we spotted over the local mountains Big Sur looming behind it. That makes for about 120 miles of visibility. The redwoods were cool, but not as cold as I feared, though Lisa's toes went numb. The stream down West Alpine road was as broad as I had ever seen it, and together with the Redwoods in the area was a sight to behold.

Climbing Haskins Hill was nice and cool, and the descent down the other side was as smooth and pretty as always. Pescadero Creek looked like a miniature version of the Russian River, swollen with recent rains, and brown with sediment. We rode into Pescadero and made a turn onto North road to visit the baby goats newly born at the farm. Lunch immediately followed at the Pescadero grocery, where 3 loafs of the artichoke garlic bread was quickly consumed in short order. Catherine and Tiffany showed up to help cart the partially baked bread we bought for the hostel, and reported that the hostel staff was cranky that so many of us were showing up as a group.

Li had hurt his knee and so opted to ride to the hostel in the SAG with Catherine while the rest of us went back over to Cloverdale road and went down it to Gazos Creek road. On Gazos Creek road, I had front flat, but Scotty and Katelyn kept us company while I fixed it.

Upon reaching Highway 1, we faced a painful headwind but fortunately it was only 2 miles to the hostel. Once we were checked in and hot tub spots were reserved, we commenced eating. This was the first time we had ever had this much food at the hostel, and as far as I could tell, the eating started at 5pm and did not stop until 9pm. The hot tub was as good as ever, with the cold wind outside serving as a lovely contrast for the warmth of the tub.

The next morning, I woke up a full hour later than I had wanted to. This meant that with all the cooking and eating (yes, more eating, though for the first time I did a pigeon point trip without waking up the second morning hungry), and then we made our way up the coast towards Bean Hollow Road. I cheated by leaving my CPAP machine with the car, saving us 1.5 pounds (or the weight of a full water bottle). Kyle wanted to try China grade, so opted to ride South. Heather and Chris wanted to do more mileage, so chose to ride North to Half-Moon Bay. Li's knee still wasn't any better, so he opted to ride the SAG. That left only 9 riders riding to Pescadero from Pigeon Point. The descent from Bean Hollow Road into Pescadero Road was glorious, causing Scotty to say, "Piaw knows all the pretty little roads."

Stage road was pretty as well, and we got to it after a quick stop in San Gregorio where we ran into Western Wheelers Bob & Betty on a tandem. Tunitas Creek was amazingly beautiful as well, with little waterfalls flowing at a high, and the Redwoods providing ample shade for our climb. Once at the top, we quickly decided that the fastest route home was appreciated, so I jettisoned original plans to add more climbing to the ride and we headed home on Sand Hill Road and Foothill Expressway, getting home around 3:00pm with 99 miles on the odometer.

Congratulations to Phil for finishing his first ride to the coast and back, and special thanks to Catherine and Tiffany for the amazing SAG service. What an amazing ride.

[Update: Li Moore's Pictures]

[Update: Heather & Chris's Pictures]

Compensation is hard, let's go shopping!

Whenever you hear about a spate of acquisitions by big companies of innovative small companies, there's always a temptation to point fingers and laugh at how the behemoth can't innovate any more. Of course, that's a myth, as How the Mighty Fall shows: most big companies are quite innovative (especially in capital intensive industries), and failures of large successful firms aren't usually caused by too little innovation. The problem usually has to do with incentives.

With the cost of starting a startup decreasing by the year, it is far easier for entrepreneurial employees to leave big companies and start their own thing than to push through the bureaucracy at a large company to launch their product. Part of it is because large companies have a lot to protect (compare YouTube's early days with Google video's early days, and you'll see that the innovation differences had very little to do with technology), and the other part is that a large company like Microsoft cannot launch a product without it having to scale immediately, while a small unknown startup has the luxury of making mistakes and trying several ideas out in order to find one that gains traction.

But what about incentives? Leaving aside the fact that it's very difficult to use compensation to reward creative problem solving, it turns out that it's very difficult to reward entrepreneurial activity in a large firm. You might think that you could for instance, offer an entrepreneur a higher risk/reward ratio by asking him (and possibly his team as well) to risk a portion of the salary in order to potentially receive startup-like rewards. Now, you can't offer everyone this, or you'll discover that everyone who's part of your existing fast growing revenue engine will take the deal and get out-sized rewards without actually taking any risks. The problem lies in that any project/employee who has strong enough connections to get this offer, by their very nature also has the political capital to negotiate their own goals and metrics by which they can get that out-sized rewards. This leads to extremely negative incentives, like launching a product while knowing you can't possibly scale to meet demand, or launching a product missing a critical feature that would have been needed to drive adoption in order to meet an artificial, pre-negotiated deadline. In fact, this problem is so endemic that even for external-acquisitions, earn-outs are being abandoned because of the costs and undesirable side-effects associated with them.

Ok, pre-negotiated goals don't work. How about post-facto awards? Since those aren't expected, you won't have negative incentives, and people would stay on if they believe in their projects, right? It turns out that people are actually pretty good at figuring out that a project is successful or going to be successful. Someone I know was on a project that obviously had great trajectory, and he was amazingly unhappy about large groups of senior engineers and managers suddenly descending on his (previously under-the-radar) project trying to take credit for a piece of it in order to get an out-sized award. The resulting feeding frenzy isn't good for morale, and obviously leads to entrepreneurs thinking that starting their own companies just isn't that bad an idea after all. Worse, after you hand out that out-sized awards, everybody now has an incentive to leave that project in order to find the next big thing so they can repeat the process. Of course, not rewarding such successful projects doesn't work either, since you then risk losing valuable employees to other companies.

If you ask me, there's no real easy solution to any of these problems. You'd have to have an amazing top-level manager, who is so aware of everything that happens at every level of the company to be able to avoid all of the pitfalls I detailed above, which doesn't even scratch the surface of the fundamental problems in compensation. This is one reason why when faced with these issues, many top-level executives just throw up their hands and say, "Compensation is hard, let's go shopping for acquisitions instead!"

Why the rush to get big?

I really have to wonder. What was going on in the Board of Directors when Microsoft decided to become a 60,000 person company? Were they thinking, "We are doing so well now at 30,000 people. If we were bigger, we would be even better!" Because from where I'm sitting, the bigger companies get, past a certain point (that point can be very different for different companies --- Google when it was at 1500 people was more agile than many 200 person companies), the more they suck. Now, if you take a survey of all the managers in a given company, very few of them would say, "I'm over-staffed. Take some of my folks away, please." That's because generally, the more people a manager manages, the better his chances of promotion, since he's seen as having more responsibility. So no one ever says, "I've got enough people."

This might make sense in the industrial setting, where more people means more widgets you can build. But we build software. More people usually makes a late project later. More people adds to confusion, and leads to more communications overhead. Even if you add more people and had them work on different projects, unless they're all in completely different spaces to the point where they might as well be different companies (in which case, maybe they should be!), you still have the overhead of coordinating strategy and making sure that the products fit together. So why the hurry to get big? What goes on in the head of the executives and board members' heads that lead them to think that you can double or triple the growth in headcount without dire consequences down the road? Is it always just foolish optimism? Or is it that when you're at that 30,000' level running the business, all you see are opportunities everywhere that you could get to, if only you had another 500 people here and 500 people there?

Or maybe, just maybe, there's the thought that you could lock up all the smart talent in one company and then all your competitors would suffer?

I don't know. All I can say is, the thrill of fast growth is fun, but you really pay for the consequences a few years down the road, and as far as I can tell, it just isn't worth it. Far better to grow at a pace where new people can be assimilated thoroughly, and new people always have enough old-timers around to show them what's going on. I think the ideal growth rate is somewhere between 20-50% a year, not doubling every year that many fast-growth advocates are so fond of.

Review: Manifold Time

Manifold Time is Stephen Baxter's novel about The Carter Catastrophe.

Baxter starts the story off about commercial space flight, getting you to think that it's a classic "entrepreneur-in-space" novel. The commercial space-flight section of the book (mostly about mining asteroids) is kinda hokey, since I don't see the point of sending water-based creatures into space. The amount of additional ballast required would be really prohibitive.

The Carter Catastrophe piece is not really all that convincing, either. Then Baxter brings in time-travel, casual time-like loops, and a mysterious new race of super-smart human children called the Blues. No reason is given for the rise of the Blues, and the human reaction to these seems highly suspect.

Everything escalates to a climax and then we get a let down at the end, when we see the fate of humanity in a surprising conclusion that's not very satisfying.

As is usual with Baxter novels, the characters are wooden. The female protagonist is even called Emma Stoney, and she's definitely appears to move through life as though she's stoned, hanging on after the male protagonist even after a divorce. You could see the puppet strings behind the characters.

Ultimately, I don't know why I even bothered to finish the book. I guess the plot felt a lot like an scab I can't resist picking at. It did take me more than a month though! Not recommended. (Disclosure: I picked this up during one of the many Kindle giveaways)

Checking Your Accountant's Work

Someone recently at work wrote the following in favor of getting someone else to do your taxes:

Has anybody talked about time spent and stress yet? I'm in and out between 1 and 2 hours, sip some coffee while sitting on a leather chair, and I'm out to have lunch right after.

Well, because of my Munich trip 2 years ago, I still have lingering after-effects due to tax-equalization, foreign tax credits, and what not. Therefore, my employer paid a tax-accountant to do my taxes. Unfortunately, with such a complicated tax return, the number of ways for errors to creep in multiply, not just from the accountant doing things wrong, but also from your very own communications to him as well. This makes it imperative that even if someone else does your taxes for you, you still have to check the numbers yourself. Here's how:

1. Check all the sources. That means that for every 1099-INT/DIV/B, those numbers need to show up in your tax forms. If they don't or they've been aggregated, you need to do the aggregation yourself to make sure that everything lines up. The same goes for W-2, etc.
2. Look for obvious missing items: If you usually have to file a Form 2210 but don't have to this time, make sure you find out why.
3. Schedule E (if you have one) needs to be triple-checked, as it's very hard for someone else to guess what expenses of a business are deductible, and what are not. You really can't just dump the receipts onto someone else and have them guess.
4. If you have incoming tax credits (such as first time home buyer's, energy efficiency improvements, etc), make sure those line items exist in the tax forms as well.
5. If you have foreign tax credits and they are large, make sure that all the forms exist (you need one for each form of foreign tax credit, and one for each tax system, so that's 4 Form 1116).
6. Run Turbo Tax and verify your own numbers for income tax purposes and make sure that your accountant isn't very far off. If he is far off, make sure you get an explanation in writing.

Does this all seem like so much work that you might as well do your own taxes? Yes. Even though you paid someone else to do it, you are still liable for paying the taxes, and ensuring that nothing gets lost. You're the one stuck dealing with the IRS if your accountant gets too aggressive. Unfortunately, by paying for someone else to do it, you can't just make the changes to the tax form, you'll have to get your accountant to understand why his numbers are wrong, and get him to fix it.

Over the years, if I've learned anything about finances it's this: You cannot outsource or abdicate responsibility for your finances. No one else will know or even care as much about our financial situation as you do, and if you think otherwise, you'll end up learning that lesson expensively.

Paperback edition of "An Engineer's Guide to Silicon Valley Startups" now available!

It took over a week for CreateSpace to deliver, but the books are now here and ready to ship. You can now place orders at the book's web-site. Pre-orders will ship on Monday (Post Office doesn't open on Sundays).

Before the books arrived, I had nightmares about how I'd screwed up in the final order process and ended up with 100 books with nothing but blank pages. But now that the books are here and are as specified I feel relieved. I'm also glad that I kept my initial order size small, as 100 copies of a relatively thin book still takes up a sizable amount of my tiny home office. I have no idea how John Reed copes with 1500 copy runs. Then again, if I sold enough books I'm sure I could find a way to store books vertically rather than just having boxes lying around on the floor.

It does amuse me that the envelopes and the books all came in same-size boxes. Of course, lifting the book box is back breaking work!

A Game for Old-Timers

When I was in college, I spent way too much time at the Workstations in Evans Basement playing Net-Trek. Well, if you're feeling nostalgic, here's Altitude, a Net-Trek clone with wacky 2-D physics and a choice of planes, brought up to date with modern graphics. Cute, addictive, and a lot of fun. Give it a shot! Each session is short (5-10 mins), but you'll keep playing just one more session. Recommended.