Auto Ads by Adsense

Friday, April 30, 2010

Review: Predictably Irrational

Predictably Irrational: The Hidden Forces That Shape Our Decisions is Dan Ariely's book about his adventures with behavioral economics. It's a compelling read, but has also woven itself so much into popular culture that you may already be familiar with most of it.

The book covers many human foibles, with a chapter devoted to each one. These are:
  • Comparison Failures: We judge objects and people by comparison, rather than by absolute worth, so we get anchored to the comparables rather than to true value.
  • Anchoring: We get locked to the first number we see, even if it's utterly random. That's why Steve Jobs hawks the iPad by saying, how much would you pay for the device? $1000? $800? How about $499. So the $499 price looks like a bargain in comparison to the $1000 price.
  • Free almost overpoweringly over-rides everything else. Even if it is a worse value than a non-free product. The experiment here was great. He offered people a $8 coupon or a $20 coupon for $10. Even though the $20 coupon was a better deal, most people took the $8 coupon.
  • Social norms versus market norms. Basically, when you do a favor for a friend, an offer to pay is an insult. That's why companies try to motivate employees by trying to lock employees into social norm, so you feel bad for saying no to working on weekends (after all, you wouldn't say no to visiting your mom on a weekend). The problem comes when employers make business decisions and then cause the employee to feel betrayed (your mom wouldn't lay you off!).
  • Decisions made in the heat of the moment. It turns out that we under-estimate our lack of judgement when we are in emotionally heated situations. That's why major decisions are best made after sleeping on them, or better yet, if there's a time delay between when you feel the emotion and when you make the decision.
  • Overvaluing what we own. This one is all over popular culture. Once you own something, you feel much worse about giving it up than if you had never tried it in the first place. That's why free trials work.
  • Analysis Paralysis. I see this all the time, when people try to keep so many options open that they fail to see that they would have been better off committing to one in the first place. My favorite example in this book was of the woman who still kept trying to keep her old boyfriend even though she liked her new one better. She felt bad losing the option to her old boyfriend, even though it was irrational to do so.
  • The Placebo Effect. This one is so well known I was surprised he bothered to spend so much time on it in the book.
  • Cheating. It turns out that we cheat less if we are reminded of the usual high minded values before being exposed to temptation, and we cheat more if the cheating is one-step removed from tangibles. In other words, it's way easier to steal credit card numbers than it is to steal cash, and it's even easier to sell CDOs and bad mortgages than to actively bilk you of your money.
Quite a lot of topics were covered, but and the book's an easy read. The only disappointment, I guess, is that so much of it already permeates popular knowledge, so it's easy to go through the book nodding at every chapter and then asking, "Why does it take so many experiments to show what we already know?" Of course, that's precisely the state of behavioral economics: it's so new that smart people like Ariely have to prove what should be first principles. Ultimately, I recommend this book, but I do hope that in the future, we see less obvious irrationalities from Ariely's research. I'm glad I borrowed it from the library, because I would have felt about paying even the Kindle price for a bunch of stuff I already knew. If you don't have time to read the book, watch Ariely on YouTube:

Loyalty is a two way street

I was recently getting together with a friend/former colleague from a previous startup. We were discussing engineers who had spent a long time at the startup (which never went anywhere). It brought back to me Mark Suster's article about job-hoppers. It reminded me that loyalty has to be a two way street. Yet companies and employers lay-off long-time employees all the time as business decisions.

You can't have it both ways. You can't claim that the company is a family and that you owe it to a company to be loyal when someone else comes by with a 30% raise, and then turn around and layoff employees when business gets rough. If a family member did that to you, you'd feel incredibly betrayed, and the typical rank-and-file worker has been continually betrayed in this fashion for the last 30 years or so. An investor or venture capitalist can invest in 20 startups at once, and if 19 of them fail and the 20th turns out to be the next Google, the investors still win. An employee/entrepreneur can work in only one startup at a time. If after 3-4 years, there's no exit in sight, and you're not being granted new options, a hefty title, or some other compensation, then I would start looking for a new job! Note that many startups don't have an active retention program in the form of refresh options for employees (even high performers), and I think that's a major mistake. Obviously, a startup founder can't just leave, since if he leaves that's a signal of major disaster.

In retrospect, the time when every one of us employees should have exited was when we saw the Chief Architect leave. He had all the incentives to make the startup successful, but if he was leaving, then what reason did the rest of us had to stay? Incidentally, he eventually ended up at Google, where he got handsomely rewarded, so his departure was well-timed. I discuss other reasons to leave in An Engineer's Guide to Silicon Valley Startups. By the way, the print copy is current sold out, but digital editions are still available, and I expect the second printing to be available next week.

Wednesday, April 28, 2010

WM2016 Followup

Our LG WM2016 somehow became unleveled, so I called the GeekSquad and had them come over. The technician first showed me how to level the machine myself, and then told me about several issues:
  • Don't use bleach for cleaning the tub. Use dishwasher fluid instead. That gets rid of oil, which bleach doesn't do. Run the machine in "Hot Wash"/Cotton mode. You only have to do it every 2-3 months.
  • Hair tends to accumulate at the bottom of the glass door. You must clean off the hair, or it will eventually cause a door leak.
  • When using detergent, use half of the "1" line on the Tide cup. The machine is a small machine, and using too much detergent would eventually destroy the machine. A 96 load version of the Tide bottle will last 1-2 years.
Wow, I didn't know quite so much about the machine even after reading the manual. I guess the extended warranty/GeekSquad coverage wasn't as much of a waste of money as I expected!

Tuesday, April 27, 2010

Diet and exercise don't work?

In the Atlantic, Marc Ambinder wrote:Stigma might be more bearable—an unpleasant way station on the path to a thinner, healthier life—if diet and exercise, the most prescribed solutions to obesity, worked. But they don’t. Qualification: if you eat less and exercise more, you’ll lose weight. But the chances that you’ll stick with that regimen are slim, and the chances that you’ll regain the weight, and then some, are quite high. A systematic review of weight-loss programs, by Thomas A. Wadden and Adam Gilden Tsai of the University of Pennsylvania, found that the evidence that commercial and self-help weight-loss programs work is “suboptimal.” People who diet often regain more weight than they lose.

I can think of two counter examples. A friend of mine at work was diagnosed with a heart problem. It was exacerbated by a sedentary lifestyle, and his doctor wanted to immediately put him on drugs. He said, "Wait, wait. Can I solve this with a lifestyle change? Change my diet and exercise?" His doctor replied, "That doesn't work. Statistically, nobody sticks with such regiments." My friend wasn't willing to give up, however, and told the doctor, "Let me try it for a month. If that doesn't work you can put me on drugs." This guy went from zero exercise to biking to work 4-5 days a week, hiking and running with his kids on weekends, and started cutting his portion sizes and eating more greens. A month later, his doctor pronounced him completely fit, and at little risk from his cardiovascular disease. Six months later, he was still going strong, and still biking to work nearly every day. Note that if this same man had lived outside California, he probably wouldn't be able to bike through the entire winter. If he had even lived in San Francisco, his bike probably would have gotten stolen within that time period.

Example two: In 2005, I was diagnosed with osteopenia. My doctor immediately put me on a regiment of calcium and vitamin D supplements, and I embarked on a program of hiking and weight lifting, that I continue to this day (it's been 5 years). My bones are back to normal, even though doctors and others were incredulous at the improvement.

The lesson, perhaps, is that if you're a Google engineer, all these rules of thumb about lifestyle changes not working? They're probably inapplicable to you. The question in my mind is: Is there an easy way to predict what kind of persons do well with this kind of diet/exercise regime, and what kinds of persons don't?

Review: Up in the Air

Ok, so this review is actually a bait and switch. Last night, Lisa wanted to watch a movie. I had neglected to sign up for Netflix, but then remembered that the PS3 had on-line movie rentals through the Playstation Store, so I fired up the game machine and looked through the store looking for something to watch.

It wasn't hard to find something to watch, since Up in the Air [Blu-ray] was a movie that everyone on our St. Vincent Trip talked about. It felt odd not to have the Amazon reviews available while browsing, but I got over that since we wanted to watch the movie.

Purchasing is a bit of a snag, since you can only buy Playstation Store dollars in units of $5 at a time, but I could see myself making regular use of this feature, so there wasn't a lot of resistance on my part. Once the download started, I could see that my WiFi connection while good, wasn't going to instantaneously download the movie for me to watch right away. However, once the PS3 had downloaded about 5% of the movie, an option appeared to watch the movie while downloading. It took no more than 3 minutes to get to this point, but getting Lisa ready to watch the movie took at least 10 minutes, so it was no big deal.

The nice thing about watching the movie this way, as opposed to say, Redbox is that there are no ads. You click play, and the movie immediately starts. No trailers, no menu, nothing. Just what all online rental experiences should be like. The picture quality is absolutely amazing. As mentioned before, the PS3 upscaling works smoothly and magically, and Up in the Air wasn't the kind of movie that needed HD anyway.

Twice, however, the playback stopped when the playback went faster than the download could handle. Each time, we'd stop and do other stuff for a bit so that we wouldn't get stuttering. I think if I had about 30 minutes of buffer time before watching the movie, I don't think we would have even gotten a pause. I would certainly do this kind of rental again, as opposed to renting from Redbox or paying a monthly fee to Netflix (though as a cheapskate, I would of course get a free trial from Netflix first). For HD movies, however, there's some planning needed since it would take quite a while to download the entire movie to disk. That's probably better done through a regular Blu-Ray rental or Netflix.

As for the movie itself: I liked it. Unlike Hollywood movies, there's a plot twist that doesn't end happily for the protagonist, which I thought was unusual. Worth the rental, but I don't think we needed to see it in the theater.

Monday, April 26, 2010

Guide to Startups Visits San Francisco

LOLapps, a startup in San Francisco, has invited me to give a talk there on May 5th. It will be on-site, and the talk will be open to all comers, including non-employees and walk-ins. (I'll post contact info for the talk later) I certainly didn't expect to give a talk at a San Francisco startup before one in Silicon Valley, but "Silicon Valley" has really come to include the greater Bay Area in recent years. The Berkeley area, for instance, has always had a fair number of startups that became successful, including Inktomi, Ingres, Sybase, and Perforce (Perforce was mentioned in my book).

I'm expecting to keep the talk short (30 minutes or less) so as to have time for questions. If you're in the neighborhood and read the book (or haven't read the book, but want to find out more), please show up and ask questions.

The location is:
116 New Montgomery
Suite 700
San Francisco, CA 94105

The cross street is Mission, and it's 1 block away from Montgomery Street station

I'm down to a little more than a handful of print copies of An Engineer's Guide to Silicon Valley Startups left. I've placed an order for a second printing, but it usually takes a week or so for it to show up. Thanks to all of you for helping make this book a success!

Friday, April 23, 2010

Review: Perfect Rigor

Perfect Rigor is Masha Gessen's book about Grisha Perelman, the mathematician who proved the Poincaré Conjecture. It's a short book and a compelling read: I got it home from the library at 5pm yesterday, and finished it at 11pm, with a couple of hours off for cooking, eating, and mowing the lawn.

Perelman doesn't do interviews, and hates talking to people, so Gessen was forced to go about writing this book by interviewing his friends, teachers, and associates. Perelman's childhood was filled with mathematics, having been enrolled by his math tutor into a math club, where Perelman did very well. The book does a great job at describing what it was like growing up in the Soviet system for mathematicians during the 80s, with widespread anti-Semetic sentiments. Perelman was brilliant, however, and was lucky in having great mentors who nurtured and developed him as a problem solver. This first part of the book makes for great reading.

After Perelman leaves the Soviet Union, however, the book flounders a bit. First of all, Gessen had to describe that Poincaré Conjecture to laymen. This is not easy to do with text only: there are no diagrams throughout the book. I feel that this is the weakest part of the book, and Gessen made a hash out of it. Fortunately, the Wikipedia entry is thorough and does a good job of explanation.

Because Gessen did not have access to Perelman, we never understand Perelman's view of the whole thing. Why did he decline the Clark medal? We are led to believe that Perelman felt that he did not get the recognition that he deserved. Gessen also implies that Perelman might have Asperger syndrome, or was so idealistic that when the politics (again, really sociology) inside Mathematics came into play with regards to his solution of the problem, he decided to resign from Mathematics rather than put up with the problem. Perelman also declined the Millenium Prize.

Given that even his former mentors and teachers don't really have access to Perelman, we'll never know the complete story behind this.

20 years ago, when Fermat's Last Theorem was declared proven, I remember sitting down at dinner with one of my CS instructors. He told me about the result, and I said, "OK, so does this prove P=NP or anything interesting like that?" "No," came the reply. "Well then, I don't see why it's news." "You have no soul, Piaw!"

Unfortunately, I still feel like that. I don't think Gessen explains why the Poincaré Conjecture was important or interesting, and unfortunately neither this book nor Wikipedia does a good job of explaining the motivation behind the drive to solve this problem, which even after Perelman's publication, took teams of mathematicians nearly two years to fully explain. Nevertheless, a fun read, and worth the 3 hours of your time to read for a good understanding of the sociology behind Mathematics. Not that I could perform at the level these guys regularly do.

Thursday, April 22, 2010

Review: YOU: The Owner's Manual

YOU: The Owner's Manual, Updated and Expanded Edition: An Insider's Guide to the Body that Will Make You Healthier and Younger was on sale in the Kindle store for $2, and I read the excerpt and liked it, so I bought it and devoured it over the course of a week. It's miles better than the previous health book I read.

The book considers the various pieces in your body that are vital to on-going health. Much likethe Nintendo game Brain Age, the book takes the view that you have a RealAge, which may or may not correspond to your calendar age. Everyone knows that a car that receives routine maintenance and oil changes will last much longer than a car that's been abused and not been maintained.

A lot of emphasis is on things that you can do. For instance, the authors explain why your stool is brown, and then tell you what's going on if your stool is of a different color, as well as what happens if there's blood in your stool. There's even an FAQ at the end of the book, covering questions readers have sent in for the earlier edition of the book.

Now, nearly every health book covers diet and exercise, but this book really does the right things. For instance, it recommends 3 sessions of 10 minutes of weight lifting a week, since that's apparently enough to provide good health. The book discourages you from running a marathon, since that's bad for your joints. In fact, exercising too much is considered bad as well, because of the increased in oxidation. This is in fact documented, but of course, some of us like traveling too much to give it up.

The book also covers diet, and even goes as far as to provide a comprehensive diet plan, complete with recipes, how to choose how much to eat, right down to how you snack and when you snack. This is very impressive: the entire book is a one-stop shop. If you can't be bothered to read any other book on health, this will do you very nicely. All the latest research is in this book as well, including the studies on aerobic exercise improving intelligence.

The only criticism I have on this book is that sometimes, the writing tries to be too clever by half. The analogies and references to popular culture, celebrities, and so forth feel a bit forced, as though the geek doctor in them was trying to be cool. Similarly, the chapters all start with a series of myths that the doctors then debunk. The problem is, by the end of the chapter, most people will remember the myths, rather than remembering the debunking.

Nevertheless, these criticisms are minor compared to how good the book is. Highly recommended. This is a book actually worth owning and referring to, rather than just checking out from the library.

Wednesday, April 21, 2010

Xianhang Zhang Returns!

Xianhang Cooks

After his last visit, I was very lucky to have more and longer conversations with Xianhang, including a long hike on Saturday, and he agreed to cook for us once more. This time, it's for the crew that's coming along for this year's Tour of the German-speaking Alps.

My kitchen was sadly under-equipped, even though the stove and oven was brand new. So we first started off the cooking expedition with a visit to the Restaurant Supply store to pick up pots, a frying pan, and various other utensils. I was so glad that someone with real expertise was helping me pick out stuff. I would have just gone down to Safeway and picked up stuff off the rack. Now I have a full set of mixing bowls, baking pans, etc.

Then it was off to Ranch 99 and then Whole Foods to pick up groceries, and upon returning home Xianhang basically started cooking up a Swiss-themed dinner in anticipation of the alps trip. We started off with sausage and polenta, followed by potato salad, then soup, and finally desert was a baked Alaska: a meringue topping off rocky road ice cream, which was on top of sponge cake, all baked in the oven (the first time we ever baked anything in my oven), followed with a sprinkle of strawberry sauce. All very yummy. Cynthia and Kekoa brought wine and plates (when I say under-equipped I mean it), Phil brought more wine, and we all had a great time.

Thanks, Xianhang!

Monday, April 19, 2010

Review: Planet Earth

Planet Earth is the BBC HD documentary series covering all the various landscapes and wildlife of the planet. It also seems to be the Blu-Ray used to show off people who've bought new entertainment systems.

The cinematography is just amazing. I can't watch even half an episode without thinking to myself that the cinematographers must have spent weeks sitting in blinds trying to capture the images and shots in the series. The sound is perfect, and the narration by David Attenborough brings back days of sitting in classrooms getting to watch a documentary instead of listening to another boring lecture.

I do have complaints, and it's that the series as a whole seem shallow. In nearly every episode, the narration raises more questions than it answers, and you wish it occasionally would stop to explain something rather than just barrel along to the next wonder of nature. Of course, you could watch the series with a laptop so you can Google questions as they come up, but to take your eyes off screen after screen of beautiful shots would be silly.

If you own a Blu-Ray player and a home entertainment system capable of showing it off, buy Baraka. After you're done watching that, then get a copy of Planet Earth (there're so many floating around, I'm sure you could just borrow it from someone) and view it. Don't take my word for it, Amazon's released Episode 1: Pole to Pole in HD for free. So go watch it already.

Tips for Tech Leads

One of the unending sources of frustrations of being an experienced engineer in a company full of fresh graduates is having to deal with new tech leads---many engineers fresh out of school are great designers and engineers, but couldn't lead a team out of a paper bag. It didn't help that Google's internal leadership training was derived from team-building exercises for experienced executives, rather than leadership 101 that would have been appropriate in this situation.

My favorite remedy for this is the Peopleware, which I also mentioned in An Engineer's Guide to Silicon Valley Startups. However, that book seems to be out of print available only direct from the publisher, so I'll do my best to summarize my tips for new tech leads:
  • Have lunch with your team at least once a week. If you learn nothing else about leadership, learn to eat with your team. (If you're an industry veteran, I know this sounds basic, but I assure you that I ran into multiple tech leads at Google who did not do this, because no one told them that this is necessary and essential) If necessary, put it on the calendar. Communal eating is such a major bonding ritual since the dawn of humanity that not to do this marks you as an idiot tech lead or manager. One of the best tech leads I know at Google, Arup Mukherjee does this not just once a week, but nearly every day. In Peopleware, Demarco and Lister describe a situation where a crafty tech lead turned making dinner into a team-building exercise. That's the way tech leads should be: even when they're leading, it's not obvious. When a great leader is finished, the team says, "We did it all by ourselves without a leader!"
  • Schedule 1-on-1s with every team member, at least once every two weeks. This ensures that your team member has time to catch up with you and tell you about problems that are important but not urgent. Many people have a fear of speaking in public, and private meetings ensures that little nagging things that could blow up don't blind side you. You can also use this time to provide feedback. Most people crave feedback, and being able to provide useful feedback regularly distinguishes a great leader from the rest.
  • Make sure that members of your team don't hear from you only when things go wrong. Many tech leads only talk to team members only if something went wrong. Do this enough times, and your team mates will start avoiding you out of repeated Pavlovian training. If this means you have to find ways to praise them every other time you see them, so be it. In Orson Scott Card's book, Ender's Game, the protagonist instinctively knows something that many executives do not: bad news should come from the top level, and good news should come from line management. This is something that every executive should learn and do.
  • Make a big deal out of milestones. Celebrate every milestone. Reed Hastings was a great cheer-leader for every person in his company. When I applied that to small teams of 3 or 4, it was just as motivating. Most technical leaders err too much on the side of not providing sufficient positive feedback to encourage teams.
  • Try to give team members whatever they need in order to get things done. Your role is to support them. Avoid grabbing the glory by taking all the interesting jobs and treating them as minions to do work you don't want to do. Note: if you work at Google, this might not apply to you. It's more important to get promoted early at Google so your evaluations of your team-mates matter more so they can get promotions --- this is another way incentives get mixed up in big companies in a way that don't at startups.
  • You're spending well over $100k a year per engineer. There's no reason to skimp on $5,000-$10,000 worth of hardware sitting at his workstation so that he's more productive. Yes, it's nice to be frugal, but one big sign for me that a company has gone too far is when I'm tempted to bring my home machine to work so that I can have a faster machine at work. And no, I don't consider upgrades every 18 months a waste of money if your business depends on the productivity of engineers, but many companies do. I just don't understand it.
Do all of these things and you still might not be a great leader (so much of leadership is built into who you are, and not really teachable), but you'll at least not be the kind of tech lead that people and talk about years later about how bad a tech lead you are. And ultimately, if you eventually wish to have a career at a startup or even start your own company, you'll discover that these soft skills are as important as your ability to code.

(I now have a book full of useful tips like these: Startup Engineering Management)

Friday, April 16, 2010

Singapore-Malaysia-Thailand Trip Pictures

I took a vacation for two weeks in Singapore, Thailand, and Malaysia with my girlfriend and my parents. My parents mostly because they knew the area well, thereby ensured that I was eating mostly the best of the best that Singapore and Malaysia had to offer, and secondly, Malaysia is still enough of a podunk country that simply speaking English wouldn't have been enough. My dad speaks fluent Malay. =)

Anyhow, pictures and video! Pictures are in chronological order, the videos, not so much. I ended up with about 500 pictures culling from 1000, so not too bad. =)

Videos are uploaded on youtube because picasa doesn't support 720p yet. To see the difference between 480p and 720p, simply do full screen and the difference will be obvious.


Promotion Systems

When doing research for my book, I interviewed several Googler old-timers. What surprised me was that two ex-Googlers said the same thing pretty consistently: they felt that the Engineering Ladder was a very bad idea. At the time, I disagreed. I believed that in general, incentives systems worked and it was a good idea to spur everyone along with carrots. Since then, I've come around to agreeing with these two very smart people.

What changed my mind? The first was my reading of the book Drive. To a large extent, software engineering is a field full of ambiguity and complexity, and that makes compensation hard. So incentives tend to fail or backfire. Then I tried to invert the problem. What if I tried to design a promotion system to piss off as many employees as possible? What characteristics would it have?
  • No pleasant surprises. In other words, you can only be disappointed if you didn't get a promotion, you can't be pleasantly surprised by a promotion.
  • Create unhappiness by dependence on scarce resources. In other words, gate promotions based on scarce resources so that even people who would otherwise be qualified could become disgruntled through no fault of their own.
  • Eliminate accountability from people who make the promotion decisions (e.g., through a committee). That way, promotion decisions can seem arbitrary.
  • Ensure that promotions are competitive races between all qualified candidates. This ensures that people who manipulate that packet in such a way as to have the best looking packets will win over people who are trying to get feedback and improve, which is supposedly the point behind all these feedback systems.
When I looked at Google's promotion system through this lens, I was very impressed. It seemed as though the system was designed to create disgruntled employees out of people who might otherwise be perfectly happy. And note that the ex-Googlers I interviewed were people who benefited from the system, as in: they were very well respected and were hence ranked highly. They were people who couldn't care less what their ranking was, yet they disliked the system anyway, purely because they thought that having a ranking system actually had a deleterious effect on the exchange of ideas. Note that at one time the system did work: when the company was small enough that everyone knew what everyone else was working on (more or less), and hence felt accountable when making promotion decisions. The company also had so few managers that they didn't have time to put people up for promotion anyway, so putting the onus on the employee made sense. (Actually, at one point the promotion committees would consider everyone for a promotion, not just the ones who applied) Furthermore, if you tried to game your packet, chances were, someone on the promotion committee would know, so it was futile. This went away when the company got big, and got so many managers that there was no excuse for your manager not to say good things about you if he wanted to see you promoted. Someone did tell me that he thought that this evaluation of the system was biased: the alternative is the traditional management system, where everyone tries to suck up to the boss. When I discussed this with Reed Hastings, he said that he's still the biggest fan of the traditional management system executed well. When I thought about it, that makes sense, because of the following:
  • You have to select good managers anyway. Even in a peer based system, a bad manager can still sink good engineers by selecting bad goals, or blatantly playing favorites.
  • Managers are typically incentivized through business goals. If they deliberately promote poor engineers, good engineers tend to leave their teams and the business goals will be much harder to achieve. In general, most companies do a very good job of properly aligning managers with the corporate objectives. It's much harder to do so for rank and file employees.
  • A manager can be held accountable for her promotion decisions. It's much harder to hold a faceless committee accountable.
No matter what promotion system is in use, there'll be flaws. Worse, these flaws are insidious. Since superstars are those who get promoted frequently, they see nothing wrong with the system. In fact, they'll usually try to torpedo any changes that don't benefit them, and since they're the superstars, their opinions will carry more weight. Instead, it's the misfits who see the flaws of the system most often. Talented misfits will stick around for as long as compensation is adequate, and leave once it's not. And any manager will tell you that once compensation is the only thing holding employees to a company, you're in a very tenuous and dangerous situation. Silicon Valley has a history replete with companies who were stripped nearly bare of talent by smaller startups that had the ability to offer higher compensation (usually through stock options) to under-valued employees. Obviously, if you're one of the misfits, you should check your compensation package often and leave the moment it's inadequate.

Ultimately, having to have an engineering ladder at all is why I consider startups better places to work. If you've negotiated substantial stock compensation, then it doesn't matter whether or not you're a misfit or superstar under the promotion system. If the company does well, you're going to do very well. If the company does badly, even the superstars aren't going to be adequately compensated. That makes doing the right thing much easier for everyone, and explains why startups tend to have much less office politics than bigger companies.
[Update: Follow up Post]

Thursday, April 15, 2010

Selling Stock Post-IPO

Google was unusual in that it had 3 lock up periods: a 30 day lockup, a 90 day lockup, and the 180 day lockup. The idea was to let pre-IPO stock (and option) holders sell 5% of their holdings after 30 days, an additional 10% after 90 days and any amount after 180 days. The other company I heard of which got an exemption from the usual 180 day lockup rule was Netscape. In my book I discussed some of the pitfalls behind early selling, but here I want to talk a bit about selling strategy.

The first thing I observed was that the float at the time was tiny. This meant that it was conceivable for Google employees who are divesting to move the stock. I also observed that employees (especially employees with large stock holdings) were mostly housed in Mountain View. Many of the large stock holders were engineers, and were Owls, rather than Larks. Since I was effectively locked out of the 30 day lockup due to my vesting date, I got to observe market movements.

Indeed, what I saw was that the highest prices of the day tended to come before 10am PST, and the stock would take a consistent drop after that and stay low for the rest of the day, before another morning spike the next day. I saw this repeated day after day, week after week, and that convinced me that my theory was right.

Obviously, for me to take advantage of the idea, I had to ensure that I was the only person who would execute on it. Otherwise, everyone else would pile on and I would lost my chance to capture the morning spikes. So I deliberately unsubscribed myself from the internal financial-planning discussion mailing lists, and deliberately kept myself from blogging during that period, internally or externally. And yes, I did capture the morning spike when I finally did sell (I'm a lark by nature, so getting up at 6:30am PDT/PST to trade wasn't a problem).

Note that this only occurred at the 30 day and 60 day lockups. At the 180 day lockup periods, institutional investors/venture fund holders could sell, and those guys could move huge chunks of stock in a day, and employee movement would be lost in that noise.

In late 2007, I did the inverse. I took an informal poll of relatively naive employees, and asked if they were going to sell at the all-time high of $700+/share. When everyone else told me that they were holding on for more capital gains, I remembered Warren Buffett's adage: "Be fearful when everyone else is being greedy, and be greedy when everyone else is fearful", and sold until my hands shook from pushing the "sell" button. It was one of the hardest things I ever did, but was also incredibly lucrative. My only regret was not selling even more than I did.

I am not normally a market timer: those of you who've read my financial posts over the years have probably noticed that I have no faith in anybody's ability to time the markets, including my own, and I follow a rough asset-allocation model. But as William Bernstein said during his visit to Google on one of my birthdays: Just because you believe in the efficient market does not absolve you of the responsibility to do the math and look at what makes sense.

An investor always has the responsibility to check current conditions against common sense. As the above two examples illustrate, common sense means that when you know the market is wrong, acting on your beliefs quickly and decisive.

Wednesday, April 14, 2010

Review: Dollhouse

Dollhouse is Eliza Dushku's vehicle for showing off her versatility (lead actress and producer), and it's Joss Whedon's world for exploring the nature of identity.

The show is superficially a science fiction show, but there's very little science. The premise of the show is that you've got an outfit that can imprint personalities into people (called Dolls), and then use those people to fulfill fantasies, or perform more sinister roles. The obvious uses are touched upon, but the show really hits its strides when the Dollhouse's sinister past and secrets come back to revisit it.

The acting is generally good, with Olivia Williams stealing every scene she's in, and lending what would otherwise be a very dull character a surprisingly sympathetic life. The writing is intelligent, and with none of the teen buffyisms you almost come to expect out of a Joss Whedon show. With extremely strong female leads, however, it is very clear that this show plays to his strengths.

The technology, however, requires massive suspension of disbelief, not just of the basic tech piece of "imprinting", but from the giant ass plot-holes in the show. For instance, we're supposed to believe that the brilliant people who build all this amazing technology have never heard off-site backups (despite the Dollhouse being a widely distributed operation). There are basic security issues that are never addressed, and of course, many episodes leave you wondering why the equivalent of the New York Times in this universe hasn't already exploded the story.

All that aside, however, I enjoyed the themes that are explored, and definitely enjoyed the plot twists: they were both surprising and believable at the same time. Ultimately, however, I agree with Grant McCracken (I'd provide a link, but his blog seems to be down) as to why the show failed to connect with a broader audience:
Ours is a transformational world, and to that extent Dollhouse should speak to us.

But there are two kinds of transformation: involuntary and voluntary. We hate the former. We love the latter even more.

Which transformation is Dollhouse about? It's about a woman (Eliza Dushku, pictured) trapped in an endless series of involuntary transformations. A mysterious corporation wipes her soul, inserts a new persona, and hires her out. This is the stuff of our worst realities and dreams.

The trouble is not that we can not identify with "Echo." The problem is we can identify too well. We too have been the captives of forced transformations, and it gives us no comfort to see someone else endure this horrible condition.

All in all, it's a short Season (13 episodes), and I don't think it was a waste of time at all. It's not on par with the best Buffy season, but I consider it a show that's at least as good as Firefly.

I look forward to checking out Season 2 from the library.

Facebook is stealing Adsense Revenue

Just before I left Google, I heard rumors that if you told your manager that you were leaving to join Facebook, you would be escorted out of the building right away. What's interesting to me about this rumor is that it was quite believable, and also despite all that publicity about the Apple-Google rivalry, there was no such rumor circulating about that happening if you were to leave Google to join Apple.

Now, first of all, I will state my anti-Apple bias: I think that Apple would be a horrible place to work if you're a Googler. It is in every sense the anti-Google in outlook. I have friends who worked there (and one who still does), and I know for one thing it would drive me crazy. People tell me that there's so much secrecy that when you have to debug something, getting enough information to even do your work if the bug spans up or down one level of abstraction would be a challenge. And obviously, there's a lot more secrecy than there is at Google, even for internal employees.

On the other hand, while Apple might be able to start establishing ads on mobile devices, Facebook is already stealing ad revenue out of adsense. How? Take this blog for instance. I run Adsense Ads on the side and bottom (unobtrusive, but valuable if Google does a good job of monetization). The blog, however gets syndicated to various places: Buzz, Friendfeed and Facebook. The first is controlled by Google, the second doesn't try to monetize, and in fact helps drive traffic to the blog. The last, however, strips out all Google ads, and replaces it with Facebook ads!

Not only does Facebook see all the revenue that Google otherwise would see, it also grabs all the revenue that I would otherwise see, since Google shares Adsense revenue with the content provider (in this case, me). You can imagine a world in which all content thus gets syndicated through Facebook and Google's content ads network goes to zero.

So if the rumors are true, then indeed Google is terrified of Facebook, and has every reason to be. The potential for disruption to its content ads network is huge. The caveat, of course, is that the high traffic sites which have lots of traffic clearly have a disincentive to syndicate their content through Facebook, so it's only low traffic sites like mine that are affected by this. My guess though, is that Facebook will eventually provide some sort of revenue sharing arrangements with the high traffic sites, at which point it would really start chipping into Google's content ads network.

Then of course, there's that other insidious threat: Facebook is currently in the position Google was 6.5 years ago, being able to offer hot pre-IPO stock to potential employees. This is a cycle in Silicon Valley, however, and the correct response is to properly compensate your employees rather than threaten to escort them out of the buildings when they give notice. I doubt if upper-management at Google is unaware of that, given how recently the shoe was on the other foot.

Tuesday, April 13, 2010


Google has an unusual perk: on-site laundry machines. When I first joined, it was just one laundry machine sitting in building 0. By the time I left, there were no less than 4 laundry rooms with entire banks of washers and dryers. When I first joined, I didn't think much of the perk: one laundry machine that operated at European speeds meant that it was never available when you needed it.

When the first laundry room with a bank of machines showed up, I started using them, because it was more work to dig up quarters for the laundry machines at my apartment building than to just dump the laundry onto my trailer and tow it to work on my bicycle. In California, you can do this most of the year and not even need a waterproof bag for the laundry. I also started keeping a week's worth of clothes at work and then never even bringing those home: this let me do a bike commute sans saddlebag, if I so chose to.

As mentioned in my book, Google had an unusual number of fresh graduates. This led to the kind of "accidents" you would expect in a college dorm. One day, I came in to the laundry room to find one of our administrative assistants in a huff. "Darn Engineers with PhDs shouldn't be allowed to do laundry," she muttered under her breath. I asked her what happened, and she said, "Some idiot PhD went and stuck washing powder into the dryer. Now the whole thing's jammed up and there's powder all over the place." She went out, grabbed a marker and a sheet of paper, and taped up the dryer with an "Out of Order" sign. Then she had a better idea. She went out, grabbed a ton of paper and tape, and wrote on them: "Washing machine. Use detergent here. Do not put dryer sheets here.", and "Dryer. DO NOT using detergent. Use dryer sheets here!" And proceeded to label every machine in the laundry room.

Since then, we have not had a repeat of the "detergent in drying machine" incident. So at least we know that even engineers with PhDs can read.

Monday, April 12, 2010

Referral Programs

Most companies, startups included, have employee referral programs. Nearly every corporation will tell you that referrals are the best source of their employees. If you compare the cost of an employee acquisition through a retained recruiter and through a referral, it's usually an order of magnitude. A retained recruiter usually gets 30% of the recruit's first year salary: for a $100k/year engineer, that's $30k a year. The typical referral bonus paid out to an employee is less than $3000.

One response that companies make when faced with urgent need for hiring is to raise the referral bonus. Strangely enough, this makes very little difference to the rate of recruits unless the referral fee is increased dramatically. In fact, the biggest way you can get more referrals from your employees is to make them happy. Google was the master at this: the campus is filled with fun exhibits, so when you invite your friends over for lunch, you pretty much have to take them all over campus and show them the various cool things. It makes you feel good, and it makes your friends want to work at this great environment. The mind-blowing free food perk is also amazing, and Google did not enforce strict limits on guests: these guests frequently became future full-time employees.

People recruit friends because they want their friends to be happy too, and even new employees are very good at filtering people whom they instinctively know won't fit in the culture. It wasn't uncommon to see a note from an employee referring a friend who wouldn't fit thus: "Hi, this is my friend X. He's a major pain in the ass to work with, and I wouldn't want him here. I don't want to offend him by not referring him, so could you please write him a note saying that there's no position here for him?" For some positions, you might want to ignore such notes, but by and large, if the person with a financial incentive to get someone hired would forgo that incentive so as to never have to work with that person, that's a pretty strong signal.

Finally, if you start to see the number of referrals drop off, this is a signal not that your carrot isn't good enough, but that your employees are no longer as happy at work. Nobody wants to bring their friends into a toxic work environment. Unfortunately, very few companies pay attention to this as a signal.

Sunday, April 11, 2010

Review: Garmin Edge 500

Someone I knew decided he didn't like the color of his Garmin Edge 500, so I bought it off him for $200. You can buy the version that's bundled with a HRM and Cadence/Speed sensor, but it's cheaper to buy those pieces separately. Since I didn't care about having a Cadence/Speed sensor, I just bought the Garmin Heart Rate Monitor. [Update: I bought the Cadence/Speed Sensor after all]

First, a note about what the Edge 500 is and is not. It's not a navigation unit like the GPSMap 76CSx or the Edge 705. There's no slot for an SD card to store maps, and it won't even give you route pointers or directions. It is purely a replacement for a traditional bike computer. My Sigma 2006MHR had given up the ghost recently, and I wanted a stoker computer for the bike position anyway, and the Edge 500, while expensive, had the benefit of not needing wheel magnets and sensors, as well as not needing adjustment when I switched it from bike to bike. Having nice looking tracks when you're done is a plus, though a minor one.

The unit came with 2 bike mounts, though you can buy more mounts if you have other bikes. The tie-on is very similar to the Sigma's with 2 rubber bands. The computer slots in and then twists 90 degrees to secure. I found the mounting not as reliable as I expected, since in pushing the buttons I ended up twisting the computer back towards the unlock position a couple of times.

The computer itself is fairly easy to use. There's a power button, and a start-stop button. A mode button switches between 3 different configurable displays, and accessing the menu system to switch settings, bikes, etc., did not require reading the manual. One note that the person who sold it to me said to do is to ensure that you always stop the timer and immediately reset the GPS unit while you still have a GPS signal, as otherwise the unit will lose all the data from your ride. I can definitely see this being very annoying.

Connecting the unit to my Windows 7 PC through a USB cord was straightforward, though I had to use the control panel to force Windows to recognize the unit. Once that was done, tracks uploaded quickly and easily. If you want to see my tracks, here's a hilly ride without the HRM and the next day's recovery ride with a HRM.

GPS speeds are not very accurate. In the first ride, the GPS unit recorded a top speed of 55.2mph. I had a separate, traditional bike computer on that bike (it was a tandem), and it recorded a top speed of 41.5mph. That's a 20% error. In other words, if you went on a Tour of the Alps, and all you had was a GPS track telling me your top speed of 100+kph, I won't believe you. Use a traditional bike computer if you want accurate speed measurements. Or get the speed/cadence sensor. The temperature sensor is also not very accurate, and doesn't react quickly to changes in ambient temperatures. This is of concern because without accurate temperature sensor, elevation changes measured by the barometric altimeter might not be accurate.

One improvement from previous years is that Garmin's Connect software now does appropriate smoothing for elevation gained/lost if you turn on the Elevation Corrections on the web-site. This makes Garmin's elevation gained/lost reports much more believable than in previous years.

Battery life with the Edge 500 is much better than in previous units. I've recorded about 7 hours of elapsed time with the unit on, and it reports 75% battery life. [Update: After two back to back rides (King Ridge, Coleman Valley Road]without a recharge, the unit was down to 30% from 100%. This was with both the HRM and the GSC 10 in use, which means that a 15 hour battery life is a reasonable expectation] The quoted 18 hour battery life seems very achievable. The unit is also light enough that I wouldn't consider leaving it behind on the Tour of the Alps just to save weight. All in all, I'm satisfied with this purchase as a bike computer replacement. It costs about $100 more than a similar non-GPS bike computer, but does quite a bit more. However, if you care about navigation features and absolute minimum weight isn't a big issue, I would still recommend the Edge 705 instead. I'm certainly not leaving behind my 76CSx for the upcoming tour, but would happily use the Garmin Edge 500 on my single and tandem for local rides.

Facebook: Google Redux?

It amuses me that what people say about Facebook today sound substantially the same as what people said about Google six and a half years ago. What were they saying to me back then?

It must be too late to join Google now. There's no way you'd become a millionaire by joining them at such a late stage.

How could they possibly make money? Those little ads? Nobody clicks on those! I don't know any one who clicks on those things, and I myself never click on those!

The former statement obviously might or might not be true, depending on how generous the stock compensation package handed out to employees today are. The latter statement, however, is easily verified: try placing an ad on Facebook. The last time I tried, it was a minimum of $0.60 per click, and they wouldn't take my $1/day budget. For reference: Google will happily take a $1/day ad campaign, and will take bids as low as $0.01.

When I visited Facebook campus a few weeks ago, it reminded me of Google campus 6 years ago. The parking lots were overflowing, and people were clearly working at high intensity. Even the cafeteria looked familiar (well, it should, since it was for all intents and purposes Google's old Cafe-14, complete with the same staff).

Ultimately, two things convinced me of Facebook's value. The first was when the Caja team needed a contact with Facebook 2 years ago. I dug through my Facebook contacts and came up with Jeff Rothschild, who's the VP of Technology at Facebook. I made the connection and they had their meeting. (Jeff by the way is featured in my book) Now, there might not be an easy way to monetize this (though see above about the price per click on Facebook today), but I was convinced of the value of Facebook after that.

The second thing that convinced me of Facebook's value happened when I was in Europe, and again when I was in Australia. Over and over again, whenever I met someone, instead of giving me their e-mail address, they'd tell me to friend them on Facebook. This by the way, is a very bad idea for most Europeans: there are so many name duplicates on Facebook that it's impossible to disambiguate the two hundred "Ben Smiths" for instance. I tell people to friend me instead since my name is much more unique than a typical European/Australian/American.

Thirdly, people I talked to who weren't in Silicon Valley all thought that Facebook must have been a giant corporation with more than 10,000 employees. More than anything else, having small companies have such a huge impact on the rest of the world is the major reason why Silicon Valley is still the place to be, if you're a computer scientist.

Finally, if it is true that Facebook has found a way to monetize social networks, this is huge news. The Google-MySpace deal famously fell short of expectations, so if Facebook really has trounced Google at the social-network monetization game, that would be huge.

Saturday, April 10, 2010

AMT and stock options

An e-mail came into my mail box addressed to a group of high net-worth friends. Apparently, a group of individuals were trying to put together a sum of money structured as a loan to an employee of a high profile startup in Silicon Valley. The startup has been noted as "the next hot IPO" in the Valley, and an early employee had recently exercised all his or her stock options, which triggered a large AMT capital gain immediately on the difference between the current "fair market value" price and his option price.

Let's take a look at this from an employee point of view: as a private company employee, he's in a very bad spot, since he can't actually sell enough stock on the open market to pay the AMT due. If there was any way he could sell the stock at all, that's the correct way to go. Alternatively, he could attempt a private placement (i.e., sell it to accredited investors) in order to raise the funds. That would however require expensive investment bankers, etc., in order to structure a deal. So it's understandable that he would try to structure a loan using some stock as collateral.

From an investor's point of view, however, this is a bad deal. If the hot startup collapses instead of going public, then the investor loses all his money anyway, since there would be no recourse other than the aid private company stock, which would be worthless. If the hot startup goes public, the investor would have his upside limited to the agreed upon interest rate. The only way such a deal would be attractive was if the investor got equity as well.

Obviously, the best thing to have done would have been for the employee to have read my book An Engineer's Guide to Silicon Valley Startups several years ago, but since it was only published this year, it's understandable that the advice (which is not at all common knowledge) wasn't followed.

Preparing Employees to be Wealthy

Of all the companies I ever worked for, Google put the most effort into educating its employees about financial planning prior to its IPO. I was impressed by the folks brought on campus to talk to us. These included Burton Malkiel, Bill Sharpe, and Robert Shiller. If after this world-class education in financial planning, investing, and basic economics, you couldn't figure out what to do with your money, it wasn't Google's fault for not providing opportunities to learn!

Google also organized financial fairs, where the big name financial firms, local financial advisors, brokerages, and even Vanguard would show up and talk to you one on one to try to sell you on their services, if that's what you felt you need. I didn't go, since I pretty much have my strategy all laid out, but I know others did make use of those services. Over the months coming up to the IPO, I saw more than one high level engineer/old-timer schedule meetings with these crooks financial professionals.

One of my friends did attend, however, and told me this story:
So I was at the Morgan Stanley booth, and O. came up and started talking to the MS rep. As you know, O. is very financially sophisticated, and started asking all these complicated questions about forward-contracts, asset allocation, options, and hedging strategies. You could see the rep's eyes get bigger and bigger. He clearly decided that O. was a potential big huge customer, and put lots of effort into selling him. Then C. came in (C was a serious old-timer with a huge amount of stock), and asked a couple of naive-sounding questions. You could see the rep getting irritated that this guy who by the looks of him wasn't in the same class as O. did was wasting his time and replied very curtly and tried to get rid of C.

At that point I couldn't help myself and laughed very hard. As with everything else, appearances are deceiving, and the sales rep had no way of knowing who the big fish was. Here in the valley, judging people's net-worths by their appearance or mode of dress is a serious mistake.

Review: Freedom

I found Daemon enjoyable, so I looked forward to reading Freedom when it came out. Unfortunately, it came out during a particularly busy time of my life, so rather than buy it, I waited in a virtual queue at the library.

Freedom continues where Daemon left off. We've got a networked AI of sorts slowly infiltrating the world, while a group of plutocrats try to fight back and take advantage of its capabilities for its own ends. As with the previous book, the characters are nothing to write home about: their motivations are simple, and even the (very mild) romance thread seems mishandled.

What the book does do a good job of, however, is to imagine a world where a MMORPG designer had the tools to reshape the way the world worked according to game design principles. You would have quests, levels, and ability to gather resources based on resources, and on top of that, a reputation based economy. But what would people do with it? Could you really run a community, a country, or the world this way? How would the non-digerati react? Dan Suarez hints at some answers, but is better at demonstrating the immediate consequences that unrolling the full implications of the world he has built.

All in all, if you turn off your brain, and then treat this book as a techno-thriller, you won't be disappointed: loud explosions, death matches, and full on fights are common. You could see a Hollywood movie made from this book. Furthermore, there's actually a conclusion: it's a bit pat, and I find it difficult to imagine someone who would actually execute such a vision to have the humility to build in the kind of checks and balances Suarez has him do, but there are no hanging threads and if there's a sequel, it'll have to involve different characters.

This novel is unusual in that it has a bibliography. It's an unusually well-read bibliography as well, including The Transparent Society, a book all too often ignored in the privacy debates, but is a much better book than most privacy advocates give it credit for. Recommended, but I'm not sure I would buy it at full price.

Friday, April 09, 2010


If there's anything Google loves to build, it's infrastructure. Google had entire buildings worth of machines, and lots of ways to make use of all of them. There's MapReduce, Bigtable, Blobstore, and all sorts of other distributed infrastructure. So much so that engineers frequently told me that they thought that developing without all that infrastructure would be crippling, and would slow them down too much.

The irony, of course, is that companies like FriendFeed gives the lie to that statement. Friendfeed was launched in weeks! If you're not Google, you don't have to scale to billions of users right away. Existing tools can be made to be extremely scalable. For instance, even MySQL can be made to scale. The truth is, launching products at a big company takes a much longer time than at a startup because of non-coding related reasons. In fact, many of the engineers who made that above statement would find creative ways around missing infrastructure if they were at a startup: context is everything.

I remember attending a talk by YouTube engineers after the acquisition (this was at OSCON, so I know it's unclassified information). What impressed me was how closed they always seemed to be to falling over completely. Yet they never did. Then it occurred to me that a startup should always be running at the ragged edge of what their systems can handle: to do otherwise would mean that you're not using all your resources efficiently. By contrast, Google can afford a few under-utilized machines. In addition, all that generic infrastructure has overhead. Generic cluster management software, for instance, doesn't (and can't) know enough about the overall job structure of your tasks to put compute-intensive tasks on the same machine as network bound tasks. But a startup with a customized software stack can do that (and frequently must do so) because they don't have enough machines to do otherwise.

In short, I think startups have to be very careful about building generic infrastructure just because that's the way Google did things. Google built generic infrastructure because its big problem (search) had to have massive scalability right away. Even with a single user, a search engine still has to search as much of the web as possible. But what applied to Google doesn't apply to all startups. Build only the tools you need as the need arises.

GOOG digger

After about 6 months, I finally had enough time to sit down at Charlie's and have a meal, rather than eating at my desk, or more frequently, just snacking rather than eating meals. So sitting down alone in the cafeteria, I was about to get started when I heard an unfamiliar voice say, "Mind if I join you?" I looked up from my plate and saw a gorgeous Asian woman with a tray of food. Well, any geek will tell you that us dorky guys rarely get pretty women inviting themselves to sit with us when we're alone eating dinner, so I agreed.

She was in [department redacted], and we spent the next 15 minutes exchanging details of our work, what we were working on. Since we were in completely different departments, we had lots to talk about that was new to each other (or so I thought). Then she asked me this question: "So... how long have you been here?" My response, "About 6 months."

Not five minutes later, she looked at her watch and said, "Oh no! I have a meeting to run to!" It was already 7pm, and at that time we didn't have distributed offices to introduce time-zone craziness. Off she went, and I never saw her again. I thought nothing of the event and finished my dinner. When I told my girlfriend Lisa the story, she laughed hard for 10 minutes and said, "Next time a pretty girl in the cafetaria asks how long you've been working there, try saying 6 years!" I then realized what had happened and laughed along with her.

Thursday, April 08, 2010

Unicode Names in E-mail headers

If you've corresponded with folks via gmail, you might have noticed that some folks write their English names, and then put their non-English names in parenthesis after that. For instance, in my case, I write: Piaw Na(蓝俊彪).

This use started in my office in 2005. I was complaining to Pengtoh about the pain of using Chinese input methods, and how they weren't actually easy to use on my Linux workstation anyway. Pengtoh pointed me at Mandarin Tools, and I used that to construct my Chinese name in pinyin. We then wondered whether Gmail would handle unicode in the headers fine, so we tested that by sticking our Chinese names in the Gmail settings. (You can find this in the "Accounts and Import" tab of the Gmail settings)

It turns out that Gmail handled Unicode in e-mail headers just fine. We then decided we wanted to see if we could make this convention common. We did this by ourselves, and then decided that we had to recruit non-Asians to make the meme stick as well. So Pengtoh contacted a few more of our non-Asian co-workers, and offered to construct Chinese names for them.

I knew we had succeeded when I started seeing Japanese names in parenthesis, then Hindi over the years. The unfortunate part in the early days was that many other mail readers could not handle unicode characters, either blanking them out, or turning them into gibberish. To solve this problem, I also constructed a non-unicode name for use with those mailing lists and corresponding with people who used these mailers. You can do this by adding more than one entry in the "Send Mail" as part of the Gmail settings. My hope is that all mailers everywhere can handle non-Unicode names in headers eventually.

It's not inconceivable that somewhere else, someone else came up with the same convention. However, I do not recall seeing this convention pre-2005, so at this moment, my best guess is that Unicode names in parenthesis originated in my/Pengtoh/Ovidiu's office in 2005. If you know of an alternate origin, please let me know in the comments!

Wednesday, April 07, 2010

Corporate Culture

Corporate culture is a nebulous term. People use it to include everything from free food to whether or not you can bring your dog to work. I personally think that far more important is the way we interact with each other, and how things get done.

One winter, I decided that it would be a good thing to run a bicycle repair workshop as a series of winter classes. I didn't (and really couldn't) teach every class myself, so I put up a shared spreadsheet, and listed a couple of sessions that I would teach, and then Mark Logan, Ryan Kauffman, Bob Sutterfield all pitched in and taught classes I couldn't teach. It was typical of Google culture that volunteer-ism was common. You could almost always get help from someone if you went up to them and asked about something. If they didn't know, they'd point you to someone who did.

Unfortunately, corporate culture is one of the first things to get diluted when a company gets big. And indeed, when Google got to 5,000, then 10,000 and finally 20,000 people events like this got less common. An attempt was made to formalize events like this as "Googles Teaching Googlers." Pardo's wheel-building lectures were made available that way. While it worked to a certain extent, the formal version to my eyes, were always a pale reflection of the spirit of community that prevailed the the company was much smaller.

Tuesday, April 06, 2010

Food Story 2

Growing up in Singapore meant that I really loved food, and always appreciated good food. However, I'm also cheap, and so never went to any fancy restaurants. It wasn't until Google that I actually had a formal, multi-course Western-style dinner.

By 2005, those of us who were on the Sarbanes-Compliance projects were deemed to be "done". At other companies, this might mean a bonus, but Google had something much better: in-house Chefs who could prepare a fancy meal on a budget. Many people think that the in-house Chefs at Google are an ostentatious perk that was an expensive luxury for employees, but in reality, I think the value Google got out of them in terms of extra work from employees and being able to run award-type events really cheaply meant that the culinary staff more than paid for themselves.

We sat down in a room near Charlie's (this room would eventually become the B40 gym), which had been laid out like a restaurant, complete with a special door from which food would arrive. I'd never seen table-cloths so white, nor had I ever had a place-setting with this many implements. I thought, "OMG, I'm in for a treat."

The first entrée arrived. It was a salad with dressing. I had been trained to get hungry by 6:30, and was starving, so I ate it with relish. Then came the Ceviche. That was really tasty too! Then came the sorbet. I was shocked. That's it? That was my fancy dinner? I was still hungry. "Oh well," I thought, "I can still go grab a burger at Charlie's afterwards." When the main dish arrived after the sorbet (it was Filet Mignon), I finally realized that the sorbet was a palate cleanser, not desert. The rest of the meal was fantastic.

Google ran many other "event"-type dinners. One of my favorites was the chocolate-themed dinner that Chris persuaded Charlie and the culinary team to run. That was a $20 dinner, but my goodness, you got $100 worth of food out of it. Another week, Google's culinary team ran the Cafe Crawl: visit all the Cafes in a week:
Lea and I used bicycles to visit all the Cafes in one lunch period. The reward: a special meal, and a pass that let you skip to the front of the line for a week! Needless to say, I took advantage of that pass to get a huge amount of sushi.

Google cafeteria reached their height in 2007 --- when I visited Paris in 2008 for a culinary tour, I unfavorably compared some restaurants in Paris to Google's Cafe 5IVE, for instance. I was sad to return from Germany to discover that many of my favorite Chefs had left. Olivia Wu and Scott Giambastiani are still at Google though, and they still turn out meals that could blow your mind if you were used to the typical corporate cafeteria.

There's a talent war brewing in the valley over corporate Chefs, so hopefully, having great food as part of your compensation package will be more common.

P.S. I'm fasting for an annual physical/blood-test, and writing this post while fasting was a mistake!

Monday, April 05, 2010


An friend of mine was very upset at work. "I look at my bug list and I just want to cry," she said. Now, this was a person who was single-handedly developing and supporting a program used by millions of users. It was significant, important work, and she knew it. But in the face of this hugely negative feedback, even the most self-confident of us would falter.

I had just gotten my first fan-mail as a result of the book, and asked if she had gotten any? "No! Do you think I would be so pissed if I had fan mail?" Here's the thing: there was nowhere in that program that exposed who the developer was. No about box, no credits page nothing.

Compare this to a movie, where everyone from the Key Grip to the Best Boy gets named. Open Source software at least makes a step in the right direction: the Firefox about box gives credit to everyone involved.

This made me think about all the hype I've heard over the last few years about how few engineers there were in general, and how few women engineers there were in particular. One problem is that we hide away all the people behind the amazing products we use every day. Who was the chip-designer behind the iPod's touch technology? Do you know? I know, because I went to college with her. The main excuse most corporations give is that "if we exposed the engineers behind the products, we would be giving competitors a list of people they should poach from us."

I personally think that's a piss poor excuse. Chrome was promoted by a comic book. The comic listed names of many engineers who worked on that product. As far as I know RIM, who is in dire need of a useful browser on their phone, hasn't called any of those engineers asking if they could build one for them. For the engineers who were named, however, the delight of having their names, and faces enshrined in a comic book drawn by Scott McCloud, however, has got to be at least as good as the Founder's Award they got!

Personally, I think until engineers start demanding that they be credited for important software they contributed to, and corporations start recognizing and treating them as people who are deserving the credit rather than being hidden behind a corporate brand, I think we shouldn't be surprised that kids who grow up with iPods, iPads, Android Phones, and other products chose not to go into engineering. After all, those products weren't made by people (whom they could aspire to be), they were made by faceless corporations. And court orders aside, no kid aspires to grow up to be a corporation.

Hiring Committee Stories

Google's engineering hiring is unique as far as I know. Patterned after faculty hiring by top universities, Google engineering had no hiring managers. None. This meant that all the usual job search advice by any number of books and web-sites didn't apply to Google. Bypassing HR and trying to get to a hiring manager didn't do any good. About the only thing that could have been useful would have been to get a strong employee referral (i.e., get an employee who thought the world of you to say so when he submitted your resume). The interview process went like this: you would interview with a panel of engineers, who would then write feedback to an internal database, which would then go to a hiring committee (also composed of engineers) to evaluate the feedback and provide a go/no-go decision. Since engineers might have to live with the code (and personality) of a bad hire, the hiring committee tended to be conservative on hires. Phone interviews and in person interviews were conducted almost exclusively by engineers, with directors brought in only if the candidate requested a meeting with a manager explicitly, or if the candidate himself was interviewing for a manager/director position.

For reasons explained in my book, I ended up on the Site Reliability hiring committee. At that time, the hiring committee was composed out of relatively senior engineers: Lucas, Ben, Bogdan, and various engineering big-wigs like Bill and Urs. Frequently, when the committee found feedback on hiring to be ambiguous, it would assign another interview to an engineer well-known to be decisive (i.e., someone who would be willing to stick his neck out and say "hire" or "no-hire"). This happened surprisingly frequently because many people dislike rejecting people, and occasionally, someone would write feedback that wasn't really informative enough.

We didn't always have the luxury of a second-interview, however, since some folks had to be flown in from far-away places. Google was truly a global company, and in its pursuit of talent would consider resumes from literally anywhere in the world. Now, I didn't think that Google's interviews were particularly hard, compared to startups and other well-known firms in the industry. In general, quality companies reject a large number of engineers because most people who call themselves programmers can't code.

A few incidents came to mind as being particularly funny:

One day, I came to the hiring committee and started reading feedback from interviewers. One of them turned out to be a candidate I had interviewed earlier in the week. I was doing as much as 5 interviews a week at this time, so I didn't always remember the candidate by the time I got to the committee. Lucas's feedback for the candidate started with, "I spent the first five minutes of my interview calming the candidate down after his interview with Piaw..." When the others got to this part of the feedback there was a lot of laughter. I think that was the moment I realized that Bogdan and I would get along, because he high-fived me across the table. The candidate was a no-hire, but I don't think it was because I was particularly harsh.

At one point, we came across a candidate who had to be flown over from the other side of the world. Since we knew there was to be no chance of re-evaluating this candidate if the feedback was insufficient, we asked the recruiters to make sure that we had decisive, experienced interviewers for this candidate, who seemed pretty senior. She replied, "How about Piaw, Ben, Bogdan, and Lucas?" When he heard this, Bill put his head in his hands and said, "Why don't we just save ourselves and the candidate some time and just send him a rejection letter now?" The room burst into laughter.

Our committee took hiring seriously. We agonized over many hiring (and no-hiring) decisions for many years, learned the idiosyncrasies of many interviewers, and tried to match them up to candidates as well as possible. It was very high intensity work, and on one or two occasions I had to go head-to-head and argue my case in front of VPs because I felt strongly about one candidate or another. I didn't always win, but at every point everyone's opinions were considered. I'm sure we made mistakes, but looking back, I'm not sure I would have found a better process. I for one think that the decisions the committee made were far better than the decisions each of us individually would have made.

Sunday, April 04, 2010

Turns out I lied...

I didn't retire from Google, I was handed a pink slip! Thanks, Stephan, Larry, Lea, Pardo, Roberto, Catherine, Mike, and Parisa. With such appreciative colleagues, I must have been a fool to even consider retiring. Thank goodness they set me straight.

(I swear it looked more pink in person)
Posted by Picasa

Orkut Story

As many people know, Google ran an annual ski trip for employees, and by the time I joined, it was getting pretty big, with Google taking up most of Squaw Valley's lodging. I remember checking in, and then heading over to my room to shower.

When I got to the room, I noticed that not only it was a suite with separate party area from the bedrooms, but the suite had a table that was piled with Vodka and alcohol floor to ceiling! I thought to myself, "Wow. Google's incredibly generous. Not only did they book an entire suite just for 2 people, they must have also negotiated some amazing deal to give everyone enough Vodka to keep them drunk for weeks!"

I took my shower and then finally noticed that my roommate had already checked in before me and left his luggage in the room. I took a look at the name tag, and that explained everything. It was Orkut Büyükkökten. The alcohol didn't come with the room, it came with Orkut! I was momentarily horrified, since I actually entertained thoughts of sleeping at night so I could go cross-country skiing the next day, but Orkut had clearly intended an all night party. I then realized that I could easily swap with someone who wanted to party all night!

Mike Samuel came to my rescue and bravely agreed to swap places with me, so I got a good night's sleep, and he had to put up with all that alcohol. Over the years, I learned to bring ear plugs with me to the ski trip, since even if I wasn't sharing a room with Orkut, the guy next door could be a hard partier.

One note about Google parties: the music is universally set too loud. I always felt as though the parties were for people about 25 years younger than I was. I thought I was alone in thinking that, and that I was being a fuddy duddy, but then the 30-year old and 25-year old Googlers told me that too! Until the company split up the departments so each department could have its own party, I never did attend what I considered a good party: one in which you could hear your colleagues talk in a social setting.

Saturday, April 03, 2010

Getting confused with Meng

One of the perpetual sources of amusement to others (and occasionally myself) at Google is that for whatever reason, I get frequently confused with Meng by other people. This is very confusing to me, because Meng looks like this:

Yes, that's right. He's always shaking hands with a president.

I look like this:
From Pigeon Point 2010

(And that's my good side)

Clearly, Meng is much better looking than I am, yet people confuse me for him! I got annoyed at this at first, but then decided that since it was mostly white people making this mistake, I could understand. When I first moved to this country, all the white people looked alike to me too. It took me a while to realize that color of hair and eyes could be used as distinguishing features.

Then one day, an Asian woman mistook me for Meng. What's worse, I was even riding a bicycle. Meng doesn't ride a bicycle. About the only thing we share in common is that we both grew up in Singapore, and are friends with Pengtoh.

Then one day, Meng came up to me and said that this guy walked up to him and started talking about cycling. I felt mortified that at least once in a while the mistaken identities went the othe way.

Anyway, now that I'm no longer a googler, there's no excuse for mistaking Meng and I. Just look for the Google badge. If there's one, then it's Meng. If there's none, then it's me. Until Meng retires from Google, that is... I don't know what to do if that happens.

Friday, April 02, 2010

Google food story

When I first joined Google, the menus seemed so exotic to me that I resorted to having to search the web to figure out what was being served. The internal mailing list, food-discuss, also had fairly active discussions of certain menu items.

One particularly memorable exchange came up when the dinner menu included the entrée John Dory. The conversation went like this:

"What's John Dory?"
"It's a fish from New Zealand."
"Oh thank god! I was afraid that some hapless Googler pissed off Charlie and got turned into the dinner entrée!"

Last Day at Google

Today was my last day at Google. It's been 6.5 years since I first started at Google, and let me tell you when I first joined I did not expect to stick around for that long. My previous longest tenure was at Mpath Interactive for 3.5 years. I remember joining the company, and thinking, "Boy, this company is so huge. I'm going to get lost here." I became very pleasantly surprised that after my very first TGIF, I ran into Eric Schmidt on the way back to my cube and he knew my name, had obviously read my resume, and knew that I had "been around the block a few times as far as startups were concerned." Looking back at it and reflecting on what my starter project at Google was, I shouldn't have been so surprised.

Financially, it stopped making sense for me to stay at Google about 2 years ago. The risk-reward ratio had mostly tipped down to much less risk and correspondingly less reward, and while regular refreshers were handed out, they weren't really enough to really make a big difference to my net-worth. However, I still had interesting things to do, including my Munich assignment. Munich was such a small office that I felt like I was at a startup again, which was awesome. I felt like I did a lot there, and it was enough to keep me excited and motivated.

After my Australia trip last year, I became a part-time employee, going to a 4-day work week. Amongst other things, I scanned 10 years worth of slides, bought a house, toured Japan, and wrote a book, but somehow still never quite got caught up to everything I wanted to do. And having found a taste for long-term travel (as I got more time, I discovered that I didn't want to do more trips, but wanted to do longer trips), I wanted to do still more! I still never did find time to visit my friends, or even do any of the other nerdier things I thought I could make time for.

The final straw that made me decide to retire from Google was the realization that as an engineer and a professional, I'm highly optimized for startups. As Google got bigger, the pressure to specialize and stick to a formal role grew to the point where everything I did (and that included the assignment in Munich) came at the cost of professional advancement. I supposed I could have sat back and coasted, or as a friend of mine used to say, "rest and vest", but that's not in my personality. John T. Reed's book, Succeeding, made a very good point, which is that it's very difficult to change your personality, and trying to do so would make you very unhappy. However, it's possible and quite easy to change your context and your environment, and your life should be about finding a context and environment where your personality makes succeeding easy, rather than trying to fit your personality into an increasingly ill-fitting context.

In any case, I don't have any plans to jump right away into another work-place, startup or not. For one thing, I have at least one more book I want to write (it has nothing to do with computers or startups), and Lisa and I have booked a 5 week trip to Europe this summer, along with some Googler friends. I have hopes of doing a photography trip in the fall, and yes, I would like to do another sailing trip. Then there's friends to visit, and maybe for once I should try to attend WorldCon or GenCon, events that I have always thought of attending, but never did it because when you have a limited amount of vacation time, you would never waste it on indoor activities when there's so much left of the world you want to see.

For the immediate future, however, I plan to spend the next few weeks writing up some of my experiences at Google (no, I won't divulge any trade secrets), so if you enjoy that sort of thing, stay tuned.