I will confess that I never did follow the Theranos story very carefully. Unlike many others, I've never had a fear of blood tests, nor am I by nature an early adopter, so the prospect of only getting a "finger prick" rather than a venous blood draw never got me very excited.
Bad Blood covers the story, and in great detail. It reveals the tricks and techniques that Elizabeth Holmes and her executives used to brow-beat, intimidate, and trick employees, investors, and famous people into investing in the company, aiding it in its lies, and then intimidate those who would expose its illegal acts to the public.
There are many moments in the narrative where I think to myself, "My goodness, how did this story ever get made? The bad actors in the story are so powerful!" Then I realized that of course, the "technology" they were selling never worked, and they would have eventually been caught, though perhaps not before they hurt a ton of people with inaccurate or misleading blood tests.
The story is exciting, interesting, and of course, impeccably researched. It's interesting to me how easily most of the media was taken by personality, while nobody actually followed up and looked at the product by doing the kind of comparison study that John Carreyrou did (get an assay done by the Theranos product, and get one done by Labcorp).
In any case, the book comes highly recommended, and it's a good reminder that staying away from sociopaths is a good idea. Even if the good guys eventually win, the bad guys can still make your life very painful in the mean time. Buy or borrow your copy and read it!
Showing posts with label startups. Show all posts
Showing posts with label startups. Show all posts
Monday, July 16, 2018
Monday, October 06, 2014
Gaming the Coding Interview
Paul Graham's essay on how you can't really game startups had me thinking about the coding interview. Google had a lot of studies showing that the interview as practiced by Google wasn't very effective: in other words, interview scores don't really correlate with actual job performance. In part, this is because Google's not a startup any more --- political ability probably determines your promotions and effectiveness within Google than simply being good at engineering. But a major part is also that the coding interview is very susceptible to being gamed.
For instance, if you read Cracking the Coding Interview and were diligent about it (i.e., actually worked through the problems and practiced at them), you'd stand a good chance of doing really well during Google's interview process. Lest you think that this is a recent phenomena, even in 2003, Google's interview process was very similar. I remember being asked to reverse all the words in a sentence, and a few other puzzler type questions, and even during my interview, I remembered one interviewer telling the next one as the hand-off was happening, "this guy knows all the standard interview questions." Back then, Gayle's book didn't exist, but 10 years of interviewing for startups and interviewing at startups had hit me with every interview question that could be easily covered in a 45 minute session.
I will note that Facebook does have tougher interviews today than Google (they're hiring slower and therefore can be more picky), but from what I've seen their interviews are no less subject to being gamed.
When I look back at the interviewing process, there's really only one company that's stood out for having an interview process that couldn't be easily gamed, and that's Wealthfront in late 2012. I only include the date because in between, startups can change a lot and for all I know they could be interviewing like Google today.
The way Wealthfront conducted their interview was by pair programming. The candidate would come in, and pair program real problems with their "interviewer". The experience is intense, and in many ways eliminates the possibility of hiring someone who couldn't even write correct java syntax, or construct unit tests for code he'd just written. It's a good way to go and difficult to game, since you have to actually be able to design, structure, and turn ideas into code all the way to the testing and debugging steps.
Another good idea I've seen at certain startups is to put the culture fit interview first, before any technical interviews get done. The reason for this is if you get a candidate who's stellar on the technical side, it's actually very difficult to reject him for cultural reasons. I can attest to this, as one of my early hires at Google bombed out precisely for that reason, though without doing much damage. By putting the cultural fit interview first, you eliminate the bias to hire, even though you might waste a bit of time.
For instance, if you read Cracking the Coding Interview and were diligent about it (i.e., actually worked through the problems and practiced at them), you'd stand a good chance of doing really well during Google's interview process. Lest you think that this is a recent phenomena, even in 2003, Google's interview process was very similar. I remember being asked to reverse all the words in a sentence, and a few other puzzler type questions, and even during my interview, I remembered one interviewer telling the next one as the hand-off was happening, "this guy knows all the standard interview questions." Back then, Gayle's book didn't exist, but 10 years of interviewing for startups and interviewing at startups had hit me with every interview question that could be easily covered in a 45 minute session.
I will note that Facebook does have tougher interviews today than Google (they're hiring slower and therefore can be more picky), but from what I've seen their interviews are no less subject to being gamed.
When I look back at the interviewing process, there's really only one company that's stood out for having an interview process that couldn't be easily gamed, and that's Wealthfront in late 2012. I only include the date because in between, startups can change a lot and for all I know they could be interviewing like Google today.
The way Wealthfront conducted their interview was by pair programming. The candidate would come in, and pair program real problems with their "interviewer". The experience is intense, and in many ways eliminates the possibility of hiring someone who couldn't even write correct java syntax, or construct unit tests for code he'd just written. It's a good way to go and difficult to game, since you have to actually be able to design, structure, and turn ideas into code all the way to the testing and debugging steps.
Another good idea I've seen at certain startups is to put the culture fit interview first, before any technical interviews get done. The reason for this is if you get a candidate who's stellar on the technical side, it's actually very difficult to reject him for cultural reasons. I can attest to this, as one of my early hires at Google bombed out precisely for that reason, though without doing much damage. By putting the cultural fit interview first, you eliminate the bias to hire, even though you might waste a bit of time.
Monday, September 22, 2014
Review: Gobble Dinner Service
In the past, there's been plenty of food startups, from kitchit to gastronauts. None of them have addressed what I consider the best possible market: busy parents. We ran into Gobble and decided to give them a try, since they were promising fast meals that were done right.
The idea behind Gobble is this: you get pre-packaged, pre-prepared gourmet food delivered to your door in refrigerated packages. Each box comes with 3 meals, and you're in a subscription service, so you can cancel any time. Each meal comes with a preparation card, and it takes about 10 minutes to prepare each meal, and you'll be done. It's a nice concept, though as with all sorts of food, everything depends on the execution.
The central premise behind any kind of delivered food service like this is Sous Vide. Since the food has been already vacuum-packed while cooking, it's an easy step to simply go the next step to freeze it and then deliver it to your door. The biggest problem is that most people don't have a sous vide setup, so I was curious as to how they did the reheating.
It turned out that about only 2 out of 3 meals are done via sous vide. The fish and seafood dishes have ingredients that are so easily cooked that stir fry does it. The other sous vide meals are finished via either stir fry, or a searing step followed by an oven. This last method means that Gobble cheated on their marketing: it takes way longer than 10 minutes to pre-heat the oven and then for you to stir fry and present the meal.
The other problem I had with them was the delivery. The service uses On-Trac, which has a history of extremely late deliveries to my home. Indeed, the first delivery was so late that our Gobble meal turned into Pizza take out by the time the van driver showed up at my home. I called customer service and they apologized and gave me a $10 credit, but if I'd had hungry kids and a hungry wife, $10 wouldn't have come close to making up for it. There's also the problem of picking up the old container. I have no idea when they intend to collect them or if I'm supposed to throw them away.
As for value for money, the cost of the meal is about $12/person. This is approximately about the cost of eating out, except you don't have to tip. The variety of meals are decent, though the portion size ranges from barely adequate to substantial. It's very clear that each meal is sized not by calorie needs but by how much each ingredient costs: the chicken dishes are substantial, the beef dishes are usually supplemented by beans, and the seafood dishes would not keep a teenage boy well-fed.
The meals are decent, though everything is Americanized, so the curry tastes kinda bland and the chili is very mild. But it's all been very good, though not as good as if you went all modernist cuisine on it.
In any case, since we do have a sous vide machine, I'm not sure we'll continue after a month's trial, but I can recommend them to people without sous vide machine. It's also a nice way to get recipe ideas. In any case, if you do want a referral code for a trial e-mail me and I'll arrange for you to get one. Or you can just click through above if you're impatient and do without.
This is one of the few services that I think deserves success, and serves the South Bay quite well. Recommended.
The idea behind Gobble is this: you get pre-packaged, pre-prepared gourmet food delivered to your door in refrigerated packages. Each box comes with 3 meals, and you're in a subscription service, so you can cancel any time. Each meal comes with a preparation card, and it takes about 10 minutes to prepare each meal, and you'll be done. It's a nice concept, though as with all sorts of food, everything depends on the execution.
The central premise behind any kind of delivered food service like this is Sous Vide. Since the food has been already vacuum-packed while cooking, it's an easy step to simply go the next step to freeze it and then deliver it to your door. The biggest problem is that most people don't have a sous vide setup, so I was curious as to how they did the reheating.
It turned out that about only 2 out of 3 meals are done via sous vide. The fish and seafood dishes have ingredients that are so easily cooked that stir fry does it. The other sous vide meals are finished via either stir fry, or a searing step followed by an oven. This last method means that Gobble cheated on their marketing: it takes way longer than 10 minutes to pre-heat the oven and then for you to stir fry and present the meal.
The other problem I had with them was the delivery. The service uses On-Trac, which has a history of extremely late deliveries to my home. Indeed, the first delivery was so late that our Gobble meal turned into Pizza take out by the time the van driver showed up at my home. I called customer service and they apologized and gave me a $10 credit, but if I'd had hungry kids and a hungry wife, $10 wouldn't have come close to making up for it. There's also the problem of picking up the old container. I have no idea when they intend to collect them or if I'm supposed to throw them away.
As for value for money, the cost of the meal is about $12/person. This is approximately about the cost of eating out, except you don't have to tip. The variety of meals are decent, though the portion size ranges from barely adequate to substantial. It's very clear that each meal is sized not by calorie needs but by how much each ingredient costs: the chicken dishes are substantial, the beef dishes are usually supplemented by beans, and the seafood dishes would not keep a teenage boy well-fed.
The meals are decent, though everything is Americanized, so the curry tastes kinda bland and the chili is very mild. But it's all been very good, though not as good as if you went all modernist cuisine on it.
In any case, since we do have a sous vide machine, I'm not sure we'll continue after a month's trial, but I can recommend them to people without sous vide machine. It's also a nice way to get recipe ideas. In any case, if you do want a referral code for a trial e-mail me and I'll arrange for you to get one. Or you can just click through above if you're impatient and do without.
This is one of the few services that I think deserves success, and serves the South Bay quite well. Recommended.
Thursday, July 24, 2014
Startup Engineering Management Gets a 2nd Edition
Startup Engineering Management has been doing so well that I added what I learned over the last few years to it and gave it a 2nd Edition. It's a book that's attracted a surprising following, indicating that there's interest in the no-nonsense, non-political approach to management that I espouse for startups.
This new edition includes a whole new chapter on process analysis, sections on justifying hardware selection based on the great reception my Wharton talk got, and also a foreword by Harper Reed, who endorsed the book early in its life.
Along with the new edition, the price has gone up from $21.95 to $24.95 for the digital edition, and the paper version has also risen to match the price with An Engineer's Guide to Silicon Valley Startup. If you've bought Startup Engineering Management in digital edition since April 23rd, 2014, you've already received a free upgrade to the 2nd Edition in the mail.
If you bought a digital copy earlier, the upgrade price is $5, and what you need to do is e-mail me the original receipt from Paypal or Google checkout. Once I've verified the purchase, you'll get an invoice via paypal and an upgrade. Thank you all for your support!
This new edition includes a whole new chapter on process analysis, sections on justifying hardware selection based on the great reception my Wharton talk got, and also a foreword by Harper Reed, who endorsed the book early in its life.
Along with the new edition, the price has gone up from $21.95 to $24.95 for the digital edition, and the paper version has also risen to match the price with An Engineer's Guide to Silicon Valley Startup. If you've bought Startup Engineering Management in digital edition since April 23rd, 2014, you've already received a free upgrade to the 2nd Edition in the mail.
If you bought a digital copy earlier, the upgrade price is $5, and what you need to do is e-mail me the original receipt from Paypal or Google checkout. Once I've verified the purchase, you'll get an invoice via paypal and an upgrade. Thank you all for your support!
Friday, May 30, 2014
Introducing Jemstep
After I met with the founder of Jemstep, I realized that his product was not for me, but a large group (probably even the majority) of Americans would find it to be a great product fit. When William Bernstein visited Google, he asked for a show of hands as to how many people in the room had sizable 401(k)s or IRAs as a percentage of their total portfolio. Nobody raised their hands. He said, "This is completely atypical of American society, where most people's financial assets are largely tied up in their 401(k) or IRA."
The net result is that my approach, and the approach of many of the products I introduce such as Wealthfront, are completely targeted towards people like me. When most of your financial assets are in after-tax portfolio, tax-loss harvesting becomes important, as does qualifying for long term capital gains taxes. When much of your assets are in IRAs or 401(k)s, however, then asset location becomes important.
Jemstep does the hard work of figuring out what the correct asset location for every asset is, and managing multiple assets across all counts. Unfortunately, because everything is done through TrustE, Jemstep cannot place trades for you, or actually manage your disparate accounts. What this means is that you have to manually enter the trades, and deal with the tax consequences thereof. This is in contrast to Wealthfront, which does place trades for you, etc.
Jemstep is fairly cheap. It charges about $70 for an unlimited sized portfolio, and correspondingly less if you have less than $500,000 in assets. But it does a lot less for you than Wealthfront, and you're still stuck with whatever transaction costs are involved. It also doesn't do any of the sophisticated tax-loss harvesting that Wealthfront does. For folks with large taxable portfolios, Wealthfront's fees more than pay for themselves when you take into account tax loss harvesting.
This all sounds really negative, but it isn't. If you're the beneficiary of a tech IPO, or just won the lottery or sold your business, this service is not for you. If you have a huge legacy portfolio in taxable accounts that are unconsolidated, you might want to try Jemstep, but my suspicion is that you're best off slowly migrating your account to Wealthfront, and Jemstep is at best a stop-gap. However, if you're middle-aged, and have a large tax-sheltered portfolio, then asset location matters a lot to you and you're better off with Jemstep than with Wealthfront, especially since it's unlikely that you're able to move your 401(k) to Wealthfront while you're working at the same employer for the next few decades.
In short, I'd suggest checking out Jemstep, but only if your tax-sheltered portfolio is a significant percentage of your net-worth (more than about 20%).
The net result is that my approach, and the approach of many of the products I introduce such as Wealthfront, are completely targeted towards people like me. When most of your financial assets are in after-tax portfolio, tax-loss harvesting becomes important, as does qualifying for long term capital gains taxes. When much of your assets are in IRAs or 401(k)s, however, then asset location becomes important.
Jemstep does the hard work of figuring out what the correct asset location for every asset is, and managing multiple assets across all counts. Unfortunately, because everything is done through TrustE, Jemstep cannot place trades for you, or actually manage your disparate accounts. What this means is that you have to manually enter the trades, and deal with the tax consequences thereof. This is in contrast to Wealthfront, which does place trades for you, etc.
Jemstep is fairly cheap. It charges about $70 for an unlimited sized portfolio, and correspondingly less if you have less than $500,000 in assets. But it does a lot less for you than Wealthfront, and you're still stuck with whatever transaction costs are involved. It also doesn't do any of the sophisticated tax-loss harvesting that Wealthfront does. For folks with large taxable portfolios, Wealthfront's fees more than pay for themselves when you take into account tax loss harvesting.
This all sounds really negative, but it isn't. If you're the beneficiary of a tech IPO, or just won the lottery or sold your business, this service is not for you. If you have a huge legacy portfolio in taxable accounts that are unconsolidated, you might want to try Jemstep, but my suspicion is that you're best off slowly migrating your account to Wealthfront, and Jemstep is at best a stop-gap. However, if you're middle-aged, and have a large tax-sheltered portfolio, then asset location matters a lot to you and you're better off with Jemstep than with Wealthfront, especially since it's unlikely that you're able to move your 401(k) to Wealthfront while you're working at the same employer for the next few decades.
In short, I'd suggest checking out Jemstep, but only if your tax-sheltered portfolio is a significant percentage of your net-worth (more than about 20%).
Saturday, March 16, 2013
Wharton Business School Presentation
One question that came up was where can you learn to code? While I'm not a big fan of the various programming boot camps for professional programmers, for MBAs, it could very well be just the thing. The school itself had fantastic facilities, and the lecture hall was the best room I'd ever given a presentation in, with stadium-style seating and the lecturer standing in the pit subject to questions all around. Definitely an experience to be had.
As usual, the above slides are sanitized for public consumption. The presentation given to the business school students is a lot more peppered with interesting case studies from industry. Having said that, with my new full time job, my time for speaking engagements is a lot more limited and I will curtail them going forward.
Friday, February 08, 2013
Startup Engineering Management visits Wharton School of Business in San Francisco
Did you know that Wharton School of Business had a San Francisco Branch? I didn't, until James Kilpatrick, affiliated with their entrepreneurship program contacted me and asked if I was willing to give a talk to the students in the program. Given the prestige of Wharton, who was I to turn them down?
The talk will happen in Wharton's San Francisco campus on Saturday, March 16th around noon. (Yes, it's a weekend MBA program) The talk will be directed towards MBA students who are mostly not technical. Hence, it will be about "attracting, recruiting, retaining, and keeping engineers happy at a startup." The school of business has about 30 free spaces available for non-MBA students who are interested in Wharton's MBA program to attend, in addition to its current students who may attend the talk.
If you're interested in going to the talk, please send e-mail to James Kilpatrick telling him you want to attend the talk, and he'll send you the details if there's enough room. Those attending the talk from outside will need to stay fora short admissions information session with Director of admissions Katherine Lilygren.
The talk will happen in Wharton's San Francisco campus on Saturday, March 16th around noon. (Yes, it's a weekend MBA program) The talk will be directed towards MBA students who are mostly not technical. Hence, it will be about "attracting, recruiting, retaining, and keeping engineers happy at a startup." The school of business has about 30 free spaces available for non-MBA students who are interested in Wharton's MBA program to attend, in addition to its current students who may attend the talk.
If you're interested in going to the talk, please send e-mail to James Kilpatrick telling him you want to attend the talk, and he'll send you the details if there's enough room. Those attending the talk from outside will need to stay fora short admissions information session with Director of admissions Katherine Lilygren.
Sunday, January 06, 2013
My New Job
When I announced my retirement almost 3 years ago, I got three reactions:
- "I can't imagine staying home to feed the cats and watch TV." Those folks couldn't be more wrong. I don't have any cats, and I barely had time to finish a couple of video games, let alone watch TV. Instead, I wrote 3 books, traveled a lot, started a consulting business, got married, had a baby, and in general lived a pretty good life that never left me feeling bored or unfulfilled. My personal experience is that the kind of people who make those statements are people lacking in imagination: they can't imagine leading a self-directed life, so they imagine a life of boredom if they left work.
- "You're too young to retire, you'll be back at work." They were partly right. Writing books is significant work, and my consulting business was also work, as is getting married and coping with baby.
- "Would you ever consider leaving retirement?" My response was "Of course, for an appropriate role and an opportunity that gets me excited enough."
So when the team asked me to join as the VP of Engineering for the company I was delighted to say yes and prove that yes, there are opportunities that will get me to return to the workforce. (I did discuss positions at other Silicon Valley startups, and yes, I am happy to report that Silicon Valley is as full of opportunity as ever for those of us for whom big companies are undesirable work environments)
Yes, Quark Games does have job openings, including engineering positions. All engineers will report to me, so if you've enjoyed working with me in the past or you know my management philosophy and like it, consider applying for a job!
Yes, Quark Games does have job openings, including engineering positions. All engineers will report to me, so if you've enjoyed working with me in the past or you know my management philosophy and like it, consider applying for a job!
This has implications for my negotiation business. Effective immediately, I will take on no more new customers. If you're an existing customer, don't worry, I won't abandon you. I will keep on your case until you are satisfied or give you a full refund. Existing customers who basically treat me as a sounding board for financial advice don't have to worry either --- I will keep servicing your requests as there's no conflict of interest with my new job. My books will always be available on Amazon and on-line.
Tuesday, December 18, 2012
Bloomreach Talk
Tuesday, September 18, 2012
Wealthfront's Startup Equity Calculator
Wealthfront has a great startup equity calculator post on their blog that's worth reading for every engineer who intends to work at a startup.
It has a few interesting data points, one of which is that Hardware Engineers tend to get more equity on average than software engineers. This could be because on average, companies that require hardware engineers tend to go for experienced hardware engineers, while it's not unusual for startups to hire new grads for software engineering work.
Rachleff also makes the point that it makes much more sense to manage your career than to manage your wealth. That's true for new graduates. It's definitely not true for people at the mid-point of their careers, which in Silicon Valley is a much younger age than anyone outside the Valley would believe. I think Rachleff understates how important proper handling of your finances are. After I left Google, I had the privilege of speaking to many people about wealth management. There were a surprising number of people who had more stock than I did with worse financial outcomes almost a decade later. The difference between having a good financial plan and trying to time the markets or relying on a crooked financial advisor are enormous!
In any case, a number of my clients implicitly understand this --- some of them have explicitly turned down multi-million dollar retention packages at big companies (or in some cases refused to even start negotiating for those) in favor of unknown outcomes at startups. Even if those startups do not succeed, the skills they learn and exposure to an environment that requires all their talents, rather than a subset of them, will eventually lead to far more success.
Friday, September 14, 2012
Market Efficiency
Most companies understand not to cheap out on machines for engineers. The cost of a top end development machine is about $5,000, and when you're shelling out $100K a year, even a brand new machine every year just doesn't cost that much (in practice, nobody upgrades every year --- there's enough overhead when switching machines that doing it every other Moore's cycle --- 36 months makes much more sense). And every bit of increased productivity means you get that much more out of the $100K/year asset.
But once in a while, companies come across an under-priced engineer. Whether because they're inexperienced and didn't know how to negotiate, or whether they were initially offered less compensation because they were an unproven quantity, it's extremely tempting to keep under paying them for as long as you can get away with under paying them. What's happening in this case is that management thinks that they've stumbled across an engineer who's an idiot savant --- that somehow it's possible to be a great engineer who happens to be clueless as to his net-worth. There's no doubt that such people exist (I know some of them), and if you're at a big company that risk might pay off, but it's an insane risk to take at a startup where every engineer matters.
Here's what eventually happens. The engineer has friends, and eventually his curiosity will lead him to compare compensation with those friends. When he learns that he's significantly underpaid, he'll get pissed off enough to interview, and if you're lucky enough, to ask you for a raise. At this point, you'll have to give him a raise. If you're smart, you'll give him a raise, and compensate him for his lost wages as a result of you underpricing him in the first place. Most companies might do the first but rarely do they do the second. The consequences of not doing the second is that the engineer you've pissed off is out interviewing, and will end up with higher offers and you'll end up paying back those lost wages anyway, assuming you manage to keep him. If you don't, you'll spend that money recruiting and training a new engineer to replace him.
You might argue that a startup can't afford cash and raises. You might be right. But there's no excuse even then: if you can't afford cash, then you can provide additional equity. Equity is even better, since you can tie that to a vesting period, which would keep the employee loyal for years to come, and raise the bar for anyone else trying to poach your employees.
One of the things that impressed me about Google was its willingness to raise new graduates to market rapidly --- it was not unusual for a new graduate at Google to get a 30% raise on her first promotion, reflecting her increased value. However, Facebook was even better in that regard. I've had reports of Facebook granting retention packages even before the new employee's first year is up for review! This is a great approach, because the employee considers this unasked-for raise a gift. What happens to you when you receive a gift? You feel obliged to give back. So not only have you made a high performing employee happy, you've ensured that he's going to work even harder for you, at least in the near term! Contrast this with the employee mentioned above who had to ask for his raise (after realizing that he was underpaid): he didn't feel like he got a gift, he got screwed. Even raises that come from promotions don't feel like a gift, because the employee felt like he had to work for it.
Regardless of the performance of its stock, I've made the statement in recent years that Facebook probably has the best engineering management in Silicon Valley, and this is just one example of what they do better than anybody else. (Note: the author does not own any Facebook stock)
But once in a while, companies come across an under-priced engineer. Whether because they're inexperienced and didn't know how to negotiate, or whether they were initially offered less compensation because they were an unproven quantity, it's extremely tempting to keep under paying them for as long as you can get away with under paying them. What's happening in this case is that management thinks that they've stumbled across an engineer who's an idiot savant --- that somehow it's possible to be a great engineer who happens to be clueless as to his net-worth. There's no doubt that such people exist (I know some of them), and if you're at a big company that risk might pay off, but it's an insane risk to take at a startup where every engineer matters.
Here's what eventually happens. The engineer has friends, and eventually his curiosity will lead him to compare compensation with those friends. When he learns that he's significantly underpaid, he'll get pissed off enough to interview, and if you're lucky enough, to ask you for a raise. At this point, you'll have to give him a raise. If you're smart, you'll give him a raise, and compensate him for his lost wages as a result of you underpricing him in the first place. Most companies might do the first but rarely do they do the second. The consequences of not doing the second is that the engineer you've pissed off is out interviewing, and will end up with higher offers and you'll end up paying back those lost wages anyway, assuming you manage to keep him. If you don't, you'll spend that money recruiting and training a new engineer to replace him.
You might argue that a startup can't afford cash and raises. You might be right. But there's no excuse even then: if you can't afford cash, then you can provide additional equity. Equity is even better, since you can tie that to a vesting period, which would keep the employee loyal for years to come, and raise the bar for anyone else trying to poach your employees.
One of the things that impressed me about Google was its willingness to raise new graduates to market rapidly --- it was not unusual for a new graduate at Google to get a 30% raise on her first promotion, reflecting her increased value. However, Facebook was even better in that regard. I've had reports of Facebook granting retention packages even before the new employee's first year is up for review! This is a great approach, because the employee considers this unasked-for raise a gift. What happens to you when you receive a gift? You feel obliged to give back. So not only have you made a high performing employee happy, you've ensured that he's going to work even harder for you, at least in the near term! Contrast this with the employee mentioned above who had to ask for his raise (after realizing that he was underpaid): he didn't feel like he got a gift, he got screwed. Even raises that come from promotions don't feel like a gift, because the employee felt like he had to work for it.
Regardless of the performance of its stock, I've made the statement in recent years that Facebook probably has the best engineering management in Silicon Valley, and this is just one example of what they do better than anybody else. (Note: the author does not own any Facebook stock)
Wednesday, May 16, 2012
An Inflated Sense of Risk
I was at the Berkeley alumni panel organized by Dan Wallach a week or so back, and was very fortunate to be seated next to Jon Blow when someone asked if joining a startup, or doing an indie game was risky. Jon immediately went into this rant about how in this modern world, we have an inflated sense of risk: almost all risk has been taken out of society by civilization --- if you startup fails, you don't go to debtor's prison, and you are not at risk of getting eaten by tigers or wolves. As a result, we have people who inflate the little risks in life to a ridiculous amount, to the point of not letting kids walk or bike to school because they might get abducted by strangers!
This came to me again recently when someone asked me for advice. He was about to leave his current giant employer for another employer that was much smaller, but was well on its way to being public. From a business point of view, neither company had significant risk. Yet he wavered. "I have two pre-school kids, and I can't afford to take the risk of even joining a high quality startup such as OSMeta."
Here's the deal. If you join a smaller company from a giant firm or even a tiny startup, and then face new challenges, overcome them, and work with really smart people, there's zero chance that if everything goes belly up over the next few years the giant corporation will not hire you. Zero. By contrast, if you do spend your time at a large corporation with reduced risk and perhaps a fat package that comes to you every year no matter what happens, would you push yourself as hard? (Some people might: if you're one of them, this does not apply to you, and you're probably a super star at the large corporation anyway and are not reading this) And if you don't manage to push yourself as hard and giant corporation falls onto bad times, what's going to happen? You could be pushed out of your job (by better politicians, perhaps), and now you're actually less employable than if you'd taken the small company "higher risk" job.
So what I think is that people inflate the risk of going solo, doing the startup thing, or even joining a smaller firm, and forget the risk of staying stagnant at a large corporation where you might not get a chance to stretch, and the political game is far more vicious as it's practiced by many people who have nothing better to do all day precisely because the business faces effectively no risk. Startups eliminate such behavior because if you don't pull your weight, it's very obvious and the business is at risk, and good startups fire such people.
This came to me again recently when someone asked me for advice. He was about to leave his current giant employer for another employer that was much smaller, but was well on its way to being public. From a business point of view, neither company had significant risk. Yet he wavered. "I have two pre-school kids, and I can't afford to take the risk of even joining a high quality startup such as OSMeta."
Here's the deal. If you join a smaller company from a giant firm or even a tiny startup, and then face new challenges, overcome them, and work with really smart people, there's zero chance that if everything goes belly up over the next few years the giant corporation will not hire you. Zero. By contrast, if you do spend your time at a large corporation with reduced risk and perhaps a fat package that comes to you every year no matter what happens, would you push yourself as hard? (Some people might: if you're one of them, this does not apply to you, and you're probably a super star at the large corporation anyway and are not reading this) And if you don't manage to push yourself as hard and giant corporation falls onto bad times, what's going to happen? You could be pushed out of your job (by better politicians, perhaps), and now you're actually less employable than if you'd taken the small company "higher risk" job.
So what I think is that people inflate the risk of going solo, doing the startup thing, or even joining a smaller firm, and forget the risk of staying stagnant at a large corporation where you might not get a chance to stretch, and the political game is far more vicious as it's practiced by many people who have nothing better to do all day precisely because the business faces effectively no risk. Startups eliminate such behavior because if you don't pull your weight, it's very obvious and the business is at risk, and good startups fire such people.
Tuesday, May 15, 2012
Don't Blame The Engineer
I was talking to someone at a big company recently, and we talked about their retention problem. The person said something to the effect that "We have so many engineers that you just can't cover everyone. There's too many places to hide!" My reaction was: "Wait a minute, why is it a problem with the individual engineers! In an organization this size, your problem is not the 300 bad eggs hiding in your organization. Your problem is the 300 directors (and up) and VPs you have of whom 50% are corrupting your culture intentionally, and 90% wasn't even aware of what your original culture was because nearly every manager you have was hired from the outside!"
Here's the hard truth about organizations. It's almost never the problem of a few "bad egg" engineers that's the root of your culture problem. I've a number of very bright friends at Yahoo, and they keep telling me that Yahoo still has great people, but you couldn't tell that from the outside: the problem is that at the top, the organization is dysfunctional, and no amount of great engineering can save you when you can't get your act together: even if one engineer somewhere in the organization were to invent the next billion dollar idea, that person wouldn't get heard, and wouldn't get sufficient resources in order to execute and deliver it to the market in order to generate that value.
Yet engineers persist on blaming other engineers for the problem. I'll never forget sitting down with a very senior engineer at Google several years ago discussing G-Drive. "The tech lead wasn't any good. If he was more persistent or more forceful, G-Drive might have launched." My jaw dropped. Knowing what we now know about how G-Drive was originally killed (it's documented in great detail in Steve Levy's book, In The Plex), we now know there was no way any engineer, no matter how forceful or brilliant, could have kept the project from the chopping block no matter what. The kind of people who could have saved that project were the people who were politically savvy enough to have the ear of Larry Page. Most of them were not engineers, they were managers, directors or "executives." I have no idea why the engineers I talk to feel the need to blame the engineers: it could be that just like with family quarrels, it's easy to turn the anger on the people you know well rather than the strangers.
Yes, there are circumstances under which engineers can and should take the blame. If you chose to build an entire website on PHP, or tried to scale a web-site based on Ruby on Rails, you deserve all the derision you get from your engineering peers. But even such screw ups, by and large, do not tank the company. And as long as you don't screw up management big time, you'll get a chance to rectify those errors. And yes, if you're at a 3 person startup and the product sucks because of engineering decisions, then blame the engineer(s) involved. But seriously, at a large company (anything over 200 people), blaming the engineer simply means that the management sucks and won't take responsibility for its mistakes. If you're such an engineer in such a firm, go get yourself a new job. Waiting for management that sucks to admit its made a mistake could take a really long time.
In Startup Engineering Management, I note that peer based systems cannot scale past Dunbar's number. One of the unnoted pernicious side effects, however, is that peer based systems also make it easy for management to shirk the important management tasks: that of choosing new managers as well as promoting the right people.
Here's the hard truth about organizations. It's almost never the problem of a few "bad egg" engineers that's the root of your culture problem. I've a number of very bright friends at Yahoo, and they keep telling me that Yahoo still has great people, but you couldn't tell that from the outside: the problem is that at the top, the organization is dysfunctional, and no amount of great engineering can save you when you can't get your act together: even if one engineer somewhere in the organization were to invent the next billion dollar idea, that person wouldn't get heard, and wouldn't get sufficient resources in order to execute and deliver it to the market in order to generate that value.
Yet engineers persist on blaming other engineers for the problem. I'll never forget sitting down with a very senior engineer at Google several years ago discussing G-Drive. "The tech lead wasn't any good. If he was more persistent or more forceful, G-Drive might have launched." My jaw dropped. Knowing what we now know about how G-Drive was originally killed (it's documented in great detail in Steve Levy's book, In The Plex), we now know there was no way any engineer, no matter how forceful or brilliant, could have kept the project from the chopping block no matter what. The kind of people who could have saved that project were the people who were politically savvy enough to have the ear of Larry Page. Most of them were not engineers, they were managers, directors or "executives." I have no idea why the engineers I talk to feel the need to blame the engineers: it could be that just like with family quarrels, it's easy to turn the anger on the people you know well rather than the strangers.
Yes, there are circumstances under which engineers can and should take the blame. If you chose to build an entire website on PHP, or tried to scale a web-site based on Ruby on Rails, you deserve all the derision you get from your engineering peers. But even such screw ups, by and large, do not tank the company. And as long as you don't screw up management big time, you'll get a chance to rectify those errors. And yes, if you're at a 3 person startup and the product sucks because of engineering decisions, then blame the engineer(s) involved. But seriously, at a large company (anything over 200 people), blaming the engineer simply means that the management sucks and won't take responsibility for its mistakes. If you're such an engineer in such a firm, go get yourself a new job. Waiting for management that sucks to admit its made a mistake could take a really long time.
In Startup Engineering Management, I note that peer based systems cannot scale past Dunbar's number. One of the unnoted pernicious side effects, however, is that peer based systems also make it easy for management to shirk the important management tasks: that of choosing new managers as well as promoting the right people.
Monday, May 14, 2012
Choosing between jobs
One question that occasionally comes up at the end of a Negotiation cycle is, "Ok, now I have all these offers, how do I choose?" Typically, if you end your negotiation cycles in this state, it means that you've done an amazing job in your negotiations, since frequently, there's one stand out company and it's clear which one you should go. (When I had a choice between Yahoo, Google, and Versign in 2003, the answer was pretty obvious. Similarly, in 2010, when one of my friends had a choice between Google and Facebook, the answer was also obvious, even when the compensation numbers were ostensibly close)
In An Engineer's Guide to Silicon Valley Startups I mention creating a spreadsheet so you can compare the companies involved. Well, one of the contributors to the third edition, Santhosh Srinivasan, has actually gone ahead and created it and shared it on Google Docs for all to use.
Santhosh writes that the inspiration behind the spreadsheet was LAAAM, the Lightweight Architecture Alternative Assessment Method. The idea is that you create a weighting that's important to you, and then rank each job offer independent of the weightings and then the highest scoring total would be your preferred offer.
In reality, I've never actually had to use such a spreadsheet, and neither do most of my clients. The intuitive approach works for most of us because ultimately, if you do a good job with negotiating compensation, the money difference should be so minor that who you want to work with should determine where you land. Since people are the least fungible of all, that approach works well unless you end up at a company so unstable that people come and go without your having an opportunity to work with the people you joined in order to work with. (That can happen, but your interview process should weed out such companies)
I do have friends who've built compensation models using spreadsheets, and then used that to get corporations to bid up their offers by showing that spreadsheet around to various companies. That's a viable approach.
In An Engineer's Guide to Silicon Valley Startups I mention creating a spreadsheet so you can compare the companies involved. Well, one of the contributors to the third edition, Santhosh Srinivasan, has actually gone ahead and created it and shared it on Google Docs for all to use.
Santhosh writes that the inspiration behind the spreadsheet was LAAAM, the Lightweight Architecture Alternative Assessment Method. The idea is that you create a weighting that's important to you, and then rank each job offer independent of the weightings and then the highest scoring total would be your preferred offer.
In reality, I've never actually had to use such a spreadsheet, and neither do most of my clients. The intuitive approach works for most of us because ultimately, if you do a good job with negotiating compensation, the money difference should be so minor that who you want to work with should determine where you land. Since people are the least fungible of all, that approach works well unless you end up at a company so unstable that people come and go without your having an opportunity to work with the people you joined in order to work with. (That can happen, but your interview process should weed out such companies)
I do have friends who've built compensation models using spreadsheets, and then used that to get corporations to bid up their offers by showing that spreadsheet around to various companies. That's a viable approach.
Wednesday, April 11, 2012
AirBnB's delightful offices
I'd been a big fan of AirBnB ever since I met one of their founders at the Sunfire offices, but I had never visited the office, so when XiaoQin and I were in San Francisco today for an unrelated matter, I asked if we could pay them a visit and was pleased that it was OK.
The exterior of the building is non-descript, but the interior is gorgeous. The lobby is the giant cereal box of Obama Os from way back during the startup phase when they were timing the launch with political conventions.
The meeting rooms inside are styled after some of their apartment BnB listings.
It is standard now for Silicon Valley startups to have their own kitchen and chef, and AirBnB's kitchen was open, in full view of everyone, and most importantly, the food was very tasty:
In a nice touch, even the bathrooms are deliberately non-corporate, styled after a comfortable home's bathroom, albeit a little bigger. In a tip towards environmental consciousness, they don't offer paper towels, but instead reusable cloth towels that are laundered.
There were large numbers of bicycles in the hallways, as 80% of the employees bike to work. It's very popular in Silicon Valley to say that all Silicon Valley companies are similar. I think people who say that are full of it, as every Silicon Valley startup I've worked at has been very different with its own unique culture. The ones with the strongest cultures tend to be very successful, and if what we saw today is characteristic of AirBnB, I'll be surprised if they do not have a bright future ahead of them.
The exterior of the building is non-descript, but the interior is gorgeous. The lobby is the giant cereal box of Obama Os from way back during the startup phase when they were timing the launch with political conventions.
 |
| From BayArea |
The meeting rooms inside are styled after some of their apartment BnB listings.
 |
| From BayArea |
It is standard now for Silicon Valley startups to have their own kitchen and chef, and AirBnB's kitchen was open, in full view of everyone, and most importantly, the food was very tasty:
 |
| From BayArea |
In a nice touch, even the bathrooms are deliberately non-corporate, styled after a comfortable home's bathroom, albeit a little bigger. In a tip towards environmental consciousness, they don't offer paper towels, but instead reusable cloth towels that are laundered.
 |
| From BayArea |
There were large numbers of bicycles in the hallways, as 80% of the employees bike to work. It's very popular in Silicon Valley to say that all Silicon Valley companies are similar. I think people who say that are full of it, as every Silicon Valley startup I've worked at has been very different with its own unique culture. The ones with the strongest cultures tend to be very successful, and if what we saw today is characteristic of AirBnB, I'll be surprised if they do not have a bright future ahead of them.
Friday, December 09, 2011
Starting Up While Employed
Someone recently asked me for advice on working on a startup while he was still employed at a large tech company. My first response was to point him at my response on Quora. Even though I provide details on pretty much how to do this, I still don't think it's a great idea.
- Working at a startup is tough. Doing so while trying to hold down a full time job at a large tech firm is going to slow you down a lot. Unless your idea has such a high technical barrier that you're the only person who could conceivably execute on it, someone else is very likely to get to market before you do.
- It sends the wrong message to your potential investors and co-founders. As a co-founder, I want to see someone else dedicated enough to cut the safety net and go full time. As an investor, I'd be wondering about your commitment if you were only willing to work on it part time. Yes, nearly everyone knows that Reed Hastings did his startup while contracting for other Silicon Valley firms. But note: he was not an employee, and no one doubted his commitment to the startup because he took out a second mortgage on his home!
- If you're working everywhere but California, which has extremely employee friendly laws and courts, then even my reply to the above question might not save you. In particular, are you so good at keeping a secret that you can avoid talking about your idea at lunch? And of course, you'd better be scrupulous about never doing any work on that company laptop or on a company paid-for internet connection. (A surprising number of people at large tech companies don't even own their own computer, and trust me, if you end up going to court and this comes up, you are losing all your intellectual property)
- Finally, if you really are startup material, then quitting your job to do a startup and coming back to a big company if it fails will always be an option. If you're not, then what makes you think your startup will be worth all that extra time you're putting in?
Monday, November 07, 2011
Product Endorsement: Wealthfront
Andy Rachleff gave XiaoQin and I a presentation four weeks ago on their latest wealth management product, and just lifted the embargo on me writing about it on my blog. I walked through the demo and worked through various scenarios, and I'm impressed.
Those of you who've been to my talks or read my financial posts know that I dislike wealth managers, and traditional banks that take your money and charge you huge amounts of money (usually 1% of total wealth) for what could easily be done with a spreadsheet or computer program. But there are people who can't manage their wealth themselves, either through lack of interest, inability to deal with the numbers involved, or (most common among engineers) lack of emotional control, which has very little correlation with intelligence or success in other areas of life.
What Wealthfront has done is to write that computer program. The result is one that interviews you to figure out your risk tolerance, then does Mean-Variance Optimization (MVO) and provide recommendations for performing asset allocation. There are many MVO programs that exist, but the biggest problem is actually getting the data: Wealthfront pays about $300k/year to get access to historical data so the MVO actually isn't garbage-in/garbage-out. This is the kind of stuff you get if you got William Bernstein, for instance, to manage your money.
What's more, the output isn't just provided to you with no context. You get an explanation of why the program picked certain ETFs, and you get to over-ride the results of the risk-tolerance analysis if you believe that the interview did not provide correct/optimum results. The service does automatic rebalancing, automatic accumulation of dividends to as part of the rebalancing scheme, and might in the future also offer automatic divestment of company stock. This is excellent stuff, and stuff I tried to get the Google Finance team interested in doing instead of creating yet another day-trader web-site.
The fee for all this? They've decided to go with a Freemium model: free management up to the first $25,000, and 0.25% for all money above that. (Yes, you can run A-B comparisons between your wealth manager/spreadsheet against Wealthfront to see who does better --- note that if you do that, you should take risk into account! A riskier portfolio could out perform Wealthfront, and if you have a good few years you could be fooled into taking more risk than you desire) More interestingly, past a certain level, they'll offer access to alternative assets such as hedge-funds and the absolute-return funds that university endowments get access to, which is responsible for much of their outsized returns in recent years.
Even better, if you choose to do it yourself, you can use their website to gauge what your efficient frontier is and then execute those trades yourself manually without paying wealthfront. The risk in doing that is that you won't have the emotional control and financial discipline to rebalance your portfolio, which is critical for high performance and wealth preservation. In that area, Wealthfront actually does something real nice, which is to accumulate dividends so that transaction costs are minimized and do the rebalancing all at once. Yes, they also take into account taxes, and so won't do small transactions that cause you to have a tax reporting headache. If you don't already have a legacy portfolio, these are the first wealth management folks that I am willing to endorse. I am not an employee, shareholder, investor, or otherwise compensated by wealthfront for making the above statement. In fact, I do occasionally get paid as a financial adviser for high and medium networth individuals to manage their portfolio, so in some sense I'm working against my self-interest --- people paying them certain won't be paying me!
Visit Wealthfront's Beta site so you can get an access code.
Those of you who've been to my talks or read my financial posts know that I dislike wealth managers, and traditional banks that take your money and charge you huge amounts of money (usually 1% of total wealth) for what could easily be done with a spreadsheet or computer program. But there are people who can't manage their wealth themselves, either through lack of interest, inability to deal with the numbers involved, or (most common among engineers) lack of emotional control, which has very little correlation with intelligence or success in other areas of life.
What Wealthfront has done is to write that computer program. The result is one that interviews you to figure out your risk tolerance, then does Mean-Variance Optimization (MVO) and provide recommendations for performing asset allocation. There are many MVO programs that exist, but the biggest problem is actually getting the data: Wealthfront pays about $300k/year to get access to historical data so the MVO actually isn't garbage-in/garbage-out. This is the kind of stuff you get if you got William Bernstein, for instance, to manage your money.
What's more, the output isn't just provided to you with no context. You get an explanation of why the program picked certain ETFs, and you get to over-ride the results of the risk-tolerance analysis if you believe that the interview did not provide correct/optimum results. The service does automatic rebalancing, automatic accumulation of dividends to as part of the rebalancing scheme, and might in the future also offer automatic divestment of company stock. This is excellent stuff, and stuff I tried to get the Google Finance team interested in doing instead of creating yet another day-trader web-site.
The fee for all this? They've decided to go with a Freemium model: free management up to the first $25,000, and 0.25% for all money above that. (Yes, you can run A-B comparisons between your wealth manager/spreadsheet against Wealthfront to see who does better --- note that if you do that, you should take risk into account! A riskier portfolio could out perform Wealthfront, and if you have a good few years you could be fooled into taking more risk than you desire) More interestingly, past a certain level, they'll offer access to alternative assets such as hedge-funds and the absolute-return funds that university endowments get access to, which is responsible for much of their outsized returns in recent years.
Even better, if you choose to do it yourself, you can use their website to gauge what your efficient frontier is and then execute those trades yourself manually without paying wealthfront. The risk in doing that is that you won't have the emotional control and financial discipline to rebalance your portfolio, which is critical for high performance and wealth preservation. In that area, Wealthfront actually does something real nice, which is to accumulate dividends so that transaction costs are minimized and do the rebalancing all at once. Yes, they also take into account taxes, and so won't do small transactions that cause you to have a tax reporting headache. If you don't already have a legacy portfolio, these are the first wealth management folks that I am willing to endorse. I am not an employee, shareholder, investor, or otherwise compensated by wealthfront for making the above statement. In fact, I do occasionally get paid as a financial adviser for high and medium networth individuals to manage their portfolio, so in some sense I'm working against my self-interest --- people paying them certain won't be paying me!
Visit Wealthfront's Beta site so you can get an access code.
Monday, October 03, 2011
Scalability
I've been helping various startups, and one of the things that's impressing me is how frequently these startups are launched on new infrastructure pieces like Ruby on Rails, written mostly by people who're not really trained formally in computer science or software engineering. This is a triumph of modern programming tools: I certainly didn't think that we'd get to a point where essentially product managers can actually write code, and asking these teams to execute in C++ or even Java would have set them back months. Even better, these startups don't even run or own their machines, choosing instead to use Amazon Web Services to launch and scale.
That is, until they suddenly hit a scaling inflection point and then someone like me gets called in to help out with the architecture and scalability problem. This is a good thing, by the way. Too many software engineers fresh out of school get hung up on the latest performance or scalability techniques and use them too aggressively when there's no need. It's far more important to launch a product quickly and get it to the point where you have product acceptance before you worry about performance. Even then, some of the latest techniques get you nowhere. One of my favorite examples came when I was at Mirapoint. A team of folks were working on the mail transfer agent. This is a relatively straight forward piece of code but when designing for high performance, they had decided on an architecture that made use of multiple threads and that made debugging hell. Brad Taylor came in, took a look at it, and rewrote the entire thing using a single-threaded select loop. Not only was that single-threaded select loop easier to debug, it actually ran faster, because the processor wasn't doing all those crazy context-switches just to get simple things done. In general, when scaling a product, your best bet is to first design for easy debugging and easy replication (so you can get horizontal scaling) by spinning up new processes rather than throwing the universe into an address space and launching multiple threads. The first person to articulate that philosophy to me was Michael Wolf at a dinner conversation with me and Steve Grimm.
Back to the startup. The problem with launching a startup on AWS and on Ruby on Rails is that when you hit the scalability inflection point, the easy solutions are not available. For instance, one company I helped were hammering their MySQL database with too many requests. While the number of queries were potentially large (and there was a fair bit of writing, so read-caching wasn't helping), their actual database was small. If they were hosting their own infrastructure instead of running on AWS, a simple straightforward solution that would have required no programming, would have been to just install a SSD (or if you have money burning a hole in your pocket, one of those crazy Fusion IO PIC boards, which bypass the SATA limitations and use the full bandwidth of the PCI bus for IO). SSDs are expensive, but even $400 or $8000 a pop is cheaper than the time it would take to rework the database to a NoSQL solution. Unfortunately, none of the existing cloud solutions will let you specify SSD-type performance for the machines you request.
Since they were essentially using MySQL as a blob store, they thought about exploring one of the NoSQL solutions. But the amount of data they had was so small it would have fit into main memory (of a relatively large server), so they could have potentially could use a simple shadow hashtable approach with a background thread to write the shadowed hashtable to disk for persistence. Unfortunately, that requires real threads and some simple locking, and the default implementation of Ruby doesn't support kernel threads (though JRuby does). Furthermore, by using Ruby On Rails, they'd written themselves into a corner where it would be difficult to extract the data layer out of their code so they could write to a NoSQL database anyway. What's interesting to me is that these were the people smart enough to know that they'd run into scalability limits with their software and infrastructure. There were probably many others who did not and chose to muddle along.
When I finally read about how Twitter was a big user of Ruby, all their scalability problems finally made sense. At some point, you do have to throw away your prototype and rewrite everything if you want to scale.
That is, until they suddenly hit a scaling inflection point and then someone like me gets called in to help out with the architecture and scalability problem. This is a good thing, by the way. Too many software engineers fresh out of school get hung up on the latest performance or scalability techniques and use them too aggressively when there's no need. It's far more important to launch a product quickly and get it to the point where you have product acceptance before you worry about performance. Even then, some of the latest techniques get you nowhere. One of my favorite examples came when I was at Mirapoint. A team of folks were working on the mail transfer agent. This is a relatively straight forward piece of code but when designing for high performance, they had decided on an architecture that made use of multiple threads and that made debugging hell. Brad Taylor came in, took a look at it, and rewrote the entire thing using a single-threaded select loop. Not only was that single-threaded select loop easier to debug, it actually ran faster, because the processor wasn't doing all those crazy context-switches just to get simple things done. In general, when scaling a product, your best bet is to first design for easy debugging and easy replication (so you can get horizontal scaling) by spinning up new processes rather than throwing the universe into an address space and launching multiple threads. The first person to articulate that philosophy to me was Michael Wolf at a dinner conversation with me and Steve Grimm.
Back to the startup. The problem with launching a startup on AWS and on Ruby on Rails is that when you hit the scalability inflection point, the easy solutions are not available. For instance, one company I helped were hammering their MySQL database with too many requests. While the number of queries were potentially large (and there was a fair bit of writing, so read-caching wasn't helping), their actual database was small. If they were hosting their own infrastructure instead of running on AWS, a simple straightforward solution that would have required no programming, would have been to just install a SSD (or if you have money burning a hole in your pocket, one of those crazy Fusion IO PIC boards, which bypass the SATA limitations and use the full bandwidth of the PCI bus for IO). SSDs are expensive, but even $400 or $8000 a pop is cheaper than the time it would take to rework the database to a NoSQL solution. Unfortunately, none of the existing cloud solutions will let you specify SSD-type performance for the machines you request.
Since they were essentially using MySQL as a blob store, they thought about exploring one of the NoSQL solutions. But the amount of data they had was so small it would have fit into main memory (of a relatively large server), so they could have potentially could use a simple shadow hashtable approach with a background thread to write the shadowed hashtable to disk for persistence. Unfortunately, that requires real threads and some simple locking, and the default implementation of Ruby doesn't support kernel threads (though JRuby does). Furthermore, by using Ruby On Rails, they'd written themselves into a corner where it would be difficult to extract the data layer out of their code so they could write to a NoSQL database anyway. What's interesting to me is that these were the people smart enough to know that they'd run into scalability limits with their software and infrastructure. There were probably many others who did not and chose to muddle along.
When I finally read about how Twitter was a big user of Ruby, all their scalability problems finally made sense. At some point, you do have to throw away your prototype and rewrite everything if you want to scale.
Monday, September 26, 2011
Execution Strategy and Business Strategy
Rebecca Frankel shared on her Google Reader Feed this image:
I think it's a great example of execution strategy. A frequently used execution strategy is to standardize on one platform and optimize on delivery that way. The nice thing about this strategy is that if you're the startup and the incumbent has built in legacy systems and costs that keep them from executing on the same strategy, you'll have a built-in advantage that will continue for a good long time. For instance, it's hard for an airline to switch the hubs it's using after it's been locked into a long term contract. And obviously, it's hard for incumbent airlines to sell all their fleets and standardize on the 737. The strategy that RyanAir uses, by the way, was first pioneered by SouthWest Airlines. A good execution strategy like this can be easily explained to all your employees to the point where everyone knows what the strategy is, leading to broader alignment.
At Google circa 2003, the execution strategy was straight-forward: build clusters of commodity machines driven by a common software infrastructure so scaling was straightforward. Once MapReduce was adopted, for instance, your mapreduce jobs were datacenter independent and could be run on any number of datacenters. Similarly, all projects could share a single set of site reliability engineers, release engineers, etc., and your resource constraints could be relieved by effectively hiring in all these areas.
The problem comes when you have a new product that doesn't fit your existing execution strategy. Orkut, for instance, was initially written using Microsoft's .NET framework. That didn't fit in with what the rest of Google was doing. It rapidly became a popular product, and the system started falling over from the huge number of requests. The team was staff-constrained, and few engineers inside Google wanted to work on a product that was clearly way out on the left field with respect to Google's execution strategy. To be honest, nobody knew how big social networks were going to be. The result was that it took a while to rewrite Orkut to conform with Google's execution strategy, and by the time it was done enough time had passed and enough users had migrated to another social network that the rest was history. In retrospect, the right thing to do should have been to spin Orkut off, have it raise its own funding round (with Google kicking money), and race quickly to deal with its scaling problems independent of the rest of Google's infrastructure. Facebook might still have won, but at least Orkut wouldn't have been paying Google's strategy tax. Microsoft has its own strategy tax here in that everything has to be tied to windows, but I think even Microsoft's starting to move away from that with its phone and tablet entries.
Business strategy is a whole different animal. In some ways, it's like playing a strategic board game with your business. I'll give you an example. Microsoft invested $240M in Facebook in 2007 at the then stunning valuation of $15B. At that time, Facebook was not profitable (and would become profitable based on Microsoft's guaranteed revenue), and it looked like Microsoft was desperate, throwing money at Facebook. Well, 4 years later, it looks like a brilliant move. Not only does it look like Microsoft's investment will pay off (at least 4X, maybe more), Facebook's been a tremendous thorn on Google's side, probably accounting for no small amount of management distraction, time spent launching (and re-launching) competitive products, and I'm sure no small drain on Google's engineering team. Time will tell as to whether Microsoft's acquisition of Nortel's patent portfolio, essentially forcing Google to buy Motorola will be similarly smart, but spending $4.5B (and that's split between Apple/RIM/Sony/EMC, etc) so that your competition has to spend $12.5B (and ends up having to run a hardware business that's not particularly profitable --- Motorola's the weakest of the Android manufacturers) looks pretty smart right now.
Another company which is good at this is Amazon. The Kindle, for instance, unusually attacked the market from a completely different angle. It's early adopters were not the usual hip 20-somethings, but were the older generation: people who still read and whose deteriorating eye-sight and arthritis made the Kindle an almost must-have. This was so much ignored by other vendors (Apple included) that by the time other ebook stores launched, nobody else has made a dent in electronic book distribution. By ensuring that the Kindle App is available for nearly every platform, Amazon has gotten a choke-hold on electronic book distribution that's only starting to be realized at this point.
Someone told me a little bit back that Amazon's S3 services were priced at below cost at launch. Basically, they bet that they could drive costs down a bit, and that customers would see the move to their cloud services as a no-brainer at those prices and thereby gain them further economies of scale. At this point, I've run into lots of companies that have based their businesses on S3, but comparatively few who've done so on Google's AppEngine infrastructure AppEngine is a bit of a red-headed stepchild at Google because it can't be priced to produce high margins, while Amazon's very much used to low margins.
Obviously, business strategy is of no use if you screw up your execution strategy (or if your product sucks --- nothing ever saves you then), but ideally you want everything in place. Amazon's been the stealth surprise in the past few years in places that looked completely unrelated to e-commerce because of this. At the same time, Microsoft's not being doing so well at execution but its business strategy successes are being ignored by the press, and I think that counting them out would be a mistake.
I think it's a great example of execution strategy. A frequently used execution strategy is to standardize on one platform and optimize on delivery that way. The nice thing about this strategy is that if you're the startup and the incumbent has built in legacy systems and costs that keep them from executing on the same strategy, you'll have a built-in advantage that will continue for a good long time. For instance, it's hard for an airline to switch the hubs it's using after it's been locked into a long term contract. And obviously, it's hard for incumbent airlines to sell all their fleets and standardize on the 737. The strategy that RyanAir uses, by the way, was first pioneered by SouthWest Airlines. A good execution strategy like this can be easily explained to all your employees to the point where everyone knows what the strategy is, leading to broader alignment.
At Google circa 2003, the execution strategy was straight-forward: build clusters of commodity machines driven by a common software infrastructure so scaling was straightforward. Once MapReduce was adopted, for instance, your mapreduce jobs were datacenter independent and could be run on any number of datacenters. Similarly, all projects could share a single set of site reliability engineers, release engineers, etc., and your resource constraints could be relieved by effectively hiring in all these areas.
The problem comes when you have a new product that doesn't fit your existing execution strategy. Orkut, for instance, was initially written using Microsoft's .NET framework. That didn't fit in with what the rest of Google was doing. It rapidly became a popular product, and the system started falling over from the huge number of requests. The team was staff-constrained, and few engineers inside Google wanted to work on a product that was clearly way out on the left field with respect to Google's execution strategy. To be honest, nobody knew how big social networks were going to be. The result was that it took a while to rewrite Orkut to conform with Google's execution strategy, and by the time it was done enough time had passed and enough users had migrated to another social network that the rest was history. In retrospect, the right thing to do should have been to spin Orkut off, have it raise its own funding round (with Google kicking money), and race quickly to deal with its scaling problems independent of the rest of Google's infrastructure. Facebook might still have won, but at least Orkut wouldn't have been paying Google's strategy tax. Microsoft has its own strategy tax here in that everything has to be tied to windows, but I think even Microsoft's starting to move away from that with its phone and tablet entries.
Business strategy is a whole different animal. In some ways, it's like playing a strategic board game with your business. I'll give you an example. Microsoft invested $240M in Facebook in 2007 at the then stunning valuation of $15B. At that time, Facebook was not profitable (and would become profitable based on Microsoft's guaranteed revenue), and it looked like Microsoft was desperate, throwing money at Facebook. Well, 4 years later, it looks like a brilliant move. Not only does it look like Microsoft's investment will pay off (at least 4X, maybe more), Facebook's been a tremendous thorn on Google's side, probably accounting for no small amount of management distraction, time spent launching (and re-launching) competitive products, and I'm sure no small drain on Google's engineering team. Time will tell as to whether Microsoft's acquisition of Nortel's patent portfolio, essentially forcing Google to buy Motorola will be similarly smart, but spending $4.5B (and that's split between Apple/RIM/Sony/EMC, etc) so that your competition has to spend $12.5B (and ends up having to run a hardware business that's not particularly profitable --- Motorola's the weakest of the Android manufacturers) looks pretty smart right now.
Another company which is good at this is Amazon. The Kindle, for instance, unusually attacked the market from a completely different angle. It's early adopters were not the usual hip 20-somethings, but were the older generation: people who still read and whose deteriorating eye-sight and arthritis made the Kindle an almost must-have. This was so much ignored by other vendors (Apple included) that by the time other ebook stores launched, nobody else has made a dent in electronic book distribution. By ensuring that the Kindle App is available for nearly every platform, Amazon has gotten a choke-hold on electronic book distribution that's only starting to be realized at this point.
Someone told me a little bit back that Amazon's S3 services were priced at below cost at launch. Basically, they bet that they could drive costs down a bit, and that customers would see the move to their cloud services as a no-brainer at those prices and thereby gain them further economies of scale. At this point, I've run into lots of companies that have based their businesses on S3, but comparatively few who've done so on Google's AppEngine infrastructure AppEngine is a bit of a red-headed stepchild at Google because it can't be priced to produce high margins, while Amazon's very much used to low margins.
Obviously, business strategy is of no use if you screw up your execution strategy (or if your product sucks --- nothing ever saves you then), but ideally you want everything in place. Amazon's been the stealth surprise in the past few years in places that looked completely unrelated to e-commerce because of this. At the same time, Microsoft's not being doing so well at execution but its business strategy successes are being ignored by the press, and I think that counting them out would be a mistake.
Friday, September 23, 2011
Location Still Matters
This year's been interesting as I got invited to several startups either to talk or to give advice. Usually, I try to mix it with another visit if it's in the city, so I don't make the trip to San Francisco just for one thing.
One of the fascinating things is that San Francisco has a pretty active startup scene, but many of them are hurting for engineers. After talking to several startups with ambitions of growth but who can't seem to hire decent engineers no matter what, I'm coming to the conclusion that the more technically challenging your startup, the more important it is that you be in Silicon Valley, rather than being able to locate elsewhere.
Why is that? For technically challenging problems, you want people with a decent amount of experience doing the engineering. I'll take an example: Facebook managed to get Jeff Rothschild to lead its engineering team fairly early on. Jeff, by the way, doesn't get nearly enough credit for making Facebook as successful as it has been. It is doubtful that Facebook would have been able to recruit and retain Jeff if it was in San Francisco rather than Palo Alto. The same probably would have gone for Google's Jeff Dean and Sanjay Ghemewat. For whatever reason (people tell me it has a lot to do with schools), parents prefer living in the South Bay. I've lost count of the number of people I know who moved to the city when they were single and childless, and then moved back down south once they had a kid.
The result: if you want to grow past your first 50 engineers or so, you'll either have to settle for a less technically competent population, or you'd have to move south. What's surprising to me is how things I wouldn't have expected to be technically challenging turn out to be such. For instance, I would have guessed that Twitter wouldn't need Google-quality engineers, but that turns out not to be true.
This doesn't mean that San Francisco startups can't be successful and make lots of money. For instance, AirBnB and Zynga will be incredibly successful. Zynga has a famously low technical bar, and one of my friends came back from an interview saying that being the smartest person there wasn't enough even if it did make her rich. Obviously, being in Silicon Valley is also no guarantee that you'd be able to attract technically competent employees (Friendster was in Mountain View, for instance). But by and large, I've been amused to watch Cloudera move steadily south (from Burlingame to San Mateo and now Palo Alto). By the way, I don't think Zynga's wrong to have a low technical bar: there's no need to pay for top-end talent if your problems don't need top-end talent to solve it.
Many designers have argued to me that design talent is easier to get in San Francisco. I'm not a designer, so I don't really know, but let's say I give you that point. The problem is, to realize your design, you probably need only 1 designer for every 10-20 engineers. And of course, Apple is right in Silicon Valley, and whatever you might say about Apple, you can't argue that their design is inferior.
Ultimately, if you're a startup, think carefully about what your business is. If you never need more than about 50 engineers, I think San Francisco is fine. If you believe you're really in the media business, San Francisco's probably better (be very careful, though --- Yahoo! thought it was in the media business --- that turned out to be false!). But if your startup idea needs a sizable number of Google-quality engineers to succeed in the long term, you really should be in the valley.
Now the real puzzle to me is that there should be far more startups in the Berkeley area than they are. They've got Cal, which has a strong computer science department, so recruiting for engineers should be no problem. My guess is that the city of Berkeley does not view startups in a friendly fashion, and it would be very difficult to find cheap space in the area. Inktomi (founded by Cal professor Eric Brewer), for instance, famously moved out of Berkeley (to Foster City) as soon as it had to scale.
One of the fascinating things is that San Francisco has a pretty active startup scene, but many of them are hurting for engineers. After talking to several startups with ambitions of growth but who can't seem to hire decent engineers no matter what, I'm coming to the conclusion that the more technically challenging your startup, the more important it is that you be in Silicon Valley, rather than being able to locate elsewhere.
Why is that? For technically challenging problems, you want people with a decent amount of experience doing the engineering. I'll take an example: Facebook managed to get Jeff Rothschild to lead its engineering team fairly early on. Jeff, by the way, doesn't get nearly enough credit for making Facebook as successful as it has been. It is doubtful that Facebook would have been able to recruit and retain Jeff if it was in San Francisco rather than Palo Alto. The same probably would have gone for Google's Jeff Dean and Sanjay Ghemewat. For whatever reason (people tell me it has a lot to do with schools), parents prefer living in the South Bay. I've lost count of the number of people I know who moved to the city when they were single and childless, and then moved back down south once they had a kid.
The result: if you want to grow past your first 50 engineers or so, you'll either have to settle for a less technically competent population, or you'd have to move south. What's surprising to me is how things I wouldn't have expected to be technically challenging turn out to be such. For instance, I would have guessed that Twitter wouldn't need Google-quality engineers, but that turns out not to be true.
This doesn't mean that San Francisco startups can't be successful and make lots of money. For instance, AirBnB and Zynga will be incredibly successful. Zynga has a famously low technical bar, and one of my friends came back from an interview saying that being the smartest person there wasn't enough even if it did make her rich. Obviously, being in Silicon Valley is also no guarantee that you'd be able to attract technically competent employees (Friendster was in Mountain View, for instance). But by and large, I've been amused to watch Cloudera move steadily south (from Burlingame to San Mateo and now Palo Alto). By the way, I don't think Zynga's wrong to have a low technical bar: there's no need to pay for top-end talent if your problems don't need top-end talent to solve it.
Many designers have argued to me that design talent is easier to get in San Francisco. I'm not a designer, so I don't really know, but let's say I give you that point. The problem is, to realize your design, you probably need only 1 designer for every 10-20 engineers. And of course, Apple is right in Silicon Valley, and whatever you might say about Apple, you can't argue that their design is inferior.
Ultimately, if you're a startup, think carefully about what your business is. If you never need more than about 50 engineers, I think San Francisco is fine. If you believe you're really in the media business, San Francisco's probably better (be very careful, though --- Yahoo! thought it was in the media business --- that turned out to be false!). But if your startup idea needs a sizable number of Google-quality engineers to succeed in the long term, you really should be in the valley.
Now the real puzzle to me is that there should be far more startups in the Berkeley area than they are. They've got Cal, which has a strong computer science department, so recruiting for engineers should be no problem. My guess is that the city of Berkeley does not view startups in a friendly fashion, and it would be very difficult to find cheap space in the area. Inktomi (founded by Cal professor Eric Brewer), for instance, famously moved out of Berkeley (to Foster City) as soon as it had to scale.
Subscribe to:
Posts (Atom)
