Review: Principles

 I was assigned Principles as part of an onboarding reading. I'd never heard of Ray Dalio or Bridgewater Associates before, and I was dubious about yet another business book. But I gave it a shot anyway.

The first 2/3rds of the book were irritating. I generally agreed with Dalio that the most important thing in the world was to face reality, and that you need to iterate constantly towards your understanding of reality and to challenge your view. But I got very annoyed by his choice of words. For instance, he used "evolve" when he meant "improvement." This is a pet peeve of mine, since evolution doesn't actually have any direction or aim. I was also very annoyed by various statements about how the system he found himself in was a good one. I certainly don't think that viruses or mosquitoes do any good. We just have to live with them. I suppose if you become mega rich like Ray Dalio did, you can't help but believe that the universe you live in must be the best of all possible worlds, even if objective reality of the majority of people living around you says otherwise.

The last 3rd of the book ("Work Principles") were however a treasure trove of great ideas and deep insight. For instance, here's a brilliant section on why you need to talk to your skip-level reports, a major mistake many directors (new and experienced) make:

Probe to the level below the people who report to you. You can’t understand how the person who reports to you manages others unless you know their direct reports and can observe how they behave. f. Have the people who report to the people who report to you feel free to escalate their problems to you. This is a great and useful form of upward accountability. g. Don’t assume that people’s answers are correct. People’s answers could be erroneous theories or spin, so you need to occasionally double-check them, especially when they sound questionable. Some managers are reluctant to do this, feeling it is the equivalent of saying they don’t trust their people. These managers need to understand that this process is how trust is earned or lost. Your people will learn to be much more accurate in what they tell you if they understand this (page 459)

Here's another one about how to properly run a post-mortem:

 Avoid the anonymous “we” and “they,” because they mask personal responsibility. Things don’t just happen by themselves—they happen because specific people did or didn’t do specific things. Don’t undermine personal accountability with vagueness. Instead of the passive generalization or the royal “we,” attribute specific actions to specific people: “Harry didn’t handle this well.” Also avoid “We should . . .” or “We are . . .” and so on. Since individuals are the most important building blocks of any organization and since individuals are responsible for the ways things are done, mistakes must be connected to those individuals by name. Someone created the procedure that went wrong or made the faulty decision. Glossing over that can only slow progress toward improvement. (Page 479)

 Use “public hangings” to deter bad behavior. No matter how carefully you design your controls and how rigorously you enforce them, malicious and grossly negligent people will sometimes find a way around them. So when you catch someone violating your rules and controls, make sure that everybody sees the consequences. (Page 514)

The book has a ton of common sense, and ultimately, if I'd summarize the theme of the book, it's that the management team need to approach an organization the way an engineer approaches code:

 Great managers are not philosophers, entertainers, doers, or artists. They are engineers. They see their organizations as machines and work assiduously to maintain and improve them. They create process-flow diagrams to show how the machine works and to evaluate its design. They build metrics to light up how well each of the individual parts of the machine (most importantly, the people) and the machine as a whole are working. And they tinker constantly with its designs and its people to make both better. They don’t do this randomly. They do it systematically, always keeping the cause-and-effect relationships in mind. (Page 451)

You will note that many management books aspire to inculcate this sort of thinking. One good one, for instance, is the Fifth Discipline. I loved the Beer Game as described in that book, because it made a big deal out of how long feedback cycles derail non-systematic thinking. But the rest of the book didn't cover the practical details of how to build a learning organization, whereas Dalio's book does.

Dalio claims that one key approach that makes Bridgewater different, is that it makes its potential employees take a personality test. The resultant attributes are placed together with your track record to create a personnel summary (I imagine this to be something like a D&D character sheet). Then, when you need a team to do X, instead of randomly grabbing people who happen to be available, you can search through your employee database to find the best matches in terms of personality and track record, much like a team of D&D players might say, "We need a cleric, a bard, 2 fighters, and a wizard."

There's also an emphasis on "believability weighted decisions." The idea is that when making a decision, you want the people with the highest believability about that domain. Dalio claims that the believability is based on 2 principles: (1) having done the task (or similar tasks) at least 3 times before, and (2) being able to explain how and why the decision should be made. This is a nice balance between the dummy "democratic voting approach" or the even more dummy "I'm your boss so I'm going to call this shot for you" approach.

I tried the personality test, and got some results. Unfortunately, it still reads a little bit like a horoscope to me --- too generic and not generally all that interesting.

Nevertheless, I really thought the last 3rd of this book more than paid for itself in time spent reading it. It's in a far better class than books like Beyond Entrepreneurship or The Making of a Manager. Your time will be much better spent reading this book instead of any of those.

Highly recommended.

Post-COVID home and office design

 Recently someone showed me a group photo from a Pokemon GoFest. My immediate reaction was visceral: this looks way too dangerous during COVID19 times --- too crowded, too many people in a small space, never mind that it was outdoors. Before this year, there was hope that with a vaccine and good public health measures we could return to post-COVID19 times, but now it's looking more and more like COVID19 will be endemic.

In the short-term regardless, remote-work has become the norm, but I think that architects and office designers are still behind the curve on designing for a post-pandemic world. I'll start with the home. Prior to the pandemic, great halls were the fashion for home design. In a post-pandemic world where work-from-home is the norm, the great hall is the biggest waste of space you can imagine. Consider:

• Tall ceilings amplifies noise and creates echo-y environments, meaning that the space cannot be used for more than one zoom call at a time
• The open space does not provide isolation, whether you're doing home work, writing code, or even writing a report.
• The big empty space  does not afford power sockets which are still necessary for power or large monitors, even if your wi-fi coverage was fast enough or you had a mesh router.
• Finally, any one cooking or eating in the great hall will disturb anyone who's trying to work.

It is far better in the post-COVID environment to have a lot of small enclosable spaces than to have one big space, and home designs in the past 10 years have not caught up to that reality, and many home buyers have fallen prey to fashion rather than the practicalities of working from home.

Going to the office, the situation is even worse. Office designs in the past 10-15 years have been constrained by the costs in high rent areas such as Silicon Valley and the need to pack as many people as possible in a work environment. All the space recommendations of Peopleware for knowledge workers (engineers, artists, etc) have been deprecated in favor of open-floor plans with no walls or doors. There is no way any high end creative technical talent will put up with that sort of environment in a COVID-endemic environment. So you get announcements like DropBox moving out of their offices in favor of pre-reserved collaborative spaces.

I think for very small teams (3-4 people) it's possible to do long term remote work. But if you have a true multi-disciplinary development, you'll soon outstrip the capabilities of Zoom. Even the best remote work environments cannot beat standing together in front of a white board for impromptu design discussions. And for the most collaborative creative teamwork (think video games, or storyboarding a Pixar movie), you will require in person work. Despite my best efforts I have to constantly push people to jump into zoom calls instead of slacking at each other in a slack channel: the bandwidth provided by even an imperfect Zoom call with a shared screen far outstrips most people's ability to express themselves in the written medium!

A big company like Google/Facebook/Dropbox will probably not miss the creativity hit from daily collaborative work (though I'd argue that they do, but just as described in Peopleware, there's no way to measure the business loss from creative ideas not being put into practice, they don't know what they're missing), but if you're a startup (or in a creative endeavor like Pixar or Naughty Dog), you cannot afford to lose this, and if you visit offices like Pixar's, you'll discover that they never adopted the mass open-space fashion of Silicon Valley. (Peopleware cites examples of "skunkworks" projects where the managers successfully placed their teams in non-traditional offices precisely to maximize team work --- the only reason any startup can perform a large company is that they have focus and team work in ways that big companies cannot do) I suspect that the more creative the work, and the more multi-disciplinary the work, the more likely it is that it will benefit from in-person collaboration and team work. Hence, you might want your accounting department to be entirely remote (nobody wants creative accounting), and payroll processing maintenance and programming could probably be done remotely, but putting together a movie, high quality video game, or solving new technical problems might benefit from in person collaboration.

Unlike pre-COVID days, however, you can no longer mandate that your talent walk in the office every day. You have to make them want to do so. A lot of this is building teams where people are eager to collaborate and see each other in person, but making the office a more desirable workspace than most people's homes (which are, as described above, not configured for decent individual creative work, let alone collaborative work) is a good first step.

Those recommendations from Peopleware include:

• At least 100 square feet of private work space per person, with a door you can close for privacy and/or noise isolation. (Sorry, head phones do not cut it!)
• Collaborative work environments that are well ventilated, preferably with windows
• A gradation of private to collaborative to public workspace

Ironically, the pre-built spaces that have these characteristics turn out to be single-family homes built in the 1950s, with low ceilings, individual rooms, and a shared living room work environment. They sometimes even have kitchens big enough for a team to make and eat a meal together. It probably isn't a surprise that many successful startups had houses as office space rather than an actual office building.

If I were to design an office for the future, I would create a hub and spoke design, with large teams divided into smaller teams, each with a collaboration area, and bigger collaboration areas for cross team communications, brain storming, or design. Instead of the monolithic cafetarias of the past, you would construct smaller dining areas that let teams dine together without putting huge numbers of people together to spread disease.

It's fashionable to denigrate offices in favor of remote work now, but I suspect that the future success stories will come out of in person collaboration for the spark and serendipity that cannot occur through scheduled zoom calls. It will take real courage (not the Apple kind) to build these workspaces of the future that cannot look anything like the sardine-packed workplaces of the past, but the ones who succeed will discover that it is well worth the effort, and the reduced cost of offices in the future will be but one component of that.

Additional Reading

Has the Pandemic transformed the Office Forever? (The author seems afraid to draw any conclusions in this article, but it does a good job discussing trends prior to the pandemic)

Welcome to the Post-Pandemic dream home

Review: The Intelligence Trap - Why Smart People Make Dumb Mistakes

I checked out The Intelligence Trap from the library half expecting it to be a let-down. I thought it might turn out to be another rehash of Khaneman's book, but it turned out not to be that, though it did reference his work.

The interesting thing about this book is that it reveals a new area of study called evidence-based wisdom, a lot of the insights in this book are interesting:

• higher humility scores appear to predict scholastic performance (and on-the-job performance as well) even better than IQ.
• teams that have too many super-stars/high performers (more than about 30%) actually underperform teams of fewer super-stars.. In other words, you can actually build a team/company with too many super-stars. This is a counter-intuitive result, and is supported by examples in the book with references to literature.
• once designated a leader, executives frequently become less likely to cooperate, reaching impasses at a far higher rate than less powerful employees lower down in the hierarchy
• experts take many short cuts to get quick decisions fast. However, in that rush, they can fall prey to motivated reasoning, avoiding taking the hard decision to re-examine their work from first principles.
• Asian educational systems are actually better at cultivating evidence-based wisdom, emphasizing thoughtfulness over quickness and confidence.

All in all, the book's well worth the time, and certainly for leaders looking to build teams, has important implications for team building. Recommended.

Review: LG E42.5" UHD 4K monitor

I normally like to wait for a bit before reviewing electronics, but there's a limited time deal on the 43UD79-B monitor ($530), so I'll make an exception.

I've used wide-screen monitors like the Dell 3818 before, but I've found them to be impractical if you're a writer or programmer: the extra width might give you multiple side-by-side windows for comparing photos or viewing graphs, but when you are writing, it's the amount of text you can fit in vertically that matters. So when I thought about what was practical, the standard 4:3 ratio is what you really want, along with a big screen.

4K is not useful on TVs because of the viewing distance. But 4K is great for computer monitors. With a big screen like the LG 43" display, you can use the screen without turning on resolution scaling, giving you huge amounts of text for writing. The screen comes with 4HDMI inputs, 2 of which allow for 60fps when driven by say, an Nvidi GTX 1030 (yes, I had to upgrade my wife's Optiplex 790 so it could drive a 4K display) , and a display port input as well as a USB-C alt-mode-capable input. The device won't switch automatically between them, requiring you to push "OK" on the remote (yes, the monitor comes with a remote!) in order to flip between inputs.

I was skeptical of the remote at first, but it's turned out to be much better than the usual buttons at the bottom or sides of Dell's monitors, which respond slowly (if at all) to text input and are frustrating to use. The monitor also comes with speakers, which lets you declutter your desk by eliminating the speakers.

Like all screens, your first impression upon unboxing it is that "it's huge." But by the next day, you're going to think: "all the other screens are so small! How the heck did I get anything done on them?" It is amusing that the various collection of monitors in the house show their age by the kind of connectors available on them: DVI-D inputs are no longer provided on the modern monitors, and one monitor actually has a VGA-input, and composite-video input which are no longer found at all anywhere. All this in the space of 10 years. It looks like mini-display ports are going to go next.

By far the worst feature of a 4K screen is that compression artifacts such as those imposed by Google Photos are readily visible. I used to think that Google Photos did an acceptable job of compressing pictures, but now that I regularly see those same photos on a 4K screen I no longer think so: edges that are even a bit off look blurry, and faces don't have the same clarity that you can clearly see from the original Canon RAW file and/or the Lightroom uncompressed JPG output. I'm glad that I have a decent backup solution for my RAW images, as I'm pretty sure I will have to re-render many of them to not be annoyed by looking at them on the 4K screen. I'm doubly happy that I have high resolution cameras and don't go through life shooting pictures of my kids with crappy cell phone cameras. I'm afraid the price of a monitor this good is that the limitations of your photo gear will become readily apparent, and you'll have to buy better cameras to keep up.

I'm red-green color-blind, so I can't comment on color accuracy. If you care about it, use a color-calibration tool.

It's very rare that the cheaper device (the LG 43" monitor) outperforms and is more practical than the more expensive device (the Dell 3818), but this is clearly the case here, especially if you use your screen for real work (programming, writing, photography). If you fit into one of those categories, take advantage of the current prices and get one (and if you're a manager of a team of programmers/writers, you owe it to your team to buy one for every member of the team --- this is one of the best bang for the buck upgrades you can get for your team, right up there with SSDs back when they were first introduced).  If you don't like buying from Amazon, Costco has it for a bit more ($550). Given Costco's longer warranty coverage that might be worth the extra $30 anyway.

Monitors last forever (though my HP ZR2740W had to be replaced under warranty once), so it's not worth replacing them unless something dramatically better comes along. It looks like the LG 43UD79-B is that something. Recommended.

Review: Radical Candor

Radical Candor is an ex-Googler's book about management. Kim Scott was the manager for Adsense's sales team, and grew the team for several years before joining Apple and then working with Twitter and Dropbox. That gives her resume great credibility.

She's not afraid to illustrate the number one rule to getting ahead in big corporations: know the senior boss personally (in this case Sheryl Sandberg), and have her support you no matter how you screw up. After she joined Google (immediately as a manager, by the way --- she didn't have to work there as a leaf node), she managed to piss off enough of her team to lose several team members to transfers and departures. She writes:

The great thing about working at Google was that the company gave me a chance to fix my mistake. My boss explained exactly what I’d done wrong—and then let me hire people to replace those I’d lost. I was able to bring several people who’d worked for me at Juice to Google. (Kindle Loc. 1558)

Sounds kinda like she got rewarded for pissing off  and demoralizing her existing team, doesn't it? In my experience, that was par for the course at large corporations, so don't hold it against her.

In any case, the book is actually a good one.  Her thesis is that everything in management starts from relationships. Fundamentally, you have to have great relations with your team, to the point where when you provide negative feedback, they see it as being helpful, rather than becoming defensive or quitting. The tools she provides in the book to do so are labeled "Radical Candor." Her example is that if you see someone with their fly down, you should call it out instead of ignoring it and not giving them a chance to correct it. The same applies to verbal tics, annoyances, and of course, poor performance on the job. The book covers many such examples.

One of the best points of the book is that you need both "Rock Stars" and "Super Stars." The idea is that "Rock Stars" are high performers who are satisfied with their role, while "Super Stars" are constantly looking for the next challenge who will leave if you don't move them up quickly enough. This initially sounded to me like she was encouraging you to pigeon hole your employees but fortunately she mentions that the whole point of relationship building with your team is that you understand what phase of life she's in, and what she expects out of her work. She points out that because it is human nature to over-worship "Super Stars", you shouldn't actually make a big deal out of promotions:

Announcing promotions breeds unhealthy competition for the wrong things: documentation of status rather than development of skill. (Kindle Loc. 3656)

Note: Google isn't a great example: promotions were always a big deal, at least in engineering. Similarly, I'll note that Twitter had a singularly poor engineering culture, so her constant use of Dick Costolo as being a great manager kinda lost points with me rather than being the great examples she intended. Of course, Costolo himself might or might not have been responsible for Twitter's poor engineering culture, but bear in mind that her book's probably not intended to apply to engineering management.

With all that in mind, I enjoyed the book. Everything she writes about 1:1s, skip reporting, and management by walking around rings true. The emphasis on asking for feedback in order to model desired behavior (you want every employee to be constantly asking for feedback in order to improve) is first rate. The book's readable and full of specific examples and case studies.

My biggest criticism of the book is that Scott's ego-centricism means that she barely references prior work and doesn't even mention classics of management literature (I suspect that this means that she never read them!). But that in itself is not enough for me to avoid recommending this book for every manager, engineer or not.

Review: The Hard Thing about Hard Things

I was all prepared to find The Hard Thing About Hard Things disappointing. CEO books are frequently about self-aggrandizement (e.g., Richard Branson's memoir) and rarely give you advice you can use. Horowitz's book, however, turned out to be a rare breath of fresh air.

Sure, there's a certain amount of bragging in the book (taking a company to $1.6B is no mean feat, and Horowitz isn't ashamed to take credit for it), but that's not why you should read books anyway. You read books for practical tips that will let you avoid making mistakes when hiring, firing, or screwing up. One avoided bad hire at the executive will more than pay for the time you spend reading this book, so in that respect the book's more than well worth your time.

The book's intended audience is the CEO of a technology company. It doesn't really

Here are a few insights from my reading of the book:

• When hiring an executive, focus on whether his strengths fits your needs, rather than avoiding weaknesses. For that to work, you have to have a strong idea of what you need in that executive. Unfortunately, there's no easy way to get that insight, other than by running the organization that you need him to run for a while --- but that's OK. Better to get it the hard way than by randomly hiring people! (Note that there's a lack of emphasis on avoiding people with no integrity --- Horowitz assumes you know that and doesn't spend any time on it)
• When hiring anyone (executive or not), don't ask questions like: "He's great for the job right now, but will he scale as the company scale?" Those questions lead you to make prejudicial judgements about your executive. There's no way to know whether anyone can "scale" in a particular way, so the best thing to do is to be honest and say, "I'm looking for the best fit for the  current job. Next year, if the company changes, we'll still be looking for the best fit for that job. This applies to everyone in the company, even me!"

Deciding (with woefully incomplete data) that someone who works their butt off, does a terrific job, and loyally contributes to your mission won’t be with you three years from now takes you to a dark place. It’s a place of information hiding, dishonesty, and stilted communication. It’s a place where prejudice substitutes for judgment. It’s a place where judgment replaces teaching. It’s a place where teamwork becomes internal warfare. Don’t go there. (Kindle Loc. 2887)

• Training is a competitive advantage for startups. Don't outsource your training. In particular, you need to get your best people involved in training new people. In order to do that, however, you have to make training seem valued. The best way to do so is for the CEO to spend time training others, to lead by example. Training is also where you set expectations for your team. In particular, Horowitz include a training document he wrote for Opsware called "Good Product Manager/Bad Product Manager". In it, he defines what a product manager does, what best practices are, and how product managers will be evaluated. It's a great document that defines what a product manager does.
• Managing politics is counter-intuitive for the CEO. You have to actively manage it by setting up processes, whether it's for performance evaluation, pay raises, or promotion opportunities. If you do not do so, expect everyone on the executive team to become a squeaky wheel and playing politics. That means that the more actively non political you try to be by avoiding the usual big company BS, the more you encourage politics as people try to achieve the same goals out of band.
• Similarly, the CEO has to actively give feedback all the time. Again, this is counter-intuitive, since typically people managers are generally told to provide feedback in a "shit-sandwich" fashion --- sandwich the negative feedback between the positive feedback. The problem with the latter approach is that experienced executives see through this right away, so the only way to be consistent is to provide both positive and negative feedback as soon as you see work that deserves it.
• Organizations and Processes should be designed for the sake of the employee at the leaf node level, not for the sake of the managers. In other words, when designing or reorganizing a company, you should consider what it's like being a customer support rep or an engineer, rather than what it's like being one of the managers having to work in the environment. This may seem obvious, but keep in mind that most CEOs hang out mostly with executives and rarely reach down to the leaf nodes where people actually do the work, so it makes sense for Horowitz to emphasize this point. Furthermore, Horowitz makes the great point that organization designs all suck --- you're basically optimizing certain paths of communications between organizations while making certain other communication paths harder or even not happen at all. What this means is that along with the organization design you'll have to put into effect a plan to monitor the issues arising from the consciously designed sub-optimal paths so that if things get too bad you can redesign or hack around it.

There are other things that Horowitz say that I felt aren't necessary expressed well or are too general to be practical:

• He makes the distinction between a wartime CEO and a peacetime CEO. While that distinction has some merit, I'm not sure it makes a ton of sense. For instance, Steve Jobs was an effective asshole (and effectively an asshole) even when Apple was no longer fighting for survival. I wonder how many CEOs are going to use this chapter as a justification for being a jerk to everyone around them.
• There's an assumption that the CEO is technical, so the book doesn't discuss engineering management or the challenges peculiar to engineering management. I'm OK with that, since I did actually write a book about engineering management.
• The book actively encourages the kind of non-poaching agreement that Apple/Google/Intuit/Intel were involved in that's illegal, though of course, none of the companies involved were ever punished in any significant fashion. But you'll love Horowitz's excuse for encouraging this kind of illegal behavior: CEOs are already very lonely people and can only commiserate with fellow CEOs, so pissing off other CEOs is a bad idea since you need all the friends you can get when you're a CEO. I have no idea if Horowitz is as actively un-self-aware in real life as he seems to be when he wrote this section of the book, but there you go. Apparently, having been a successful CEO is a license to encourage others to do illegal stuff.

Finally, at the end of the book Horowitz reveals that the entire book was a marketing brochure for his VC firm, Andressen Horowitz. That's probably OK --- you didn't expect anything else, did you?

Regardless, the book's well worth a read. Like I mentioned, if it helps you avoid one of the errors I noted above, it's worth the time. Recommended.

Review: Matterhorn

My standard for Vietnam War books is The Things They Carried, by Tim O'Brien. But Matterhorn might very well replace that. Matterhorn is the name of a firebase during the Vietnam war, and the novel follows the travails of Bravo company, amongst which is one Lieutenant Mellas, a college educated Marine Corps officer who for idealistic reasons, opted not to go for deferment and ended up in the infantry instead.

One by one, you get to know other members of the company, officers, NCOs, machine-gunners and yes, the guy who's only got a few weeks left to go on his tour and is dreaming of going back to the girlfriend he left behind in Thailand. The story is good, with Bravo company getting screwed over by senior military officers who're trying to make themselves look good at the expense of the men they command.

If you're wondering why a Vietnam War novel might be relevant to a software engineer, I think this short passage might change your mind:

“You know why we’re really strung out in this fucking death canyon?” Mellas didn’t know, so he just grunted. “Because Fitch doesn’t know how to play the fucking game. That’s why. He’s a good combat leader. I’d literally follow him to my death. But he’s not a good company commander in this kind of war. He got on Simpson’s bad side because he got his picture in the paper too often and never gave Simpson credit, which by the way he doesn’t deserve, but that’s the point. The smart guy gives the guy with the power the credit, whether he deserves it or not. That way the smart guy is dangling something the boss wants. So the smart guy now has power over the boss.” (Loc. 3841-47)

Over and over again, the novel doesn't flinch from the power politics that are played at high levels in a corporation (and in this case, the Marine Corps is just as functional or dysfunctional as any large corporation). At one point, Bravo company is tasked with digging trenches and building bunkers to defend a hill --- only to be told to abandon it to prepare for another assault elsewhere in Vietnam. Whereupon the North Vietnamese Army (NVA) promptly take over those defenses and Bravo company is then tasked with assaulting the very defenses they had built from a disadvantageous position. The poor Google engineers who built the very first version of Google Drive were similarly told to abandon it, only to have to launch again after Dropbox proved that the market existed and is pretty lucrative must have felt very similar to the marines in Bravo company. In fact, just as some of the high performing officers were unfairly blamed by their commanding officers for incompetence, I myself heard a Senior Staff Engineer at Google blame the former tech lead for Google Drive for failure to push against the killing of the project.

There's a passage where Mellas thinks about the Colonel in charge of the operation:

Mellas would probably have said that Blakely didn’t have what it takes, but Mellas would have been wrong. Blakely would have performed a lower-level job just as well as he performed his current job—competently, not perfectly, but well enough to get the work done and stay out of trouble. He’d make the same sorts of small mistakes, but they’d have a smaller effect. Instead of sending a company out without food, he might place a machine gun at a disadvantage. But the Marines under him would make up for mistakes like that. They’d fight well with the imperfect machine-gun layout. The casualties would be slightly higher, with slightly fewer enemy dead, but the statistics of perfection never show up in any reporting system. A victory is reported with the casualties it takes to secure that victory, not the casualties it would have taken if the machine gun had been better placed. There was nothing sinister in this. Blakely himself would not be aware that he’d positioned the machine gun poorly. He’d feel bad about his casualties for a while. But reflecting on why or for what wasn’t something Blakely did. Right now the problem before him was to engage the enemy and get the body count as high as possible. He wanted to do a good job, as any decent person would, and now he’d finally figured out a way to do so. He might actually get to use the entire battalion in a battle all at one time, an invaluable experience for a career officer.  (Loc. 6174-84)

That's the reality of management in a big organization, and an inherent limitation in the data-driven management techniques used today. Suboptimal code (or machine gun placement) sure as heck matters to the marines who get killed because of it (and to the engineers who have to maintain or work-around the problems), but it's not visible at all in the aggregate level to senior management. As a result, incompetent managers with serious political skills get promoted far more frequently than competent managers who lack such skills. In a high quality organization (like the Marine Corps or Google), the rank-and-file who get hired (or enlisted) are so good that they can make even incompetent managers look great. In fact, in certain circumstances, high casualties, constant war-rooms, and constant enemy engagement can make such managers look like stars, even though a better manager could have avoided all of the above. (And no, I have no idea whether the Marine Corps or Google's rank and file are really that far above average nowadays, but back when I was at Google, the average engineer was really really good, and in many cases much better than the average manager)

I'm at risk at this point of making this novel sound like a treatise in office politics, self-promotion, and lessons in how to make yourself (and your boss) look good rather than a great novel.  Let me try to disabuse you of that. It's a great novel. It's got great characters, a transparent prose style, an interesting plot and setting. It explains why the North Vietnamese beat the Americans despite the latter's overwhelming technology advantage: the terrain and weather negated most of the advantages the Marines had over their enemies, and organizational dysfunction took care of the rest.

But at this point, the novel has won so many awards and accolades (it took 30 years to write and publish!) that anything I can say about the conventional aspects of the novel can be (and probably has been) better said elsewhere by professional reviewers. The novel delivers everything a novel should deliver, and provides lessons and entertainment in spades. I paid $2 during a Kindle sale for it, but knowing what I know now would not hesitate to pay full freight. Buy it, read it, and enjoy the heck out of it. And as you do read it, the management/political lessons it provides might turn out be really useful in your career. That makes this book highly recommended.

Review: Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future

One chapter in Peopleware that always resonates in my mind is the chapter on "Spanish Theory Management":

Some years ago I was swapping war stories with the manager of a large project in southern California. He began to relate the effect that his project and its crazy hours had had on his staff. There were two divorces that he could trace directly to the overtime his people were putting in, and one of his worker's kids had gotten into some kind of trouble with drugs, probably because his father had been too busy for parenting during the past years. Finally, there had been the nervous breakdown of the test team leader. As he continued through these horrors, I began to realize that in his own strange way, the man was bragging. You might suspect that with another divorce or two and a suicide, the project would have been a complete success, at least in his eyes.

Elon Musk is a biography of the man, and if you weren't aware of the era that both books were written in, you might well suspect that Elon Musk was the manager Tom DeMarco was referring to. Consider this: in this book alone, he scolded an employee for attending the birth of his child instead of attending a work event. He repeatedly set impossible schedules, and then push employees past the breaking point and then discards them:

“Elon’s worst trait by far, in my opinion, is a complete lack of loyalty or human connection,” said one former employee. “Many of us worked tirelessly for him for years and were tossed to the curb like a piece of litter without a second thought. Maybe it was calculated to keep the rest of the workforce on their toes and scared; maybe he was just able to detach from human connection to a remarkable degree. What was clear is that people who worked for him were like ammunition: used for a specific purpose until exhausted and discarded.”  (Loc. 4911-15)

At one point, he even fires his administrator who'd been with him for more than 10 years:

Brown often felt like an extension of Musk—the one being who crossed over into all of his worlds. For more than a decade, she gave up her life for Musk, traipsing back and forth between Los Angeles and Silicon Valley every week, while working late into the night and on weekends. Brown went to Musk and asked that she be compensated on par with SpaceX’s top executives, since she was handling so much of Musk’s scheduling across two companies, doing public relations work and often making business decisions. Musk replied that Brown should take a couple of weeks off, and he would take on her duties and gauge how hard they were. When Brown returned, Musk let her know that he didn’t need her anymore, and he asked Shotwell’s assistant to begin scheduling his meetings. Brown, still loyal and hurt, didn’t want to discuss any of this with me. Musk said that she had become too comfortable speaking on his behalf and that, frankly, she needed a life. (Loc 4926-32)

There's also a section where Jeff Bezos poaches one of SpaceX's employees by doubling his salary. Characteristically, Musk, rather than consider whether he underpaid that employee, thinks that Bezos and the employee betrayed him.

Keep in mind that I'm sympathetic to Elon Musk's goals and background. Not only was Musk a huge science nerd and programmer, he also played D&D in his youth, and of course, if electric cars replace the internal combustion engine, the world would be a much better place. I also enjoyed the section on Musk bringing startup-style mentality to the aerospace, which apparently needs a huge kick in the pants and massive cost-cutting.

What's unfortunate about this book is that Ashlee Vance treats Musk's approach to engineering, scheduling, and design as being par for the course: that abusing employees, creating impossible schedules through optimistic CEO-level views on how long something ought to take was the only way for Elon Musk to achieve his goals and get his results.

Imagine an alternate world in which Musk was a better leader: it could be that instead of having a large number of rocket failures and massive amounts of drama, his rockets could have had fewer test cycles, and finished in approximately the same amount of time. Of course, maybe launching something without drama and having it work properly the first time wouldn't merit a book.

In any case, it's worth reading the book, as it does provide a behind the scenes look at Tesla and SpaceX that's entertaining and interesting. But you do have to read between the lines to see a few interesting underlying principles:

• Certain non-tech related fields like Space/Aerospace and Cars are ripe for disruption by Silicon Valley startups. In particular, fields that have fossilized and gotten used to fat margins and inefficiency workflows are vulnerable to attacks from Silicon Valley.
• Ironically, part of this attack is due to the ease of exploitation of the underlying workforce: nobody who's actually a good mechanical or aerospace engineer enjoys working under the bureaucracy of the entrenched businesses. You can therefore lure such people to work for you at below market pay and work them hard for an extended period because you offer effectively more responsibility and freedom of action than the bureaucracy. When those people burn out, replace them with more fresh graduates. This is known as the EA model of HR management.
• If you succeed, you'll get lauded in the business press, and then have books written about you.

This is obviously excessively cynical, and as noted above, I do agree with Elon Musk's goals, and think that in the coming battle between Silicon Valley and Detroit, there's no question Detroit is going to lose. But it's still sad to see obnoxious business practices praised and lauded as though there aren't better alternatives.

Nevertheless, read the book, and see if you agree with me. Recommended.

Indexing Google's Source Code

I couldn't talk about this before, but now that Wired magazine has disclosed the size of Google's code base (2 billion lines), I can discuss my authorship of gtags and what it did for Google, as well as some funny stories arising from that.

I wrote the first version of gtags in 1991 (yes, gtags is older than Google!), when I was at Geoworks. GEOS was several million lines of assembly, including every freaking app written for that OS. Since every object could potentially call any other object, the entire code base was relevant. Needing to get my head around that code base, I tried to build a TAGS database and that immediately caused my Emacs to start swapping. The performance was unacceptable.

The core insight was this: there's no reason to use a disk based search on a TAGS database. Stick the entire database into RAM, and use a hash-table to lookup your keywords, and search performance would go from multiple seconds (half a minute in some cases) to sub-second response time. So one weekend I coded up the hash-table, wrote code to load up a TAGS database into memory, and implemented a simple UI that let me browse code in Emacs. Soon, I enjoyed sub-second search times and could grok code that would have been impossible to do any other way.

If I ever needed validation that the tool-building approach to coping with large-scale software was the right approach, this was it. Once the senior engineers (remember, I was an intern at Geoworks then) got hold of the tool, I saw even loyal-vi users switch over to Emacs just to get their hands on the code browsing functionality (going from half a minute per search to sub-seconds was critical).

After I left Geoworks, most of my code was in C, C++, or other high level languages. Computers got so fast, and IDEs so sophisticated that I never dealt with a code base that couldn't be loaded into the IDE. It seemed to me that the need for such functionality had been obviated by ever more powerful machines.

That was, until I joined Google in 2003. By then, Google's code based was already approaching 1 billion lines, but in multiple languages. I needed to wrap my head around that code base in a hurry. Various teams were using random tricks to subset Google's code base into their IDEs, which I thought was a kludgy and unsatisfactory way to work. So in my 20% time, I rewrote my old tool using Google infrastructure (thanks to Craig Silverstein, who was the only person who believed in my code enough to waste precious time code reviewing it --- even then he was skeptical that my tool would be widely used or even substantially useful, given the huge amount of effort people had put into subsetting the codebase). I coded up the UI again in Emacs Lisp. I actually had to put some effort into the UI this time, given that C++ (and Java) overloading meant you had multiple search results for any given search term. Thankfully, Arthur Gleckler came in to lend a hand. Reading Arthur's Lisp code was like reading poetry: you can't believe the succinctness and elegance that can be expressed in so little space. It's worth your time to learn Emacs Lisp just so you can read Arthur's code.

Just as I expected, gtags took off in a huge way inside Google's engineering team. (By the time I left, 2500 daily active users was the metric, or about 25% of Google's engineering workforce. The internal tools team did a survey once and discovered that nearly every engineering workstation had a copy of Stephen Chen's gtags-mixer running on it) There wasn't a whole scale conversion from vi to Emacs though: Laurence Gonsalves stepped in and wrote a vim script that emulated the Emacs code. I don't even remember how I managed to do the code review for that checkin, but anything to help gtags, so I must have just gritted my teeth and done the code review or asked Laurence to find someone competent to review it.

But I wasn't nearly even close to done. Because of the huge amount of ambiguity and overloading involved in C++ and Java, I wanted gtags to do a better job of ranking the results of any given search. Phil Sung took a first crack at it, introducing Sung-ranking and later on, an include-rank that mirrored page-rank, except for code. Stephen Chen solved the problem of how to intermix protocol buffer files into the search results. Matei Zaharia (now a professor at MIT) spent a summer integrating a parser into the indexer for gtags, so that it was no longer a dumb lexical scanner but a full-on type-comprehension system for both C++ and Java. He also designed and implemented incremental indexing on Google's code base, no mean feat. Leandro Groisman and Nigel D'Souza both also made major contributions to gtags.

For several years, I had the entire Google source repository downloaded and checked out on a dedicated gtags indexing machine sitting under my desk. It was a standard underpowered workstation of that era: dual core, 2GB of RAM, and 500GB of disk: it had a special p4 client that eliminated the need to download any binary assets, since it was only interested in code! It was probably a major security hole, but I figured since Bill Coughran knew about it, I wasn't violating any corporate policies.

This illustrates a very important point: 2 billion lines of code sounds like a lot of code, but if you do the math (assuming 50 characters per line) you'll get only about 100GB of data (uncompressed). After dropping comments, white space, and lines that don't perform any declarations, your index is going to be pretty tiny, and you need to split that code base into several corpora (C++, Java, protocol buffer declarations, python), so each individual server could easily handle its entire corpus in RAM without any fancy sharding. Too many people get caught up in trying to apply fancy Google techniques required to manage terabytes of data when they're dealing with tiny amounts of data that fit into RAM and can be managed by traditional programming techniques.

In any case, gtags was a very hardware light project: it never took more than one machine to index all of Google's code base (and we never had to apply any fancy MapReduce techniques), nor did the serving cluster ever exceed more than about 10 machines. We came close to maxing out the RAM available on 32-bit machines for a while, but between Phil's string table optimization reducing memory use by 75% and the switch to a 64-bit architecture we never ever had to split indexes for any given language (there was a server for each language) across multiple servers. Those servers were under-utilized of course (they could probably have served 25,000 or 250,000 users at once), but on the flip side, you always got sub 10ms response times out of gtags. We switched from dedicated gtags server desktops sitting under people's desks to Google's cloud internally fairly early on, with Ken Ashcraft doing much of the work of converting gtags into a borg-ready service.

This came to a head when Google added the China office sometime in 2005 or so. After that, the powers that be decided that high intellectual property (HIP) code needed special permissions to access. Since I wasn't HIP enough, I simply stopped indexing that code. This burdened the HIP people so much that eventually some of them (including Sandor Dornbush) contributed to gtags. A HIP guy would take on the burden of downloading HIP code and indexing it using our indexer, and then put up the gtags server with that code behind a HIP firewall. The gtags-mixer would then be configured to talk to the HIP server and mix-in the result if you were HIP enough.

One of my prouder moments at Google was when Rob "Commander" Pike came to me and asked me how gtags worked. It turned out that he didn't want to talk to the gtags mixer or the gtags server, but just wanted his programming environment/editor to directly grok the output of the indexer. I was happy to give him access to the index for him to do whatever he wanted with it. I forget the mechanism by which this happened: he might have simply scp'd the index over to his machine, or I might have had the indexer push the index to his machine whenever it was done. This was great, because Rob became one of the folks who would notice whenever the indexer was broken because the file wouldn't get updated!

In any case, as with many things at Google, after I left gtags got replaced by some cloud solution that took way more resources than me, Arthur, and a bunch of interns, and I'm sure everything I wrote has been long retired by now, with the possible exception of the Emacs Lisp front-end.

Even after I left Google, gtags paid me back. Soon after I met my wife, she talked to some of her friends at Google about who she was dating. One of them did a p4 lookup on my changes, and said to her, "Hey wow, this guy has code commited everywhere, even the protocol-compiler." So I guess that worked out as far as a back-door reference check was concerned. (That change in the protocol-compiler was necessitated because I wanted to inject a clue in its output: that clue enabled the gtags indexer to map a generated C++ source file back to its original .proto form --- it was far easier to do that by having the protocol compile emit the clue than to try to guess --- it was a trivial change and Sanjay approved it in seconds)

If it seemed unbelievable to you that during that period of time I had such an illustrious group of people on a tiny 20% project, it should be. But I maintain that the test of a high quality engineering organization is whether or not that organization is able and willing to invest time, money, and effort into building tools that enable that organization to move faster and produce higher quality code. Google met that test and passed with flying colors.

Fictional Leadership

I've often maintained that fiction is as good a source of leadership training as non-fiction, since much non-fiction is written in terms of platitudes and generalities, while fiction frequently presents specific situations. Of course, leaders in fiction are always superb, but humans always learn better from positive examples than from negative examples, so that's not a bad thing.

I was watching the Battlestar Galactica reboot's Hand of God episode the other day with Xiaoqin, and was particularly struck by how great an example of leadership the episode had in the form of commander Adama. We see him in various different roles that illustrate what role leadership plays, and what a great manager should do.

The episode begins with a planning meeting. Adama's role here is simple: he needs the proper diversity of thinking and expertise in the planning stage to design the best plan for the assault on the Cylon base. Note that while the news media loves to consider diversity of races as a proxy for diversity of thinking, here Adama cares very much about having a diverse of mindset. He turns to Starbuck (Kara Thrace) to provide that. Not only that, at the meeting he carefully backs up Starbuck, by telling both Apollo and Colonel Tigh: "With all due respect, none of us are as crazy as Starbuck." There are lots of subtleties about that meeting, including that Starbuck and Tigh hate each other, and Adama is aware of that. Notice how deftly he shuts down the name calling that the two of them were about to start, which would have been unproductive and prevented a good plan for being formed. Leadership is frequently about bringing the right people in the room and managing the context so that you can get the most out of everyone, and this is a great example.

Starbuck is a great pilot. In the arena of technical management, she'd be considered a great tech lead. That made her a natural to lead the assault, but her healing leg meant that she had to be kept out of the fight. Adama convinces Starbuck of this not by giving her an order, but by showing her that her legs are not yet strong enough to  let her perform at her best. This is another great example of leadership as persuasion: it's not enough to say "no, you shouldn't do this." You have to provide examples why.

Then there's the scene where Apollo is up the night before the mission. He's anxious, and already defensive because everyone knows that Starbuck's the best pilot in the fleet but he's having to substitute for her. Adama's aware of this, and carefully steers Apollo's anxieties away from this, providing assurance that he's going to do a great job. He even hands Apollo his personal lucky charm to assist. He then tells Apollo to get some rest in preparation for the mission.

During the execution of the mission, Adama provides leadership mentoring to Starbuck, telling her that once she's laid down the plan, the execution is in the hands of the others, and that her obvious anxiousness would actually undermine the operation if she doesn't retain a good grip on herself in the operations room. Not only does that calm her, he's clearly also grooming her as a future leader.

Finally, when the operation is successful and everyone's celebrating, notice what Adama does. He carefully places himself at the edge of the celebration. He's cheering folks on, and there to receive the lucky charm back from Apollo, but at no point does he attempt to steal the credit for the success. His people are allowed to say to themselves, "We did this great thing!" This is servant leadership at its greatest, and you very rarely see it in real life.

Of course, the fictional universe Adama is in is very stark. Adama isn't racing for the next promotion (there's no one to promote him), and the stakes are high, so it's easier to motivate people to do the right thing. But that's actually not that much different from a startup (getting a promotion at an unsuccessful startup isn't going to do much good). But it's still great to watch a great leader in action, and I can only think of a very small handful of people who in real life could match what Adama did in this episode.

If you haven't already seen this, it's well worth the $1.99 to watch this episode for these leadership lessons. As a technical manager, it's well worth the 44 minutes of your time.

Interviewing and Performance

Recently, someone asked me a deep question: we all know (and Google has the data) that interviews do a poor job of predicting on-the-job performance. If that is the case, would a different form of interviewing (say, pair programming) or other form of testing do a better job?

My answer to that question is "no." What most of the other articles do not note is that Google actually does have data as to the major factor influencing on-the-job performance (at least, performance as viewed by Google's notorious promotion committees). It turns out that even in 2003-2004, Google had data indicating that your first tech lead at the company strongly predicted how well you would do in the future inside Google.

There are several reasons for this. One obvious one is that the promotion committee is likely to weigh your tech lead's comments on your performance more heavily than some other random engineer's. The deeper reason, however, can be found in the book, Chasing Stars. Fundamentally, all organizations have stated or unstated rules for how they work. Whether the on-boarding systems do a good job of explaining that to new employees and indoctrinating them in the culture very much explains future performance.

Google at the time when I joined was a largely oral culture. A typical noogler joining the engineering team would during his first week of working through the engineering training document find several bugs a day in the documentation necessitating a documentation change, if he were conscientious. Old documentation or out of date documentation was rampant, and the tech docs team had their hands full trying to keep up with the amount of code and internal APIs continually being churned. If you actually had to get work done, your most important tool wasn't the documentation or the internal search engine (which was laughably bad), but knowing who to talk to. For instance, if you needed to make a change to crawl, and your tech lead knew to say, "Go talk to Arup Mukherjee and ask him how you would do this", you were in luck and you'd be productive and efficient. If your tech lead said, "Go read the documentation," or worse, "Use the Source, Luke", not only would you waste a lot of time reading both code and documentation (as I was forced to once when dealing with the search results mixer), chances are when you were done you would probably have done it wrong, and your code reviewer would spend gobs of time correcting the way you did things, and forcing you to do everything over and over until you got it right. If that happened, you might as well kiss your "Exceeds Expectations" performance review goodbye. (And yes, I lucked into knowing people who wouldn't just tell me who to talk to, but walked me to their cube, provided introductions, and made it clear that what I was supposed to do was important enough to deserve help)

I'm fond of saying that context matters a lot when it comes to performance. This type of context-sensitive performance isn't necessarily because the tech lead deliberately led the poor engineer wrong. It was because the tech lead did not provide a suitable context for the engineer to work with, and in the process makes the job much much harder (or in some cases nearly impossible) for the new engineer. Hence if your interview process is successful in eliminating people who can't actually do the job, but you end up with variable performance or unexpectedly poor performance on the job from people who should be doing well, you need to examine your on-boarding process or the training process for your leads/managers.

The follow up to this question then is, "If performance is so context determined, why do we bother with interviews?" The answer to that is that the goal of the interview isn't to predict performance in the future. The goal of the interview is to ensure sufficient technical competency and cultural compatibility so that with a good on-boarding process or a decent tech lead/manager, the new engineer ought to be able to do a great job. Hence, when I run interviews, I don't favor esoteric problems that require dynamic programming (for instance), but basic data structure questions. While I consider basic tests such as the Fizz Buzz Test way too simple and insufficiently indicative of someone with basic computer science knowledge, coding questions that approximate that level of complexity (while still testing basic computer science concepts) is all that is typically needed to weed out people who simply can't code and shouldn't be allowed access to your source control system.

Review: How to Win Friends and Influence People

How to Win Friends and Influence People is currently $2.99 on the Kindle store. I'd gotten this far in life without reading the book, but a friend of mine told me she used a technique from the book and it worked, though she felt slimy about it afterwards. That's intriguing enough to get me to buy and read the book.

This is a great book, as far as being an effective politician and getting what you want from people goes. Fundamentally, the book is all about helping you tell people what they want to hear, as opposed to what reality is. For example, in one anecdote, the manager of a singer who refused to get on stage simply lied to him over and over again until he did so. In another example, Dale encourages you not to tell people that they are wrong, but to pretend that you could be wrong and asking to check the facts. In certain circumstances, that could easily win you favors, sales, and deals. In other circumstances, it could make you look like an easy pushover and mark, and you will get out-maneuvered by more politically savvy folks, especially if you're an engineer. Dale Carnegie, however, doesn't tell you how to distinguish between those circumstances. For instance, if Galileo had read this book, he might easily have avoided the Roman inquisition. It would have done immense harm to the scientific enterprise, however, so I'm glad the world is not full of people who've read Dale Carnegie's book.

People occasionally ask me for advice on their careers. Given that I'm completely oblivious to office politics, I'm a bad person to ask. But I do refer them to books such as Career Warfare. It's quite clear to me that How to Win Friends and Influence People is also a great book to read if you want to succeed at a large company, where perception is much more important than reality. Keep in mind, however, that if you're an engineer, you're a much worse liar than anyone who's not an engineer, so some of these techniques absolutely will not work for you.

Highly Recommended.

Quantifying the Apple Tax

The last time I was a Mac user was in 2009 when I upgraded to the Canon 5D Mk 2 and got rid of my Mac Mini as being too slow to run Lightroom. Before 2006, I was a PC user and never bothered paying the Apple tax. It wasn't until I joined Quark Games, which was a mostly Mac shop that I ran into the Apple tax again.

The Apple tax impacts small development shops. Large corporations  like Adobe or Google aren't cash constrained. In fact,  at Google, most developers wouldn't even be aware of the Apple tax because most of their computation is done in the cloud. At a small shop like Quark, however, we are cash constrained and most of our computation is done locally, at the developer's desk.

Most of Apple's desktops are incredibly under-powered. For instance, the iMacs don't even let you replace a hard drive, which means that you have to pay Apple's incredible markup for SSDs, and in the case of the smaller iMacs, you can't even upgrade the memory yourself. For a developer workstation that potentially needs more than one SSDs, this is unacceptable. Yes, you can upgrade to a 3.4 GHz Core i7, but that's even more expensive than a Mac Pro and you end up with a machine you cannot upgrade.

Then there's the Mac Pro. It's mid-2013, and they cost $2500. What's worse, is that they use a 2009 Xeon CPU which under-performs my 2008 home desktop! And that machine cost me $1200 back in 2008! You can compare it with a current Dell with the latest Haswell i7-4770 processor. That machine would cost $750, with twice the processing power of the Mac Pro! Sure, the Mac Pro has a nicer case which makes it easier to upgrade. And it has ECC RAM (for all the good that'll do you --- I can't remember a single instance where I wanted ECC RAM for any of my development needs). The fact is, Apple has no mid-range towers, but if you need to deliver iOS applications to your customers you have no choice: you have to buy an Apple product. Yes, I'm aware Apple has a new Mac Pro at the end of the year. However, the new machine has no room for hard drive expansion at all, so I might as well buy an iMac!

At Quark, our solution has been to buy the 13 inch non-Retina Macbook Pros. With a couple of screwdrivers you can take those apart and upgrade the memory and hard drive. Unfortunately, when you need to process a lot of art and music assets, the CPUs on those machines bog down. Even then, using Macbook Pros save you because when we bought our Mac Pros, we could take the hard drives out of the laptops and stuff them into the desktops and get immediate productivity gains, without the pain of reinstalling all our software and losing a day in the process.

What's amusing to me is that the art team tells me that the rest of the industry has gone to Windows PCs for 3D-modeling and other art needs precisely to escape the Apple tax (and these despite Apple's reputation as the go-to computer for artists!). So it's only engineering that's stuck paying the Apple tax. Certainly, if Android were too crush iOS devices, small development shops will be the first to switch sides completely just to avoid paying the Apple tax, which stands at $1700/developer. I know I'll be switching our art team over at the next available opportunity. In the mid-1990s, I dreamed of the days when we'd escape the Microsoft hegemony. But now that we've largely escaped it's clear the Apple overlords are much worse than the Microsoft ones ever were.

The "Science" in Computer Science

When I was an undergraduate, it was very common for many of us to view the "science" in Computer Science as an oxymoron. The proof was that all the "real" sciences had names like "Physics", "Chemistry", "Astronomy", and "Biology", while we were lumped in with "Political Science", "Social Science", and of course, "Military Science". Many took the position that Computer Science should be considered a branch of mathematics, while those of us who were liberal arts majors (I count myself and Jonathan Blow as the major advocates of this) considered Software Engineering a branch of the literary arts as the goal was ultimately to produce code that was easy for humans to read, since machines didn't care what your code looked like.

If you read Thomas Kuhn's The Structure of Scientific Revolutions, however, there is a sense in which Computer Science is a science. Consider the construction of a program to be a sociological construction of a theory about how best to approach a problem. You start out with version 1, which solves some portion of a problem. Later on, as the problem is better understood through the lens of your theory (i.e., your users start using your program and start providing you with feedback), you tinker with your theory to make it better fit the evidence (user feedback or market feedback). As a result, your program becomes more complicated and your program's structure (theory) starts to show it's datedness. When things go to a head, however, you either refactor or rewrite all the offending crufty code, throwing it away and replacing it with a new program (theory )that accommodates all the evidence to date. This is equivalent to perhaps relativity supplanting Newtonian physics. Note that the analogy even holds here --- old versions of your program continue to work, but the newer program (better theory) is more elegant, and fits better with the problem space. If your rewrite fails, the result is less useful than the previous version and society refuses to adopt your new program. For instance, Vista was not widely adopted and most users stayed on Windows XP instead.

There's even space in there for unit tests and systems tests: those tests are the empirical experiments by which you attempt to prove that your theory (program) works. In effect, when writing tests, you're trying to prove that your theory about how the problem should be solved is wrong. If you have the resources, you might even want such experiments be run/written by a third party, so they have no cognitive biases with which to approach the problem.

Obviously, this view of software engineering as actually "doing science" can only be carried up to a limit, but I find it to be an interesting analogy, and would be interested in hearing your thoughts.

Market Efficiency

Most companies understand not to cheap out on machines for engineers. The cost of a top end development machine is about $5,000, and when you're shelling out $100K a year, even a brand new machine every year just doesn't cost that much (in practice, nobody upgrades every year --- there's enough overhead when switching machines that doing it every other Moore's cycle --- 36 months makes much more sense). And every bit of increased productivity means you get that much more out of the $100K/year asset.

But once in a while, companies come across an under-priced engineer. Whether because they're inexperienced and didn't know how to negotiate, or whether they were initially offered less compensation because they were an unproven quantity, it's extremely tempting to keep under paying them for as long as you can get away with under paying them. What's happening in this case is that management thinks that they've stumbled across an engineer who's an idiot savant --- that somehow it's possible to be a great engineer who happens to be clueless as to his net-worth. There's no doubt that such people exist (I know some of them), and if you're at a big company that risk might pay off, but it's an insane risk to take at a startup where every engineer matters.

Here's what eventually happens. The engineer has friends, and eventually his curiosity will lead him to compare compensation with those friends. When he learns that he's significantly underpaid, he'll get pissed off enough to interview, and if you're lucky enough, to ask you for a raise. At this point, you'll have to give him a raise. If you're smart, you'll give him a raise, and compensate him for his lost wages as a result of you underpricing him in the first place. Most companies might do the first but rarely do they do the second. The consequences of not doing the second is that the engineer you've pissed off is out interviewing, and will end up with higher offers and you'll end up paying back those lost wages anyway, assuming you manage to keep him. If you don't, you'll spend that money recruiting and training a new engineer to replace him.

You might argue that a startup can't afford cash and raises. You might be right. But there's no excuse even then: if you can't afford cash, then you can provide additional equity. Equity is even better, since you can tie that to a vesting period, which would keep the employee loyal for years to come, and raise the bar for anyone else trying to poach your employees.

One of the things that impressed me about Google was its willingness to raise new graduates to market rapidly --- it was not unusual for a new graduate at Google to get a 30% raise on her first promotion, reflecting her increased value. However, Facebook was even better in that regard. I've had reports of Facebook granting retention packages even before the new employee's first year is up for review! This is a great approach, because the employee considers this unasked-for raise a gift. What happens to you when you receive a gift? You feel obliged to give back. So not only have you made a high performing employee happy, you've ensured that he's going to work even harder for you, at least in the near term! Contrast this with the employee mentioned above who had to ask for his raise (after realizing that he was underpaid): he didn't feel like he got a gift, he got screwed. Even raises that come from promotions don't feel like a gift, because the employee felt like he had to work for it.

Regardless of the performance of its stock, I've made the statement in recent years that Facebook probably has the best engineering management in Silicon Valley, and this is just one example of what they do better than anybody else. (Note: the author does not own any Facebook stock)