Review: A Philosophy of Software Design

 Over time, I've learned to be very suspicious of thick programming books. The thicker they are, the more likely it is that they be crammed with worthless code listings, and pages after pages of diagrams and pontification that's worthless. Thin books like K&R tend to be much more useful. When Steve Grimm recommended A Philosophy of Software Design, I was very happy to discover that it was a thin book.

You probably know many of the things Ousterhout points out, but even if you do, you've probably never seen them articulated this way before. For instance, he mentions that you should aim for API designs that have depth. His expression of "depth", however, might be different in principle from what you've heard before, which is that you should have few APIs/interfaces that are composable in ways that provide maximum functionality and abstraction.  He explicitly compares the UNIX file "open" type system calls vs the Java InputStream set of classes and finds that the InputStream type much to his distaste --- using it requires creating multiple classes stacked on top of one another, leading to verbosity, while the UNIX api just requires an "open."

There are many other common design mistakes that Ousterhout points to, including avoiding pass through methods, avoiding decorators, putting different abstractions for different layers of an API design, and taking on as much configuration complexity as possible as part of the design of the lowest layer of the API, rather than exporting it as tunable knobs at higher levels. In particular, he also points out that the error API like exceptions and return codes are part of a system's interface, and that programmers and designers need to think carefully before exposing them as part of a design --- if you can eliminate an error code, you've simplified the design considerably, and it's worth implementation complexity to do so.

I loved the sections of commenting ("write the comments first" is counter intuitive advice that might be worth trying), and naming hygiene, where he describes a bug that took him 6 months to find because a the same variable name was used to hold both physical blocks and logical blocks, and that one piece of code that confused the two slipped by multiple code reviews and inspections while silently corrupting files in a file system. These are design issues that can only be learned through hard fought experience, and Ousterhout has the scars to prove it.

I also enjoyed his perspective on object-oriented programming, agile development, and test-driven development. (Spoiler: he thinks that test driven development is bad because by writing the unit tests first, you've unconsciously made the cost of changing a design during development higher, thereby making it more likely that you'll try to patch a bad design than radically refactor it)

The book does have limitations. For instance, a lot of the complexity in modern APIs are because of the need to support legacy applications. Ousterhout has no experience with this (and neither do his reviewers, many of whom are Google engineers), and so doesn't comment on it, but it's a major issue in any company that's built a platform or is building a platform, since usually those are built in a hurry and only after success do you realize that you screwed up (the failures never need to be fixed since they have no customers). Many modern programming environments and systems make it easy to pull in libraries and dependencies from open source or other repositories, but this explosion of dependencies actually makes it really hard to debug and fix code. Ousterhout doesn't comment on that either, since most of his designs were base layer operating system type code. Another source of complexity I've found in modern systems is a penchant for putting behavioral controls in config files rather than in code. For instance, many Java logging systems assume you'd want to control logging in a properties file, which I've found to be backwards --- the logging systems should expose a programmer API, and if the application needs a config file to control behavior then there should be libraries available to do that reading and configuration, but making it hard to control behavior through an API actually makes it harder. SpringBoot (where you can spend all your time debugging inscrutable config files) is another example of this sort of misdesign.

No matter the limitations, this is a great book and well worth your time to read if you're an engineer. It's a great discussion of microarchitecture (not systems design) and good API design as well as good programming practices that makes you think. Recommended.

Update: The above was a review of the first edition of this book. There's now a second edition, and Prof Ousterhout has kindly published the additional chapters as a PDF on his web-site.

Review: JavaScript: The Good Parts

While building my Shared Checklist application, I learned JavaScript mostly by reading examples on the web as well as a library copy of Javascript: The Definitive Guide. The problem with learning a language this way is that you're generally clueless about the idiomatic uses of various language features, as well as what general good practice is.

JavaScript: The Good Parts promised to be a good introduction to the latter, so I checked it out from the library. The book's very reasonable if you have a good understanding of Scheme, Self, and one other "classical" object-oriented programming language, and have read SICP. Since I qualify as all of the above, I found the book to be a breeze, reading it in a couple of hours.

In general, I found the explanation to be clear, to the point, and comprehensible. There are a few sections that I thought weren't very good: for instance, he introduces object-oriented programming in JavaScript through the use of a programming paradigm meant to emulate "classical" object-oriented languages. The idea is that most readers come from C++, Java, or some other OO language in wide use. Then he tells you to throw away all that and switch to the Self-like prototypes-based object-oriented language instead. From a pedagogic point of view, he should have gone straight for the prototype based learning approach --- never teach someone "the wrong way" to do something first, and then teach them "the right way."

Nevertheless, reading this book before I'd gone ahead with my JavaScript implementation of Shared Checklists would have been very useful. My code's much messier than it had to be, and there are a few "best practices" that would have made everything neater.

Reading the Amazon reviews, it seems like most of the people who rate this book badly do not have the required background I mentioned above. If you have the right background, reading this book is as much fun as reading Dennis Ritchie's The C Programming Language. That's high praise, and the book deserves it.

Recommended.

Building Shared Checklist (Part 2): The Web Application

After the Android App was done, I turned my attention to building a web application. There were several motivations for this: first, I do enjoy sitting down in front of a desktop (or laptop) with a real keyboard. Nothing makes for fast data entry than a real physical device that's optimized for typing. Secondly, I wanted to explore the world of web applications. Remember that as a back-end guy for much of my career, I'd skipped the entire javascript-heavy universe of web application frameworks.

In the old days, web applications used to have 3 tiers: the client, the application server, and the database. The application server would keep tracks of things like sessions and cookies, using the database only for persistent data. The "new" approaches are forked into 2 types: the single page Javascript-oriented web application (which Firebase was designed to facilitate), where the client web-browser's Javascript engine would keep track of all the state, and went directly to the database for persistent data, and the REST-ful approach, where each click on a link would encode all the session data so that the application server was essentially stateless and didn't have to maintain state for the client. Both approaches eliminate the scaling problems of the traditional 3-tier approach, but introduced several new problems of their own, as I was about to find out.

My first thought was to try it using GWT. GWT had several nice features: one would have been that I wouldn't have to learn any Javascript. Another was a one-click deploy into Google's App Engine, which was also very appealing. Finally, I'm a big fan of static type-checking being able to eliminate an entire class of bugs.

Imagine my disappointment when I discovered that there was no integration package between Firebase and GWT whatsoever. OK, no problem, I could write my own wrapper around the Firebase javascript stuff and GWT. I tried that and discovered that Javascript objects passed into the GWT Java code became opaque objects that couldn't be used! This was very disappointing. I was even more disappointed when I discovered that it also made debugging very painful --- the GWT-generated Javascript was even more impenetrable than I imagined, and I'd have to end up learning Javascript anyway to do any kind of intense debugging. So I gave up on GWT and went full-on into writing a Javascript single-webpage web application.

Not being much of a UI guy, I looked around for a Javascript UI library, and found webix. The Treeview looked like exactly what I wanted, and I jumped right into using it. As with the Android App, the Firebase Web API (really Javascript API --- there's no server->server integration in Firebase land) is callback-oriented.

This is really weird, since if I wanted to get the user to Login right away, there's no easy way to say, "Hey, show this UI only after the user's done logging in." I tried some forms of deferred execution, but there's a real weird interaction between "login using a popup window" and the use of the Javascript firebase API.

In the end, I got it all to work, but it was clunky. For instance, I was forced to pop up dialog boxes, etc., to get the user to input a new checklist, or even a new checklist item. This was unsatisfying: in practice, I wanted people to be able to push a "+"  button somewhere on the tree and then edit a text-field inline like in the Android app. But since Webix wasn't really designed to do so without you jumping in and modifying the code, I couldn't easily do this. In the end I stuck with dialog boxes all over the place.

All in all, I'm definitely unsatisfied with the result. The debugging was also a pain: any kind of error in the Javascript causes the entire web page to freeze, and then you'll be digging through the debugger's console trying to spot and reproduce what's going on. It's not even fast compared with the Android app despite not having to download to a device. I guess that makes responsive, fast UIs in Javascript applications like GMail even more impressive to me now than they were in the past.

Building Shared Checklists (Part 1): The Android App

After determining that there's no app that did what I wanted, I decided to create Shared Checklists, which was an Android app that does do what I want.

I started with an Android app, since I did some development for my wife's app a year or so ago. The requisite tools are all free: Android Studio, my Windows PC, and my already registered custom domain. At first, I thought that I would have to write all the cloud-syncing code, but this being 2016, it turned out that I didn't have to: Firebase (acquired by Google in 2014) did all this fancy cloud-synchronization.

This is ironic, because as a back-end guy, the development of a cloud-database with syncing capability would have been fun for me, as opposed to the mostly UI-oriented development that an Android app would turn out to be.

Firebase's real-time database is a JSON-tree in-cloud database that lets you get at any subtree as a Java object inside an Android app, or treat each leaf-node as a subfield at whatever granularity you wanted. This is very nice: you get to do a write() using a Java object and have the serialization all done for you.

The penalty to this is that the Firebase API is a callback-oriented API. What this means is that updates coming in from the cloud can happen at any time, and you need to update the App's UI to reflect that. If not coded carefully, this can turn into a mess of synchronization bugs and crashes (not a surprise, right)? Even when coded correctly, you could spend a lot of time working through UI glitches, etc.

I started with a basic simple approach: a sequence of Android activities (screens) in which you deal with one aspect of the real-time database at a time: there'd be a checklist management screen, and a checklist screen. The checklist screen would let you add and remove items from a given checklist, while the checklist management screen would handle adding new checklists, sharing new checklists, and deleting checklists.

This was all pretty good: I eventually figured out that the Android App UI needed to be a dumb reflection of whatever the database was seeing, and rather than screwing around with the UI's model whenever the user made an edit or a change, I would just reflect the change in the real-time database in the cloud, and let the inevitable notifications filter down and then refresh the UI based on the cloud-activated events.

One surprising pain point was integration with cloud single-signon services. I don't know about you, but I get very annoyed now with apps and/or websites that force me to create an account. Don't make me create yet one more (likely insecure) password for your website. Make use of Facebook login or Google login services.

Facebook login was surprisingly easy. I basically got it working on the first try, which was pretty awesome. Again, it's a callback-oriented API, so I had to jump through hoops and play all sorts of tricks to get the UI to force the user to login first before I could show the contents. And that was OK. One major pain point is the integration between a database-oriented app and Android's concepts of activity screens and restore events. The documentation is horrible, and when restored from a stopped state, there's every chance that the real-time database connection had been severed! My quick and dirty solution was to basically force a Firebase reconnect and authentication every time the app was resumed from a stopped state. It's not very satisfying, but from a development point of view it was the easiest way to prevent crashes. Android studio has no way of simulating this sort of event, so it was a major pain doing it manually, fixing the crashes, and then attempting to get the OS to evict the process again. You would think that Android Studio would have a means of injecting such events into your app for testing purposes, but no.

Google sign-in integration, however, was a nightmare. Superficially, the steps seem similar. In reality, what you have to do is to run a backend server somewhere, serving a particular JSON file with certain values. You'd think that Google would automate this for someone who was running an app-engine backend on a Google hosted custom domain, but no dice. It could never get it to work! I eventually gave up completely and stuck with just Facebook login. This, by the way, now explains to me why you see Facebook login everywhere, while non-Google sites don't tend to provide Google login: the process is sufficiently onerous that you can't expect the typical web-site maintainer to use it.

After using the app for a couple of days, I grew unsatisfied with the UI. Upon reflection, I realized that the Navigation Drawer approach was the correct UI for the app. This effectively required a rewrite for the entire app to use the new approach. It was only a little painful: every Activity had to be turned into a Fragment, and then I had to rejigger some of the database schema to reflect the new approach. This approach also eliminated an entire class of bugs related to having a default checklist, so I was relatively happy with the end result.

In the end, the experience was surprisingly good. While Android Studio's layout tool still leaves much to be desired (and to be honest, for it to be a useful tool would require some sort of standardization of device that's pretty much impossible in the Android universe), it was usable most of the time. The compilation time had now sped up to the point where a full signed APK build took only 35s or so, while incremental debug builds during the crucial edit/compile/debug cycle was now under 3s. In practice, that was fast enough that the transfer to my debug device over USB was now as much of a bottleneck as build time.

All in all, given that the entire tool chain was free, I thought this was relatively satisfying as a hobbyist project.