On Open Floor Plans

Way back in 2006, I was working for a startup, originally called Sharefare and later renamed to Ludi Labs. When I first joined Sharefare, we had an old office space in Los Altos. It was a "traditional" office space, divided up into offices, with only two large open areas. One was the front reception, and other was a break room area. While there, we had around ten engineers. We would put 2-3 engineers per office. It worked pretty well.

After I had been at Sharefare a few months, we received another round of funding, but it came with a catch. Our board wanted us to hire a "real" CEO, since our CEO/founder was a very technical guy, not an MBA type at all. Our CEO came from Under Armour, you know the sports apparel maker. He was an east coast guy, who I'm sure we paid a ton to relocate to the Bay Area. When he arrived, we were running out of space in our little old office in Los Altos. So he decided that one of his first priorities would be getting us a new office. He had a vision for the new office -- the floor of the Wall Street stock exchange. He wanted an open floor plan that would encourage collaboration and keep everyone energized. So we moved into a brand new office (we were the first tenants) with a completely wide open space.

Now I always attributed this hubris to our CEO being an east coast guy who had never really been around engineers. You see, engineers do not work well in hectic, loud environments like the floor of the Wall Street stock exchange. A very large part of their job is thinking, and people running around making noise does not help an engineer think. Lots of other folks have written about what kind of environment does work well for engineers, so I won't go too much into it. I'm just here to tell you what does not work.

The open floor plan is complete fail. The most obvious problem is the noise. First, most companies have some non-engineers who work for them. Some of these non-engineers really need to do a lot of talking to do their job. They need to be on the phone ordering office supplies, or talking to potential business partners, or screening new hires. These people constantly generate noise. Put that in an open floor plan, and there is always noise. This is a constant drain on other folks, like engineers and designers, whose work requires them to think a lot.

The next problem is that engineers and designers, do indeed need to talk sometime as well. Sometimes this is talking about work stuff, but there is also a lot of non-work stuff that engineers talk about. Maybe they are debating closures in Java or the merits of Python vs. Ruby. You want engineers who bullshit about stuff like that. Or maybe they are just talking about the Super Bowl or Lost. There is value in this too (y'know that whole "team building" thing.) Anyways, when you are in an open environment where noise is already problem, then it really discourages this kind of engineering conversations. If you are lucky enough to need to talk to your fellow engineer when it is relatively quiet in the office, you may be reluctant to engage him/her because you want to relish the peace. If instead it is during a typically noisy time, you will ask yourself if whatever topic is really worth talking about if it means shouting over the din. Instead of encouraging collaboration, the open floor plan does the opposite.

Back to Sharefare... It did not take long for everyone to realize that the open office was not working. First, we tried to put the engineers and non-engineers at opposite ends of the office. We had a handful of real offices that were intended for execs. Instead they were kept open, so that people could use them for longer phone conversations. I personally spent a lot of time in those rooms, doing phone interviews with potential new hires. Finally, the CEO came clean and admitted that the whole thing had been a bad decision on his part. He bought everyone Bose noise canceling headphones, supposedly out of his own pocket. This was his high point of popularity with engineering. Now when you walked into the office, everyone constantly had the Bose headphones on. I think this also hurt collaboration. I know I am personally less likely to try to start a conversation with somebody with headphones on.

Anyways, that was our failed experiment. It seems like the idea has been picked up more and more in the Valley. A lot of companies are opting for this configuration. I think some do it as a cost cutting measure. You can spend a lot less on furniture with an open floor plan, and you can pack in a lot more people in a given space. Others claim that it is all about encouraging collaboration. Again, my experience is that it will do the opposite. Still others do it because they want to break away from the traditional cubicle environment. To those folks I would say: "function over form." An ugly office full of productive engineers is better than a stylish office full of miserable engineers.

Minimalism? Programming? Huh?

Last week, I was at the Strange Loop conference, and it was a great conference. It inspired one blog post that managed to piss some Clojure people off. So here is another. This time I'm going after the closing keynote, Alex Payne. Here are his slides, including detailed notes.

Strange Loop 2009 Keynote: Minimalism in Computing

View more presentations from Alex Payne.

So where to begin... Even though I am going to harsh on this, I will give it very high marks. Why? Because it was thought provoking. If something makes you think long and hard about how and why you do your job, that is a good thing. I worry about people who do not question their assumptions on a regular basis.
Ok, so now on to what you've been waiting for -- the harshness. I hate software engineering analogies. Well, not all, but most. I hate when people try to romanticize about what they do by making a far-fetched analogy to something more glamorous. If you're a programmer, you're not a rock star. You're not an artist. You're not a musician. You're not even a scientist. Sorry. Despite having the word "architect" in my official job title, I would add that you're not an architect either.

You're a programmer. At best, you're an engineer, and that is really stretching it at times. So obviously if I find it ridiculous to compare programming to things like creating art or making music, it seems even more ridiculous to compare the output of programming to art or music.

That being said, I enjoy programming. It is not just a job. I also take pride in my work. I like to show off my code as much as the next guy.

From this perspective, I can definitely appreciate Alex's thoughts on minimalism. Or perhaps more generally, associating some subjective qualities with programming. If an analogy to art or construction or whatever helps one express those subjective qualities, then fine. Just don't forget who you are, and maybe even try to find some pride in just that.

Now I should really end this post now, on kind of a feel good note. But I won't. Instead, I want to touch on some of the *ahem* deeper thoughts that Alex's talk provoked. I was reminded of a paper I read several years ago that tried to explain two obviously overly general (and probably not politically correct) things: why Americans are good at software but not cars, while the Japanese are good at cars, but not software. I tried to find a link to this paper, and was quite sure that I had saved it in Delicious. However I could not find, and I'm convinced that Delicious lost it. Maybe it fell out of cache...

Anyways, the crux of the argument is that Americans are good at getting something done quickly, even though the quality may be poor. So we are good innovators, but terrible craftsmen. This is a good fit for software, but obviously not cars.

If you accept this idea, and it has its merits, then in a world of low quality, rapidly written code, is there any room for subjective qualities? Why should you take any pride in your work at all if its value is directly proportional to how quickly you can produce it, throw it away, and move on to something else? To some degree, doesn't the software world's affection with agile development codify these almost nihilistic ideals? Perhaps the propensity to try and compare programming to art is simply an act of desperation caused by the realization that you'd be better off giving up on good design and instead you should invest in duct-tape.

Anyways, this is a close-to-home topic for me. My day job often involves telling developers the "right" way to do something, and the more unsavory flip-side of this, telling developers when they've done things the "wrong" way. You don't make a lot of friends, but I'm an INTJ so it works for me. Every once in awhile, I work with a programmer who writes beautiful code. Yeah, I said it. Any such programmer knows that they write good code, even if they don't like to talk about it. When you write good code, you constantly see bad code around you, and you can't help but get a big ego. I always compliment the good code, and it always surprises the arrogant pricks. They think I'm blown away and have never seen such good code. In reality, I just feel sorry for them.

Social Technology Fail

This is the kind of posting that needs a disclaimer. I'm going to talk a little about recent changes at Facebook and Twitter, but strictly from a technology perspective. It goes without saying that I have no idea what I'm talking about. I am fortunate enough to be acquaintances with several engineers at both companies, and I have a college classmate (and fellow Pageboy) who seems to be a pretty important dude at Facebook, but I have no extra knowledge of these companies' technology than anybody else. So just to repeat: I have no idea what I'm talking about. You should really stop reading.

Since you are still reading, then I will assume that you too enjoy being an armchair architect. Since my day job is as an architect at eBay, I tell myself that exercises such as this make me better at my job. Heh heh. Let's start with Facebook.

For several months now, I've noticed an interesting phenomenon at Facebook. My news feed would often have big gaps in it. I have about 200 friends on Facebook, and I'd say that around 70% of these friends are active, and probably 20-25% are very active Facebook users. So at any time I could look at my feed, and there would be dozens of posts per hour. However, if I scrolled back around 3-4 hours, I would usually find a gap of say 4-6 hours of no posts. The first time I ever noticed this, it was in the morning. So I thought that this gap must have been normal -- people were asleep. Indeed, most of my friends are in the United States. However, I started noticing this more and more often, and not always in the morning. It could be the middle of the day or late at night, and I would still see the same thing: big gaps. So what was going on?

Well here's where the "I don't know what I'm talking about" becomes important. Facebook has been very happy to talk about their architecture, so that has given me speculation ammo. It is well known that Facebook has probably the biggest memcached installation in the world, with many terabytes of RAM dedicated to caching. Facebook has written about how they have even used memcached as a way to synchronize databases. It sure sounds a lot like memcached has evolved into something of a write-through cache. When you post something to Facebook, the web application that you interact with only sends your post to the cache.

Now obviously reads are coming from cache, that's usually the primary use case for memcached. Now I don't know if the web app can read from either memcached and a data store (either a MySQL DB, or maybe Cassandra?) or if Facebook has gone for transparency here too, and augmented memcached to have read-through cache semantics as well. Here's where I am going to speculate wildly. If you sent all your writes to a cache, would you ever try to read from anything other than the cache? I mean, it would be nice to only be aware of the cache -- both from a code complexity perspective and from a performance perspective as well. It sure seems like this is the route that Facebook has taken. The problem is that not all of your data can fit in cache, even when your cache is multiple terabytes in size. Even if your cache was highly normalized data (which would be an interesting setup, to say the least) a huge site like Facebook is not going to squeeze all of their data into RAM. So if your "system of record" is something that cannot fit all of your data... inevitably some data will be effectively "lost." News feed gaps anyone?

Maybe this would just be another useless musing -- an oddity that I noticed that maybe few other people would notice, along with a harebrained explanation. However, just this week Facebook got a lot of attention for their latest "redesign" of their home application. Now we have the News Feed vs. the Live Feed. The News Feed is supposed to be the most relevant posts, i.e. incomplete by design. Now again, if your app can only access cache, and you can't store all of your data in cache, what do you do? Try to put the most "relevant" data in cache, i.e. pick the best data to keep in there. Hence the new News Feed. The fact that a lot of users have complained about this isn't that big of a deal. When you have a very popular application, any changes you make are going to upset a lot of people. However, you have to wonder if this time they are making a change not because they think it improves their product and will benefit users overall, but if instead it is a consequence of technology decisions. Insert cart before horse reference here...

Facebook has a great (and well deserved) reputation in the technology world. I'm probably nuts for calling them out. A much easier target for criticism is Twitter. I was lucky enough to be part of their beta for lists. Now lists are a great idea, in my opinion. Lots of people have written about this. However, the implementation has been lacking to say the least. Here is a very typical attempt to use this feature, as seen through the eyes of Firebug:

It took my five attempts to add a user to a list. Like I said, this has been very typical in my experience. I've probably added 100+ users to lists, so I've got the data points to back up my statement. What the hell is going on? Let's look at one of these errors:

Ah, a 503 Service Unavailable response... So it's a temporary problem. In fact look at the response body:

I love the HTML tab in Firebug... So this is the classic fail whale response. However, I'm only getting this on list requests. Well, at the very least I'm only consistently getting this on list requests. If the main Twitter site was giving users the fail whale at an 80% clip... In this case, I can't say exactly what is going. I could try to make something up (experiments with non-relational database?)
However, this is much more disturbing to me than what's going on at Facebook. I don't get how you can release a feature, even in beta, that is this buggy. Since its release, Twitter has reported a jump in errors. I will speculate and say that this is related to lists. It would not be surprising for a feature having this many errors to spill over and affect other features. If your app server is taking 6-10 seconds to send back (error) responses, then your app server is going to be able to handle a lot less requests overall. So not only is this feature buggy, but maybe it is making the whole site buggier.
Now, I know what we (eBay) would do if this was happening: We'd wire-off the feature, i.e. disable it until we had fixed what was going wrong. Twitter on the other hand...

Huh? You've got a very buggy feature, so you're going to roll it out to more users? This just boggles my mind. I cannot come up with a rationale for something like this. I guess we can assume that Twitter has the problem figured out -- they just haven't been able to release the fix for whatever reason. Even if that was the case, shouldn't you roll out the fix and make sure that it works and nothing else pops up before increasing usage? Like I said, I just can't figure this one out...

The Age Question

I've avoided joining the echo chamber on the question of age and entrepreneurs, but then Marc Andressen wrote something really interesting about it. One of the important "scientific" conclusions he sites is that in some disciplines (math, physics) people peak in their late 20's to early 30's, but in others (writing, history) they peak much later, in their 40's or even 50's. So then the question becomes are entrepreneurs more like mathematicians or historians?

Of course most people would say that entrepreneurs are more like mathematicians, so the younger the better. I will argue that entrepreneurs are nothing like mathematicians and physicists, but that does not mean that they are not more like those types than they are like historians and writers.

First some disclaimers. I have a degree in mathematics, and like to humor myself by considering myself a mathematician at my core. Truth is that I am a software engineer, a very different discipline, and I cannot deny this.

Ok, first off entrepreneurs are completely full of themselves for trying to enter this kind of comparison. By even making these analogies, you are equating the creativity and skill of scientists and scholars with the ... attributes ... of entrepreneurs. This is ridiculous. It is like trying to compare the skills of a microsurgeon with the skills of a race horse. It just doesn't make any sense.

Entrepreneurs definitely have creativity, intelligence, and skill, but it is not of the same species as scientists and scholars. Entrepreneurs have any measurement of their prowess: money. If their ideas make money, they are good ideas. If they do not make money, they are not good ideas. It's painfully simple.

Scientists and scholars are held to very different standards. Now it is true that scientific endeavor is open to experimentation and verification, a kind of litmus test similar to the success or failure of an entrepreneur's business. But a lot of the really great scientists have theories that are so theoretical and advanced that they cannot just be easily verified in a lab experiment. Similarly mathematicians must write proofs of their theories. These proofs must judged both objectively and subjectively by peers. Of course scholars have no "out" in terms of being proven right or wrong. Thus they all share something similar: their success is measured in the near term by the scrutiny of the most brilliant minds in the world.

This is a kind of scrutiny that no entrepreneur can experience. If the top two dozen computer scientists all thought that Google had a great search engine eight years ago, it would not have meant anything for Google. If those same elite scientists decided Google was crap, again it meant nothing.

The other flip side to the world of science and scholarship is that the very peers who decided your success or failure, are often wrong. That's why there are numerous examples of scientists whose work is not appreciated until after their death.

I could easily make arguments about how scientists and scholars have to be so much "smarter" than entrepreneurs, but I think just looking at the criteria for success is sufficient.

Again, given my background in mathematics, I have a little insight into why mathematicians peak in their late 20's or early 30's. The obvious reason is that solving highly theoretical problems requires tremendous time and submersion into you work. This is true, but it's only part of the reason. The other reason is that to solve those kind of problems, you need to have very little else to think about. I'm not talking about things outside of your work and study, I'm talking about things inside it.

As you progress in any field, you absorb more and more knowledge. This knowledge is tapped in some way, even if subconsciously, whenever you set out to solve a new problem. Knowing less, and thus have more focussed concentration on what you do know, is exactly the kind of thing needed to solve certain kinds of problems. Those kinds of problems are much more common in mathematics and physics than in history and literature.

So even if I humor the self-loving entrepreneurs out there and compare them to scientists and scholars, I cannot really say who they are more similar to. I do think that engineers play a significant role in the kind of technological endeavors common to entrepreneurs. As an engineer who was once a mathematician, I can say with confidence that engineers are much more like writers and historians than like mathematicians and physicists. Extra knowledge and experience definitely help an engineer to be more creative and productive. Computer scientists may be more like physicists. I wouldn't know, as I am an engineer, not a computer scientist.