Distributed Development

First, I’d like to apologize for not writing more for this blog. Actually, I’ve been writing quite a lot recently, but it’s mainly been in Ocaml, not English. More on that later.

But one thing I’d like to do is lay out the design for an application that I’d like to see someone do- although it’s not something I’d be really good at. The topic came up of how you would do a distributed development tool. Yes, all you really need (as demonstrated by a large number of open source projects) is a mailing list and a remote version control repository. But why aren’t all software projects run this way? Because there is a huge cost to having development distributed to that extent in terms of efficiency, and thus a huge advantage to having all the members of a team in a single office. A cost which open source projects are willing to accept- but which many businesses are not. Thus, while open source seems to be just fine with email and subversion, business wants all of it’s programmers under one roof, generally. But does this have to be the case? I don’t think so, but to understand why, we first need to understand why the common office is still so popular for programming teams.

Why is the office still popular? The answer is not generally power-tripping on the part of the management. While there is some of that, too many places that are willing to be very flexible with their programmers on just about everything else- from dress code to what hours are worked etc.- are completely inflexible on having a shared office. This includes managers who still think of themselves as hackers, like Joel Spolsky or Paul Graham. It’s not just the Bill Lumberghs of the world who insist on this.

The reason for this, I think, communication. Programming, as it’s actually practiced, is a very communication-heavy job. Programmers, in the course of their work, are constantly having to communicate and exchange information. And I’m not just talking about the official book a conference room and print out an agenda style meetings, and I’m also not talking about just exchanging bits of (hopefully) working code. I’m talking about informal discussions about various issues that arise naturally from programming.

If you work in a office as a programming, watch over the next week how many informal little conversations happen around you- not just the ones you participate in, but the ones you’re aware of. That conversation in the hall you happen to walk past. The friendly argument over lunch the next table. The person wandering in and asking a question two cubicles over. How often do you go out for beers after work, and to chew over some shop talk, with coworkers? And so on. How many times do you get sucked into these conversations, meetings in all but name, that otherwise you wouldn’t have even known about? “Oh, hey- you might be interested in this. Dave, Mike, and I were just discussing…”

The examples I’m listing above are all, in some broad sense, informal communication. Emails, memos, organized meetings, etc. are all, in a broad sense, formal communication. The difference is that in formal communication you need to think about who is communicated to- and those not explicitly on the communication list don’t hear anything at all.

The thing is, as humans, we’re really good at informal communication, but kind of spotty at formal communication. Think about it- we evolved, and until very recently lived, as small hunter-gatherer tribes. Everyone more or less in one spot, and most communication was informal. Formal communication- tribal councils, etc.- were either to officially announce what everyone had already decided on already, or were held to communicate with other groups where close cooperation wasn’t necessary. It doesn’t matter how the next tribe over is going to hunt antelope, while we’re all here let’s decide how we’re going to hunt antelope. Like it or not, this is how humans work together. The advantage localized (office-centric) development has is that it can leverage this informal, unstructured communication- while to date distributed development has had to rely solely on formal, structured communication.

Even if it’s as low a bar to entry as email, you still have to remember to send the email. Which is the problem with the current maillist/cvs approach to distributed development. If you follow (or participate in) an open source project, how many times do you see a post that begins “a bunch of us were at the same conference/in the same irc chat, and talking about things, we decided …” followed by people responding “I wasn’t at that conference/on that irc chat, why wasn’t I consulted? I hate the fact that this decision was made without my input.” Even if it’s not intentional, this slipping into informal communication modes creates the perception of an in-clique and an out-group- with the in-clique naturally wielding more power, as they make decisions via informal communication, and then present them fiat-accompli by the formal means of communication. Since the out-group doesn’t have any sort of similar organization (which is why they’re a group and not a clique), they’re not going to be able to effectively respond.

This is not even impugning motives to the members of the in-clique- they might not even be intending for things to work this way. They may not mean to be, or realize they are, excluding people who weren’t there from the decision making process, but it sure as heck feels like that is what is happening to the people being excluded.

As a side comment, this is why I hate doors as a programmer, Joel not withstanding. I’ve programmed in an office with a door, I’ve programmed in a cubical, I’ve programmed in a communal room, and I’ve programmed on a trading floor. My least favorite was the office of the door. Because of the increased isolation, this discourages informal means of communication. The biggest advantage of having an office to yourself with a door (that you can’t hear the conversation the guy next to you is having) is also the biggest disadvantage- especially considering, since you’re all in the same office anyways, more official means of communication don’t have to be developed. Now, not only do you have an in-clique and an out-group, you also don’t have a common official means of communication. Being able to not overhear the conversation with the guy next to you is what God invented headphones for.

So now that we’ve identified the core advantage of having a common office, is there a way to provide the same, or at least a similar, advantage to a development team scattered over a continent? The answer is, I think, yes. The core idea is that the primary information carrier for informal means of communication is voice. Go back and consider all of the examples I listed of informal communication- they all depended upon the recipient being within the sound of someone’s voice.

Even better, voice communication tends to be very tolerant of low-fidelity reproduction, allowing for cheap compression schemes that get incredible compression rates at the cost of some fidelity. For example, while not exactly “pin-drop” clear, G.728 compression is more than good enough for phone communications, and only requires 2 kbyte/sec (16 kbit/sec) communication speeds. So long as you have something better than a 14.4 kbaud modem, you’re good to go. And it’s simple enough that an 8-bit CPU can do- and vaguely modern CPU won’t even notice the load. This isn’t the case with video- and, for the most part, video isn’t necessary.

So that’s the core of my idea- have a server set up somewhere with good bandwidth (colocs are cheap and common). Every client connects directly to the server, and sends upstream the sound there. The server merges them all together and ships the combined sound stream out- so every client only has to deal with 2 sound streams (one out, one in). This creates a virtual “common room” for the informal communication to happen in.

Combine this with a shared white board and a distributed editor, and the room metaphor becomes even stronger (and the tool more useful). The two most useful things when holding an impromptu programming discussion is a white board (it’s amazing how drawing some circles and squares with arrows between them helps an explanation, even if you’re not doing proper UML), and a monitor/keyboard so you can joint edit code. These don’t have to be complicated- notepad and paint are powerful enough- as you’re not doing “real” (aka official) communication with them. You’re not doing presentations with them, you’re just doing one step above hand waving with them.

This admits to a whole raft of potential elaborations (not all of which have to be in version 1.0). For example, the server could annotate the outgoing sound stream as to where the sound is (mostly) coming from- allowing the client, if it has photos of the various people, to adjust the photos based on who is talking- so the person talking is currently most central and largest, while the people not talking are more towards the periphery and smaller- mimicing, in a sense, the fact that we tend to turn and look at whoever is talking.

This also sidesteps another problem with having all programmers in the same room, that of incidental interruptions. If I have to take a phone call, or my kid runs in having a minor kid-emergency of some sort, that doesn’t interrupt everyone else. I just put the speaker on hold, or simply take it off and put it aside. Since “rooms” are effectively free, they can be created for all sorts of purposes- both dynamically and statically. It’s easy for some small subset of programmers to create one-off virtual conference rooms for discussions that either aren’t public, or just don’t need to be. Gives a whole new meaning to “get a room you two”, doesn’t it?

There is also differing levels of participation. The obvious dichotomy is being able to listen and talk, being able to listen only, or not being able to listen at all. If you’re using Scrum, imagine being able to hold a scrum meeting where only the pigs can talk- the chickens can listen in, but can’t participate (or can only participate if explicitly allowed to). For those of you not familiar with Scrum, the terms come from the old joke about bacon and eggs for breakfast- the chicken is involved, but the pig is committed. Or think of more official meetings where Robert’s Rules of Order can be enforced- if you don’t have the floor, you can’t talk.

It works the other way as well- the individual can also choose their own level of participation- from listening and participating, to just listening, to not listening at all (but possibly interruptible by anyone or some subset of people). This also deals with the problem of “I’m trying to hold a complicated problem all in my head at once- don’t bother me unless it’s important”. If that’s what you’re doing, then you’re not listening, but you may still be interruptible.

Another possibility is being able to record and save conversations. 2 KByte/sec means only 86.4M/day, assuming a 12 hour-day, or 22.5G/year assuming a 260-day work year. Which really isn’t that much disk space by today’s standard. This sounds Orwellian at first blush, but consider for a moment how often you would like to be able to go back and review discussions or meetings you had in the past. The key trick here is indexing the recordings for searching later- adding time-specific keywords would be useful, but some sort of voice recognition would be even more useful.

Actually, I think I’m going to take back something I said earlier- you might want a better editor than notepad. Distributed pair programming becomes a distinct possibility in this system. Of course, this leads you directly into the editor religious wars, which make the Crusades look civilized (and well organized) by comparison. But at least you’ve solved the one-keyboard “problem”.

The question becomes why don’t I write up this program? Two problems. First is that I’ve got more than enough projects going on already, I don’t need another one. Second is that it’s primarily a UI problem. There are a couple of interesting points kicking around, but it’s mainly UI. My preferred development environment is vi and a unix command line- you don’t want me doing UI work, nor am I particularly interested in doing UI work.

But I would like this program to exist, even if I don’t write it. There are problems with a distributed development group it doesn’t address- for example, notice how I cleverly sidestepped the problems of having developers scattered over more than a couple of time zones. And there is an advantage to having a “work space”, where psychologically just being there means you’re working, not goofing off, and minimizing non-work distractions. I can’t work from home- I won’t do work. And I’ve known more than one kid who couldn’t, or wouldn’t, learn that “I’m bored” doesn’t constitute an emergency.

But being able to join a company without actually having to move to where that company is located would be a huge advantage. But an even bigger advantage is having a conference room with an infinite amount of white board space, a handy terminal, and in a park so I don’t miss a warm sunny day sitting in office, that’s worth real money to me.