From a Washington Post article on the software glitch that briefly grounded United Airlines:
“When the two airlines decided to merge, they had a choice to make: Would they take the Continental system, or would they take the United system?” Seaney said. “They decided to go with the Continental system and basically dumped Travelport.”
Flipping one reservation system into a notably different system is complicated and problematic, and it has led to headaches for United.
“There wasn’t a lot of commonality, and there were a lot of glitches along the way,” Seaney said. “They had issues with reservations, with elite status, all their awards programs had issues.”
I feel United’s pain. About 17 years ago, my homefair.com web site (which I now have had nothing to do with for 15 years) bought a related web site. We had very similar business models, so our software did similar tasks, but theirs was written in Perl and ours was written in Java. Rewriting their software to work with ours cost more than building both of our individual systems. Still, I think it was worth it.
I am a huge believer in rewriting software. In particular:
–As soon as you finish the first working version of an application, you should rewrite it from scratch. The rewrite will go quickly and will involve a much better design.
–Letting any software application go more than two years without a total rewrite will create maintenance headaches. You will have accumulated patches that turn the code into a mess. Your developers will not remember enough to be able to discern what sorts of changes could blow up the system.
The Perl code that we inherited was not only hard to follow (all Perl code is hard to follow–that’s why I hate it) but had undergone continual patching without rewrites. Most corporate software evolves that way, and sooner or later that ends up costing a lot.
Related: Zeynep Tufekci writes,
In the nineties, I paid for parts of my college education by making such old software work on newer machines. Sometimes, I was handed a database, and some executable (compiled) code that nobody had the source code for. The mystery code did some things to the database. Now more things needed to be done. The sane solution would have been to port the whole system to newer machines, fully, with new source code. But the company neither had the money nor the time to fix it like that, once and for all. So I wrote more code that intervened between the old programs and the old database, and added some options that the management wanted. It was a lousy fix. It wouldn’t work for the next thing that needed to be done, either, but they would probably hire one more person to write another layer of connecting code. But it was cheap (for them). And it worked (for the moment).
It amazes me that anyone a) would try to write a large software system in Perl, even back in the wild-west days of the dot-com bubble, and b) could actually succeed in any meaningful sense of the word. It’s very good at doing what it was designed for: Writing small, self-contained scripts of under a thousand lines or so. But why anyone would want to undertake any major project without static typing is beyond me.
Although, after typing this, I’m suddenly reminded just how much code you need to write to do anything at all in Java, and I kind of get it. I’m still shocked that they actually produced a working product, and that people manage to do it in Python these days.
Dude, isn’t facebook nominally written PHP.
I don’t mean that as a defense of PHP, just as an example of what you can mange to do if you sink enough costs.
I would be fascinated to see an application of the three axis model to software design. I split my time between the Freedom-Coercion and Civilization-Barbarianism axes, and I’m much more skeptical of rewrites of large applications. I’m skeptical because there’s an awful lot of accumulated corner-case dealing in there, and lessons you have to re-learn in a rewrite. To me, this is why modularity is such a fundamentally important idea when writing software; you can rewrite a part without destroying everything.
Joel Spolsky agrees with your point about accumulated corner cases. He believes that re-writing code from scratch is the single worst strategic mistake that any software company can make.
http://joelonsoftware.com/articles/fog0000000069.html
I have often agreed with Joel in the past, but on this one I disagree. The issue concerns software patches. In Joel’s view, they are solutions that have stood the test of time. In my view, they are land mines waiting for the next maintenance programmer to step on. When you are maintaining code and you are constantly saying, “I don’t know why that’s there, but I am afraid to touch it,” watch out.
I’m curious if Joel has updated his view – the original article looks to be from 2000. Testing and tools (to make rewriting/refactoring safer) have probably come a long way since then?
My data point:
At [large software company where I work], we’re rewriting/migrating stuff all the time; the joke is that every service is either deprecated or not-yet-ready.
“When you are maintaining code and you are constantly saying, ‘I don’t know why that’s there, but I am afraid to touch it,’ watch out.
If we were discussing politics, I would refer you to Chesterton’s Fence (http://www.chesterton.org/taking-a-fence-down/). That’s the intersection I find interesting: is that the same mental pattern, or not? Is it more appropriate in one case than the other, and why?
Chesteron’s fence is important because in real life (unlike in Chesterton’s example) everything is so interconnected that you can’t understand the results of removing the fence. If such a situation arises in software, that means somone has already made a mistake.
Of course, it still happens all the time. But that is a challenge to (carfully!) correct the poor modularisation and get rid of the fence.
I was still at Continental during the merger and had a product (not reservations) that I designed which ended up being chosen over the United system. This kind of knife fight — to see which code survived and absorbed the other — happened for many individual systems. It was a mess. I was lucky. Most of the more innovative systems lost because in the face of impending furloughs no one was willing to take any risk.
In the case of the reservations system (SHARES-A) and with respect to your comments about rewriting old code, I am still amazed that this mainframe stuff from at least the ’70s (if not earlier) has never been rewritten. Continental’s IT department (long before United) went to great lengths to get the overall enterprise rewired so the stranglehold the mainframe had on ongoing development could be broken. EDS (or whatever they call themselves now) was the best at those political knife fights even then, and we never managed it. The amount of cash flow required to keep patching SHARES is incredible. Equally incredible is how difficult any innovative new software service or feature is to build around it.
Most of the visionaries that fought for an end to SHARES are long gone now. I think your comments about rewrites (I’m rewriting my simulation software for my PhD right now, actually) is spot-on. I think it also applies to organizations. The org chart usually needs a rewrite before the political will for a software rewrite can be found.
Yeah, mergers are terrible. The combined system has to accomodate the union of all the data and functionality of the individual systems. You have to map between the two kinds of identifiers all over the place.
I worked on the merger of billing systems when Sirius and XM merged. Those two systems were actually based on the same off-the-shelf billing system, and the decision was to migrate all data from system B into system A, and it was *still* a nightmare due to the customizations each party had made to the system over the years, the variations in how each company did business, etc.
Arnold,
Your rewrite from scratch position is in the foundational document of software process: The 1970 Royce paper that defines the waterfall software process suggests that the ONLY way to do a waterfall project well is to do the full requirements, analysis, design, code, etc. approach, then throw away the result, and start over now that you have enough information to do something intelligent.
This is the single largest overlooked claim in software.
“Letting any software application go more than two years without a total rewrite will create maintenance headaches.”
Eek. Granting for the sake of argument that all software applications need to be rewritten periodically, I think some of them would need the period pushed out to considerably more years than that, in some cases to more than a decade. Look at a reasonably capable renderer like povray, luxrender, or mitsuba. Or, for better-known examples, look at various optimizing compilers, or TeX, or emacs. Many of them are showing their age, and might well have benefited from rewrites every 5-15 years, but I don’t think it’s really feasible to rewrite them every 2 years, and I doubt it would be helpful.
I agree with the idea of constant rewrites for living software, but for larger programs (dozens of engineers), you need to do the rewrites of one modular section at a time.
Also, for larger teams, you are never doing the rewrite with the same team, so it’s easy to overestimate how much better you will do the next time. The mostly new team understands some of what the software does, but is also unfamiliar with a lot.
> Letting any software application go more than two years without a total rewrite will create maintenance headaches.
I have developed a bias toward refactoring applications in place, rather than rewriting them. I have found that programmers love the idea of rewriting as a default position, but that their (our! 😉 optimism masks the true enormity of effort required. As a business proposition, a rewrite is much more costly. For example who will actually *do* the rewrite, and who will train them? Will they be new employees, or existing ones? If new, who will train them? If existing, who maintains the existing application? Indeed, what is to be done with the existing application while the rewrite takes place?
So while it may be true that a ground-up rewrite every couple of years reduces maintenance headaches, I assert it will be at the cost of increasing business-as-a-whole headaches. “There are no solutions, only tradeoffs” etc.
Again, in the interest of full disclosure, I have a book and an online course about refactoring applications, so my bias may be coloring my opinion. It is a hard-won bias, for good or ill.