One specialist said that as many as five million lines of software code may need to be rewritten before the Web site runs properly.
1. There is zero chance that rewriting five million lines of code is the answer. Either the solution is a lot simpler or there is no solution other than to start over.
My instinct from the outset was that starting over was the right answer. I am not alone.
2. The other day, President Obama said, “No one is madder about the Web site than I am, which means it’s going to get fixed.”
Or, as Michael Palin and John Cleese would put it, Wake up, Polly!
3. In response to the WaPo story, I wrote a letter to the editor, which they published (mine is the third letter on this page). This is not a technical screw-up, and it will not be fixed by technical people. It is an organizational screw-up. And until that is recognized, it probably will get worse. I write,
In my experience, communication failures between technical staff and management reflect an atmosphere of fear and lack of mutual respect.
I call this the suits-geeks divide. I saw it during the financial crisis, when it was evident that many mortgage credit-risk geeks warned of problems at their firms but management went out of the way not to listen. Merrill Lynch and Freddie Mac were particularly well-documented cases.
4. Every year, I have my high school students pair off and present proposals to start a new business. Two questions I always ask are “What are the critical management functions?” and “What would somebody experienced in this business know that you do not know?”
Suppose that President Obama and Secretary Sebelius were in the class, and they proposed starting the world’s largest health insurance brokerage. I would expect them to be able to identify as key management functions: marketing, customer education, insurance company partnership management, pricing and underwriting standards, and operations.
Somebody who had experience with creating a health insurance brokerage business would know that the systems problems are more complicated than just putting up a web site. In the background, the system needs to communicate with the systems at several government agencies and at the insurance companies. That changes it from a simple technical project to a complex, time-consuming, project involving business and technical staff.
You build a complex, mission-critical system through a process of continual negotiations among business units and technical people. You do not treat it as a procurement process. You cannot just write up a spec, put it up for bid, and parcel it out to dozens of contractors.
The development of the computer system probably would fall under operations, but you would want a project executive with a lot of authority to negotiate with all of the business units and to make project decisions. When conflicts arise, the project executive should be able to go straight to the CEO and get them resolved.
The project executive’s main focus is keeping the project’s complexity from getting out of control. The project executive must have the authority to trim features in order to meet deadlines.
You go through a lot of analysis and many painful meetings before anyone writes a line of code. The technical staff have to be able to challenge the business units, because sometimes the business unit asks for something to be done in a really complicated way, when a much simpler solution is available to solve the business problem.
One of the worst things that can happen on a systems project is to find yourself revisiting the business-technical negotiations process after writing a lot of code. If that is what is happening now, this project is in an unbelievable amount of trouble.
5. I suspect that the technical problems are mere symptoms. Probably what is fundamentally messed up in this health insurance brokerage business is the org chart.
6. In business, you need clear lines of authority and accountability. The bureaucratic tendency is to seek the opposite–to blur authority and avoid blame. This is a big challenge in the private sector. However, I think it tends to be even more difficult to overcome in government.
7. For Christmas, someone should give President Obama and Secretary Sebelius a copy of The Mythical Man-month.
[update: good PBS interview with several mythical-man-month allusions]
[update: Clay Shirky’s tweet echoes this theme.
I’ve said it before. I’ll say it again.
The ACA law was NEVER designed to produce a workable healthcare finance system. The real design objective of the ACA law – every single aspect of ACA – was to attempt a system that was/is so glaringly, patently, obviously unworkable as to make the U.S. public believe that the “only” solution is for the U.S. Federal Government to provide health care finance on the “single-payer” European/Canadian model.
Had anyone in Fed Gov just proposed that, prior to ACA and all of it’s now-becoming-apparent problems, the U.S. public would have shouted them out of office.
Now, it’s going to be (very soon) that, “Okay, we’ve tried to do this using private market-oriented approach and they completely screwed it up! Now it’s time for Government to step in and provide ALL real health care finance.”
It might be too late for single payer in the USA. If my calculations are correct single payer would require a tax increase equal to 9% of GDP. I think that much more taxes would require a new tax, like a 9% VAT because if you increase the income tax, avoidance will increase, for one thing more wives will opt to produce for in-family consumption avoiding the taxed economy.
Floccina:
I don’t disagree – with 2 caveats …
1.) I think your tax increase projection of 9% of GDP is conservative.
2.) I suspect the effective costs of ACA will quickly, vastly, and quite apparently exceed even non-conservative estimates of tax increase.
It will be the realization of #2 caveat that will make the #1 caveat appear “attractive” to the U.S. public.
Beautifully said Arnold. Shayne is unfortunately also correct. Sec. Sibelius’s words prove that.
Given how well this is going, why should we think a “single payer” system (done in partnership with 50 states ….) would work any better? The model wouldn’t be the UK NHS, it would be the UK NHS expanded to cover all of Europe.
The rest, well, pretty much what Arnold said…
The model could be the Canadian system instead of the UK system. While Canada’s population is much smaller and somewhat less diverse, it is spread over a very wide area and has many of the same organizational problems that the US has when creating national programs (ours is often called Quebec). Medicare in Canada is organized mostly on a provincial level, not a federal level (which the US states would like). Funding comes from both the province and the federal government (94% from the provinces, currently). Different provinces do cover slightly different things with their Medicare (there’s an absolute minimum standard of what they can cover, but most cover more than that in different areas). Alberta’s been talking on and off about mixing government-funded care with private clinics for those who can afford them, although the act that brought the Medicare program into being specifically prohibits private clinics. It’s not a perfect system, but it is a lot less expensive per patient than the American system and our care is just as good, if occasionally slower.
You know what the response of the rest of the world to the US implementing single-payer healthcare would be? It’s about damn time! Yes, the implementing of such a system in the US would have some unique problems that all the little countries in Europe don’t face, but they’re not the reason your broken system continues to exist.
The reaction from those in the rest-of-the-world who actually know something? “WTF are you doing messing with our safety net/research pot-of-gold???”
“Now, it’s going to be (very soon) that, “Okay, we’ve tried to do this using private market-oriented approach and they completely screwed it up! Now it’s time for Government to step in and provide ALL real health care finance.””
Somebody is having a bad day…. 🙁
$20 bucks says by the end of 2014 we’ll all still be able to buy private health insurance, and healthcare.gov will have enrolled millions of people.
Sadly, I think both will be true. The healthcare site will be up and running with a giant backlog of bugs and a stripped down feature list. And the ACA will be steadily on its way to turning insurance companies into public utilities, effectively taking over health care finance.
josh:
A “bet” is an intriguing idea. But I think I have 2 “surprises” for you …
Surprise #1.) If I were to bet, I’d take your side of the bet. Understand that if what I’m postulating (the government “single payer” eventuality) actually happens, there will be “millions of people” automatically enrolled – whether that is under the auspices of a modified ACA, or under the auspices of a greatly expanded Medicare/Medicaid construct. And, everyone will still be able to buy “private insurance” as well. They’ll basically have to, just as current Medicare recipients have to buy additional “private insurance” (part B, part D, etc.) to cover health care costs excluded from Medicare coverage.
Surprise #2.) I actually consider the “single payer” (government/taxpayer funded) system economically preferable to ACA, in its current incarnation. ACA is an economic albatross, not (necessarily) a political albatross.
ACA, in its current form is economically unworkable and impossible. A “single payer” (government/taxpayer) system on the European/Canadian model is workable, but grossly sub-optimal.
I’ve proposed an alternative that I believe is optimal – see my comment to David Henderson’s post today on EconLib. And John Cochran’s paper linked there.
shane, i would take my side of the bet too that is why i proposed it :). pretty sure arnold wouldn’t take his own bet either, I think he was just venting (understandably so, because the rollout was managed unbelievably poorly both from tech and management side). however perhaps arnold would propose a differently structured bet that would get more clearly at the point he was trying to make about healthcare.gov being a trojan horse that is certain to remove private healthcare. I understand the concern but I’m fairly certain he wouldn’t risk money on it, because it’s extremely low probability.
My whole thing is that a lot of the people complaining aren’t about obamacare aren’t being constructive or caring about the country they’re just trying to inflict political damage for their own self interest. ACA has been passed, and whether anybody would admit it it’s clearly not optimal… even for democrats. I’m a right of center independent. I would agree with david frum in that the republicans made a huge mistake in adopting the ‘hell no’ approach to healthcare changes, because they’ve given themselves very little room to improve it. If they had committed to work with obama from the start we would’ve ended up with something quite different than obamacare. It’s just unfortunate that the GOP has become so anti everything, because it’s getting to the point where I’ll be forced to vote for democrat simply because they are more responsible and collaborative (at the federal level).
Somebody on econlog or in the libertarian blogosphere (makkiw?) recently made the point that few would care about the mandate if it turns out to be mostly dollar neutral. It seems like in some places, particularly well populated urban areas w/lot sof competition this may end up being somewhat true. In the rural or lower income places it’s going to be a lot more medicaid as you state, at least in the current version of ACA. However this is something that can be improved within the context of the mandate (which I don’t think is going away)…. To this point allowing providers to compete across state lines would create competition in all areas and this is something many republicans considered supporting in the ACA before the political decision to go nuclear was undertaken. Likewise I think the exchanges also have features that can be tweaked to facilitate more competition.
I also read your post and I agree that playing with tax incentives is another piece of the puzzle. I actually think all of these things are going to be bigger problems than the website (which as horrible as it’s been rolled out will probably be fixed fairly quickly, given they’ve already fixed enough to enroll hundreds of thousands in the last week or so).
All of these things need to be fixed and they need conservative ideas would make them better. We only need to get conservatives with the smarts (and balls) to work across party lines and not cater to narrow and increasingly irrational points of view. ACA is here to stay.
whoa, Josh. I was not the one who said that Obamacare was a Trojan horse. I don’t think that the system problems are a conspiracy designed to bring on single payer. Never attribute to conspiracy would can be explained by stupidity.
It’s pretty amusing that Josh claims the GOP has not been cooperative. Let me just say, thank God they haven’t! First of all, all politicians, even liberals, are primarily concerned with winning the next election, not with doing the right thing. Note the Democrats now calling for delay on the mandate.
More to the point, though, during the year and a half that Congress was putting together the Frankenstein’s Monster that is ACA, the GOP offered many, many elements THEY THOUGHT would improve it. Every one of them was dismissed. The reason no Republicans could vote for it is because it was Obama/Pelosi/Reid who were intractable on several issues that no Republican could accept.
There are still too many Americans dead-set against single-payer for it to fly here, regardless of how badly ACA damages the various aspects of the health insurance and health care markets. And comparing apples to apples, which single-payer advocates generally resist, US HEALTH CARE (as distinguished from health INSURANCE) is for most Americans the best in the world. It is primarily Medicaid dependents who experience sub-par care. In short order, all non-wealthy Americans would be enjoying a Medicaid-level experience, if single-payer were to ever pass.
Obamacare, as it meets its inevitable demise over the next three years, will prove to be the high-water mark of the Progressive project in universal health insurance. When Obamacare proves unworkable, a great deal more freedom will gradually be introduced, such as private health insurance exchanges that offer employees policies from out-of-state providers, concierge medicine from doctors who refuse to act as indentured servants, transparent pricing of health care services, and rational insurance policies that leave the first several thousand dollars of medical expenses to the discretion of the consumer.
As always, competition is a powerful net positive in restraining costs and improving consumer choice and quality. (It’s remarkable to me that health care/insurance is the one area, other than perhaps education, where the clear victory of Mises/Hayek/Friedmas, et al finds a befuddling exception. Markets can’t function without the information provided by prices!) When that becomes evident, the US will lead a worldwide trend to reduce the role of the state in health insurance. This may now be more hope than expectation, but I believe fiscal reality will bring things to such a pass. And I don’t think that trend will wait 10 years.
As for the conspiracy idea, I’m with Arnold–most of those who advocate state-provided health insurance really do think it would be better. Obamacare is simply the best they could cobble together in this country, where so many individuals still value their liberty, and many still appreciate that economic liberty cannot be separated from political liberty.
The road to hell, and the road to socialized medicine, are both paved with good intentions.
Is this project capable of failure? Is there any point where someone will say: “no, we’ve spent too much money and time on this.” I don’t think there’s a limit, and it’s too important to too many people to cancel.
I think we’re witnessing a premature birth, and after several months or years and billions of dollars, eventually there will be a functional though probably stagnant system.
This may well be true. But it’s peanuts compared to the cost of the ACA itself.
With the current executive leadership, no. Which is a problem.
With future leadership, yes. Which could also be a problem.
I agree with Shane Cook that ObamaCare was not designed to succeed. However, it was not supposed to fail at the onset. The plan was to get everybody signed up and using it. Normal (and planned) government inefficiencies would cause shortages of clinicians, overuse of nurse practitioners and physician assistants, longer wait times to see specialists or get elective surgery, etc. The left-wing politicians would work with their mass media partners and blame all the problems and failures on greedy clinicians, hospitals, labs, drug companies, etc. The outraged public then would support complete socialization of health care.
The current disaster will make it harder for left-wingers to blame future ObamaCare problems on the greed of healthcare providers.
From the PBS interview:
> We have actually analyzed some of the code, and seen that while there were some fixes, they actually created some new problems, some of which involve security. So, they’re going to be — this is a long process. And, as you just heard, there’s going to be some rearchitecting that has to happen to make all of this hook together.
The security problems are going to be the next elephant in the room for the White House. It’s very hard to do security well, and that’s when it’s all under one roof. You could not pay me enough money to put my personal information into that system.
I haven’t figured out the right angle yet, but there is a helluva story in the comparison between Obama’s campaign tech team, approach, and results versus the Obamacare tech team, approach, and results.
The first speak to some shockingly irresponsible development practices. The process of setting up an application so you can confidently make changes is well-known. The reasons for not using them are also well-known. As are the ramifications of not doing so.
Case in point.
I’ve been wondering about the security around the site as well. Especially after seeing some of the terrible development practices that almost have to be underlying the quotes that are hitting the media.
http://arstechnica.com/information-technology/2013/10/healthcare-gov-deferred-final-security-check-could-leak-personal-data/
I have tried the exchange a couple of times. One time I could not get in but I was able to get in a couple of other times and signed up successfully. But how many large scale projects start perfectly initially? The latest Boeing jet, the Dreamliner, delivered year’s late and then had battery problems. I thin the first couple NASA rockets blew up and we went to the moon. These things work out.
Bob- Did Boeingactually start delivering the Dreamliner before it could get off the ground? They actually made the decision to delay, yes, even with problems. The administration could have made the decision to delay and yes there probably would have been problems even then, but the system probably would have worked a lot better than it did on Oct 1. The comparison is that Boeing would deliver the Dreamliner on time to a client, meanwhile the engineers are going, “What are you doing? We don’t even know if it will fly! We have only tested the engines and they work.”
Boeing’s FIA satellite project never “worked out,” nor was it really designed to work effectively at all, except in the sense that it effectively wooed government people:
http://www.nytimes.com/2007/11/11/washington/11satellite.html?pagewanted=all&_r=0
Even if the website gets fixed, I imagine there will be plenty more issues to come, regardless of what the House does- intended or not.
“You build a complex, mission-critical system through a process of continual negotiations among business units and technical people. You do not treat it as a procurement process. You cannot just write up a spec, put it up for bid, and parcel it out to dozens of contractors.”
This is the core of the Agile software development methodology, and as I’ve said on my blog, they clearly didn’t use that style in this project.
I also have issue with the “No one is madder about the Web site than I am, which means it’s going to get fixed.” from Obama. It never helps to create a hostile relationship between executives and the project team. Especially a publicly hostile relationship. Does he not realize the effect morale has on a project? Evidently not.
As far as the “5 million lines of code” goes, the whole “lines of code” metric is meaningless and is only used by PR-types who don’t understand programming. I’ve refactored 5000 lines of code in an afternoon and come up with roughly the same “amount” of code, just organized more efficiently. I’ve struggled for days on a small fraction of that amount until I found a good way of handling the problem.
The Lines of Code metric is also used by Labor Theory of Value true believers* who may or may not “understand programming”.
*slaves of some defunct economist, usually Karl Marx.
I had to laugh, listening to the Congressional hearings, when the executive said, “we would have rather had a couple of months” to test the software. If you weren’t testing from the very first, then you weren’t using industry standard best practices.
Back in the 70s and 80s we first wrote the code then tested it at the end. This led to gargantuan failures. These days, you start testing the minute there is code to test and you don’t stop until the product is end-of-lifed. Big-time outfits, like Amazon or Netflix, actually test their code while it’s running in production. Netflix, for example, uses it’s “Simian Army” to randomly kill off running nodes to make sure that their failure-handling systems can tolerate equipment failures and crashes.
I wish someone would research and write a book on this whole development project, it would provide an excellent “how not to do it” case study for future project managers.
I’m not in a position to write a book, but the quotes in the media outline project problems I’ve seen way too many times. Just not all at once. I wrote a few articles about it on http://mattosbun.com if you’re interested.
Testing is something I really focused on because the comment you have above, along with things like not being able to fix problems without breaking other things speaks to a great failure in testing strategy. Unit testing would have saved a lot of pain. Test Driven Development even more. Periodic functionality reviews by stakeholders (We call them “Sprint Reviews”) would shake out a ton of issues before they become problems and scheduled User Acceptance Testing (Like you said- NOT done just at the end of the project) would verify whether or not the site does what it needs to while problems can be easily managed.
Clearly, none of these things happened. Very serious failings on the parts of both the project management staff and the development management.
I agree. Iterative development and testing is the way to go. The UI launched on Oct 1 had so many issues that no user could have possibly signed off on it. You couldn’t even shop before registering. Who in their right mind would open a “market” where you could browse before signing up.
It would have been much simpler to mandate that employers can no longer provide health insurance and standardize insurance regulations so companies could compete nationwide. Companies would advertise and innovate to attract customers. An instant marketplace! The insurance companies already have on-line portals to shop. Then health insurance would no longer be tied to employment (just like auto insurance and home owners insurance) as it should be. The relationship should be between you and your provider. Adding a huge bureaucratic middleman is silly.
Beautifully said Arnold. Shayne is unfortunately also correct. Sec. Sibelius’s words prove that.
From the review of “The Mythical Man-month” at Amazon, one of its key principles…
“When designing a new kind of system, a team should factor in the fact that they will have to throw away the first system that is built since this first system will teach them how to build the system. The system will then be completely redesigned using the newly acquired insights during building of the first system…”
Oooops!
Love the love for Brooks’ “Mythical Man-Month” – should be required reading for any manager of any significant engineering project, whether SW or HW or both. Sounds to me like the Obaminable care website used the classic “waterfall” development process, where you’re presumed to have developed all of the detailed requirements before you write a line of code, and you assume that you can deliver full functionality on the first build (plus one “correction of deficiencies” build, which fixes the relatively few bugs you find in testing, which is only done on the full product); however, waterfall development has been thoroughly discredited everywhere except government procurement for at least 20 years. For a new system, the day when you know all of the requirements is the day that the last implementation is retired.
The people who draft laws like the ACA are predominantly congressional staffers, academics, and liberals.
This does not make them bad people, but it does mean they are out of the loop on actual software and business development, They are also innocent of the willingness to cheat and cut corners in the insurance industry, the labor market, and the public in general.
I was intrigued by the comment that the Canadian Medicare is funded mainly by the provinces. Obviously there are no provinces in Canada with the political culture of our Deep South or Mountain West. If American states will not accept an expansion of Medicaid that costs them next to nothing for three years, they sure aren’t going to accept a single payer system that would cost a minimum of 14% of payroll immediately (based on the Vermont single payer system in proposal stage now)
a 3rd possibility: 24/7 driven news media are creating hysteria with the most dramatic quotes they can get, and supposedly smart people are buying into it…2 or 3 months from now, problems are small, i hope you have the grace to admit that your fell for the hype
I will gladly admit that I fell for the melodrama, if that turns out to be the case, but I would not blame it on the media. I think that anyone who looks at other people’s code usually hates it. So the melodrama could be coming from that. Or, it could be that there are real problems.