very powerful economic forces drive centralization of a successful decentralized system. . .
the fundamental problem is that decentralized systems inherently provide users a worse experience than centralized systems along the axes that the vast majority of users care about.
Pointer from Tyler Cowen.
The argument over whether software should be centralized or decentralized is analogous to the argument about command vs. a market. In Specialization and Trade, I describe two forms of coordination, or resource allocation. A command system is used within a firm. A price system is used in the market.
I discovered during my business career that software does not evolve independently of the context in which it is created. I used to say that every organization gets the information system it deserves. Tightly-run organizations end up with very reliable systems. More free-flowing organizations end up with very fragmented systems.
People’s needs differ from and conflict with one another. In a command system, a central planner determines which needs will be met. In a market system, the price and profit system directs entrepreneurs to which needs will be met.
A command system is fine if the pattern of needs is given, or if you have enough power over people to treat their needs as given. A central planner can seek to optimize to meet a given set of needs. But a market system works better at discovering needs.
The original communication network–the telephone system–was centralized. That is because switches were expensive relative to bandwidth. But as computers took over switching, the cost of switching plummeted, obeying Moore’s Law. This opened the way for the Internet to take over communications around the turn of the 21st century.
When the Web first arrived, people did not know how it was going to be used. The challenge was one of discovering needs, and decentralization was most appropriate.
Eventually, some needs coalesced, and we started to see well-worn paths through the Internet jungle. So there emerged big, centralized systems, such as caching servers and search engines.
Amazon, Google, Facebook, and Apple are able to take people’s needs as given. They try to optimize to meet those needs.
One of them could falter if and only if it gets caught flat-footed by a new service that has discovered needs that its customers have that are not being met.
If you don’t know exactly what your software will need to do, then a decentralized architecture might make sense. But once you find a clear pattern of usage, you will want to optimize the software for that pattern, and one can predict that the architecture will evolve in a centralized direction.
You’re missing a stage. The stages are:
1) Discovery. Decentralized systems are optimal for discovering unmet needs.
2) Maturity. Needs are known, and centralized systems are optimal for meeting those needs efficiently.
3) Obsolescence. There is a centralized system that is extremely efficient at meeting the needs of the past, which increasingly diverge from the needs of the present. The system is large and expensive, many things remain dependent on it, and it was not designed to scale down gracefully.
Nay, ’tis essential to this blest existence
To keep itself within the will divine,
Whereby our very wishes are made one;
So that, as we are station above station
Throughout this realm, to all the realm ’tis pleasing,
As to the King, who makes his will our will.
– Dante
While there is an element of the analogy to the market’s discovery that you draw in tech, the consistent historical pattern is the other way around: an early innovator like DEC creates a new market, the minicomputer that presaged the PC or Apple with the full-screen smartphone, around a product they design and manufacture mostly themselves, and then once the market is proven, a bunch of suppliers dive in and make it much more decentralized.
Two good examples would be Microsoft and Intel, who formed the Wintel duopoly for two decades, but completely missed the mobile transition a decade ago and were replaced by a host of smaller suppliers. Intel’s vertically integrated model of designing and manufacturing their own chips was largely replaced by a panoply of ARM, Samsung, TSMC, Qualcomm, and so on, while Microsoft’s Windows OS is a footnote compared to Android, which is built by google, Samsung, Qualcomm, and a host of random companies and people that contribute to the open source linux kernel at its base. This decentralized end game is much better at mixing and matching components to create a variety of products to match consumers’ widely differing desires.
Similarly, online software was centralized because internet connections were slow and flaky, but decentralized tech like bitcoin or bit torrent has become popular with faster internet connections. To me, this common pattern suggests that the current Big Tech giants are already doomed, as close observers see the same signs of the rot that set in at IBM, SGI, Yahoo, or Microsoft long before.
Let me end by pointing out that while economic forces shape a lot of what happens, giant successes like Apple are sui generis and driven by the unique talents of a founder who can even somewhat upend those forces, such as the joke that the Apple org chart was Jobs as the sun that everybody reported to, which nobody else has been able to replicate with much success.
From the article:
I spent nearly two decades building and operating in production the LOCKSS system, a small-ish system that was intended, but never quite managed, to be completely decentralized. I agree with Marlinspike’s conclusion…. It is always comforting to find someone coming to the same conclusion via a completely different route
This is called confirmation bias.
many of the techniques routinely used by centralized systems to improve the user experience, such as A/B testing, are difficult if not impossible to apply to decentralized systems
He’s talking about centralized/decentralized systems, not markets. In a decentralized market you get inherent A/B testing between different competitors.
Privacy. … real data protection requires end-to-end encryption, but metadata protection requires innovation. Both will happen faster in centralized systems because they can change faster
If centralized means bigger, then why would they change faster? And I think innovation would also happen faster in several smaller firms than in one big centralized firm. He seems to be trying to apply attributes of centralized systems to centralized markets….
Censorship resistance
I think he makes a better argument for censorship resistance, but I still see it more as a political problem than a technology problem.
Availability
Here he goes back to systems, without considering markets. Considering the recent outages, I would guess a more decentralized market would give users better availability. Even if average availability of one big centralized company is better than average availability of several smaller companies, one outage doesn’t bring down the services of the whole market for service X.
Control
I agree with him that what happened with XMPP is very exemplifying of the limitations of trying to deploy a common protocol for interoperability. But that is interoperability. What I think most users would understand as control is the ability of saying: “I’m done with Amazon/Googlemaps/Youtube, screw them, I’m switching to their competitor”. ¿How is centralization better at giving users more control?
Discovery is the core problem.
If the goal is known, high profile & measurable (zero covid cases in Wuhan within 60 days), then a centralized system can in fact get there faster.
If the goals are unknown, diffuse, and difficult to quantify (just about everything else in an evolving consumer economy), then decentralized exploration is required.
But note that military victory is known, high profile, and measurable. This has extreme implications for our relationship with China.
Arnold, I think there is something fundamentally wrong with your analogy. Software is a tool on which the authority relies to give instructions to execute actions needed for a specific outcome. Most authorities still rely on other tools to give those instructions (the most simple example is how you order a bank to debit your checking account to pay the seller of what you bought –today we can use some software but we can still write a check). The discussion about resource allocation is about authority: the one each of us may have to determine how we earn and spend income.