How Do We Explain the Unreasonable Effectiveness of IT?
Joseph Campbell: As Schopenhauer says, when you look back on your life, it looks as though it were a plot, but when you are into it, it’s a mess: just one surprise after another. Then, later, you see it was perfect. So, I have a theory that if you are on your own path things are going to come to you. Since it’s your own path, and no one has ever been on it before, there’s no precedent, so everything that happens is a surprise and is timely.
Why is the IT industry so darn effective? Just think about these amazing advancements. A little over 30 years ago the Apple Mac went on sale. In 2020 Benedict Evans estimates 80% of adults on earth will have a smartphone. And about at that same time applications were typically monoliths that ran on one computer. Now applications can deploy with the push of a button on cloud native architectures that exploit many thousands of CPUs using datacenter scale operating systems. And software used to be this strange specialized niche only nerds cared about or understood. Now software is in everything and is so ubiquitous it’s becoming nearly invisible. The examples could go on and on and on...and on.
These advances have evolved step-by-step over time, so we don’t even realize the full weight of the transformative changes we’ve experienced. What can account for such astonishingly rapid progress?
Stepping stones.
What the heck do stepping stones have to do with anything? Here’s a clue...do you remember the Connections TV Series by the incredible James Burke?
For an explanation we turn to Ken Stanley, Computer scientist, artificial intelligence researcher, Associate Professor at the University of Central Florida, who wrote a new book Why Greatness Cannot Be Planned: The Myth of the Objective, with a fascinatingly counterintuitive premise:
The greatest achievements become less likely when they are made objectives. The best way to achieve greatness, the truest path to “blue sky” discovery or to fulfill boundless ambition, is to have no objective at all. To achieve our highest goals, we must be willing to abandon them.
The Big Idea
Note: this is a mashup from the book and an interview with Ken.
Almost no prerequisite to any major invention was invented with that invention in mind. The genius of the Wright brothers wasn’t to invent every necessary component for flight from scratch, but to recognize that we were only a stepping stone away from flight given past innovations. Great invention is defined by the realization that the prerequisites are in place, laid before us by predecessors with entirely unrelated ambitions, just waiting to be combined and enhanced. The path of prerequisites to an invention can be thought of as stepping stones.
There’s no systematic way to know how solve ambitious problems. We just don’t know what will lead to what. The future is a fog. An ambitious problem is not deciding what to have for lunch, it’s more like curing cancer or creating an AI.
All discovery requires risk because discovery requires exploration. Looking in new places without knowing exactly what you’ll find, how you’ll get there, or even if there’s anything to find.
Increasingly we are trying to derisk discovery by defining objectives backed by metrics.
Objectives are a way of saying prove to me first that you know where you are going before you go. This attempt to derisk discovery will not work.
Objective based systems cause people to avoid risk because there's a risk of not meeting the objectives. This leads to an aversion to trying anything new.
The idea that an improving score [metric] guarantees that you’re approaching the objective is wrong. It’s perfectly possible that moving closer to the goal actually does not increase the value of the objective function, even if the move brings you closer to the objective. The game of Go is not won by taking the most direct route and life is far more complicated than Go.
Everything can’t be dealt with by some quantifiable measure. We’ve come to believe too much in metrics and accountability. Everything can’t be reduced to some sort of scientific experiment where we can use an empirical result that will cause us to go in the correct direction.
It’s possible to explore a search space intelligently without an objective. Novelty often acts as a stepping stone detector because anything novel is a potential stepping stone to something even more novel. In other words, novelty is a rough shortcut for identifying interestingness. This is called a non-objective system of discovery, like natural evolution and human innovation. Nature produced us humans by not trying to produce us.
The key problem is that the stepping stones that lead to ambitious objectives tend to be pretty strange. That is, they probably aren’t what you would predict if you were thinking only of your objective. For example, vacuum tubes had nothing to do with computers. People like Thomas Edison who were originally interested in vacuum tubes were investigating electricity, not computing. tubes has nothing to do with computers. Stepping stones do not resemble the final product.
Systems that are designed to proliferate stepping stones are the ones that lead to the greatest discoveries. Nature is a stepping-stone collector. Stepping stones are collected not because they may lead to some far off distant objective, but because they are well-adapted in their own right.
In the long run, it’s the accumulation of stepping stones that leads to the greatest innovations. When each small step is a revelation, the chain itself is nothing less than a revolution. So while betting on revolution may be dangerous, over time it does happen. But as with all great processes of discovery, the revolutions are rarely the objectives of the stepping stones that lead to them.
If you want to cure cancer explore the space around cancer. Study cancer not as an objective, but as an area of inquiry. This may lead in the end to cancer being cured, probably by someone not even trying to cure cancer. All these stepping stones will intertwine at some point in the future.
Pick a problem. Work on it. Seeking consensus prevents traveling down interesting stepping stones because people don’t agree on what the most interesting stepping stones are. Disagreement is a sign something interesting is happening. Everyone working on a problem that interests them will end up going somewhere. Don’t force great minds to waste their lives pondering a distant dream, let great minds pursue their own interests. Some will go in a direction that centuries later may lead to some great achievement. Some will go in other directions, but at least they will be marching forward, stepping stone by stepping stone, which is ultimately the only realistic path to the future.
It’s the combination of many minds with many different interests that ultimately plunders the search space in the long run, not any individual objective or person.
IT is Great at Creating, Collecting, and Proliferating Stepping Stones
We are now in a position to understand why the IT industry is so productive: IT is great at making stepping stones.
Like no other industry IT builds on itself, rapaciously integrating any bits that might prove useful. IT is a great novelty driven machine for exploring the technology search space. Many independent interests work in parallel generating artifacts--code, new discoveries, frameworks, platforms, processes, hardware, answers, ideas, content, new paradigms, new business models, etc--that feed forward into a great autocatalytic reaction.
The root of the IT innovation engine is the incredible generosity of people. Yes, there’s often a strong profit motive, but people are inexplicably free with their knowledge and time in the IT industry. Why that would be I’m not sure. Perhaps because IT inherits its schema of operation from the open inquiry process of the science domain? Perhaps because deep down we all recognize we are participating together in the Great Going Forward that is modernity? Perhaps it’s simply how humans work within a community that loves what they do?
Once you look stepping stones are everywhere in IT:
The Open Source movement is the most obvious and powerful of stepping stone generators and collectors. It’s hard to imagine what the world would look like today without the rise of the Open Source ethos. It’s perhaps the greatest invention of our age.
People in IT seem especially open to following their own muse. Instead of trying to keep the Union together, if an individual or a small group doesn’t like where something is heading, they’ll break off and form their own movement. Consensus is not required or even desired. By holding many paths open simultaneously without knowing where they will lead, new stepping stones are continually being created.
Innumerable discussion forums like StackExchange minimizing the cycle time from problem to solution such that overall productivity is greatly multiplied.
Hundreds of conferences where people get together and exchange ideas. While conferences are usually not free they often make available papers from the talks as well as videos.
In addition to conferences there is a continuous stream of meetups and company sponsored talks. And don’t forget all the blogs!
YouTube and many other platforms contain instructional videos on any topic imaginable.
Universities are great producers of papers and new ideas and example software. While way too many papers sit unnecessarily behind paywalls, enough leaks through to make a difference, though not the difference they can make with a more open policy.
Companies and universities are continually researching new materials and capabilities.
For the first time in history advances in manufacturing and logistics have made it possible to truly reach a global market.
Massive government investment helps create the basic science that keeps the whole ball rolling.
If that’s not enough, innovation is catalyzed with Xprize type contests that set audacious goals while not trying to control how they are achieved. Companies like Google and Facebook are actively persuing their own Moon Shots.
Finally there's the VC industry that now includes a bevy of crowdsourcing options. VCs are a betting market in which ideas that are one stepping stone away from success are nurtured and made part of the energy flow. Without this carrot a lot of innovation would be lost.
All these forces and more help keep the universe of stepping stones expanding. It’s sort of inflationary model of progress radiating out from the Big Bang of the Enlightenment.
There are Some Challenges Ahead
Private IPOs. The great dream of many developers is to work really hard and reap great rewards with a public IPO. Like the lottery, winning doesn’t happen often, but it’s the dream that keeps people in the game. Private IPOs seem like part of a general move to Winner Take All scenarios and in the Private IPO the developer is not the winner. If we don’t reward developers will we be choking off the air supply to some of our best stepping stone builders?
Walled Gardens. Data is the future yada yada. While Google prospered on an open web the open web is being divided into a few Winner Take All data silos. Without open data how can the next generation of stepping stones be built? It goes against all reason to think a few closed companies can be as innovative as our great IT innovation machine. Closed data is not a good development.
Be Fruitful and Create Stepping Stones
There’s a line of thought out there that programmers should focus their skills on solving problems that really matter. That’s a lot of pressure. Not everyone has a passionate interest in problems that “matter.” And it may just be bad advice.
Even if you don’t think what you are doing is curing cancer or making the world a better place, what you may be doing is creating a stepping stone. And that has value. So be fruitful and create stepping stones.
Related Articles
For a great interview with Ken Stanley take a listen to this interview on the Science Radio Cafe.
Novelty Search and the Myth of the Objective (at EPIA 2013)
Picbreeder: a collaborative art application based on an idea called evolutionary art, which is a technique that allows pictures to be bred almost like animals.
The Unreasonable Effectiveness of Mathematics in the Natural Sciences
More Interesting Quotes from Why Greatness Cannot Be Planned: The Myth of the Objective
Interestingly, Jobs himself told a great story of the value in following the interesting without worrying about long-term objectives: If I had never dropped out, if I had never dropped in on that calligraphy class, then personal computers might not have the wonderful typography that they do. Of course it was impossible to connect the dots looking forward when I was in college, but it was very, very clear looking backwards.
For example, computer programmer Markus “Notch” Persson realized in 2009 that a new kind of video game was possible through combining concepts from three recent games: “Dwarf Fortress,” “Roller Coaster Tycoon,” and “Infiniminer.” Unlike nearly all modern games, his resulting lo-fi game, “Minecraft,” has outdated graphics, few frills, little content, and provides players with no explicit goals. Instead, players are free to discover, build, and create in a pixelated massive open world composed of cubes and recombinable resources. Few would have predicted that such a strange game could be successful, let alone that it might fundamentally alter the video game industry. Yet Notch understood the value of a new concrete possibility: Melding recent game innovations together could result in a new video game that was much like an interactive digital version of Legos, the children’s toy.
Instead of judging every activity for its potential to succeed, we should judge our projects for their potential to spawn more projects. If we really behave as treasure hunters and stepping stone collectors, then the only important thing about a stepping stone is that it leads to more stepping stones, period. The worst stepping stone is one that leads nowhere beyond itself, no matter how nice it may feel to stand upon it for the moment. As treasure hunters, our interest is in collecting more stepping stones, not in reaching a particular destination. The more stepping stones we find, the more opportunities there are to depart to somewhere greater.
In today’s culture, it’s rare that we follow this kind of path. The prevailing philosophy is to chain exploration, to enslave it to our objectives.
If your objective was to invent a microwave oven, you would not be working on radars. If you wanted to build a flying machine (as countless failed inventors did over the years), you wouldn’t spend the next few decades instead trying to invent an engine. If you were like Charles Babbage in the 1820s [5] and wanted to build a computer, you wouldn’t dedicate the rest of your life to refining vacuum tube technology.
The paradox is that the key stepping stones were perfected only by people without the ultimate objective of building microwaves, airplanes, or computers. The structure of the search space—the great room of all possible things—is just plain weird.
Deception is the key reason that objectives often don’t work to drive achievement. If the objective is deceptive, as it must be for most ambitious problems, then setting it and guiding our efforts by it offers little help in reaching it.
Focusing only on the objective leads to deception. Robots that come closer to the goal are often actually running into cul-de-sacs that are far from the correct path to the real solution. These cul-de-sacs, which you can see in Fig. 5.2, are like any other deceptive trap—they’re really no different from being stuck in a Chinese finger trap. The direction that seems to bring you closer ends up being the wrong direction.
When the quest for progress is packaged into a measure, the result is an objective-driven approach. If the objective is ambitious, then a drive to increase objective performance is likely to produce deception, preventing the best possible result from being discovered. For example, GDP (gross domestic product) is a measure of national productivity. Of course, as a nation, the hope is to maximize GDP, which makes maximal GDP a kind of national objective. But once again, an increase in GDP does not mean that economic policies, if continued, will lead to even higher GDP. The economy could be stuck in a Chinese finger trap—a decrease might be needed to produce a much larger increase. In fact, economists recognize that relying too heavily upon GDP makes little sense—even while in practice it is the most widely-touted economic indicator. This paradox has been called “GDP fetishism [62].”
A more poisonous and extreme form of Campbell’s law is the problem of perverse incentives. Strangely, sometimes rewards or measures chosen to make things better actually make them far worse. For example, when India was under British rule, the British government tried to exterminate poisonous snakes by paying citizens for every dead snake they handed over. But it didn’t work out the way it was intended: Instead it led to citizens literally breeding cobras just to kill them for the bounty. Ultimately, the number of venomous snakes in India actually increased [63]. So the incentive system produced the opposite of the intended effect. And the same thing happened in Hanoi but with rats—leading not to fewer rats, but to rat farms [64].
“You can’t control what you can’t measure [69].” Over 35 years later, DeMarco published an article revealing that his viewpoint had reversed over time: While “the book’s deep message seems to be, metrics are good, more would be better, and most would be best,” it turns out that they “must be used with careful moderation [70].” With more complex software, composed of millions of lines of code and countless interacting parts, simple metrics become uninformative.
But there’s a problem with accuracy that’s also tied to the myth of the objective: Surprisingly, accuracy doesn’t necessarily help increase performance in a pursuit driven by objectives. That’s probably not the kind of news you want to hear if you’re involved in the push for more accuracy in assessment.
But when experts radically disagree with each other, something interesting is happening. Darwin’s theory of evolution was dismissed by many experts when it was first introduced [83]—a good sign! Thomas Kuhn spoke of paradigm shifts in which the present framework of science begins to crack. At those moments discord is the mark of revolution [84]. For all these reasons, some of our resources should go towards rewarding disagreement rather than consensus.
For example, how many predicted that advances in commodity consumer electronics would lead to the first mass-produced fully-electric sports car, the Tesla Roadster? Yet by bundling together what are in effect thousands of lithium ion laptop batteries, it became possible to create practical electric cars that were both lighter and more powerful [99]. There’s nothing like suddenly realizing that we’re one stepping stone away from some yet unrealized potential.
Of course many objective-driven endeavors must and should proceed, and nothing in this book should contradict that. Houses should still be constructed according to the blueprint, software should still be designed to meet its specification, and there’s no harm in following the recipe when you cook your next dinner. You might even increase your endurance if you pick up running. All of these are modest objectives and not the target of the arguments in this book.
For example, if you discover an ancient artifact while exploring an uncharted river, it’s not by complete chance—you found it because you were exploring, even if you didn’t know what you would find.
