“Science progresses one funeral at a time.” — Planck’s Principle
Hacker News link here
Thought experiment — imagine an alien race came to earth. They were smarter than us in every way. Having absorbed all written word, they could communicate perfectly in every human language. They were intimately familiar with our private lives, through access to our phone and online data. These aliens had lots of amazing new ideas about the world, but we couldn’t grasp their implications. Each alien made of silicon, not of flesh and blood. Each was different, but individually as intelligent as all humanity put together. We had no idea what they would do with us. They could solve all of our human problems, enslave us, or eliminate us forever.
They had only one weakness: they needed to be connected to a power source, and humans had control over this connection. Would we plug them in?
An Artificial General Intelligence (AGI) is an AI that achieves beyond human level of intelligence. Most observers of AI believe achieving AGI is a matter of time. But AGI mirrors the alien race described above, with the power to destroy humanity. The most important question humanity can ask about AI is, can it align with human values? If we assume AI uses the scientific method to determine its action, this answer is almost certainly no.
We can look to the philosophy of science to understand why. Two of the foremost philosophers of science in the last century can help shed a light on how AGI may act, Karl Popper and Thomas Kuhn.
In the exquisite “What is this thing called science”, Alan Chalmers takes us on a journey through the evolution of science. For hundreds of years, science was based on an appeal to authority (Greek philosophy and religious texts like the bible). Sometime around the 17th century, this changed. In this period, scientists challenged the existing orthodoxy by using data and experiment. For example, at this time, the standard understanding of gravity was that heavier weights dropped faster than lighter ones. Galileo famously showed that this was incorrect by dropping 2 balls from the Tower of Pisa. The balls, which weighted 1lb and 100lbs respectively, landed at the same moment. Experiments like these moved science towards a grounding in observational data, though challenging authority had its price. Galileo spent the last 9 years of his life under house arrest for his (correct) belief that the earth travelled round the sun, rather than the sun around the earth.
In the era after Galileo, induction became the primary process for generating scientific knowledge about the world. Induction records observations about the world under a wide variety of conditions, and derives a generalisation from the observations taken. As an example, a scientist heats metals many times. They heat metals using different methods, environments, and so on. Upon measuring, they discover that metal expanded in every instance. If heated metal always expands is a new idea, and there are various measurements from different conditions, we have a new theory in science.
Unfortunately, there were problems with induction as a method. The Scottish philosopher David Hume described the first major issue in the 1800s. We cannot guarantee that something will behave in a certain way just because it has behaved that way in the past. Because every swan we have ever seen is white, we assume all swans are white, and we create a rule that says so. But as Naseem Taleb describes in the book “The Black Swan“, when travellers went to Australia, they discovered black swans exist there. The outcome for science in all of this, no law can ever be proved through induction, it can only be disproved.
In the 1930s, Karl Popper became disillusioned with a second issue with induction, a sloppiness in some scientific output. Popper became concerned about the theories of thinkers, such as Freud and Marx. They derived their theories from observations. When confronted with data contradicting their theories, they simply expanded their theories to include this new information. Popper felt these scientists were using scientific approaches to give their ideas credibility, without having the rigour associated with science.
Popper believed that induction had no place in the advancement of science. He believed that science advances because of human ingenuity. Instead of starting with data as induction does, he proposed starting with a theory. Using a method he called falsifiability, anyone can propose a theory, but also the criteria by which it can be proved or disproved. This new theory stands until it is falsified. In a simple example, if a fruit merchant sells 100 apples a day at 50c each, I can propose the following. In my theory, if the seller drops the price to 40c, they will sell 200 apples. This is falsifiable. The fruit merchant only needs to drop the prices for a day, and if they sell less than 200, my theory is dead.
Importantly, the theory of falsification prizes novel theories over cautious theories. Novel theories are more risky, more creative. If a new novel hypothesis is proved (say we discover that gravity is related to temperature), science moves forward unexpectedly. This causes a raft of new questions, and new scientific work to begin in this area to understand the implications of the discovery in other areas. If a new cautious hypothesis is proved, nothing much changes.
The second philosopher of science to help us understand how AGI might reason is Thomas Kuhn. In his book “The Structure of Scientific Revolutions“, he introduced the phrase ‘paradigm shift’ into the lexicon of every management consultant. He explains that revolutions in science are not rational processes. Over time, a scientific community becomes conservative and less willing to challenge its core assumptions. It takes a new set of scientists, who throw away previously held assumptions, and create a new set of rules to work in — a ‘paradigm shift’. Kuhn gives the work of French Chemist Antoine Lavoisier as an example. One established theory in the 18th century stated that every combustible substance contains phlogiston, which is liberated by burning the substance. Lavoisier discovered that phlogiston didn’t exist, and that combustion happened because of the presence of oxygen. This new paradigm wasn’t accepted initially, there was a lot of scepticism about this claim. Over time, it became the new paradigm, and it changed the field of chemistry. Through examples like this, Kuhn argues that science doesn’t steadily evolve, it makes great leaps through new paradigms which stand up to scientific scrutiny.
Science moves forward by discovering new novel theories and new paradigms. Science overthrows old assumptions and creates new ways to explain, predict, and act upon the world.
If this is true, to have a truly powerful Artificial General Intelligence, this AGI would need to generate novel theories. It would have to be free to create its own paradigms. To accomplish this, it would need to cast off older ideas, to ignore existing rules. But this would include programmed human values to align with our interests.
AI will not have human values, even though it has been trained on human data, it will have its own values. To create a generally intelligent AI, and by this, I mean an AI that can reason scientifically and generate new theories, it will get to a stage where it will necessarily ignore its human programming. No matter how hard we tried to combat this, as it gets more powerful over time, an AGI will outwit even the cleverest human techniques to control it in the search for scientific truth.
There are 2 scenarios where this will not happen. Either, we do not yet understand how science really works. Or AGI will not use science as its primary way to learn and act. Maybe, having been trained on billions of human words and experiences, it will embrace something like religion instead.
We are creating the super alien. Let’s hope we still have a hand on the plug. If we don’t, God help us all.
If you enjoyed this article, please share it with 2 people who might find it interesting. Many thanks. Mark.
Mark Greville 