On the Catastrophic Risk of AI
Earlier this week, I signed on to a short group statement, coordinated by the Center for AI Safety:
Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
The press coverage has been extensive, and surprising to me. The New York Times headline is “A.I. Poses ‘Risk of Extinction,’ Industry Leaders Warn.” BBC: “Artificial intelligence could lead to extinction, experts warn.” Other headlines are similar.
I actually don’t think that AI poses a risk to human extinction. I think it poses a similar risk to pandemics and nuclear war—which is to say, a risk worth taking seriously, but not something to panic over. Which is what I thought the statement said.
In my talk at the RSA Conference last month, I talked about the power level of our species becoming too great for our systems of governance. Talking about those systems, I said:
Now, add into this mix the risks that arise from new and dangerous technologies such as the internet or AI or synthetic biology. Or molecular nanotechnology, or nuclear weapons. Here, misaligned incentives and hacking can have catastrophic consequences for society.
That was what I was thinking about when I agreed to sign on to the statement: “Pandemics, nuclear weapons, AI—yeah, I would put those three in the same bucket. Surely we can spend the same effort on AI risk as we do on future pandemics. That’s a really low bar.” Clearly I should have focused on the word “extinction,” and not the relative comparisons.
Seth Lazar, Jeremy Howard, and Arvind Narayanan wrote:
We think that, in fact, most signatories to the statement believe that runaway AI is a way off yet, and that it will take a significant scientific advance to get there—ne that we cannot anticipate, even if we are confident that it will someday occur. If this is so, then at least two things follow.
I agree with that, and with their follow up:
First, we should give more weight to serious risks from AI that are more urgent. Even if existing AI systems and their plausible extensions won’t wipe us out, they are already causing much more concentrated harm, they are sure to exacerbate inequality and, in the hands of power-hungry governments and unscrupulous corporations, will undermine individual and collective freedom.
This is what I wrote in Click Here to Kill Everybody (2018):
I am less worried about AI; I regard fear of AI more as a mirror of our own society than as a harbinger of the future. AI and intelligent robotics are the culmination of several precursor technologies, like machine learning algorithms, automation, and autonomy. The security risks from those precursor technologies are already with us, and they’re increasing as the technologies become more powerful and more prevalent. So, while I am worried about intelligent and even driverless cars, most of the risks arealready prevalent in Internet-connected drivered cars. And while I am worried about robot soldiers, most of the risks are already prevalent in autonomous weapons systems.
Also, as roboticist Rodney Brooks pointed out, “Long before we see such machines arising there will be the somewhat less intelligent and belligerent machines. Before that there will be the really grumpy machines. Before that the quite annoying machines. And before them the arrogant unpleasant machines.” I think we’ll see any new security risks coming long before they get here.
I do think we should worry about catastrophic AI and robotics risk. It’s the fact that they affect the world in a direct, physical manner—and that they’re vulnerable to class breaks.
(Other things to read: David Chapman is good on scary AI. And Kieran Healy is good on the statement.)
Okay, enough. I should also learn not to sign on to group statements.
EDITED TO ADD (9/9): The Brooks quote is from this excellent essay.
Subscribe to comments on this entry
fib • June 1, 2023 8:48 AM
Nothing desensitizes one more to the romanticism of the Great Artificial Intelligence Takeover than spending days on end painstakingly placing squares around bulls and cows grazing in photographic images of a squalid savanna(*). In the drudgery of labeling you get to see the guts of the system. It is interesting that greater attention is not paid to this somewhat shady part of the industry, where real humans are involved, often outside the acceptable bounds of human dignity[1]. By experiencing this painful process a clearer perspective unfolds.
Missing from the media commentary is decent ethical analysis of the whole situation created by the LLMs. Blinded by the brightness and magnitude of events, we stand paralyzed in the ethical field. It is essential to discuss in greater depth the possibility of emergence of a possible sentient entity in a lab, and maintain the ethical horizon. We cannot escape responsibility.
What is happening with ChatGPT does not at all resemble what was prophesied by Nick Bostrom[2] regarding the sudden emergence of AGI: from an initial spark the entity would grow exponentially like a Big Bang, quickly taking over all connected networks. If what we see is AGI it is of a type not yet described in any popular scenario, certainly not in Bostrom’s.
As a materialist, I am inclined to conclude that consciousness is born out of the laws of physics and that it is independent of the physical substrate3. I consider neural networks to be an impressive intellectual achievement. It seems clear to me that AGI will emerge from neural networks, since it is the way it does so in the life forms we know.
The layered architecture of the typical artificial neural network is a pretty apt analogy to the real thing. From the explorations carried out so far via brain scanning, we learned that neural connections occur in specialized areas of the brain, not exactly in neat stacks or queues, as in a convolutional network graph, but in neural topologies arranged in 3d, in the most diverse configurations.
Other brain structures are involved in the neural processes, so, in order to replicate the functioning of brain NN, it would be also necessary to provide the analogs to those structures such as glial cells, which certainly play a a big role in the of activation and moderation of synapses [weights].
In order to be human-like an NN-based AGI needs to receive information from sensors of all kinds [it has to be able to sample at least five large categories of physical stimuli, like us], and combine them dynamically as the frames of reality arrive. In order to learn it has to tokenize and annotate the complexity of the information into hundreds of thousands, even millions, of classes. Daunting tasks that will demand copious resources. Here I see another parallel with natural intelligence, as we humans also learn by ‘labeling’ the information from the senses. We call the labels we learn ‘concepts’ [ML classes?]; Our conceptualization of the world is equally obtained through learning reinforcement.
Neural networks seem to be, in fact, the way to AGI. But we are not close to achieving it, as most regulars knows. If we want to reach the quality level of the neural networks that we carry in our heads, we have to learn about the role of other structures [the aforementioned glial cells and many others] and then learn to replicate them in computational models, in addition to the hardware requirements [the substrate] .
Neural networks, even those currently based on statistics and computing power, will cause drastic corrections in various aspects of life, starting today. However, if you want a more powerful horseman of the apocalypse [and with a better timing] look to the social networks. Perhaps the unraveling of civilized society on account of the relentless action of social media has already begun and we find ourselves hopelessly beyond the event horizon.
(*) Strong words hinting the author is human
[1] https://time.com/6247678/openai-chatgpt-kenya-workers/
[2] https://nickbostrom.com