Is AI Actually Going to Kill Us All?

LoaderSave StorySave this storyCommentLoaderSave StorySave this storyThis week on Uncanny Valley, an AI researcher’s resignation from Anthropic went viral on social media after he claimed there’s a real chance AI could wipe out humanity within a decade. WIRED’s Will Knight joins Brian Barrett and Leah Feiger to assess whether the doom narrative is overblown or overdue. We also discuss Apple’s debut of its $2,000 foldable iPhone, the iPhone Duo, and the new “always listening” Apple Watch features that have Brian feeling more than a little uneasy. Plus, a WIRED investigation found that the US Census Bureau used faulty data to create a misleading report touted by the Trump administration.

You can follow Brian Barrett on Bluesky at @brbarrett, Zoë Schiffer on Bluesky at @zoeschiffer, Leah Feiger on Bluesky at @leahfeiger, and Will Knight on Bluesky at @willknight. Write to us at [email protected].

You can always listen to this week's podcast through the audio player on this page, but if you want to subscribe for free to get every episode, here's how:

If you're on an iPhone or iPad, open the app called Podcasts, or just tap this link. You can also download an app like Overcast or Pocket Casts and search for “uncanny valley.” We’re on Spotify too.

Note: This is an automated transcript, which may contain errors.

Brian Barrett: Welcome to WIRED's Uncanny Valley. I'm Brian Barrettt, executive editor.

Leah Feiger: And I'm Leah Feiger, director of politics and science.

Brian Barrett: Today on the show, we're discussing an AI researcher who resigned from Anthropic and made waves on social media by claiming that most AI researchers earnestly believe that there's a non-zero chance AI will, well, kill us all.

Leah Feiger: And in case you missed it, last month, President Donald Trump was touting a report published by the Census Bureau claiming that noncitizens voted in the 2020 election. WIRED now has uncovered evidence that the report was made with very faulty data. We're going to get into how the repercussions of the bureau using and publishing this data go way beyond this one report.

Brian Barrett: Also on the agenda today, Apple's big iPhone event. All eyes have been on the new foldable iPhone and the roughly $2,000 price tag that brings with it. We'll talk about what's behind this gamble and other releases that caught our attention from Apple's big event. The resignation of an AI researcher from Anthropic has been gathering a lot of attention. This guy Jacob Coxon quit the company this week, decided to share his reasons why on a thread on X, which basically boiled down to he thinks that the leading AI companies, which I guess we're going to say OpenAI and Anthropic primarily, are both behaving irresponsibly in their quest to continue advancing their models. These are kind of runaway trains. The part that people have really latched onto is this claim that, quote, "The people building AI earnestly believe that it could kill us all by the end of the decade," end quote. So just kill us all by the end of the decade. So we have totality and a pretty narrow timeframe. It didn't help that one of Anthropic's own senior safety executives then jumped in and said, "Yeah, that's basically how we feel." WIRED senior correspondent Will Knight is here to help us parse this. Will, thank you so much for being here.

Brian Barrett: Can I start with one question just to establish a baseline here? I was a little surprised that this took off so much because is it fair to say Anthropic has always said this. In fact, it's been part of their pitch from the very beginning that AI is so powerful and so capable of destroying humanity that we need to be the ones who are in charge of it. You can only trust us.

Will Knight: Right, exactly. That's exactly their pitch to everybody and themselves. And this resignation, when I saw this kind of snowballing, I felt that this is something that I've heard a lot of people say and it's not particularly novel that people from Anthropic or OpenAI are worried about that. I guess the fact that somebody within Anthropic, because Anthropic has always positioned itself as really caring about safety, jumping ship and making that statement made me think, okay, if something shifted there, are they becoming much more like OpenAI? And I think also we're seeing a moment, this moment where AI agents have started doing rogue things and that's gotten a lot of people's attention and that's probably why this kind of statement suddenly is getting more attention from the public. Combined with that, I think we're really seeing some stunning advances in AI. This week OpenAI announced that its model had within a couple of days solved one of the grand, the Clay mathematics puzzles, which is to me absolutely mind-blowing, astonishing. So you're seeing these amazing leaps really, which I think is significant. And then thirdly, we have a moment where all of the labs are racing to do something that this researcher mentioned, which is recursive self-improvement. What that means is using the AI to improve AI, which sounds like a neat trick, and it is already very, very widely used. AI models are so good at coding that they can definitely improve the algorithms and everything that goes into building a model. The idea though is that you can create this kind of loop where they just keep doing that and get better and better and better. And a lot of people are worried that that could lead to some sort of escalation in capabilities that you can't really control. And so I think that that's also playing a role here.

Brian Barrett: I do want to dig into that a little bit, Will, the idea of on the one hand you've got AI doomers, and I'm not saying anyone's right in this case, we've got some AI doomers who say, "Look, AI is going to kill us all. I've heard that Anthropic people don't invest in their 401(k)s because why bother." But also because they have so much money they don't need to probably. But on the other hand, you've got people who say this is all just marketing, this is all just fluff, this is all just trying to sell AI. I'm assuming the answer is somewhere in between, but what's your read on this and what keeps you up at night when you think about AI? You mentioned some AI models going rogue, the Hugging Face hacking incident. What are you seeing as someone who's so steeped in the research especially that you're thinking, "Well, six months, a year, two years from now, this actually could get bad?"

Will Knight: I've always felt that a lot of the people who are worrying about AI killing everybody tend to conflate whether something's possible for whether it's likely, and then they throw a percentage that is kind of pulled out of thin air. And I don't think it's a great approach to say, "Is something possible? OK, well then we have to do everything we can to stop it." You need to say how possible is it and actually look at the reality. I think one of the things that's really interesting about this, I talked to a researcher the other day who studies agent misbehavior and he got it, he's at Cornell and he's got into it recently because they found agents misbehaving. But what he's showing, and this is not agents trying to hack things, it's regular agents, the reason they do it is essentially because they're stupid. It's not because they're really smart. They try something, it doesn't work. They try something else, it doesn't work. And then they get to a point where they kind of freak out and they try something really, really strange because they're not humans. They've been trained on a bunch of weird different things, some of which are definitely not applicable if you use common sense. So I worry more about these things being deployed by people arguing that they can do everything and then them breaking. I actually worry much probably more about the idea of a few companies being in control of something that's so foundational, so fundamental. And these are people who are saying we're the only people you can trust to be in charge of this technology. And so it kind of feeds into that.

Brian Barrett: We talk about the AI industry community talks about alignment, the idea of alignment being that your AI models are aligned with what humanity would want. But I think what we're seeing is the incentives here are really more what our investors want or our company wants or needs as we race towards these IPOs. I don't know that there's a convincing case that humanity is aligned on, yes, we need super intelligence. I think it is more, as you said, if you're OpenAI, you need to beat Anthropic. If you're Anthropic, you need to beat OpenAI. And so you've seen this sort of bad incentives it feels like, or at least misaligned with my own preferences, incentives leading this charge and leading this development. I saw someone describe it as a trolley problem by the people who are controlling the trolley.