AI researchers 'earnestly believe' it could kill all humans within the next decade, Anthropic researcher says

Gammatek ISPL

Sep 10

11 min read

Author: Gammatek ISPL , Published: Sep 2026 An Anthropic researcher whose entire job is making AI systems safer just said, in public, that he thinks there's more than a 10% chance AI kills every human being on Earth within the next ten years. He wasn't fired for it. His own company didn't walk it back. If the people building the technology you're increasingly running your business on are putting numbers like that in writing, the question for anyone responsible for risk at their company isn't whether this is hyperbole, it's whether your enterprise risk management framework has a category for it at all. For most companies, it currently doesn't.

What was actually said, and by whom

On September 9, 2026, a former Anthropic and OpenAI researcher named Jacob Coxon posted a lengthy resignation thread on X explaining why he'd left Anthropic. His central claim: <cite index="21-1">the people building AI genuinely believe it could kill everyone by the end of the decade, and this isn't a publicity stunt</cite>. Coxon said he'd spent three years doing pretraining research at both companies and that neither was acting responsibly, accusing them of racing toward self-improving superintelligence while gambling with everyone's safety.

What made this different from the usual wave of AI-doom commentary is who responded, and how fast. Evan Hubinger, Anthropic's alignment science lead, the person literally in charge of trying to make sure Anthropic's models stay under human control, replied within hours confirming Coxon's account rather than distancing the company from it. <cite index="19-3,19-4">Hubinger said he personally believes there's a greater than 10% chance of this outcome within the next decade, and that Anthropic doesn't yet have a plan to solve alignment for superintelligence and isn't clearly on track to develop one</cite>.

He was careful to clarify that the risk isn't coming from current models, he described that risk as low, the concern is about what comes after. Samuel Marks, who leads scalable oversight at Anthropic, added his own post arguing AI developers keep building despite believing in catastrophic outcomes because of <cite index="20-4">a mix of commercial pressure and the belief that they're locked in a race against less careful competitors</cite>. Separately, OpenAI's chief scientist Jakub Pachocki published his own post around the same time acknowledging that no AI company has solved alignment and monitoring well enough yet either.

Coxon's original thread has reportedly been viewed more than 70 million times. Whatever you think of the underlying probability estimate, the fact that it's coming from inside the safety teams at the two most prominent frontier labs, stated in public, under their own names, is the part worth sitting with.

Why this is different from the usual AI doom cycle

AI extinction warnings aren't new. Researchers have debated existential AI risk in academic and policy circles for over a decade. What's new here is the source and the register. This wasn't an outside critic, an op-ed writer, or a rival company's competitive jab. It was a resignation, followed by confirmation from a sitting alignment lead at the company being criticized, using his real name and title, on a public platform, during a week when both Anthropic and OpenAI are reportedly moving toward historic IPOs.

That combination, insiders confirming the concern rather than a company spokesperson downplaying it, is a different category of signal than most enterprise risk functions are set up to evaluate. Traditional risk management has well-developed tools for parsing vendor claims, analyst reports, and competitor positioning. It has almost no established process for weighing a safety researcher's public probability estimate about a risk category that doesn't have an actuarial history, an insurance product, or a regulatory framework built around it yet. That gap is exactly the opportunity, and the exposure, for whoever owns enterprise risk at your organization right now.

This isn't the first warning, but it's a different kind of one

Public warnings about catastrophic AI risk have a longer history than this week's headlines suggest, and it's worth being precise about how this moment differs from what came before, because the distinction changes how a risk function should weigh it. In 2023, hundreds of AI researchers and executives, including the CEOs of OpenAI, Anthropic, and Google DeepMind, signed a one-sentence statement from the Center for AI Safety declaring that mitigating AI extinction risk should be a global priority alongside pandemics and nuclear war. Geoffrey Hinton and Yoshua Bengio, two of the field's most cited researchers, have spent the years since publicly warning about loss-of-control scenarios. None of that was new information to anyone tracking AI policy closely.

What changed this week wasn't the existence of the warning, it was the register it was delivered in. The 2023 statement was a carefully negotiated, consensus-built sentence signed by dozens of people with competing interests, designed to be uncontroversial enough that everyone could sign it. Coxon's resignation and Hubinger's confirmation were neither negotiated nor consensus-built. They were two individuals, in the middle of a normal work week, stating a specific number in public and then not retracting it. For a risk function evaluating credibility, spontaneous, costly, unscripted statements from people with strong incentives to stay quiet carry more weight than a polished consensus statement, precisely because the incentives point the opposite direction: Coxon walked away from his job, and Hubinger's employer had every commercial reason to want him to say something reassuring instead.

There's also a liability dimension worth naming directly, since it's the part most likely to eventually show up in a board meeting rather than a trade publication. If a company's own safety leadership is on record estimating a double-digit probability of catastrophic harm from the technology it's building, that record doesn't disappear. It becomes discoverable. Directors' and officers' liability insurers, and the boards who buy that coverage, increasingly ask what a company knew and when, in almost every major litigation involving a foreseeable harm. A public statement from an alignment lead is exactly the kind of documented foresight that shows up in that conversation, for the lab itself and, by extension, for any company whose risk exposure depends heavily on that lab's products.

Where catastrophic AI risk sits, or doesn't, on a standard risk register

Pull up almost any company's existing risk register and you'll find familiar categories: financial risk, operational risk, credit risk, cyber risk, regulatory risk, reputational risk. Frontier AI catastrophic risk, the kind Hubinger and Coxon are describing, doesn't map cleanly onto any of them, which is precisely why it tends to fall through the cracks rather than getting formally assessed and then dismissed.

It's not quite cyber risk, because it's not about your systems being breached, it's about the AI systems your vendors are racing to build becoming misaligned in ways that could affect everyone, including companies with no direct relationship to the labs building them. It's not quite vendor risk in the traditional sense either, because the exposure isn't "this vendor might fail to deliver," it's "this vendor's own safety leadership has stated they don't yet have a plan to prevent a catastrophic outcome." And it's not quite regulatory risk, because for most of this category, the regulation doesn't exist yet to force the assessment the way, say, data privacy law forces a privacy risk assessment.

This is exactly the gap that enterprise risk management software and enterprise governance software vendors are starting to build categories for: a slot for systemic, low-probability, high-severity technology risk that sits outside the usual buckets, gets escalated to the board rather than buried in an IT risk appendix, and gets revisited on a schedule instead of only after a headline forces the question.

A comparison: traditional ERM categories vs. frontier AI risk

Dimension	Traditional operational/cyber risk	Frontier AI catastrophic risk
Source of the risk	Your own systems, vendors, or processes failing or being attacked	Third-party AI labs' models becoming misaligned or misused at a scale beyond any one company's control
Actuarial history	Decades of loss data, established insurance products	Essentially none; no historical base rate exists to calibrate against
Who's sounding the alarm	Security researchers, auditors, regulators	The safety researchers employed by the companies building the systems, an unusually credible source structurally incentivized to downplay rather than overstate the risk
Regulatory backstop	Established frameworks (SOC 2, ISO 27001, GDPR, sector-specific rules)	Emerging and inconsistent; the EU AI Act addresses high-risk deployment, not existential-scale model risk directly
Typical ERM software handling today	Dedicated risk category, quantified, tracked, reported to the board	Usually absent entirely, or folded loosely into generic "AI/technology risk" without a specific assessment methodology
Appropriate response horizon	Continuous monitoring, incident response plans	Scenario planning and vendor concentration review, since no single company can mitigate the underlying risk alone

The last row is the most important one operationally. No individual company, including yours, can reduce the probability that a frontier lab's model becomes misaligned. What you can control is how exposed your operations are to that outcome: how concentrated your dependency is on any single AI vendor, whether critical processes have a non-AI fallback, and whether your board has actually been briefed on this category rather than assuming it's someone else's problem to track.

Implementation considerations for building AI risk into your ERM function

Add a distinct line item for frontier AI systemic risk to your risk register, separate from generic "technology risk." Folding it into an existing catch-all category guarantees it never gets a dedicated owner or a specific review cadence.
Track AI vendor concentration the way you'd track single-supplier dependency in procurement. If a critical business process depends entirely on one foundation model provider, that's a concentration risk worth quantifying regardless of how you feel about the extinction-risk debate itself.
Build a non-AI fallback path for anything genuinely mission-critical. This isn't about distrusting AI day to day, current-model risk is what Anthropic's own researchers describe as low, it's about not having a single point of failure tied to a technology category its own safety teams say they can't yet fully control.
Put frontier AI risk on the board's risk committee agenda at least annually, with a named owner. A risk category serious enough that a lab's alignment lead is publicly estimating a double-digit chance of catastrophic outcome deserves more than an informal mention in a broader technology briefing.
Revisit your enterprise risk management software's category taxonomy. Most ERM platforms were built around financial, operational, and compliance risk categories from a decade or more ago. If yours doesn't have a configurable slot for systemic third-party technology risk, that's a gap worth raising with your vendor or your internal risk team now, not after the next viral resignation thread.
Separate the probability debate from the exposure-management task. You don't need to resolve whether Hubinger's 10% estimate is right to do useful work here. You need to know how exposed your specific operations are if any version of this risk category materializes, which is a tractable, boring, unglamorous risk-management exercise regardless of where you land on the underlying debate.
Ask your D&O and cyber insurance brokers directly whether frontier AI dependency is contemplated in your current policy language. Most existing cyber and technology E&O policies were written before this specific risk category was part of the public conversation, and coverage gaps tend to surface only after a loss event, which is the most expensive possible time to discover one.
Document the review, even if the conclusion is "acceptable exposure for now." The value of this exercise isn't necessarily changing your AI vendor relationships this quarter. It's having a dated record that your risk function actually considered the category, assessed your specific exposure, and made a deliberate decision, rather than never having looked at it at all. That record is exactly what a board, an auditor, or an insurer will ask for later.

This is the same discipline behind the accountability-chain requirement we covered when OpenAI's policy team raised its own concentration-of-power warning, and behind the human-oversight obligations we walked through in where AI is creating jobs for humans. The pattern across all three: the industry itself keeps naming risks it hasn't fully solved, faster than most companies' risk frameworks are updating to account for them.

What this looks like in practice for a mid-sized company

Most companies reading this aren't running frontier model training and don't need a bespoke AI safety team. What they typically have is a handful of vendor relationships, Microsoft Copilot, an AI-enabled CRM feature, an AI customer support layer, that all trace back, one or two steps removed, to the same small handful of frontier labs this week's story is about. The practical exposure isn't "our company causes an extinction event." It's "our operations depend on a small number of AI providers whose own leadership says they don't have a solved plan for controlling what they're building, and we've never formally written that down as a tracked risk."

Closing that gap doesn't require a philosophy debate at the next board meeting. It requires the same unglamorous inventory work every other risk category gets: list the AI-dependent processes, note which vendor each one traces back to, flag the ones with no manual fallback, and put a single line in the risk register acknowledging the category exists. That's roughly a week of work for someone who already owns vendor risk or IT risk, not a new department.

Frequently asked questions

Is a 10% extinction-risk estimate from one researcher something a business should actually plan around? Not as a precise probability to build a model on, the number itself is a personal estimate, not a peer-reviewed forecast. What's worth planning around is the underlying fact pattern: a company's own alignment leadership stating in public that they don't yet have a solved plan for controlling the systems they're racing to build. That's a vendor-concentration and dependency question your risk function can act on regardless of whether the real probability is 10%, 1%, or something else entirely.

Does this mean companies should stop using AI tools? No, and the researchers making these statements are explicit that the concern is about future superintelligent systems, not the AI tools currently in production use, which they describe as low-risk. The actionable takeaway is exposure management and vendor concentration awareness, not abandoning current AI tooling.

How is this different from the general AI safety debate that's been going on for years? The debate itself isn't new. What's new is a sitting alignment science lead at a major AI lab publicly confirming, under his own name, a departing colleague's claim that the people building the technology privately believe it could be catastrophic, rather than the company distancing itself from the claim. That's a credibility signal risk managers don't usually get to see this directly.

What's the single highest-leverage first step for a company with no current AI risk category on its books? Add the line item to the risk register and name an owner. That single step, done this quarter, does more than any amount of debate about the underlying probability, because it converts an abstract worry into a tracked, revisited, board-visible responsibility.

Should this affect how we evaluate enterprise risk management software vendors? Yes, worth asking directly: does the platform support a configurable category for systemic third-party technology risk, distinct from generic IT risk, with its own scoring methodology and reporting cadence? Many legacy ERM platforms don't yet, and that's a reasonable question to raise in your next vendor review regardless of which platform you're on today.

Is this the same thing as the 2023 "AI extinction risk" statement signed by AI CEOs? Related, but not the same signal. The 2023 Center for AI Safety statement was a consensus document signed by dozens of people, designed to be broadly agreeable. This week's statements were unscripted, made by two individuals mid-career with direct personal cost, a resignation and a public confirmation from a sitting alignment lead, which most risk analysts would weight as a stronger credibility signal than a negotiated group statement, even though the underlying concern is the same one that's been circulating since at least 2023.

Could statements like these create legal exposure for AI companies, and by extension their enterprise customers?It's plausible, though untested. Documented, public statements from a company's own safety leadership acknowledging an unsolved catastrophic risk are exactly the kind of foreseeability evidence that tends to surface in litigation and insurance disputes. That's a reason for enterprise customers to ask their AI vendors directly about how this is being handled internally, not just a reason to worry about the vendor's own legal exposure.

Not sure whether your current risk register or ERM software actually has a home for this category, or who inside your company should own it?

Our AI governance and compliance review maps your AI vendor dependencies, identifies where systemic AI risk currently has no owner in your risk framework, and helps you build the board-level reporting structure before an incident, or another viral resignation thread, forces the question.

Book an AI governance and compliance review : https://www.gammateksolutions.com/post/the-best-worst-and-strangest-ways-ai-is-really-being-used-at-work https://www.gammateksolutions.com/post/big-tech-profits-get-160bn-boost-from-gains-on-stakes-in-other-ai-companies https://www.gammateksolutions.com/post/amazon-prime-members-just-got-a-huge-new-perk-here-s-how-to-use-it