No, one researcher quitting over fears about self-improving superintelligence doesn't mean you should stop using AI tools in your firm. It does mean it's worth asking a more useful question. Who in your business is actually accountable for what your AI tools do, and would you notice if something went wrong?
What did the researcher actually say?
On 8 September 2026, Jacob Coxon announced his resignation from Anthropic in a post on X. He's a 27-year-old mathematics graduate who had spent three years doing pretraining research (the initial, resource-heavy phase of training a large language model on huge volumes of data, before it's fine-tuned for a specific job) at both OpenAI and Anthropic, Newsweek reported.
In the post itself, he wrote that the two companies "are racing straight to self-improving superintelligence and gambling with our lives." Superintelligence here means a system more capable than humans across nearly every domain. Self-improving means it could keep making itself more capable with limited human involvement. As TechCrunch reported, Coxon told the Wall Street Journal the same day that he no longer believes any single lab can build such a system responsibly, and that the industry needs government coordination or a "pacing agreement" among labs to slow the race down.
By his own account, Coxon's decision came down to two conclusions: "it's obvious that things are speeding up, and two, they're not under control." He says he's leaving the AI industry entirely, not moving to a competitor — a detail that matters, because it separates his warning from a rival lab's talking points.
He wasn't a lone voice for long. The same TechCrunch report noted that Anthropic colleague Evan Hubinger said publicly that the team believes AI could kill all humans with a likelihood above 10% within a decade. Hubinger also admitted Anthropic doesn't yet have a settled plan to keep a superintelligent system aligned — meaning reliably behaving as intended — with human interests.
Is this a new argument, or did it just go viral?
The underlying debate isn't new. Researchers inside frontier AI labs have raised alignment concerns publicly for several years now, usually while still drawing a salary from the company they're warning about.
In 2024, OpenAI's own safety-focused "superalignment" team lost both its leaders within a day of each other: Ilya Sutskever and Jan Leike. Leike said safety work had been "sailing against the wind" and losing out to product speed. What's different in Coxon's case is that he resigned specifically to say it, and a colleague who stayed backed the same order-of-magnitude risk estimate within days.
None of this means the two companies have stopped shipping products. Anthropic and OpenAI are both still releasing new models on their usual cadence. Neither has publicly committed to Coxon's proposed "pacing agreement," and the disagreement is playing out inside the labs building the tools your business already uses, not on the sidelines.
For a business owner in Sydney or Melbourne, the detail that matters isn't the resignation itself. It's that the people who build these systems for a living openly disagree about how much control exists over their own products right now. We track stories like this on the AI Smarter blog because AI news moves faster than most SMBs have time to translate into a plain business decision.
What does "AI risk" actually mean for a professional-services firm?
Extinction-level superintelligence is not the AI risk that will cost your accounting, legal or advisory practice money this year. Three much closer risks will, and none of them require a rogue superintelligent system to bite.
Concentration risk means your entire client workflow depends on one AI vendor with no fallback. If a bookkeeping practice routes every client query through a single AI assistant and that vendor changes pricing, deprecates the model, or has an outage, the practice stops functioning until someone fixes it manually.
Model dependency means trusting AI output without documenting why. A financial adviser who lets an AI tool draft a Statement of Advice (the formal document ASIC requires advisers to give clients recommendations in) needs a documented human sign-off step. Without one, there's no answer ready if a regulator or an unhappy client asks how the recommendation was actually formed.
Unreviewed automation means a workflow runs unattended long enough that nobody notices when it drifts. An automated email-response agent that quietly starts sending slightly wrong answers to client questions can run for weeks before anyone checks the log.
Stack these three together — one vendor, no sign-off step, no audit trail — and a consultancy has effectively built the same kind of system Coxon is worried about, just at a smaller scale. The failure here isn't an extinction event; it's a wrong invoice or a wrong client email that nobody caught.
None of these three showed up in a single headline this month. All three are the reason a documented methodology matters more than which model you happen to be using. That's the gap our AIOS approach is built to close — not by picking the "safest" model, but by putting a reviewable structure around how AI touches your business at all.
Should you stop using AI tools because of this?
No. Coxon's warning is about frontier labs racing to build systems more capable than any human, not about a small business running a customer-service chatbot with a documented review process behind it. Those are different problems at a different scale.
The near-term risk isn't a superintelligent model deciding it doesn't need you anymore. It's a client's confidential file pasted into a public chatbot with no data-handling policy behind it, or an AI-drafted email leaving a partner's inbox without anyone reading it first. As we set out on our homepage, the difference between reckless AI adoption and disciplined AI adoption was never about whether you use AI. It's about whether someone in the business is accountable for reviewing what it does.
What should you actually do about it this month?
Four steps, in order of urgency.
This week: Write down every place AI touches client-facing output — drafting, intake, reporting — and mark who reviews it before it goes out. If you can't list them from memory in under two minutes, that's the finding.
This month: Name one person accountable for spot-checking AI output weekly, and give them ten minutes on the calendar to actually do it. "It's probably fine" is not a review process, and nobody will build one without it being somebody's named job.
This month: Avoid locking anything critical to a single AI vendor with no manual fallback. Ask what would happen on Monday morning if the tool disappeared over the weekend — a price change, an outage, a shut-down feature. Write down the actual answer, not the assumption that it "probably won't happen."
This quarter: If you're not sure whether your current AI use already counts as unreviewed automation, that's the same audit our Sydney AI automation engagements start with. You can see what a properly governed setup looks like in our client work.
Do not panic and ban AI tools firm-wide over a story about frontier research labs. And do not wait for the AI industry to "sort out safety" before you fix basic governance gaps inside your own business. Those are two entirely different timelines, and only one of them is in your control.
Coxon's resignation is a genuinely significant story about where frontier AI development is heading. It is not, on its own, a reason to change what your firm does on Monday morning. It is a reason to finally write down who is watching your AI tools, and what they're supposed to do if something looks wrong.
If you want a plain-language look at where AI risk actually sits in your business — not the industry's, yours — book a free initial AI consultation and we'll walk through it together.