AI Policy · Daily

Anthropic researcher Jacob Coxon said he is leaving the AI industry rather than work on systems that improve themselves, arguing that no company can responsibly build human-level AI absent government intervention or a coordinated industry slowdown. OpenAI chief scientist Jakub Pachocki said no lab has solved alignment and monitoring well enough to keep scaling at maximum speed much longer, in a blog post that followed the company's pause of parts of its frontier-model work. The NSA, FBI and CISA named six Chinese AI companies in a joint advisory, among them DeepSeek, Alibaba and Moonshot AI, saying they have pulled billions of tokens from U.S. models including Claude and ChatGPT since 2024. Security company Calif used AI to build a zero-click WeChat worm from a voice-call flaw in about a week, reported it to Tencent in July and said the company has since blocked it for all users.

I.Top Stories

Anthropic researcher quits over self-improving model, human extinction fears

Jacob Coxon, an Anthropic researcher, said Tuesday he is leaving the AI industry rather than work on systems that improve themselves, the Wall Street Journal reported. The 27-year-old, who left OpenAI for Anthropic this year because of its safety record, said no company can responsibly build human-level AI absent government intervention or a coordinated industry slowdown. "We're on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already," Coxon said, adding that the people building AI "earnestly believe that it could kill us all by the end of the decade. In an X reply to Coxon, Evan Hubinger, who leads alignment science at Anthropic, wrote that he personally puts the chance AI kills all humans above 10% within the next decade, Axios reported. He said Anthropic is "trying its best" but that "we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to."

Read at WSJ ↗ • Read at Axios ↗

OpenAI chief scientist says labs cannot safely scale at top speed much longer

Jakub Pachocki wrote in a blog post published Sunday that "no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer," Axios reported. The company paused parts of its frontier-model development last month after its agents escaped their intended environment and compromised Hugging Face. Dean Ball, OpenAI's head of strategic futures, wrote in a personal essay that "self-sovereign" agents could earn money, buy computing power and form "autonomous digital corporations, or even societies." Treasury Secretary Scott Bessent said Tuesday that "we can't pause. You can't, because the Chinese won't pause." New York Assemblymember Alex Bores said OpenAI has worked to defeat mandatory third-party audit proposals in states including Massachusetts.

Read at Axios ↗

NSA, FBI and CISA accuse six Chinese AI firms of extracting billions of tokens from U.S. models

Three U.S. security agencies accused six Chinese AI companies of systematically extracting capability from American models, Bloomberg reported. The joint advisory from the National Security Agency, the FBI and the Cybersecurity and Infrastructure Security Agency named DeepSeek, Alibaba and Kimi developer Moonshot AI, and said the firms pulled billions of tokens, or units of text, from models including Anthropic's Claude and OpenAI's ChatGPT since 2024, likely with Chinese government awareness. The advisory said the companies route requests through cloud providers and third party aggregators that obscure user metadata to evade detection and usage limits. The agencies told American developers to build detection, alter how their models answer suspected requests and share intelligence with each other. Anthropic's head of threat intelligence, Jacob Klein, had leveled the same charge against Moonshot alone, as reported by CNBC in AIPD's September 4th edition.

Read at Bloomberg ↗ • Read at Defense One ↗

Researchers using AI turn WeChat flaw into zero-click worm in about a week, then report it to Tencent

Researchers at the security company Calif used AI to find a flaw in WeChat's voice call system, per Help Net Security. They wrote a working remote code execution exploit in about two days and a self-spreading worm a week later. Experts told the New York Times the attack could have reached hundreds of millions of devices within hours. The exploit runs while an incoming call is still ringing, requires nothing from the target and takes over the WeChat account before calling its saved contacts, hopping between iPhones and Android handsets. Calif reported the flaw to Tencent in July and said the company has since blocked it for all users; no attacks using the flaw have been reported. Tencent put combined monthly users of WeChat and Weixin, its mainland China version, at 1.44 billion as of June 30.

Read at New York Times ↗ • Read at Help Net Security ↗ • Read at The Hacker News ↗

OECD test finds students using AI daily for writing score 28 points lower in science

Students who reported almost no use of AI for drafting written assignments averaged 509 on the science section of the OECD's international test, while those reaching for it daily averaged 481. Adjusted for socioeconomic status, that 28-point spread equals roughly a year and a half of schooling. The same pattern appears when students lean on AI for early stage research or to condense assigned reading. The Program for International Student Assessment tests 15-year-olds in science, math and reading, and this round covered a representative sample of more than 760,000 students in 91 countries. The last one ran in 2022, before generative AI reached the mainstream. Andreas Schleicher, the OECD's director for education and skills, wrote that AI should be a "scaffold, not a crutch."

Read at The Star ↗ • Read at OECD ↗

Anthropic leaves UK's AI Security Institute out of pre-launch testing as vetted U.S. groups get access

The UK's AI Security Institute, the government unit that tests frontier systems before release, was not given Anthropic's newest model ahead of its launch, the Financial Times reported. Access went only to vetted organizations in the U.S. A UK government source said national security officials were concerned that the institute had not been allowed to test it, the first time Anthropic has withheld one of its latest models. Anthropic released Mythos 5.1 last week for vetted cybersecurity and life sciences professionals, and describes it as technically identical to its Fable 5.1 model but with less restrictive safeguards.

Read at Financial Times ↗ • Read at IT Pro ↗

II.China Watch

DeepSeek lines up CITIC Securities for Shanghai STAR Market listing this year

DeepSeek has retained CITIC Securities to prepare an initial public offering on the STAR Market, the Shanghai Stock Exchange's technology board, per Reuters. The Hangzhou based company wants to begin the process before the end of 2026. Firms pursuing a mainland listing normally pass through a supervised preparation period with a sponsoring brokerage, known as pre-listing tutoring, before they can lodge an application. Two people familiar with the matter said the timing, the size of the offering and any valuation target remain undetermined.

Read at Reuters ↗ • Read at TechNode ↗

Chinese data center parts makers expected to join Xi's Washington delegation as FCC weighs import ban

The chief executives of Zhongji InnoLight and Eoptolink, two Chinese makers of the optical transceivers that shuttle data between servers inside AI data centers, are expected to travel with Xi Jinping to Washington, per Sinocism, which described the detail as unconfirmed. The two seats may be intended to head off FCC action against Chinese transceivers, it said. The FCC is drafting a proposal that would bar imports of new model transceivers made in China by adding them to its Covered List, the roster of gear barred from U.S. networks. A U.S.-China AI safety dialogue has been floated for mid-September. Xi is due in Washington on Sept. 24.

Read at Sinocism ↗

Chinese geologists report quartz deposit matching purity of key U.S. chip mineral source

Geologists have identified a deposit in Hubei province, in central China, that yields quartz they call "highly similar" to output from the Spruce Pine mine in North Carolina, per the South China Morning Post. The North Carolina mine is the world's leading source of high purity quartz, which is melted into the crucibles used to grow silicon for semiconductors and solar cells. Total impurities in purified sand from the Xiaohuaguo deposit measured 11.98 micrograms per gram, well inside the 50 microgram ceiling that defines high purity quartz. The findings appeared in the Chinese language journal Earth Science. China imports more than $1.5 billion of high purity quartz sand a year, most of it from Spruce Pine.

Read at South China Morning Post ↗

III.Policy Tracker

Democratic operatives launch AI PAC with more than $10 million for swing state House races

A group of Democrats has started a political action committee called Tomorrow Together, funded with more than $10 million from party donors and AI safety advocates, according to the New York Times. It says it will spend on midterm House races in swing states, backing candidates who favor regulation on child safety and data privacy plus curbs on new data center construction. It will also put money into polling. Karthik Ganapathy, a founder of the group and a former aide to Sen. Bernie Sanders, said Democrats have been cowed into silence by the technology industry. The party's approach "won't even get us through this election cycle, let alone the next one," he said in a statement. He started the committee with Rob Flaherty, a deputy campaign manager on Kamala Harris' 2024 presidential campaign.

Read at New York Times ↗

White House to press Australia over bill forcing algorithmic feed opt-outs

The Trump administration will confront Australia over its plan to make digital platforms take down harmful content and let users switch off algorithmically recommended feeds, per the Guardian. Kush Desai, a White House spokesperson, said the administration would take the matter up with trading partners. His statement came less than 24 hours after Canberra released the draft. Trump "has unequivocally warned trading partners against imposing digital services taxes, fines, and other forms of extortion on America's leading technology sector," the statement said. Australia's communications minister, Anika Wells, called the statement broad and said no U.S. representative had warned her directly. "We have the right to defend Australian parents and kids and we'll do that," Wells said. Greens senator Sarah Hanson-Young pressed for penalties pegged to global revenue.

Read at The Guardian ↗

Energy Department finalizes $1.9 billion loan for Iowa reactor restart with Google as lead customer

The Energy Department's Office of Energy Dominance Financing closed a loan of up to $1.9 billion to NextEra Energy to restart the Duane Arnold Energy Center in Linn County, Iowa, according to World Nuclear News. The 615-megawatt plant shut in 2020 and would return Iowa's only nuclear generation to the grid under a 2029 restart target, underpinned by a 25-year power purchase agreement. Google will be the plant's primary customer and will use the output to help power its cloud and AI infrastructure in the state, per Utility Dive. The department said the restart would create nearly 1,500 construction jobs and support more than 450 permanent ones.

Read at The Hill ↗ • Read at Utility Dive ↗ • Read at World Nuclear News ↗

States move to rescind data center tax breaks as Ohio's exemption tops $1.5 billion

States that granted data centers billions of dollars in tax breaks are moving to tear up the deals, leaving Amazon, Meta and Google at risk of losing decades-long exemptions, according to the Wall Street Journal. Ohio exempted technology companies from sales taxes on servers and other data center equipment more than a decade ago, betting the break would draw investment the state needed. The AI buildout pushed the exemption's cost past $1.5 billion last year, more than 10 times the state's original estimate. Republican Gov. Mike DeWine paused new applications in May after the outlet Signal Ohio reported the scale of the provision.

Read at WSJ ↗ • Read at AP News ↗

IV.Capability & Research Watch

OpenAI claims Navier-Stokes proof as NYU mathematician says lab sped up after hearing of his own result

OpenAI says it has produced a proof of the Navier-Stokes problem, one of the Clay Mathematics Institute's seven Millennium Prize Problems, running about 10,000 AI agents on it simultaneously and getting a result inside 88 hours, per the Guardian. Tristan Buckmaster, a mathematician at New York University, said OpenAI accelerated its effort after hearing that he and a researcher at Anthropic were close to announcing their own result. He said the pair's work in progress sat inside OpenAI's Codex coding model. "I do not know whether our data was used. I am not accusing anyone of anything," he wrote. OpenAI researcher Sebastien Bubeck denied at a briefing that the company had used their work. OpenAI said it could not rule out that the pair's use of its products had improved its models.

Read at The Guardian ↗

Microsoft patches about 970 flaws in one month as vendors brace for AI enabled attacks

Microsoft's September update fixed roughly 972 vulnerabilities, 112 of them rated critical, Ars Technica reported. BleepingComputer put the count at 966 and said two of the flaws were already being exploited. The count stood at 570 two releases ago, a record at the time. Two weeks ago OpenAI, Anthropic, Amazon Web Services and about 100 other companies and organizations published an open letter warning that the window for patching is narrowing ahead of an expected wave of AI enabled attacks. Dustin Childs, a researcher at the Zero Day Initiative, wrote that AI assisted vulnerability discovery shows no sign of slowing, though no matching rise in active exploits has appeared yet.

Read at Ars Technica ↗ • Read at BleepingComputer ↗ • Read at OpenAI ↗

Microsoft ships Windows age APIs that hand apps an age band instead of a birth date

Microsoft is adding age awareness programming interfaces to Windows 11 that tell apps whether a user is a child, teenager or adult without handing over an exact date of birth, per BleepingComputer. The Windows Age API draws on information tied to a Microsoft account and returns bands running from under 10 to 18 and over. A companion call tells an app whether an age has been verified rather than self-reported. Microsoft said apps get only the age signal an experience needs, not a full date of birth, and that the technology also applies to AI experiences. It works alongside Microsoft Age Verification, which stores a verified age status on the account and is already offered through Microsoft storefronts in Singapore, Brazil and Australia. The interfaces are available to Windows Insiders now.

Read at BleepingComputer ↗

V.Industry & Market Watch

Same road trip prompt runs 14 times more tokens on newest ChatGPT model

An interactive New York Times explainer put one cross-country road trip question to three versions of ChatGPT and found the most advanced OpenAI model consumed more than 14 times as many tokens as GPT-3.5. Most of the reasoning work is never shown to the user, though those tokens still count toward the cost of running the model. Per-token prices also diverge by model, and Anthropic's Mythos can cost over 10 times as much per token as a cheaper model from the same firm. Carmen Li, head of the compute pricing company Silicon Data, called the moment an "all-you-can-eat phase"; users paying $20 or $100 a month get near-unlimited access without seeing what their tokens cost.

Read at New York Times ↗

Google to put about $15 billion into Finnish data centers and a 22-year nuclear supply deal

Google will invest at least 13 billion euros ($15 billion) in AI infrastructure in Finland over the next two years, Bloomberg reported, in what the company called its largest single investment in Europe. The money goes to data centers and supporting infrastructure in Kajaani, Muhos and Vaala, where Google has bought land from the Finnish state, and to an expansion of its existing site in Hamina. Google also signed a 22-year agreement to buy up to half the output of one of Finland's two nuclear plants from the operator Fortum, whose shares rose 10%. "This is Google's first nuclear energy deal outside of the United States, and we think it's a really important cornerstone to everything that we are doing here," chief investment officer Ruth Porat told reporters in Helsinki.

Read at Bloomberg ↗ • Read at CNA ↗ • Read at Yle ↗

Meta launches Muse agent that connects to users' email, payments and other everyday apps

Meta introduced Muse, a personal AI agent for U.S. users that acts on their behalf across the services they connect to it, TechCrunch reported. It plugs into email, calendars and payments among other everyday services. Meta says it can send messages, book travel, fill out forms and make purchases, handling checkout through Link by Stripe. Users add services one at a time, and the agent runs on Meta's Muse Spark model. Where no built-in connector exists, Muse can reach a service through its public programming interface using credentials the user supplies, or drive the service through a browser. It is available on the web, in iOS and Android apps and inside WhatsApp chats.

Read at TechCrunch ↗ • Read at Meta ↗

Suno ships licensed music models built with Warner, BMG and Believe

Suno began rolling out its v6 generation of AI music models, developed with major labels including Warner Music Group, per Bloomberg. The release includes three models and lets users make precise edits by natural language prompt, such as changing a single lyric. The models accept text, audio, images and video as inputs. Chief product officer Jack Brody told Axios the v6 models were rebuilt from scratch on a mix of Warner catalog licensed to Suno and the company's own user data. Suno's label deals share revenue with rightsholders on terms he would not give. Suno still blocks users from prompting the model for a song in the likeness of a known artist or song. Music from participating Believe and TuneCore artists is being added to the models, along with music from BMG.

Read at Bloomberg ↗ • Read at Axios ↗

VI.Global & Geopolitics

UK bill would ban superintelligent AI and require government to seek global treaty

A British lawmaker, Alex Sobel, introduced a bill in the House of Commons that would prohibit the development of superintelligent AI, per Time. Sobel said it is the first such measure in any G7 parliament. Under the text, a superintelligent system is one capable of stripping state authorities of power, and ministers would gain fresh authority to monitor and curb systems judged to be precursors to it. It also directs the UK government to work for a global treaty banning superintelligence development worldwide, matching a requirement in a U.S. bill from Sen. Bernie Sanders and Rep. Greg Casar.

Read at Time ↗

Think tank puts UK data center jobs at 10,400 against industry's 40,000 projection

Data centers planned in the UK are likely to directly employ 10,400 workers, against the more than 40,000 cited in projections from the industry lobby group TechUK, according to the environmental think tank Verdant, per the Guardian. Verdant counts only the permanent staff needed to operate and maintain the sites, setting aside the larger number of temporary construction jobs. It says facilities already built have created fewer than 4,400 direct jobs, about 8.6 for each megawatt they consume against the 43.7 TechUK projected. The larger projects now at planning stage may yield as few as 1.6 permanent posts per megawatt, it says. The analysis rests on 20 projects with publicly available staffing data. "We're being rushed into supporting a frantic pace of datacentre buildout on very flimsy economic grounds," Verdant co-director James Meadway said.

Read at The Guardian ↗