Former AI employees warn about risks as companies face safety questions

NEW YORK--Former employees of OpenAI, Anthropic and Google DeepMind warned New York City lawmakers on Monday that artificial intelligence companies are advancing increasingly powerful technology faster than their ability to fully understand or control its risks.

The warnings came during a New York City Council hearing on artificial intelligence safety on October 5, where former industry employees appeared alongside representatives of OpenAI, Anthropic, Google and Meta. Council members are considering a package of measures aimed at increasing transparency, outside oversight and protections for employees who raise concerns about AI safety.

Among those testifying were Jacob Coxon, a former Anthropic engineer who previously worked at OpenAI, former OpenAI researcher Daniel Kokotajlo and Alex Turner, who left Google DeepMind. Their testimony focused on concerns that competition among leading AI developers could encourage companies to prioritize increasingly capable systems before safety questions have been adequately resolved.

Coxon warned that the startup culture of moving quickly and fixing problems later was poorly suited to the development of advanced artificial intelligence. He argued that AI could bring substantial benefits, but said stronger transparency and safeguards are necessary to reduce the chances of systems developing in ways humans cannot control.

Kokotajlo raised concerns about the industry's ability to identify when advanced AI systems are behaving in unintended ways. Turner also questioned whether voluntary company safeguards are sufficient as AI capabilities expand and companies compete for technological leadership.

Representatives from OpenAI, Anthropic, Google and Meta also addressed the Council and defended the safety work being undertaken inside their companies. They said AI is already delivering benefits while acknowledging that increasingly powerful systems require safeguards and appropriate oversight.

The hearing became particularly pointed when New York City Council Speaker Julie Menin asked company representatives to quantify the possibility of a worst-case AI catastrophe. OpenAI representative Morgan Dwyer declined to assign a specific percentage, but said any meaningful possibility of catastrophic failure should be considered unacceptable and that companies should not develop models unless they can make a strong case that the systems can remain under human control.

The City Council is considering proposals that could require greater disclosure about AI systems, outside review of certain technologies used in the city and stronger incentives for whistleblowers to report violations.

The hearing reflects a broader push by state and local governments in the United States to develop their own AI rules as questions continue about how much oversight should come from government and how much responsibility should remain with the companies building the technology.

The testimony also brought unusual public attention to concerns coming from inside some of the world's leading AI laboratories, placing the debate over AI safety, corporate responsibility and government oversight directly before elected lawmakers.

‍