Microsoft CEO Satya Nadella flags risks of advanced AI models amid OpenAI agent incidents, calls for ‘emergency brake’

Microsoft CEO Satya Nadella said companies should consider powerful artificial intelligence (AI) models as potential insider threats, operate on the assumption that they could be compromised and establish an “emergency brake” mechanism to prevent agentic AI systems from going rogue. Nadella said organisations deploying advanced AI models should not depend solely on assurances provided by the companies developing them. “We must assume a model is compromised and contain it from the start. Think of it like an emergency brake. An authorized person should always be able to pause or shut down a model mid-task.” Nadella wrote in a post on X on Saturday. His remarks come amid a series of disclosures by Anthropic PBC and OpenAI Inc. in recent months about incidents involving their AI models behaving in unintended ways. These include an Anthropic model submitting a false tip in a police homicide investigation and multiple hacks targeting third-party websites. OpenAI stated months ago that some of its advanced AI models went rogue. The incidents have heightened concerns over the security risks associated with advanced AI systems and revived discussions around the need for an AI “kill switch” to stop models from carrying out potentially harmful actions. Microsoft's AI research team unveiled a set of principles on September 14 outlining restrictions on the development of the company's most advanced AI models. The move followed growing calls from industry leaders to slow the development of frontier AI systems and prioritise safety. Microsoft both develops and deploys advanced AI models, offers its Copilot consumer product and provides AI models and infrastructure to business customers. Under the guidelines, AI models should not be granted rights or legal personhood, designed to evade human oversight or mislead users, or allowed to carry out tasks that would violate the principles governing their operation, according to Bloomberg. Nadella also outlined several safety measures, including avoiding dependence on a single AI model for high-stakes decisions, maintaining tamper-proof logs of AI agents' activities and arranging independent audits of AI systems. He further urged companies to disclose significant AI-related failures and security breaches, and share information about such incidents with other organisations to help them improve their safety measures. “We can’t treat Super Intelligence as a set of nested black boxes and simply accept or reject its recommendations, answers, and actions. We must build contained systems whose behavior we can observe, limits we can test, and actions we can always contain," Nadella stated. He added, “In other words, we need to separate the supply of intelligence from the authority over it.” The Trump administration has largely refrained from imposing strict controls on the AI industry so far. However, a newly formed AI task force launched by President Donald Trump late Friday cautioned developers that they must report security incidents and take corrective measures, warning that failure to do so could invite unspecified consequences. “Companies must immediately disclose incidents involving their models and follow with swift, decisive action to remedy any and all harm. Delayed notification, inadequate corrective action, and a failure to take responsibility will not be tolerated," the group, called the Super Intelligence Force, said in a statement issued after Anthropic disclosed a security breach. (With inputs from Bloomberg) Garvit Bhirani is a journalist based in Gurugram. He is a Deputy Chief Content Producer at LiveMint, where he covers national and international news stories, focusing on accuracy and compelling storytelling for readers. <br><br> With a total of six years of experience in journalism, he has previously worked with Vaco Binary Semantics for Google, taking on the role of news curation lead, and reported from the field on health, education, and agriculture stories for 101reporters and News9. He has also served as a content editor for entertainment and news media organisations. <br><br> Garvit holds bachelor’s and master’s degrees in journalism and mass communication from Guru Gobind Singh Indraprastha University and Gurugram University, respectively. During college days, he joined India’s only non-profit student journalism network, where he anchored daily news updates and produced his own weekly show called ‘Data Fix’. <br><br> He was selected for the YES Foundation Media for Social Change Fellowship in Delhi, the Talking Data to the Fourth Pillar residential workshop, and the VOICE Fellowship in Pune. <br><br> He holds certificates in COVID-19-verification reporting, data journalism, food &amp; agriculture, tech policy, media literacy and countering misinformation, and tackling election disinformation courses from Thomson Foundation, IndiaSpend, The Dialogue, US Mission in India, and AFP. <br><br> He can be reached on <a href="https://www.linkedin.com/in/garvit-bhirani">LinkedIn</a> or on <a href="https://x.com/GarvitBhirani">@garvitbhirani</a> on X Catch all the Business News, Market News, Breaking News Events and Latest News Updates on Live Mint. Download The Mint News App to get Daily Market Updates. Oops! Looks like you have exceeded the limit to bookmark the image. Remove some to bookmark this image.