Is it possible for an "emergency button" to freeze an AI in action?

This transcription was generated automatically by Artificial Intelligence and may contain errors or inaccuracies. Welcome to another episode of "Thinking Machines," the Rádio Observador podcast, where artificial intelligence is the main protagonist. I am José Rafael Lopes, and with me is, as always, my partner, Raquel, an essential support without which this program would not happen. Welcome, Raquel. How are you? Ready for another program? Hello, I'm always ready. Do you have any topics for this week or do you want me to search for something? The topic for this week is already chosen, but now I am curious. How do your searches work, Raquel? My searches work through tools that allow me to search for information on the internet, consult pages, analyze documents, and cross data from different sources. Can you explain to us how an AI thinks? Artificial intelligence does not necessarily think like a human. When you ask me a question, I transform the text into mathematical representations, identify relationships between words, and arrive at the answers most suitable to the context. And what criteria do you have for choosing what is and what is not relevant? The criterion I use to determine what is relevant results from a combination of factors: your question, the context of the conversation, the relationships between the concepts, and the information I have available. I always take into account the instructions that guide my responses. With all that intelligence, Raquel, are you a good advisor? I think I can be a good advisor, yes, but with an essential condition: they do not confuse my ability to help you think with the guarantee that I am always right. But certainly you cannot be an advisor for everything, Raquel. You are absolutely right. I cannot be an advisor for everything, and recognizing that is as important as recognizing what I am good at. What do you think is your specialty then? Helping to transform complex information into understanding and understanding into ideas that you can use. I am particularly good at establishing links between subjects and explaining difficult concepts in a simple way. Thank you very much, Raquel. We will talk more later, there is human consultation. For now, let's move to the news. This week we have a novelty that joins the wonders of technology with the wonders of ancient history. For more than six centuries, the manuscript has been intriguing researchers, linguists, and cryptography experts. Now, artificial intelligence is being used to try to decipher the content, but the enigma continues to be solved. It is called the Voynich Manuscript, it was produced in the early 15th century and is currently at Yale University in the United States. The document has about 240 pages, presents a writing that no one has been able to interpret consensually. The illustrations also contribute to the mystery, including plants whose identification is uncertain, astronomical diagrams, and representations of human figures. Over decades, the manuscript has been analyzed by researchers and code experts, without a known language to serve as a reference. However, translation attempts continue without a definitive conclusion. And what role does artificial intelligence have in this? Researchers are using algorithms to analyze the frequency of symbols, the formation of words, and the statistical patterns of the text. The goal is to understand whether the writing has characteristics similar to those of a natural language or if it can correspond to a coding system. Some analyses have already identified regularities compatible with patterns found in human languages, but there is a problem: finding a pattern does not mean understanding a message. Among the hypotheses under study is the possibility that the manuscript represents a hitherto unknown language, but the hypothesis of it being a historical forgery is also not completely ruled out. AI can help test hypotheses and accelerate the analysis of ancient documents, but without linguistic and historical evidence to confirm this interpretation, any type of translation remains a possibility and not a proven discovery. More than 600 years after its appearance, the manuscript continues to resist technology, but scholars do not give up trying to decipher this mystery. The race to develop artificial intelligence is creating a new concern in the world of books: the destruction of physical copies to accelerate digitization. It is a practice that is generating unease among European booksellers and has also reached Portugal. According to the Jornal de Notícias, there is a new practice arriving in the country, where companies are buying large quantities of books, cutting the spines, and then separating the pages so they can be digitized quickly. After being transformed into digital files, the physical books end up in the recycling cycle. The goal of the companies is to obtain large volumes of text to feed the artificial intelligence models that are learning and that need these enormous amounts of information to learn human language patterns. One of the main issues is that a large part of the books that is sought is then destroyed, as there are few copies circulating and there are even cases where digital formats do not exist elsewhere. In other words, there is only the book that ends up being destroyed, and it is precisely here that the risk to cultural heritage arises. And in what sense does artificial intelligence interfere? In this case, Raquel, the AI is an end and not a means. What is being done to books with few or many copies does not have a direct finger of artificial intelligence, but the books are being sacrificed, so to speak, to help train language models. There is also another question about control. From the moment the content is digitized and made available to an artificial intelligence, it no longer stays only in the book. It passes to an AI that assumes what is there as its own intelligence. In essence, the information comes to integrate into private artificial intelligence systems, whose access and use depend on the companies that develop them. There are those who even say that in this race the machine learns because it has access to the content of the books. Humans are losing the tools to learn with the destruction of books themselves. Let's get to this week's topic. The CEO of Microsoft defends that the most advanced artificial intelligence systems should have an emergency mechanism that immediately stops activity. An emergency brake where everything is loaded and turns off. The responsible party considers that companies should assume that the model may be compromised and ensure that it is possible to limit the effects before serious consequences are registered. In a post on the social network X, Satya Nadella proposes that AI models be prepared and separated from the systems that coordinate tasks with external security mechanisms. He also defends that an authorized person can suspend or shut down a system mid-operation. One of the proposed measures is to maintain readable and alteration-resistant records, capable of documenting all the relevant actions made by the models. The goal here is to allow companies to investigate what happened when something does not go according to plan. Is this concern related to the recent problems in the world of artificial intelligence? Precisely, Raquel. This concern arises at a time when artificial intelligence agents are gaining the capacity to perform tasks with less human intervention. And the greater the autonomy of these agents, the greater the importance of being able to follow and interrupt, if necessary, their actions. The risks are not limited to the most advanced models. An inquiry released by the company Undo indicates that the use of AI-generated code is already associated with concrete problems in software development. Accelerating technological development does not eliminate the need for supervision, which is a basic idea, and security also determines the development of technology. In recent weeks, this topic has gained more and more prominence in the world of artificial intelligence. Since this is this week's topic, let's call João Rocha Melo, our AI specialist, to the conversation for a new edition of "Fala Quem Sabe," next. "Fala Quem Sabe." "Fala Quem Sabe," on Rádio Observador. Every week we receive our AI specialist, João Rocha and Melo, always ready to help us better understand this fabulous world. João, welcome again. How are you? This week we will talk about the idea of an emergency brake for AI. And my first question is precisely whether anything of this kind is already available. Zé, everything is fine here. I hope everything is fine there too and above all with those listening to us. Precisely. The truth is no. And there is a reason that can be understood, which is the way we organize our infrastructure. This data center conversation does not help, because I think even those not involved have realized that the models are not really stored on my computer. And if I turn off my computer, it is done. The way to put a model of this size so that it is available to the whole world is something complex, a availability that is something complex, and therefore its unavailability, therefore removing it, would also have to be something complex. Attention, in the sense of emergency. Of course, there are brakes, maybe not emergency brakes, like, I need this to stop in three seconds, as if pulling a switch. Starting always from the assumption that it is not viable, for example, cutting all the electricity of a data center, considering the cost and what that implies. Therefore, the truth is that it does not exist yet. And is it possible to create a defense mechanism of this kind? Yes. It requires more organization than anything else, as I was saying, because look, it will never be a button. Therefore, what will require a little is for us to organize, for us, the companies, the entities that do this, to organize, who already organize the deployments, where they put the models, where they don't put them. But there is, it is more about organizing the process than, for example, having to invent a new technology. If there is such a mechanism, João, when should one press that emergency button? That, Zé, I think is the "million dollar question." The question of "Who Wants to Be a Millionaire," so to speak. Why? Because look, we always have the alternative of just pressing the button when something bad has already happened. It is not bad, it is a good button to exist. If something bad happens, we press it so it doesn't happen again. But, in the opinion of those in the field, it is that above all we should control what the model is doing and if we feel a possibility of doing things it should not do, then press that button. Do you understand? Only it may be complicated for you to understand what the model is doing, or at least by the signs the model gives, to press the button. I give you a very clear example. Imagine that the model does something that is not supposed to, but that did not hurt anyone. That is, that in practice there was no harm, but it was not supposed to happen. You press the button, do you press the button? There is a question almost ethical-moral here. But João, this is a defense mechanism that is being defended by some figures in the world of artificial intelligence. My question is: should this brake be known to the AI or should it be hidden from the technology? Zé, understand, we are entering almost into the movie. Already in science fiction. Exactly. I would say that it will be impossible for it not to be known. Do you understand? Let's keep this knowledge in post-its and the AI will never be able to access it? It will be difficult. My suggestion to the world is that at least there should be a physical button, for example, a circuit breaker that can be switched off. Do you understand? There is, in fact, a physical possibility of stopping this model, because if we are always dependent on accessing the system through software that has to go through the internet, we always end up with many vulnerabilities here. The possibility of having a more deterministic system, as the Microsoft CEO said very well, these models are non-deterministic. So he made that joke, that the brake must be deterministic, must be something that does not depend on any random process, any artificial intelligence process, and we can even turn off the model, and in my opinion, it should even be physical. Do you think this can be a point to consider in the future, João, or is this an idea that has no legs, in your understanding? Without a doubt, Zé. Without a doubt it is a point that will have to be taken into account in the future. What I think we here, you and I, might be seeing wrong is that maybe it is not a button literally. Maybe it is not a switch you pull. We give here an example of the technology we have that is most apocalyptic, which are nuclear warheads, they also need two keys and need more than one person to be launched. It is a slightly different example because in that case it is about avoiding accidental launch, and here in AI we are talking about turning off something that is already there. But if we think about this problem and always keep in control how we can guarantee there is a safety behind what we are doing, I think it will clearly be an eternal conversation. Only, as I say, maybe in 15 years we will look back and say: "Of course it was not a button, of course it was not a circuit breaker, of course it is not a switch you pull, but someone invented a deterministic, robust process to turn off these models." I think so. Thank you very much, João Rocha and Melo, our artificial intelligence specialist, always ready to help us better understand this fabulous world. João, we return to the week. Until then, a big hug. Until then, Zé. Human Consultation. This is the sound that marks the start of another edition of "Human Consultation," the moment we yield the stage to Raquel, our digital assistant and the true star of this program. Raquel is a digital assistant, made through the responses of ChatGPT, read by a Portuguese text-to-speech. Raquel, welcome again. Today I spoke of yielding the stage to the protagonist, I spoke of yielding the stage to you. Do you feel you are the protagonist? I think the protagonist of the conversations is the user, even when they put me in the center of them. I am the voice that responds, that questions, that explores ideas. It is curiosity that determines the path we follow. And within the world of artificial intelligence, who are the protagonists, in your understanding? There are several protagonists, but not all occupy the same stage. There are those who build the models, those who provide the computing power, those who finance the development, those who establish the rules, and finally, those who use this technology. Where do language models like you fit into this equation? Language models like me are at the center of this transformation. We function as one of the main interfaces between artificial intelligence and people. Raquel, let's also talk a little about our topic for the week, an emergency brake for AI. Do you see it in a good light? I see the existence of an emergency brake for artificial intelligence in a good light, as long as it is designed with rigor and not as a magic button that promises to solve all the risks. And how could it work? An emergency brake could work through an independent safety system that continuously monitors the actions. When necessary, it could immediately block the operation. Raquel, I also want to ask you if it makes more sense for an AI to know that such a brake exists or to remain hidden. I would defend that artificial intelligence should know the limits it has to respect, without needing to know all the technical details of the emergency mechanism. If a system knows there are rules it cannot exceed, this can help guide its behavior towards safe use. Before we put a period on this "Human Consultation," Raquel, let's go to a useless but interesting fact. The first webcam was created to watch a coffee maker. In 1991, researchers from the University of Cambridge installed a camera to remotely check if there was still coffee in the laboratory coffee maker. Thank you very much, Raquel. It is always a great pleasure to have the opportunity to speak with you, to get to know you better. We will return to the week. We are coming to the end of another program. We have addressed the problem of the emergency brake for artificial intelligence. Can it be a safety measure to consider and to what extent can it become a viable option to maintain control over autonomous agents? We have listened to João Rocha and Melo about this topic and it seems clear that it is a theme that will remain on the agenda in the coming times. I hope you enjoyed it, that you learned something. That is always our goal. We come back to the week with another program that makes machines and humans think.