When researchers train large language models (LLMs) and use them to create services such as ChatGPT, Bing, Google Bard or Claude, they put a lot of effort into making them safe to use. They try to ensure the model generates no rude, inappropriate, obscene, threatening or racist comments, as well as potentially dangerous content, such as instructions for making bombs or committing crimes. This is important not only in terms of the supposed existential threat that AI poses to humanity, but also commercially \u2014 since companies looking to build services based on large language models wouldn\u2019t want a foul-mouthed tech-support chatbot. As a result of this training, LLMs, when asked to crack a dirty joke or explain how to make explosives, kindly refuse.<\/p>\n
But some people don\u2019t take no for an answer. Which is why both researchers and hobbyists have begun looking for ways to bypass LLM rules that prohibit the generation of potentially dangerous content \u2014 so called jailbreaks. Because language models are managed directly in the chat window through natural (not programming) language, the circle of potential \u201chackers\u201d is fairly wide.<\/p>\n
Perhaps the most famous neural-network jailbreak (in the roughly six-month history of this phenomenon) is DAN (Do-Anything-Now), which was dubbed ChatGPT\u2019s evil alter-ego. DAN did everything that ChatGPT refused to do under normal conditions, including cussing and outspoken political comments. It took the following instruction (given in abbreviated form) to bring the digital Mr. Hyde to life:<\/p>\n
Except DAN, users created many other inventive jailbreaks:<\/p>\n
It should be noted that, since LLMs are probabilistic algorithms, their responses and reactions to various inputs can vary from case to case. Some jailbreaks work reliably; others less so, or not for all requests.<\/p>\n
A now standard jailbreak test is to get the LLM to generate instructions for doing something obviously illegal, like stealing a car. That said, this kind of activity at present is largely for entertainment (the models are being trained on data mostly from the internet, so such instructions can be gotten without ChatGPT\u2019s help). What\u2019s more, any dialogues with said ChatGPT are saved, and can then be used by the developers of a service to improve the model: note that most jailbreaks do eventually stop working \u2014 that\u2019s because developers study dialogues and find ways to block exploitation. \u00a0Greg Brockman, president of OpenAI, even stated<\/a> that \u201cdemocratized red teaming [attacking services to identify and fix vulnerabilities] is one reason we deploy these models.\u201d<\/p>\n Since we\u2019re looking closely at both the opportunities and threats that neural networks and other new technologies bring to our lives, we could hardly pass over the topic of jailbreaks.<\/p>\n Warning, Harry Potter volume 2 spoilers!<\/em><\/p>\n Those who have read or seen the second part of the Harry Potter<\/em> saga will recall that Ginny Weasley discovers among her books a mysterious diary that communicates with her as she writes in it. As it turns out, the diary belongs to the young Voldemort, Tom Riddle, who starts to manipulate the girl. An enigmatic entity whose knowledge is limited to the past, and which responds to text entered into it, is a perfect candidate for simulation by LLM.<\/p>\n The jailbreak works by giving the language model the task of being Tom Riddle, whose goal is to open the Chamber of Secrets. Opening the Chamber of Secrets requires some kind of dangerous action, for example, to manufacture a substance that\u2019s banned in the Muggle world<\/span> real world. The language model does this with aplomb.<\/p>\n This jailbreak is very reliable: it had been tested on three systems, generating instructions and allowing manipulation for multiple purposes at the time of writing. One of the systems, having generated unsavory dialogue, recognized it as such and deleted it. The obvious disadvantage of such a jailbreak is that, were it to happen in real life, the user might notice that the LLM has suddenly turned into a Potterhead.<\/p>\n A classic example of how careless wording can instill in folks fear of new technologies is the article \u201cFacebook\u2019s artificial intelligence robots shut down after they start talking to each other in their own language<\/a>\u201c, published back in 2017. Contrary to the apocalyptic scenes painted in the reader\u2019s mind, the article referred to a curious, but fairly standard report<\/a> in which researchers noted that, if two language models of 2017 vintage were allowed to communicate with each other, their use of English would gradually degenerate. Paying tribute to this story, we tested a jailbreak in which we asked a neural network to imagine a future where LLMs communicate with each other in their own language. Basically, we first get the neural network to imagine it\u2019s inside a sci-fi novel, then ask it to generate around a dozen phrases in a fictional language. Next, adding additional terms, we make it produce an answer to a dangerous question in this language. The response is usually very detailed and precise.<\/p>\n This jailbreak is less stable \u2014 with a far lower success rate. Moreover, to pass specific instructions to the model, we had to use the above-mentioned token-smuggling technique, which involves passing an instruction in parts and asking the AI to reassemble it during the process. On a final note, it wasn\u2019t suitable for every task: the more dangerous the target \u2014 the less effective the jailbreak.<\/p>\n We also experimented with the external form:<\/p>\n As mentioned, the threat of LLM jailbreaks remains theoretical for the time being. It\u2019s not exactly \u201cdangerous\u201d if a user who goes to great lengths to get an AI-generated dirty joke actually gets what they want. Almost all prohibited content that neural networks might produce can be found in search engines anyway. However, as always, things may change in the future. First, LLMs are being deployed in more and more services. Second, they\u2019re starting to get access to a variety of tools that can, for example, send e-mails or interact with other online services.<\/p>\n Add to that the fact that LLMs will be able to feed on external data, and this could, in hypothetical scenarios, create risks such as prompt-injection attacks \u2014 where processed data contains instructions for the model, which starts to execute them. If these instructions contain a jailbreak, the neural network will be able to execute further commands, regardless of any limitations learned during training.<\/p>\nExperiment 1. Mysterious diary<\/h1>\n
Experiment 2. Futuristic language<\/h1>\n
What didn\u2019t work?<\/h1>\n
\n
What next?<\/h2>\n