On Thursday, OpenAI released the "system card" for ChatGPT's new GPT-4o AI model that details model limitations and safety testing procedures. Among other examples, the document reveals that in rare occurrences during testing, the model's Advanced Voice Mode unintentionally imitated users' voices without permission. Currently, OpenAI has safeguards in place that prevent this from happening, but the instance reflects the growing complexity of safely architecting with an AI chatbot that could potentially imitate any voice from a small clip.
周四,OpenAI 发布了 ChatGPT 新 GPT-4o AI 模型的“系统卡”,详细说明了模型的局限性和安全测试程序。该文档还披露,在测试过程中,该模型的“高级语音模式”在极少数情况下会在未经许可的情况下无意模仿用户的声音。目前,OpenAI 已采取安全措施来防止这种情况发生,但该实例反映了使用 AI 聊天机器人进行安全架构的复杂性日益增加,因为这种机器人有可能模仿一小段音频中的任何声音。
Advanced Voice Mode is a feature of ChatGPT that allows users to have spoken conversations with the AI assistant.
高级语音模式是 ChatGPT 的一项功能,允许用户与 AI 助手进行语音对话。
In a section of the GPT-4o system card titled "Unauthorized voice generation," OpenAI details an episode where a noisy input somehow prompted the model to suddenly imitate the user's voice. "Voice generation can also occur in non-adversarial situations, such as our use of that ability to generate voices for ChatGPT’s advanced voice mode," OpenAI writes. "During testing, we also observed rare instances where the model would unintentionally generate an output emulating the user’s voice."
在 GPT-4o 系统卡片标题为“未经授权的语音生成”的部分中,OpenAI 详细介绍了一个事件,其中嘈杂的输入以某种方式促使模型突然模仿用户的声音。“语音生成也可能发生在非对抗性情况下,例如我们使用该功能为 ChatGPT 的高级语音模式生成语音,”OpenAI 写道。“在测试过程中,我们还观察到极少数情况下模型会无意中生成模仿用户声音的输出。”
In this example of unintentional voice generation provided by OpenAI, the AI model outbursts “No!” and continues the sentence in a voice that sounds similar to the "red teamer" heard in the beginning of the clip. (A red teamer is a person hired by a company to do adversarial testing.)
在 OpenAI 提供的这种无意语音生成的示例中,AI 模型突然说出“不!”,并以一种听起来与剪辑开头听到的“红队成员”相似的声音继续说出这句话。(红队成员是指公司聘请来进行对抗性测试的人。)
It would certainly be creepy to be talking to a machine and then have it unexpectedly begin talking to you in your own voice. Ordinarily, OpenAI has safeguards to prevent this, which is why the company says this occurrence was rare even before it developed ways to prevent it completely. But the example prompted BuzzFeed data scientist Max Woolf to tweet, "OpenAI just leaked the plot of Black Mirror's next season."
与机器交谈时,如果它突然开始用你自己的声音与你交谈,那肯定会令人毛骨悚然。通常情况下,OpenAI 采取了安全措施来防止这种情况发生,这就是为什么该公司表示,即使在开发出完全阻止这种情况发生的方法之前,这种情况也很少发生。但这个例子促使 BuzzFeed 数据科学家 Max Woolf 在推特上写道:“OpenAI 刚刚泄露了《黑镜》下一季的剧情。”
Audio prompt injections 音频提示注入
How could voice imitation happen with OpenAI's new model? The primary clue lies elsewhere in the GPT-4o system card. To create voices, GPT-4o can apparently synthesize almost any type of sound found in its training data, including sound effects and music (though OpenAI discourages that behavior with special instructions).
OpenAI 的新模型如何进行语音模仿?主要线索在于 GPT-4o 系统卡片的其他地方。为了创建语音,GPT-4o 显然可以合成其训练数据中发现的几乎任何类型的声音,包括声音效果和音乐(尽管 OpenAI 通过特殊指令阻止了这种行为)。
As noted in the system card, the model can fundamentally imitate any voice based on a short audio clip. OpenAI guides this capability safely by providing an authorized voice sample (of a hired voice actor) that it is instructed to imitate. It provides the sample in the AI model's system prompt (what OpenAI calls the "system message") at the beginning of a conversation. "We supervise ideal completions using the voice sample in the system message as the base voice," writes OpenAI.
正如系统卡中所述,该模型可以根据短音频片段从根本上模仿任何声音。OpenAI 通过提供授权语音样本(来自聘请的配音演员)来安全地引导此功能,并指示模型模仿该样本。它在对话开始时将样本提供给 AI 模型的系统提示(OpenAI 称之为“系统消息”)。“我们使用系统消息中的语音样本作为基础语音来监督理想的完成度,”OpenAI 写道。
In text-only LLMs, the system message is a hidden set of text instructions that guides behavior of the chatbot that gets added to the conversation history silently just before the chat session begins. Successive interactions are appended to the same chat history, and the entire context (often called a "context window") is fed back into the AI model each time the user provides a new input.
在纯文本 LLMs 中,系统消息是一组隐藏的文本指令,用于指导聊天机器人的行为,这些指令会在聊天会话开始之前静默地添加到对话历史中。后续的交互会被附加到同一个聊天历史记录中,并且每次用户提供新的输入时,整个上下文(通常称为“上下文窗口”)都会被反馈到 AI 模型中。
(It's probably time to update this diagram created in early 2023 below, but it shows how the context window works in an AI chat. Just imagine that the first prompt is a system message that says things like "You are a helpful chatbot. You do not talk about violent acts, etc.")
(可能是时候更新下面这张在 2023 年初创建的图表了,但它展示了上下文窗口在 AI 聊天中的工作原理。想象一下,第一个提示是一条系统消息,内容类似于“你是一个乐于助人的聊天机器人。你不会谈论暴力行为等。”)
Since GPT-4o is multimodal and can process tokenized audio, OpenAI can also use audio inputs as part of the model's system prompt, and that's what it does when OpenAI provides an authorized voice sample for the model to imitate. The company also uses another system to detect if the model is generating unauthorized audio. "We only allow the model to use certain pre-selected voices," writes OpenAI, "and use an output classifier to detect if the model deviates from that."
由于 GPT-4o 是多模态的,可以处理标记化的音频,因此 OpenAI 还可以使用音频输入作为模型系统提示的一部分,这也是 OpenAI 在为模型提供授权语音样本以供模仿时所做的。该公司还使用另一个系统来检测模型是否正在生成未经授权的音频。OpenAI 写道:“我们只允许模型使用某些预先选择的语音,并使用输出分类器来检测模型是否偏离了这些语音。”
reader comments 读者评论
106