Verco: Learning Coordinated Verbal Communication for Multi-agent Reinforcement Learning
Verco：多智能体强化学习中的协调性语言交流学习

LLMs trained on large datasets have shown remarkable abilities in various downstream tasks. Recent works have gradually applied LLMs to complex tasks such as robot control [23, 24], planning generation [25], embodied agent [26, 27, 28, 29], etc. Among them, SayCan [26] and LLM+P [25] use the LLM as a high-level planner to generate long-term plans for agents based on task goals but often do not directly interact with the environment. Voyager [29] uses GPT-4 to accomplish highly complex tasks in the Minecraft game and continuously generates new skills during the learning process. ReAct [30] combines reasoning and action to enhance LLMs’ reasoning ability for high-level goals, while improving the interpretability and credibility of LLMs’ decisions. As a supplement, [31] added additional reporters to provide useful information for the planner. However, these methods cannot be improved based on environmental feedback, making it difficult to adapt to different environments. In recent works, GLAM [21] and TWOSOME [20] both use reinforcement learning (RL) to enable LLMs to interact and learn in a single-agent environment, and the rich prior experience of LLMs itself can significantly reduce the learning cost of agents.
经过大规模数据集训练的模型在多种下游任务中展现出卓越能力。近期研究逐步将此类模型应用于机器人控制[23, 24]、规划生成[25]、具身智能体[26, 27, 28, 29]等复杂任务。其中，SayCan[26]与LLM+P[25]采用LLM作为高层规划器，依据任务目标为智能体制定长期计划，但通常不直接与环境互动。Voyager[29]利用 GPT-4 在 Minecraft 游戏中完成高度复杂任务，并在学习过程中持续生成新技能。ReAct[30]结合推理与行动，增强LLMs针对高层目标的推理能力，同时提升LLMs决策的可解释性与可信度。此外，[31]增设额外报告者，为规划器提供有用信息。然而，这些方法难以基于环境反馈进行优化，导致适应不同环境的能力受限。 在近期研究中，GLAM[21]与 TWOSOME[20]均采用强化学习（RL）技术，使LLMs能在单一代理环境中互动学习，而LLMs自身丰富的先前经验能显著降低代理的学习成本。

In recent works, finetuning has been employed to improve the performance of LLMs in specific tasks across different domains. Among these approaches, the use of RL to enhance the consistency between the LLM and human preferences is a common practice [32]. Many reinforcement learning with human feedback (RLHF) methods utilize PPO [33] to learn a reward function from human datasets and fine-tune LLMs according to the learned reward function. A significant challenge in fine-tuning LLMs is how to reduce training costs. Parameter-efficient finetuning (PEFT)
近期作品中，微调技术被广泛应用于提升LLMs在跨领域特定任务中的表现。在这些方法中，利用强化学习（RL）增强LLM与人类偏好之间的一致性已成为常见做法[32]。众多基于人类反馈的强化学习（RLHF）方法采用 PPO[33]从人类数据集中学习奖励函数，并据此对LLMs进行微调。微调LLMs时面临的一大挑战是如何降低训练成本。参数高效微调（PEFT）¹¹1https://github.com/huggingface/peft. [34, 35] can significantly reduce the number of LLM parameters while avoiding excessive performance loss. As a recent technology in PEFT, low-rank adapters (LoRA) indirectly train the dense layers of neural networks by learning low-rank matrices. TWOSOME [20] fine-tuned a LoRA as the actor model, thus enabling efficient fine-tuning while avoiding mutual interference between actor and critic.
[34, 35]能显著减少LLM参数的数量，同时避免过度的性能损失。作为 PEFT 领域的一项最新技术，低秩适配器（LoRA）通过学习低秩矩阵间接训练神经网络的稠密层。TWOSOME [20]将 LoRA 微调为行动者模型，从而在避免行动者与评判者间相互干扰的同时，实现了高效的微调。

Existing multi-agent cooperation algorithms have made significant progress in many complex scenarios, but the realistic situation of partial observability greatly limits the degree of cooperation among agents. To address this issue, researchers have proposed a series of communication-based cooperation paradigms [15, 18, 19, 36, 17] to facilitate agents’ understanding of the environment and teammates by exchanging local observations or intentions. However, most of the communication messages generated by existing communication algorithms are difficult for humans to understand, thus not interpretable and cannot be improved explicitly. To make the communication messages understandable and ineterpretable, using verbal text as a communication message naturally becomes a solution. The FAMA [37] algorithm extends GLAM to multi-agent settings and introduces a verbal communication module. However, the communication module in the FAMA method cannot be learned through interactions with the environment and can only select from a pre-defined set of message candidates, significantly reducing the degree of message generation freedom and relying on the quality of the pre-specified message candidates.
现有的多智能体协作算法在众多复杂场景中取得了显著进展，但部分可观测性的现实情况极大限制了智能体间的协作程度。为解决这一问题，研究者们提出了一系列基于通信的协作范式[15, 18, 19, 36, 17]，通过交换局部观测或意图来促进智能体对环境和队友的理解。然而，现有通信算法生成的多数通信信息对人类而言难以理解，不具备可解释性，也无法明确改进。为使通信信息变得可理解和可解释，采用口头文本作为通信信息自然成为一种解决方案。FAMA[37]算法将 GLAM 扩展至多智能体环境，并引入了口头通信模块。 然而，FAMA 方法中的通信模块无法通过与环境的互动来学习，只能从预定义的消息候选集中选择，这显著降低了消息生成的自由度，并依赖于预设消息候选集的质量。

We assume a textual MARL setting here with language vocabulary $\mathcal{V}$. The multi-agent problem here can be described as a partially observable Markov game (POMG), which is defined by the following tuple $G=\left<S,U,\mathcal{N}{},\Omega,\mathcal{T},O,r,\gamma,\mathcal{V}\right>$, with $S$ the state space, $U\subset\mathcal{V}$ the action space. At each discrete time step $t$, each agent $i\in\mathcal{N}{}:=\{1,\dots,n{}\}$ will select an action $u_{{i}}\in U_{{i}}\subset\mathcal{V}$. $\mathcal{T}(s^{\prime}|s,\boldsymbol{u)}:S\times U\times S\rightarrow P(S)$ is the state transition function, where $\boldsymbol{u}=\{u_{{1}},\dots,u_{{n{}}}\}\in\boldsymbol{U}\equiv U^{n}{}$ is the joint action. Each agent $i$ can get its textual partial observation $o_{{i}}\in\Omega$ by the observation function $O(s,i):S\times\mathcal{N}\rightarrow\Omega$. $r_{{i}}(s,u_{{i}}):S\times U_{{i}}\rightarrow\mathbb{R}$ is the reward function for each agent $i$. The $\gamma$ is the discount factor. The goal for each agent is to maximize the expected return. In order to meet the demands of communication and decision-making, we designed a dual-LLM structure for agents, enabling them to send text messages based on their local observations while making decisions based on the received messages from other agents. By considering both local observations and teammates’ messages, the agents can select corresponding coordinated actions. Our entire framework consists of three main components: the teacher policy $\pi^{tch}$, the message policy $\pi_{\eta}$, and the action policy $\pi_{\theta}$.
在此，我们假设一个文本多智能体强化学习（MARL）环境，其中语言词汇为 $\mathcal{V}$ 。此处的多智能体问题可描述为部分可观测马尔可夫博弈（POMG），由以下元组 $G=\left<S,U,\mathcal{N}{},\Omega,\mathcal{T},O,r,\gamma,\mathcal{V}\right>$ 定义，其中 $S$ 为状态空间， $U\subset\mathcal{V}$ 为动作空间。在每个离散时间步 $t$ ，每个智能体 $i\in\mathcal{N}{}:=\{1,\dots,n{}\}$ 将选择一个动作 $u_{{i}}\in U_{{i}}\subset\mathcal{V}$ 。 $\mathcal{T}(s^{\prime}|s,\boldsymbol{u)}:S\times U\times S\rightarrow P(S)$ 是状态转移函数，其中 $\boldsymbol{u}=\{u_{{1}},\dots,u_{{n{}}}\}\in\boldsymbol{U}\equiv U^{n}{}$ 为联合动作。每个智能体 $i$ 通过观测函数 $O(s,i):S\times\mathcal{N}\rightarrow\Omega$ 获得其文本部分观测 $o_{{i}}\in\Omega$ 。 $r_{{i}}(s,u_{{i}}):S\times U_{{i}}\rightarrow\mathbb{R}$ 是每个智能体 $i$ 的奖励函数。 $\gamma$ 是折扣因子。每个智能体的目标是最大化期望回报。为满足通信与决策需求，我们为智能体设计了双LLM结构，使其能基于局部观测发送文本消息，并根据接收自其他智能体的消息做出决策。通过综合考虑局部观测与队友消息，智能体可选择相应的协同动作。我们的整体框架包含三个主要组件：教师策略 $\pi^{tch}$ 、消息策略 $\pi_{\eta}$ 及动作策略 $\pi_{\theta}$ 。

We consider using LLMs as interactive agents in embodied environments, the LLMs are trained on massive amounts of text data and can generate human-like responses to questions or prompts. In order to improve the performance of LLMs in various specific tasks, fine-tuning is commonly employed. Among these, LoRA (Low-Rank Adaptation) [22] serves as a popular fine-tuning method, effectively reducing computational costs while maintaining satisfactory performance. Specifically, LoRA training the dense layers with weights matrix, $W_{0}\in\mathbb{R}^{d\times k}$ and injects trainable rank decomposition matrices into the Transformer by $W_{0}+\Delta W=W_{0}+BA$, where $B\in\mathbb{R}^{d\times r},A\in\mathbb{R}^{r\times k}$, since the rank $r$ is much smaller than $d$ and $k$, it can significantly reduce the scale of trainable parameters.
我们考虑将LLMs作为实体环境中的交互代理，这些LLMs经过大量文本数据的训练，能够生成类似人类的回答来响应问题或提示。为了提升LLMs在各种特定任务中的表现，通常采用微调技术。其中，LoRA（低秩适应）[22]作为一种流行的微调方法，能在保持良好性能的同时有效降低计算成本。具体而言，LoRA 通过 $W_{0}\in\mathbb{R}^{d\times k}$ 训练带有权重矩阵的密集层，并通过 $W_{0}+\Delta W=W_{0}+BA$ 向 Transformer 注入可训练的秩分解矩阵，其中 $B\in\mathbb{R}^{d\times r},A\in\mathbb{R}^{r\times k}$ ，由于秩 $r$ 远小于 $d$ 和 $k$ ，它能显著减少可训练参数的规模。

In the context of human collaborative problems, the diversity in the information received by individuals and their distinct modes of thinking often render simple independent actions insufficient for achieving coordinated consensus. Consequently, the use of natural language has become a prevalent and significant method for facilitating coordination and cooperation among humans. From another perspective, the domain of large language models is precisely dedicated to and adept at enhancing communication and interaction in a human-like manner. Therefore, it is a natural progression to integrate large language models into agents as communication modules. These agents can employ the communication module to convey information or intentions observed by themselves to others, thereby enabling more effective collaboration.
在人类协作问题的背景下，个体接收信息的多样性及其独特的思维模式常使得简单的独立行动不足以达成协调共识。因此，自然语言的使用已成为促进人类间协调与合作的一种普遍且重要的方法。从另一角度看，大型语言模型领域正是致力于并擅长以类人的方式增强沟通与互动。因此，将大型语言模型整合进代理作为通信模块，是自然的发展趋势。这些代理可利用通信模块传递自身观察到的信息或意图给他人，从而实现更有效的协作。

Relying solely on environment interaction to train a communication model that automatically generates reasonable messages from scratch is highly inefficient, while utilizing a large-scale model (e.g., GPT-4, LLaMA-65B) for communication is both costly and difficult to fine-tune. To facilitate the generation of coordinated and consistent messages in the communication module, we employ GPT-4 as a coordinated message generator, taking the local observations of all agents as input and generating customized message labels for each agent $\{m_{i}\}^{n}_{i=1}\sim\pi^{tch}(p^{tch})$. The corresponding prompt $p^{tch}=\rho_{tch}(\{o_{i}\}^{n}_{i=1},n)$ is obtained through the teacher LLM’s prompt function $\rho_{tch}$:
仅依赖环境交互来训练一个从零开始自动生成合理信息的通信模型效率极低，而利用大规模模型（如 GPT-4、LLaMA-65B）进行通信既成本高昂又难以微调。为促进通信模块中协调一致信息的生成，我们采用 GPT-4 作为协调信息生成器，以所有代理的本地观察为输入，为每个代理生成定制的消息标签 $\{m_{i}\}^{n}_{i=1}\sim\pi^{tch}(p^{tch})$ 。相应的提示 $p^{tch}=\rho_{tch}(\{o_{i}\}^{n}_{i=1},n)$ 通过教师LLM的提示函数 $\rho_{tch}$ 获得。

Supervised learning training is conducted on the message samples using LLaMA-7B, which is updated by LoRA. The message prediction of agent $i$ is generated by message policy $\hat{m}_{i}\sim\pi_{\eta}(\cdot|\rho_{m}(o_{i}))$. The SFT loss can be expressed as follows:
监督学习训练采用 LLaMA-7B 对消息样本进行处理，该模型通过 LoRA 进行更新。代理 $i$ 的消息预测由消息策略 $\hat{m}_{i}\sim\pi_{\eta}(\cdot|\rho_{m}(o_{i}))$ 生成。SFT 损失可表示如下：

where $CROSS\_ENTROPY$ represents the cross entropy loss function. The entire workflow of SFT is shown in Figure 3(a) To control the training scale, we let all agents share the action policy and message module. Besides, We use initialized agents to interact in the environment for several rounds to collect trajectory data. Empirically, with only a small amount of data and training steps, the communication policy $\pi_{\eta}$ can already generate a reasonably coherent verbal message.
其中 $CROSS\_ENTROPY$ 代表交叉熵损失函数。SFT 的整体工作流程如图 3(a)所示。为控制训练规模，我们让所有代理共享行动策略与信息模块。此外，我们使用初始化代理在环境中交互数轮，以收集轨迹数据。经验表明，仅需少量数据及训练步骤，通信策略 $\pi_{\eta}$ 便能生成相当连贯的口头信息。

The action module also employs LLMs, which brings two distinct advantages: i) LLMs can directly and effectively comprehend verbal messages from teammates without the need for further training. ii) LLMs have been demonstrated to have learned a substantial amount of physical rules present in the real world [38], and this knowledge can significantly reduce the number of interactions required between the policy and the environment, thereby enhancing sample efficiency. To align general purpose LLMs with the specific environment, feedback from the environment is utilized to adapt to the environment better and yield better performance.
行动模块还采用了LLMs，带来两大显著优势：i) LLMs能直接有效理解队友的口头信息，无需额外训练；ii) LLMs已证明掌握了现实世界中大量物理规则[38]，这些知识能显著减少策略与环境间的交互次数，从而提升样本效率。为使通用LLMs适应特定环境，利用环境反馈进行调整，以更好地适应环境并提升性能。

Inspired by previous works [21, 20], we do not directly prompt the LLM to generate the actions, instead, we query the LLM for the probabilities of all available actions. Assume the $k$-th action of $i$-th agent $u_{i,k}\in U_{i}$ is a sequence of tokens $u_{i,k}=\{w^{1}_{i,k},\dots,w^{N_{i,k}}_{i,k}\}$, where $N_{k}$ is the token number of the $k$-th action. So the token-level probability of $u_{i,k}$ can be calculated by:
受先前作品[21, 20]的启发，我们并非直接指示LLM生成动作，而是向LLM查询所有可用动作的概率。假设 $k$ 号动作是 $i$ 号代理 $u_{i,k}\in U_{i}$ 的一系列令牌 $u_{i,k}=\{w^{1}_{i,k},\dots,w^{N_{i,k}}_{i,k}\}$ ，其中 $N_{k}$ 是 $k$ 号动作的令牌编号。因此， $u_{i,k}$ 的令牌级概率可通过以下方式计算：

where $\rho_{u}$ is the action prompt function. Therefore, the action policy $\pi$ can be got by the softmax over the token-level probabilities over actions:
其中 $\rho_{u}$ 为动作提示函数。因此，通过动作级概率上的 softmax 运算，可以得到动作策略 $\pi$ ：

In our work, we employ PPO [39] to optimize the action policy. PPO is a state-of-the-art actor-critic RL method, where each agent learns an actor parameterized by $\theta$ and a critic parameterized by $\phi$.
在我们的工作中，我们采用 PPO[39]来优化行动策略。PPO 是一种先进的演员-评论家强化学习方法，其中每个代理学习一个由 $\theta$ 参数化的演员和一个由 $\phi$ 参数化的评论家。

For each agent $i$, the policy loss can be formulated as follows:
对于每个代理 $i$ ，策略损失可以表述如下：

where $A^{t}_{i}$ is the Generalized Advantage Estimation (GAE) [40]. The critic network is composed of an additional MLP added to the last transformer block of the LLaMA model.
其中 $A^{t}_{i}$ 代表广义优势估计（GAE）[40]。批评家网络由附加在 LLaMA 模型最后一个变换器模块上的额外多层感知机（MLP）构成。

where $\hat{V}^{t}_{i}=A^{t}_{i}+V_{\phi_{old}}(\rho_{u}(o^{t}_{i}))$. 其中 $\hat{V}^{t}_{i}=A^{t}_{i}+V_{\phi_{old}}(\rho_{u}(o^{t}_{i}))$ 。

The overall learning loss is:
总体学习损失为：

where $\mathcal{H}(\pi_{i,\theta})$ denotes the entropy of agent $i$’s action policy $\pi_{i,\theta}$.
其中 $\mathcal{H}(\pi_{i,\theta})$ 表示代理 $i$ 的动作策略 $\pi_{i,\theta}$ 的熵。

Given that the goals of the action policy and message policy are distinct, we load two independent LoRA weights separately to ensure that they do not interfere with each other. During the entire action policy alignment stage, the message policy will be frozen to stabilize the training of the action policy. The actor, critic, and message networks share the same LLaMA model and the gradients do not affect each other, therefore greatly reducing our training costs and memory requirements. The entire algorithm can be described as shown in Algorithm 1.
鉴于行动策略与信息策略的目标各异，我们分别加载两套独立的 LoRA 权重，确保彼此互不干扰。在行动策略对齐阶段，信息策略将保持冻结状态，以稳定行动策略的训练。行动者、批评者及信息网络共享同一 LLaMA 模型，且梯度互不影响，从而大幅降低训练成本与内存需求。整个算法可如算法 1 所示描述。

The Overcooked environment is a popular complex environment for decision-making problems. The goal for agents is to make different types of salads with the provided raw materials and tools in a 7x7 grid-size kitchen. Our work extends the single-agent textual version of Overcooked in Tan et al. to multi-agent systems. In our setting, two agents need to collaborate to complete tasks in the environment. As shown in the figure, there are two types of maps A and B. In Map A, two agents are in the same space, so there may be collisions. In Map B, the two agents are separated and need to complete the task by passing items through the table. The environment is partially observable, and each agent can only observe the objects within 5×5 square centered of itself. As to the reward, chopping the correct item will be +0.2, providing the correct dish will be +1, -0.1 for delivering any wrong item, -0.01 for each collision between agents, and -0.001 for each time step. 

We compare our method with two trainable baselines and a baseline for direct decision-making by GPT-4. The description for each baseline is as follows: 

TWOSOME [20]. TWOSOME proposes an efficient single-agent LLM decision fine-tuning framework, and it balances the joint probability of candidate actions by several regularizing methods. 

Symbolic PPO [33, 39]. The symbolic PPO takes the raw numerical observation as its input and uses MLPs as the backbone network. 

The results on Overcooked are shown in Figure LABEL:fig:overcooked. In the method based on raw symbolic input, CommNet showed better performance than the Symbolic-PPO. However, it is difficult to observe the communication patterns between agents in a way that humans can understand. The results also indicate that the LLM-based methods have significantly higher sample efficiency. Moreover, Our Verco further achieves higher episode returns. We believe this improvement benefits from the communication information that can effectively coordinate the actions among agents. In addition, verbal messages can also promote our understanding of the cooperation mechanisms between agents. Moreover, the results show that Verco has significantly lower episode length and entropy compared to other algorithms. This indicates that the introduction of verbal communication can also encourage agents to complete tasks more efficiently and reduce the uncertainty of action policy. Although GPT-4 has rich prior knowledge, there are still biases in making complex decisions in the environment, which can easily lead to task failure.

To further analyze the differences brought about by the introduction of verbal communication, we visualize the replay of Verco and non-communication algorithms (TWOSOME) in detail. In the Single Room scenario as shown in Figure 6(a,d), communication is crucial to coordinate and perform different tasks, as there may be conflicting actions between agents. Specifically, Alice suggested that Bob should pick up the tomatoes and Bob suggested that Alice should pick up the bowl. After receiving the message from another agent, the two agents choose different actions which avoid conflicts. In the non-communication situation, both agents choose to directly pick up the tomato which will collide and waste time. In Separate Rooms, agents are separated by table and can only transfer items through the middle table. Therefore, communication is also important for agent cooperation. As shown in Figure 6(b), after Alice reminds Bob, Bob will put the tomatoes on the middle table for Alice to chop. Otherwise, Bob may not be aware of the need to pass the tomato to Alice without communication as shown in Figure 6(e). Overall, by directly sending verbal messages, we can have a more intuitive understanding of the cooperative motivations between agents.