Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
大型模型的参数高效微调：全面调查

Index Terms-Large Language Model, Parameter-Efficient Fine-tuning, Computer System, Distributed System.
索引词-大型语言模型，参数高效微调，计算机系统，分布式系统。

Large Models (LMs) have recently captured considerable public interest. Their ability to understand context and nuances enables them to proficiently handle diverse tasks across multiple domains, including natural language processing (NLP), computer vision (CV), etc. In the field of NLP, Large Language Models (LLMs) have achieved significant advancements across various tasks including text generation [1], [2], translation [3], [4], personalized chat-bots [5], [6], [7], and summarization [8], demonstrating remarkable proficiency.
大型模型（LMs）最近引起了相当大的公众关注。它们理解上下文和细微差别的能力使它们能够熟练处理跨多个领域的各种任务，包括自然语言处理（NLP）、计算机视觉（CV）等。在自然语言处理领域，大型语言模型（LLMs）在各种任务中取得了显著进展，包括文本生成[1]，[2]，翻译[3]，[4]，个性化聊天机器人[5]，[6]，[7]和摘要[8]，展示了卓越的熟练度。

Earlier studies [1] has suggested that LLMs exhibit high levels of generalization, enabling them to apply their acquired knowledge to new tasks not included in their original training. This capability is commonly known as zero-shot learning. Nevertheless, fine-tuning remains essential to further enhance LLMs for optimal performance on new user datasets and tasks.
较早的研究[1]表明LLMs表现出高水平的泛化能力，使它们能够将所获知识应用于原始训练中未包含的新任务。这种能力通常被称为零样本学习。然而，微调仍然是必不可少的，以进一步增强LLMs在新用户数据集和任务上的最佳性能。

Due to its scale, a widely adopted strategy for fine-tuning LLMs involves adjusting a limited number of LLM parameters while keeping the remainder unchanged. This technique, termed Parameter-Efficient-Fine-Tuning (PEFT), involves selectively adjusting a small proportion of their parameters, while keeping the rest unaltered. Furthermore, the application of PEFT extends beyond the realm of NLP and quickly attracts interest in the CV community for handling fine-tuning vision models with large parameters, such as Vision Transformers (ViT) and diffusion models, as well as disciplinary models such as vision-language models.
由于其规模，广泛采用的微调LLMs的策略涉及调整有限数量的LLM参数，同时保持其余参数不变。这种技术被称为参数高效微调（PEFT），涉及有选择地调整一小部分参数，同时保持其余参数不变。此外，PEFT 的应用不仅限于 NLP 领域，还迅速引起了 CV 社区的兴趣，用于处理具有大参数的微调视觉模型，如 Vision Transformers（ViT）和扩散模型，以及视觉-语言模型等学科模型。

In this survey, we systematically review and categorize recent advancements in PEFT algorithms as well as the system implementation costs associated with various PEFT algorithms across diverse scenarios. Figure 1 presents the overview content for this survey. In section , we present some fundamental concepts for LLM and PEFT, including computational flow for LLM, basic knowledge of PEFT, and commonly used datasets and tasks.
在这项调查中，我们系统地审查和分类了最近在 PEFT 算法方面的进展，以及不同场景中与各种 PEFT 算法相关的系统实施成本。图 1 展示了本调查的概述内容。在第 节中，我们介绍了一些关于LLM和 PEFT 的基本概念，包括LLM的计算流程、PEFT 的基本知识以及常用的数据集和任务。

We categorized all types of PEFT algorithms in Section III according to their computational flow. In Section III-A, we introduce additive algorithms that either introduce additional weight parameters or modify activations. For algorithms that exclusively require fine-tuning with existing parameters, they fall under the category of selective approaches, and their introduction can be found in Section III-B, In Section III-C, we explore reparameterized PEFT, which constructs a (lowdimensional) reparameterization of original model parameters for training while transforms the weights back to maintain the inference speed. Additionally, there exist algorithms that combine the above techniques, and we have classified these as hybrid approaches, elaborating on them in Section III-D We also investigate strategies for further reducing the computational complexity of different PEFT algorithms, including KV-cache management, pruning, quantization, and memory optimization, in Section IV
我们根据它们的计算流程在第三部分对所有类型的 PEFT 算法进行了分类。在第 III-A 节中，我们介绍了引入额外权重参数或修改激活的加法算法。对于仅需要与现有参数进行微调的算法，它们属于选择性方法的范畴，其介绍可以在第 III-B 节中找到。在第 III-C 节中，我们探讨了重新参数化的 PEFT，它为训练构建了原始模型参数的（低维）重新参数化，同时将权重转换回以保持推理速度。此外，还存在结合上述技术的算法，我们已将其分类为混合方法，并在第 III-D 节中对其进行了详细阐述。我们还研究了进一步减少不同 PEFT 算法的计算复杂性的策略，包括 KV 缓存管理、修剪、量化和内存优化，在第 IV 节中。

In Section , we expand the scope of this survey beyond the computational perspective to involve various potential application scenarios. We explore innovations that applying PEFT techniques to different model architecture, including LLMs
在第 节中，我们将这项调查的范围扩大到涉及各种潜在应用场景，从而超越计算视角。我们探索将 PEFT 技术应用于不同模型架构的创新，包括LLMs。

Fig. 1: A content overview covered in the survey.
图 1：调查涵盖的内容概述。

(Section V-A), Vision Transformer (Section V-B), VisionLanguage alignment models (Section V-C), and Diffusion models (Section V-D), for varied downstream tasks, underscoring PEFT's versatility and applicability in a range of scenarios.
（第五部分-A），视觉 Transformer（第五部分-B），视觉语言对齐模型（第五部分-C）和扩散模型（第五部分-D），用于各种下游任务，强调 PEFT 在各种场景中的多功能性和适用性。

In Section VI, we explores the system design challenge for PEFT methods. The discussion includes three advanced system solutions for practical PEFT deployment: distributed tuning (Section VI-B), PEFT query serving (Section VI-C), and concurrent PEFT tuning (Section VI-D).
在第六部分，我们探讨了 PEFT 方法的系统设计挑战。讨论包括三种实际 PEFT 部署的高级系统解决方案：分布式调整（第 VI-B 节），PEFT 查询服务（第 VI-C 节）和并发 PEFT 调整（第 VI-D 节）。

In the last Section VII we summarize our survey and propose several potential future directions from both algorithm and system perspectives, hoping to provide valuable insights for further research and development in the field.
在最后的第七部分，我们总结了我们的调查，并从算法和系统的角度提出了几个潜在的未来方向，希望为该领域的进一步研究和发展提供有价值的见解。

In this section, we first discussed the computation flow of LLM, including its fundamental components, computational complexity, and the flow of computations it involves as a case study. We then provide a brief overview of different PEFT algorithms in section II-B
在本节中，我们首先讨论了LLM的计算流程，包括其基本组成部分、计算复杂性以及涉及的计算流程作为案例研究。然后在第 II-B 节中提供了对不同 PEFT 算法的简要概述。

In order to gain a deeper understanding of LLM and other Transformer-based models, we employ LLaMA-7B, a cuttingedge open-source LLM model, to scrutinize the architecture of LLM as well as Transformer. As shown in Figure 2 (a), LLaMA consists of three major components: an embedding block, a stack of decoder blocks and a head block which consists of linear and softmax layer. The embedding layer's primary role is to transform unstructured textual information, into chunks of discrete numerical vectors (tokens) to facilitate subsequent processing. The embedded tokens are then delivered to the decoder layers for further processing. Each LLaMA decoder is composed of two fundamental components: Multihead Self-Attention (MSA) and Feedforward Network (FFN). In the MSA module, each of the tokens will be clustered by an attention map obtained by a dot production between two linear mappings of the input tokens. Then the grouped tokens will be further processed by a Feedforward Neural network.
为了更深入地了解LLM和其他基于 Transformer 的模型，我们采用 LLaMA-7B，这是一种尖端的开源LLM模型，来审查LLM和 Transformer 的架构。如图 2（a）所示，LLaMA 由三个主要组件组成：嵌入块、一堆解码器块和一个由线性和 softmax 层组成的头块。嵌入层的主要作用是将非结构化的文本信息转换为离散的数字向量块（标记），以便后续处理。嵌入的标记然后传递给解码器层进行进一步处理。每个 LLaMA 解码器由两个基本组件组成：多头自注意力（MSA）和前馈网络（FFN）。在 MSA 模块中，每个标记将通过输入标记的两个线性映射之间的点积获得的注意力图进行聚类。然后，分组的标记将通过前馈神经网络进一步处理。
Additionally, Root Mean Square Layer Normalization (RMSNorm) [9] is adopted in LLaMA as a replacement for Layer Normalization to ensure efficient training.
此外，均方根层归一化（RMSNorm）[9]在 LLaMA 中被采用作为层归一化的替代，以确保高效训练。

LLM distinguishes itself from other deep neural network (DNN) models such as convolutional neural networks (CNN) in two significant ways. Firstly, LLM exhibits an inherent autoregressive nature, necessitating multiple iterations to complete the generation task. Moreover, LLM incorporates an attention mechanism, a component with computational complexity that scales quadratically with the length of the inputs. On the other hand, the inherent computation characteristic of LLM lies in the attention blocks inside each decoder layer. Figure 2 (c) depicts the high-level overview of the computation flow in the attention block.
LLM以两种显著方式区别于其他深度神经网络（DNN）模型，如卷积神经网络（CNN）。首先，LLM表现出固有的自回归特性，需要多次迭代才能完成生成任务。此外，LLM融入了注意力机制，这是一个具有计算复杂度的组件，随着输入长度的增加呈二次方增长。另一方面，LLM的固有计算特性在每个解码器层内的注意力块中。图 2（c）展示了注意力块中计算流程的高级概述。

During the inference process, each decoder takes a 4dimensional tensor as the input tokens. The input tokens are first multiplied with three weight matrices , and , producing the output referred to as query and value . Given the MSA module's inability to recognize positional data and the inherent autoregressive nature of LLMs, the query and key will undergo a process using Rotary Positional Embedding [10] (RoPE, denoted as . Subsequently, the key and value will be combined with prior tokens.
在推理过程中，每个解码器将一个 4 维张量 作为输入标记。首先，输入标记将与三个权重矩阵 和 相乘，产生所谓的查询 和数值 的输出。鉴于 MSA 模块无法识别位置数据以及LLMs固有的自回归特性，查询和键将经过使用旋转位置嵌入[10]（RoPE，表示为 的过程。 随后，键和数值将与先前的标记组合在一起。

After the positional embedding, the intermediate activation will then undergo a series of multiplication, softmax, and residual addition to generate MSA output as described in Eq9 To be noted here, in the equation refers to the number of feature dimensions in the multi-head attention mechanism.
在位置嵌入之后，中间激活将经历一系列乘法、softmax 和残差相加，以生成 Eq9 中描述的 MSA 输出。需要注意的是，在方程中 指的是多头注意力机制中的特征维度数量。

The SA output will then be forwarded to the FFN blocks for further processing. The FFN block will have another three
SA 输出将被转发到 FFN 块进行进一步处理。FFN 块将有另外三

Fig. 2: (a) LLaMA architecture. (b) LLaMA auto-regressive pattern. (c) Three common PEFT operations. All the learnable components are highlighted in red, while the frozen components are highlighted in grey. LoRA is applied on all the Query, Key, and Value blocks. The adapter targets the FFN module. Soft-Prompt focused on tuning the input activation of each decoder. We only show one decoder for illustration simplicity.
图 2：(a) LLaMA 架构。(b) LLaMA 自回归模式。(c) 三种常见的 PEFT 操作。所有可学习组件都用红色突出显示，而冻结组件则用灰色突出显示。LoRA 应用于所有查询、键和值块。适配器针对 FFN 模块。Soft-Prompt 专注于调整每个解码器的输入激活。为了简化说明，我们只展示一个解码器。

matrices , and and the computation can be illustrated by:
矩阵 和 ，计算过程如下：

where denotes the input of the FFN layer, and SiLU is the nonlinear function used in LLaMA. In the original Transformer, the FFN block can be demonstrated by:
其中 表示 FFN 层的输入，SiLU 是 LLaMA 中使用的非线性函数。在原始 Transformer 中，FFN 块可以通过以下方式展示：

The output of the last decoder layer will be sent to a linear layer, which then generates a probability distribution spanning the complete vocabulary to predict the next token in the sequence. The produced token will then be concatenated with the previous tokens and used as the input for the next round of processing. This generating process repeats in an auto-regressive manner until a full sequence of tokens, referred to as a completion, is produced (Figure 2 (b)). For training, the computation flow is similar to that for inference, except that the generated sentences are directly compared to the ground truth output and generate the training loss. Gradients will then be computed across the LLM weights to minimize this training loss.
最后一个解码器层的输出将被发送到一个线性层，该线性层然后生成一个横跨完整词汇表的概率分布，以预测序列中的下一个标记。生成的标记然后将与先前的标记连接起来，并用作下一轮处理的输入。这个生成过程以自回归的方式重复，直到生成一个完整的标记序列，称为完成（图 2（b））。对于训练，计算流程与推理过程类似，不同之处在于生成的句子直接与地面真实输出进行比较并生成训练损失。然后将计算梯度跨越LLM个权重以最小化这个训练损失。

To analyze the computation cost and memory overhead in LLM, we also set a series of parameters used in later section III Table I shows the parameter size and computation dimension in the LLaMA-7B model as a starting example.
为了分析LLM中的计算成本和内存开销，我们还设置了一系列在后文第 III 节中使用的参数。表 I 显示了 LLaMA-7B 模型中的参数大小和计算维度，作为一个起始示例。

LLM models generate tokens (words) one for each round, depicted in Fig 2, based on the previous prompt (input) and previously generated sequence. This process will be repeated until the model outputs hits and termination token. To accelerate the inference process in LLM models, people take the strategy of storing the previous Keys and Values in KeyValue cache (KV-cache), so they don't need to recalculate them for each new token. Mathematically, we can represent the total decoders' kv-cache memory cost in equation 6. In the equation, 1 and are the context length and batch size and refers to the number of layers. The is the head dimension and is the number of heads.
LLM模型生成令牌（单词），每轮一个，如图 2 所示，基于先前的提示（输入）和先前生成的序列。这个过程将重复进行，直到模型输出命中和终止令牌。为了加速LLM模型中的推理过程，人们采取了将先前的键和值存储在 KeyValue 缓存（KV 缓存）中的策略，这样他们就不需要为每个新令牌重新计算它们。在数学上，我们可以用方程 6 表示总解码器的 kv-cache 内存成本。在方程中，1 和 分别是上下文长度和批处理大小， 表示层数。 是头维度， 是头的数量。

Fine-tuning remains essential to enhance LLM performance on unseen user datasets and tasks. With the size of the model growing (e.g. 1.5B in GPT-2 to 175B in GPT-3), standard full fine-tuning paradigm requires thousands of GPU work in parallel, which is highly inefficient and unsustainable. A type of algorithm has been raised namely Parameter-efficient fine-tuning (PEFT) which aims to tune minimal parameters to achieve better performance over full tuning on downstream tasks.
微调仍然是提高LLM在未见用户数据集和任务上性能的关键。随着模型规模的增长（例如，GPT-2 中的 15 亿到 GPT-3 中的 175 亿），标准的全微调范式需要成千上万个 GPU 并行工作，这是非常低效和不可持续的。一种算法被提出，即参数高效微调（PEFT），旨在调整最小参数以在下游任务上实现比全面调整更好的性能。

In parallel developments, large-scale pre-trained models in vision and multimodal domains have also demonstrated their effective representational learning capabilities, enabling adaptation from large datasets to smaller ones or across various data modalities through fine-tuning. Consequently, this capability has made PEFT increasingly attractive to the wider research community.
在平行发展中，视觉和多模态领域的大规模预训练模型也展示了它们有效的表征学习能力，使得能够通过微调从大型数据集适应到较小的数据集或跨越各种数据模态。因此，这种能力使得 PEFT 对更广泛的研究社区越来越具吸引力。

We categorized the PEFT algorithms into additive, selective, reparameterized, and hybrid fine-tuning based on their operations. As Figure 3 depicts, three major additive finetuning algorithms are normally used: (1) Adapter; (2) Soft Prompt; (3) Others. They differ from each other in terms of the different additional tunable modules or parameters. Selective fine-tuning, on the other hand, doesn't require any additional parameters, it selects a small subset of parameters from the backbone model and only makes them tunable while keeping the majority of parameters untouched during finetuning on downstream tasks. We categorized selective finetuning based on the grouping of chosen parameters: (1) Unstructural Masking; (2) Structural Masking. Reparametrization
我们根据它们的操作将 PEFT 算法分类为加性、选择性、重新参数化和混合微调。如图 3 所示，通常使用三种主要的加性微调算法：（1）适配器；（2）软提示；（3）其他。它们在不同的可调模块或参数方面有所不同。另一方面，选择性微调不需要任何额外参数，它从骨干模型中选择一小部分参数，并只使它们可调，同时在下游任务的微调过程中保持大多数参数不变。我们根据所选参数的分组将选择性微调分类为：（1）非结构化掩蔽；（2）结构化掩蔽。重新参数化

TABLE I: Configuration parameters and computation operation for LLaMA-7B architecture
表 I：LLaMA-7B 架构的配置参数和计算操作

represents transforming model parameters between two equivalent forms. Specifically, reparametrized fine-tuning introduces additional low-rank trainable parameters during training, which are then integrated with the original model for inference. This approach is categorized into two main strategies: (1) Lowrank Decomposition, and (2) LoRA Derivatives. Hybrid finetuning explores the design spaces of different PEFT methods and combines their advantages.
代表将模型参数在两种等效形式之间进行转换。具体而言，重新参数化微调在训练过程中引入了额外的低秩可训练参数，然后将其与原始模型集成以进行推断。这种方法分为两种主要策略：（1）低秩分解，和（2）LoRA 导数。混合微调探索了不同 PEFT 方法的设计空间，并结合它们的优势。

Two types of tasks have been widely used for LLM evaluation, the first type is the General Language Understanding Evaluation (GLUE) [11] benchmark, which integrates nine sentence or sentence-pair language understanding tasks (CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, RTE, and WNLI), chosen for their diversity in dataset sizes, text genres, and difficulty levels, and is based on established existing datasets. It also includes a diagnostic dataset specifically designed to evaluate and analyze model performance across various linguistic phenomena inherent in natural language. Additionally, it features a public leaderboard to track performance on the benchmark and a dashboard to visualize model performance on the diagnostic set.
两种类型的任务已广泛用于LLM评估，第一种类型是通用语言理解评估（GLUE）[11]基准测试，它整合了九个句子或句对语言理解任务（CoLA，SST-2，MRPC，STS-B，QQP，MNLI，QNLI，RTE 和 WNLI），这些任务因其数据集大小、文本流派和难度级别的多样性而被选择，并且基于已建立的现有数据集。它还包括一个专门设计的诊断数据集，用于评估和分析模型在自然语言中固有的各种语言现象上的表现。此外，它还设有一个公开排行榜，用于跟踪基准测试的表现，并有一个仪表板来可视化模型在诊断集上的表现。

The other type of dataset that has been used in recent LLM papers is common sense reasoning which integrated into our study caters to a variety of research facets: (1) OpenBookQA [12] is curated to foster research in advanced question-answering, delving into a profound understanding of both the subject matter and the language in which it is articulated. (2) PIQA [13] primarily emphasizes everyday scenarios, demonstrating a predilection for unconventional solutions. (3) Social IQA [14] emerges as a novel questionanswering benchmark tailored for gauging social commonsense intelligence. (4) HellaSwag [15] serves as a dataset, the essence of which is to ascertain the capability of machines in aptly concluding sentences. (5) BoolQ [16] is a dataset dedicated to question-answering, particularly for binary responses (yes/no queries). (6) WinoGrande [17] is introduced as a fresh compilation, encompassing a substantial 44,000 problems. (7) -easy [18] presents itself as a novel dataset constituting genuine grade-school level multiple-choice science questions, designed to invigorate research in intricate question-answering. (8) ARC-challenges [18], distinctively, encompasses solely those questions that were inaccurately addressed by both a retrieval-based algorithm and a word co-occurrence algorithm.
最近LLM篇论文中使用的另一种数据集是融入到我们研究中的常识推理，适用于各种研究方面：（1）OpenBookQA [12] 旨在促进高级问答研究，深入理解主题内容及表达语言。 （2）PIQA [13] 主要强调日常场景，展示对非传统解决方案的偏好。 （3）Social IQA [14] 是一种新颖的问题回答基准，旨在评估社会常识智能。 （4）HellaSwag [15] 是一个数据集，其本质在于确定机器在恰当地得出句子结论方面的能力。 （5）BoolQ [16] 是一个专门用于问答的数据集，特别适用于二元响应（是/否查询）。 （6）WinoGrande [17] 被介绍为一个新的编译，包含大量的 44,000 个问题。 （7） -easy [18] 作为一个新的数据集，包含真实的小学水平的多项选择科学问题，旨在激发复杂问答研究。 （8）ARC 挑战[18]，独特地，仅包括那些由检索算法和词共现算法都错误地处理的问题。

Image recognition is the primary benchmark and application for vision models, exemplified by benchmarks such as fine- grained visual categorization (FGVC) and visual task adaptation benchmark (VTAB). Beyond image classification, video action recognition is another key application area, involving datasets like Kinetics-400 [19], SSv2 [20], and HMDB51 [21]. Additionally, PEFT has been utilized for dense prediction tasks, using datasets like MSCOCO [22], ADE20K [23], and PASCAL VOC [24].
图像识别是视觉模型的主要基准和应用，以细粒度视觉分类（FGVC）和视觉任务适应基准（VTAB）等基准为例。除了图像分类之外，视频动作识别是另一个关键应用领域，涉及 Kinetics-400 [19]、SSv2 [20]和 HMDB51 [21]等数据集。此外，PEFT 已被用于密集预测任务，使用 MSCOCO [22]、ADE20K [23]和 PASCAL VOC [24]等数据集。

The PEFT strategies can be broadly classified into four categories: additive PEFT (Section III-A), which modifies the model architecture by injecting new trainable modules or parameters; selective PEFT (Section III-B), which makes a subset of parameters trainable during fine-tuning; reparameterized PEFT (Section III-C), which constructs a (lowdimensional) reparameterization of the original model parameters for training, then equivalently transforms it back for inference; and hybrid PEFT (Section III-D), which combines advantages from different PEFT methods to build a unified PEFT model. A overview of different types of PEFT algorithms is depicted in Figure 4.
PEFT 策略可以广泛分为四类：加性 PEFT（第 III-A 节），通过注入新的可训练模块或参数来修改模型架构；选择性 PEFT（第 III-B 节），在微调期间使一部分参数可训练；重新参数化 PEFT（第 III-C 节），为训练构建原始模型参数的（低维）重新参数化，然后等效地将其转换回推断；混合 PEFT（第 III-D 节），结合不同 PEFT 方法的优势构建统一的 PEFT 模型。图 4 展示了不同类型的 PEFT 算法概述。

Standard full fine-tuning entails substantial computational expenses and also could potentially harm the model's generalization ability. To mitigate this problem, a widely employed approach is to maintain the pre-trained backbone unchanged and introduce only a minimal number of trainable parameters that are strategically positioned within the model architecture. While fine-tuning for a specific downstream task, only the weights of these additional modules or parameters are updated, which results in a substantial reduction in storage, memory, and computational resource requirements. Due to their characteristic of adding parameters, these techniques can be termed as Additive Tuning, as shown in Figure 4 (a). Next, we discuss several popular Additive PEFT algorithms.
标准的全面微调需要大量的计算开销，还可能损害模型的泛化能力。为了缓解这个问题，一个广泛采用的方法是保持预训练的主干网络不变，并仅引入一小部分可训练参数，这些参数被战略性地放置在模型架构中。在为特定的下游任务微调时，只更新这些额外模块或参数的权重，这导致存储、内存和计算资源需求大幅减少。由于增加参数的特性，这些技术可以称为增量调整，如图 4（a）所示。接下来，我们将讨论几种流行的增量 PEFT 算法。

Fig. 3: Taxonomy of Parameter-Efficient Fine-Tuning Methods for Large Models.
图 3：大型模型参数高效微调方法的分类。

(a) Additive PEFT 添加剂 PEFT

(c) Reparameterization PEFT
(c) 重新参数化 PEFT

Fig. 4: Different types of PEFT algorithms.
图 4：不同类型的 PEFT 算法。

The concept of adapters in the field of NLP was initially introduced by Serial Adapter [25] as shown in Figure 5 (a). In their approach, each Transformer block is enhanced by adding two adapter modules, with one positioned after the self-attention layer and the other after the FFN layer, respectively. Subsequent research has aimed to address the additional computational cost associated with adapter layers. A modified framework AdapterFusion [29] was proposed, where adapter layers are inserted only after the 'Add & Norm' step following the FFN layer to enhance the computational efficiency. The adapters mentioned above follow a sequential design, placing adapter layers as bottlenecks within the Transformer blocks. This approach may potentially reduce the model's parallelism and require a trade-off between inference efficiency and accuracy. In contrast, [26] introduced a parallel adapter (PA) approach as depicted in Figure 5 (b), which reorganizes the traditionally sequential adapter layers into a parallel side-network that runs alongside each Transformer sublayer. Similarly, CIAT [27], CoDA [28] and KronA [72] also adopts a parallel adapter design. Except for the parallel design, CoDA employs a sparse activation mechanism to improve the inference efficiency as shown in Figure 5 (c).
在自然语言处理领域，适配器的概念最初是由 Serial Adapter [25]引入的，如图 5(a)所示。在他们的方法中，每个 Transformer 块通过添加两个适配器模块进行增强，一个位于自注意力层之后，另一个位于 FFN 层之后。随后的研究旨在解决适配器层带来的额外计算成本。提出了一种修改后的框架 AdapterFusion [29]，其中适配器层仅在 FFN 层后的“加和归一化”步骤之后插入，以增强计算效率。上述适配器遵循顺序设计，将适配器层放置在 Transformer 块内部作为瓶颈。这种方法可能会降低模型的并行性，并需要在推理效率和准确性之间进行权衡。相比之下，[26]引入了一种并行适配器（PA）方法，如图 5(b)所示，将传统的顺序适配器层重新组织为一个并行的侧网络，与每个 Transformer 子层并行运行。类似地，CIAT [27]、CoDA [28]和 KronA [72]也采用了并行适配器设计。 除了并行设计外，CoDA 还采用了稀疏激活机制来提高推理效率，如图 5（c）所示。
Specifically, CoDA use a soft top- selection process that identifies important tokens in each layer, which will be processed by both the frozen pre-trained Transformer layer and the adapter branch to maintain model accuracy. In contrast, those unimportant tokens are only processed by the adapter branch while skip the heavy pre-trained layer, therefore optimizing for inference efficiency without compromising overall performance.
具体来说，CoDA 使用软顶 选择过程，识别每一层中的 个重要标记，这些标记将由冻结的预训练 Transformer 层和适配器分支处理，以保持模型准确性。相比之下，那些不重要的标记只由适配器分支处理，跳过繁重的预训练层，因此在不影响整体性能的情况下优化推理效率。

To enhance the performance and generalization of adapters, various studies have implemented multi-task learning strategies, such as AdapterFusion [29], AdaMix [30], PHA [31], AdapterSoup [32], MerA [33], and Hyperformer [34]. AdapterFusion keeps all pre-trained adapters in the model and employs a fusion module to merge the multi-task informations. Unlike AdapterFusion, MerA merges pretrained adapters into a single one through optimal transport based on weights and activations. This approach avoids introducing any additional trainable parameters, thereby enhancing computational efficiency. Hyperformer stores the multi-task information in a shared hypernetwork, which generates task and layer-specific adapter parameters conditioned on task and layer id embeddings. Given a new task, only an additional task embedding needs to be learned, therefore reducing the number of trained parameters.
为了提高适配器的性能和泛化能力，各种研究已经实施了多任务学习策略，例如 AdapterFusion [29]，AdaMix [30]，PHA [31]，AdapterSoup [32]，MerA [33]和 Hyperformer [34]。AdapterFusion 保留模型中的所有预训练适配器，并采用融合模块来合并多任务信息。与 AdapterFusion 不同，MerA 通过基于权重和激活的最优传输将预训练适配器合并为一个单一适配器。这种方法避免引入任何额外的可训练参数，从而提高了计算效率。Hyperformer 将多任务信息存储在共享的超网络中，该超网络生成基于任务和层 ID 嵌入的任务和层特定适配器参数。对于新任务，只需要学习一个额外的任务嵌入，从而减少了训练参数的数量。

(a) Serial Adapter 串行适配器

(b) Parallel Adapter 并行适配器

(d) Adapter Layer 适配器层

Fig. 5: Illustration of three representative adapter-based fine-tuning algorithms. Blue represents frozen, while yellow represents trainable.
图 5：三种代表性的基于适配器的微调算法示例。蓝色代表冻结，黄色代表可训练。

where is the sequence of input tokens for layer , including soft prompt tokens followed by the original input tokens is the number of soft prompt tokens, and is the number of original input tokens.
其中 是第 层的输入令牌序列，包括软提示令牌 和原始输入令牌 ， 是软提示令牌的数量， 是原始输入令牌的数量。

Prefix-tuning [35] introduces learnable vectors that are prepended to keys and values across all Transformer layers. To ensure stability during the optimization process, Prefixtuning adopts a reparameterization strategy, which utilizes an MLP layer to generate these prefix vectors rather than optimizing them directly. After fine-tuning, only the prefix vectors are saved for inference. This technique is adapted and improved in several studies [36], [37], [38]. For instance, ptuning v2 [37] removes reparameterization and expands its usage to broader model scales and NLP tasks. APT (Adaptive Prefix Tuning) [38] enhances Prefix-tuning by introducing adaptive gate mechanism to control the prefix importance in each layer. Concurrent work p-tuning [39] and prompttuning [40] apply learnable vectors only at the initial word embedding layer rather than all layers to enhance training and inference efficiency. It's important to highlight that prompttuning demonstrates its effectiveness primarily in the context of large models, specifically those with over 11 billion parameters [40]. Complementing this, Xprompt [41] eliminates the negative prompt tokens through a hierarchical structured pruning, which closes the performance gap at smaller model scales. The work by [97] provides some theoretical analysis towards prompt tuning, demonstrating its universality and limitations in limited-depth Transformers. IDPG (InstanceDependent Prompt Generation) [42] improves prompt tuning by generating prompts based on each input sentence with a lightweight prompt generator. In a related approach, LPT (Late Prompt Tuning) [43] also leverages a prompt generator to obtain instance-aware prompt. Unlike previous work, LPT adds these prompts only after an intermediate layer, rather than at the initial or all layers. This strategic placement eliminates the gradient calculation below the intermediate layer, thereby significantly accelerating the training speed. Simultaneously, LPT can improve the overall performance due to the shorter backpropagation path preserves more task-related information. Inspired by LPT, SPT (Selective Prompt Tuning) [44] delves deeper into the importance of prompt inserting strategies. It introduces a learnable probabilistic gate in each layer to determine whether to use the prompt propagated from the previous layer or inject a newly generated prompt. APrompt [45] employs another prompt inserting strategy. In addition to input prompts inserted at the beginning of the input sequence for each Transformer layer, APrompt also prepends additional learnable prompts to the respective query, key, and value matrices in the self-attention blocks to learn new attention patterns. Besides, APrompt incorporates the learning of a taskspecific head.
Prefix-tuning [35] 引入了可学习的向量，这些向量被添加到所有 Transformer 层的键 和值 之前。为了确保在优化过程中的稳定性，Prefixtuning 采用了重新参数化策略，该策略利用 MLP 层生成这些前缀向量，而不是直接优化它们。微调后，只保存前缀向量以进行推断。这种技术在几项研究中得到了改进和应用[36]，[37]，[38]。例如，ptuning v2 [37] 去除了重新参数化，并将其用法扩展到更广泛的模型规模和 NLP 任务。APT（自适应前缀调整）[38] 通过引入自适应门机制来增强 Prefix-tuning，以控制每层中前缀的重要性。同时进行的工作 p-tuning [39] 和 prompttuning [40] 仅在初始词嵌入层应用可学习向量，而不是所有层，以增强训练和推断效率。值得强调的是，prompttuning 主要在大型模型的背景下展示了其有效性，特别是那些具有超过 110 亿参数的模型[40]。 与此相辅相成的是，Xprompt [41] 通过分层结构修剪消除了负面提示标记，从而在较小的模型规模下缩小了性能差距。[97] 的工作对提示调整进行了一些理论分析，展示了其在有限深度的 Transformer 中的普适性和局限性。IDPG（InstanceDependent Prompt Generation）[42] 通过基于每个输入句子生成提示的轻量级提示生成器改进了提示调整。在相关方法中，LPT（Late Prompt Tuning）[43] 也利用提示生成器获得实例感知提示。与以往的工作不同，LPT 仅在中间层之后添加这些提示，而不是在初始或所有层。这种策略性的放置消除了中间层以下的梯度计算，从而显著加快了训练速度。同时，由于更短的反向传播路径保留了更多与任务相关的信息，LPT 可以提高整体性能。受到 LPT 的启发，SPT（Selective Prompt Tuning）[44] 深入探讨了提示插入策略的重要性。 它在每一层引入了一个可学习的概率门，用于确定是使用从前一层传播的提示还是注入新生成的提示。APrompt [45]采用另一种提示插入策略。除了在每个 Transformer 层的输入序列开头插入输入提示外，APrompt 还在自注意力块中的相应查询、键和值矩阵前置额外的可学习提示，以学习新的注意力模式。此外，APrompt 还结合了任务特定头的学习。

The concept of soft prompts has been employed for various downstream tasks [98], [99], although their training can be prone to instability and slow convergence. To address this, SPoT [46] uses a source prompt learned from one or multiple tasks to initialize prompts for new tasks. Similarly, transfer of soft prompts from one task to initialize another is proposed in TPT (transferable prompt tuning) [47], which demonstrates that a better prompt initialization results in a large training convergence speedup. InfoPrompt [48] develops two mutual information based loss functions, i.e., head loss and representation loss, to find better prompt initialization and learn sufficient task-relevant information, thereby also expediting convergence. PTP [49] delves into the root causes of training instability. It identifies the steep nature of the loss landscape in conventional prompt tuning, where minor variations in input data can lead to significant loss fluctuations. To mitigate this, PTP introduces perturbation-based regularizers to smooth the loss landscape and consequently stabilize the training process. DePT [52] decomposes the soft prompt into a shorter soft prompt with a pair of low-rank matrices, which are optimised with two distinct learning rates. This strategy not only improves the performance but also enhances training and inference efficiency. SMoP (Sparse Mixture-of-Prompts) [51] reduce the training and inference cost by utilizing short soft prompts. During training, multiple short soft prompts are trained, each tailored to specific subsets of the dataset. During inference, SMoP integrates a gating mechanism that routes each input instance to an appropriate short prompt. This technique not only increases efficiency in both training and inference stages but also retains performance comparable to those achieved with longer soft prompts. To further cut down the number of soft prompt parameters, IPT (Intrinsic Prompt Tuning) [50] identifies an intrinsic task subspace by training an auto-encoder on multiple tasks. Tuning on new tasks then requires adjusting only a few parameters within this subspace, significantly reducing the number of training parameters.
软提示的概念已经被用于各种下游任务[98]，[99]，尽管它们的训练可能容易不稳定且收敛缓慢。为了解决这个问题，SPoT [46] 使用从一个或多个任务中学习的源提示来初始化新任务的提示。类似地，TPT（可转移提示调整）[47] 提出了从一个任务转移软提示以初始化另一个任务的方法，表明更好的提示初始化会导致训练收敛速度加快。InfoPrompt [48] 开发了两个基于互信息的损失函数，即头损失和表示损失，以找到更好的提示初始化并学习足够的任务相关信息，从而加快收敛速度。PTP [49] 深入研究了训练不稳定的根本原因。它确定了传统提示调整中损失景观的陡峭特性，其中输入数据的微小变化可能导致显著的损失波动。为了缓解这一问题，PTP 引入基于扰动的正则化器来平滑损失景观，从而稳定训练过程。 DePT [52] 将软提示分解为一对低秩矩阵的较短软提示，这些矩阵使用两个不同的学习率进行优化。这种策略不仅提高了性能，还增强了训练和推断效率。SMoP（Sparse Mixture-of-Prompts）[51] 通过利用短软提示来减少训练和推断成本。在训练过程中，会训练多个短软提示，每个提示都针对数据集的特定子集。在推断过程中，SMoP 集成了一个门控机制，将每个输入实例路由到适当的短提示。这种技术不仅提高了训练和推断阶段的效率，而且保持了与较长软提示实现的性能相媲美。为了进一步减少软提示参数的数量，IPT（Intrinsic Prompt Tuning）[50] 通过在多个任务上训练自动编码器来识别内在任务子空间。然后，对新任务进行调整只需要调整这个子空间内的少量参数，从而显著减少训练参数的数量。

(a)  （a）

(b) SSF
Fig. 6: Illustration of (IA) and SSF. Blue represents frozen, while yellow represents trainable.
图 6：(IA) 和 SSF 的示意图。蓝色代表冻结，黄色代表可训练。

rescaling vectors: , and , to rescale the key, value, and FFN activations, respectively, as depicted in Figure 6 (a). The operations within the self attention block can be described as follows:
重新调整向量： ，和 ，以重新调整键、值和 FFN 激活，如图 6（a）所示。自注意力块内的操作可以描述如下：

In FFN, the rescaling can be denoted as:
在 FFN 中，重新缩放可以表示为：

where is Hadamard product. Furthermore, the scale vectors and can be seamlessly integrated into the weight matrices of and . This integration effectively eliminates the extra computational costs during inference. A similar technique SSF [55] also performs linear transformation to the model activations, as illustrated in Figure 6 (b). Specifically, after each operation (i.e., MSA, FFN and layer normalization) in the pre-trained model, a SSF-ADA layer is injected, which performs scaling and shifting to the features generated from the operation. During fine-tuning, only those SSF-ADA layers can be updated, while during inference, similar to (IA) , these SSF-ADA layers can be merged into model weights, so no additional inference overhead would be incurred. IPA (InferenceTime Policy Adapters) [56] offers a novel approach to align LLMs, such as GPT-4, with user-specific requirements without modifying the base model's parameters. This is particularly significant when dealing with models whose parameters are extremely large and often not directly accessible. IPA achieves this by combining (through multiplication and normalization) the output distribution of a base LLM (base policy) with that of a smaller-sized model (adapter policy) during the decoding phase. During training, the policy adapter's parameters are fine-tuned using reinforcement learning, while the base policy's parameters remain fixed. During inference, IPA decodes with the combined distribution of the base model and the trained policy adapter, tailoring it to fulfill specific user-defined criteria.
其中 是 Hadamard 乘积。此外，比例向量 和 可以无缝集成到 和 的权重矩阵中。这种集成有效地消除了推断过程中的额外计算成本。类似的技术 SSF [55]也对模型激活进行线性变换，如图 6（b）所示。具体来说，在预训练模型的每个操作（即 MSA、FFN 和层归一化）之后，注入了一个 SSF-ADA 层，对从操作生成的特征进行缩放和移位。在微调过程中，只有那些 SSF-ADA 层可以更新，而在推断过程中，类似于（IA） ，这些 SSF-ADA 层可以合并到模型权重中，因此不会产生额外的推断开销。IPA（推断时间策略适配器）[56]提供了一种新颖的方法来使LLMs（如 GPT-4）与用户特定要求对齐，而无需修改基础模型的参数。当处理参数极大且通常无法直接访问的模型时，这一点尤为重要。 IPA 通过在解码阶段将基础LLM（基础策略）的输出分布与较小尺寸模型（适配器策略）的输出分布进行组合（通过乘法和归一化）来实现这一点。在训练期间，通过强化学习微调策略适配器的参数，而基础策略的参数保持不变。在推断期间，IPA 使用基础模型和经过训练的策略适配器的组合分布进行解码，以满足特定用户定义的标准。

Rather than additive PEFT, which increases the model complexity by adding more parameters, selective PEFT finetunes a subset of the existing parameters to enhance model performance over downstream tasks, as depicted in Figure 4 (b).
与增加参数的 PEFT 不同，选择性 PEFT 通过微调现有参数的子集来提高模型在下游任务中的性能，如图 4（b）所示。

Fig. 7: Illustration of how two parameter masking methods. Blue represents frozen, while yellow represents trainable.
图 7：两种参数掩码方法的示例。蓝色代表冻结，黄色代表可训练。

Specifically, given a model with parameters where each denotes an individual model parameter and represents the total count of these parameters, the process of selective PEFT is represented by applying a binary mask to these parameters. Each in is either 0 or 1 , indicating whether the corresponding parameter is selected (1) or not selected (0) for fine-tuning. The updated parameter set after fine-tuning is given by:
具体来说，给定具有参数 的模型，其中每个 表示一个单独的模型参数， 表示这些参数的总数，选择性 PEFT 的过程通过将二进制掩码 应用于这些参数来表示。 中的每个 都是 0 或 1，表示相应参数 是否被选择（1）或未被选择（0）进行微调。微调后的更新参数集 如下：

where represents the learning rate, and is the gradient of the loss function with respect to the parameter . In this formulation, only the parameters that are selected (i.e., are updated during backpropagation.
其中 代表学习率， 代表损失函数相对于参数 的梯度。在这个公式中，只有被选中的参数（即 ）在反向传播过程中被更新。

Diff pruning [57] is a representative work that applies a learnable binary mask to the model weights during finetuning. To achieve parameter efficiency, the mask is regularized by a differentiable approximation of the -norm penalty. PaFi [59] simply select model parameters with the smallest absolute magnitude as trainable. FishMask [60] determines parameter importance using the approximate Fisher information. It then selects the top parameters based on this information to form the mask M. Similarly, Fish-Dip [61] also uses Fisher information to calculate , but the mask will be re-calculated dynamically in each train period. LT-SFT [62] introduces another technique to determines parameter importance inspired by the Lottery Ticket Hypothesis [100], [101], where the the subset of parameters that change the most during an initial fine-tuning stage is selected to form the mask . SAM [63] proposes a second-order approximation method, which approximates the original problem with an analytically solvable optimization function, to help decide the parameter mask. Child-tuning [64] proposes two approaches to select a child network during each training iteration, where only the parameters within this child network can be updated.
差分修剪[57]是一项代表性工作，它在微调过程中将可学习的二进制掩码应用于模型权重。为了实现参数效率，该掩码通过对 -范数惩罚的可微近似进行正则化。PaFi [59]简单地选择具有最小绝对值的模型参数作为可训练参数。FishMask [60]使用近似的 Fisher 信息确定参数重要性。然后根据这些信息选择前 个参数来形成掩码 M。类似地，Fish-Dip [61]也使用 Fisher 信息来计算 ，但掩码将在每个训练周期动态重新计算。LT-SFT [62]引入了另一种技术，该技术受到“彩票票据假设”[100]，[101]的启发，该假设选择在初始微调阶段中变化最大的参数子集以形成掩码 。SAM [63]提出了一种二阶近似方法，该方法用可解析的优化函数近似原始问题，以帮助决定参数掩码。 Child-tuning [64]在每次训练迭代中提出了两种选择子网络的方法，只有该子网络内的参数可以被更新。

However, above unstructured parameter masking results in an uneven distribution of non-zero masks and diminished hardware efficiency when implementing PEFT. As shown in Figure 7, the structured mask organize parameter masking in regular patterns, unlike unstructured ones that apply it randomly, thus can enhances computational and hardware efficiency during training. Therefore, various structured selective PEFT techniques have undergone extensive investigation. Diff pruning proposes a structured pruning strategy by partitioning the weight parameters into local groups and strategically eliminating them together. Similarly, FAR [65]
然而，上述的非结构化参数掩码会导致非零掩码的分布不均匀，并在实施 PEFT 时降低硬件效率。如图 7 所示，结构化掩码以规则模式组织参数掩码，不像非结构化掩码那样随机应用，因此可以增强训练过程中的计算和硬件效率。因此，各种结构化选择性 PEFT 技术已经进行了广泛的研究。差异剪枝提出了一种结构化剪枝策略，通过将权重参数分成局部组并有策略地一起消除它们。类似地，FAR【65】

(a) LoRA

(b) DyLoRA DyLoRA

(c) DoRA

Fig. 8: Illustration of three representative reparameterized PEFT algorithms. Blue represents frozen, while yellow represents trainable.
图 8：三种代表性的重新参数化 PEFT 算法示例。蓝色代表冻结，黄色代表可训练。

fine-tunes BERT models by grouping weights of the FFN in Transformer blocks into nodes, then rank and select the learner nodes using norm. To further reduce the memory access frequency, they also reconfigure the FFN by grouping the learner nodes together. Bitfit [66] is proposed to only finetunes the bias parameters of each DNN layer, and achieves competitive results for small models. However, this method fails to handle large models. The work by [58] applies NAS to Bitfit, where S-BitFit keeps the structural nature in Bitfit that restrict NAS algorithm must choose whether or not for each bias module. Similar to Bitfit that fine-tunes a specific module in Transformer, Xattn Tuning [67] finetunes only the cross-attention layers. SPT (sensitivity-aware visual parameter-efficient fine-tuning) [68] first identifies the sensitive parameters measured by the loss reduction when being tuned. This sensitivity is calculated using a first-order Taylor expansion, derived from a single forward and backward pass before fine-tuning in oneshot. Next, SPT finds the weight matrices whose number of sensitive parameters exceeds a predefined threshold, and then applies a selected PEFT technique (e.g., LoRA and Adapter) to these targeted weights to achieve structural tuning.
通过将 Transformer 块中 FFN 的权重分组到节点中，然后使用 范数对学习节点进行排名和选择，来微调 BERT 模型。为了进一步减少内存访问频率，他们还通过将学习节点分组来重新配置 FFN。Bitfit [66] 提出仅微调每个 DNN 层的偏置参数，并且对于小模型取得了竞争性结果。然而，这种方法无法处理大模型。[58]的工作将 NAS 应用于 Bitfit，其中 S-BitFit 保留了 Bitfit 中的结构性质，限制 NAS 算法必须为每个偏置模块选择是否 。与 Bitfit 类似，Xattn Tuning [67] 仅微调交叉注意力层。SPT（敏感性感知视觉参数高效微调）[68]首先确定通过调整时损失减少来测量的敏感参数。这种敏感性是使用一阶泰勒展开计算的，在微调之前通过单次前向和后向传递导出。 接下来，SPT 找到敏感参数数量超过预定义阈值的权重矩阵，然后对这些目标权重应用选定的 PEFT 技术（例如，LoRA 和 Adapter）以实现结构调整。

Reparameterization stands for equivalently transforming a model's architecture from one to another via transforming its parameters. In the context of PEFT, this often means constructing a low-rank parameterization to achieve the goal of parameter efficiency during training. For inference, the model can be converted to its original weight parameterization, ensuring unchanged inference speed. This procedure is depicted in Figure 4 (c).
重新参数化是指通过转换模型的参数，等效地将模型的架构从一个转换为另一个。在 PEFT 的背景下，这通常意味着构建一个低秩参数化，以在训练过程中实现参数效率的目标。对于推断，模型可以转换为其原始权重参数化，确保推断速度不变。该过程如图 4（c）所示。

Earlier research studies [69] have shown that common pre-trained models exhibit an exceptionally low intrinsic dimensionality. In other words, it is possible to find a lowdimensional reparameterization that is effective for fine-tuning as the entire parameter space. Intrinsic SAID [69] is the pioneering work in investigating the intrinsic dimension feature during the fine-tuning of LLMs. However, the most widely recognized reparameterization technique is LoRA (Low Rank Adaptation) [70], [102], as shown in Figure 8] (a). For a given pre-trained weight matrix , LoRA introduces two trainable weight matrices, and where the rank , operating in parallel to . Let represent the input. Under normal conditions, the output through is . Instead, LoRA modifies this output by introducing an incremental update that encapsulates task-specific knowledge:
较早的研究[69]表明，常见的预训练模型表现出异常低的内在维度。换句话说，可以找到一个低维重参数化，对整个参数空间进行微调是有效的。内在 SAID[69]是在LLMs微调过程中研究内在维度特征的开创性工作。然而，最广泛认可的重参数化技术是 LoRA（低秩适应）[70]，[102]，如图 8]（a）所示。对于给定的预训练权重矩阵 ，LoRA 引入两个可训练的权重矩阵 和 ，其中秩 ，并行操作 。让 代表输入。在正常情况下，通过 的输出是 。相反，LoRA 通过引入一个包含特定任务知识的增量更新 来修改这个输出。

where denotes a scaling factor. At the onset of training, is initialized using a random Gaussian distribution, while is initialized to zero, ensuring that initially holds a value of zero. LoRA is straightforward to implement and has been evaluated on models with up to 175 billion parameters. Fig 8 (c) used a single decoder as an example, the frozen and learnable components are highlighted in grey and red, respectively. Once fine-tuning is complete, LoRA's adaptive weights seamlessly integrate with the pre-trained backbone weights. This integration ensures that LoRA maintains the model's efficiency, adding no extra burden during inference.
其中 表示缩放因子。在训练开始时， 使用随机高斯分布进行初始化，而 初始化为零，确保 最初保持为零。LoRA 易于实现，并已在具有高达 1750 亿参数的模型上进行了评估。图 8（c）以单个解码器为例，冻结和可学习组件分别用灰色和红色突出显示。一旦微调完成，LoRA 的自适应权重将与预训练的骨干权重无缝集成。这种集成确保 LoRA 保持模型的效率，在推理过程中不增加额外负担。

In LoRA training, selecting an appropriate rank has always been a challenging issue. To address this, DyLoRA [76], as depicted in Figure 8 (b), trains LoRA module on a range of ranks within a predefined training budget, rather than adhering to a single, fixed rank. Specifically, for a given rank range , DyLoRA dynamically chooses a rank at each iteration of the training process. Consequently, the matrices and are tailored for the selected rank , resulting in truncated versions and , and the subsequent forward and backward pass during this iteration will be restricted on and instead of and . With this dynamic and search-free approach, DyLoRA significantly reduces the training time required to find an optimal and fixed LoRA rank for specific tasks. AdaLoRA [77] reformulates the with a singular value decomposition (SVD), denoted as , where and are orthometric, is a diagonal matrix containing sigular values . All the three weight matrices are made learnable. During training, the singular values are pruned iteratively based on their importance scores, which are constructed from moving average of the magnitude of gradient-weight product.
在 LoRA 训练中，选择适当的秩一直是一个具有挑战性的问题。为了解决这个问题，DyLoRA [76]，如图 8（b）所示，对 LoRA 模块进行训练，使用预定义的训练预算范围内的一系列秩，而不是坚持使用单一的固定秩。具体而言，对于给定的秩范围 ，DyLoRA 在训练过程的每次迭代中动态选择一个秩 。因此，矩阵 和 被调整为选定的秩 ，导致截断版本 和 ，在此迭代期间的后续前向和后向传递将受到限制 和 ，而不是 和 。通过这种动态和无搜索的方法，DyLoRA 显著减少了寻找特定任务的最佳和固定 LoRA 秩所需的训练时间。AdaLoRA [77]重新构造了 ，使用奇异值分解（SVD）表示为 ，其中 和 是正交的， 是包含奇异值 的对角矩阵。所有三个权重矩阵都是可学习的。 在训练过程中，奇异值根据其重要性分数进行迭代修剪，这些分数是根据梯度权重乘积的幅度的移动平均值构建的。

To ensure the orthogonality between and , i.e., , an additional regularizer term is included in the loss:
为了确保 和 之间的正交性，即 ，损失函数中包含了一个额外的正则化项：

This adaptive approach enables the model to dynamically adjust the rank within each LoRA module, effectively managing its parameter counts based on the significance of the weight matrices. However, according to SoRA [78], the importance scores used in AdaLoRA is heuristically constructed, which lacks rigorous theoretical motivation. Additionally, both moving average operation and calculation of Eq. 13 introduces extra computation cost during training. To address this, SoRA eliminates the orthogonality premise of and . Instead, a gating unit between and is directly applied and optimized:
这种自适应方法使模型能够动态调整每个 LoRA 模块内的排名，根据权重矩阵的重要性有效地管理其参数计数。然而，根据 SoRA [78]，AdaLoRA 中使用的重要性分数是经验性构建的，缺乏严格的理论动机。此外，在训练过程中，移动平均操作和计算 Eq. 13 都会引入额外的计算成本。为了解决这个问题，SoRA 消除了 和 之间的正交前提。相反，直接应用和优化了 和 之间的一个门控单元 ：

where is Hadamard product. The gate is updated using a variation of proximal gradient iteration for loss [103], [104], which has a clear mathematical meaning and do not need the heuristic premise. After training, the zeroed-out gate units are pruned by removing the corresponding columns and rows in and .
其中 是 Hadamard 乘积。门 使用一种变体的近端梯度迭代来更新 损失[103]，[104]，具有明确的数学意义，不需要启发式前提。训练后，通过在 和 中移除相应的列和行来修剪归零门单元。

Several subsequent studies have aimed to improve LoRA's performance in various aspects. For instance, LaplaceLoRA [81] notices that fine-tuned LLMs often exhibit overconfidence. To enhance the calibration of fine-tuned LLMs, Laplace-LoRA utilizes a Bayesian approach, specifically a post-hoc Laplace approximation [105], [106], to the posterior over the LoRA parameters. LoRA Dropout [82] introduces random noises to the learnable low-rank matrices and increases parameter sparsity to reduce the risk of overfitting. LoRA+ [84] proposes to set different learning rates for the LoRA matrices and , such that with fixed and tune .
几项后续研究旨在改善 LoRA 在各个方面的性能。例如，LaplaceLoRA [81] 发现，经过微调的LLMs通常表现出过度自信。为了增强经过微调的LLMs的校准性，Laplace-LoRA 采用了一种贝叶斯方法，具体来说是一种事后 Laplace 近似[105]，[106]，用于对 LoRA 参数的后验进行建模。LoRA Dropout [82] 引入随机噪声到可学习的低秩矩阵中，并增加参数稀疏性以减少过拟合的风险。LoRA+ [84] 建议为 LoRA 矩阵 和 设置不同的学习率，使得 固定而 。

Thanks to the modular design of LoRA, many studies incorporate multiple LoRA modules in their frameworks to enhance performance. For example, LoRAHub aggregates various LoRA modules trained on different tasks. Given a handful of examples from a new task, LoRAHub can autonomously compose compatible LoRA modules without human intervention via a gradient-free method Shiwa [107]. MOELoRA employs a Mixture-of-Experts (MOE) approach to train LoRA in a multi-task setting, resulting in multiple expert LoRA modules. To retrieve parameters for certain task, MOELoRA utilizes a task-motivated gate function that assigns contribution weights to each expert based on the task ID, and the final parameters is calculated through a weighted sum of all experts.
由于 LoRA 的模块化设计，许多研究在其框架中结合多个 LoRA 模块以增强性能。例如，LoRAHub 聚合了在不同任务上训练的各种 LoRA 模块。给定一个新任务的一些示例，LoRAHub 可以通过无梯度方法 Shiwa[107]自动组合兼容的 LoRA 模块，无需人工干预。MOELoRA 采用专家混合（MOE）方法在多任务设置中训练 LoRA，从而产生多个专家 LoRA 模块。为了检索特定任务的参数，MOELoRA 利用一个任务驱动的门函数，根据任务 ID 为每个专家分配贡献权重，最终参数通过所有专家的加权和计算得出。

In addition to LoRA, several other reparameterization techniques are emerging with significant potential. For instance, Compacter [71] introduces a light-weight adapter modules by parameterizing the and as , where , and denotes the Kronecker product. They further decrease the parameter count by designating as shared parameters and reparameterizing using the product of two low-rank matrices, effectively reducing the parameter complexity from to . Related studies, such as KronA [72] and KAdaptation [73], also employ the Kronecker product to reparameterize adapter weights, aiming to achieve parameter reduction. HiWi [59] proposes an adapter fine-tuning method that applies an adapter directly to pretrained parameters instead of hidden representations as:
除了 LoRA 之外，还出现了几种具有重要潜力的重新参数化技术。例如，Compacter [71] 引入了一个轻量级的适配器模块，通过将 和 参数化为 ，其中 ， 表示 Kronecker 积。他们进一步通过将 指定为共享参数并重新参数化 为两个低秩矩阵的乘积，有效地将参数复杂度从 降低到 。相关研究，如 KronA [72] 和 KAdaptation [73]，也利用 Kronecker 积重新参数化适配器权重，旨在实现参数减少。HiWi [59] 提出了一种适配器微调方法，将适配器直接应用于预训练参数而不是隐藏表示，如下所示：

where denotes the weights or biases within the Transformer block's feed-forward layer. Notably, during inference, this method computes in advance, ensuring that the model's inference latency remains on par with that of traditional full fine-tuning. VeRA (Vector-based Random Matrix Adaptation) [74] employs a single pair of frozen low-rank matrices and that are shared across all layers, and adapts these matrices by learning small, trainable scaling vectors represented as and (formally denoted by diagonal matrices and . Specifically, the reparameterization is given by:
其中 表示 Transformer 块的前馈层内的权重或偏置。值得注意的是，在推断期间，该方法提前计算 ，确保模型的推断延迟与传统的完全微调相当。VeRA（基于向量的随机矩阵适应）[74]采用一对冻结的低秩矩阵 和 ，这些矩阵在所有层之间共享，并通过学习表示为 和 的小型可训练缩放向量来调整这些矩阵（正式表示为对角矩阵 和 ）。具体而言，重新参数化如下：

where both and are initialized using a random Gaussian distribution. Similar to LoRA, the scaling vector is initialized to zeros to ensure that the weight matrix is unaffected during the first forward pass. This method significantly reduces the number of trainable parameters compared to LoRA yet maintains the same performance, enabling the fine-tuning of larger models on a single GPU. DoRA (WeightDecomposed Low-Rank Adaptation) [75] presents a novel approach as illustrated in Figure 8 (c) by decomposing model weights into magnitude and direction as follows:
其中 和 都是使用随机高斯分布进行初始化的。与 LoRA 类似，缩放向量 被初始化为零，以确保在第一次前向传递期间权重矩阵不受影响。这种方法与 LoRA 相比显著减少了可训练参数的数量，但保持了相同的性能，使得可以在单个 GPU 上对更大的模型进行微调。DoRA（WeightDecomposed Low-Rank Adaptation）[75]提出了一种新颖的方法，如图 8（c）所示，通过将模型权重 分解为大小和方向。

where is the magnitude vector, is the directional matrix, with being the vector-wise norm of a matrix across each column. Subsequently, DoRA adopts a unique fine-tuning strategy for and . While both are tunable, only undergoes LoRA reparameterization, defined as:
其中 是幅度向量， 是方向矩阵， 是矩阵在每列上的向量范数。随后，DoRA 采用了一种独特的 和 的微调策略。虽然两者都是可调的，但只有 经历 LoRA 重新参数化，定义为：

where is the incremental directional update learned by LoRA, and the underlined parameters denote the trainable parameters. Through this methodology, DoRA consistently outperforms LoRA across various tasks and models, demonstrating its superiority.
其中 是 LoRA 学习到的增量方向更新，下划线参数表示可训练参数。通过这种方法，DoRA 在各种任务和模型中始终优于 LoRA，展示了其优越性。

The efficacy of various PEFT methods can significantly differ across different tasks. As a result, numerous studies aim to either combine the advantages of diverse PEFT approaches or seek to establish a unified perspective by analyzing the similarities among these methods. For instance, UniPELT[90] integrates LoRA, prefix-tuning, and adapters into each Transformer block. To control which PEFT submodules should be activated, they also introduce a gating mechanism. This
各种 PEFT 方法的功效在不同任务中可能会有显著差异。因此，许多研究旨在要么结合不同 PEFT 方法的优势，要么通过分析这些方法之间的相似之处来建立统一的视角。例如，UniPELT[90]将 LoRA、前缀调整和适配器集成到每个 Transformer 块中。为了控制哪些 PEFT 子模块应该被激活，他们还引入了一个门控机制。
mechanism consists of three small FFNs that each produce a scalar value , which is then applied to the LoRA, prefix, and adapter matrices, respectively. Across various setups, UniPELT has consistently shown improvements in accuracy ranging from to . S4 [91] explores design spaces for several PEFT methods (i.e., Adapter (A), Prefix (P), BitFit (B), and LoRA (L)) to uncover underlying design patterns. After a series experiments, their findings include: (1) Applying the spindle grouping partitioning for Transformer layers, which results in four layer groups for . Layers in one group have similar behaviors together, which means should be apply similar PEFT strategies. (2) Allocating the number of trainable parameters to layers uniformly. (3) Tuning all the groups. (4) Assigning different PEFT strategies in different group. The resulting design space that has the best performance is:
机制由三个小的 FFN 组成，每个都产生一个标量值 ，然后分别应用于 LoRA、前缀和适配器矩阵。在各种设置中，UniPELT 一直显示出准确性的改进，范围从 到 。S4 [91] 探索了几种 PEFT 方法（即适配器（A）、前缀（P）、BitFit（B）和 LoRA（L））的设计空间，以揭示潜在的设计模式。经过一系列实验，他们的发现包括：（1）为 Transformer 层应用主轴分组分区，结果是四个层组 为 。一个组中的层具有相似的行为，这意味着应该应用相似的 PEFT 策略。 （2）将可训练参数的数量均匀分配给各层。 （3）调整所有组。 （4）在不同组中分配不同的 PEFT 策略。具有最佳性能的结果设计空间是：

MAM Adapter[26] explores the intrinsic similarity between three additive PEFT methods: adapters, prefix-tuning, and LoRA, which leads to the development of three variants: Parallel Adapter, which places adapter layers alongside specific layers (SA or FFN) instead of after them; Multi-head Parallel Adapter, which divides the parallel adapter into multiple heads, each affecting the head attention output in SA; and Scaled Parallel Adapter, which adds a scaling term after the parallel adapter layer, similar to LoRA. Extensive experimentation revealed that the most effective configuration involves using prefix-tuning in the SA layer and the scaled parallel adapter in the FFN layer, which is called MAM Adapter. LLM-Adapters [94] builds an easy-to-use framework that incorporates various PEFT techniques into LLMs. Through comprehensive benchmarking across multiple datasets, the study reveals several key insights: (1) The most effective locations for series adapters, parallel adapters, and LoRA are after the MLP layers, alongside the MLP layers, and simultaneously following the Attention layers and MLP layers, respectively. (2) Smaller LLMs utilizing PEFT can achieve competitive or even superior results on certain tasks when compared to their larger counterparts. (3) With appropriate in-distribution fine-tuning data, smaller models are capable of surpassing larger models in task-specific performance.
MAM 适配器[26] 探索了三种附加 PEFT 方法之间的内在相似性：适配器、前缀调整和 LoRA，这导致了三种变体的发展：并行适配器，将适配器层放置在特定层（SA 或 FFN）旁边，而不是在它们之后；多头并行适配器，将并行适配器分成多个头部，每个头部影响 SA 中的头部注意力输出；以及缩放并行适配器，在并行适配器层之后添加一个缩放项，类似于 LoRA。广泛的实验揭示了最有效的配置涉及在 SA 层中使用前缀调整和在 FFN 层中使用缩放并行适配器，这被称为 MAM 适配器。LLM-适配器[94] 构建了一个易于使用的框架，将各种 PEFT 技术整合到 LLMs 中。通过在多个数据集上进行全面基准测试，研究揭示了几个关键见解：（1）系列适配器、并行适配器和 LoRA 的最有效位置分别是在 MLP 层之后、与 MLP 层并行以及同时跟随注意力层和 MLP 层之后。 (2) 利用 PEFT 的较小模型在某些任务上可以取得竞争力甚至优于较大模型的结果。(3) 通过适当的分布内微调数据，较小模型能够在特定任务的性能上超越较大模型。

Several studies leverage neural architecture search (NAS) to find better PEFT combination approaches. For example, NOAH [92] discovers that different PEFT configurations are specifically tailored for different tasks. To address this issue, NOAH employs NAS to identify the most effective PEFT configurations for each dataset. Specifically, NOAH's searching space encompasses three PEFT methods: Adapter, LoRA, and Visual Prompt Tuning (VPT). It utilizes AutoFormer [108], a one-shot NAS algorithm, for the efficient discovery of optimal prompt modules. In a related vein, AUTOPEFT [93] first establishes a searching space that includes serial adapters, parallel adapters, and prefix tuning. After that, they proposes an effective NAS methods based on a high-dimensional multidimensional Bayesian optimisation [109]. Both NOAH and AUTOPEFT demonstrate the capability of NAS in enhancing PEFT configurations across a variety of tasks.
几项研究利用神经架构搜索（NAS）来寻找更好的 PEFT 组合方法。例如，NOAH [92] 发现不同的 PEFT 配置专门针对不同的任务定制。为了解决这个问题，NOAH 利用 NAS 来识别每个数据集中最有效的 PEFT 配置。具体来说，NOAH 的搜索空间包括三种 PEFT 方法：Adapter、LoRA 和 Visual Prompt Tuning（VPT）。它利用 AutoFormer [108]，一种一次性 NAS 算法，高效地发现最佳提示模块。在相关领域，AUTOPEFT [93] 首先建立了一个搜索空间，其中包括串行适配器、并行适配器和前缀调整。之后，他们提出了一种基于高维多维贝叶斯优化的有效 NAS 方法 [109]。NOAH 和 AUTOPEFT 都展示了 NAS 在增强各种任务中的 PEFT 配置方面的能力。

Processing latency and peak memory overhead are pivotal factors to consider from a computational standpoint. This section introduces a key characteristic in LLMs aimed at balancing between latency and memory usage (Section IV-A). Following this, we explore strategies for developing efficient PEFT methods to address computational challenges, including PEFT pruning (Section IV-B), PEFT quantization (Section IV-C), and memory-efficient PEFT techniques (Section IV-D), each designed to enhance model performance while minimizing resource consumption. It is noteworthy that quantization inherently addresses memory overhead concerns. However, given its distinct characteristics, we address these quantization methods separately rather than incorporating them under the memory-efficient PEFT section.
处理延迟和峰值内存开销是从计算角度考虑的关键因素。本节介绍了LLMs中旨在平衡延迟和内存使用之间的关键特征（第 IV-A 节）。在此之后，我们探讨了开发高效 PEFT 方法以解决计算挑战的策略，包括 PEFT 修剪（第 IV-B 节）、PEFT 量化（第 IV-C 节）和内存高效 PEFT 技术（第 IV-D 节），每种方法都旨在提高模型性能同时最大限度地减少资源消耗。值得注意的是，量化本质上解决了内存开销问题。然而，鉴于其独特特性，我们将这些量化方法单独处理，而不是将它们纳入内存高效 PEFT 部分。

The core of the LLMs model lies an autoregressive Transformer model, depicted in Figure 2 When we look at autoregression characteristic, it becomes a major challenge in designing an inference system, because every time a new token is generated, the entire LLM model has to transfer all the weights from different memories to the memory of the graphics processor, which is very unfriendly to single-user task scheduling or multi-user work-load balance. The challenging part of serving the auto-regressive paradigm is all previous sequences have to be cached and saved for the next proceeding iteration, the cached activation generated from the previous sequences is stored as the Key-Value Cache (KV-cache).
LLMs模型的核心是一个自回归 Transformer 模型，如图 2 所示。当我们看到自回归特性时，它成为设计推理系统的一个主要挑战，因为每次生成一个新的标记时，整个LLM模型都必须将所有权重从不同的内存传输到图形处理器的内存，这对于单用户任务调度或多用户工作负载平衡非常不友好。为提供自回归范式的挑战在于必须缓存和保存所有先前的序列以供下一个迭代使用，从先前序列生成的缓存激活被存储为键值缓存（KV-cache）。

The storage of KV-cache will cost both memory space and IO performance, yielding in workload memory-bounded and under-utilizing the computation power of the system. Previous works proposed a series of solutions like KV-cache control management [133] or KV-cache compression [134] to improve throughput or reduce latency. When designing PEFT methods, it is crucial to consider the characteristics of the KV-cache to complement its features. For instance, when applying soft prompts in the inference phase, efficiently leveraging the KVcache for these additional inputs can help accelerate response times by ensuring prompt-related data is readily accessible.
KV-cache 的存储将消耗内存空间和 IO 性能，导致工作负载受内存限制且系统的计算能力被低效利用。先前的研究提出了一系列解决方案，如 KV-cache 控制管理[133]或 KV-cache 压缩[134]，以提高吞吐量或减少延迟。在设计 PEFT 方法时，关键是考虑 KV-cache 的特性以补充其功能。例如，在推理阶段应用软提示时，有效利用 KV-cache 来处理这些额外输入可以通过确保相关数据可立即访问来加快响应时间。

The inclusion of pruning can substantially enhance the efficiency of PEFT methods. In particular, AdapterDrop [110] explores the removal of adapters from lower transformer layers and multi-task adapters in AdapterFusion [29], which shows that the pruning can improve the training and inference efficiency with minimal decrease in performance. SparseAdapter [111] investigates different pruning methods and finds that high sparsity ratios can outperform standard adapters. Additionally, the Large-Sparse configuration, which increases the bottleneck dimension while maintaining a constant parameter budget (e.g., doubling dimensions with a sparsity), substantially enhances the model's capacity, resulting in improved performance. SPLoRA [112] adopts channel-based pruning to the LoRA weights and .
修剪的加入可以显著提高 PEFT 方法的效率。具体而言，AdapterDrop [110] 探索了从较低的变压器层和 AdapterFusion [29] 中的多任务适配器中移除适配器，表明修剪可以在最小性能降低的情况下提高训练和推理效率。SparseAdapter [111] 研究了不同的修剪方法，并发现高稀疏度比率可以胜过标准适配器。此外，增加瓶颈维度同时保持恒定参数预算（例如，通过 1 稀疏度加倍维度）的 Large-Sparse 配置显著增强了模型的容量，从而提高了性能。SPLoRA [112] 采用基于通道的修剪来修剪 LoRA 权重。

BI-Adapter [115], PEQA [116], QLoRA [117], LoftQ [118], LQ-LoRA [119], QA-LoRA [120], INT2.1 [121], QDyLoRA [122], BitDelta [123]

Fig. 9: Taxonomy of Efficient PEFT Design.
图 9：高效 PEFT 设计分类。

This pruning affects not only the source weights , but also the LoRA parameters and . Similarly, LoRAPruning [113] adopts structured pruning not only to the pretrained model weights but also to the LoRA weights. In contrast to unstructured LoRA pruning methods, which primarily focus on sparsifying model weights while leaving LoRA weights dense, thus making weight merging challenging to achieve, LoRAPruning enables the weights to be merged easily. Additionally, this work also introduces a novel criterion that utilizes LoRA's gradients as an approximation of the gradients for the pre-trained weights, enabling the estimation of weight importance. ProPETL [114] constructs a single shared prototype (e.g., adapter, prefix, or LoRA) across layers and tasks. In addition, ProPETL learns binary masks to prune different sub-networks in different layers and tasks. As a result, the parameters can be reused in across layers and tasks, largely increasing the parameter efficiency.
这种修剪不仅影响源权重 ，还影响 LoRA 参数 和 。类似地，LoRAPruning [113]采用结构化修剪，不仅适用于预训练模型的权重，还适用于 LoRA 的权重。与非结构化的 LoRA 修剪方法相比，后者主要专注于稀疏化模型权重，同时保持 LoRA 权重密集，从而使权重合并难以实现，LoRAPruning 使权重可以轻松合并。此外，该工作还引入了一种新颖的标准，利用 LoRA 的梯度作为预训练权重梯度的近似，从而实现权重重要性的估计。ProPETL [114]在层和任务之间构建一个共享的原型（例如，适配器、前缀或 LoRA）。此外，ProPETL 学习二进制掩码，以修剪不同层和任务中的不同子网络。因此，参数可以在层和任务之间重复使用，大大提高了参数效率。

Quantization serves as another popular technique for improving computational efficiency and reduce memory usage. For example, by investigating the loss landscape of adapters, BI-Adapter [115] finds that adapters are resistant to noise in parameter space. Building on this insight, the authors introduce a clustering-based quantization approach. Remarkably, they demonstrate that a 1 -bit quantization of adapters not only minimizes storage requirements but also achieve superior performance among all precision settings. PEQA (Parameter-Efficient and Quantization-aware Adaptation) [116] uses a two-stage pipeline to achieve parameterefficient and quantization-aware fine-tuning. In the first stage, the pre-trained FFN weight matrix is quantized to , where represents per-channel scales and denotes the quantized weight. In the second stage, remains fixed, and fine-tuning is only conducted on . This approach not only ensures memory efficiency but also facilitates parameter efficiency. QLoRA [117] proposes several novel techniques, including a 4-bit NormalFloat, a Double Quantization, and a Paged Optimizers, to backpropagate a 4-bit quantized pretrained language model into LoRA. These techniques enable the fine-tuning for a 65B language model on a single 48GB GPU while maintaining similar performance to the full 16-bit fine-tuning. Similar to the original implementation [70], QLoRA attaches the fixed zero initialized LoRA weights to the quantized pre-trained model as the training start point. However, when applying the extreme low-bit (e.g., 2-bit) quantization, the huge quantization error can adversely impact the initialization of LoRA fine-tuning, i.e., quantization where , which will harm the fine-tuning performance as shown in [127]. To solve this, several quantization strategies are proposed to eliminate the quantization error. For example, LoftQ (LoRA-Fine-Tuningaware Quantization) [118] presents an innovative framework that provides a superior initialization point of quantized backbone weights and LoRA weights for subsequent LoRA finetuning. This approach addresses the discrepancies caused by quantization through the optimization of a Frobenius norm objective during network initialization, which takes both the LoRA weights and the quantized pre-trained backbone into consideration. LoftQ exhibits superior performance in 2-bit quantization over QLoRA, as well as greater generalization for downstream tasks. LQ-LoRA [119] uses an iterative algorithm inspired by robust principal components analysis [135], [136] which decomposes the weight such that to resolve the inaccuracy caused by the quantization error, where is the quantized component which remains fixed and is the trainable low-rank component. Moreover, this approach leverages integer linear programming to determine a mixed quantization strategy, enabling dynamic quantization configurations for each weight matrix while adhering to a predetermined total bit rate limit. QA-LoRA [120] address another limitation of QLoRA, which struggles to preserve its quantized property post fine-tuning. In QLoRA, the quantized pre-trained weight (NF4) have to be recovered to FP16 to match the LoRA weight precision (FP16) during weight merging. Instead, QA-LoRA uses INT4 quantization and introduces group-wise operators to enable quantization during inference stage, therefore improve the efficiency and accuracy compared with QLoRA. BitDelta [123] introduces a novel 1-bit posttraining quantization method that acts on the weight delta between a fine-tuned model and its underlying pre-trained model. Specifically, given the weight matrices and from the fine-tuned and base models respectively, the weight delta is binarized as . Here, , a high-precision scalar, is initialized based on the mean absolute delta value , with indicating the sign of . BitDelta further calibrates the scaling factors via distillation on a compact calibration dataset, while the binary matrices remain unchanged. This approach notably streamlines the deployment of multiple fine-tuned models on shared servers by utilizing a singular full-precision base model alongside efficiently batched 1 -bit deltas.
量化是另一种提高计算效率和减少内存使用的流行技术。例如，通过研究适配器的损失景观，BI-Adapter [115] 发现适配器对参数空间中的噪声具有抗性。基于这一洞察力，作者们引入了一种基于聚类的量化方法。值得注意的是，他们证明了适配器的 1 位量化不仅最小化了存储需求，而且在所有精度设置中实现了卓越的性能。PEQA（参数高效和量化感知调整）[116] 使用两阶段流水线实现参数高效和量化感知微调。在第一阶段，预训练的 FFN 权重矩阵 被量化为 ，其中 表示每通道比例， 表示量化权重。在第二阶段， 保持不变，微调仅在 上进行。这种方法不仅确保了内存效率，还促进了参数效率。 QLoRA [117]提出了几种新颖的技术，包括 4 位 NormalFloat、双量化和分页优化器，以将 4 位量化的预训练语言模型反向传播到 LoRA。这些技术使得在单个 48GB GPU 上对 65B 语言模型进行微调成为可能，同时保持与完整 16 位微调相似的性能。与原始实现[70]类似，QLoRA 将固定的零初始化 LoRA 权重附加到量化的预训练模型上作为训练起点。然而，当应用极低位（例如 2 位）量化时，巨大的量化误差可能会对 LoRA 微调的初始化产生不利影响，即量化 ，这将损害微调性能，如[127]所示。为了解决这个问题，提出了几种量化策略来消除量化误差。例如，LoftQ（LoRA-Fine-Tuningaware 量化）[118]提出了一个创新框架，为后续 LoRA 微调提供了量化骨干权重和 LoRA 权重的优越初始化点。 这种方法通过在网络初始化期间优化 Frobenius 范数目标来解决由量化引起的差异，考虑了 LoRA 权重和量化的预训练骨干。LoftQ 在 2 位量化方面表现出比 QLoRA 更优越的性能，同时对下游任务具有更好的泛化能力。LQ-LoRA [119] 使用受鲁棒主成分分析 [135]，[136] 启发的迭代算法，将权重分解为 以解决量化误差引起的不准确性，其中 是保持不变的量化分量， 是可训练的低秩分量。此外，这种方法利用整数线性规划确定混合量化策略，为每个权重矩阵提供动态量化配置，同时遵守预定的总比特率限制。QA-LoRA [120] 解决了 QLoRA 的另一个限制，即在微调后难以保持其量化属性。在 QLoRA 中，量化的预训练权重（NF4）必须恢复为 FP16，以匹配权重合并期间的 LoRA 权重精度（FP16）。 相反，QA-LoRA 使用 INT4 量化并引入分组操作符，以在推断阶段实现量化，从而提高效率和准确性，与 QLoRA 相比。BitDelta [123]引入了一种新颖的 1 位后训练量化方法，作用于经过微调的模型与其基础预训练模型之间的权重增量。具体来说，给定来自经过微调和基础模型的权重矩阵 和 ，权重增量 被二元化为 。在这里， ，一个高精度标量，基于平均绝对增量值 初始化， 表示 的符号。BitDelta 通过蒸馏在紧凑的校准数据集上进一步校准缩放因子，而二进制矩阵保持不变。这种方法通过利用单个全精度基础模型以及高效批处理的 1 位增量，显着简化了在共享服务器上部署多个经过微调的模型。

Fine-tuning the full LLMs necessitates substantial training memory owing to their considerable size. While most PEFT methods primarily target parameter efficiency, they still incur a significant memory overhead during training because gradient computation and backpropagation are still necessary for these methods. For example, prevalent PEFT techniques such as adapters and LoRA can only reduce memory usage to approximately compared to full model fine-tuning according to some literatures [125], [130]. From a computational perspective, memory efficiency also remains a critical factor that cannot be overlooked.

To improve memory efficiency, various techniques have been developed to minimize the need for caching gradients for the entire LLM during fine-tuning, thereby reducing memory usage. For example, both Side-Tuning [124] and LST (Ladder-Side Tuning) [125] introduces a learnable network branch parallel to the backbone model. By channeling the backpropagation exclusively through this parallel branch, it circumvents the need to store gradient information for the main model's weights, thus markedly reducing memory requirements during training. Similarly, Res-Tuning [126] disentangles the PEFT tuners (e.g., prompt tuning, adapter) from the backbone model. On top of the disentanglement, a memoryefficient fine-tuning framework named Res-Tuning-Bypass is proposed, which generates a bypass network in parallel with the backbone model by removing the data flow from the decoupled tuners to the backbone. This eliminates the requirement for gradient caching within the backbone model during backpropagation. MEFT [127] (memory-efficient fine-tuning) is an approach inspired by the reversible model [137]. During the training of a reversible model, intermediate activations are not required to be cached in the forward pass. During backpropagation, they can be recalculated from the final output. To save the memory during fine-tuning, MEFT investigates how to transform an LLM to its reversible counterparts without additional pre-training. A critical aspect of this transformation is the careful initialization of newly-introduced parameters in the pre-trained models. MEFT demonstrates the importance of the parameter initialization, and suggests that these parameters must be initialized in a manner that preserves the pre-trained model's starting point, ensuring that the fine-tuning of the modified model achieves performance on par with full finetuning methods. With this key consideration, MEFT introduces three distinct methods, each significantly curtailing the memory demands traditionally required for storing activations. LoRA-FA [128] addresses a limitation about memory overhead in LoRA fine-tuning. During training, LoRA modules still require high activation memory consumption. This is because, during backpropagation, large input activations must be stored during the forward pass to compute gradients. LoRA-FA resolves this issue by freezing both the pre-trained weights and the projection-down weights , and only updating the projection-up weights . Consequently, the input activation no longer needs to be stored, as the intermediate activation is adequate for gradient computation for . Given that , the memory requirement for activations in LoRA-
为了提高内存效率，已经开发了各种技术来最小化在微调期间对整个LLM进行梯度缓存的需求，从而减少内存使用量。例如，Side-Tuning [124] 和 LST（Ladder-Side Tuning）[125] 都引入了一个可学习的网络分支，与主干模型并行。通过仅通过这个并行分支传递反向传播，它规避了需要存储主模型权重的梯度信息，从而在训练期间显着减少了内存需求。类似地，Res-Tuning [126] 将 PEFT 调谐器（例如，提示调整器，适配器）与主干模型解耦。在解耦的基础上，提出了一种名为 Res-Tuning-Bypass 的内存高效微调框架，通过从解耦的调谐器到主干模型的数据流中移除数据流，生成一个与主干模型并行的旁路网络。这消除了在反向传播期间主干模型内部的梯度缓存要求。MEFT [127]（内存高效微调）是受可逆模型 [137] 启发的方法。 在可逆模型的训练过程中，不需要在前向传播中缓存中间激活。在反向传播过程中，可以从最终输出中重新计算它们。为了在微调过程中节省内存，MEFT 研究了如何将一个LLM转换为可逆对应项，而无需额外的预训练。这种转换的一个关键方面是在预训练模型中谨慎初始化新引入的参数。MEFT 展示了参数初始化的重要性，并建议这些参数必须以一种方式初始化，以保留预训练模型的起始点，确保修改后模型的微调能够达到与完全微调方法相当的性能。在这个关键考虑下，MEFT 提出了三种不同的方法，每种方法都显著减少了传统上用于存储激活所需的内存需求。LoRA-FA [128] 解决了 LoRA 微调中关于内存开销的限制。在训练过程中，LoRA 模块仍然需要高激活内存消耗。 这是因为在反向传播过程中，需要在前向传播期间存储大的输入激活以计算梯度。LoRA-FA 通过冻结预训练权重 和投影下权重 ，仅更新投影上权重 来解决这个问题。因此，输入激活 不再需要被存储，因为中间激活 足以用于 的梯度计算。鉴于 ，在 LoRA 中激活的内存需求-
FA can be significantly reduced.
FA 可以显著减少。