Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
大型模型的参数高效微调：全面调查

Index Terms-Large Language Model, Parameter-Efficient Fine-tuning, Computer System, Distributed System.
索引词条-大型语言模型、参数高效微调、计算机系统、分布式系统。

Large Models (LMs) have recently captured considerable public interest. Their ability to understand context and nuances enables them to proficiently handle diverse tasks across multiple domains, including natural language processing (NLP), computer vision (CV), etc. In the field of NLP, Large Language Models (LLMs) have achieved significant advancements across various tasks including text generation [1], [2], translation [3], [4], personalized chat-bots [5], [6], [7], and summarization [8], demonstrating remarkable proficiency.
大型模型（LM）最近引起了公众的极大兴趣。它们理解上下文和细微差别的能力使其能够熟练处理多个领域的各种任务，包括自然语言处理（NLP）、计算机视觉（CV）等。在 NLP 领域，大型语言模型 (LLMs) 在文本生成[1]、[2]、翻译[3]、[4]、个性化聊天机器人[5]、[6]、[7]和摘要[8]等各种任务中都取得了重大进展，显示出非凡的能力。

Earlier studies [1] has suggested that LLMs exhibit high levels of generalization, enabling them to apply their acquired knowledge to new tasks not included in their original training. This capability is commonly known as zero-shot learning. Nevertheless, fine-tuning remains essential to further enhance LLMs for optimal performance on new user datasets and tasks.
早先的研究[1]表明，LLMs ，他们表现出很高的概括能力，能够将获得的知识应用于原来训练中没有包括的新任务。这种能力通常被称为 "零点学习"。不过，要进一步提高LLMs 在新的用户数据集和任务中的最佳性能，微调仍然是必不可少的。

Due to its scale, a widely adopted strategy for fine-tuning LLMs involves adjusting a limited number of LLM parameters while keeping the remainder unchanged. This technique, termed Parameter-Efficient-Fine-Tuning (PEFT), involves selectively adjusting a small proportion of their parameters, while keeping the rest unaltered. Furthermore, the application of PEFT extends beyond the realm of NLP and quickly attracts interest in the CV community for handling fine-tuning vision models with large parameters, such as Vision Transformers (ViT) and diffusion models, as well as disciplinary models such as vision-language models.
由于其规模庞大，一种被广泛采用的微调LLMs 的策略是调整数量有限的LLM 参数，其余参数保持不变。这种技术被称为 "参数-系数-微调"（Parameter-Efficient-Fine-Tuning，PEFT），即有选择地调整一小部分参数，其余参数保持不变。此外，PEFT 的应用范围还超出了 NLP 领域，并迅速引起了 CV 界对处理微调大参数视觉模型（如视觉变换器（ViT）和扩散模型）以及学科模型（如视觉语言模型）的兴趣。

In this survey, we systematically review and categorize recent advancements in PEFT algorithms as well as the system implementation costs associated with various PEFT algorithms across diverse scenarios. Figure 1 presents the overview content for this survey. In section , we present some fundamental concepts for LLM and PEFT, including computational flow for LLM, basic knowledge of PEFT, and commonly used datasets and tasks.
在本调查中，我们系统地回顾了 PEFT 算法的最新进展，并对其进行了分类，同时还介绍了不同场景下与各种 PEFT 算法相关的系统实施成本。图 1 展示了本调查的概览内容。在 部分，我们介绍了LLM 和 PEFT 的一些基本概念，包括LLM 的计算流程、PEFT 的基本知识以及常用数据集和任务。

We categorized all types of PEFT algorithms in Section III according to their computational flow. In Section III-A, we introduce additive algorithms that either introduce additional weight parameters or modify activations. For algorithms that exclusively require fine-tuning with existing parameters, they fall under the category of selective approaches, and their introduction can be found in Section III-B, In Section III-C, we explore reparameterized PEFT, which constructs a (lowdimensional) reparameterization of original model parameters for training while transforms the weights back to maintain the inference speed. Additionally, there exist algorithms that combine the above techniques, and we have classified these as hybrid approaches, elaborating on them in Section III-D We also investigate strategies for further reducing the computational complexity of different PEFT algorithms, including KV-cache management, pruning, quantization, and memory optimization, in Section IV
我们在第三节中根据计算流程对所有类型的 PEFT 算法进行了分类。在第 III-A 节中，我们介绍了引入额外权重参数或修改激活度的加法算法。在第 III-C 节中，我们探讨了重参数化 PEFT 算法，该算法对原始模型参数进行（低维）重参数化以进行训练，同时将权重转换回来以保持推理速度。此外，还有将上述技术结合起来的算法，我们将其归类为混合方法，并在第 III-D 节中详细阐述。

In Section , we expand the scope of this survey beyond the computational perspective to involve various potential application scenarios. We explore innovations that applying PEFT techniques to different model architecture, including LLMs
在第 节，我们将调查范围从计算角度扩展到各种潜在应用场景。我们探讨了将 PEFT 技术应用于不同模型架构的创新，包括LLMs

Fig. 1: A content overview covered in the survey.
图 1：调查涉及的内容概览。

(Section V-A), Vision Transformer (Section V-B), VisionLanguage alignment models (Section V-C), and Diffusion models (Section V-D), for varied downstream tasks, underscoring PEFT's versatility and applicability in a range of scenarios.
(第 V-A 节）、Vision Transformer（第 V-B 节）、VisionLanguage 对齐模型（第 V-C 节）和 Diffusion 模型（第 V-D 节），以完成不同的下游任务，凸显了 PEFT 在各种情况下的多功能性和适用性。

In Section VI, we explores the system design challenge for PEFT methods. The discussion includes three advanced system solutions for practical PEFT deployment: distributed tuning (Section VI-B), PEFT query serving (Section VI-C), and concurrent PEFT tuning (Section VI-D).
在第六部分，我们探讨了 PEFT 方法的系统设计挑战。讨论包括实用 PEFT 部署的三种先进系统解决方案：分布式调整（VI-B 节）、PEFT 查询服务（VI-C 节）和并发 PEFT 调整（VI-D 节）。

In the last Section VII we summarize our survey and propose several potential future directions from both algorithm and system perspectives, hoping to provide valuable insights for further research and development in the field.
在最后的第七部分，我们总结了我们的调查，并从算法和系统的角度提出了几个潜在的未来发展方向，希望能为这个领域的进一步研究和发展提供有价值的见解。

In this section, we first discussed the computation flow of LLM, including its fundamental components, computational complexity, and the flow of computations it involves as a case study. We then provide a brief overview of different PEFT algorithms in section II-B
在本节中，我们首先讨论了LLM 的计算流程，包括其基本组成部分、计算复杂度以及作为案例研究的计算流程。然后，我们在第 II-B 节中简要介绍了不同的 PEFT 算法。

In order to gain a deeper understanding of LLM and other Transformer-based models, we employ LLaMA-7B, a cuttingedge open-source LLM model, to scrutinize the architecture of LLM as well as Transformer. As shown in Figure 2 (a), LLaMA consists of three major components: an embedding block, a stack of decoder blocks and a head block which consists of linear and softmax layer. The embedding layer's primary role is to transform unstructured textual information, into chunks of discrete numerical vectors (tokens) to facilitate subsequent processing. The embedded tokens are then delivered to the decoder layers for further processing. Each LLaMA decoder is composed of two fundamental components: Multihead Self-Attention (MSA) and Feedforward Network (FFN). In the MSA module, each of the tokens will be clustered by an attention map obtained by a dot production between two linear mappings of the input tokens. Then the grouped tokens will be further processed by a Feedforward Neural network.
为了更深入地了解LLM 和其他基于 Transformer 的模型，我们采用了最先进的开源LLM 模型 LLaMA-7B 来仔细研究LLM 和 Transformer 的架构。如图 2 (a)所示，LLaMA 由三个主要部分组成：嵌入块、解码器块堆栈以及由线性层和软最大层组成的头部块。嵌入层的主要作用是将非结构化文本信息转化为离散数字向量（标记）块，以便于后续处理。然后，嵌入的标记被传送到解码器层进行进一步处理。每个 LLaMA 解码器都由两个基本组件组成：多头自注意（MSA）和前馈网络（FFN）。在 MSA 模块中，每个词组都将根据输入词组的两个线性映射之间的点生成所获得的注意力图进行分组。然后，分组后的词组将由前馈神经网络进一步处理。
Additionally, Root Mean Square Layer Normalization (RMSNorm) [9] is adopted in LLaMA as a replacement for Layer Normalization to ensure efficient training.
此外，LLaMA 采用了均方根层归一化（RMSNorm）[9] 来替代层归一化，以确保高效训练。

LLM distinguishes itself from other deep neural network (DNN) models such as convolutional neural networks (CNN) in two significant ways. Firstly, LLM exhibits an inherent autoregressive nature, necessitating multiple iterations to complete the generation task. Moreover, LLM incorporates an attention mechanism, a component with computational complexity that scales quadratically with the length of the inputs. On the other hand, the inherent computation characteristic of LLM lies in the attention blocks inside each decoder layer. Figure 2 (c) depicts the high-level overview of the computation flow in the attention block.
LLM 与卷积神经网络（CNN）等其他深度神经网络（DNN）模型有两个显著的不同之处。首先， 表现出固有的自回归特性，需要多次迭代才能完成生成任务。此外， 还包含注意力机制，该机制的计算复杂度与输入长度成二次方关系。另一方面， 固有的计算特性在于每个解码器层内的注意力块。图 2（c）描述了注意力区块计算流程的高层概览。LLM LLM LLM

During the inference process, each decoder takes a 4dimensional tensor as the input tokens. The input tokens are first multiplied with three weight matrices , and , producing the output referred to as query and value . Given the MSA module's inability to recognize positional data and the inherent autoregressive nature of LLMs, the query and key will undergo a process using Rotary Positional Embedding [10] (RoPE, denoted as . Subsequently, the key and value will be combined with prior tokens.
在推理过程中，每个解码器将 4 维张量 作为输入标记。输入标记首先与三个权重矩阵 和 相乘，产生的输出称为查询 和值 。鉴于 MSA 模块无法识别位置数据以及LLMs 固有的自回归性质，查询和密钥将使用旋转位置嵌入法 [10]（RoPE，表示为 ）进行处理。 随后，密钥和值将与之前的标记相结合。

After the positional embedding, the intermediate activation will then undergo a series of multiplication, softmax, and residual addition to generate MSA output as described in Eq9 To be noted here, in the equation refers to the number of feature dimensions in the multi-head attention mechanism.
如公式 9 所述，在位置嵌入之后，中间激活将经过一系列乘法、软最大值和残差加法来生成 MSA 输出。需要注意的是，公式中的 指的是多头注意力机制中的特征维数。

The SA output will then be forwarded to the FFN blocks for further processing. The FFN block will have another three
然后，SA 输出将被转发到 FFN 模块进行进一步处理。FFN 块将有另外三个

Fig. 2: (a) LLaMA architecture. (b) LLaMA auto-regressive pattern. (c) Three common PEFT operations. All the learnable components are highlighted in red, while the frozen components are highlighted in grey. LoRA is applied on all the Query, Key, and Value blocks. The adapter targets the FFN module. Soft-Prompt focused on tuning the input activation of each decoder. We only show one decoder for illustration simplicity.
图 2： (a) LLaMA 架构。(b) LLaMA 自动回归模式。(c) 三种常见的 PEFT 操作。所有可学习的组件以红色标出，冻结的组件以灰色标出。LoRA 应用于所有查询、键和值块。适配器以 FFN 模块为目标。Soft-Prompt 专注于调整每个解码器的输入激活。为便于说明，我们只展示了一个解码器。

matrices , and and the computation can be illustrated by:
矩阵 ，以及 ，计算过程可通过以下方式说明：

where denotes the input of the FFN layer, and SiLU is the nonlinear function used in LLaMA. In the original Transformer, the FFN block can be demonstrated by:
其中 表示 FFN 层的输入，SiLU 是 LLaMA 中使用的非线性函数。在最初的变换器中，FFN 模块可以通过以下方式进行演示：

The output of the last decoder layer will be sent to a linear layer, which then generates a probability distribution spanning the complete vocabulary to predict the next token in the sequence. The produced token will then be concatenated with the previous tokens and used as the input for the next round of processing. This generating process repeats in an auto-regressive manner until a full sequence of tokens, referred to as a completion, is produced (Figure 2 (b)). For training, the computation flow is similar to that for inference, except that the generated sentences are directly compared to the ground truth output and generate the training loss. Gradients will then be computed across the LLM weights to minimize this training loss.
最后一个解码层的输出将被发送到线性层，然后线性层生成一个跨越完整词汇的概率分布，以预测序列中的下一个标记。然后，生成的标记将与之前的标记连接起来，作为下一轮处理的输入。这一生成过程以自动递归的方式重复进行，直到生成一个完整的标记序列（称为 "完成"）（图 2 (b)）。对于训练，计算流程与推理类似，只是生成的句子会直接与地面实况输出进行比较，并产生训练损失。然后将计算LLM 权重的梯度，以最小化训练损失。

To analyze the computation cost and memory overhead in LLM, we also set a series of parameters used in later section III Table I shows the parameter size and computation dimension in the LLaMA-7B model as a starting example.
为了分析LLM 中的计算成本和内存开销，我们还设置了后面第三节中使用的一系列参数 表 I 显示了 LLaMA-7B 模型中的参数大小和计算维度，并以此为起始例子。

LLM models generate tokens (words) one for each round, depicted in Fig 2, based on the previous prompt (input) and previously generated sequence. This process will be repeated until the model outputs hits and termination token. To accelerate the inference process in LLM models, people take the strategy of storing the previous Keys and Values in KeyValue cache (KV-cache), so they don't need to recalculate them for each new token. Mathematically, we can represent the total decoders' kv-cache memory cost in equation 6. In the equation, 1 and are the context length and batch size and refers to the number of layers. The is the head dimension and is the number of heads.
LLM 模型根据前一个提示（输入）和之前生成的序列，每轮生成一个标记（词），如图 2 所示。这个过程会一直重复，直到模型输出命中和终止标记。为了加快 模型的推理过程，人们采取的策略是将之前的键值和值存储在键值缓存（KV-cache）中，这样就不需要为每个新标记重新计算键值和值了。在数学上，我们可以用公式 6 来表示解码器的 KV 缓存内存总成本。式中，1 和 是上下文长度和批量大小， 指层数。 是磁头维度， 是磁头个数。LLM

Fine-tuning remains essential to enhance LLM performance on unseen user datasets and tasks. With the size of the model growing (e.g. 1.5B in GPT-2 to 175B in GPT-3), standard full fine-tuning paradigm requires thousands of GPU work in parallel, which is highly inefficient and unsustainable. A type of algorithm has been raised namely Parameter-efficient fine-tuning (PEFT) which aims to tune minimal parameters to achieve better performance over full tuning on downstream tasks.
微调对于提高LLM 在未知用户数据集和任务上的性能仍然至关重要。随着模型规模的不断扩大（例如，GPT-2 中的 1.5B 到 GPT-3 中的 175B），标准的完全微调范式需要数千个 GPU 并行工作，效率极低，难以为继。有人提出了一种算法，即参数高效微调（PEFT），其目的是调整最小参数，以获得比对下游任务进行全面调整更好的性能。

In parallel developments, large-scale pre-trained models in vision and multimodal domains have also demonstrated their effective representational learning capabilities, enabling adaptation from large datasets to smaller ones or across various data modalities through fine-tuning. Consequently, this capability has made PEFT increasingly attractive to the wider research community.
与此同时，视觉和多模态领域的大规模预训练模型也展示了其有效的表征学习能力，能够通过微调从大型数据集适应小型数据集或跨各种数据模态。因此，这种能力使 PEFT 对更广泛的研究界越来越有吸引力。

We categorized the PEFT algorithms into additive, selective, reparameterized, and hybrid fine-tuning based on their operations. As Figure 3 depicts, three major additive finetuning algorithms are normally used: (1) Adapter; (2) Soft Prompt; (3) Others. They differ from each other in terms of the different additional tunable modules or parameters. Selective fine-tuning, on the other hand, doesn't require any additional parameters, it selects a small subset of parameters from the backbone model and only makes them tunable while keeping the majority of parameters untouched during finetuning on downstream tasks. We categorized selective finetuning based on the grouping of chosen parameters: (1) Unstructural Masking; (2) Structural Masking. Reparametrization
我们根据 PEFT 算法的操作，将其分为加法微调、选择性微调、重参数微调和混合微调。如图 3 所示，通常使用三种主要的加法微调算法：(1) 适配器；(2) 软提示；(3) 其他。它们之间的区别在于可调整的附加模块或参数不同。另一方面，选择性微调不需要任何附加参数，它从骨干模型中选择一小部分参数，只对其进行微调，而在下游任务的微调过程中保持大部分参数不变。我们根据所选参数的分组对选择性微调进行了分类：（1）非结构性屏蔽；（2）结构性屏蔽。重新参数化

TABLE I: Configuration parameters and computation operation for LLaMA-7B architecture
表 I：LLaMA-7B 架构的配置参数和计算操作

represents transforming model parameters between two equivalent forms. Specifically, reparametrized fine-tuning introduces additional low-rank trainable parameters during training, which are then integrated with the original model for inference. This approach is categorized into two main strategies: (1) Lowrank Decomposition, and (2) LoRA Derivatives. Hybrid finetuning explores the design spaces of different PEFT methods and combines their advantages.
表示在两种等效形式之间转换模型参数。具体来说，重参数化微调在训练过程中引入额外的低秩可训练参数，然后与原始模型整合进行推理。这种方法分为两种主要策略：(1) 低阶分解和 (2) LoRA 衍生。混合微调探索了不同 PEFT 方法的设计空间，并结合了它们的优势。

Two types of tasks have been widely used for LLM evaluation, the first type is the General Language Understanding Evaluation (GLUE) [11] benchmark, which integrates nine sentence or sentence-pair language understanding tasks (CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, RTE, and WNLI), chosen for their diversity in dataset sizes, text genres, and difficulty levels, and is based on established existing datasets. It also includes a diagnostic dataset specifically designed to evaluate and analyze model performance across various linguistic phenomena inherent in natural language. Additionally, it features a public leaderboard to track performance on the benchmark and a dashboard to visualize model performance on the diagnostic set.
有两类任务被广泛用于LLM 评估，第一类是通用语言理解评估（GLUE）[11] 基准，它整合了九个句子或句子对语言理解任务（CoLA、SST-2、MRPC、STS-B、QQP、MNLI、QNLI、RTE 和 WNLI），这些任务因其数据集大小、文本流派和难度级别的多样性而被选中，并基于现有的数据集。它还包括一个诊断数据集，专门用于评估和分析模型在自然语言固有的各种语言现象中的表现。此外，它还提供了一个公共排行榜来跟踪基准性能，以及一个仪表盘来直观显示模型在诊断集上的性能。

The other type of dataset that has been used in recent LLM papers is common sense reasoning which integrated into our study caters to a variety of research facets: (1) OpenBookQA [12] is curated to foster research in advanced question-answering, delving into a profound understanding of both the subject matter and the language in which it is articulated. (2) PIQA [13] primarily emphasizes everyday scenarios, demonstrating a predilection for unconventional solutions. (3) Social IQA [14] emerges as a novel questionanswering benchmark tailored for gauging social commonsense intelligence. (4) HellaSwag [15] serves as a dataset, the essence of which is to ascertain the capability of machines in aptly concluding sentences. (5) BoolQ [16] is a dataset dedicated to question-answering, particularly for binary responses (yes/no queries). (6) WinoGrande [17] is introduced as a fresh compilation, encompassing a substantial 44,000 problems. (7) -easy [18] presents itself as a novel dataset constituting genuine grade-school level multiple-choice science questions, designed to invigorate research in intricate question-answering. (8) ARC-challenges [18], distinctively, encompasses solely those questions that were inaccurately addressed by both a retrieval-based algorithm and a word co-occurrence algorithm.
LLM 最近的论文中使用的另一种数据集是常识推理，它与我们的研究相结合，迎合了各种研究方面的需要：(1) OpenBookQA [12] 的目的是促进高级问题解答的研究，深入理解主题和表述主题的语言。(2) PIQA[13]主要强调日常情景，倾向于非常规的解决方案。(3) Social IQA[14]是为衡量社会常识智能而量身定制的一种新颖的答题基准。(4) HellaSwag[15]是一个数据集，其本质是确定机器在恰当地总结句子方面的能力。(5) BoolQ[16]是一个专门用于回答问题的数据集，尤其是二元回答（是/否查询）。(6) WinoGrande [17]是一个全新的汇编，包含大量 44,000 个问题。(7) -easy [18]是一个新颖的数据集，由真正的小学水平多选科学问题组成，旨在促进复杂问题解答的研究。(8) ARC-challenges [18]与众不同，它只包含那些用基于检索的算法和词语共现算法都无法准确解答的问题。

Image recognition is the primary benchmark and application for vision models, exemplified by benchmarks such as fine- grained visual categorization (FGVC) and visual task adaptation benchmark (VTAB). Beyond image classification, video action recognition is another key application area, involving datasets like Kinetics-400 [19], SSv2 [20], and HMDB51 [21]. Additionally, PEFT has been utilized for dense prediction tasks, using datasets like MSCOCO [22], ADE20K [23], and PASCAL VOC [24].
图像识别是视觉模型的主要基准和应用，例如细粒度视觉分类（FGVC）和视觉任务适应基准（VTAB）。除了图像分类，视频动作识别是另一个关键应用领域，涉及的数据集有 Kinetics-400 [19]、SSv2 [20] 和 HMDB51 [21]。此外，PEFT 还被用于密集预测任务，使用的数据集包括 MSCOCO [22]、ADE20K [23] 和 PASCAL VOC [24]。

The PEFT strategies can be broadly classified into four categories: additive PEFT (Section III-A), which modifies the model architecture by injecting new trainable modules or parameters; selective PEFT (Section III-B), which makes a subset of parameters trainable during fine-tuning; reparameterized PEFT (Section III-C), which constructs a (lowdimensional) reparameterization of the original model parameters for training, then equivalently transforms it back for inference; and hybrid PEFT (Section III-D), which combines advantages from different PEFT methods to build a unified PEFT model. A overview of different types of PEFT algorithms is depicted in Figure 4.
PEFT 策略可大致分为四类：添加式 PEFT（III-A 节），通过注入新的可训练模块或参数来修改模型结构；选择式 PEFT（III-B 节），在微调过程中使参数子集成为可训练参数；重参数化 PEFT（III-C 节），对原始模型参数进行（低维）重参数化以进行训练，然后等价转换回来进行推理；混合式 PEFT（III-D 节），结合不同 PEFT 方法的优点以建立统一的 PEFT 模型。不同类型的 PEFT 算法概览见图 4。

Standard full fine-tuning entails substantial computational expenses and also could potentially harm the model's generalization ability. To mitigate this problem, a widely employed approach is to maintain the pre-trained backbone unchanged and introduce only a minimal number of trainable parameters that are strategically positioned within the model architecture. While fine-tuning for a specific downstream task, only the weights of these additional modules or parameters are updated, which results in a substantial reduction in storage, memory, and computational resource requirements. Due to their characteristic of adding parameters, these techniques can be termed as Additive Tuning, as shown in Figure 4 (a). Next, we discuss several popular Additive PEFT algorithms.
标准的全面微调需要大量的计算费用，还可能损害模型的泛化能力。为了缓解这一问题，一种被广泛采用的方法是保持预先训练的骨干模块不变，只引入极少量的可训练参数，这些参数在模型架构中处于战略位置。在针对特定下游任务进行微调时，只更新这些附加模块或参数的权重，从而大幅减少对存储、内存和计算资源的需求。如图 4 (a)所示，由于这些技术具有增加参数的特点，因此可将其称为 "加法调整"。接下来，我们将讨论几种流行的加法 PEFT 算法。

Fig. 3: Taxonomy of Parameter-Efficient Fine-Tuning Methods for Large Models.
图 3：大型模型的参数效率微调方法分类学。

(a) Additive PEFT (a) 加法 PEFT

(c) Reparameterization PEFT
(c) 重参数化 PEFT

Fig. 4: Different types of PEFT algorithms.
图 4：不同类型的 PEFT 算法。

The concept of adapters in the field of NLP was initially introduced by Serial Adapter [25] as shown in Figure 5 (a). In their approach, each Transformer block is enhanced by adding two adapter modules, with one positioned after the self-attention layer and the other after the FFN layer, respectively. Subsequent research has aimed to address the additional computational cost associated with adapter layers. A modified framework AdapterFusion [29] was proposed, where adapter layers are inserted only after the 'Add & Norm' step following the FFN layer to enhance the computational efficiency. The adapters mentioned above follow a sequential design, placing adapter layers as bottlenecks within the Transformer blocks. This approach may potentially reduce the model's parallelism and require a trade-off between inference efficiency and accuracy. In contrast, [26] introduced a parallel adapter (PA) approach as depicted in Figure 5 (b), which reorganizes the traditionally sequential adapter layers into a parallel side-network that runs alongside each Transformer sublayer. Similarly, CIAT [27], CoDA [28] and KronA [72] also adopts a parallel adapter design. Except for the parallel design, CoDA employs a sparse activation mechanism to improve the inference efficiency as shown in Figure 5 (c).
如图 5（a）所示，NLP 领域的适配器概念最初由 Serial Adapter [25] 提出。在他们的方法中，通过添加两个适配器模块来增强每个 Transformer 模块，其中一个模块位于自我注意层之后，另一个模块位于 FFN 层之后。随后的研究旨在解决与适配器层相关的额外计算成本问题。有人提出了一个改进的框架 AdapterFusion [29]，即在 FFN 层之后的 "添加与规范 "步骤之后才插入适配器层，以提高计算效率。上述适配器采用顺序设计，将适配器层作为瓶颈置于变换器块中。这种方法可能会降低模型的并行性，需要在推理效率和准确性之间做出权衡。与此相反，[26] 引入了并行适配器 (PA) 方法，如图 5 (b) 所示，该方法将传统的顺序适配器层重组为并行侧网络，与每个变换器子层并行运行。同样，CIAT [27]、CoDA [28] 和 KronA [72] 也采用了并行适配器设计。除了并行设计，CoDA 还采用了稀疏激活机制来提高推理效率，如图 5（c）所示。
Specifically, CoDA use a soft top- selection process that identifies important tokens in each layer, which will be processed by both the frozen pre-trained Transformer layer and the adapter branch to maintain model accuracy. In contrast, those unimportant tokens are only processed by the adapter branch while skip the heavy pre-trained layer, therefore optimizing for inference efficiency without compromising overall performance.
具体来说，CoDA 使用软顶部 选择过程，在每一层中识别出 重要的标记，这些标记将由冻结的预训练转换器层和适配器分支处理，以保持模型的准确性。相反，那些不重要的词组只由适配器分支处理，而跳过重度预训练层，从而在不影响整体性能的情况下优化推理效率。

To enhance the performance and generalization of adapters, various studies have implemented multi-task learning strategies, such as AdapterFusion [29], AdaMix [30], PHA [31], AdapterSoup [32], MerA [33], and Hyperformer [34]. AdapterFusion keeps all pre-trained adapters in the model and employs a fusion module to merge the multi-task informations. Unlike AdapterFusion, MerA merges pretrained adapters into a single one through optimal transport based on weights and activations. This approach avoids introducing any additional trainable parameters, thereby enhancing computational efficiency. Hyperformer stores the multi-task information in a shared hypernetwork, which generates task and layer-specific adapter parameters conditioned on task and layer id embeddings. Given a new task, only an additional task embedding needs to be learned, therefore reducing the number of trained parameters.
为了提高适配器的性能和通用性，许多研究都采用了多任务学习策略，如 AdapterFusion [29]、AdaMix [30]、PHA [31]、AdapterSoup [32]、MerA [33] 和 Hyperformer [34]。AdapterFusion 在模型中保留了所有预先训练好的适配器，并采用融合模块来合并多任务信息。与 AdapterFusion 不同的是，MerA 通过基于权重和激活度的优化传输，将预先训练好的适配器合并成一个。这种方法避免了引入任何额外的可训练参数，从而提高了计算效率。Hyperformer 将多任务信息存储在共享超网络中，并根据任务和层 ID 嵌入生成特定于任务和层的适配器参数。对于新任务，只需学习额外的任务嵌入，从而减少了训练参数的数量。

(a) Serial Adapter (a) 串行适配器

(b) Parallel Adapter (b) 并行适配器

(d) Adapter Layer (d) 适配器层

Fig. 5: Illustration of three representative adapter-based fine-tuning algorithms. Blue represents frozen, while yellow represents trainable.
图 5：三种具有代表性的基于适配器的微调算法示意图。蓝色代表冻结，黄色代表可训练。

where is the sequence of input tokens for layer , including soft prompt tokens followed by the original input tokens is the number of soft prompt tokens, and is the number of original input tokens.
其中 是 层的输入令牌序列，包括软提示令牌 和原始输入令牌 是软提示令牌的数量， 是原始输入令牌的数量。

Prefix-tuning [35] introduces learnable vectors that are prepended to keys and values across all Transformer layers. To ensure stability during the optimization process, Prefixtuning adopts a reparameterization strategy, which utilizes an MLP layer to generate these prefix vectors rather than optimizing them directly. After fine-tuning, only the prefix vectors are saved for inference. This technique is adapted and improved in several studies [36], [37], [38]. For instance, ptuning v2 [37] removes reparameterization and expands its usage to broader model scales and NLP tasks. APT (Adaptive Prefix Tuning) [38] enhances Prefix-tuning by introducing adaptive gate mechanism to control the prefix importance in each layer. Concurrent work p-tuning [39] and prompttuning [40] apply learnable vectors only at the initial word embedding layer rather than all layers to enhance training and inference efficiency. It's important to highlight that prompttuning demonstrates its effectiveness primarily in the context of large models, specifically those with over 11 billion parameters [40]. Complementing this, Xprompt [41] eliminates the negative prompt tokens through a hierarchical structured pruning, which closes the performance gap at smaller model scales. The work by [97] provides some theoretical analysis towards prompt tuning, demonstrating its universality and limitations in limited-depth Transformers. IDPG (InstanceDependent Prompt Generation) [42] improves prompt tuning by generating prompts based on each input sentence with a lightweight prompt generator. In a related approach, LPT (Late Prompt Tuning) [43] also leverages a prompt generator to obtain instance-aware prompt. Unlike previous work, LPT adds these prompts only after an intermediate layer, rather than at the initial or all layers. This strategic placement eliminates the gradient calculation below the intermediate layer, thereby significantly accelerating the training speed. Simultaneously, LPT can improve the overall performance due to the shorter backpropagation path preserves more task-related information. Inspired by LPT, SPT (Selective Prompt Tuning) [44] delves deeper into the importance of prompt inserting strategies. It introduces a learnable probabilistic gate in each layer to determine whether to use the prompt propagated from the previous layer or inject a newly generated prompt. APrompt [45] employs another prompt inserting strategy. In addition to input prompts inserted at the beginning of the input sequence for each Transformer layer, APrompt also prepends additional learnable prompts to the respective query, key, and value matrices in the self-attention blocks to learn new attention patterns. Besides, APrompt incorporates the learning of a taskspecific head.
前缀调整[35]引入了可学习的向量，这些向量在所有变换器层的键 和值 前缀。为了确保优化过程中的稳定性，前缀调整采用了重新参数化策略，利用 MLP 层生成这些前缀向量，而不是直接对其进行优化。微调后，仅保存前缀向量用于推理。一些研究对这一技术进行了调整和改进 [36]、[37]、[38]。例如，ptuning v2 [37] 取消了重新参数化，并将其应用扩展到更广泛的模型规模和 NLP 任务。APT（自适应前缀调整）[38] 通过引入自适应门机制来控制每一层的前缀重要性，从而增强了前缀调整功能。同时进行的 p-tuning [39] 和 prompttuning [40] 只在初始词嵌入层而不是所有层应用可学习向量，以提高训练和推理效率。需要强调的是，prompttuning 主要是在大型模型，特别是参数超过 110 亿的模型中显示出其有效性[40]。作为补充，Xprompt [41] 通过分层结构化剪枝消除了负面提示标记，从而缩小了较小模型规模下的性能差距。文献[97]对提示调整进行了一些理论分析，证明了其在有限深度变换器中的普遍性和局限性。IDPG（InstanceDependent Prompt Generation）[42] 通过使用轻量级提示生成器根据每个输入句子生成提示来改进提示调整。在一种相关的方法中，LPT（晚期提示调整）[43] 也利用提示生成器来获得实例感知提示。与之前的研究不同，LPT 仅在中间层后添加这些提示，而不是在初始层或所有层。这种策略性的安排省去了中间层以下的梯度计算，从而大大加快了训练速度。同时，由于反向传播路径更短，保留了更多与任务相关的信息，LPT 可以提高整体性能。受 LPT 的启发，SPT（选择性提示调整）[44] 深入探讨了提示插入策略的重要性。它在每一层都引入了一个可学习的概率门，以决定是使用上一层传播的提示，还是注入一个新生成的提示。APrompt [45] 采用了另一种提示插入策略。除了在每个转换器层的输入序列开头插入输入提示外，APrompt 还会在自我注意区块中的相应查询、键和值矩阵中预置额外的可学习提示，以学习新的注意模式。此外，APrompt 还加入了任务特定头的学习。

The concept of soft prompts has been employed for various downstream tasks [98], [99], although their training can be prone to instability and slow convergence. To address this, SPoT [46] uses a source prompt learned from one or multiple tasks to initialize prompts for new tasks. Similarly, transfer of soft prompts from one task to initialize another is proposed in TPT (transferable prompt tuning) [47], which demonstrates that a better prompt initialization results in a large training convergence speedup. InfoPrompt [48] develops two mutual information based loss functions, i.e., head loss and representation loss, to find better prompt initialization and learn sufficient task-relevant information, thereby also expediting convergence. PTP [49] delves into the root causes of training instability. It identifies the steep nature of the loss landscape in conventional prompt tuning, where minor variations in input data can lead to significant loss fluctuations. To mitigate this, PTP introduces perturbation-based regularizers to smooth the loss landscape and consequently stabilize the training process. DePT [52] decomposes the soft prompt into a shorter soft prompt with a pair of low-rank matrices, which are optimised with two distinct learning rates. This strategy not only improves the performance but also enhances training and inference efficiency. SMoP (Sparse Mixture-of-Prompts) [51] reduce the training and inference cost by utilizing short soft prompts. During training, multiple short soft prompts are trained, each tailored to specific subsets of the dataset. During inference, SMoP integrates a gating mechanism that routes each input instance to an appropriate short prompt. This technique not only increases efficiency in both training and inference stages but also retains performance comparable to those achieved with longer soft prompts. To further cut down the number of soft prompt parameters, IPT (Intrinsic Prompt Tuning) [50] identifies an intrinsic task subspace by training an auto-encoder on multiple tasks. Tuning on new tasks then requires adjusting only a few parameters within this subspace, significantly reducing the number of training parameters.
软提示的概念已被用于各种下游任务 [98]、[99]，但其训练可能容易出现不稳定和收敛缓慢的问题。为了解决这个问题，SPoT [46] 使用从一个或多个任务中学到的源提示来初始化新任务的提示。同样，TPT（可转移提示调整）[47] 中也提出了从一个任务中转移软提示来初始化另一个任务的方法，该方法证明了更好的提示初始化会带来很大的训练收敛速度提升。InfoPrompt [48] 开发了两个基于互信息的损失函数，即头部损失和表征损失，以找到更好的提示初始化并学习足够的任务相关信息，从而也加快了收敛速度。PTP [49] 深入研究了训练不稳定的根本原因。它发现了传统即时调整中损失景观的陡峭性质，在这种情况下，输入数据的微小变化就会导致显著的损失波动。为了缓解这一问题，PTP 引入了基于扰动的正则器来平滑损失景观，从而稳定训练过程。DePT [52] 将软提示分解为带有一对低秩矩阵的更短软提示，并以两种不同的学习率对其进行优化。这种策略不仅能提高性能，还能提高训练和推理效率。SMoP（Sparse Mixture-of-Prompts）[51] 利用短软提示降低了训练和推理成本。在训练过程中，会对多个短软提示进行训练，每个提示都是针对数据集的特定子集量身定制的。在推理过程中，SMoP 集成了一种门控机制，可将每个输入实例路由到适当的短提示。这种技术不仅提高了训练和推理阶段的效率，还能保持与使用较长软提示符时相当的性能。为了进一步减少软提示参数的数量，IPT（内在提示调整）[50] 通过在多个任务上训练自动编码器来识别内在任务子空间。在新任务上进行调整时，只需在该子空间内调整几个参数，从而大大减少了训练参数的数量。

(a)  (a)

(b) SSF
Fig. 6: Illustration of (IA) and SSF. Blue represents frozen, while yellow represents trainable.
图 6：(IA) 和 SSF 的图示。蓝色代表冻结，黄色代表可训练。

rescaling vectors: , and , to rescale the key, value, and FFN activations, respectively, as depicted in Figure 6 (a). The operations within the self attention block can be described as follows:
重定向向量： 和 ，分别对键、值和 FFN 激活进行重新缩放，如图 6 (a) 所示。自我关注区块内的操作可描述如下：

In FFN, the rescaling can be denoted as:
在 FFN 中，重定标可表示为

where is Hadamard product. Furthermore, the scale vectors and can be seamlessly integrated into the weight matrices of and . This integration effectively eliminates the extra computational costs during inference. A similar technique SSF [55] also performs linear transformation to the model activations, as illustrated in Figure 6 (b). Specifically, after each operation (i.e., MSA, FFN and layer normalization) in the pre-trained model, a SSF-ADA layer is injected, which performs scaling and shifting to the features generated from the operation. During fine-tuning, only those SSF-ADA layers can be updated, while during inference, similar to (IA) , these SSF-ADA layers can be merged into model weights, so no additional inference overhead would be incurred. IPA (InferenceTime Policy Adapters) [56] offers a novel approach to align LLMs, such as GPT-4, with user-specific requirements without modifying the base model's parameters. This is particularly significant when dealing with models whose parameters are extremely large and often not directly accessible. IPA achieves this by combining (through multiplication and normalization) the output distribution of a base LLM (base policy) with that of a smaller-sized model (adapter policy) during the decoding phase. During training, the policy adapter's parameters are fine-tuned using reinforcement learning, while the base policy's parameters remain fixed. During inference, IPA decodes with the combined distribution of the base model and the trained policy adapter, tailoring it to fulfill specific user-defined criteria.
其中 是哈达玛乘积。此外，尺度向量 和 可以无缝集成到 和 的权重矩阵中。这种整合有效地消除了推理过程中的额外计算成本。类似的技术 SSF [55] 也对模型激活进行线性变换，如图 6 (b) 所示。具体来说，在预训练模型的每个操作（即 MSA、FFN 和层归一化）之后，都会注入一个 SSF-ADA 层，该层会对操作生成的特征进行缩放和移位。在微调过程中，只能更新这些 SSF-ADA 层，而在推理过程中，类似于（IA） ，这些 SSF-ADA 层可以合并到模型权重中，因此不会产生额外的推理开销。IPA（推理时间策略适配器）[56] 提供了一种新颖的方法，在不修改基础模型参数的情况下，使LLMs （如 GPT-4）与用户的特定要求保持一致。这在处理参数极其庞大且通常无法直接访问的模型时尤为重要。IPA 通过在解码阶段将基础LLM （基础策略）的输出分布与较小模型（适配器策略）的输出分布相结合（通过乘法和归一化）来实现这一点。在训练过程中，策略适配器的参数通过强化学习进行微调，而基础策略的参数则保持不变。在推理过程中，IPA 利用基础模型和训练有素的策略适配器的组合分布进行解码，使其符合用户定义的特定标准。

Rather than additive PEFT, which increases the model complexity by adding more parameters, selective PEFT finetunes a subset of the existing parameters to enhance model performance over downstream tasks, as depicted in Figure 4 (b).
与通过增加参数来提高模型复杂度的加法 PEFT 相比，选择性 PEFT 是对现有参数的一个子集进行微调，以提高模型在下游任务中的性能，如图 4 (b) 所示。

Fig. 7: Illustration of how two parameter masking methods. Blue represents frozen, while yellow represents trainable.
图 7：两种参数屏蔽方法的示意图。蓝色代表冻结，黄色代表可训练。

Specifically, given a model with parameters where each denotes an individual model parameter and represents the total count of these parameters, the process of selective PEFT is represented by applying a binary mask to these parameters. Each in is either 0 or 1 , indicating whether the corresponding parameter is selected (1) or not selected (0) for fine-tuning. The updated parameter set after fine-tuning is given by:
具体来说，给定一个带有参数 的模型，其中每个 表示一个单独的模型参数， 表示这些参数的总数，选择性 PEFT 的过程是通过对这些参数应用二进制掩码 来表示的。 中的每个 都是 0 或 1，表示微调时选择（1）或不选择（0）相应的参数 。微调后的更新参数集 由以下公式给出：

where represents the learning rate, and is the gradient of the loss function with respect to the parameter . In this formulation, only the parameters that are selected (i.e., are updated during backpropagation.
其中 代表学习率， 是损失函数相对于参数 的梯度。在这一公式中，反向传播过程中只更新选定的参数（即 ）。

Diff pruning [57] is a representative work that applies a learnable binary mask to the model weights during finetuning. To achieve parameter efficiency, the mask is regularized by a differentiable approximation of the -norm penalty. PaFi [59] simply select model parameters with the smallest absolute magnitude as trainable. FishMask [60] determines parameter importance using the approximate Fisher information. It then selects the top parameters based on this information to form the mask M. Similarly, Fish-Dip [61] also uses Fisher information to calculate , but the mask will be re-calculated dynamically in each train period. LT-SFT [62] introduces another technique to determines parameter importance inspired by the Lottery Ticket Hypothesis [100], [101], where the the subset of parameters that change the most during an initial fine-tuning stage is selected to form the mask . SAM [63] proposes a second-order approximation method, which approximates the original problem with an analytically solvable optimization function, to help decide the parameter mask. Child-tuning [64] proposes two approaches to select a child network during each training iteration, where only the parameters within this child network can be updated.
Diff pruning [57] 是一项具有代表性的工作，它在微调过程中对模型权重应用了可学习的二进制掩码。为实现参数效率，掩码通过 -norm 惩罚的可微分近似值正则化。PaFi [59] 简单地选择绝对值最小的模型参数作为可训练参数。FishMask [60] 利用近似费雪信息确定参数的重要性。 同样，Fish-Dip [61] 也使用 Fisher 信息来计算 ，但掩码将在每个训练期动态地重新计算。LT-SFT [62] 受彩票假设 [100], [101] 的启发，引入了另一种确定参数重要性的技术，即选择在初始微调阶段变化最大的参数子集来形成掩码 。SAM [63] 提出了一种二阶近似法，用一个可分析求解的优化函数来近似原始问题，以帮助决定参数掩码。子调整法 [64] 提出了两种方法，在每次训练迭代中选择一个子网络，只有该子网络中的参数可以更新。

However, above unstructured parameter masking results in an uneven distribution of non-zero masks and diminished hardware efficiency when implementing PEFT. As shown in Figure 7, the structured mask organize parameter masking in regular patterns, unlike unstructured ones that apply it randomly, thus can enhances computational and hardware efficiency during training. Therefore, various structured selective PEFT techniques have undergone extensive investigation. Diff pruning proposes a structured pruning strategy by partitioning the weight parameters into local groups and strategically eliminating them together. Similarly, FAR [65]
然而，上述非结构化参数掩码会导致非零掩码分布不均，降低实现 PEFT 时的硬件效率。如图 7 所示，结构化掩码以规则模式组织参数掩码，不像非结构化掩码那样随机应用，因此可以提高训练过程中的计算和硬件效率。因此，各种结构化选择性 PEFT 技术得到了广泛的研究。Diff pruning 提出了一种结构化剪枝策略，它将权重参数划分为局部组，并有策略地将它们一起消除。同样，FAR [65］

(a) LoRA (a) 土地註冊處

(b) DyLoRA

(c) DoRA (c) 土改局

Fig. 8: Illustration of three representative reparameterized PEFT algorithms. Blue represents frozen, while yellow represents trainable.
图 8：三种具有代表性的重新参数化 PEFT 算法示意图。蓝色代表冻结，黄色代表可训练。

fine-tunes BERT models by grouping weights of the FFN in Transformer blocks into nodes, then rank and select the learner nodes using norm. To further reduce the memory access frequency, they also reconfigure the FFN by grouping the learner nodes together. Bitfit [66] is proposed to only finetunes the bias parameters of each DNN layer, and achieves competitive results for small models. However, this method fails to handle large models. The work by [58] applies NAS to Bitfit, where S-BitFit keeps the structural nature in Bitfit that restrict NAS algorithm must choose whether or not for each bias module. Similar to Bitfit that fine-tunes a specific module in Transformer, Xattn Tuning [67] finetunes only the cross-attention layers. SPT (sensitivity-aware visual parameter-efficient fine-tuning) [68] first identifies the sensitive parameters measured by the loss reduction when being tuned. This sensitivity is calculated using a first-order Taylor expansion, derived from a single forward and backward pass before fine-tuning in oneshot. Next, SPT finds the weight matrices whose number of sensitive parameters exceeds a predefined threshold, and then applies a selected PEFT technique (e.g., LoRA and Adapter) to these targeted weights to achieve structural tuning.
微调 BERT 模型的方法是将变换器块中的 FFN 权重分组为节点，然后使用 规范对学习节点进行排序和选择。为了进一步降低内存访问频率，他们还通过将学习节点分组来重新配置 FFN。Bitfit [66] 建议只对每个 DNN 层的偏置参数进行微调，并对小型模型取得了有竞争力的结果。但是，这种方法无法处理大型模型。文献[58]将 NAS 应用于 Bitfit，其中 S-BitFit 保留了 Bitfit 中的结构特性，即限制 NAS 算法必须为每个偏置模块选择是否 。与 Bitfit 微调 Transformer 中的特定模块类似，Xattn Tuning [67] 只微调交叉注意层。SPT（灵敏度感知可视参数高效微调）[68] 首先要确定微调时损耗降低所测量的敏感参数。这种灵敏度是通过一阶泰勒扩展计算得出的，而泰勒扩展是在单次微调之前通过一次前向和后向传递得出的。接下来，SPT 会找出敏感参数数量超过预定阈值的权重矩阵，然后对这些目标权重应用选定的 PEFT 技术（如 LoRA 和 Adapter），以实现结构调整。

Reparameterization stands for equivalently transforming a model's architecture from one to another via transforming its parameters. In the context of PEFT, this often means constructing a low-rank parameterization to achieve the goal of parameter efficiency during training. For inference, the model can be converted to its original weight parameterization, ensuring unchanged inference speed. This procedure is depicted in Figure 4 (c).
重新参数化（Reparameterization）是指通过转换参数，将模型的结构从一个模型转换为另一个模型。就 PEFT 而言，这通常意味着在训练过程中构建一个低秩参数化，以实现参数效率的目标。在推理时，可将模型转换为原始权重参数化，确保推理速度不变。这一过程如图 4 (c) 所示。

Earlier research studies [69] have shown that common pre-trained models exhibit an exceptionally low intrinsic dimensionality. In other words, it is possible to find a lowdimensional reparameterization that is effective for fine-tuning as the entire parameter space. Intrinsic SAID [69] is the pioneering work in investigating the intrinsic dimension feature during the fine-tuning of LLMs. However, the most widely recognized reparameterization technique is LoRA (Low Rank Adaptation) [70], [102], as shown in Figure 8] (a). For a given pre-trained weight matrix , LoRA introduces two trainable weight matrices, and where the rank , operating in parallel to . Let represent the input. Under normal conditions, the output through is . Instead, LoRA modifies this output by introducing an incremental update that encapsulates task-specific knowledge:
早期的研究[69]表明，普通的预训练模型表现出极低的内在维度。换句话说，我们有可能找到一种低维的重参数化方法，它能有效地对整个参数空间进行微调。本征 SAID [69] 是研究LLMs 微调过程中的本征维度特征的开创性工作。不过，最广泛认可的重参数化技术是 LoRA（低秩自适应）[70], [102]，如图 8 所示。］(a).对于给定的预训练权重矩阵 ，LoRA 引入了两个可训练的权重矩阵 和 ，其中 的秩与 平行运行。让 代表输入。在正常情况下，通过 的输出是 。相反，LoRA 通过引入增量更新 来修改输出，增量更新囊括了特定任务的知识：

where denotes a scaling factor. At the onset of training, is initialized using a random Gaussian distribution, while is initialized to zero, ensuring that initially holds a value of zero. LoRA is straightforward to implement and has been evaluated on models with up to 175 billion parameters. Fig 8 (c) used a single decoder as an example, the frozen and learnable components are highlighted in grey and red, respectively. Once fine-tuning is complete, LoRA's adaptive weights seamlessly integrate with the pre-trained backbone weights. This integration ensures that LoRA maintains the model's efficiency, adding no extra burden during inference.
其中 表示缩放因子。训练开始时， 使用随机高斯分布初始化，而 则初始化为零，确保 初始值为零。LoRA 的实现非常简单，已在多达 1 750 亿个参数的模型上进行了评估。图 8 (c) 以单个解码器为例，冻结部分和可学习部分分别以灰色和红色标出。微调完成后，LoRA 的自适应权重将与预训练的骨干权重无缝集成。这种整合可确保 LoRA 保持模型的效率，在推理过程中不会增加额外负担。

In LoRA training, selecting an appropriate rank has always been a challenging issue. To address this, DyLoRA [76], as depicted in Figure 8 (b), trains LoRA module on a range of ranks within a predefined training budget, rather than adhering to a single, fixed rank. Specifically, for a given rank range , DyLoRA dynamically chooses a rank at each iteration of the training process. Consequently, the matrices and are tailored for the selected rank , resulting in truncated versions and , and the subsequent forward and backward pass during this iteration will be restricted on and instead of and . With this dynamic and search-free approach, DyLoRA significantly reduces the training time required to find an optimal and fixed LoRA rank for specific tasks. AdaLoRA [77] reformulates the with a singular value decomposition (SVD), denoted as , where and are orthometric, is a diagonal matrix containing sigular values . All the three weight matrices are made learnable. During training, the singular values are pruned iteratively based on their importance scores, which are constructed from moving average of the magnitude of gradient-weight product.
在 LoRA 训练中，选择合适的等级一直是一个具有挑战性的问题。为了解决这个问题，如图 8 (b) 所示，DyLoRA [76]在预定义的训练预算范围内对一系列等级进行 LoRA 模块训练，而不是遵循单一、固定的等级。具体来说，对于给定的秩范围 ，DyLoRA 会在训练过程的每次迭代中动态选择一个秩 。因此，矩阵 和 将根据选定的等级 进行调整，从而产生截断版本的 和 ，在此迭代过程中的后续前向和后向传递将受限于 和 ，而不是 和 。通过这种动态的免搜索方法，DyLoRA 大大减少了为特定任务寻找最佳和固定 LoRA 秩所需的训练时间。AdaLoRA [77] 通过奇异值分解（SVD）重新计算 ，表示为 ，其中 和 为正交矩阵， 为包含奇异值的对角矩阵 。所有三个权重矩阵都是可学习的。在训练过程中，奇异值会根据其重要性得分进行迭代修剪，重要性得分由梯度-权重乘积大小的移动平均值计算得出。

To ensure the orthogonality between and , i.e., , an additional regularizer term is included in the loss:
为确保 和 （即 ）之间的正交性，损耗中加入了额外的正则项：

This adaptive approach enables the model to dynamically adjust the rank within each LoRA module, effectively managing its parameter counts based on the significance of the weight matrices. However, according to SoRA [78], the importance scores used in AdaLoRA is heuristically constructed, which lacks rigorous theoretical motivation. Additionally, both moving average operation and calculation of Eq. 13 introduces extra computation cost during training. To address this, SoRA eliminates the orthogonality premise of and . Instead, a gating unit between and is directly applied and optimized:
这种自适应方法使模型能够动态调整每个 LoRA 模块内的等级，根据权重矩阵的重要性有效管理其参数计数。不过，根据 SoRA [78]，AdaLoRA 中使用的重要性分数是启发式构建的，缺乏严格的理论依据。此外，移动平均操作和公式 13 的计算都会在训练过程中引入额外的计算成本。为了解决这个问题，SoRA 取消了 和 的正交性前提。取而代之的是直接应用和优化 和 之间的门控单元 ：

where is Hadamard product. The gate is updated using a variation of proximal gradient iteration for loss [103], [104], which has a clear mathematical meaning and do not need the heuristic premise. After training, the zeroed-out gate units are pruned by removing the corresponding columns and rows in and .
其中 是哈达玛乘积。门 采用近似梯度迭代的一种变体来更新 loss [103], [104]，这种方法有明确的数学含义，不需要启发式前提。训练结束后，通过删除 和 中的相应列和行，对归零的门单元进行剪枝。

Several subsequent studies have aimed to improve LoRA's performance in various aspects. For instance, LaplaceLoRA [81] notices that fine-tuned LLMs often exhibit overconfidence. To enhance the calibration of fine-tuned LLMs, Laplace-LoRA utilizes a Bayesian approach, specifically a post-hoc Laplace approximation [105], [106], to the posterior over the LoRA parameters. LoRA Dropout [82] introduces random noises to the learnable low-rank matrices and increases parameter sparsity to reduce the risk of overfitting. LoRA+ [84] proposes to set different learning rates for the LoRA matrices and , such that with fixed and tune .
随后的一些研究旨在提高 LoRA 的各方面性能。例如，LaplaceLoRA [81]注意到微调后的LLMs 经常表现出过度自信。为了加强微调后的LLMs 的校准，Laplace-LoRA 采用了贝叶斯方法，特别是对 LoRA 参数的后验进行事后拉普拉斯近似[105]、[106]。LoRA Dropout [82]在可学习的低阶矩阵中引入随机噪音，增加参数稀疏性，以降低过拟合风险。LoRA+ [84] 建议为 LoRA 矩阵 和 设置不同的学习率，如 与 固定，并调整 。

Thanks to the modular design of LoRA, many studies incorporate multiple LoRA modules in their frameworks to enhance performance. For example, LoRAHub aggregates various LoRA modules trained on different tasks. Given a handful of examples from a new task, LoRAHub can autonomously compose compatible LoRA modules without human intervention via a gradient-free method Shiwa [107]. MOELoRA employs a Mixture-of-Experts (MOE) approach to train LoRA in a multi-task setting, resulting in multiple expert LoRA modules. To retrieve parameters for certain task, MOELoRA utilizes a task-motivated gate function that assigns contribution weights to each expert based on the task ID, and the final parameters is calculated through a weighted sum of all experts.
得益于 LoRA 的模块化设计，许多研究将多个 LoRA 模块纳入其框架，以提高性能。例如，LoRAHub 聚合了针对不同任务训练的各种 LoRA 模块。给定新任务中的少量示例后，LoRAHub 可通过无梯度方法 Shiwa [107]，在无需人工干预的情况下自主组成兼容的 LoRA 模块。MOELoRA 采用专家混合（MOE）方法在多任务环境中训练 LoRA，从而产生多个 LoRA 专家模块。为了检索特定任务的参数，MOELoRA 利用任务驱动门函数，根据任务 ID 为每位专家分配贡献权重，并通过所有专家的加权和计算最终参数。

In addition to LoRA, several other reparameterization techniques are emerging with significant potential. For instance, Compacter [71] introduces a light-weight adapter modules by parameterizing the and as , where , and denotes the Kronecker product. They further decrease the parameter count by designating as shared parameters and reparameterizing using the product of two low-rank matrices, effectively reducing the parameter complexity from to . Related studies, such as KronA [72] and KAdaptation [73], also employ the Kronecker product to reparameterize adapter weights, aiming to achieve parameter reduction. HiWi [59] proposes an adapter fine-tuning method that applies an adapter directly to pretrained parameters instead of hidden representations as:
除 LoRA 外，其他几种重新参数化技术也在不断涌现，并具有巨大潜力。例如，Compacter [71] 通过将 和 参数化为 ，其中 和 表示 Kronecker 乘积，引入了轻量级适配器模块。他们将 指定为共享参数，并使用两个低阶矩阵的乘积对 进行重新参数化，从而进一步减少了参数数量，有效地将参数复杂度从 降至 。相关研究，如 KronA [72] 和 KAdaptation [73]，也采用克朗内克乘法对适配器权重进行重新参数化，以达到减少参数的目的。HiWi [59] 提出了一种适配器微调方法，它将适配器直接应用于预训练参数，而不是隐藏表示：

where denotes the weights or biases within the Transformer block's feed-forward layer. Notably, during inference, this method computes in advance, ensuring that the model's inference latency remains on par with that of traditional full fine-tuning. VeRA (Vector-based Random Matrix Adaptation) [74] employs a single pair of frozen low-rank matrices and that are shared across all layers, and adapts these matrices by learning small, trainable scaling vectors represented as and (formally denoted by diagonal matrices and . Specifically, the reparameterization is given by:
其中 表示变换器模块前馈层的权重或偏置。值得注意的是，在推理过程中，这种方法会提前计算 ，确保模型的推理延迟与传统的完全微调相同。VeRA（基于向量的随机矩阵适应）[74] 采用了一对在所有层中共享的冻结低秩矩阵 和 ，并通过学习以 和 表示的小型可训练缩放向量来适应这些矩阵（正式表示为对角矩阵 和 。具体来说，重新参数化的方法如下

where both and are initialized using a random Gaussian distribution. Similar to LoRA, the scaling vector is initialized to zeros to ensure that the weight matrix is unaffected during the first forward pass. This method significantly reduces the number of trainable parameters compared to LoRA yet maintains the same performance, enabling the fine-tuning of larger models on a single GPU. DoRA (WeightDecomposed Low-Rank Adaptation) [75] presents a novel approach as illustrated in Figure 8 (c) by decomposing model weights into magnitude and direction as follows:
其中 和 都使用随机高斯分布初始化。与 LoRA 类似，缩放向量 初始化为零，以确保权重矩阵在第一次前向传递时不受影响。与 LoRA 相比，这种方法大大减少了可训练参数的数量，但保持了相同的性能，从而可以在单个 GPU 上对更大的模型进行微调。DoRA（权重分解低级适应）[75] 提出了一种新颖的方法，如图 8 (c) 所示，它将模型权重 分解为大小和方向，具体如下：

where is the magnitude vector, is the directional matrix, with being the vector-wise norm of a matrix across each column. Subsequently, DoRA adopts a unique fine-tuning strategy for and . While both are tunable, only undergoes LoRA reparameterization, defined as:
其中， 是幅值向量， 是方向矩阵， 是矩阵每列的矢量-向规范。随后，DoRA 对 和 采用独特的微调策略。虽然两者都是可调的，但只有 会进行 LoRA 重参数化，其定义如下：

where is the incremental directional update learned by LoRA, and the underlined parameters denote the trainable parameters. Through this methodology, DoRA consistently outperforms LoRA across various tasks and models, demonstrating its superiority.
其中 是 LoRA 学习到的增量方向更新，下划线参数表示可训练参数。通过这种方法，DoRA 在各种任务和模型中的表现始终优于 LoRA，证明了其优越性。

The efficacy of various PEFT methods can significantly differ across different tasks. As a result, numerous studies aim to either combine the advantages of diverse PEFT approaches or seek to establish a unified perspective by analyzing the similarities among these methods. For instance, UniPELT[90] integrates LoRA, prefix-tuning, and adapters into each Transformer block. To control which PEFT submodules should be activated, they also introduce a gating mechanism. This
在不同的任务中，各种 PEFT 方法的功效会有很大不同。因此，许多研究旨在结合不同 PEFT 方法的优势，或通过分析这些方法之间的相似性来建立统一的观点。例如，UniPELT[90] 将 LoRA、前缀调谐和适配器集成到每个转换器块中。为了控制哪些 PEFT 子模块应被激活，他们还引入了门控机制。这
mechanism consists of three small FFNs that each produce a scalar value , which is then applied to the LoRA, prefix, and adapter matrices, respectively. Across various setups, UniPELT has consistently shown improvements in accuracy ranging from to . S4 [91] explores design spaces for several PEFT methods (i.e., Adapter (A), Prefix (P), BitFit (B), and LoRA (L)) to uncover underlying design patterns. After a series experiments, their findings include: (1) Applying the spindle grouping partitioning for Transformer layers, which results in four layer groups for . Layers in one group have similar behaviors together, which means should be apply similar PEFT strategies. (2) Allocating the number of trainable parameters to layers uniformly. (3) Tuning all the groups. (4) Assigning different PEFT strategies in different group. The resulting design space that has the best performance is:
该机制由三个小型 FFN 组成，每个 FFN 产生一个标量值 ，然后分别应用于 LoRA、前缀和适配器矩阵。在不同的设置下，UniPELT 的准确率一直在 到 之间。S4 [91] 探索了几种 PEFT 方法（即 Adapter (A)、Prefix (P)、BitFit (B) 和 LoRA (L)）的设计空间，以发现潜在的设计模式。经过一系列实验，他们的发现包括(1) 对 Transformer 层应用主轴分组分区，结果产生了四个层组 ， 。一个组中的层具有相似的行为，这意味着应采用相似的 PEFT 策略。(2) 将可训练参数的数量统一分配给各层。(3) 调整所有组。(4) 在不同组中分配不同的 PEFT 策略。最终得出的性能最佳的设计空间是

MAM Adapter[26] explores the intrinsic similarity between three additive PEFT methods: adapters, prefix-tuning, and LoRA, which leads to the development of three variants: Parallel Adapter, which places adapter layers alongside specific layers (SA or FFN) instead of after them; Multi-head Parallel Adapter, which divides the parallel adapter into multiple heads, each affecting the head attention output in SA; and Scaled Parallel Adapter, which adds a scaling term after the parallel adapter layer, similar to LoRA. Extensive experimentation revealed that the most effective configuration involves using prefix-tuning in the SA layer and the scaled parallel adapter in the FFN layer, which is called MAM Adapter. LLM-Adapters [94] builds an easy-to-use framework that incorporates various PEFT techniques into LLMs. Through comprehensive benchmarking across multiple datasets, the study reveals several key insights: (1) The most effective locations for series adapters, parallel adapters, and LoRA are after the MLP layers, alongside the MLP layers, and simultaneously following the Attention layers and MLP layers, respectively. (2) Smaller LLMs utilizing PEFT can achieve competitive or even superior results on certain tasks when compared to their larger counterparts. (3) With appropriate in-distribution fine-tuning data, smaller models are capable of surpassing larger models in task-specific performance.
MAM 适配器[26] 探索了三种相加 PEFT 方法之间的内在相似性：适配器、前缀调整和 LoRA，并由此开发出三种变体：并行适配器，将适配器层与特定层（SA 或 FFN）放在一起，而不是放在它们之后；多头并行适配器，将并行适配器分为多个头，每个头都会影响 SA 中的头注意输出；缩放并行适配器，在并行适配器层之后添加一个缩放项，与 LoRA 类似。大量实验表明，最有效的配置是在 SA 层使用前缀调整，在 FFN 层使用缩放并行适配器，即 MAM 适配器。LLM-适配器 [94] 构建了一个易于使用的框架，将各种 PEFT 技术整合到LLMs 中。通过对多个数据集进行全面的基准测试，该研究揭示了几个关键的见解：（1）串联适配器、并联适配器和 LoRA 的最有效位置分别在 MLP 层之后、MLP 层旁边，以及同时在 Attention 层和 MLP 层之后。(2) 利用 PEFT 的小型LLMs 在某些任务上可以取得比大型同类产品更有竞争力甚至更优越的结果。(3) 利用适当的分布内微调数据，较小的模型能够在特定任务的性能上超越较大的模型。

Several studies leverage neural architecture search (NAS) to find better PEFT combination approaches. For example, NOAH [92] discovers that different PEFT configurations are specifically tailored for different tasks. To address this issue, NOAH employs NAS to identify the most effective PEFT configurations for each dataset. Specifically, NOAH's searching space encompasses three PEFT methods: Adapter, LoRA, and Visual Prompt Tuning (VPT). It utilizes AutoFormer [108], a one-shot NAS algorithm, for the efficient discovery of optimal prompt modules. In a related vein, AUTOPEFT [93] first establishes a searching space that includes serial adapters, parallel adapters, and prefix tuning. After that, they proposes an effective NAS methods based on a high-dimensional multidimensional Bayesian optimisation [109]. Both NOAH and AUTOPEFT demonstrate the capability of NAS in enhancing PEFT configurations across a variety of tasks.
一些研究利用神经架构搜索（NAS）来寻找更好的 PEFT 组合方法。例如，NOAH [92] 发现，不同的 PEFT 配置专门针对不同的任务。为了解决这个问题，NOAH 利用 NAS 来确定每个数据集最有效的 PEFT 配置。具体来说，NOAH 的搜索空间包括三种 PEFT 方法：Adapter、LoRA 和 Visual Prompt Tuning (VPT)。它利用 AutoFormer [108]（一种一次性 NAS 算法）高效地发现最佳提示模块。与此相关，AUTOPEFT [93] 首先建立了一个搜索空间，其中包括串行适配器、并行适配器和前缀调整。之后，他们提出了一种基于高维多维贝叶斯优化的有效 NAS 方法 [109]。NOAH 和 AUTOPEFT 都证明了 NAS 在各种任务中增强 PEFT 配置的能力。

Processing latency and peak memory overhead are pivotal factors to consider from a computational standpoint. This section introduces a key characteristic in LLMs aimed at balancing between latency and memory usage (Section IV-A). Following this, we explore strategies for developing efficient PEFT methods to address computational challenges, including PEFT pruning (Section IV-B), PEFT quantization (Section IV-C), and memory-efficient PEFT techniques (Section IV-D), each designed to enhance model performance while minimizing resource consumption. It is noteworthy that quantization inherently addresses memory overhead concerns. However, given its distinct characteristics, we address these quantization methods separately rather than incorporating them under the memory-efficient PEFT section.
从计算角度来看，处理延迟和峰值内存开销是需要考虑的关键因素。本节将介绍LLMs 中旨在平衡延迟和内存使用的关键特性（第 IV-A 节）。随后，我们将探讨开发高效 PEFT 方法的策略，以应对计算挑战，包括 PEFT 剪枝（第 IV-B 节）、PEFT 量化（第 IV-C 节）和内存高效 PEFT 技术（第 IV-D 节），每种方法都旨在提高模型性能，同时最大限度地减少资源消耗。值得注意的是，量化从本质上解决了内存开销问题。不过，考虑到其独特性，我们将这些量化方法单独讨论，而不是将其纳入内存效率 PEFT 部分。

The core of the LLMs model lies an autoregressive Transformer model, depicted in Figure 2 When we look at autoregression characteristic, it becomes a major challenge in designing an inference system, because every time a new token is generated, the entire LLM model has to transfer all the weights from different memories to the memory of the graphics processor, which is very unfriendly to single-user task scheduling or multi-user work-load balance. The challenging part of serving the auto-regressive paradigm is all previous sequences have to be cached and saved for the next proceeding iteration, the cached activation generated from the previous sequences is stored as the Key-Value Cache (KV-cache).
LLMs 模型的核心是一个自回归变换器模型，如图 2 所示。当我们看到自回归特性时，它成为设计推理系统的一大挑战，因为每生成一个新的标记，整个LLM 模型就必须将所有权重从不同的存储器转移到图形处理器的存储器中，这对单用户任务调度或多用户工作量平衡非常不友好。自动回归范式的挑战在于必须缓存和保存所有先前序列，以备下一次迭代之用，从先前序列生成的缓存激活被存储为键值缓存（KV-cache）。

The storage of KV-cache will cost both memory space and IO performance, yielding in workload memory-bounded and under-utilizing the computation power of the system. Previous works proposed a series of solutions like KV-cache control management [133] or KV-cache compression [134] to improve throughput or reduce latency. When designing PEFT methods, it is crucial to consider the characteristics of the KV-cache to complement its features. For instance, when applying soft prompts in the inference phase, efficiently leveraging the KVcache for these additional inputs can help accelerate response times by ensuring prompt-related data is readily accessible.
KV 缓存的存储会耗费内存空间和 IO 性能，导致工作负载受内存限制，系统计算能力利用不足。之前的研究提出了一系列解决方案，如 KV 缓存控制管理 [133] 或 KV 缓存压缩 [134]，以提高吞吐量或减少延迟。在设计 PEFT 方法时，考虑 KV 缓存的特性以补充其功能至关重要。例如，在推理阶段应用软提示时，有效利用 KV 缓存来处理这些额外的输入，可确保提示相关的数据可随时访问，从而有助于加快响应时间。

The inclusion of pruning can substantially enhance the efficiency of PEFT methods. In particular, AdapterDrop [110] explores the removal of adapters from lower transformer layers and multi-task adapters in AdapterFusion [29], which shows that the pruning can improve the training and inference efficiency with minimal decrease in performance. SparseAdapter [111] investigates different pruning methods and finds that high sparsity ratios can outperform standard adapters. Additionally, the Large-Sparse configuration, which increases the bottleneck dimension while maintaining a constant parameter budget (e.g., doubling dimensions with a sparsity), substantially enhances the model's capacity, resulting in improved performance. SPLoRA [112] adopts channel-based pruning to the LoRA weights and .
剪枝可以大大提高 PEFT 方法的效率。其中，AdapterDrop[110]探索了从低变压器层移除适配器的方法，以及 AdapterFusion[29]中的多任务适配器，结果表明剪枝可以提高训练和推理效率，而性能下降最小。SparseAdapter [111] 研究了不同的剪枝方法，发现高稀疏比的 可以优于标准适配器。此外，Large-Sparse 配置在保持参数预算不变的情况下增加了瓶颈维度（例如，在 稀疏度的情况下增加一倍维度），大大增强了模型的容量，从而提高了性能。SPLoRA [112] 对 LoRA 权重 和 采用了基于信道的剪枝。

BI-Adapter [115], PEQA [116], QLoRA [117], LoftQ [118], LQ-LoRA [119], QA-LoRA [120], INT2.1 [121], QDyLoRA [122], BitDelta [123]
BI-Adapter[115]、PEQA[116]、QLoRA[117]、LoftQ[118]、LQ-LoRA[119]、QA-LoRA[120]、INT2.1[121]、QDyLoRA[122]、BitDelta[123]

Fig. 9: Taxonomy of Efficient PEFT Design.
图 9：高效 PEFT 设计的分类标准。

This pruning affects not only the source weights , but also the LoRA parameters and . Similarly, LoRAPruning [113] adopts structured pruning not only to the pretrained model weights but also to the LoRA weights. In contrast to unstructured LoRA pruning methods, which primarily focus on sparsifying model weights while leaving LoRA weights dense, thus making weight merging challenging to achieve, LoRAPruning enables the weights to be merged easily. Additionally, this work also introduces a novel criterion that utilizes LoRA's gradients as an approximation of the gradients for the pre-trained weights, enabling the estimation of weight importance. ProPETL [114] constructs a single shared prototype (e.g., adapter, prefix, or LoRA) across layers and tasks. In addition, ProPETL learns binary masks to prune different sub-networks in different layers and tasks. As a result, the parameters can be reused in across layers and tasks, largely increasing the parameter efficiency.
这种剪枝不仅影响源权重 ，也影响 LoRA 参数 和 。同样，LoRAPruning [113] 不仅对预训练模型权重，而且对 LoRA 权重采用了结构化剪枝。非结构化 LoRA 修剪方法主要侧重于稀疏化模型权重，而让 LoRA 权重保持密集，从而使权重合并难以实现，相比之下，LoRAPruning 使权重很容易合并。此外，这项工作还引入了一种新标准，利用 LoRA 的梯度作为预训练权重梯度的近似值，从而能够估计权重的重要性。ProPETL [114] 构建了跨层和跨任务的单一共享原型（如适配器、前缀或 LoRA）。此外，ProPETL 还能学习二进制掩码，以剪除不同层和任务中的不同子网络。因此，参数可以跨层和跨任务重复使用，大大提高了参数效率。

Quantization serves as another popular technique for improving computational efficiency and reduce memory usage. For example, by investigating the loss landscape of adapters, BI-Adapter [115] finds that adapters are resistant to noise in parameter space. Building on this insight, the authors introduce a clustering-based quantization approach. Remarkably, they demonstrate that a 1 -bit quantization of adapters not only minimizes storage requirements but also achieve superior performance among all precision settings. PEQA (Parameter-Efficient and Quantization-aware Adaptation) [116] uses a two-stage pipeline to achieve parameterefficient and quantization-aware fine-tuning. In the first stage, the pre-trained FFN weight matrix is quantized to , where represents per-channel scales and denotes the quantized weight. In the second stage, remains fixed, and fine-tuning is only conducted on . This approach not only ensures memory efficiency but also facilitates parameter efficiency. QLoRA [117] proposes several novel techniques, including a 4-bit NormalFloat, a Double Quantization, and a Paged Optimizers, to backpropagate a 4-bit quantized pretrained language model into LoRA. These techniques enable the fine-tuning for a 65B language model on a single 48GB GPU while maintaining similar performance to the full 16-bit fine-tuning. Similar to the original implementation [70], QLoRA attaches the fixed zero initialized LoRA weights to the quantized pre-trained model as the training start point. However, when applying the extreme low-bit (e.g., 2-bit) quantization, the huge quantization error can adversely impact the initialization of LoRA fine-tuning, i.e., quantization where , which will harm the fine-tuning performance as shown in [127]. To solve this, several quantization strategies are proposed to eliminate the quantization error. For example, LoftQ (LoRA-Fine-Tuningaware Quantization) [118] presents an innovative framework that provides a superior initialization point of quantized backbone weights and LoRA weights for subsequent LoRA finetuning. This approach addresses the discrepancies caused by quantization through the optimization of a Frobenius norm objective during network initialization, which takes both the LoRA weights and the quantized pre-trained backbone into consideration. LoftQ exhibits superior performance in 2-bit quantization over QLoRA, as well as greater generalization for downstream tasks. LQ-LoRA [119] uses an iterative algorithm inspired by robust principal components analysis [135], [136] which decomposes the weight such that to resolve the inaccuracy caused by the quantization error, where is the quantized component which remains fixed and is the trainable low-rank component. Moreover, this approach leverages integer linear programming to determine a mixed quantization strategy, enabling dynamic quantization configurations for each weight matrix while adhering to a predetermined total bit rate limit. QA-LoRA [120] address another limitation of QLoRA, which struggles to preserve its quantized property post fine-tuning. In QLoRA, the quantized pre-trained weight (NF4) have to be recovered to FP16 to match the LoRA weight precision (FP16) during weight merging. Instead, QA-LoRA uses INT4 quantization and introduces group-wise operators to enable quantization during inference stage, therefore improve the efficiency and accuracy compared with QLoRA. BitDelta [123] introduces a novel 1-bit posttraining quantization method that acts on the weight delta between a fine-tuned model and its underlying pre-trained model. Specifically, given the weight matrices and from the fine-tuned and base models respectively, the weight delta is binarized as . Here, , a high-precision scalar, is initialized based on the mean absolute delta value , with indicating the sign of . BitDelta further calibrates the scaling factors via distillation on a compact calibration dataset, while the binary matrices remain unchanged. This approach notably streamlines the deployment of multiple fine-tuned models on shared servers by utilizing a singular full-precision base model alongside efficiently batched 1 -bit deltas.
量化是提高计算效率和减少内存使用的另一种流行技术。例如，通过研究适配器的损失情况，BI-Adapter [115] 发现适配器对参数空间的噪声具有抵抗力。基于这一发现，作者提出了一种基于聚类的量化方法。值得注意的是，他们证明对适配器进行 1 位量化不仅能最大限度地减少存储需求，还能在所有精度设置中实现卓越性能。PEQA（参数效率和量化感知适配）[116] 采用两阶段流水线实现参数效率和量化感知微调。在第一阶段，预训练的 FFN 权重矩阵 被量化为 ，其中 表示每个通道的比例， 表示量化后的权重。在第二阶段， 保持不变，只对 进行微调。这种方法不仅能确保内存效率，还能提高参数效率。QLoRA [117] 提出了几种新技术，包括 4 位 NormalFloat、双量化和分页优化器，将 4 位量化预训练语言模型反向传播到 LoRA 中。这些技术可在单个 48GB GPU 上对 65B 语言模型进行微调，同时保持与完整 16 位微调类似的性能。与最初的实现[70]类似，QLoRA 将固定为零的初始化 LoRA 权重附加到量化的预训练模型上作为训练起点。然而，在应用极端低位（如 2 位）量化时，巨大的量化误差会对 LoRA 微调的初始化产生不利影响，即量化 其中 ，这将损害微调性能，如文献[127]所示。为了解决这个问题，人们提出了几种量化策略来消除量化误差。例如，LoftQ（LoRA-Fine-Tuningaware Quantization）[118] 提出了一种创新框架，为随后的 LoRA 微调提供了量化主干权重和 LoRA 权重的卓越初始化点。这种方法通过优化网络初始化过程中的 Frobenius 准则目标来解决量化造成的差异，该目标同时考虑了 LoRA 权重和量化的预训练骨干网。LoftQ 在 2 位量化方面的性能优于 QLoRA，而且对下游任务的通用性也更强。 LQ-LoRA [119] 采用了一种受稳健主成分分析[135]、[136]启发的迭代算法，该算法分解权重 ，使 ，以解决量化误差造成的不准确性，其中 是保持固定的量化成分， 是可训练的低秩成分。此外，这种方法利用整数线性规划来确定混合量化策略，使每个权重矩阵都能进行动态量化配置，同时遵守预定的总比特率限制。QA-LoRA [120] 解决了 QLoRA 的另一个局限性，即在微调后难以保持其量化属性。在 QLoRA 中，量化的预训练权重（NF4）必须恢复到 FP16，以便在权重合并时与 LoRA 权重精度（FP16）相匹配。相反，QA-LoRA 使用 INT4 量化，并引入了分组运算符，以便在推理阶段进行量化，因此与 QLoRA 相比，QA-LoRA 提高了效率和精度。BitDelta [123] 引入了一种新颖的 1 位训练后量化方法，该方法作用于微调模型与其底层预训练模型之间的权重三角洲。具体来说，给定分别来自微调模型和基础模型的权重矩阵 和 ，将权重三角 二值化为 。 是一个高精度标量，根据平均绝对三角洲值 进行初始化， 表示 的符号。BitDelta 在二进制矩阵保持不变的情况下，通过对紧凑校准数据集的蒸馏进一步校准缩放因子。这种方法通过利用奇异的全精度基础模型和高效分批的 1 位三角积分，显著简化了在共享服务器上部署多个微调模型的过程。