Build a Large Language Model (From Scratch) Version 8
构建一个大型语言模型（从头开始）第 8 版

Copyright 2024 Manning Publications
版权所有 2024 曼宁出版社

welcome 欢迎

• Sebastian Raschka 塞巴斯蒂安·拉施卡

brief contents 简要内容

CHAPTERS 章节

1

Understanding Large Language Models
理解大型语言模型

This chapter covers 本章涵盖

• High-level explanations of the fundamental concepts behind large language models (LLMs)
  关于大型语言模型的基本概念的高层次解释(LLMs)
• Insights into the transformer architecture from which LLMs, such as the ones used on the ChatGPT platform, are derived
  从中派生出LLMs等 ChatGPT 平台使用的变换器架构的见解
• A plan for building an LLM from scratch
  从头开始建造LLM的计划

1.1 What is an LLM?
1.1 什么是LLM？

1.2 Applications of LLMs
1.2 LLMs

1.3 Stages of building and using LLMs
建立和使用LLMs的 1.3 阶段

1.4 Introducing the transformer architecture
1.4 引入变换器架构

TRANSFORMERS VERSUS LLMS
变形金刚对战LLMS

1.5 Utilizing large datasets
利用大型数据集

GPT-3 DATASET DETAILS GPT-3 数据集详情

1.6 A closer look at the GPT architecture
1.6 深入了解 GPT 架构

• Improving Language Understanding by Generative Pre-Training (2018) by Radford et al. from OpenAI, http://cdn.openai.com/researchcovers/language-unsupervised/language understanding_paper.pdf
  通过生成式预训练提高语言理解(2018 年)，Radford 等人来自 OpenAI,http://cdn.openai.com/researchcovers/language-unsupervised/language understanding_paper.pdf

The model is simply trained to predict the next word

该模型仅被训练来预测下一个词语

1.7 Building a large language model
建立一个大型语言模型

1.8 Summary 1.8 总结

• LLMs have transformed the field of natural language processing, which previously mostly relied on explicit rule-based systems and simpler statistical methods. The advent of LLMs introduced new deep learningdriven approaches that led to advancements in understanding, generating, and translating human language.
  LLMs已经改变了自然语言处理领域,之前该领域主要依赖于基于明确规则的系统和更简单的统计方法。LLMs的出现带来了新的基于深度学习的方法,推动了人类语言的理解、生成和翻译方面的进步。
• Modern LLMs are trained in two main steps.
  现代LLMs是通过两个主要步骤训练的。
• First, they are pretrained on a large corpus of unlabeled text by using the prediction of the next word in a sentence as a "label."
  首先,它们通过使用预测句子中下一个词作为"标签"而在一个大规模的未标注文本语料库上进行预训练。

• LLMs are based on the transformer architecture. The key idea of the transformer architecture is an attention mechanism that gives the LLM selective access to the whole input sequence when generating the output one word at a time.
  LLMs基于变换器架构。变换器架构的关键思想是注意力机制,该机制使LLM在逐个生成输出单词时能够有选择地访问整个输入序列。
• The original transformer architecture consists of an encoder for parsing text and a decoder for generating text.
  原始的 Transformer 架构由一个编码器用于解析文本和一个解码器用于生成文本组成。
• LLMs for generating text and following instructions, such as GPT-3 and ChatGPT, only implement decoder modules, simplifying the architecture.
  LLMs
• Large datasets consisting of billions of words are essential for pretraining LLMs. In this book, we will implement and train LLMs on small datasets for educational purposes but also see how we can load openly available model weights.
  包含数十亿个字的大型数据集对于预训练LLMs至关重要。在本书中,我们将在小型数据集上实现和训练LLMs,以供教育目的使用,同时也将了解如何加载公开可用的模型权重。
• While the general pretraining task for GPT-like models is to predict the next word in a sentence, these LLMs exhibit "emergent" properties such as capabilities to classify, translate, or summarize texts.
  虽然 GPT 类模型的常规预训练任务是预测句子中的下一个单词,但这些LLMs模型展现了"突现"属性,例如分类、翻译或总结文本的能力。
• Once an LLM is pretrained, the resulting foundation model can be finetuned more efficiently for various downstream tasks.
  一旦LLM经过预训练,所得到的基础模型可以更高效地进行微调以用于各种下游任务。
• LLMs finetuned on custom datasets can outperform general LLMs on specific tasks.
  在特定任务上,针对定制数据集进行微调的LLMs可以超越一般的LLMs。

Working with Text Data
处理文本数据

Thls chapter covers 本章涵盖

• Preparing text for large language model training
  为大型语言模型训练准备文本
• Splitting text into word and subword tokens
  将文本分割成单词和子词令牌
• Byte pair encoding as a more advanced way of tokenizing text
  字节对编码作为更先进的文本分词方式
• Sampling training examples with a sliding window approach
  使用滑动窗口方法对训练样本进行采样
• Converting tokens into vectors that feed into a large language model
  将令牌转换为输入大型语言模型的向量

2.1 Understanding word embeddings
2.1 理解词嵌入

2.2 Tokenizing text 2.2 文本分词

Listing 2.1 Reading in a short story as text sample into Python
列 2.1 将短篇小说作为文本样本读取到 Python

with open("the-verdict.txt", "r", encoding="utf-8") as f:
    raw_text = f.read()
print("Total number of character:", len(raw_text))
print(raw_text[:99])

TEXT SAMPLE SIZES 文本样本大小

import re
text = "Hello, world. This, is a test."
result = re.split(r'(\s)', text)
print(result)

result = [item for item in result if item.strip()]
print(result)

REMOVING WHITESPACES OR NOT
去除空白或不去除

text = "Hello, world. Is this-- a test?"
result = re.split(r'([,.:;?_!"()\']|--|\s)', text)
result = [item.strip() for item in result if item.strip()]
print(result)

Input text 输入文本

preprocessed = re.split(r'([,.:;?_!"()\']|--|\s)', raw_text)
preprocessed = [item.strip() for item in preprocessed if item.strip()]
print(len(preprocessed))

print(preprocessed[:30])

2.3 Converting tokens into token IDs
将令牌转换为令牌 ID

all_words = sorted(set(preprocessed))
vocab_size = len(all_words)
print(vocab_size)

Listing 2.3 Implementing a simple text tokenizer
列表 2.3 实现简单的文本分词器

def
    _init__(self, vocab):
    self.str_to_int = vocab #A
    self.int_to_str = {i:s for s,i in vocab.items()} #B
    def encode(self, text): #C
        preprocessed = re.split(r'([,.?_!"()\']|--|\s)', text)
        preprocessed = [item.strip() for item in preprocessed if item.strip()]
        ids = [self.str_to_int[s] for s in preprocessed]
        return ids
    def decode(self, ids):
        #D
        text = " ".join([self.int_to_str[i] for i in ids])
        text = re.sub(r'\s+([,.?!"()\'])', r'\1', text) #E
        return text

tokenizer = SimpleTokenizerV1(vocab)
text = """"It's the last he painted, you know," Mrs. Gisburn said with pardonable
pride."""
ids = tokenizer.encode(text)
print(ids)

2.4 Adding special context tokens
2.4 添加特殊上下文标记

all_tokens = sorted(list(set(preprocessed)))
all_tokens.extend(["<|endoftext|>", "<|unk|>"])
vocab = {token:integer for integer,token in enumerate(all_tokens)}
print(len(vocab.items()))

('younger', 1156)
('your', 1157)
('yourself', 1158)
('<|endoftext|>', 1159)
('<|unk|>', 1160)

Listing 2.4 A simple text tokenizer that handles unknown words
列表 2.4 一个简单的文本分词器,可以处理未知词

class SimpleTokenizerV2:
    def __init__(self, vocab):
        self.str_to_int = vocab
        self.int_to_str = { i:s for s,i in vocab.items()}
    def encode(self, text):
        preprocessed = re.split(r'([,.?_!"()\']|--|\s)', text)
        preprocessed = [item.strip() for item in preprocessed if item.strip()]
        preprocessed = [item if item in self.str_to_int #A
                else "<|unk|>" for item in preprocessed]
        ids = [self.str_to_int[s] for s in preprocessed]
        return ids
    def decode(self, ids):
        text = " ".join([self.int_to_str[i] for i in ids])
        text = re.sub(r'\s+([,.?!"()\'])', r'\1', text)
        #B
    return text

text1 = "Hello, do you like tea?"
text2 = "In the sunlit terraces of the palace."
text = " <|endoftext|> ".join((text1, text2))
print(text)

• [BOS] (beginning of sequence): This token marks the start of a text. It signifies to the LLM where a piece of content begins.
  [BOS]这个标记表示文本的开始。它向LLM表示内容的起始位置。
• [EOS] (end of sequence): This token is positioned at the end of a text, and is especially useful when concatenating multiple unrelated texts, similar to <|endoftext|>. For instance, when combining two different Wikipedia articles or books, the [EOS] token indicates where one article ends and the next one begins.
  [EOS]这个标记位于文本的末尾,当拼接多个无关的文本时特别有用,类似于<|endoftext|>。例如,当合并两篇不同的维基百科文章或书籍时,[EOS]标记指示一篇文章的结尾和下一篇的开始。
• [PAD] (padding): When training LLMs with batch sizes larger than one, the batch might contain texts of varying lengths. To ensure all texts have the same length, the shorter texts are extended or "padded" using the [PAD] token, up to the length of the longest text in the batch.
  [PAD] (填充):在训练批量大小大于 1 的LLMs时,批量可能包含长度不同的文本。为确保所有文本长度相同,使用[PAD]标记对较短的文本进行扩展或"填充",直至达到批量中最长文本的长度。

2.5 Byte pair encoding
2.5 字节对编码

from importlib.metadata import version
import tiktoken
print("tiktoken version:", version("tiktoken"))

tokenizer = tiktoken.get_encoding("gpt2")

text = "Hello, do you like tea? <|endoftext|> In the sunlit terraces of
someunknownPlace."
integers = tokenizer.encode(text, allowed_special={"<|endoftext|>"})
print(integers)

Text sample with 文本样本

unknown words

EXERCISE 2.1 BYTE PAIR ENCODING OF UNKNOWN WORDS
第 2.1 练习 未知词的字节对编码

2.6 Data sampling with a sliding window
使用滑动窗口进行数据采样 2.6

with open("the-verdict.txt", "r", encoding="utf-8") as f:
    raw_text = f.read()
enc_text = tokenizer.encode(raw_text)
print(len(enc_text))

context_size = 4 #A
x = enc_sample[:context_size]
y = enc_sample[1:context_size+1]
print(f"x: {x}")
print(f"y: {y}")

for i in range(1, context_size+1):
    context = enc_sample[:i]
    desired = enc_sample[i]
    print(context, "---->", desired)

for i in range(1, context_size+1):
    context = enc_sample[:i]
    desired = enc_sample[i]
    print(tokenizer.decode(context), "---->", tokenizer.decode([desired]))

and ----> established
and established ----> himself
and established himself ----> in
and established himself in ----> a

import torch
from torch.utils.data import Dataset, DataLoader
class GPTDatasetV1(Dataset):
    def __init__(self, txt, tokenizer, max_length, stride):
        self.input_ids = []
        self.target_ids = []
        token_ids = tokenizer.encode(txt) #A
        for i in range(0, len(token_ids) - max_length, stride): #B
            input_chunk = token_ids[i:i + max_length]
            target_chunk = token_ids[i + 1: i + max_length + 1]
            self.input_ids.append(torch.tensor(input_chunk))
            self.target_ids.append(torch.tensor(target_chunk))
    def __len__(self): #C
    def __getitem__(self, idx): #D
        return self.input_ids[idx], self.target_ids[idx]

Listing 2.6 A data loader to generate batches with input-with pairs
列表 2.6 生成带有输入输出对的批次数据的数据加载器

def create_dataloader_v1(txt, batch_size=4, max_length=256,
    stride=128, shuffle=True, drop_last=True, num_workers=0)
tokenizer = tiktoken.get_encoding("gpt2") #A
dataset = GPTDatasetV1(txt, tokenizer, max_length, stride) #B
dataloader = DataLoader(
dataloader = DataLoader(
    dataset,
    batch_size=batch_size,
    shuffle=shuffle,
    drop_last=drop_last, #C
    num_workers=0 #D
)

with open("the-verdict.txt", "r", encoding="utf-8") as f:
    raw_text = f.read()
dataloader = create_dataloader_v1(
    raw_text, batch_size=1, max_length=4, stride=1, shuffle=False)
data_iter = iter(dataloader)
    #A
first_batch = next(data_iter)
print(first_batch)

second_batch = next(data_iter)
print(second_batch)

EXERCISE 2.2 DATA LOADERS WITH DIFFERENT STRIDES AND CONTEXT SIZES
练习 2.2 具有不同步幅和上下文大小的数据加载器

dataloader = create_dataloader_v1(raw_text, batch_size=8, max_length=4, stride=4)
data_iter = iter(dataloader)
inputs, targets = next(data_iter)
print("Inputs:\n", inputs)
print("\nTargets:\n", targets)
This prints the following:
Inputs:
    tensor([[ 40, 367, 2885, 1464],
        [ 1807, 3619, 402, 271],
        [10899, 2138, 257, 7026],
        [15632, 438, 2016, 257],
        [ 922, 5891, 1576, 438],
        [ 568, 340, 373, 645],
        [ 1049, 5975, 284, 502],
        [ 284, 3285, 326, 11]])
Targets:
    tensor([[ 367, 2885, 1464, 1807],
        [ 3619, 402, 271, 10899],
        [ 2138, 257, 7026, 15632],
        [ 438, 2016, 257, 922],
        [ 5891, 1576, 438, 568],
        [ 340, 373, 645, 1049],
        [ 5975, 284, 502, 284],
        [ 3285, 326, 11, 287]])

2.7 Creating token embeddings
创建令牌嵌入

input_ids = torch.tensor([2, 3, 5, 1])

torch.manual seed(123)
embedding_layer = torch.nn.Embedding(vocab_size, output_dim)
print(embedding_layer.weight)

Parameter containing:
tensor([[ 0.3374, -0.1778, -0.1690],
    [ 0.9178, 1.5810, 1.3010],
    [ 1.2753, -0.2010, -0.1606],
    [-0.4015, 0.9666, -1.1481],
    [-1.1589, 0.3255, -0.6315],
    [-2.8400, -0.7849, -1.4096]], requires_grad=True)

EMBEDDING LAYERS VERSUS MATRIX MULTIPLICATION
嵌入层与矩阵乘法

2.8 Encoding word positions
2.8 编码词位置

vocab_size = 50257
output_dim = 256
token_embedding_layer = torch.nn.Embedding(vocab_size, output_dim)

max_length = 4
dataloader = create_dataloader_v1(
    raw_text, batch_size=8, max_length=max_length, stride=max_length, shuffle=False)
data_iter = iter(dataloader)
inputs, targets = next(data_iter)
print("Token IDs:\n", inputs)
print("\nInputs shape:\n", inputs.shape)

Token IDs:
    tensor([[ 40, 367, 2885, 1464],
        [ 1807, 3619, 402, 271],
        [10899, 2138, 257, 7026],
        [15632, 438, 2016, 257],
        [ 922, 5891, 1576, 438],
        [ 568, 340, 373, 645],
        [ 1049, 5975, 284, 502],
        [ 284, 3285, 326, 11]])

token_embeddings = token_embedding_layer(inputs)
print(token_embeddings.shape)

context_length = max_length
pos_embedding_layer = torch.nn.Embedding(context_length, output_dim)
pos_embeddings = pos_embedding_layer(torch.arange(context_length))
print(pos_embeddings.shape)

input_embeddings = token_embeddings + pos_embeddings
print(input_embeddings.shape)
torch.Size([8, 4, 256])

2.9 Summary 2.9 总结

• LLMs require textual data to be converted into numerical vectors, known as embeddings since they can't process raw text. Embeddings transform discrete data (like words or images) into continuous vector spaces, making them compatible with neural network operations.
  LLMs需要将文本数据转换为数值向量,称为嵌入,因为它们无法处理原始文本。嵌入将离散数据(如单词或图像)转换为连续向量空间,使其与神经网络操作兼容。

• As the first step, raw text is broken into tokens, which can be words or characters. Then, the tokens are converted into integer representations, termed token IDs.
  作为第一步,原始文本被分解成可以是单词或字符的标记。然后,这些标记被转换成整数表示,称为标记 ID。
• Special tokens, such as <|unk|> and <|endoftext|>, can be added to enhance the model's understanding and handle various contexts, such as unknown words or marking the boundary between unrelated texts.
  <代码 0>特殊标记,如<|unk|>和<|endoftext|>,可以添加以增强模型的理解能力,并处理各种上下文,如未知词或标记不相关文本之间的边界。
• The byte pair encoding (BPE) tokenizer used for LLMs like GPT-2 and GPT3 can efficiently handle unknown words by breaking them down into subword units or individual characters.
  字节对编码(BPE)分词器用于像 GPT-2 和 GPT3 这样的模型,可以通过将未知词分解成子词单元或个别字符来高效处理未知词。
• We use a sliding window approach on tokenized data to generate inputtarget pairs for LLM training.
  我们对分词数据使用滑动窗口方法来生成LLM训练的输入目标对。
• Embedding layers in PyTorch function as a lookup operation, retrieving vectors corresponding to token IDs. The resulting embedding vectors provide continuous representations of tokens, which is crucial for training deep learning models like LLMs.
  在 PyTorch 中,嵌入层函数作为查找操作,检索与令牌 ID 对应的向量。得到的嵌入向量提供了令牌的连续表征,这对于训练深度学习模型（如LLMs）至关重要。
• While token embeddings provide consistent vector representations for each token, they lack a sense of the token's position in a sequence. To rectify this, two main types of positional embeddings exist: absolute and relative. OpenAI's GPT models utilize absolute positional embeddings that are added to the token embedding vectors and are optimized during the model training.
  虽然令牌嵌入为每个令牌提供了一致的向量表示,但它们缺乏对令牌在序列中位置的感知。为了纠正这一点,存在两种主要类型的位置嵌入:绝对和相对。OpenAI 的 GPT 模型利用绝对位置嵌入,它们被添加到令牌嵌入向量中,并在模型训练期间进行优化。

Coding Attention Mechanisms
编码注意力机制

This chapter covers 本章涵盖

• Exploring the reasons for using attention mechanisms in neural networks
  探索在神经网络中使用注意力机制的原因
• Introducing a basic self-attention framework and progressing to an enhanced selfattention mechanism
  介绍一个基本的自注意力框架并发展到一个增强的自注意力机制
• Implementing a causal attention module that allows LLMs to generate one token at a time
  实施一个因果注意力模块,使LLMs能够逐个生成标记
• Masking randomly selected attention weights with dropout to reduce overfitting
  使用随机遮蔽注意力权重进行 dropout 以减少过拟合
• Stacking multiple causal attention modules into a multi-head attention module
  将多个因果注意力模块堆叠成一个多头注意力模块

3.1 The problem with modeling long sequences
3.1 对于建模长序列的问题

3.2 Capturing data dependencies with attention mechanisms
通过注意力机制捕获数据依赖关系

3.3 Attending to different parts of the input with self-attention
通过自注意力关注输入的不同部分

THE "SELF" IN SELF-ATTENTION
自注意力中的"自我"

3.3.1 A simple self-attention mechanism without trainable weights
3.3.1 没有可训练权重的简单自注意力机制

import torch
inputs = torch.tensor(
    [[0.43, 0.15, 0.89], # Your (x^1)
    [0.55, 0.87, 0.66], # journey (x^2)
    [0.57, 0.85, 0.64], # starts (x^3)
    [0.22, 0.58, 0.33], # with (x^4)
    [0.77, 0.25, 0.10], # one (x^5)
    [0.05, 0.80, 0.55]] # step (x^6)
)

query = inputs[1] #A
attn_scores_2 = torch.empty(inputs.shape[0])
for i, x_i in enumerate(inputs):
    attn_scores_2[i] = torch.dot(x_i, query)
print(attn_scores_2)

tensor([0.9544, 1.4950, 1.4754, 0.8434, 0.7070, 1.0865])

UNDERSTANDING DOT PRODUCTS
了解点积

res = 0.
for idx, element in enumerate(inputs[0]):
    res += inputs[0][idx] * query[idx]
print(res)
print(torch.dot(inputs[0], query))

tensor(0.9544)
tensor(0.9544)

def softmax_naive(x):
    return torch.exp(x) / torch.exp(x).sum(dim=0)
attn_weights_2_naive = softmax_naive(attn_scores_2)
print("Attention weights:", attn_weights_2_naive)
print("Sum:", attn_weights_2_naive.sum())

attn_weights_2 = torch.softmax(attn_scores_2, dim=0)
print("Attention weights:", attn_weights_2)
print("Sum:", attn_weights_2.sum())

query = inputs[1] # 2nd input token is the query
context_vec_2 = torch.zeros(query.shape)
for i,x_i in enumerate(inputs):
    context_vec_2 += attn_weights_2[i]*x_i
print(context_vec_2)

tensor([0.4419, 0.6515, 0.5683])

3.3.2 Computing attention weights for all input tokens
3.3.2 为所有输入标记计算注意力权重

attn_scores = torch.empty(6, 6)
for i, x_i in enumerate(inputs):
    for j, x_j in enumerate(inputs):
        attn_scores[i, j] = torch.dot(x_i, x_j)
print(attn_scores)

attn_scores = inputs @ inputs.T
print(attn_scores)

attn_weights = torch.softmax(attn_scores, dim=-1)
print(attn_weights)

row_2_sum = sum([0.1385, 0.2379, 0.2333, 0.1240, 0.1082, 0.1581])
print("Row 2 sum:", row_2_sum)
print("All row sums:", attn_weights.sum(dim=-1))

Row 2 sum: 1.0
第 2 行和：1.0

3.4 Implementing self-attention with trainable weights
使用可训练权重实现 self-attention

3.4.1 Computing the attention weights step by step
逐步计算注意力权重

x_2 = inputs[1] #A
d_in = inputs.shape[1] #B
d_out = 2 #C

torch.manual_seed(123)
W_query = torch.nn.Parameter(torch.rand(d_in, d_out), requires_grad=False)
W_key = torch.nn.Parameter(torch.rand(d_in, d_out), requires_grad=False)
W_value = torch.nn.Parameter(torch.rand(d_in, d_out), requires_grad=False)

query_2 = x_2 @ W_query
key_2 = x_2 @ W_key
value_2 = x_2 @ W_value
print(query_2)

tensor([0.4306, 1.4551])

WEIGHT PARAMETERS VS ATTENTION WEIGHTS
权重参数与注意力权重

keys = inputs @ W_key
values = inputs @ W_value
print("keys.shape:", keys.shape)
print("values.shape:", values.shape)

keys_2 = keys[1]
attn_score_22 = query_2.dot(keys_2)
print(attn_score_22)

tensor([1.2705, 1.8524, 1.8111, 1.0795, 0.5577, 1.5440])

d_k = keys.shape[-1]
attn_weights_2 = torch.softmax(attn_scores_2 / d_k**0.5, dim=-1)
print(attn_weights_2)

tensor([0.1500, 0.2264, 0.2199, 0.1311, 0.0906, 0.1820])

THE RATIONALE BEHIND SCALED-DOT PRODUCT ATTENTION
缩放点积注意力的基本原理

context_vec_2 = attn_weights_2 @ values
print(context_vec_2)

WHY QUERY, KEY, AND VALUE?
为什么是查询、键和值？

3.4.2 Implementing a compact self-attention Python class
3.4.2 实现一个紧凑的自注意力力 Python 类

Listing 3.1 A compact self-attention class
3.1 节 一个紧凑的自注意力类

import torch.nn as nn
class SelfAttention_v1(nn.Module):
    def __init__(self, d_in, d_out):
        super().__init__()
        self.d_out = d_out
        self.W_query = nn.Parameter(torch.rand(d_in, d_out))
        self.W_key = nn.Parameter(torch.rand(d_in, d_out))
        self.W_value = nn.Parameter(torch.rand(d_in, d_out))
    def forward(self, x):
        keys = x @ self.W_key
        queries = x @ self.w_query
        values = x @ self.w_value
        attn_scores = queries @ keys.T # omega
        attn_weights = torch.softmax(
            attn_scores / keys.shape[-1]**0.5, dim=-1)
        context_vec = attn_weights @ values
        return context_vec

torch.manual seed(123)
sa_v1 = SelfAttention_v1(d_in, d_out)
print(sa_v1(inputs))

Listing 3.2 A self-attention class using PyTorch's Linear layers
使用 PyTorch 的 Linear 层的自注意力类

class SelfAttention_v2(nn.Module):
    def __init__(self, d_in, d_out, qkv_bias=False):
        super().__init__()
        self.d_out = d_out
        self.W_query = nn.Linear(d_in, d_out, bias=qkv_bias)
        self.W_key = nn.Linear(d_in, d_out, bias=qkv_bias)
        self.W_value = nn.Linear(d_in, d_out, bias=qkv_bias)
    def forward(self, x):
        keys = self.W_key(x)
        queries = self.W_query(x)
        values = self.W_value(x)
        attn_scores = queries @ keys.T
        attn_weights = torch.softmax(attn_scores / keys.shape[-1]**0.5, dim=-1)
        context_vec = attn_weights @ values
        return context_vec

torch.manual_seed(789)
sa_v2 = SelfAttention_v2(d_in, d_out)
print(sa_v2(inputs))

EXERCISE 3.1 COMPARING SELFATTENTION_V1 AND SELFATTENTION_V2
练习 3.1 比较 SELFATTENTION_V1 和 SELFATTENTION_V2

3.5 Hiding future words with causal attention
3.5 使用因果注意力隐藏未来词

3.5.1 Applying a causal attention mask
3.5.1 应用因果注意力遮罩

queries = sa_v2.w_query(inputs)
    #A
keys = sa_v2.w_key(inputs)
attn_scores = queries @ keys.T
attn_weights = torch.softmax(attn_scores / keys.shape[-1]**0.5, dim=1)
print(attn_weights)

context_length = attn_scores.shape[0]
mask_simple = torch.tril(torch.ones(context_length, context_length))
print(mask_simple)

masked_simple = attn_weights*mask_simple
print(masked_simple)

row_sums = masked_simple.sum(dim=1, keepdim=True)
masked_simple_norm = masked_simple / row_sums
print(masked_simple_norm)

INFORMATION LEAKAGE 信息泄露

mask = torch.triu(torch.ones(context_length, context_length), diagonal=1)
masked = attn_scores.masked_fill(mask.bool(), -torch.inf)
print(masked)

attn_weights = torch.softmax(masked / keys.shape[-1]**0.5, dim=1)
print(attn_weights)

3.5.2 Masking additional attention weights with dropout
3.5.2 使用 dropout 遮蔽额外的注意力权重

torch.manual_seed(123)
dropout = torch.nn.Dropout(0.5) #A
example = torch.ones(6, 6) #B
print(dropout(example))

torch.manual_seed(123)
print(dropout(attn_weights))

3.5.3 Implementing a compact causal attention class
实现紧凑因果注意力类

batch = torch.stack((inputs, inputs), dim=0)
print(batch.shape)

def __init__(self, d_in, d_out, context_length, dropout, qkv_bias=False):
    super().__init__()
    self.d_out = d_out
    self.W_query = nn.Linear(d_in, d_out, bias=qkv_bias)
    self.W_key = nn.Linear(d_in, d_out, bias=qkv_bias)
    self.w_value = nn.Linear(d_in, d_out, bias=qkv_bias)
    self.dropout = nn.Dropout(dropout) #A
    self.register_buffer(
        'mask',
        torch.triu(torch.ones(context_length, context_length),
        diagonal=1)
    ) #B
    def forward(self, x):
        b, num_tokens, d_in = x.shape #C
        keys = self.w_key(x)
        queries = self.W_query(x)
        values = self.w_value(x)
        attn_scores = queries @ keys.transpose(1, 2) #C
        attn_scores.masked_fill_( #D
            self.mask.bool()[:num_tokens, :num_tokens], -torch.inf)
    attn_weights = torch.softmax(attn_scores / keys.shape[-1]**0.5, dim=-1)
    attn_weights = self.dropout(attn_weights)
    context_vec = attn_weights @ values
    return context_vec

torch.manual_seed(123)
context_length = batch.shape[1]
ca = CausalAttention(d_in, d_out, context_length, 0.0)
context_vecs = ca(batch)
print("context_vecs.shape:", context_vecs.shape)

3.6 Extending single-head attention to multi-head attention
将单头注意力扩展到多头注意力

3.6.1 Stacking multiple single-head attention layers
3.6.1 堆叠多个单头注意力层

class MultiHeadAttentionWrapper(nn.Module):
    def __init__(self, d_in, d_out, context_length,
        dropout, num_heads, qkv_bias=False):
        super().__init__()
        self.heads = nn.ModuleList(
            [CausalAttention(d_in, d_out, context_length, dropout, qkv_bias)
                for _ in range(num_heads)]
        )
    def forward(self, x):
        return torch.cat([head(x) for head in self.heads], dim=-1)

torch.manual_seed(123)
context_length = batch.shape[1] # This is the number of tokens
d_in, d_out = 3, 2
mha = MultiHeadAttentionWrapper(d_in, d_out, context_length, 0.0, num_heads=2)
context_vecs = mha(batch)
print(context_vecs)
print("context_vecs.shape:", context_vecs.shape)

EXERCISE 3.2 RETURNING 2-DIMENSIONAL EMBEDDING VECTORS
练习 3.2 返回二维嵌入向量

3.6.2 Implementing multi-head attention with weight splits
3.6.2 实施权重拆分的多头注意力机制

class MultiHeadAttention(nn.Module):
    def
\(\qquad\)
                    init__(self, d_in, d_out,
                        context_length, dropout, num_heads, qkv_bias=False):
    super().__init__()
    assert d_out % num_heads == 0, "d_out must be divisible by num_heads"
    self.d_out = d_out
    self.num_heads = num_heads
    self.head_dim = d_out // num_heads
\#A
    self.W_query = nn.Linear(d_in, d_out, bias=qkv_bias)
    self.W_key = nn.Linear(d_in, d_out, bias=qkv_bias)
    self.W_value = nn.Linear(d_in, d_out, bias=qkv_bias)
    self.out_proj = nn.Linear(d_out, d_out)
    #B
    self.dropout = nn.Dropout(dropout)
    self.register_buffer(
        'mask',
        torch.triu(torch.ones(context_length, context_length), diagonal=1)
    )
def forward(self, x):
    b, num_tokens, d_in = x.shape
    keys = self.w_key(x) #C
    queries = self.w_query(x) #C
    values = self.W_value(x) #C
    keys = keys.view(b, num_tokens, self.num_heads, self.head_dim) #D
    values = values.view(b, num_tokens, self.num_heads, self.head_dim) #D
    queries = queries.view(b, num_tokens, self.num_heads, self.head_dim)#D
        © Manning Publications Co. To comment go to liveBook

keys = keys.transpose(1, 2) #E
queries = queries.transpose(1, 2) #E
values = values.transpose(1, 2) #E
attn_scores = queries @ keys.transpose(2, 3) #F
mask_bool = self.mask.bool()[:num_tokens, :num_tokens] #G
attn_scores.masked_fill_(mask_bool, -torch.inf) #H
attn_weights = torch.softmax(
    attn_scores / keys.shape[-1]**0.5, dim=-1)
attn_weights = self.dropout(attn_weights)
context_vec = (attn_weights @ values).transpose(1, 2) #I
    #J
context_vec = context_vec.contiguous().view(b, num_tokens, self.d_out)
context_vec = self.out_proj(context_vec) #K
return context_vec

Perform two matrix multiplications to obtain
进行两次矩阵乘法以获得

print(a @ a.transpose(2, 3))

first_head = a[0, 0, :, :]
first_res = first_head @ first_head.T
print("First head:\n", first_res)
second_head = a[0, 1, :, :]
second_res = second_head @ second_head.T
print("\nSecond head:\n", second_res)

torch.manual_seed(123)
batch_size, context_length, d_in = batch.shape
d_out = 2
mha = MultiHeadAttention(d_in, d_out, context_length, 0.0, num_heads=2)
context_vecs = mha(batch)
print(context_vecs)
print("context_vecs.shape:", context_vecs.shape)

tensor([[[0.3190, 0.4858],
    [0.2943, 0.3897],
    [0.2856, 0.3593],
    [0.2693, 0.3873],
    [0.2639, 0.3928],
    [0.2575, 0.4028]],
    [[0.3190, 0.4858],
    [0.2943, 0.3897],
    [0.2856, 0.3593],
    [0.2693, 0.3873],
    [0.2639, 0.3928],
    [0.2575, 0.4028]]], grad_fn=<ViewBackward0>)
context_vecs.shape: torch.Size([2, 6, 2])

EXERCISE 3.3 INITIALIZING GPT-2 SIZE ATTENTION MODULES
练习 3.3 初始化 GPT-2 尺寸注意力模块

3.7 Summary 3.7 总结

• Attention mechanisms transform input elements into enhanced context vector representations that incorporate information about all inputs.
  注意力机制将输入元素转换为增强的上下文向量表示,这些表示包含了所有输入的信息。
• A self-attention mechanism computes the context vector representation as a weighted sum over the inputs.
  自注意力机制将上下文向量表示计算为输入的加权和。
• In a simplified attention mechanism, the attention weights are computed via dot products.
  在简化的注意力机制中,注意力权重通过点积计算得出。
• A dot product is just a concise way of multiplying two vectors elementwise and then summing the products.
  点积只是一种简洁的方式来逐元素地乘以两个向量,然后对这些乘积求和。
• Matrix multiplications, while not strictly required, help us to implement computations more efficiently and compactly by replacing nested forloops.
  矩阵乘法虽然不是严格必需的,但通过代替嵌套的 for 循环,可以更有效和紧凑地实现计算。
• In self-attention mechanisms that are used in LLMs, also called scaled-dot product attention, we include trainable weight matrices to compute intermediate transformations of the inputs: queries, values, and keys.
  在用于LLMs的自注意力机制(也称为缩放点积注意力)中,我们包括可训练的权重矩阵来计算输入的中间转换:查询、值和键。
• When working with LLMs that read and generate text from left to right, we add a causal attention mask to prevent the LLM from accessing future tokens.
  当使用从左到右读取和生成文本的LLMs时，我们添加因果注意力掩码以防止LLM访问未来的 Token。
• Next to causal attention masks to zero out attention weights, we can also add a dropout mask to reduce overfitting in LLMs.
  除了使用因果注意力掩码来减少注意力权重外，我们还可以添加 dropout 掩码来降低LLMs中的过拟合。
• The attention modules in transformer-based LLMs involve multiple instances of causal attention, which is called multi-head attention.
  基于变压器的LLMs中的注意力模块涉及多个因果注意力的实例,这称为多头注意力。
• We can create a multi-head attention module by stacking multiple instances of causal attention modules.
  我们可以通过堆叠多个因果注意力模块来创建一个多头注意力模块。
• A more efficient way of creating multi-head attention modules involves batched matrix multiplications.
  创建多头注意力模块的更有效方式涉及批量矩阵乘法。

Implementing a GPT model from Scratch To Generate Text
从头实现一个 GPT 模型以生成文本

Thls chapter covers 本章涵盖

• Coding a GPT-like large language model (LLM) that can be trained to generate human-like text
  编码一个类似 GPT 的大型语言模型（LLM）,可以训练生成人类文本
• Normalizing layer activations to stabilize neural network training
  稳定神经网络训练的正则化层激活
• Adding shortcut connections in deep neural networks to train models more effectively
  在深度神经网络中添加快捷连接以更有效地训练模型
• Implementing transformer blocks to create GPT models of various sizes
  实现 transformer 模块来创建不同规模的 GPT 模型
• Computing the number of parameters and storage requirements of GPT models
  计算 GPT 模型的参数数量和存储需求

4.1 Coding an LLM architecture
4.1 编码 LLM 架构

GPT-2 VERSUS GPT-3 GPT-2 与 GPT-3

GPT_CONFIG_124M = {
    "vocab_size": 50257, # Vocabulary size
    "context_length": 1024, # Context length
    "emb_dim": 768, # Embedding dimension
    "n_heads": 12, # Number of attention heads
    "n_layers": 12, # Number of layers
    "drop_rate": 0.1, # Dropout rate
    "qkv_bias": False # Query-Key-Value bias
}

• "vocab_size" refers to a vocabulary of 50,257 words, as used by the BPE tokenizer from chapter 2.
  "vocab_size"指的是 50,257 个单词的词汇表,这是在第 2 章中使用 BPE 分词器时采用的。
• "context_length" denotes the maximum number of input tokens the model can handle, via the positional embeddings discussed in chapter 2 .
  "context_length"表示模型可处理的最大输入令牌数量,这是通过第 2 章讨论的位置嵌入实现的。
• "emb_dim" represents the embedding size, transforming each token into a 768-dimensional vector.
  "emb_dim"表示嵌入大小,将每个令牌转换为 768 维向量。
• "n_heads" indicates the count of attention heads in the multi-head attention mechanism, as implemented in chapter 3.
  "n_heads"表示在第 3 章中实现的多头注意力机制中的注意力头的数量。
• "n_layers" specifies the number of transformer blocks in the model, which will be elaborated on in upcoming sections.
  "n_layers"指定模型中变压器块的数量，这将在后续部分中详细解释。
• "drop_rate" indicates the intensity of the dropout mechanism (0.1 implies a drop of hidden units) to prevent overfitting, as covered in chapter 3 .
  "drop_rate"表示 dropout 机制的强度(0.1 意味着 隐藏单元的下降),以防止过拟合,如第 3 章所述。
• "qkv_bias" determines whether to include a bias vector in the Linear layers of the multi-head attention for query, key, and value computations. We will initially disable this, following the norms of modern LLMs, but will revisit it in chapter 6 when we load pretrained GPT-2 weights from OpenAI into our model.
  "qkv_bias"决定是否在多头注意力的查询、键和值计算的线性层中包含偏置向量。我们最初会禁用此功能,遵循现代LLMs的规范,但在第 6 章加载来自 OpenAI 的预训练 GPT-2 权重到我们的模型时会重新审视它。

import torch
import torch.nn as nn
class DummyGPTModel(nn.Module):
    def __init__(self, cfg):
        super().__init__()
        self.tok_emb = nn.Embedding(cfg["vocab_size"], cfg["emb_dim"])
        self.pos_emb = nn.Embedding(cfg["context_length"], cfg["emb_dim"])
        self.drop_emb = nn.Dropout(cfg["drop_rate"])
        self.trf_blocks = nn.Sequential(
            *[DummyTransformerBlock(cfg) for _ in range(cfg["n_layers"])]) #A
        self.final_norm = DummyLayerNorm(cfg["emb_dim"]) #B
        self.out_head = nn.Linear(
            cfg["emb_dim"], cfg["vocab_size"], bias=False
        )
    def forward(self, in_idx):
        batch_size, seq_len = in_idx.shape
        tok_embeds = self.tok_emb(in_idx)
        pos_embeds = self.pos_emb(torch.arange(seq_len, device=in_idx.device))
        x = tok_embeds + pos_embeds
        x = self.drop_emb(x)
        x = self.trf_blocks(x)
        x = self.final_norm(x)
        logits = self.out_head(x)
        return logits
class DummyTransformerBlock(nn.Module): #C
    def __init__(self, cfg):
        super().__init__()
    def forward(self, x): #D
        return x
class DummyLayerNorm(nn.Module): #E
    def __init__(self, normalized_shape, eps=1e-5): #F
        super().__init__()
    def forward(self, x):
        return x

import tiktoken
tokenizer = tiktoken.get_encoding("gpt2")
batch = []
txt1 = "Every effort moves you"
txt2 = "Every day holds a"
batch.append(torch.tensor(tokenizer.encode(txt1)))
batch.append(torch.tensor(tokenizer.encode(txt2)))
batch = torch.stack(batch, dim=0)
print(batch)

tensor([[ 6109, 3626, 6100, 345], #A
    [ 6109, 1110, 6622, 257]])

torch.manual_seed(123)
model = DummyGPTModel(GPT_CONFIG_124M)
logits = model(batch)
print("Output shape:", logits.shape)
print(logits)

Output shape: torch.Size([2, 4, 50257])
tensor([[[-1.2034, 0.3201, -0.7130, ..., -1.5548, -0.2390, -0.4667],
        [-0.1192, 0.4539, -0.4432, ..., 0.2392, 1.3469, 1.2430],
        [ 0.5307, 1.6720, -0.4695, ..., 1.1966, 0.0111, 0.5835],
        [ 0.0139, 1.6755, -0.3388, ..., 1.1586, -0.0435, -1.0400]],
        [[-1.0908, 0.1798, -0.9484, ..., -1.6047, 0.2439, -0.4530],
        [-0.7860, 0.5581, -0.0610, ..., 0.4835, -0.0077, 1.6621],
        [ 0.3567, 1.2698, -0.6398, ..., -0.0162, -0.1296, 0.3717],
        [-0.2407, -0.7349, -0.5102, ..., 2.0057, -0.3694, 0.1814]]],
        grad_fn=<UnsafeViewBackward0>)

4.2 Normalizing activations with layer normalization
使用层归一化对激活值进行归一化

torch.manual_seed(123)
batch_example = torch.randn(2, 5) #A
layer = nn.Sequential(nn.Linear(5, 6), nn.ReLU())
out = layer(batch_example)
print(out)

tensor([[0.2260, 0.3470, 0.0000, 0.2216, 0.0000, 0.0000],
    [0.2133, 0.2394, 0.0000, 0.5198, 0.3297, 0.0000]],
    grad_fn=<ReluBackward0>)

mean = out.mean(dim=-1, keepdim=True)
var = out.var(dim=-1, keepdim=True)
print("Mean:\n", mean)
print("Variance:\n", var)

Mean:
    tensor([[0.1324],
        [0.2170]], grad_fn=<MeanBackward1>)
Variance:
    tensor([[0.0231],
        [0.0398]], grad_fn=<VarBackward0>)

out_norm = (out - mean) / torch.sqrt(var)
mean = out_norm.mean(dim=-1, keepdim=True)
var = out_norm.var(dim=-1, keepdim=True)
print("Normalized layer outputs:\n", out_norm)
print("Mean:\n", mean)
print("Variance:\n", var)

Normalized layer outputs:
    tensor([[ 0.6159, 1.4126, -0.8719, 0.5872, -0.8719, -0.8719],
        [-0.0189, 0.1121, -1.0876, 1.5173, 0.5647, -1.0876]],
        grad_fn=<DivBackward0>)
Mean:
    tensor([[2.9802e-08],
        [3.9736e-08]], grad_fn=<MeanBackward1>)
Variance:
    tensor([[1.],
        [1.]], grad_fn=<VarBackward0>)

torch.set_printoptions(sci_mode=False)
print("Mean:\n", mean)
print("Variance:\n", var)
Mean:
    tensor([[ 0.0000],
        [ 0.0000]], grad_fn=<MeanBackward1>)
Variance:
    tensor([[1.],
        [1.]], grad_fn=<VarBackward0>)

class LayerNorm(nn.Module):
    def __init__(self, emb_dim):
        super().__init__()
        self.eps = 1e-5
        self.scale = nn.Parameter(torch.ones(emb_dim))
        self.shift = nn.Parameter(torch.zeros(emb_dim))
    def forward(self, x):
        mean = x.mean(dim=-1, keepdim=True)
        var = x.var(dim=-1, keepdim=True, unbiased=False)
        norm_x = (x - mean) / torch.sqrt(var + self.eps)
        return self.scale * norm_x + self.shift

BIASED VARIANCE 偏差方差

ln = LayerNorm(emb_dim=5)
out_ln = ln(batch_example)
mean = out_ln.mean(dim=-1, keepdim=True)
var = out_ln.var(dim=-1, unbiased=False, keepdim=True)
print("Mean:\n", mean)
print("Variance:\n", var)

Mean:
    tensor([[ -0.0000],
        [ 0.0000]], grad_fn=<MeanBackward1>)
Variance:
    tensor([[1.0000],
        [1.0000]], grad_fn=<VarBackward0>)

LAYER NORMALIZATION VERSUS BATCH NORMALIZATION
层规范化与批量规范化

4.3 Implementing a feed forward network with GELU activations
4.3 使用 GELU 激活函数实现前馈网络

Listing 4.3 An implementation of the GELU activation function
代码示例 4.3 GELU 激活函数的实现

class GELU(nn.Module):
    def __init__(self):
        super().__init__()
    def forward(self, x):
        return 0.5 * x * (1 + torch.tanh(
            torch.sqrt(torch.tensor(2.0 / torch.pi)) *
            (x + 0.044715 * torch.pow(x, 3))
        ))

import matplotlib.pyplot as plt
gelu, relu = GELU(), nn.ReLU()
x = torch.linspace(-3, 3, 100) #A
y_gelu, y_relu = gelu(x), relu(x)
plt.figure(figsize=(8, 3))
for i, (y, label) in enumerate(zip([y_gelu, y_relu], ["GELU", "ReLU"]), 1):
    plt.subplot(1, 2, i)
    plt.plot(x, y)
    plt.title(f"{label} activation function")
    plt.xlabel("x")
    plt.ylabel(f"{label}(x)")
    plt.grid(True)
plt.tight_layout()
plt.show()

Listing 4.4 A feed forward neural network module
列举 4.4 前馈神经网络模块

class FeedForward(nn.Module):
    def __init__(self, cfg):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Linear(cfg["emb_dim"], 4 * cfg["emb_dim"]),
            GELU(),
            nn.Linear(4 * cfg["emb_dim"], cfg["emb_dim"]),
        )

The first linear layer
第一个线性层

increases the embedding dimension by a factor of 4

The three values represent the batch size (2),
批次大小(2)

number of tokens (3), and embedding size (768)

ffn = FeedForward(GPT_CONFIG_124M)
x = torch.rand(2, 3, 768) #A
out = ffn(x)
print(out.shape)

torch.Size([2, 3, 768])

4.4 Adding shortcut connections
4.4 添加快捷连接

layer_sizes = [3, 3, 3, 3, 3, 1]
sample_input = torch.tensor([[1., 0., -1.]])
torch.manual_seed(123) # specify random seed for the initial weights for reproducibility
model_without_shortcut = ExampleDeepNeuralNetwork(
    layer_sizes, use_shortcut=False
)

def print_gradients(model, x):
    # Forward pass
    output = model(x)
    target = torch.tensor([[0.]])
    # Calculate loss based on how close the target
    # and output are
    loss = nn.MSELoss()
    loss = loss(output, target)
    # Backward pass to calculate the gradients
    loss.backward()
    for name, param in model.named_parameters():
        if 'weight' in name:
            # Print the mean absolute gradient of the weights
            print(f"{name} has gradient mean of {param.grad.abs().mean().item()}")

torch.manual_seed(123)
model_with_shortcut = ExampleDeepNeuralNetwork(
    layer_sizes, use_shortcut=True
)
print_gradients(model_with_shortcut, sample_input)

4.5 Connecting attention and linear layers in a transformer block
4.5 在变换器块中连接注意力和线性层

Listing 4.6 The transformer block component of GPT
4.6 GPT 变换器块组件

from previous_chapters import MultiHeadAttention
class TransformerBlock(nn.Module):
    def __init__(self, cfg):
        super().__init__()
        self.att = MultiHeadAttention(
            d_in=cfg["emb_dim"],
            d_out=cfg["emb_dim"],
            context_length=cfg["context_length"],
            num_heads=cfg["n_heads"],
            dropout=cfg["drop_rate"],
            qkv_bias=cfg["qkv_bias"])
        self.ff = FeedForward(cfg)
        self.norm1 = LayerNorm(cfg["emb_dim"])
        self.norm2 = LayerNorm(cfg["emb_dim"])
        self.drop_shortcut = nn.Dropout(cfg["drop_rate"])
    def forward(self, x):
        #A
    shortcut = x
    x = self.norm1(x)
    x = self.att(x)
    x = self.drop_shortcut(x)
    x = x + shortcut # Add the original input back
    shortcut = x
\#B
    x = self.norm2(x)

x = self.ff(x)
x = self.drop_shortcut(x)
x = x + shortcut
#C
return x

torch.manual_seed(123)
x = torch.rand(2, 4, 768)
#A
block = TransformerBlock(GPT_CONFIG_124M)
output = block(x)
print("Input shape:", x.shape)
print("Output shape:", output.shape)

Input shape: torch.Size([2, 4, 768])
Output shape: torch.Size([2, 4, 768])

4.6 Coding the GPT model
4.6 编码 GPT 模型

Listing 4.7 The GPT model architecture implementation
4.7 章 GPT 模型架构实现

class GPTModel(nn.Module):
    def __init__(self, cfg):
        super().__init__()
        self.tok_emb = nn.Embedding(cfg["vocab_size"], cfg["emb_dim"])
        self.pos_emb = nn.Embedding(cfg["context_length"], cfg["emb_dim"])
        self.drop_emb = nn.Dropout(cfg["drop_rate"])
        self.trf_blocks = nn.Sequential(
            *[TransformerBlock(cfg) for _ in range(cfg["n_layers"])])
        self.final_norm = LayerNorm(cfg["emb_dim"])
        self.out_head = nn.Linear(
            cfg["emb_dim"], cfg["vocab_size"], bias=False
        )
    def forward(self, in_idx):
        batch_size, seq_len = in_idx.shape
        tok_embeds = self.tok_emb(in_idx)
                    #A
    pos_embeds = self.pos_emb(torch.arange(seq_len, device=in_idx.device))
    x = tok_embeds + pos_embeds
    x = self.drop_emb(x)
    x = self.trf_blocks(x)
    x = self.final_norm(x)
    logits = self.out_head(x)
    return logits

torch.manual_seed(123)
model = GPTModel(GPT_CONFIG_124M)
out = model(batch)
print("Input batch:\n", batch)
print("\nOutput shape:", out.shape)
print(out)

Input batch:
tensor([[ 6109, 3626, 6100, 345], \# token IDs of text 1
    \([\) 6109, 1110, 6622, 257]]) \# token IDs of text 2

total_params = sum(p.numel() for p in model.parameters())

EXERCISE 4.1 NUMBER OF PARAMETERS IN FEED FORWARD AND ATTENTION MODULES
第 4.1 练习 前馈和注意力模块中的参数数量

total_size_bytes = total_params * 4 #A
total_size_mb = total_size_bytes / (1024 * 1024) #B
print(f"Total size of the model: {total_size_mb:.2f} MB")

EXERCISE 4.2 INITIALIZING LARGER GPT MODELS
练习 4.2 初始化更大的 GPT 模型

4.7 Generating text 4.7 生成文本

The input context 输入内容

token 代币

6th iteration: 第 6 次迭代：

Listing 4.8 A function for the GPT model to generate text
代码清单 4.8 GPT 模型生成文本的函数

def generate_text_simple(model, idx, max_new_tokens, context_size): #A
    for _ in range(max_new_tokens):
        idx_cond = idx[:, -context_size:] #B
        with torch.no_grad():
            logits = model(idx_cond)
        logits = logits[:, -1, :] #C
        probas = torch.softmax(logits, dim=-1) #D
        idx_next = torch.argmax(probas, dim=-1, keepdim=True) #E
        idx = torch.cat((idx, idx_next), dim=1) #F

start_context = "Hello, I am"
encoded = tokenizer.encode(start_context)
print("encoded:", encoded)
encoded_tensor = torch.tensor(encoded).unsqueeze(0) #A
print("encoded_tensor.shape:", encoded_tensor.shape)
#A add batch dimension

model.eval()
\#A
out = generate_text_simple(
    model=model,
    idx=encoded_tensor,
    max_new_tokens=6,
    context_size=GPT_CONFIG_124M["context_length"]
)
print("Output:", out)
print("Output length:", len(out[0]))

decoded_text = tokenizer.decode(out.squeeze(0).tolist())
print(decoded_text)

Hello, I am Featureiman Byeswickattribute argue
你好，我是 Featureiman Byeswickattribute 争论

EXERCISE 4.3 USING SEPARATE DROPOUT PARAMETERS
练习 4.3 使用独立的 dropout 参数

4.8 Summary 4.8 总结

• Layer normalization stabilizes training by ensuring that each layer's outputs have a consistent mean and variance.
  层归一化通过确保每一层的输出具有一致的均值和方差来稳定训练。
• Shortcut connections are connections that skip one or more layers by feeding the output of one layer directly to a deeper layer, which helps mitigate the vanishing gradient problem when training deep neural networks, such as LLMs.
  快捷连接是跳过一个或多个层的连接,通过将一个层的输出直接馈入更深层的层来实现,这有助于缓解训练深度神经网络(如LLMs)时梯度消失的问题。
• Transformer blocks are a core structural component of GPT models, combining masked multi-head attention modules with fully connected feed-forward networks that use the GELU activation function.
  变换器块是 GPT 模型的核心结构组件,结合了遮掩多头注意力模块和使用 GELU 激活函数的完全连接前馈网络。
• GPT models are LLMs with many repeated transformer blocks that have millions to billions of parameters.
  GPT 模型由数以百万到数十亿参数的多个重复的 Transformer 块组成。
• GPT models come in various sizes, for example, 124, 345, 762, and 1542 million parameters, which we can implement with the same GPrModel Python class.
  GPT 模型有不同规模,例如 124、345、762 和 1542 百万参数,这些都可以使用同一个 GPrModel Python 类来实现。
• The text generation capability of a GPT-like LLM involves decoding output tensors into human-readable text by sequentially predicting one token at a time based on a given input context.
  类似 GPT 的文本生成能力涉及通过基于给定输入上下文逐个预测词元的方式将输出张量解码为可读文本。
• Without training, a GPT model generates incoherent text, which underscores the importance of model training for coherent text generation, which is the topic of subsequent chapters.
  没有训练,GPT 模型会生成不连贯的文本,这突出了模型训练对于生成连贯文本的重要性,这是后续章节的主题。

5

Pretraining on Unlabeled Data
无标签数据的预训练

This chapter covers 本章涵盖

• Computing the training and validation set losses to assess the quality of LLMgenerated text during training
  计算训练集和验证集损失以评估训练期间 LLM 生成文本的质量
• Implementing a training function and pretraining the LLM
  实施培训功能并预先训练LLM
• Saving and loading model weights to continue training an LLM
  保存和加载模型权重以继续训练一个LLM
• Loading pretrained weights from OpenAl
  从 OpenAI 加载预训练权重

WEIGHT PARAMETERS 重量参数

5.1 Evaluating generative text models
5.1 评估生成式文本模型

5.1.1 Using GPT to generate text
使用 GPT 生成文本

import torch
from chapter04 import GPTModel
GPT_CONFIG_124M = {
    "vocab_size": 50257,
    "emb_dim": 768,
    "n_heads": 12,
    "n_layers": 12,
    "drop_rate": 0.1,
    "qkv_bias": False
}
torch.manual seed(123)
model = GPTModel(GPT_CONFIG_124M)
model.eval()

import tiktoken
from chapter04 import generate_text_simple
def text_to_token_ids(text, tokenizer):
    encoded = tokenizer.encode(text, allowed_special={'<|endoftext|>'})
    encoded_tensor = torch.tensor(encoded).unsqueeze(0) # add batch dimension
    return encoded_tensor
def token_ids_to_text(token_ids, tokenizer):
    flat = token_ids.squeeze(0) # remove batch dimension
    return tokenizer.decode(flat.tolist())
start_context = "Every effort moves you"
tokenizer = tiktoken.get_encoding("gpt2")
token_ids = generate_text_simple(
    model=model,
    idx=text_to_token_ids(start_context, tokenizer),
    max_new_tokens=10,
    context_size=GPT_CONFIG_124M["context_length"]
)
print("Output text:\n", token_ids_to_text(token_ids, tokenizer))

Output text: 输出文本: 简化中文: 这是一个测试

5.1.2 Calculating the text generation loss
5.1.2 计算文本生成损失

inputs = torch.tensor([[16833, 3626, 6100], # ["every effort moves",
    [40, 1107, 588]]) # "I really like"]
Matching these inputs, the `targets` contain the token IDs we aim for the model to
produce:
targets = torch.tensor([[3626, 6100, 345 ], # [" effort moves you",
    [107, 588, 11311]]) # " really like chocolate"]

with torch.no_grad():
    #A
    logits = model(inputs)
probas = torch.softmax(logits, dim=-1) # Probability of each token in vocabulary
print(probas.shape)

torch.Size([2, 3, 50257])

token_ids = torch.argmax(probas, dim=-1, keepdim=True)
print("Token IDs:\n", token_ids)

Token IDs:
tensor([[[16657], # First batch
    [ 339],
    [42826]],
    [[49906], # Second batch
    [29669],
    [41751]]])

print(f"Targets batch 1: {token_ids_to_text(targets[0], tokenizer)}")
print(f"Outputs batch 1: {token_ids_to_text(token_ids[0].flatten(), tokenizer)}")
When we decode these tokens, we find that these output tokens are quite different from
the target tokens we want the model to generate:
Targets batch 1: effort moves you
Outputs batch 1: Armed heNetflix

text_idx = 0
target_probas_1 = probas[text_idx, [0, 1, 2], targets[text_idx]]
print("Text 1:", target_probas_1)
text_idx = 1
target_probas_2 = probas[text_idx, [0, 1, 2], targets[text_idx]]
print("Text 2:", target_probas_2)

Text 1: tensor([7.4541e-05, 3.1061e-05, 1.1563e-05])
Text 2: tensor([1.0337e-05, 5.6776e-05, 4.7559e-06])

BACKPROPAGATION 反向传播

log_probas = torch.log(torch.cat((target_probas_1, target_probas_2)))
print(log_probas)

tensor([ -9.5042, -10.3796, -11.3677, -11.4798, -9.7764, -12.2561])

avg_log_probas = torch.mean(log_probas)
print(avg_log_probas)

neg_avg_log_probas = avg_log_probas * -1
print(neg_avg_log_probas)

CROSS ENTROPY LOSS 交叉熵损失

print("Logits shape:", logits.shape)
print("Targets shape:", targets.shape)
The resulting shapes are as follows:
Logits shape: torch.Size([2, 3, 50257])
Targets shape: torch.Size([2, 3])

logits_flat = logits.flatten(0, 1)
targets_flat = targets.flatten()
print("Flattened logits:", logits_flat.shape)
print("Flattened targets:", targets_flat.shape)

loss = torch.nn.functional.cross_entropy(logits_flat, targets_flat)
print(loss)

tensor(10.7940)

PERPLEXITY 疑惑

5.1.3 Calculating the training and validation set losses
5.1.3 计算训练集和验证集损失

THE COST OF PRETRAINING LLMS
预训练的成本 LLMS

file_path = "the-verdict.txt"
with open(file_path, "r", encoding="utf-8") as file:
    text_data = file.read()

total_characters = len(text_data)
total_tokens = len(tokenizer.encode(text_data))
print("Characters:", total_characters)
print("Tokens:", total_tokens)

• An example using stride equal to the context length (here: 4 ) as shown below:
  步长等于上下文长度(此处为 4)的示例如下:
  

  

TRAINING WITH VARIABLE LENGTHS
变长度训练

train_ratio = 0.90
split_idx = int(train_ratio * len(text_data))
train_data = text_data[:split_idx]
val_data = text_data[split_idx:]

from chapter02 import create_dataloader_v1
torch.manual_seed(123)
train_loader = create_dataloader_v1(
    train_data,
    batch_size=2,
    max_length=GPT_CONFIG_124M["context_length"],
    stride=GPT_CONFIG_124M["context_length"],
    drop_last=True,
    shuffle=True,
    num_workers=0
)
val_loader = create_dataloader_v1(
    val_data,
    batch_size=2,
    max_length=GPT_CONFIG_124M["context_length"],
    stride=GPT_CONFIG_124M["context_length"],
    drop_last=False,
    shuffle=False,
    num_workers=0
)

print("Train loader:")
for x, y in train_loader:
    print(x.shape, y.shape)
print("\nValidation loader:")
for x, y in val_loader:
    print(x.shape, y.shape)

Train loader:
torch.Size([2, 256]) torch.Size([2, 256])
torch.Size([2, 256]) torch.Size([2, 256])
torch.Size([2, 256]) torch.Size([2, 256])
torch.Size([2, 256]) torch.Size([2, 256])
torch.Size([2, 256]) torch.Size([2, 256])
torch.Size([2, 256]) torch.Size([2, 256])
torch.Size([2, 256]) torch.Size([2, 256])
torch.Size([2, 256]) torch.Size([2, 256])
torch.Size([2, 256]) torch.Size([2, 256])
Validation loader:
torch.Size([2, 256]) torch.Size([2, 256])

def calc_loss_batch(input_batch, target_batch, model, device):
    input_batch, target_batch = input_batch.to(device), target_batch.to(device) #A
    logits = model(input_batch)
    loss = torch.nn.functional.cross_entropy(
        logits.flatten(0, 1), target_batch.flatten()
    )
    return loss

Listing 5.2 Function to compute the training and validation loss
代码清单 5.2 计算训练和验证损失的函数

def calc_loss_loader(data_loader, model, device, num_batches=None):
    total_loss = 0.
    if len(data_loader) == 0:
        return float("nan")
    elif num_batches is None:
        num_batches = len(data_loader) #A
    else:
        num_batches = min(num_batches, len(data_loader)) #B
    for i, (input_batch, target_batch) in enumerate(data_loader):
        if i < num_batches:
            loss = calc_loss_batch(input_batch, target_batch, model, device)
        total_loss += loss.item()
            #C
        else:
            break
    return total_loss / num_batches #D

device = torch.device("cuda" if torch.cuda.is_available() else "cpu") #A
model.to(device)
with torch.no_grad():
    #B
    train_loss = calc_loss_loader(train_loader, model, device) #C
    val_loss = calc_loss_loader(val_loader, model, device)
print("Training loss:", train_loss)
print("Validation loss:", val_loss)

5.2 Training an LLM
5.2 训练一个LLM

Listing 5.3 The main function for pretraining LLMs
5.3 节 预训练的主要函数LLMs

def evaluate_model(model, train_loader, val_loader, device, eval_iter):
    model.eval() #A
    with torch.no_grad(): #B
        train_loss = calc_loss_loader(train_loader, model, device,
num_batches=eval_iter)
        val_loss = calc_loss_loader(val_loader, model, device, num_batches=eval_iter)
    model.train()
    return train_loss, val_loss

def generate_and_print_sample(model, tokenizer, device, start_context):
    model.eval()
    context_size = model.pos_emb.weight.shape[0]
    encoded = text_to_token_ids(start_context, tokenizer).to(device)
    with torch.no_grad():
        token_ids = generate_text_simple(
            model=model, idx=encoded,
            max_new_tokens=50, context_size=context_size
        )
        decoded_text = token_ids_to_text(token_ids, tokenizer)
        print(decoded_text.replace("\n", " ")) # Compact print format
    model.train()

torch.manual_seed(123)
model = GPTModel(GPT_CONFIG_124M)
model.to(device)
optimizer = torch.optim.AdamW(model.parameters(), lr=0.0004, weight_decay=0.1) #A
num_epochs = 10
train_losses, val_losses, tokens_seen = train_model_simple(
    model, train_loader, val_loader, optimizer, device,
    num_epochs=num_epochs, eval_freq=5, eval_iter=1,
    start_context="Every effort moves you", tokenizer=tokenizer
)

Ep 1 (Step 000000): Train loss 9.781, Val loss 9.933
Ep 1 (Step 000005): Train loss 8.111, Val loss 8.339
Every effort moves you,,,,,,,,,,,,
Ep 2 (Step 000010): Train loss 6.661, Val loss 7.048
Ep 2 (Step 000015): Train loss 5.961, Val loss 6.616
Every effort moves you, and, and, and, and, and, and, and, and, and, and, and, and, and,
and, and, and, and, and, and, and, and, and, and, and,
[...] Results are truncated to save space
    #A
Ep 9 (Step 000080): Train loss 0.541, Val loss 6.393
Every effort moves you?" "Yes--quite insensible to the irony. She wanted him
vindicated--and by me!" He laughed again, and threw back the window-curtains, I had the
donkey. "There were days when I
Ep 10 (Step 000085): Train loss 0.391, Val loss 6.452
Every effort moves you know," was one of the axioms he laid down across the Sevres and
silver of an exquisitely appointed luncheon-table, when, on a later day, I had again run
over from Monte Carlo; and Mrs. Gis
#A Intermediate results removed to save space

import matplotlib.pyplot as plt
def plot_losses(epochs_seen, tokens_seen, train_losses, val_losses):
    fig, ax1 = plt.subplots(figsize=(5, 3))
    ax1.plot(epochs_seen, train_losses, label="Training loss")
    ax1.plot(epochs_seen, val_losses, linestyle="-.", label="Validation loss")
    ax1.set_xlabel("Epochs")
    ax1.set_ylabel("Loss")
    ax1.legend(loc="upper right")
    ax2 = ax1.twiny() #A
    ax2.plot(tokens_seen, train_losses, alpha=0) #B
    ax2.set_xlabel("Tokens seen")
    fig.tight_layout()
    plt.show()
epochs_tensor = torch.linspace(0, num_epochs, len(train_losses))
plot_losses(epochs_tensor, tokens_seen, train_losses, val_losses)

5.3 Decoding strategies to control randomness
5.3 控制随机性的解码策略

model.to("cpu")
model.eval()

tokenizer = tiktoken.get_encoding("gpt2")
token_ids = generate_text_simple(
    model=model,
    idx=text_to_token_ids("Every effort moves you", tokenizer),
    max_new_tokens=25,
    context_size=GPT_CONFIG_124M["context_length"]
)
print("Output text:\n", token_ids_to_text(token_ids, tokenizer))

5.3.1 Temperature scaling
5.3.1 温度缩放

vocab = {
    "closer": 0,
    "every": 1,
    "effort": 2,
    "forward": 3,
    "inches": 4,
    "moves": 5,
    "pizza": 6,
    "toward": 7,
    "you": 8,
}
inverse_vocab = {v: k for k, v in vocab.items()}

next_token_logits = torch.tensor(
    [4.51, 0.89, -1.90, 6.75, 1.63, -1.62, -1.89, 6.28, 1.79]
)

probas = torch.softmax(next_token_logits, dim=0)
next_token_id = torch.argmax(probas).item()
print(inverse_vocab[next_token_id])

torch.manual seed(123)
next_token_id = torch.multinomial(probas, num_samples=1).item()
print(inverse_vocab[next_token_id])

def print_sampled_tokens(probas):
    torch.manual_seed(123)
    sample = [torch.multinomial(probas, num_samples=1).item() for i in range(1_000)]
    sampled_ids = torch.bincount(torch.tensor(sample))
    for i, freq in enumerate(sampled_ids):
        print(f"{freq} x {inverse_vocab[i]}")
print_sampled_tokens(probas)

As we can see based on the output, the word "forward" is sampled most of the time (582 out of 1000 times), but other tokens such as "closer", "inches", and "toward" will also be sampled some of the time. This means that if we replaced the argmax function with the multinomial function inside the generate_and_print_sample function, the LLM would sometimes generate texts such as "every effort moves you toward", "every effort moves you inches", and "every effort moves you closer" instead of "every effort moves you forward".

We can further control the distribution and selection process via a concept called temperature scaling, where temperature scaling is just a fancy description for dividing the logits by a number greater than 0 :

\#A Original, lower, and higher confidence

The resulting plot is shown in Figure 5.14.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-187.jpg?height=695&width=1210&top_left_y=204&top_left_x=319)

Figure 5.14 A temperature of 1 represents the unscaled probability scores for each token in the vocabulary. Decreasing the temperature to 0.1 sharpens the distribution, so the most likely token (here "forward") will have an even higher probability score. Vice versa, increasing the temperature to 5 makes the distribution more uniform.

A temperature of 1 divides the logits by 1 before passing them to the softmax function to compute the probability scores. In other words, using a temperature of 1 is the same as not using any temperature scaling. In this case, the tokens are selected with a probability equal to the original softmax probability scores via the multinomial sampling function in PyTorch.

Also, as we can see in Figure 5.14, applying very small temperatures, such as 0.1 , will result in sharper distributions such that the behavior of the multinomial function selects the most likely token (here: "forward") almost 100\% of the time, approaching the behavior of the argmax function. Vice versa, a temperature of 5 results in a more uniform distribution where other tokens are selected more often. This can add more variety to the generated texts but also more often results in nonsensical text. For example, using the temperature of 5 results in texts such as "every effort moves you pizza" about \(4 \%\) of the time.

\section*{EXERCISE 5.1}

Use the print_sampled_tokens function to print the sampling frequencies of the softmax probabilities scaled with the temperatures shown in Figure 5.13. How often is the word "pizza" sampled in each case? Can you think of a faster and more accurate way to determine how often the word "pizza" is sampled?

\subsection*{5.3.2 Top-k sampling}

In the previous section, we implemented a probabilistic sampling approach coupled with temperature scaling to increase the diversity of the outputs. We saw that higher temperature values result in more uniformly distributed next-token probabilities, which result in more diverse outputs as it reduces the likelihood of the model repeatedly selecting the most probable token. This method allows for exploring less likely but potentially more interesting and creative paths in the generation process. However, One downside of this approach is that it sometimes leads to grammatically incorrect or completely nonsensical outputs such as "every effort moves you pizza".

In this section, we introduce another concept called top-k sampling, which, when combined with probabilistic sampling and temperature scaling, can improve the text generation results.

In top-k sampling, we can restrict the sampled tokens to the top- k most likely tokens and exclude all other tokens from the selection process by masking their probability scores, as illustrated in Figure 5.15.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-188.jpg?height=651&width=1252&top_left_y=912&top_left_x=298)

By assigning zero probabilities to the non-top-k positions, we ensure that the next token is always sampled from a top-k position

Figure 5.15 Using top-k sampling with \(k=3\), we focus on the 3 tokens associated with the highest logits and mask out all other tokens with negative infinity (-inf) before applying the softmax function. This results in a probability distribution with a probability value 0 assigned to all non-top-k tokens.

The approach outlined in Figure 5.15 replaces all non-selected logits with negative infinity value (-inf), such that when computing the softmax values, the probability scores of the non-top-k tokens are 0 , and the remaining probabilities sum up to 1 . (Careful readers may remember this masking trick from the causal attention module we implemented in chapter 3 in section 3.5.1 Applying a causal attention mask.)

In code, we can implement the top-k procedure outlined in Figure 5.15 as follows, starting with the selection of the tokens with the largest logit values:

Subsequently, we apply PyTorch's where function to set the logit values of tokens that are below the lowest logit value within our top-3 selection to negative infinity (-inf).

\#A Identifies logits less than the minimum in the top 3

\#B Assigns -inf to these lower logits

\#C Retains the original logits for all other tokens

The resulting logits for the next token in the 9-token vocabulary are as follows:

tensor([4.5100, -inf, -inf, 6.7500, -inf, -inf, -inf, 6.2800, -inf])

Lastly, let's apply the softmax function to turn these into next-token probabilities:

As we can see, the result of this top-3 approach are 3 non-zero probability scores:

We can now apply the temperature scaling and multinomial function for probabilistic sampling introduced in the previous section to select the next token among these 3 nonzero probability scores to generate the next token. We do this in the next section by modifying the text generation function.

\subsection*{5.3.3 Modifying the text generation function}

The previous two subsections introduced two concepts to increase the diversity of LLMgenerated text: temperature sampling and top-k sampling. In this section, we combine and add these concepts to modify the generate_simple function we used to generate text via the LLM earlier, creating a new generate function:

Listing 5.4 A modified text generation function with more diversity

if temperature > 0.0:

\#C

logits = logits / temperature

probs = torch.softmax(logits, dim=-1)

idx_next = torch.multinomial(probs, num_samples=1)

else:

\#D

idx_next = torch.argmax(logits, dim=-1, keepdim=True)

if idx_next == eos_id:

\(\# \mathrm{E}\)

break

idx = torch.cat((idx, idx_next), dim=1)

return idx

\#A For-loop is the same as before: Get logits, and only focus on last time step \#B In this new section, we filter logits with top_k sampling \#C This is the new section where we apply temperature scaling \#D Carry out greedy next-token selection as before when temperature scaling is disabled \#E Stop generating early if end-of-sequence token is encountered and eos_id is specified

Let's now see this new generate function in action:

The generated text is as follows:

As we can see, the generated text is very different from the one we previously generated via the generate_simple function at the beginning of section 5.3 ("Every effort moves you know," was one of the axioms he laid...!"), which was a memorized passage from the training set.

\section*{EXERCISE 5.2}

Play around with different temperatures and top-k settings. Based on your observations, can you think of applications where lower temperature and top-k settings are desired? Vice versa, can you think of applications where higher temperature and top-k settings are preferred? (It's recommended to also revisit this exercise at the end of the chapter after loading the pretrained weights from OpenAI.)

\section*{EXERCISE 5.3}

What are the different combinations of settings for the generate function to force deterministic behavior, that is, disabling the random sampling such that it always produces the same outputs similar to the generate_simple function?

So far, we covered how to pretrain LLMs and use them to generate text. The last two sections of this chapter will discuss how we save and load the trained LLM and how we load pretrained weights from OpenAI.

\subsection*{5.4 Loading and saving model weights in PyTorch}

In this chapter, we have discussed how to numerically evaluate the training progress and pretrain an LLM from scratch. Even though both the LLM and dataset were relatively small, this exercise showed that pretraining LLMs is computationally expensive. Thus, it is important to be able to save the LLM so that we don't have to rerun the training every time we want to use it in a new session.

As illustrated in the chapter overview in Figure 5.16, we cover how to save and load a pretrained model in this section. Then, in the upcoming section, we will load a more capable pretrained GPT model from OpenAI into our GPTModel instance.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-193.jpg?height=431&width=1237&top_left_y=666&top_left_x=315)

Figure 5.16 After training and inspecting the model, it is often helpful to save the model so that we can use or continue training it later, which is the topic of this section before we load the pretrained model weights from OpenAl in the final section of this chapter.

Fortunately, saving a PyTorch model is relatively straightforward. The recommended way is to save a model's so-called state_dict, a dictionary mapping each layer to its parameters, using the torch.save function as follows:

torch.save(model.state_dict(), "model.pth")

In the preceding code, "model.pth" is the filename where the state_dict is saved. The .pth extension is a convention for PyTorch files, though we could technically use any file extension.

Then, after saving the model weights via the state_dict, we can load the model weights into a new GPTModel model instance as follows:

As discussed in chapter 4, dropout helps prevent the model from overfitting to the training data by randomly "dropping out" of a layer's neurons during training. However, during inference, we don't want to randomly drop out any of the information the network has learned. Using model.eval() switches the model to evaluation mode for inference, disabling the dropout layers of the model.

If we plan to continue pretraining a model later, for example, using the train_model_simple function we defined earlier in this chapter, saving the optimizer state is also recommended.

Adaptive optimizers such as AdamW store additional parameters for each model weight. AdamW uses historical data to adjust learning rates for each model parameter dynamically. Without it, the optimizer resets, and the model may learn suboptimally or even fail to converge properly, which means that it will lose the ability to generate coherent text. Using torch.save, we can save both the model and optimizer state_dict contents as follows:

Then, we can restore the model and optimizer states as follows by first loading the saved data via torch.load and then using the load_state_dict method:

\section*{EXERCISE 5.4}

After saving the weights, load the model and optimizer in a new Python session or Jupyter notebook file and continue pretraining it for 1 more epoch using the train_model_simple function.

\subsection*{5.5 Loading pretrained weights from OpenAI}

Previously, for educational purposes, we trained a small GPT-2 model using a limited dataset comprising a short-story book. This approach allowed us to focus on the fundamentals without the need for extensive time and computational resources.

Fortunately, OpenAI openly shared the weights of their GPT-2 models, thus eliminating the need to invest tens to hundreds of thousands of dollars in retraining the model on a large corpus ourselves.

In the remainder of this section, we load these weights into our GPTModel class and use the model for text generation. Here, weights refer to the weight parameters that are stored in the .weight attributes of PyTorch's Linear and Embedding layers, for example. We accessed them earlier via model. parameters () when training the model.

In the next chapters, we will reuse these pretrained weights to finetune the model for a text classification task and follow instructions similar to ChatGPT.

Note that OpenAI originally saved the GPT-2 weights via TensorFlow, which we have to install to load the weights in Python. Moreover, the following code will use a progress bar tool called tqdm to track the download process, which we also have to install.

You can install these libraries by executing the following command in your terminal:

pip install tensorflow \(>=2.15 .0\) tqdm \(>=4.66\)

The download code is relatively long, mostly boilerplate, and not very interesting. Hence, instead of devoting precious space in this chapter to discussing Python code for fetching files from the internet, we download the gpt_download.py Python module directly from this chapter's online repository:

Next, after downloading this file to the local directory of your Python session, readers are encouraged to briefly inspect the contents of this file to ensure that it was saved correctly and contains valid Python code.

We can now import the download_and_load_gpt2 function from the gpt_download.py file as follows, which will load the GPT-2 architecture settings (settings) and weight parameters (params) into our Python session:

\section*{UPDATED DOWNLOAD INSTRUCTIONS}

If the download code does not work for you, it could be due to intermittent internet connection, server issues, or changes in how OpenAI shares the weights of the opensource GPT-2 model. In this case, please visit this chapter's online code repository at https://github.com/rasbt/LLMs-from-scratch for alternative and updated instructions, and please reach out via the Manning Forum for further questions.

After the execution of the previous code has been completed, let's inspect the contents of settings and params:

The contents are as follows:

Settings: \{'n_vocab': 50257, 'n_ctx': 1024, 'n_embd': 768, 'n_head': 12, 'n_layer': 12\} Parameter dictionary keys: dict_keys(['blocks', 'b', 'g', 'wpe', 'wte'])

Both settings and params are Python dictionaries. The settings dictionary stores the LLM architecture settings similarly to our manually defined GPT CONFIG 124 M settings. The params dictionary contains the actual weight tensors. Note that we only printed the dictionary keys because printing the weight contents would take up too much screen space, however, we can inspect these weight tensors by printing the whole dictionary via print (params) or by selecting individual tensors via the respective dictionary keys, for example, the embedding layer weights:

The weights of the token embedding layer are as follows:

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-197.jpg?height=48&width=837&top_left_y=438&top_left_x=260)
    [ 0.04034033 _. 0.08605453 0.00253983 0.04318958]

    ...
    [-0.04453601 _. 0.10435229 0.09783269 -0.06952604]
    [ 0.1860082 _.. -0.09625227 0.07847701 -0.02245961]
    [ 0.05135201 \ldots. 0.00704835
Token embedding weight tensor dimensions: (50257, 768)

First, we create a dictionary that lists the differences between the different GPT model
sizes, as explained in Figure 5.17:
model_configs = {
    "gpt2-small (124M)": {"emb_dim": 768, "n_layers": 12, "n_heads": 12},
    "gpt2-medium (355M)": {"emb_dim": 1024, "n_layers": 24, "n_heads": 16},
    "gpt2-large (774M)": {"emb_dim": 1280, "n_layers": 36, "n_heads": 20},
    "gpt2-xl (1558M)": {"emb_dim": 1600, "n_layers": 48, "n_heads": 25},
}
Suppose we are interested in loading the smallest model, "gpt2-small (124M)". We can use
the corresponding settings from the model_configs table able to update our full-length
GPT_CONFIG_124M we defined and used earlier throughout the chapter as follows:
model_name = "gpt2-small (124M)"
NEW_CONFIG = GPT_CONFIG_124M.copy()
NEW_CONFIG.update(model_configs[model_name])

NEW_CONFIG.update({"qkv_bias": True})
We can now use the updated NEW_CONFIG dictionary to initialize a new GPTModel instance:
gpt = GPTModel(NEW_CONFIG)
gpt.eval()

def assign(left, right):
    if left.shape != right.shape:
        raise ValueError(f"Shape mismatch. Left: {left.shape}, Right: {right.shape}")
    return torch.nn.Parameter(torch.tensor(right))

def load_weights_into_gpt(gpt, params):
    gpt.pos_emb.weight = assign(gpt.pos_emb.weight, params['wpe']) #A
    gpt.tok_emb.weight = assign(gpt.tok_emb.weight, params['wte'])

params["blocks"][b]["mlp"]["c_fc"]["w"].T)
    gpt.trf_blocks[b].ff.layers[0].bias = assign(
        gpt.trf_blocks[b].ff.layers[0].bias,
        params["blocks"][b]["mlp"]["c_fc"]["b"])
    gpt.trf_blocks[b].ff.layers[2].weight = assign(
        gpt.trf_blocks[b].ff.layers[2].weight,
        params["blocks"][b]["mlp"]["c_proj"]["w"].T)
    gpt.trf_blocks[b].ff.layers[2].bias = assign(
        gpt.trf_blocks[b].ff.layers[2].bias,
        params["blocks"][b]["mlp"]["c_proj"]["b"])
    gpt.trf_blocks[b].norm1.scale = assign(
        gpt.trf_blocks[b].norm1.scale,
        params["blocks"][b]["ln_1"]["g"])
    gpt.trf_blocks[b].norm1.shift = assign(
        gpt.trf_blocks[b].norm1.shift,
        params["blocks"][b]["ln_1"]["b"])
    gpt.trf_blocks[b].norm2.scale = assign(
        gpt.trf_blocks[b].norm2.scale,
        params["blocks"][b]["ln_2"]["g"])
    gpt.trf_blocks[b].norm2.shift = assign(
        gpt.trf_blocks[b].norm2.shift,
        params["blocks"][b]["ln_2"]["b"])
gpt.final_norm.scale = assign(gpt.final_norm.scale, params["g"])
gpt.final_norm.shift = assign(gpt.final_norm.shift, params["b"])
gpt.out_head.weight = assign(gpt.out_head.weight, params["wte"]) #D

load_weights_into_gpt(gpt, params)
gpt.to(device)

torch.manual_seed(123)
token_ids = generate(
    model=gpt,
    idx=text_to_token_ids("Every effort moves you", tokenizer),
    max_new_tokens=25,
    context_size=NEW_CONFIG["context_length"],
    top_k=50,
    temperature=1.5
)
print("Output text:\n", token_ids_to_text(token_ids, tokenizer))

EXERCISE 5.5 练习 5.5

EXERCISE 5.6 练习 5.6

5.6 Summary 5.6 总结

• When LLMs generate text, they output one token at a time.
  当LLMs生成文本时，它们一次输出一个标记。
• By default, the next token is generated by converting the model outputs into probability scores and selecting the token from the vocabulary that corresponds to the highest probability score, which is known as "greedy decoding."
  默认情况下，下一个令牌是通过将模型输出转换为概率分数并选择对应于最高概率分数的词汇表中的令牌生成的，这被称为"贪婪解码"。
• Using probabilistic sampling and temperature scaling, we can influence the diversity and coherence of the generated text.
  使用概率性抽样和温度缩放,我们可以影响生成文本的多样性和连贯性。
• Training and validation set losses can be used to gauge the quality of text generated by LLM during training.
  训练集和验证集的损失可用于评估LLM在训练过程中生成文本的质量。
• Pretraining an LLM involves changing its weights to minimize the training loss.
  预训练一个LLM涉及到改变其权重以最小化训练损失。
• The training loop for LLMs itself is a standard procedure in deep learning, using a conventional cross entropy loss and AdamW optimizer.
  用于LLMs本身的训练循环是深度学习中的标准流程,使用常规的交叉熵损失和 AdamW 优化器。
• Pretraining an LLM on a large text corpus is time- and resource-intensive so we can load openly available weights from OpenAI as an alternative to pretraining the model on a large dataset ourselves.
  在大型文本语料库上预训练LLM是非常耗时耗资的,因此我们可以加载 OpenAI 公开提供的权重,作为自己在大型数据集上预训练模型的替代方案。

Finetuning for Classification
用于分类的微调

Thls chapter covers 本章涵盖

• Introducing different LLM finetuning approaches
  介绍不同的LLM微调方法
• Preparing a dataset for text classification
  为文本分类准备数据集
• Modifying a pretrained LLM for finetuning
  微调预训练的LLM
• Finetuning an LLM to identify spam messages
  微调 LLM 识别垃圾邮件
• Evaluating the accuracy of a finetuned LLM classifier
  评估微调后LLM分类器的准确性
• Using a finetuned LLM to classify new data
  使用微调的LLM对新数据进行分类

6.1 Different categories of finetuning
6.1 不同类型的微调

CHOOSING THE RIGHT APPROACH
选择正确的方法

6.2 Preparing the dataset
准备数据集

import urllib.request
import zipfile
import os
from pathlib import Path
url = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
zip_path = "sms_spam_collection.zip"
extracted_path = "sms_spam_collection"
data_file_path = Path(extracted_path) / "SMSSpamCollection.tsv"
def download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path):
    if data_file_path.exists():
        print(f"{data_file_path} already exists. Skipping download and extraction.")
        return
    with urllib.request.urlopen(url) as response: #A
        with open(zip_path, "wb") as out_file:
            out_file.write(response.read())
    with zipfile.ZipFile(zip_path, "r") as zip_ref: #B
        zip_ref.extractall(extracted_path)
    original_file_path = Path(extracted_path) / "SMSSpamCollection"
    os.rename(original_file_path, data_file_path) #C
    print(f"File downloaded and saved as {data_file_path}")
download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)
#A Downloading the file
#B Unzipping the file
#C Adding a .tsv file extension

import pandas as pd
df = pd.read_csv(data_file_path, sep="\t", header=None, names=["Label", "Text"])
df

5572 rows columns
5572 行 列

Label
ham 4825
spam 747
Name: count, dtype: int64

Listing 6.2 Creating a balanced dataset
示例 6.2 创建平衡的数据集

def create_balanced_dataset(df):
    num_spam = df[df["Label"] == "spam"].shape[0] #A
    ham_subset = df[df["Label"] == "ham"].sample(num_spam, random_state=123) #B
    balanced_df = pd.concat([ham_subset, df[df["Label"] == "spam"]]) #C
    return balanced_df

train_df.to_csv("train.csv", index=None)
validation_df.to_csv("validation.csv", index=None)
test_df.to_csv("test.csv", index=None)

6.3 Creating data loaders
创建数据加载器 6.3

1. Truncate all messages to the length of the shortest message in the dataset or batch.
   将所有消息截断为数据集或批次中最短消息的长度。
2. Pad all messages to the length of the longest message in the dataset or batch.
   将所有消息的长度填充至数据集或批次中最长消息的长度。

import tiktoken
tokenizer = tiktoken.get_encoding("gpt2")
print(tokenizer.encode("<|endoftext|>", allowed_special={"<|endoftext|>"}))

import torch
from torch.utils.data import Dataset
class SpamDataset(Dataset):
    def __init__(self, csv_file, tokenizer, max_length=None, pad_token_id=50256):
        self.data = pd.read_csv(csv_file)
        self.encoded_texts = [
        tokenizer.encode(text) for text in self.data["Text"]
    ]
    if max_length is None:
        self.max_length = self._longest_encoded_length()
    else:
        self.max_length = max_length
        self.encoded_texts = [
            encoded_text[:self.max_length]
            for encoded_text in self.encoded_texts
        ]
                    #C
    self.encoded_texts = [
        encoded_text + [pad_token_id] * (self.max_length - len(encoded_text))
        for encoded_text in self.encoded_texts
    ]
    def __getitem__(self, index):
        encoded = self.encoded_texts[index]
        label = self.data.iloc[index]["Label"]
        return (
            torch.tensor(encoded, dtype=torch.long),
            torch.tensor(label, dtype=torch.long)
        )

def __len__(self):
    return len(self.data)
def _longest_encoded_length(self):
    max_length = 0
    for encoded_text in self.encoded_texts:
        encoded_length = len(encoded_text)
        if encoded_length > max_length:
            max_length = encoded_length
    return max_length

#A Pre-tokenize texts #A 预处理文本

train_dataset = SpamDataset(
    csv_file="train.csv",
    max_length=None,
    tokenizer=tokenizer
)

val_dataset = SpamDataset(
    csv_file="validation.csv",
    max_length=train_dataset.max_length,
    tokenizer=tokenizer
)
test_dataset = SpamDataset(
    csv_file="test.csv",
    max_length=train_dataset.max_length,
    tokenizer=tokenizer
)

EXERCISE 6.1 INCREASING THE CONTEXT LENGTH
练习 6.1 增加上下文长度

num_workers = 0
    #A
batch_size = 8
torch.manual_seed(123)
train_loader = DataLoader(
    dataset=train_dataset,
    batch_size=batch_size,
    shuffle=True,
    num_workers=num_workers,
    drop_last=True,
)
val_loader = DataLoader(
    dataset=val_dataset,
    batch_size=batch_size,
    num_workers=num_workers,
    drop_last=False,
)
test_loader = DataLoader(
    dataset=test_dataset,
    batch_size=batch_size,
    num_workers=num_workers,
    drop_last=False,
)

for input_batch, target_batch in train_loader:
    pass
print("Input batch dimensions:", input_batch.shape)
print("Label batch dimensions", target_batch.shape)

Input batch dimensions: torch.Size([8, 120])
Label batch dimensions torch.Size([8])

print(f"{len(train_loader)} training batches")
print(f"{len(val_loader)} validation batches")
print(f"{len(test_loader)} test batches")

This concludes the data preparation in this chapter. Next, we will prepare the model for finetuning.

\subsection*{6.4 Initializing a model with pretrained weights}

In this section, we prepare the model we will use for the classification-finetuning to identify spam messages. We start with initializing the pretrained model we worked with in the previous chapter, as illustrated in figure 6.8.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-219.jpg?height=747&width=1247&top_left_y=171&top_left_x=317)

Figure 6.8 Illustration of the three-stage process for classification-finetuning the LLM in this chapter. After completing stage 1, preparing the dataset, this section focuses on initializing the LLM we will finetune to classify spam messages.

We start the model preparation process by reusing the configurations from chapter 5 :

Next, we import the download_and_load_gpt2 function from the gpt_download.py file we downloaded in chapter 5. Furthermore, we also reuse the GPTModel class and load_weights_into_gpt function from chapter 5 to load the downloaded weights into the GPT model:

\section*{Listing 6.6 Loading a pretrained GPT model}

from gpt_download import download_and_load_gpt2

from chapter05 import GPTModel, load_weights_into_gpt

model_size = CHOOSE_MODEL.split(" ")[-1].lstrip("(").rstrip(")")

settings, params = download_and_load_gpt2(model_size=model_size, models_dir="gpt2")

model \(=\) GPTModel(BASE_CONFIG)

load_weights_into_gpt(model, params)

model.eval()

After loading the model weights into the GPTModel, we use the text generation utility function from the previous chapters to ensure that the model generates coherent text:

As we can see based on the following output, the model generates coherent text, which is an indicator that the model weights have been loaded correctly:

Every effort moves you forward.

The first step is to understand the importance of your work

Now, before we start finetuning the model as a spam classifier, let's see if the model can perhaps already classify spam messages by by prompting it with instructions:

The model output is as follows:

\begin{abstract}
Is the following text 'spam'? Answer with 'yes' or 'no': 'You are a winner you have been specially selected to receive \(\$ 1000\) cash or a \$2000 award.

The following text 'spam'? Answer with 'yes' or 'no': 'You are a winner
\end{abstract}

Based on the output, it's apparent that the model struggles with following instructions.

This is anticipated, as it has undergone only pretraining and lacks instruction-finetuning, which we will explore in the upcoming chapter.

The next section prepares the model for classification-finetuning.

\subsection*{6.5 Adding a classification head}

In this section, we modify the pretrained large language model to prepare it for classification-finetuning. To do this, we replace the original output layer, which maps the hidden representation to a vocabulary of 50,257 , with a smaller output layer that maps to two classes: 0 ("not spam") and 1 ("spam"), as shown in figure 6.9.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-222.jpg?height=960&width=1242&top_left_y=694&top_left_x=303)

Figure 6.9 This figure illustrates adapting a GPT model for spam classification by altering its architecture. Initially, the model's linear output layer mapped 768 hidden units to a vocabulary of \(\mathbf{5 0 , 2 5 7}\) tokens. For spam detection, this layer is replaced with a new output layer that maps the same 768 hidden units to just two classes, representing "spam" and "not spam."

As shown in figure 6.9, we use the same model as in previous chapters except for replacing the output layer.

\section*{OUTPUT LAYER NODES}

We could technically use a single output node since we are dealing with a binary classification task. However, this would require modifying the loss function, as discussed in an article in the Reference section in appendix B. Therefore, we choose a more general approach where the number of output nodes matches the number of classes. For example, for a 3-class problem, such as classifying news articles as "Technology", "Sports", or "Politics", we would use three output nodes, and so forth.

Before we attempt the modification illustrated in figure 6.9, let's print the model architecture via print (model), which prints the following:

Above, we can see the architecture we implemented in chapter 4 neatly laid out. As discussed in chapter 4, the GPTModel consists of embedding layers followed by 12 identical transformer blocks (only the last block is shown for brevity), followed by a final LayerNorm and the output layer, out_head.

Next, we replace the out_head with a new output layer, as illustrated in figure 6.9, that we will finetune.

\section*{FINETUNING SELECTED LAYERS VERSUS ALL LAYERS}

Since we start with a pretrained model, it's not necessary to finetune all model layers. This is because, in neural network-based language models, the lower layers generally capture basic language structures and semantics that are applicable across a wide range of tasks and datasets. So, finetuning only the last layers (layers near the output), which are more specific to nuanced linguistic patterns and task-specific features, can often be sufficient to adapt the model to new tasks. A nice side effect is that it is computationally more efficient to finetune only a small number of layers. Interested readers can find more information, including experiments, on which layers to finetune in the References section for this chapter in appendix B.

To get the model ready for classification-finetuning, we first freeze the model, meaning that we make all layers non-trainable:

Then, as shown in figure 6.9, we replace the output layer (model.out_head), which originally maps the layer inputs to 50,257 dimensions (the size of the vocabulary):

\section*{Listing 6.7 Adding a classification layer}

Note that in the preceding code, we use BASE_CONFIG["emb_dim"], which is equal to 768 in the "gpt2-small (124M)" model, to keep the code below more general. This means we can also use the same code to work with the larger GPT-2 model variants.

This new model.out_head output layer has its requires_grad attribute set to True by default, which means that it's the only layer in the model that will be updated during training.

Technically, training the output layer we just added is sufficient. However, as I found in experiments, finetuning additional layers can noticeably improve the predictive performance of the finetuned model. (For more details, refer to the References in appendix C.)

Additionally, we configure the last transformer block and the final LayerNorm module, which connects this block to the output layer, to be trainable, as depicted in figure 6.10.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-226.jpg?height=1188&width=1254&top_left_y=184&top_left_x=297)

Figure 6.10 The GPT model we developed in earlier chapters, which we loaded previously, includes 12 repeated transformer blocks. Alongside the output layer, we set the final LayerNorm and the last transformer block as trainable, while the remaining 11 transformer blocks and the embedding layers are kept non-trainable.

To make the final LayerNorm and last transformer block trainable, as illustrated in figure 6.10, we set their respective requires_grad to True:

\section*{EXERCISE 6.2 FINETUNING THE WHOLE MODEL}

Instead of finetuning just the final transformer block, finetune the entire model and assess the impact on predictive performance.

Even though we added a new output layer and marked certain layers as trainable or nontrainable, we can still use this model in a similar way to previous chapters. For instance, we can feed it an example text identical to how we have done it in earlier chapters. For example, consider the following example text:

As the print output shows, the preceding code encodes the inputs into a tensor consisting of 4 input tokens:

Then, we can pass the encoded token IDs to the model as usual:

The output tensor looks like as follows:

In chapters 4 and 5, a similar input would have produced an output tensor of [1, 4, 50257], where 50,257 represents the vocabulary size. As in previous chapters, the number of output rows corresponds to the number of input tokens (in this case, 4). However, each output's embedding dimension (the number of columns) is now reduced to 2 instead of 50,257 since we replaced the output layer of the model.

Remember that we are interested in finetuning this model so that it returns a class label that indicates whether a model input is spam or not spam. To achieve this, we don't need to finetune all 4 output rows but can focus on a single output token. In particular, we will focus on the last row corresponding to the last output token, as illustrated in figure 6.11.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-228.jpg?height=1213&width=1207&top_left_y=631&top_left_x=339)

Figure 6.11 An illustration of the GPT model with a 4-token example input and output. The output tensor consists of 2 columns due to the modified output layer. We are only focusing on the last row corresponding to the last token when finetuning the model for spam classification.

To extract the last output token, illustrated in figure 6.11, from the output tensor, we use the following code:

print("Last output token:", outputs[:, -1, :])

This prints the following:

Last output token: tensor([[-3.5983, 3.9902]])

Before we proceed to the next section, let's recap our discussion. We will focus on converting the values into a class-label prediction. But first, let's understand why we are particularly interested in the last output token, and not the 1st, 2nd, or 3rd output token.

In chapter 3, we explored the attention mechanism, which establishes a relationship between each input token and every other input token. Subsequently, we introduced the concept of a causal attention mask, commonly used in GPT-like models. This mask restricts a token's focus to only its current position and those before it, ensuring that each token can only be influenced by itself and preceding tokens, as illustrated in figure 6.12.

\section*{Tokens masked out via the causal attention mask discussed in chapter 3}

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-230.jpg?height=1080&width=886&top_left_y=412&top_left_x=396)

\section*{The last token is the only token with an attention score to all other tokens}

Figure 6.12 Illustration of the causal attention mechanism as discussed in chapter 3 , where the attention scores between input tokens are displayed in a matrix format. The empty cells indicate masked positions due to the causal attention mask, preventing tokens from attending to future tokens. The values in the cells represent attention scores, with the last token, "time," being the only one that computes attention scores for all preceding tokens.

Given the causal attention mask setup shown in figure 6.12, the last token in a sequence accumulates the most information since it is the only token with access to data from all the previous tokens. Therefore, in our spam classification task, we focus on this last token during the finetuning process.

Having modified the model, the next section will detail the process of transforming the last token into class label predictions and calculate the model's initial prediction accuracy. Following this, we will finetune the model for the spam classification task in the subsequent section.

\section*{EXERCISE 6.3 FINETUNING THE FIRST VERSUS LAST TOKEN}

Rather than finetuning the last output token, try finetuning the first output token and observe the changes in predictive performance when finetuning the model in later sections.

\subsection*{6.6 Calculating the classification loss and accuracy}

So far in this chapter, we have prepared the dataset, loaded a pretrained model, and modified it for classification-finetuning. Before we proceed with the finetuning itself, only one small part remains: implementing the model evaluation functions used during finetuning, as illustrated in figure 6.13. We will tackle this in this section.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-232.jpg?height=893&width=1223&top_left_y=204&top_left_x=322)

Figure 6.13 Illustration of the three-stage process for classification-finetuning the LLM in this chapter. This section implements the last step of stage 2 , implementing the functions to evaluate the model's performance to classify spam messages before, during, and after the finetuning.

Before implementing the evaluation utilities, let's briefly discuss how we convert the model outputs into class label predictions.

In the previous chapter, we computed the token ID of the next token generated by the LLM by converting the 50,257 outputs into probabilities via the softmax function and then returning the position of the highest probability via the argmax function. In this chapter, we take the same approach to calculate whether the model outputs a "spam" or "not spam" prediction for a given input, as shown in figure 6.14 , with the only difference being that we work with 2-dimensional instead of 50,257-dimensional outputs.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-233.jpg?height=418&width=1252&top_left_y=187&top_left_x=312)

Figure 6.14. The model outputs corresponding to the last token are converted into probability scores for each input text. Then, the class labels are obtained by looking up the index position of the highest probability score. Note that the model predicts the spam labels incorrectly because it has not yet been trained.

To illustrate figure 6.14 with a concrete example, let's consider the last token output from the previous section:

print("Last output token:", outputs[:, -1, :])

The values of the tensor corresponding to the last token are as follows:

Last output token: tensor([[-3.5983, 3.9902]])

We can obtain the class label via the following code:

In this case, the code returns 1, meaning the model predicts that the input text is "spam." Using the softmax function here is optional because the largest outputs directly correspond to the highest probability scores, as mentioned in chapter 5 . Hence, we can simplify the code as follows, without using softmax:

This concept can be used to compute the so-called classification accuracy, which measures the percentage of correct predictions across a dataset.

To determine the classification accuracy, we apply the argmax-based prediction code to all examples in the dataset and calculate the proportion of correct predictions by defining a calc_accuracy_loader function:

\section*{Listing 6.8 Calculating the classification accuracy}

\#A Logits of last output token

Let's use the function to determine the classification accuracies across various datasets estimated from 10 batches for efficiency:

Via the device setting, the model automatically runs on a GPU if a GPU with Nvidia CUDA support is available and otherwise runs on a CPU. The output is as follows:

Training accuracy: \(46.25 \%\)

Validation accuracy: \(45.00 \%\)

Test accuracy: \(48.75 \%\)

As we can see, the prediction accuracies are near a random prediction, which would be \(50 \%\) in this case. To improve the prediction accuracies, we need to finetune the model.

However, before we begin finetuning the model, we need to define the loss function that we will optimize during the training process. Our objective is to maximize the spam classification accuracy of the model, which means that the preceding code should output the correct class labels: 0 for non-spam and 1 for spam texts.

However, classification accuracy is not a differentiable function, so we use cross entropy loss as a proxy to maximize accuracy. This is the same cross entropy loss discussed in chapter 5 .

Accordingly, the calc_loss_batch function remains the same as in chapter 5, with one adjustment: we focus on optimizing only the last token, model(input_batch) [:, -1, : ], rather than all tokens, model (input_batch):

We use the calc_loss_batch function to compute the loss for a single batch obtained from the previously defined data loaders. To calculate the loss for all batches in a data loader, we define the calc_loss_loader function, which is identical to the one described in chapter 5 :

Listing 6.9 Calculating the classification loss

\#A Ensure number of batches doesn't exceed batches in data loader

\#B Disable gradient tracking for efficiency because we are not training, yet

In the next section, we will implement a training function to finetune the model, which means adjusting the model to minimize the training set loss. Minimizing the training set loss will help increase the classification accuracy, our overall goal.

\subsection*{6.7 Finetuning the model on supervised data}

In this section, we define and use the training function to finetune the pretrained LLM and improve its spam classification accuracy. The training loop, illustrated in figure 6.15, is the same overall training loop we used in chapter 5, with the only difference being that we calculate the classification accuracy instead of generating a sample text for evaluating the model.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-237.jpg?height=1266&width=1212&top_left_y=541&top_left_x=318)

Figure 6.15 A typical training loop for training deep neural networks in PyTorch consists of several steps, iterating over the batches in the training set for several epochs. In each loop, we calculate the loss for each training set batch to determine loss gradients, which we use to update the model weights to minimize the training set loss.

The training function implementing the concepts shown in figure 6.15 also closely mirrors the train_model_simple function used for pretraining the model in chapter 5 .

The only two distinctions are that we now track the number of training examples seen (examples_seen) instead of the number of tokens, and we calculate the accuracy after each epoch instead of printing a sample text:

\section*{Listing 6.10 Finetuning the model to classify spam}

def train_classifier_simple(model, train_loader, val_loader, optimizer, device, num_epochs, eval_freq, eval_iter, tokenizer):

\# Initialize lists to track losses and examples seen

train_losses, val_losses, train_accs, val_accs = [], [], [], []

examples_seen, global_step = 0, -1

\# Main training loop

for epoch in range(num_epochs):

model.train()

\#A

for input_batch, target_batch in train_loader:

optimizer.zero_grad() \#B

loss = calc_loss_batch(input_batch, target_batch, model, device)

loss.backward() \#C

optimizer.step() \#D

examples_seen += input_batch.shape[0] \#E

global_step += 1

if global_step \% eval_freq == 0: train_loss, val_loss = evaluate_model(

model, train_loader, val_loader, device, eval_iter)

train_losses.append(train_loss)

val_losses.append(val_loss)

print(f"Ep \{epoch+1\} (Step \{global_step:06d\}): "

f"Train loss \{train_loss:.3f\}, Val loss \{val_loss:.3f\}")

\#G

train_accuracy = calc_accuracy_loader(

train_loader, model, device, num_batches=eval_iter

)

val_accuracy = calc_accuracy_loader(

val_loader, model, device, num_batches=eval_iter

)

print(f"Training accuracy: \{train_accuracy*100:.2f\}\% | ", end="")

ⓒ Manning Publications Co. To comment go to liveBook

Licensed to lijin lijin <jimlee13894896995@gmail.com>

print(f"Validation accuracy: \{val_accuracy*100:.2f\}\%")

train_accs.append(train_accuracy)

val_accs.append(val_accuracy)

return train_losses, val_losses, train_accs, val_accs, examples_seen

\#A Set model to training mode

\#B Reset loss gradients from previous batch iteration

\#C Calculate loss gradients

\#D Update model weights using loss gradients

\#E New: track examples instead of tokens

\#F Optional evaluation step

\#G Calculate accuracy after each epoch

The evaluate_model function used in the preceding train_classifier_simple is identical the one we used in chapter 5 :

Next, we initialize the optimizer, set the number of training epochs, and initiate the training using the train_classifier_simple function. We will discuss the choice of the the number of training epochs after we evaluated the results. The training takes about 6 minutes on an M3 MacBook Air laptop computer and less than half a minute on a V100 or A100 GPU:

The output we see during the training is as follows:

Similar to chapter 5, we then use matplotlib to plot the loss function for the training and validation set:

Listing 6.11 Plotting the classification loss

import matplotlib.pyplot as plt

def plot_values(epochs_seen, examples_seen, train_values, val_values, label="loss"):

fig, ax1 = plt.subplots(figsize=(5, 3))

\#A

ax1.plot(epochs_seen, train_values, label=f"Training \{label\}")

ax1.plot(epochs_seen, val_values, linestyle="-.", label=f"Validation \{label\}")

ax1.set_xlabel("Epochs")

ax1.set_ylabel(label.capitalize())

ax1.legend()

ax2 = ax1.twiny()

ax2.plot(examples_seen, train_values, alpha=0) \# Invisible plot for aligning ticks ax2.set_xlabel("Examples seen")

fig.tight_layout() \#C

plt.savefig(f"\{label\}-plot.pdf")

plt.show()

\#A Plot training and validation loss against epochs

\#B Create a second \(x\)-axis for examples seen

\#C Adjust layout to make room

The resulting loss curves are shown in the plot in figure 6.16.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-242.jpg?height=698&width=1210&top_left_y=186&top_left_x=319)

Figure 6.16 This graph shows the model's training and validation loss over the five training epochs. The training loss, represented by the solid line, and the validation loss, represented by the dashed line, both sharply decline in the first epoch and gradually stabilize towards the fifth epoch. This pattern indicates good learning progress and suggests that the model learned from the training data while generalizing well to the unseen validation data.

As we can see based on the sharp downward slope in figure 6.16 , the model is learning well from the training data, and there is little to no indication of overfitting; that is, there is no noticeable gap between the training and validation set losses).

\section*{CHOOSING THE NUMBER OF EPOCHS}

Earlier, when we initiated the training, we set the number of epochs to 5 . The number of epochs depends on the dataset and the task's difficulty, and there is no universal solution or recommendation. An epoch number of 5 is usually a good starting point. If the model overfits after the first few epochs, as a loss plot as shown in Figure 6.16 could indicate, we may need to reduce the number of epochs. Conversely, if the trendline suggests that the validation loss could improve with further training, we should increase the number of epochs. In this concrete case 5 epochs was a reasonable number as there is no sign of early overfitting, and the validation loss is close to 0 .

Using the same plot_values function, let's now also plot the classification accuracies:
epochs_tensor = torch.linspace(0, num_epochs, len(train_accs))

examples_seen_tensor = torch.linspace(0, examples_seen, len(train_accs))

plot_values(epochs_tensor, examples_seen_tensor, train_accs, val_accs, label="accuracy") The resulting accuracy graphs are shown in figure 6.17.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-243.jpg?height=714&width=1228&top_left_y=466&top_left_x=319)

Figure 6.17 Both the training accuracy (solid line) and the validation accuracy (dashed line) increase substantially in the early epochs and then plateau, achieving almost perfect accuracy scores of 1.0. The close proximity of the two lines throughout the epochs suggests that the model does not overfit the training data much.

Based on the accuracy plot in figure 6.17, the model achieves a relatively high training and validation accuracy after epochs 4 and 5 .

However, it's important to note that we previously set eval_iter=5 when using the train_classifier_simple function, which means our estimations of training and validation performance were based on only 5 batches for efficiency during training.

Now, we will calculate the performance metrics for the training, validation, and test sets across the entire dataset by running the following code, this time without defining the eval_iter value:

The resulting accuracy values are as follows:

Training accuracy: \(97.21 \%\)

Validation accuracy: \(97.32 \%\)

Test accuracy: \(95.67 \%\)

The training and test set performances are almost identical.

A slight discrepancy between the training and test set accuracies suggests minimal overfitting of the training data. Typically, the validation set accuracy is somewhat higher than the test set accuracy because the model development often involves tuning hyperparameters to perform well on the validation set, which might not generalize as effectively to the test set.

This situation is common, but the gap could potentially be minimized by adjusting the model's settings, such as increasing the dropout rate (drop_rate) or the weight_decay parameter in the optimizer configuration.

\subsection*{6.8 Using the LLM as a spam classifier}

After finetuning and evaluating the model in the previous sections, we are now in the final stage of this chapter, as illustrated in figure 6.18: using the model to classify spam messages.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-245.jpg?height=754&width=1225&top_left_y=186&top_left_x=321)

Figure 6.18 Illustration of the three-stage process for classification-finetuning the LLM in this chapter. This section implements the final step of stage 3 , using the finetuned model to classify new spam messages.

Finally, let's use the finetuned GPT-based spam classification model. The following classify_review function follows data preprocessing steps similar to those we used in the SpamDataset implemented earlier in this chapter. And then, after processing text into token IDs, the function uses the model to predict an integer class label, similar to what we have implemented in section 6.6 , and then returns the corresponding class name:

Listing 6.12 Using the model to classify new texts

def classify_review(text, model, tokenizer, device, max_length=None, pad_token_id=50256)

model.eval()

input_ids = tokenizer.encode(text) \#A

supported_context_length = model.pos_emb.weight.shape[1]

input_ids = input_ids[:min(max_length, supported_context_length)] \#B

input_ids += [pad_token_id] * (max_length - len(input_ids)) \#C

input_tensor = torch.tensor(input_ids, device=device).unsqueeze(0) \#D

with torch.no_grad()

logits = model(input_tensor) \([:,-1, \quad:]\)

predicted_label = torch.argmax(logits, dim=-1).item()

return "spam" if predicted_label == 1 else "not spam" \#G

\#A Prepare inputs to the model

\#B Truncate sequences if they too long

\#C Pad sequences to the longest sequence

\#D Add batch dimension

\#E Model inference without gradient tracking

\#F Logits of the last output token

\#G Return the classified result

Let's try this classify_review function on an example text:

The resulting model correctly predicts "spam". Next, let's try another example:

Also, here, the model makes a correct prediction and returns a "not spam" label.

Finally, let's save the model in case we want to reuse the model later without having to train it again using the torch. save method we introduced in the previous chapter.

torch.save(model.state_dict(), "review_classifier.pth")

Once saved, the model can be loaded as follows:

\subsection*{6.9 Summary}
- \(\quad\) There are different strategies for finetuning LLMs, including classificationfinetuning (this chapter) and instruction-finetuning (next chapter)
- \(\quad\) Classification-finetuning involves replacing the output layer of an LLM via a small classification layer.
- In the case of classifying text messages as "spam" or "not spam," the new classification layer consists of only 2 output nodes; in previous chapters, the number of output nodes was equal to the number of unique tokens in the vocabulary, namely, 50,256
- Instead of predicting the next token in the text as in pretraining, classification-finetuning trains the model to output a correct class label, for example, "spam" or "not spam."
- The model input for finetuning is text converted into token IDs, similar to pretraining.
- Before finetuning an LLM, we load the pretrained model as a base model.
- \(\quad\) Evaluating a classification model involves calculating the classification accuracy (the fraction or percentage of correct predictions).
- \(\quad\) Finetuning a classification model uses the same cross entropy loss function that is used for pretraining the LLM.

\section*{7 \\ Finetuning to Follow Instructions}

\section*{This chapter covers}
- Introduction to the instruction finetuning process of LLMs
- Preparing a dataset for supervised instruction finetuning
- Organizing instruction data in training batches
- Loading a pretrained LLM and finetuning it to follow human instructions
- Extracting LLM-generated instruction responses for evaluation
- Evaluating an instruction-finetuned LLM

In previous chapters, we implemented the LLM architecture, carried out pretraining, and imported pretrained weights from external sources into our model. Then, in the previous chapter, we focused on finetuning our LLM for a specific classification task: distinguishing between spam and non-spam text messages. In this chapter, we implement the process for finetuning an LLM to follow human instructions, as illustrated in figure 7.1, which is one of the main techniques behind developing LLMs for chatbot applications, personal assistants, and other conversational tasks.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-249.jpg?height=575&width=1266&top_left_y=182&top_left_x=298)

Figure 7.1 A mental model of the three main stages of coding an LLM, pretraining the LLM on a general text dataset, and finetuning it. This chapter focuses on finetuning a pretrained LLM to follow human instructions.

Figure 7.1 shows two main ways of finetuning an LLM: finetuning for classification (step 8) and finetuning an LLM to follow instructions (step 9). We implemented step 8 in the previous chapter. This chapter focuses on finetuning an LLM using an instruction dataset, a process that will be further explained in the next section.

\subsection*{7.1 Introduction to instruction finetuning}

In chapter 5, we saw that pretraining an LLM involves a training procedure where it learns to generate one word at a time. The resulting pretrained LLM is capable of text completion, meaning it can finish sentences or write text paragraphs given a fragment as input.

However, pretrained LLMs often struggle with specific instructions, such as "Fix the grammar in this text" or "Convert this text into passive voice." We will examine a concrete example of this in section 7.5, where we load the pretrained LLM as the basis for instruction finetuning.

In this chapter, we focus on improving the LLM's ability to follow such instructions and generate a desired response, as illustrated in figure 7.2.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-250.jpg?height=198&width=568&top_left_y=193&top_left_x=317)

Convert 45 kilometers to meters.
The goal for the LLM is to

generate a desired response

\section*{Desired response}

45 kilometers is 45000 meters.

Provide a synonym for "bright". \(\longrightarrow\) A synonym for "bright" is "radiant".

Edit the following sentence to remove all passive voice: "The

\(\longrightarrow\) The artist composed the song.

Figure 7.2 This figure shows examples of instructions that are processed by an LLM to generate desired responses.

In the remainder of this chapter, we will implement the instruction finetuning process in several steps, beginning with the dataset preparation, as shown in figure 7.3.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-251.jpg?height=846&width=1249&top_left_y=190&top_left_x=302)

Figure 7.3 Illustration of the three-stage process for instruction finetuning the LLM in this chapter. Stage 1 involves dataset preparation. Stage 2 focuses on model setup and finetuning. Stage 3 covers the evaluation of the model.

Preparing the dataset is a key aspect of instruction finetuning where most of the time in this chapter is spent. The next section, as illustrated in figure 7.3, implements the code to download and format the dataset, which is the first step in the dataset preparation process.

\subsection*{7.2 Preparing a dataset for supervised instruction finetuning}

In this section, we download and format the instruction dataset for instruction finetuning a pretrained LLM in this chapter. The dataset consists of 1100 instruction-response pairs similar to those shown in figure 7.2. This dataset has been specifically created for this book, but interested readers can find alternative, publicly available instruction datasets in appendix B.

The following code implements and executes a function to download this dataset, which is a relatively small file, only 204 KB in size, in JSON format. JSON, or JavaScript Object Notation, mirrors the structure of Python dictionaries, providing a simple structure for data interchange that is both human-readable and machine-friendly.

Listing 7.1 Downloading the dataset

import json

import os

import urllib

def download_and_load_file(file_path, url):

if not os.path.exists(file_path):

with urllib.request.urlopen(url) as response:

text_data = response.read().decode("utf-8")

with open(file_path, "w", encoding="utf-8") as file:

file.write(text_data)

else: \#A

with open(file_path, "r", encoding="utf-8") as file:

text_data = file.read()

with open(file_path, "r") as file:

data = json.load(file)

return data

file_path = "instruction-data.json"

url = "https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/ch07/01_main-

chapter-code/instruction-data.json"

data = download_and_load_file(file_path, url)

print("Number of entries:", len(data))

\#A Skip download if file was already downloaded

The output of executing the preceding code is as follows:

Number of entries: 1100

The data list, which we loaded from the JSON file contains the 1100 entries of the instruction dataset. Let's print one of the entries to see how each entry is structured:

print("Example entry:\n", data[50])

The content of the example entry is as follows:

Example entry:

\{'instruction': 'Identify the correct spelling of the following word.', 'input': 'Ocassion', 'output': "The correct spelling is 'Occasion.'"\}

As we can see, the example entries are Python dictionary objects containing an 'instruction', 'input', and 'output'. Let's take a look at another example:

print("Another example entry:\n", data[999])

Based on the contents of this entry, the 'input' field may occasionally be empty:

Another example entry:

\{'instruction': "What is an antonym of 'complicated'?", 'input': '', 'output': "An antonym of 'complicated' is 'simple'."\}

Instruction finetuning, also known as supervised instruction finetuning, involves training a model on a dataset where the input-output pairs, like those we extracted from the JSON file, are explicitly provided. There are various methods to format these entries for LLMs. Figure 7.4 illustrates two different example formats, often referred to as prompt styles, used in the training of notable LLMs such as Alpaca and Phi-3. Alpaca was one of the early LLMs to publicly detail its instruction finetuning process. Phi-3, developed by Microsoft, is included to demonstrate the diversity in prompt styles.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-253.jpg?height=677&width=1235&top_left_y=1071&top_left_x=314)

Figure 7.4 Comparison of prompt styles for instruction finetuning in LLMs. The Alpaca style (left) uses a structured format with defined sections for instruction, input, and response, while the Phi-3 style (right) employs a simpler format with designated <|user|> and <|assistant|> tokens.

The rest of this chapter uses the Alpaca prompt style since it is one of the most popular ones, largely because it helped define the original approach to finetuning.

\section*{EXERCISE 7.1 CHANGING PROMPT STYLES}

After finetuning the model with the Alpaca prompt style, try the Phi-3 prompt style shown in figure 7.4 and observe if it affects the response quality of the model.

Let's define a format_input function that we can use to convert the entries in the data list into the Alpaca-style input format depicted in figure 7.4:

\section*{Listing 7.2 Implementing the prompt formatting function}

This format_input function takes a dictionary entry as input and constructs a formatted string. Let's test it to dataset entry data [50], which to looked at earlier:

For formatted input looks like as follows:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

\#\#\# Instruction:

Identify the correct spelling of the following word.

\section*{\#\#\# Input:}

Ocassion

\#\#\# Response:

The correct spelling is 'Occasion.'

Note that the format_input skips the optional \#\#\# Input: section if the 'input' field is empty, which we can test out by applying the format_input function to entry data[999] that we inspected earlier:

As we can see based on the following output, entries with an empty 'input' field don't contain an \#\#\# Input: section in the formatted input:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

\#\#\# Instruction:

What is an antonym of 'complicated'?

\#\#\# Response:

An antonym of 'complicated' is 'simple'.

Before we move on to setting up the PyTorch data loaders in the next section, let's divide the dataset into training, validation, and test sets analogous to what we have done with the spam classification dataset in the previous chapter. Here's how we calculate the portions:

\section*{Listing 7.3 Partitioning the dataset}

This partitioning results in the following dataset sizes:

Training set length: 935

Validation set length: 55

Test set length: 110

Having successfully downloaded and partitioned the dataset, and gained a clear understanding of the dataset prompt formatting, we are now ready for the core implementation of the instruction finetuning process. In the upcoming section, we will focus on developing the method for constructing the training batches for finetuning the LLM.

\subsection*{7.3 Organizing data into training batches}

As we progress into the implementation phase of our instruction finetuning process, the next step, illustrated in figure 7.5 , focuses on constructing the training batches effectively. This involves defining a method that will ensure our model receives the formatted training data during the finetuning process.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-256.jpg?height=788&width=1263&top_left_y=900&top_left_x=302)

Figure 7.5 After downloading the dataset and implementing text formatting utility function in the previous section, this section focuses on assembling the training batches.

In the previous chapter, the training batches were created automatically by the PyTorch DataLoader class, which employs a default collate function to combine lists of samples into batches. A collate function is responsible for taking a list of individual data samples and merging them into a single batch that can be processed efficiently by the model during training,

However, the batching process for instruction finetuning in this chapter is a bit more involved and requires us to create our own custom collate function that we will later plug into the DataLoader. We implement this custom collate function to handle the specific requirements and formatting of our instruction finetuning dataset.

In this section, we will tackle the batching process in several steps including the coding of the custom collate function, as illustrated in figure 7.6.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-257.jpg?height=1080&width=1228&top_left_y=497&top_left_x=319)

Figure 7.6 An illustration of the five substeps involved in implementing the batching process: applying the prompt template defined in the previous section, using tokenization from previous chapters, adding padding tokens, creating target token IDs, and replacing -100 placeholder tokens to mask padding tokens in the loss function.

First, to implement steps 2.1 and 2.2 as illustrated in figure 7.6 , we code an InstructionDataset class that applies format_input from the previous section and pretokenizes all inputs in the dataset, similar to the SpamDataset in chapter 6 . These two steps are illustrated in more detail in figure 7.7.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-258.jpg?height=660&width=1261&top_left_y=184&top_left_x=301)

Figure 7.7 This diagram shows how entries are first formatted using a specific prompt template and then tokenized, resulting in a sequence of token IDs that the model can process.

The 2 -step process illustrated in figure 7.7 is implemented in the init constructor method of the InstructionDataset:

Listing 7.4 Implementing an instruction dataset class

\#A Pre-tokenize texts

Similar to the approach in chapter 6, we aim to accelerate training by collecting multiple training examples in a batch, which necessitates padding all inputs to a similar length. As with the previous chapter, we use the \(<\) |endoftext|> token as a padding token.

Instead of appending the <|endoftext|> tokens to the text inputs, we can append its token ID to the pre-tokenized inputs directly. To remind us which token ID we should use, we can use the tokenizer's .encode method on an <|endoftext|> token:

In chapter 6, we padded all examples in a dataset to the same length. Moving on to step 2.3 in figure 7.6, here, we adopt a more sophisticated approach by developing a custom collate function that we can pass to the data loader. This custom collate function pads the training examples in each batch to have the same length, while allowing different batches to have different lengths, as illustrated in figure 7.8. This approach minimizes unnecessary padding by only extending sequences to match the longest one in each batch, not the whole dataset.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-260.jpg?height=330&width=1238&top_left_y=184&top_left_x=312)

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-260.jpg?height=337&width=1172&top_left_y=586&top_left_x=319)

Figure 7.8 This figure showed the padding of training examples in batches using token ID 50256 to ensure uniform length within each batch. Each batch may have different lengths, as shown by the first and second batches in this figure.

We can implement the padding process illustrated in figure 7.8 with a custom collate function as follows:

\#A Find the longest sequence in the batch
\#B Pad and prepare inputs
\#C Remove extra padded token added earlier
\#D Convert list of inputs to tensor and transfer to target device

The custom_collate_draft_1 we implemented is designed to be integrated into a PyTorch DataLoader, but it can also function as a standalone tool. Here, we use it independently to test and verify that it operates as intended. Let's try it on three different inputs that we want to assemble into a batch, where each example gets padded to the same length:

The resulting batch looks like as follows:

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-262.jpg?height=146&width=703&top_left_y=193&top_left_x=245)

As we can see based on the preceding output, all inputs have been padded to the length of the longest input list, inputs_1 containing 5 token IDs.

So, we have just implemented our first custom collate function to create batches from lists of inputs. However, as you learned in chapters 5 and 6 , we also need to create batches with the target token IDs, corresponding to the batch of input IDs. These target IDs, as illustrated in figure 7.9, are crucial because they represent what we want the model to generate and what we need during training to calculate the loss for the weight updates, similar to previous chapters.

Instruction: ... 说明：...

                            Format input into an instruction-

Response: ... 响应：...

                            response template

Figure 7.9 An illustration of the five substeps involved in implementing the batching process. We are now focusing on step 2.4, the creation of target token IDs. This step is essential as it enables the model to learn and predict the tokens it needs to generate.

As illustrated in figure 7.9, we are now modifying our custom collate function to also return the target token IDs in addition to the input token IDs.

Similar to the process described in chapter 5 for pretraining an LLM, the target token IDs match the input token IDs but are shifted one position to the right. This setup, as shown in figure 7.10, allows the LLM to learn how to predict the next token in a sequence.

The target vector does not contain the

first input ID

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-263.jpg?height=479&width=1013&top_left_y=458&top_left_x=521)

The token IDs in the target are similar to the input IDs but shifted by 1 position

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-263.jpg?height=375&width=870&top_left_y=1151&top_left_x=522)

We always add an end-of-text (padding) token

to the target

Figure 7.10 This figure illustrates the input and target token alignment used in the instruction finetuning process of an LLM. For each input sequence, the corresponding target sequence is created by shifting the token IDs one position to the right, omitting the first token of the input, and appending an end-of-text token.

The following updated collate function generates the target token IDs, as illustrated in figure 7.10 , from the input token IDs:

\#A Truncate the last token for inputs

\#B Shift +1 to the right for targets

Applied to the example batch consisting of 3 input lists we defined earlier, the new custom_collate_draft_2 function now returns the input and the target batch:

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-264.jpg?height=269&width=1047&top_left_y=1560&top_left_x=257)

\#A The 1st tensor represents inputs

\#B The 2nd tensor represents the targets

In the next step, we assign a -100 placeholder value to all padding tokens, as illustrated in figure 2.5. This special value allows us to exclude these padding tokens from contributing to the training loss calculation, ensuring that only meaningful data influences model learning.

More details on this process will be discussed after implementing this modification. (In chapter 6 , we did not have to worry about this since we only trained the model based on the last output token.)

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-265.jpg?height=1006&width=1247&top_left_y=355&top_left_x=305)

Figure 7.11 This figure illustrates step 2.5 in the token replacement process we apply to the data batches. After creating the target sequence by shifting token IDs one position to the right and appending an end-of-text token, step 2.5 focuses on replacing end-of-text padding tokens with a placeholder value (-100).

In step 2.4, as shown in figure 7.11, we replace the end-of-text tokens, which we previously used as padding tokens and are assigned token ID 50256, with -100 in the target token list. (The choice of -100 as a replacement will be clarified later.)

However, note that we retain one end-of-text token, ID 50256, in the target list, as depicted in figure 7.12. This allows the LLM to learn when to generate an end-of-text token in response to instructions, which we use as an indicator that the generated response is complete.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-266.jpg?height=432&width=1252&top_left_y=166&top_left_x=298)

Figure 7.12 This figure illustrates step 2.4 in the token replacement process in the target batch for the training data preparation. It shows the replacement of all but the first instance of the end-of-text token, which we use as padding, with the placeholder value -100 , while keeping the initial end-of-text token in each target sequence.

In the following code, we modify our custom collate function to replace tokens with ID 50256 with -100 in the target lists, as illustrated in figure 7.12. Additionally, we introduce an allowed_max_length parameter to optionally limit the length of the samples. This adjustment will be useful if you plan to work with your own datasets that exceed the 1024token context size supported by the GPT-2 model. The code for this updated collate function is as follows:

\section*{Listing 7.5 Implementing a custom batch collate function}

\#A

\#A Replace all but the first padding tokens in targets by ignore_index

\#B Optionally truncate to maximum sequence length

Again, let's try the collate function on the sample batch that we created earlier to check that it works as intended:

The results are as follows, where the first tensor represents the inputs, and the second tensor represents the targets:

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-267.jpg?height=276&width=715&top_left_y=1312&top_left_x=246)

The modified collate function works as expected, altering the target list by inserting the token ID -100. What is the logic behind this adjustment? Let's explore the underlying purpose of this modification.

For demonstration purposes, consider the following simple and self-contained example where each output logit can correspond to a potential token from the model's vocabulary. Here's how we might calculate the cross entropy loss (introduced in chapter 5) during training when the model predicts a sequence of tokens, similar to what we have done in chapter 5 when pretraining the model, or in chapter 6 when finetuning the model for classification:

The loss value calculated by the previous code is 1.1269 .

tensor(1.1269)

Adding an additional token ID will, as we would expect, affect the loss calculation.

\#A New 3rd token ID prediction

The loss value, after adding the third token, is now 0.7936 .

So far, we have carried out some more or less obvious example calculations using the cross entropy loss function in PyTorch, the same loss function we used in the training functions of chapters 5 and 6 , as well as the one we will use in this chapter.

Now, let's get to the interesting part and see what happens if we replace the third target token ID with -100 :

Based on this result, we can see that the resulting loss on these 3 training examples is identical to the loss we calculated from the 2 training examples earlier. In other words, the cross entropy loss function ignored the third entry in the targets_3 vector, the token ID corresponding to -100 . (Interested readers can try to replace the -100 value with another token IDs that is not 0 or 1 , and will see that this results in an error.)

So, what's so special about -100 that it's ignored by the cross entropy loss? The default setting of the cross entropy function in PyTorch is cross_entropy(..., ignore_index=-100). This means that it ignores targets labeled with -100 .

In this chapter, we take advantage of this ignore_index to ignore the additional end-oftext (padding) tokens that we used to pad the training examples to have the same length in each batch.

However, as shown earlier in figure 7.12, we want to keep one 50256 (end-of-text) token ID in the targets because it helps the LLM to learn to generate end-of-text tokens, which we can use as an indicator that a response is complete.

In addition to masking out padding tokens, it is also common to mask out the target token IDs that correspond to the instruction, as illustrated in figure 7.13.
![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-269.jpg?height=508&width=1256&top_left_y=956&top_left_x=292)

\#\#\# Response:

Great results were achieved by the team.<|endoftext|>

Tokenize

\([-100,-100,-100,-100,-100, \ldots, 13,50256]\)

1

The instruction tokens are replaced by -100

Figure 7.13 The left side shows the formatted input text we tokenize and then feed to the LLM during training. The right side shows the target text we prepare for the LLM where we can optionally mask out the instruction section, which means replacing the corresponding token IDs with the -100 ignore_index value.

By masking out the target token IDs that correspond to the instruction, as shown in figure 7.13, the LLM the cross entropy loss is only computed for the generated response target IDs. By masking out the instruction tokens, the model is trained to focus on generating accurate responses rather than additionally also memorizing instructions, which can help with reducing overfitting.

Currently, researchers are divided on whether masking the instructions as shown in figure 7.13 is universally beneficial during instruction finetuning. For instance, a recent paper titled "Instruction Tuning With Loss Over Instructions" demonstrated that not masking the instructions benefits the LLM performance (see the references in appendix B for more details). In this chapter, we do not apply masking and leave it as an optional exercise for the reader.

\section*{EXERCISE 7.2 INSTRUCTION AND INPUT MASKING}

After completing the chapter and finetuning the model with the InstructionDataset implemented in this section, replace the instruction and input tokens with the -100 mask to implement the instruction masking method illustrated in Figure 7.13. Then, evaluate whether this has a positive effect on model performance.

\subsection*{7.4 Creating data loaders for an instruction dataset}

In the previous section, we went through several stages to implement an InstructionDataset class and a custom_collate_fn function for the instruction dataset. In this section, as shown in figure 7.14, we can reap the fruits of our labor by simply plugging both InstructionDataset objects and the custom_collate_fn function into PyTorch data loaders. These loaders will automatically shuffle and organize the batches for the LLM instruction finetuning process.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-271.jpg?height=788&width=1270&top_left_y=167&top_left_x=298)

Figure 7.14 In previous sections, we prepared the dataset and implemented a custom collate function for batching the instruction dataset. In this section, we create and apply the data loaders to the training, validation, and test sets that we need for the LLM instruction finetuning and evaluation.

Before we implement the data loader creation step shown in figure 7.14 , we have to briefly talk about the device setting of the custom_collate_fn we implemented in the previous section.

The custom_collate_fn includes code to move the input and target tensors (for example, torch.stack(inputs_lst).to(device)) to a specified device, which can be either "cpu" or "cuda" (for GPUs), or optionally "mps" for Macs with Apple Silicon chips. (Note that using an "mps" device may result in numerical differences compared to the contents of this chapter, as Apple Silicon support in PyTorch is still experimental.)

In previous chapters, we moved the data onto the target device (for example, the GPU memory when device="cuda") in the main training loop. Having this as part of the collate function offers the advantage of performing this device transfer process as a background process outside the training loop, preventing it from blocking the GPU during model training.

The following code initializes the device variable:

if torch.backends.mps.is_available(): #A
如果 torch.backends.mps.is_available():

device = torch.device("mps")" #A
设备 = torch.device("mps")

\#A Uncomment these two lines to use the GPU on an Apple Silicon chip

Next, to reuse the chosen device setting in custom_collate_fn when we plug it into the PyTorch DataLoader class later in this section, we use the partial function from Python's functools standard library to create a new version of the function with the device argument pre-filled. Additionally, we set the allowed_max_length to 1024, which truncates the data to the maximum context length supported by the GPT-2 model we finetune later in this chapter:

Next, we can set up the data loaders as we did in previous chapters, but this time we will use our custom collate function for the batching process:

Listing 7.6 Initializing the data loaders

\#A You can try to increase this number if parallel Python process are supported by your operating system

Let's examine the dimensions of the input and target batches generated by the training Ioader:

The output is as follows (truncated for space reasons):

In the preceding output, we can see that the first input and target batch have dimensions \(8 \times 61\), where 8 represents the batch size, and 61 is the number of tokens in each training example in this batch. The second input and target batch have a different number of tokens, for instance, 76 .

As we saw in the preceding code output, thanks to our custom collate function, the data loader is able to create batches of different lengths. In the next section, we load a pretrained LLM that we can then finetune with this data loader.

\subsection*{7.5 Loading a pretrained LLM}

In the previous sections, we spent a lot of time preparing the dataset for instruction finetuning, which is a key aspect of the supervised finetuning process. Many other aspects are the same as in pretraining, allowing us to reuse much of the code from earlier chapters.

Before beginning instruction finetuning, we first load a pretrained GPT model, as shown in figure 7.15, that we want to finetune.

In the previous sections, we prepared the

dataset and data loaders

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-274.jpg?height=759&width=1250&top_left_y=276&top_left_x=299)

Figure 7.15 After the dataset preparation, the process of finetuning an LLM for instruction-following begins with loading a pretrained LLM, which serves as the foundation for subsequent training. This pretrained model, having already learned general language patterns and knowledge from vast amounts of text data, is then adapted for instruction following through the finetuning process in the next section.

As shown in the chapter overview diagram in figure 7.15, this section focuses on step 4, loading a pretrained LLM to serve as the starting point for instruction finetuning, similar to the process in previous chapters. However, instead of using the smallest 124 million parameter model as before, we load the medium-sized model with 355 million parameters. The reason for this choice is that the 124 million parameter model is too limited in capacity to achieve qualitatively satisfactory results via instruction finetuning.

This is done using the same code as in section 5.5 of chapter 5 and section 6.4 of chapter 6 except that we now specify "gpt2-medium (355M)" instead of "gpt2-small (124M)". Please note that executing the code provided below will initiate the download of the medium-sized GPT model, which has a storage requirement of approximately 1.42 gigabytes. This is roughly three times larger than the storage space needed for the small model:

Listing 7.7 Loading the pretrained model

After executing the code in the previous section, several files will be downloaded, similar to the process in earlier chapters. The downloaded files include:

Before diving into finetuning the model in the next section, let's take a moment to assess the pretrained LLM's performance on one of the validation tasks by comparing its output to the expected response. This will give us a baseline understanding of how well the model performs on an instruction-following task right out of the box, prior to finetuning, and will help us appreciate the impact of finetuning later on. We use the first example from the validation set for this assessment:

The content of the instruction are as follows:

Instruction: 指令： 简体中文： 说明：

It's important to note that the generate function returns the combined input and output text. This behavior was convenient in previous chapters since pretrained LLMs are primarily designed as text-completion models, where the input and output are concatenated to create a coherent and legible text. However, when evaluating the model's performance on a specific task, we often want to focus solely on the model's generated response.

To isolate the model's response text, we need to subtract the length of the input instruction from the start of the generated_text:

This code snippet removes the input text from the beginning of the generated_text, leaving us with only the model's generated response. The strip() function is then applied to remove any leading or trailing whitespace characters. The output is as follows:

\#\#\# Response:

The chef cooks the meal every day.

\#\#\# Instruction:

Convert the active sentence to passive: 'The chef cooks the

As we can see from the output, the pretrained model is not yet capable of correctly following the given instruction. While it does create a "Response" section, it simply repeats the original input sentence and part of the instruction, failing to convert the active sentence to passive voice as requested.

In the upcoming section, we implement the finetuning process to improve the model's ability to comprehend and appropriately respond to such requests.

\subsection*{7.6 Finetuning the LLM on instruction data}

As illustrated in the chapter overview in figure 7.16, this section focuses on finetuning the LLM. We take the pretrained model loaded in the previous section and further train it using the instruction dataset prepared earlier in this chapter.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-277.jpg?height=573&width=1233&top_left_y=1312&top_left_x=317)

Figure 7.16 In step 5 of finetuning the LLM for instruction-following, we train the pretrained model loaded in the previous section on the instruction dataset prepared earlier in this chapter.

As mentioned earlier, we already did all the hard work when we implemented the instruction dataset processing at the beginning of this chapter. For the finetuning process itself, we can reuse the loss calculation and training functions implemented in chapter 5 during the pretraining:

Before we begin training, let's calculate the initial loss for the training and validation sets:

The initial loss values are as follows; as in previous chapters, our goal is to minimize this loss:

Training loss: 3.825908660888672

Validation loss: 3.7619335651397705

\section*{DEALING WITH HARDWARE LIMITATIONS}

Note that using and training a larger model like GPT-2 medium ( 355 million parameters) is more computationally intensive compared to the smaller GPT-2 model ( 124 million parameters) used in previous chapters. If you encounter issues due to hardware limitations, you can switch to the smaller model by changing CHOOSE_MODEL \(=\) "gpt2-medium (355M)" to CHOoSE_MODEL = "gpt2-small (124M)" in section 7.5. Alternatively, to speed up the model training, consider using a GPU. The following supplementary section in this book's code repository lists several options for using cloud GPUs: https://github.com/rasbt/LLMs-from-scratch/tree/main/setup

Table 7.1 provides reference runtimes for training each model on various devices, including CPUs and GPUs. Running this code on a compatible GPU requires no code changes and can significantly speed up training. For the results shown in this chapter, I used the GPT-2 medium model and trained it on an A100 GPU.

Table 7.1 Reference runtimes for instruction finetuning GPT-2

\begin{tabular}{|l|l|l|}
\hline Model name & Device & \begin{tabular}{l} 
Runtime for 2 \\
epochs
\end{tabular} \\
\hline gpt2-medium (355M) & CPU (M3 MacBook Air) & 15.78 minutes \\
\hline gpt2-medium (355M) & GPU (NVIDIA L4) & 1.83 minutes \\
\hline gpt2-medium (355M) & GPU (NVIDIA A100) & 0.86 minutes \\
\hline gpt2-small (124M) & CPU (M3 MacBook Air) & 5.74 minutes \\
\hline gpt2-small (124M) & GPU (NVIDIA L4) & 0.69 minutes \\
\hline gpt2-small (124M) & GPU (NVIDIA A100) & 0.39 minutes \\
\hline
\end{tabular}

With the model and data loaders prepared, we can now proceed to train the model. The following code sets up the training process, including initializing the optimizer, setting the number of epochs, and defining the evaluation frequency and starting context to evaluate generated LLM responses during training based on the first validation set instruction (val_data[0]) we looked at earlier:

\section*{Listing 7.8 Instruction finetuning the pretrained LLM}

The following output displays the training progress over two epochs, where a steady decrease in losses indicates improving ability to follow instructions and generate appropriate responses:

The training output shows that the model is learning effectively, as we can tell based on the consistently decreasing training and validation loss values over the two epochs. This suggests that the model is gradually improving its ability to understand and follow the provided instructions. (Since the model demonstrated effective learning within these two epochs, extending the training to a third epoch or more is not essential, and may even be counterproductive here as it could lead to increased overfitting.)

Moreover, the generated responses at the end of each epoch let us inspect the model's progress in correctly executing the given task in the validation set example. In this case, the model successfully converts the active sentence "The chef cooks the meal every day." into its passive voice counterpart: "The meal is cooked every day by the chef."

We will revisit and evaluate the response quality of the model in more detail in a later section. But now, to conclude this section, let's examine the training and validation loss curves to gain additional insights into the model's learning process. For this, we use the plot_losses function from chapter 5:

The resulting loss plot is shown in figure 7.17.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-282.jpg?height=709&width=1230&top_left_y=487&top_left_x=316)

Figure 7.17 A plot showing the training and validation loss trends over two epochs. The solid line represents the training loss, showing a sharp decrease before stabilizing, while the dotted line represents the validation loss, which follows a similar pattern.

As we can see in the loss plot shown in figure 7.17, the model's performance on both the training and validation sets improves substantially over the course of training. The rapid decrease in losses during the initial phase indicates that the model is quickly learning meaningful patterns and representations from the data. Then, as training progresses to the second epoch, the losses continue to decrease but at a slower rate, suggesting that the model is finetuning its learned representations and converging to a stable solution.

While the loss plot in figure 7.17 indicates that the model is training effectively, the most crucial aspect is its performance in terms of response quality and correctness. In the remaining sections of this chapter, we will extract the responses and store them in a format that allows us to evaluate and quantify the response quality.

\section*{EXERCISE 7.3 FINETUNING ON THE ORIGINAL ALPACA DATASET}

The so-called Alpaca dataset by researchers at Stanford is one of the earliest and most popular openly shared instruction datasets, consisting of 52,002 entries. As an alternative to the instruction-data.json file we use in this chapter, consider finetuning an LLM on this dataset. The dataset is available at the following URL: https://raw.githubusercontent.com/tatsu-lab/stanford alpaca/main/alpaca data.json

This dataset contains 52,002 entries, which is approximately 50 times more than those we used in this chapter, and most entries are longer as well. Thus, it's highly recommended to conduct the training using a GPU to accelerate the finetuning process. If you encounter out-of-memory errors, consider reducing the batch_size from 8 to 4,2 , or even 1. Additionally, lowering the allowed_max_length from 1024 to 512 or 256 can further help manage memory issues.

\subsection*{7.7 Extracting and saving responses}

After finetuning the LLM on the training portion of the instruction dataset as described in the previous section, we now proceed to evaluate its performance on the held-out test set. To accomplish this, we first extract the model-generated responses for each input in the test dataset and collect them for manual analysis as illustrated in the chapter overview in figure 7.18.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-284.jpg?height=890&width=1235&top_left_y=182&top_left_x=314)

Figure 7.18 This section is focused on extracting and collecting the model responses on the held-out test dataset for further analysis. The next section covers model evaluation to quantify the performance of the instruction-finetuned LLM.

We start with step 7, the response instruction step illustrated in figure 7.18 , using the generate function. We then print the model responses alongside the expected test set answers for the first three test set entries, presenting them side by side for comparison:

\#A Iterate over the first 3 test set samples

\#B Use the generate function imported in section 7.5

As mentioned earlier, the generate function returns the combined input and output text, so we use slicing and the .replace() method on the generated_text contents to extract the model's response. The instructions, followed by the given test set response and model response are shown below:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

\#\#\# Instruction:

Rewrite the sentence using a simile.

Input: 输入: 制定明确的商业计划非常重要,因为它可以帮助您确定目标并制定实现这些目标的具体步骤

The car is as fast as lightning.
这辆车速度如闪电一般。
Model response: 模型回应: 简体中文译文:

The car is as fast as a bullet.
这辆车像子弹一样快。

Below is an instruction that describes a task. Write a response that appropriately

©( Manning Publications Co. To comment go to liveBook

Licensed to lijin lijin <jimlee13894896995@gmail.com>
completes the request.

\#\#\# Instruction:

What type of cloud is typically associated with thunderstorms?

Correct response:

>> The type of cloud typically associated with thunderstorms is cumulonimbus.

Model response:

>> The type of cloud associated with thunderstorms is a cumulus cloud.

Below is an instruction that describes a task. Write a response that appropriately completes the request.

\#\#\# Instruction:

Name the author of 'Pride and Prejudice'.

Correct response:

\(>>\) Jane Austen.

Model response:

>> The author of 'Pride and Prejudice' is Jane Austen. \(\qquad\)
As we can see based on the test set instructions, given responses, and the model's responses, the model performs relatively well. The answers to the first and last instructions are clearly correct, while the second answer is close but not entirely accurate. The model answers with "cumulus cloud" instead of "cumulonimbus," although it's worth noting that cumulus clouds can develop into cumulonimbus clouds, which are capable of producing thunderstorms.

Most importantly, we can see that model evaluation is not as straightforward as in the previous chapter, where we simply calculated the percentage of correct spam/non-spam class labels to obtain the classification accuracy. In practice, instruction-finetuned LLMs such as chatbots are evaluated via multiple approaches:

1. Short-answer and multiple choice benchmarks such as MMLU ("Measuring Massive Multitask Language Understanding," https://arxiv.org/abs/2009. \(\underline{03300}\) ), which test the general knowledge of a model.

2. Human preference comparison to other LLMs, such as LMSYS chatbot arena (https://arena.Imsys.org.).

3. Automated conversational benchmarks, where another LLM like GPT-4 is used to evaluate the responses, such as AlpacaEval (https://tatsulab.github.io/alpaca eval/).

In practice, it can be useful to consider all three types of evaluation methods: multiplechoice question answering, human evaluation, and automated metrics that measure conversational performance. However, since we are primarily interested in assessing conversational performance in this chapter rather than just the ability to answer multiplechoice questions, methods 2 (human evaluation) and 3 (automated metrics) may be more relevant.

Human evaluation, while providing valuable insights, can be relatively laborious and time-consuming, especially when dealing with a large number of responses. For instance, reading and assigning ratings to all 1,100 responses would require a significant amount of effort.

So, considering the scale of the task at hand, we will implement an approach similar to method 3, which involves evaluating the responses automatically using another LLM. This will allow us to efficiently assess the quality of the generated responses without the need for extensive human involvement, thereby saving time and resources while still obtaining meaningful performance indicators.

In the next section, we employ an approach inspired by AlpacaEval, leveraging another LLM to evaluate our finetuned model's responses. However, instead of relying on a publicly available benchmark dataset, we use our own custom test set. This allows for a more targeted and relevant assessment of the model's performance within the context of our intended use cases represented in our instruction dataset.

To prepare the responses for this evaluation process, we append the generated model responses to the test_set dictionary and save the updated data as an "instructiondata-with-response.json" file for record keeping. Additionally, by saving this file, we can easily load and analyze the responses in separate Python sessions later on if needed.

The following code uses the generate method in the same manner as before; however, we now iterate over the entire test_set. Also, instead of printing the model responses, we add them to the test_set dictionary:

Listing 7.9 Generating test set responses

from tqdm import tqdm

for i, entry in tqdm(enumerate(test_data), total=len(test_data)):

input_text = format_input(entry)

token_ids = generate(

model=model, idx=text_to_token_ids(input_text, tokenizer).to(device), max_new_tokens=256, context_size=BASE_CONFIG["context_length"], eos_id \(=50256\)

)

generated_text = token_ids_to_text(token_ids, tokenizer)

response_text = generated_text[len(input_text):].replace("\#\#\# Response:",

"").strip()

test_data[i]["model_response"] = response_text

with open("instruction-data-with-response.json", "w") as file:

json.dump(test_data, file, indent=4) \# "indent" for pretty-printing

Processing the dataset takes about 1 minute on an A100 GPU and 6 minutes on an M3 MacBook Air:

\(100 \% \mid\)

\(110 / 110[01: 05<00: 00, \quad 1.68 \mathrm{it} / \mathrm{s}\) ]

Let's verify that the responses have been correctly added to the test_set dictionary by examining one of the entries:

print(test_data[0])

Based on the output, we can see that the model_response has been added correctly:

\{'instruction': 'Rewrite the sentence using a simile.', 'input': 'The car is very fast.', 'output': 'The car is as fast as lightning.', 'model_response': 'The car is as fast as a bullet.'\}

Finally, we save the model as gpt2-medium355M-sft.pth file to be able to reuse it in future projects:

Remove white spaces and parentheses from file name
从文件名中删除空格和括号

\subsection*{7.8 Evaluating the finetuned LLM}

Previously, we judged the performance of an instruction finetuned model by looking at its responses on 3 examples of the test set. While this gives us a rough idea of how well the model performs, this method does not really scale well to larger amounts of responses. So, in this section, as indicated in the chapter overview in figure 7.19, we implement a method to automate the response evaluation of the finetuned LLM using another, larger LLM.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-289.jpg?height=799&width=1249&top_left_y=939&top_left_x=302)

After extracting the responses by our finetuned LLM, we use another LLM to automatically evaluate these responses

Figure 7.19 In this last step of the instruction finetuning pipeline, we implement a method to quantify the performance of the finetuned model by scoring the responses it generated for the test.

To implement step 9 shown in figure 7.19, which involves evaluating test set responses in an automated fashion, we utilize an existing instruction-finetuned 8 billion parameter Llama 3 model developed by Meta AI. This model can be run locally using the open-source Ollama application (https://ollama.com).

Ollama is an efficient application for running LLMs on a laptop. It serves as a wrapper around the open-source llama.cpp library (https://github.com/ggerganov/llama.cpp), which implements LLMs in pure C/C++ to maximize efficiency. However, note that Ollama is only a tool for generating text using LLMs (inference) and does not support training or finetuning LLMs.

\section*{USING LARGER LLMS VIA WEB APIS}

The 8 billion parameter Llama 3 model is a very capable LLM that runs locally. However, it's not as capable as large proprietary LLMs such as GPT-4 offered by OpenAI. For readers interested in exploring how to utilize GPT-4 through the OpenAI API to assess generated model responses, an optional code notebook is available within the supplementary materials accompanying this book at https://github. com/rasbt/LLMs-from-scratch/blob/main/ch07/03 model-evaluation/llm-instructioneval-openai.ipynb

To execute the following code, please install Ollama by visiting https://ollama.com and following the provided instructions for your operating system:
- For macOS and Windows users: Open the downloaded Ollama application. If prompted to install command line usage, select "yes."
- \(\quad\) For Linux users: Use the installation command available on the Ollama website.

Before implementing the model evaluation code, let's first download the Llama 3 model and verify that Ollama is functioning correctly by using it from the command line terminal.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-291.jpg?height=620&width=1235&top_left_y=176&top_left_x=311)

Figure 7.20 Two options for running Ollama. The left panel illustrates starting Ollama using ollama serve. The right panel shows a second option in macOS, running the Ollama application in the background instead of using the ollama serve command to start the application.

With the Ollama application or ollama serve running in a different terminal as shown in figure 7.20, execute the following command on the command line (not in a Python session) to try out the 8 billion parameter Llama 3 model:

ollama run llama3

The first time you execute this command, the 8 billion parameter Llama 3 model, which takes up 4.7 GB of storage space, will be automatically downloaded. The output looks like as follows:

\section*{ALTERNATIVE OLLAMA MODELS}

Note that the llama3 in the ollama run llama3 command refers to the instructionfinetuned 8 billion parameter Llama 3 model. Using Ollama with the llama3 model requires approximately 16 GB of RAM. If your machine does not have sufficient RAM, you can try using a smaller model, such as the 3.8 billion parameter phi-3 model via ollama run llama3, which only requires around 8 GB of RAM.

For more powerful computers, you can also use the larger 70 billion parameter Llama 3 model by replacing llama3 with llama3:70b. However, keep in mind that this model requires significantly more computational resources.

Once the model download is complete, we are presented with a command-line interface that allows us to interact with the model. For example, try asking the model, "What do llamas eat?":

What do llamas eat? 骆驼吃什么?
Llamas are ruminant animals, which means they have a four-chambered
胡萝卜是反刍动物，这意味着它们有四个腔室
stomach and eat plants that are high in fiber. In the wild, llamas
胃和吃纤维含量高的植物。在野外,骆驼
typically feed on: 通常以下面为食:

1. Grasses: They love to graze on various types of grasses, including tall
   草类：它们爱在各种类型的草地上放牧,包括高草
   grasses, wheat, oats, and barley.
   草类、小麦、燕麦和大麦。

Note that the response you may be seeing might differ since Ollama is not deterministic as of this writing.

You can end this ollama run llama3 session using the input /bye. However, make sure to keep the ollama serve command or the Ollama application running for the remainder of this chapter.

The following code verifies that the Ollama session is running properly before we use Ollama to evaluate the test set responses generated in the previous section:

Ensure that the output from executing the previous code displays Ollama running: True. If it shows False, please verify that the ollama serve command or the Ollama application is actively running.

\section*{RUNNING THE CODE IN A NEW PYTHON SESSION}

If you closed your Python session after section 7.7 , or if you prefer to execute the remaining code in this chapter in different Python session, you execute the following code, which loads the instruction and response data file we created in section 7.7, and it redefines the format_input function we used earlier (the tqdm progress bar utility is used later):

An alternative to the ollama run command for interacting with the model is through its REST API using Python. The following query_model function demonstrates how to use the API :

Listing 7.10 Querying a local Ollama model

\#A Create the data payload as a dictionary
\#B Convert the dictionary to a JSON formatted string and encode it to bytes
\#C Create a request object, setting the method to POST and adding necessary headers
\#D Send the request and capture the response

Before running the subsequent code cells in this notebook, ensure that Ollama is still running. The previous code cells should print "Ollama running: True" to confirm that the model is active and ready to receive requests.

Here's an example of how to use the query_1lama function we just implemented:

The resulting response is as follows:

Llamas are ruminant animals, which means they have a four-chambered stomach that allows them to digest plant-based foods. Their diet typically consists of:

1. Grasses: Llamas love to graze on grasses, including tall grasses, short grasses, and even weeds.

...

Using the query_model function defined earlier, we can evaluate the responses generated by our finetuned model with a prompt that prompts the Llama 3 model to rate our finetuned model's responses on a scale from 0 to 100 based on the given test set response as reference.

First, we apply this approach to the first three examples from the test set that we examined in a previous section:

This prints outputs similar to the following ones (note that Ollama is not fully deterministic, as of this writing, so the generated texts may vary):

The car is as fast as lightning.
这辆车速度如闪电一般。
Model response: 模型回应: 简体中文译文:

The car is as fast as a bullet.
这辆车像子弹一样快。
Score: 分数:

A scoring task! 得分任务！
To evaluate the model response "The car is as fast as a bullet.", I'll consider how well
汽车和子弹一样快。

it follows the instruction and uses a simile that's coherent, natural-sounding, and effective in conveying the idea of speed.

Here are some factors to consider:

1. **Follows instruction**: Yes, the model uses a simile to rewrite the sentence.

2. **Coherence and naturalness**: The comparison between the car's speed and a bullet is common and easy to understand. It's a good choice for a simile that conveys the idea of rapid movement.

3. **Effectiveness in conveying idea of speed**: A bullet is known for its high velocity, which makes it an excellent choice to describe a fast-moving car.

Considering these factors, I'd score the model response "The car is as fast as a bullet." around 85 out of 100. The simile is well-chosen, coherent, and effectively conveys the idea of speed. Well done, model!

Dataset response:

>> The type of cloud typically associated with thunderstorms is cumulonimbus.

Model response:

>> The type of cloud associated with thunderstorms is a cumulus cloud.

Score:

>> A scoring task!

I'll evaluate the model's response based on its accuracy and relevance to the original instruction.

**Accuracy:** The model's response is partially correct. Cumulus clouds are indeed associated with fair weather and not typically linked to thunderstorms. The correct answer, cumulonimbus, is a type of cloud that is closely tied to thunderstorm formation.

**Relevance:** The model's response is somewhat relevant, as it mentions clouds in the context of thunderstorms. However, the specific type of cloud mentioned (cumulus) is not directly related to thunderstorms.

Considering these factors, I would score the model response a **40 out of \(100 * *\). While the response attempts to address the instruction, it provides an incorrect answer and lacks relevance to the original question.

Jane Austen. 简·奥斯汀。
Model response: 模型回应: 简体中文译文:

The author of 'Pride and Prejudice' is Jane Austen.
《傲慢与偏见》的作者是简·奥斯汀。
Score: 分数:

A simple one! 很简单!
My model response: "The author of 'Pride and Prejudice' is Jane Austen."
《傲慢与偏见》的作者是简·奥斯汀。
Score: 99 分数：99
Reasoning: 推理: 人工智能系统通过分析信息、制定假设并测试这些假设来推理。推理是一种基于已知信息得出新知识的过程。机器学习算法可以用于自动推理,通过从数据中学习发现模式和关系。推理是人工智能实现智能行为的关键能力之一

• The response directly answers the question, providing the correct name of the author.
  这个回答直接回答了该问题,提供了作者的正确姓名。
• The sentence structure is clear and easy to understand.
  句子结构清晰易懂。
• There's no room for misinterpretation or ambiguity.
  没有误解或模糊的余地。
  Overall, a perfect score!
  总的来说,满分!

Based on the generated responses, we can observe that the Llama 3 model provides reasonable evaluations and is capable of assigning partial points when a model's answer is not entirely correct. For instance, if we consider the evaluation of the "cumulus cloud" answer, the model acknowledges the partial correctness of the response.

The previous prompt returns highly detailed evaluations in addition to the score. We can modify the prompt to just generate integer scores ranging from 0 to 100 , where 100 represents the best possible score. This modification allows us to calculate an average score for our model, which serves as a more concise and quantitative assessment of its performance.

The following generate_model_scores function uses a modified the prompt telling the model to "Respond with the integer number only.":

\section*{Listing 7.11 Evaluating the instruction finetuning LLM}

return scores

\#A Modified instruction line to only return the score

The results are as follows:

The evaluation output shows that our finetuned model achieves an average score above 50, which provides a useful benchmark for comparison against other models or for experimenting with different training configurations to improve the model's performance.

It's worth noting that Ollama is not entirely deterministic at the time of this writing, which means that the scores you obtain might slightly vary from the ones presented above. To obtain more robust results, you can repeat the evaluation multiple times and average the resulting scores.

To further improve our model's performance, we can explore various strategies, such as:
- Adjusting the hyperparameters during finetuning, such as the learning rate, batch size, or number of epochs.
- Increasing the size of the training dataset or diversifying the examples to cover a broader range of topics and styles.
- Experimenting with different prompts or instruction formats to guide the model's responses more effectively.
- \(\quad\) Considering the use of a larger pretrained model, which may have greater capacity to capture complex patterns and generate more accurate responses.

\section*{PERFORMANCE OF LLAMA 3 MODELS}

For reference, when using the methodology described in this section, the Llama 38 B base model, without any finetuning, achieves an average score of 58.51 on the test set. The Llama 38 B instruct model, which has been finetuned on a general instruction-following dataset, achieves an impressive average score of 82.6.

\section*{EXERCISE 7.4 PARAMETER-EFFICIENT FINETUNING WITH LORA}

To instruction finetune an LLM more efficiently, modify the code in this chapter to use the low-rank adaptation method (LoRA) from appendix E. Compare the training runtime and model performance before and after the modification.

\subsection*{7.9 Conclusions}

This chapter marks the conclusion of our journey through the LLM development cycle. We have covered all the essential steps, including implementing an LLM architecture, pretraining an LLM, and finetuning it for specific tasks, as summarized in figure 7.21.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-301.jpg?height=578&width=1252&top_left_y=180&top_left_x=298)

Figure 7.21 An overview of the different stages of implementing, pretraining, and finetuning an LLM covered in this book.

The next subsection will give you some ideas for what to look into next after the essential steps shown in figure 7.21 .

\subsection*{7.9.1 What's next?}

While we covered the most essential steps, as illustrated in figure 7.21 , there is an optional step that can be performed after instruction finetuning: preference finetuning. Preference finetuning is particularly useful for customizing a model to better align with specific user preferences. If you are interested in exploring this further, please refer to the 04 _preference-tuning-with-dpo folder in this book's supplementary GitHub repository at https://github.com/rasbt/LLMs-from-scratch/tree/main/ch07/04 preference-tuning-withdpo.

In addition to the main content covered in this book, the GitHub repository also contains a large selection bonus material that you may find valuable. To learn more about these additional resources, please visit the Bonus Material section on the repository's README page: https://github.com/rasbt/LLMs-from-scratch?tab=readme-ov-file\#bonus-material.

\subsection*{7.9.2 Staying up to date in a fast-moving field}

The fields of AI and LLM research are evolving at a rapid (and depending on who you ask, exciting) pace. One way to keep up with the latest advancements, consider exploring recent research papers on arXiv at https://arxiv.org/list/cs.LG/recent. Additionally, many researchers and practitioners are very active in sharing and discussing the latest developments on social media platforms like X (formerly Twitter) and Reddit. The subreddit r/LocalLLaMA, in particular, is a good resource for connecting with the community and staying informed about the latest tools and trends.

I also regularly share insights and write about the latest in LLM research on my blog, available at https://magazine.sebastianraschka.com and https://sebastianraschka. com/blog \(L\).

\subsection*{7.9.3 Final words}

I hope you have enjoyed this journey of implementing an LLM from the ground up and coding the pretraining and finetuning functions from scratch. In my opinion, building an LLM from scratch is the most effective way to gain a deep understanding of how LLMs work. I hope that this hands-on approach has provided you with valuable insights and a solid foundation in LLM development.

While the primary purpose of this book is educational, you may be interested in utilizing different and more powerful LLMs for real-world applications. For this, I recommend exploring popular tools such as Axolotl (https://github.com/OpenAccess-AI-Collective/ axolotl) or LitGPT (https://github.com/Lightning-AI/litg.pt), which I am actively involved in developing.

Thank you for joining me on this learning journey, and I wish you all the best in your future endeavors in the exciting field of LLMs and AI!

\subsection*{7.10 Summary}
- The instruction finetuning process adapts a pretrained LLM to follow human instructions and generate desired responses.
- \(\quad\) Preparing the dataset involves downloading an instruction-response dataset, formatting the entries, and splitting it into train, validation and test sets.
- Training batches are constructed using a custom collate function that pads sequences, creates target token IDs, and masks padding tokens.
- We load a pretrained GPT-2 medium model with 355M parameters to serve as the starting point for instruction finetuning.
- The pretrained model is finetuned on the instruction dataset using a training loop similar to pretraining.
- Evaluation involves extracting model responses on a test set and scoring them, e.g. using another LLM.
- \(\quad\) The Ollama application with an 8B parameter Llama model can be used to automatically score the finetuned model's responses on the test set, providing an average score to quantify performance.

\section*{Appendix A. Introduction to PyTorch}

\section*{Thls chapter covers}
- An overview of the PyTorch deep learning library
- Setting up an environment and workspace for deep learning
- Tensors as a fundamental data structure for deep learning
- The mechanics of training deep neural networks
- Training models on GPUs

This chapter is designed to equip you with the necessary skills and knowledge to put deep learning into practice and implement large language models (LLMs) from scratch.

We will introduce PyTorch, a popular Python-based deep learning library, which will be our primary tool for the remainder of this book. This chapter will also guide you through setting up a deep learning workspace armed with PyTorch and GPU support.

Then, you'll learn about the essential concept of tensors and their usage in PyTorch. We will also delve into PyTorch's automatic differentiation engine, a feature that enables us to conveniently and efficiently use backpropagation, which is a crucial aspect of neural network training.

Note that this chapter is meant as a primer for those who are new to deep learning in PyTorch. While this chapter explains PyTorch from the ground up, it's not meant to be an exhaustive coverage of the PyTorch library. Instead, this chapter focuses on the PyTorch fundamentals that we will use to implement LLMs throughout this book. If you are already familiar with deep learning, you may skip this appendix and directly move on to chapter 2, working with text data.

\section*{A. 1 What is PyTorch}

PyTorch (https://pytorch.orgL) is an open-source Python-based deep learning library. According to Papers With Code (https://paperswithcode.com/trends), a platform that tracks and analyzes research papers, PyTorch has been the most widely used deep learning library for research since 2019 by a wide margin. And according to the Kaggle Data Science and Machine Learning Survey 2022 (https://www.kaggle.com/c/kaggle-survey-2022), the number of respondents using PyTorch is approximately \(40 \%\) and constantly grows every year.

One of the reasons why PyTorch is so popular is its user-friendly interface and efficiency. However, despite its accessibility, it doesn't compromise on flexibility, providing advanced users the ability to tweak lower-level aspects of their models for customization and optimization. In short, for many practitioners and researchers, PyTorch offers just the right balance between usability and features.

In the following subsections, we will define the main features PyTorch has to offer.

\section*{A.1.1 The three core components of PyTorch}

PyTorch is a relatively comprehensive library, and one way to approach it is to focus on its three broad components, which are summarized in figure A.1.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-304.jpg?height=691&width=1242&top_left_y=1116&top_left_x=301)

Figure A.1 PyTorch's three main components include a tensor library as a fundamental building block for computing, automatic differentiation for model optimization, and deep learning utility functions, making it easier to implement and train deep neural network models.

Firstly, PyTorch is a tensor library that extends the concept of array-oriented programming library NumPy with the additional feature of accelerated computation on GPUs, thus providing a seamless switch between CPUs and GPUs.

Secondly, PyTorch is an automatic differentiation engine, also known as autograd, which enables the automatic computation of gradients for tensor operations, simplifying backpropagation and model optimization.

Finally, PyTorch is a deep learning library, meaning that it offers modular, flexible, and efficient building blocks (including pre-trained models, loss functions, and optimizers) for designing and training a wide range of deep learning models, catering to both researchers and developers.

After defining the term deep learning and installing PyTorch in the two following subsections, the remainder of this chapter will go over these three core components of PyTorch in more detail, along with hands-on code examples.

\section*{A.1.2 Defining deep learning}

LLMs are often referred to as AI models in the news. However, as illustrated in the first section of chapter 1 (1.1 What is an LLM?) LLMs are also a type of deep neural network, and PyTorch is a deep learning library. Sounds confusing? Let's take a brief moment and summarize the relationship between these terms before we proceed.

AI is fundamentally about creating computer systems capable of performing tasks that usually require human intelligence. These tasks include understanding natural language, recognizing patterns, and making decisions. (Despite significant progress, AI is still far from achieving this level of general intelligence.)

Machine learning represents a subfield of AI (as illustrated in figure A.2) that focuses on developing and improving learning algorithms. The key idea behind machine learning is to enable computers to learn from data and make predictions or decisions without being explicitly programmed to perform the task. This involves developing algorithms that can identify patterns and learn from historical data and improve their performance over time with more data and feedback.

\section*{Artificial intelligence (Al)}

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-306.jpg?height=646&width=1169&top_left_y=332&top_left_x=337)

Figure A. 2 Deep learning is a subcategory of machine learning that is focused on the implementation of deep neural networks. In turn, machine learning is a subcategory of Al that is concerned with algorithms that learn from data. Al is the broader concept of machines being able to perform tasks that typically require human intelligence.

Machine learning has been integral in the evolution of AI, powering many of the advancements we see today, including LLMs. Machine learning is also behind technologies like recommendation systems used by online retailers and streaming services, email spam filtering, voice recognition in virtual assistants, and even self-driving cars. The introduction and advancement of machine learning have significantly enhanced AI's capabilities, enabling it to move beyond strict rule-based systems and adapt to new inputs or changing environments.

Deep learning is a subcategory of machine learning that focuses on the training and application of deep neural networks. These deep neural networks were originally inspired by how the human brain works, particularly the interconnection between many neurons. The "deep" in deep learning refers to the multiple hidden layers of artificial neurons or nodes that allow them to model complex, nonlinear relationships in the data.

Unlike traditional machine learning techniques that excel at simple pattern recognition, deep learning is particularly good at handling unstructured data like images, audio, or text, so deep learning is particularly well suited for LLMs.

The typical predictive modeling workflow (also referred to as supervised learning) in machine learning and deep learning is summarized in figure A.3.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-307.jpg?height=839&width=1235&top_left_y=189&top_left_x=314)

Once a model is trained, we can use it to predict the labels of new data

Figure A. 3 The supervised learning workflow for predictive modeling consists of a training stage where a model is trained on labeled examples in a training dataset. The trained model can then be used to predict the labels of new observations.

Using a learning algorithm, a model is trained on a training dataset consisting of examples and corresponding labels. In the case of an email spam classifier, for example, the training dataset consists of emails and their spam and not-spam labels that a human identified. Then, the trained model can be used on new observations (new emails) to predict their unknown label (spam or not spam).

Of course, we also want to add a model evaluation between the training and inference stages to ensure that the model satisfies our performance criteria before using it in a realworld application.

Note that the workflow for training and using LLMs, as we will see later in this book, is similar to the workflow depicted in figure A. 3 if we train them to classify texts. And if we are interested in training LLMs for generating texts, which is the main focus of this book, figure A. 3 still applies. In this case, the labels during pretraining can be derived from the text itself (the next-word prediction task introduced in chapter 1). And the LLM will generate entirely new text (instead of predicting labels) given an input prompt during inference.

\section*{A.1.3 Installing PyTorch}

PyTorch can be installed just like any other Python library or package. However, since PyTorch is a comprehensive library featuring CPU- and GPU-compatible codes, the installation may require additional explanation.

\section*{PYTHON VERSION}

Many scientific computing libraries do not immediately support the newest version of Python. Therefore, when installing PyTorch, it's advisable to use a version of Python that is one or two releases older. For instance, if the latest version of Python is 3.13, using Python 3.10 or 3.11 is recommended.

For instance, there are two versions of PyTorch: a leaner version that only supports CPU computing and a version that supports both CPU and GPU computing. If your machine has a CUDA-compatible GPU that can be used for deep learning (ideally an NVIDIA T4, RTX 2080 Ti , or newer), I recommend installing the GPU version. Regardless, the default command for installing PyTorch is as follows in a code terminal:

pip install torch

Suppose your computer supports a CUDA-compatible GPU. In that case, this will automatically install the PyTorch version that supports GPU acceleration via CUDA, given that the Python environment you're working on has the necessary dependencies (like pip) installed.

\section*{AMD GPUS FOR DEEP LEARNING}

As of this writing, PyTorch has also added experimental support for AMD GPUs via ROCm. Please see https://pytorch.org for additional instructions.

However, to explicitly install the CUDA-compatible version of PyTorch, it's often better to specify the CUDA you want PyTorch to be compatible with. PyTorch's official website (https://pytorch.org.) provides commands to install PyTorch with CUDA support for different operating systems as shown in figure A.4.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-309.jpg?height=551&width=1188&top_left_y=189&top_left_x=358)

Figure A. 4 Access the PyTorch installation recommendation on https://pytorch.org to customize and select the installation command for your system.

(Note that the command shown in figure A. 4 will also install the torchvision and torchaudio libraries, which are optional for this book.)

As of this writing, this book is based on PyTorch 2.0.1, so it's recommended to use the following installation command to install the exact version to guarantee compatibility with this book:

pip install torch \(==2.0 .1\)

However, as mentioned earlier, given your operating system, the installation command might slightly differ from the one shown above. Thus, I recommend visiting the https://pytorch.org website and using the installation menu (see figure A4) to select the installation command for your operating system and replace torch with torch==2.0.1 in this command.

To check the version of PyTorch, you can execute the following code in PyTorch:

This prints:

\[
' 2.0 .1^{\prime}
\]

\section*{PYTORCH AND TORCH}

Note that the Python library is named "torch" primarily because it's a continuation of the Torch library but adapted for Python (hence, "PyTorch"). The name "torch" acknowledges the library's roots in Torch, a scientific computing framework with wide support for machine learning algorithms, which was initially created using the Lua programming language.

If you are looking for additional recommendations and instructions for setting up your Python environment or installing the other libraries used later in this book, I recommend visiting the supplementary GitHub repository of this book at https://github.com/rasbt/ LLMs-from-scratch.

After installing PyTorch, you can check whether your installation recognizes your built-in NVIDIA GPU by running the following code in Python:

This returns:

True

If the command returns True, you are all set. If the command returns False, your computer may not have a compatible GPU, or PyTorch does not recognize it. While GPUs are not required for the initial chapters in this book, which are focused on implementing LLMs for educational purposes, they can significantly speed up deep learning-related computations.

If you don't have access to a GPU, there are several cloud computing providers where users can run GPU computations against an hourly cost. A popular Jupyter-notebook-like environment is Google Colab (https://colab.research.google.com), which provides timelimited access to GPUs as of this writing. Using the "Runtime" menu, it is possible to select a GPU, as shown in the screenshot in figure A.5.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-311.jpg?height=521&width=1228&top_left_y=187&top_left_x=319)

Figure A. 5 Select a GPU device for Google Colab under the Runtime/Change runtime type menu.

\section*{PYTORCH ON APPLE SILICON}

If you have an Apple Mac with an Apple Silicon chip (like the M1, M2, M3, or newer models), you have the option to leverage its capabilities to accelerate PyTorch code execution. To use your Apple Silicon chip for PyTorch, you first need to install PyTorch as you normally would. Then, to check if your Mac supports PyTorch acceleration with its Apple Silicon chip, you can run a simple code snippet in Python:

print(torch.backends.mps.is_available())

If it returns True, it means that your Mac has an Apple Silicon chip that can be used to accelerate PyTorch code.

\section*{EXERCISE A. 1}

Install and set up PyTorch on your computer.

\section*{EXERCISE A. 2}

Run the supplementary Chapter 2 code at https://github.com/rasbt/LLMs-fromscratch that checks whether your environment is set up correctly..

\section*{A. 2 Understanding tensors}

Tensors represent a mathematical concept that generalizes vectors and matrices to potentially higher dimensions. In other words, tensors are mathematical objects that can be characterized by their order (or rank), which provides the number of dimensions. For example, a scalar (just a number) is a tensor of rank 0 , a vector is a tensor of rank 1 , and a matrix is a tensor of rank 2, as illustrated in figure A. 6

\section*{An example of a 3D vector that}

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-312.jpg?height=558&width=311&top_left_y=619&top_left_x=356)

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-312.jpg?height=568&width=212&top_left_y=607&top_left_x=731)

A matrix with 3 rows and 4 columns

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-312.jpg?height=513&width=322&top_left_y=660&top_left_x=1046)

Figure A.6 An illustration of tensors with different ranks. Here OD corresponds to rank \(0,1 \mathrm{D}\) to rank 1, and 2D to rank 2. Note that a 3D vector, which consists of 3 elements, is still a rank 1 tensor.

From a computational perspective, tensors serve as data containers. For instance, they hold multi-dimensional data, where each dimension represents a different feature. Tensor libraries, such as PyTorch, can create, manipulate, and compute with these multidimensional arrays efficiently. In this context, a tensor library functions as an array library.

PyTorch tensors are similar to NumPy arrays but have several additional features important for deep learning. For example, PyTorch adds an automatic differentiation engine, simplifying computing gradients, as discussed later in section 2.4. PyTorch tensors also support GPU computations to speed up deep neural network training, which we will discuss later in section 2.8.

\section*{PYTORCH'S HAS A NUMPY-LIKE API}

As you will see in the upcoming sections, PyTorch adopts most of the NumPy array API and syntax for its tensor operations. If you are new to NumPy, you can get a brief overview of the most relevant concepts via my article Scientific Computing in Python: Introduction to NumPy and Matplotlib at https://sebastianraschka.com/blogL \(\underline{2020 / n u m p y-\text {-intro.html. }}\)

The following subsections will look at the basic operations of the PyTorch tensor library, showing how to create simple tensors and going over some of the essential operations.

\section*{A.2.1 Scalars, vectors, matrices, and tensors}

As mentioned earlier, PyTorch tensors are data containers for array-like structures. A scalar is a 0 -dimensional tensor (for instance, just a number), a vector is a 1-dimensional tensor, and a matrix is a 2-dimensional tensor. There is no specific term for higher-dimensional tensors, so we typically refer to a 3-dimensional tensor as just a 3D tensor, and so forth.

We can create objects of PyTorch's Tensor class using the torch.tensor function as follows:

Listing A. 1 Creating PyTorch tensors

\section*{A.2.2 Tensor data types}

In the previous section, we created tensors from Python integers. In this case, PyTorch adopts the default 64-bit integer data type from Python. We can access the data type of a tensor via the .dtype attribute of a tensor:

This prints:

torch.int64

If we create tensors from Python floats, PyTorch creates tensors with a 32-bit precision by default, as we can see below:

The output is:

This choice is primarily due to the balance between precision and computational efficiency. A 32-bit floating point number offers sufficient precision for most deep learning tasks, while consuming less memory and computational resources than a 64-bit floating point number. Moreover, GPU architectures are optimized for 32-bit computations, and using this data type can significantly speed up model training and inference.

Moreover, it is possible to readily change the precision using a tensor's .to method. The following code demonstrates this by changing a 64-bit integer tensor into a 32-bit float tensor:

This returns:

\section*{torch.float32}

For more information about different tensor data types available in PyTorch, I recommend checking the official documentation at https://pytorch.org/docs/stable/tensors.html.

\section*{A.2.3 Common PyTorch tensor operations}

Comprehensive coverage of all the different PyTorch tensor operations and commands is outside the scope of this book. However, we will briefly describe relevant operations as we introduce them throughout the book.

Before we move on to the next section covering the concept behind computation graphs, below is a list of the most essential PyTorch tensor operations.

We already introduced the torch.tensor() function to create new tensors.

This prints:

\[
\begin{aligned}
& \text { tensor }([[1,2,3] \\
& [4,5,6]])
\end{aligned}
\]

In addition, the . shape attribute allows us to access the shape of a tensor:

print(tensor2d.shape)

The output is:

torch.Size( \([2,3])\)

As you can see above, .shape returns \([2,3]\), which means that the tensor has 2 rows and 3 columns. To reshape the tensor into a 3 by 2 tensor, we can use the .reshape method:

This prints:

tensor \(([1,2]\),

\([3,4]\),

\([5,6]])\)

However, note that the more common command for reshaping tensors in PyTorch is .view():

print(tensor2d.view(3, 2))

The output is:

tensor([[1, 2], \([3,4]\), \([5,6]])\)

Similar to .reshape and .view, there are several cases where PyTorch offers multiple syntax options for executing the same computation. This is because PyTorch initially followed the original Lua Torch syntax convention but then also added syntax to make it more similar to NumPy upon popular request.

Next, we can use .T to transpose a tensor, which means flipping it across its diagonal. Note that this is similar from reshaping a tensor as you can see based on the result below:

print(tensor2d.T)

The output is:

tensor([[1, 4],

\([2,5]\),

\([3,6]]\) )

Lastly, the common way to multiply two matrices in PyTorch is the .matmul method:

print(tensor2d.matmul(tensor2d.T))

The output is:

tensor([[14, 32],

\([32,77]])\)

However, we can also adopt the @ operator, which accomplishes the same thing more compactly:

print(tensor2d @ tensor2d.T)

This prints:

tensor([[14, 32],

\([32,77]])\)

As mentioned earlier, we will introduce additional operations throughout this book when needed. For readers who'd like to browse through all the different tensor operations available in PyTorch (hint: we won't need most of these), I recommend checking out the official documentation at https://pytorch.org/docs/stable/tensors.html.

\section*{A. 3 Seeing models as computation graphs}

In the previous section, we covered one of the major three components of PyTorch, namely, its tensor library. Next in line is PyTorch's automatic differentiation engine, also known as autograd. PyTorch's autograd system provides functions to compute gradients in dynamic computational graphs automatically. But before we dive deeper into computing gradients in the next section, let's define the concept of a computational graph.

A computational graph (or computation graph in short) is a directed graph that allows us to express and visualize mathematical expressions. In the context of deep learning, a computation graph lays out the sequence of calculations needed to compute the output of a neural network -- we will need this later to compute the required gradients for backpropagation, which is the main training algorithm for neural networks.

Let's look at a concrete example to illustrate the concept of a computation graph. The following code implements the forward pass (prediction step) of a simple logistic regression classifier, which can be seen as a single-layer neural network, returning a score between 0 and 1 that is compared to the true class label ( 0 or 1 ) when computing the loss:

\section*{Listing A. 2 A logistic regression forward pass}

\#A
\(y=\) torch.tensor([1.0]) ..... \#B
x1 = torch.tensor([1.1]) ..... \#C
w1 = torch.tensor([2.2])
\#D
\(b=\) torch.tensor([0.0]) ..... \#E
\(z=x 1 * w 1+b\) ..... \(\# F\)
\(\mathrm{a}=\) torch.sigmoid(z) ..... \#G
loss = F.binary_cross_entropy(a, y)
\#A This import statement is a common convention in PyTorch to prevent long lines of code
\#B true label
\#C input feature
\#D weight parameter
\#E bias unit
\#F net input
\#G activation \& output

If not all components in the code above make sense to you, don't worry. The point of this example is not to implement a logistic regression classifier but rather to illustrate how we can think of a sequence of computations as a computation graph, as shown in figure A.7.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-318.jpg?height=436&width=1254&top_left_y=352&top_left_x=295)

Figure A. 7 A logistic regression forward pass as a computation graph. The input feature \(x_{1}\) is multiplied by a model weight \(w_{1}\) and passed through an activation function \(\sigma\) after adding the bias. The loss is computed by comparing the model output \(a\) with a given label \(y\).

In fact, PyTorch builds such a computation graph in the background, and we can use this to calculate gradients of a loss function with respect to the model parameters (here w1 and b) to train the model, which is the topic of the upcoming sections.

\section*{A. 4 Automatic differentiation made easy}

In the previous section, we introduced the concept of computation graphs. If we carry out computations in PyTorch, it will build such a graph internally by default if one of its terminal nodes has the requires grad attribute set to True. This is useful if we want to compute gradients. Gradients are required when training neural networks via the popular backpropagation algorithm, which can be thought of as an implementation of the chain rule from calculus for neural networks, which is illustrated in figure A.8.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-319.jpg?height=776&width=1209&top_left_y=180&top_left_x=338)

Figure A. 8 The most common way of computing the loss gradients in a computation graph involves applying the chain rule from right to left, which is also called reverse-model automatic differentiation or backpropagation. It means we start from the output layer (or the loss itself) and work backward through the network to the input layer. This is done to compute the gradient of the loss with respect to each parameter (weights and biases) in the network, which informs how we update these parameters during training.

\section*{PARTIAL DERIVATIVES AND GRADIENTS}

Figure A. 8 shows partial derivatives, which measure the rate at which a function changes with respect to one of its variables. A gradient is a vector containing all of the partial derivatives of a multivariate function, a function with more than one variable as input.

If you are not familiar or don't remember the partial derivatives, gradients, or the chain rule from calculus, don't worry. On a high level, all you need to know for this book is that the chain rule is a way to compute gradients of a loss function with respect to the model's parameters in a computation graph. This provides the information needed to update each parameter in a way that minimizes the loss function, which serves as a proxy for measuring the model's performance, using a method such as gradient descent. We will revisit the computational implementation of this training loop in PyTorch in section 2.7, A typical training loop.

Now, how is this all related to the second component of the PyTorch library we mentioned earlier, the automatic differentiation (autograd) engine? By tracking every operation performed on tensors, PyTorch's autograd engine constructs a computational graph in the background. Then, calling the grad function, we can compute the gradient of the loss with respect to model parameter w1 as follows:

Listing A. 3 Computing gradients via autograd

import torch.nn.functional as F

from torch.autograd import grad

\(y=\) torch.tensor([1.0])

x1 = torch.tensor([1.1])

w1 = torch.tensor([2.2], requires_grad=True)

\(b=\) torch.tensor([0.0], requires_grad=True)

\(z=x 1 * w 1+b\)

\(\mathrm{a}=\) torch.sigmoid(z)

loss = F.binary_cross_entropy(a, y)

grad_L_w1 = grad(loss, w1, retain_graph=True)

\#A

grad_L_b = grad(loss, b, retain_graph=True)

\#A By default, PyTorch destroys the computation graph after calculating the gradients to free memory.

However, since we are going to reuse this computation graph shortly, we set retain_graph=True so that it stays in memory.

Let's show the resulting values of the loss with respect to the model's parameters:

print(grad_L_w1)

print(grad_L_b)

The prints:

Above, we have been using the grad function "manually," which can be useful for experimentation, debugging, and demonstrating concepts. But in practice, PyTorch provides even more high-level tools to automate this process. For instance, we can call .backward on the loss, and PyTorch will compute the gradients of all the leaf nodes in the graph, which will be stored via the tensors' .grad attributes:

The outputs are:

(tensor([-0.0898]), )

(tensor( \([-0.0817])\), )

If this section is packed with a lot of information and you may be overwhelmed by the calculus concepts, don't worry. While this calculus jargon was a means to explain PyTorch's autograd component, all you need to take away from this section is that PyTorch takes care of the calculus for us via the .backward method -- we won't need to compute any derivatives or gradients by hand in this book.

\section*{A. 5 Implementing multilayer neural networks}

In the previous sections, we covered PyTorch's tensor and autograd components. This section focuses on PyTorch as a library for implementing deep neural networks.

To provide a concrete example, we focus on a multilayer perceptron, which is a fully connected neural network, as illustrated in figure A.9.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-322.jpg?height=1091&width=1235&top_left_y=171&top_left_x=316)

Figure A. 9 An illustration of a multilayer perceptron with 2 hidden layers. Each node represents a unit in the respective layer. Each layer has only a very small number of nodes for illustration purposes.

When implementing a neural network in PyTorch, we typically subclass the torch.nn.Module class to define our own custom network architecture. This Module base class provides a lot of functionality, making it easier to build and train models. For instance, it allows us to encapsulate layers and operations and keep track of the model's parameters.

Within this subclass, we define the network layers in the \(\qquad\) init constructor and specify how they interact in the forward method. The forward method describes how the input data passes through the network and comes together as a computation graph.

In contrast, the backward method, which we typically do not need to implement ourselves, is used during training to compute gradients of the loss function with respect to the model parameters, as we will see in section 2.7, A typical training loop.

The following code implements a classic multilayer perceptron with two hidden layers to illustrate a typical usage of the Module class:

\section*{Listing A. 4 A multilayer perceptron with two hidden layers}

\#E

\#A It's useful to code the number of inputs and outputs as variables to reuse the same code for datasets with different numbers of features and classes.

\#B The Linear layer takes the number of input and output nodes as arguments.

\#C Nonlinear activation functions are placed between the hidden layers.

\#D The number of output nodes of one hidden layer has to match the number of inputs of the next layer.

\#E The outputs of the last layer are called logits.

We can then instantiate a new neural network object as follows:

model \(=\) NeuralNetwork( 50,3\()\)

But before using this new model object, it is often useful to call print on the model to see a summary of its structure:

print(model)

This prints:

Note that we used the Sequential class when we implemented the NeuralNetwork class. Using Sequential is not required, but it can make our life easier if we have a series of layers that we want to execute in a specific order, as is the case here. This way, after instantiating self.layers = Sequential(...) in the __init__ constructor, we just have to call the self.layers instead of calling each layer individually in the NeuralNetwork's forward method.

Next, let's check the total number of trainable parameters of this model:

This prints:

Total number of trainable model parameters: 2213

Note that each parameter for which requires_grad=True counts as a trainable parameter and will be updated during training (more on that later in section 2.7, A typical training loop).

In the case of our neural network model with the two hidden layers above, these trainable parameters are contained in the torch.nn. Linear layers. A linear layer multiplies the inputs with a weight matrix and adds a bias vector. This is sometimes also referred to as a feedforward or fully connected layer.

Based on the print(model) call we executed above, we can see that the first Linear layer is at index position 0 in the layers attribute. We can access the corresponding weight parameter matrix as follows:

This prints:

Since this is a large matrix that is not shown in its entirety, let's use the . shape attribute to show its dimensions:

print(model.layers[0].weight.shape)

The result is:

torch.Size( \([30,50])\)

(Similarly, you could access the bias vector via model. layers [0] .bias.)

The weight matrix above is a \(30 \times 50\) matrix, and we can see that the requires_grad is set to True, which means its entries are trainable -- this is the default setting for weights and biases in torch.nn. Linear.

Note that if you execute the code above on your computer, the numbers in the weight matrix will likely differ from those shown above. This is because the model weights are initialized with small random numbers, which are different each time we instantiate the network. In deep learning, initializing model weights with small random numbers is desired to break symmetry during training -- otherwise, the nodes would be just performing the same operations and updates during backpropagation, which would not allow the network to learn complex mappings from inputs to outputs.

However, while we want to keep using small random numbers as initial values for our layer weights, we can make the random number initialization reproducible by seeding PyTorch's random number generator via manual_seed:

The result is:

\[
\begin{aligned}
& \text { Parameter containing: } \\
& \text { tensor([[-0.0577, } 0.0047,-0.0702, \ldots, 0.0222,0.1260,0.0865] \text {, } \\
& {[0.0502,0.0307,0.0333, \ldots, \quad 0.0951,0.1134,-0.0297]} \\
& {[0.1077,-0.1108,0.0122, \ldots, 0.0108,-0.1049,-0.1063],} \\
& \ldots \text {, } \\
& {[-0.0787,0.1259,0.0803, \ldots, \quad 0.1218,0.1303,-0.1351]} \\
& {[0.1359,0.0175,-0.0673, \ldots, \quad 0.0674,0.0676,0.1058] \text {, }} \\
& [0.0790,0.1343,-0.0293, \ldots, 0.0344,-0.0971,-0.0509]], \\
& \text { requires_grad=True) }
\end{aligned}
\]

Now, after we spent some time inspecting the NeuraNetwork instance, let's briefly see how it's used via the forward pass:

The result is:tensor([[-0.1262, 0.1080, -0.1792]], grad_fn=<AddmmBackward0>)

In the code above, we generated a single random training example x as a toy input (note that our network expects 50 -dimensional feature vectors) and fed it to the model, returning three scores. When we call model \((\mathrm{x})\), it will automatically execute the forward pass of the model.

The forward pass refers to calculating output tensors from input tensors. This involves passing the input data through all the neural network layers, starting from the input layer, through hidden layers, and finally to the output layer.

These three numbers returned above correspond to a score assigned to each of the three output nodes. Notice that the output tensor also includes a grad_fn value.

Here, grad_fn=<AddmmBackward0> represents the last-used function to compute a variable in the computational graph. In particular, grad_fn=<AddmmBackward0> means that the tensor we are inspecting was created via a matrix multiplication and addition operation. PyTorch will use this information when it computes gradients during backpropagation. The <AddmmBackward0> part of grad_fn=<AddmmBackward0> specifies the operation that was performed. In this case, it is an Addmm operation. Addmm stands for matrix multiplication \((\mathrm{mm}\) ) followed by an addition (Add).

If we just want to use a network without training or backpropagation, for example, if we use it for prediction after training, constructing this computational graph for backpropagation can be wasteful as it performs unnecessary computations and consumes additional memory. So, when we use a model for inference (for instance, making predictions) rather than training, it is a best practice to use the torch.no_grad() context manager, as shown below. This tells PyTorch that it doesn't need to keep track of the gradients, which can result in significant savings in memory and computation.

The result is:

In PyTorch, it's common practice to code models such that they return the outputs of the last layer (logits) without passing them to a nonlinear activation function. That's because PyTorch's commonly used loss functions combine the softmax (or sigmoid for binary classification) operation with the negative log-likelihood loss in a single class. The reason for this is numerical efficiency and stability. So, if we want to compute class-membership probabilities for our predictions, we have to call the softmax function explicitly:

This prints:

The values can now be interpreted as class-membership probabilities that sum up to 1. The values are roughly equal for this random input, which is expected for a randomly initialized model without training.

In the following two sections, we will learn how to set up an efficient data loader and train the model.

\section*{A. 6 Setting up efficient data loaders}

In the previous section, we defined a custom neural network model. Before we can train this model, we have to briefly talk about creating efficient data loaders in PyTorch, which we will iterate over when training the model. The overall idea behind data loading in PyTorch is illustrated in figure A. 10 .

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-328.jpg?height=535&width=1237&top_left_y=501&top_left_x=315)

Figure A. 10 PyTorch implements a Dataset and a DataLoader class. The Dataset class is used to instantiate objects that define how each data record is loaded. The DataLoader handles how the data is shuffled and assembled into batches.

Following the illustration in figure A.10, in this section, we will implement a custom Dataset class that we will use to create a training and a test dataset that we'll then use to create the data loaders.

Let's start by creating a simple toy dataset of five training examples with two features each. Accompanying the training examples, we also create a tensor containing the corresponding class labels: three examples below to class 0 , and two examples belong to class 1. In addition, we also make a test set consisting of two entries. The code to create this dataset is shown below.

Listing A. 5 Creating a small toy dataset

CLASS LABEL NUMBERING PyTorch requires that class labels start with label 0 , and the largest class label value should not exceed the number of output nodes minus 1 (since Python index counting starts at 0 . So, if we have class labels \(0,1,2,3\), and 4 , the neural network output layer should consist of 5 nodes.

Next, we create a custom dataset class, ToyDataset, by subclassing from PyTorch's Dataset parent class, as shown below.

Listing A. 6 Defining a custom Dataset class

from torch.utils.data import Dataset

\#A Instructions for retrieving exactly one data record and the corresponding label \#B Instructions for returning the total length of the dataset

This custom ToyDataset class's purpose is to use it to instantiate a PyTorch DataLoader. But before we get to this step, let's briefly go over the general structure of the ToyDataset code.

In PyTorch, the three main components of a custom Dataset class are the __init__ constructor, the __getitem__ method, and the __len__ method, as shown in code listing A. 6 above.

In the __init__ method, we set up attributes that we can access later in the _getitem__ and __len__ methods. This could be file paths, file objects, database connectors, and so on. Since we created a tensor dataset that sits in memory, we are simply assigning x and y to these attributes, which are placeholders for our tensor objects.

In the \(\qquad\) method, we define instructions for returning exactly one item from the dataset via an index. This means the features and the class label corresponding to a single training example or test instance. (The data loader will provide this index, which we will cover shortly.)

Finally, the __len__ method constrains instructions for retrieving the length of the dataset. Here, we use the .shape attribute of a tensor to return the number of rows in the feature array. In the case of the training dataset, we have five rows, which we can doublecheck as follows:

The result is:

5

Now that we defined a PyTorch Dataset class we can use for our toy dataset, we can use PyTorch's DataLoader class to sample from it, as shown in the code listing below:

\section*{Listing A. 7 Instantiating data loaders}

from torch.utils.data import DataLoader

\#A The ToyDataset instance created earlier serves as input to the data loader.

\#B Whether or not to shuffle the data

\#C The number of background processes

\#D It is not necessary to shuffle a test dataset

After instantiating the training data loader, we can iterate over it as shown below. (The iteration over the test_loader works similarly but is omitted for brevity.)

The result is:

As we can see based on the output above, the train_loader iterates over the training dataset visiting each training example exactly once. This is known as a training epoch. Since we seeded the random number generator using torch.manual_seed (123) above, you should get the exact same shuffling order of training examples as shown above. However if you iterate over the dataset a second time, you will see that the shuffling order will change. This is desired to prevent deep neural networks getting caught in repetitive update cycles during training.

Note that we specified a batch size of 2 above, but the 3rd batch only contains a single example. That's because we have five training examples, which is not evenly divisible by 2 . In practice, having a substantially smaller batch as the last batch in a training epoch can disturb the convergence during training. To prevent this, it's recommended to set drop last=True, which will drop the last batch in each epoch, as shown below:

\section*{Listing A. 8 A training loader that drops the last batch}

Now, iterating over the training loader, we can see that the last batch is omitted:

The result is:

Lastly, let's discuss the setting num_workers=0 in the DataLoader. This parameter in PyTorch's DataLoader function is crucial for parallelizing data loading and preprocessing. When num_workers is set to 0 , the data loading will be done in the main process and not in separate worker processes. This might seem unproblematic, but it can lead to significant slowdowns during model training when we train larger networks on a GPU. This is because instead of focusing solely on the processing of the deep learning model, the CPU must also take time to load and preprocess the data. As a result, the GPU can sit idle while waiting for the CPU to finish these tasks. In contrast, when num_workers is set to a number greater than zero, multiple worker processes are launched to load data in parallel, freeing the main process to focus on training your model and better utilizing your system's resources, which is illustrated in figure A. 11

Data loading without multiple workers

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-333.jpg?height=524&width=592&top_left_y=787&top_left_x=293)

Data loading with multiple workers

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-333.jpg?height=554&width=641&top_left_y=751&top_left_x=922)

Figure A. 11 Loading data without multiple workers (setting num_workers=0) will create a data loading bottleneck where the model sits idle until the next batch is loaded as illustrated in the left subpanel. If multiple workers are enabled, the data loader can already queue up the next batch in the background as shown in the right subpanel.

However, if we are working with very small datasets, setting num_workers to 1 or larger may not be necessary since the total training time takes only fractions of a second anyway. On the contrary, if you are working with tiny datasets or interactive environments such as Jupyter notebooks, increasing num_workers may not provide any noticeable speedup. They might, in fact, lead to some issues. One potential issue is the overhead of spinning up multiple worker processes, which could take longer than the actual data loading when your dataset is small.

Furthermore, for Jupyter notebooks, setting num_workers to greater than 0 can sometimes lead to issues related to the sharing of resources between different processes, resulting in errors or notebook crashes. Therefore, it's essential to understand the trade-off and make a calculated decision on setting the num_workers parameter. When used correctly, it can be a beneficial tool but should be adapted to your specific dataset size and computational environment for optimal results.

In my experience, setting num_workers=4 usually leads to optimal performance on many real-world datasets, but optimal settings depend on your hardware and the code used for loading a training example defined in the Dataset class.

\section*{A. 7 A typical training loop}

So far, we've discussed all the requirements for training neural networks: PyTorch's tensor library, autograd, the Module API, and efficient data loaders. Let's now combine all these things and train a neural network on the toy dataset from the previous section. The training code is shown in code listing A. 9 below.

Listing A. 9 Neural network training in PyTorch

import torch.nn.functional as F

\#A The dataset from the previous section has 2 features and 2 classes \#B We led the optimizer needs to know which parameters to optimize \#C Set the gradients from the previous round to zero to prevent unintended gradient accumulation \#D Compute the gradients of the loss with respect to the model parameters \#E The optimizer users the gradients to update the model parameters

Running the code in listing A. 9 above yields the following outputs:

As we can see, the loss reaches zero after 3 epochs, a sign that the model converged on the training set. However, before we evaluate the model's predictions, let's go over some of the details of the preceding code listing.

First, note that we initialized a model with two inputs and two outputs. That's because the toy dataset from the previous section has two input features and two class labels to predict. We used a stochastic gradient descent (SGD) optimizer with a learning rate (lr) of 0.5 . The learning rate is a hyperparameter, meaning it's a tunable setting that we have to experiment with based on observing the loss. Ideally, we want to choose a learning rate such that the loss converges after a certain number of epochs -- the number of epochs is another hyperparameter to choose.

\section*{EXERCISE A. 3}

How many parameters does the neural network introduced at the beginning of this section have?

In practice, we often use a third dataset, a so-called validation dataset, to find the optimal hyperparameter settings. A validation dataset is similar to a test set. However, while we only want to use a test set precisely once to avoid biasing the evaluation, we usually use the validation set multiple times to tweak the model settings.

We also introduced new settings called model.train() and model.eval(). As these names imply, these settings are used to put the model into a training and an evaluation mode. This is necessary for components that behave differently during training and inference, such as dropout or batch normalization layers. Since we don't have dropout or other components in our NeuralNetwork class that are affected by these settings, using model.train() and model.eval() is redundant in our code above. However, it's best practice to include them anyway to avoid unexpected behaviors when we change the model architecture or reuse the code to train a different model.

As discussed earlier, we pass the logits directly into the cross_entropy loss function, which will apply the softmax function internally for efficiency and numerical stability reasons. Then, calling loss.backward() will calculate the gradients in the computation graph that PyTorch constructed in the background. The optimizer.step() method will use the gradients to update the model parameters to minimize the loss. In the case of the SGD optimizer, this means multiplying the gradients with the learning rate and adding the scaled negative gradient to the parameters.

PREVENTING UNDESIRED GRADIENT ACCUMULATION It is important to include an optimizer.zero_grad() call in each update round to reset the gradients to zero. Otherwise, the gradients will accumulate, which may be undesired.

After we trained the model, we can use it to make predictions, as shown below:

The results are as follows:

\[
\begin{aligned}
& \text { tensor([[ 2.8569, -4.1618], } \\
& \text { [ 2.5382, -3.7548], } \\
& \text { [ 2.0944, -3.1820], } \\
& {[-1.4814,1.4816]} \\
& [-1.7176, \quad 1.7342]])
\end{aligned}
\]

To obtain the class membership probabilities, we can then use PyTorch's softmax function, as follows:

This outputs:

\begin{tabular}{|c|c|c|}
\hline tensor([ [ & 0.9991 & \(0.0009]\) \\
\hline [ & 0.9982 & \(0.0018]\) \\
\hline [ & 0.9949 , & \(0.0051]\) \\
\hline [ & 0.0491 , & 0.9509], \\
\hline [ & 0.0307 & 0.9693]]) \\
\hline
\end{tabular}

Let's consider the first row in the code output above. Here, the first value (column) means that the training example has a \(99.91 \%\) probability of belonging to class 0 and a \(0.09 \%\) probability of belonging to class 1. (The set_printoptions call is used here to make the outputs more legible.)

We can convert these values into class labels predictions using PyTorch's argmax function, which returns the index position of the highest value in each row if we set dim=1 (setting dim=0 would return the highest value in each column, instead):

This prints:

Note that it is unnecessary to compute softmax probabilities to obtain the class labels. We could also apply the argmax function to the logits (outputs) directly:

The output is:

Above, we computed the predicted labels for the training dataset. Since the training dataset is relatively small, we could compare it to the true training labels by eye and see that the model is \(100 \%\) correct. We can double-check this using the \(==\) comparison operator:

predictions == y_train

The results are:

Using torch.sum, we can count the number of correct prediction as follows:

torch.sum(predictions == y_train)

The output is:

5

Since the dataset consists of 5 training examples, we have 5 out of 5 predictions that are correct, which equals \(5 / 5 \times 100 \%=100 \%\) prediction accuracy.

However, to generalize the computation of the prediction accuracy, let's implement a compute_accuracy function as shown in the following code listing.

Listing A. 10 A function to compute the prediction accuracy

\#A This returns a tensor of True/False values depending on whether the labels match \#B The sum operations counts the number of True values \#C This is the fraction of correct prediction, a value between 0 and 1. And .item() returns the value of the tensor as a Python float.

Note that the following code listing iterates over a data loader to compute the number and fraction of the correct predictions. This is because when we work with large datasets, we typically can only call the model on a small part of the dataset due to memory limitations. The compute_accuracy function above is a general method that scales to datasets of arbitrary size since, in each iteration, the dataset chunk that the model receives is the same size as the batch size seen during training.

Notice that the internals of the compute_accuracy function are similar to what we used before when we converted the logits to the class labels.

We can then apply the function to the training as follows:

print(compute_accuracy(model, train_loader))

The results is:

\section*{1.0}

Similarly, we can apply the function to the test set as follows:

>>> print(compute_accuracy(model, test_loader))

This prints:

\section*{1.0}

In this section, we learned how we can train a neural network using PyTorch. Next, let's see how we can save and restore models after training.

\section*{A. 8 Saving and loading models}

In the previous section, we successfully trained a model. Let's now see how we can save a trained model to reuse it later.

Here's the recommended way how we can save and load models in PyTorch:

torch.save(model.state_dict(), "model.pth")

The model's state_dict is a Python dictionary object that maps each layer in the model to its trainable parameters (weights and biases). Note that "model.pth" is an arbitrary filename for the model file saved to disk. We can give it any name and file ending we like; however, . pth and .pt are the most common conventions.

Once we saved the model, we can restore it from disk as follows:
model = NeuralNetwork(2, 2)

model.load_state_dict(torch.load("model.pth"))

The torch.load("model.pth") function reads the file "model.pth" and reconstructs the Python dictionary object containing the model's parameters while model.load_state_dict() applies these parameters to the model, effectively restoring its learned state from when we saved it.

Note that the line model \(=\) NeuralNetwork \((2,2)\) above is not strictly necessary if you execute this code in the same session where you saved a model. However, I included it here to illustrate that we need an instance of the model in memory to apply the saved parameters. Here, the NeuralNetwork \((2,2)\) architecture needs to match the original saved model exactly.

Now, we are well equipped to use PyTorch to implement large language models in the upcoming chapters. However, before we jump to the next chapter, the last section will show you how to train PyTorch models faster using one or more GPUs (if available).

\section*{A. 9 Optimizing training performance with GPUs}

In this last section of this chapter, we will see how we can utilize GPUs, which will accelerate deep neural network training compared to regular CPUs. First, we will introduce the main concepts behind GPU computing in PyTorch. Then, we will train a model on a single GPU. Finally, we'll then look at distributed training using multiple GPUs.

\section*{A.9.1 PyTorch computations on GPU devices}

As you will see, modifying the training loop from section 2.7 to optionally run on a GPU is relatively simple and only requires changing three lines of code.

Before we make the modifications, it's crucial to understand the main concept behind GPU computations within PyTorch. First, we need to introduce the notion of devices. In PyTorch, a device is where computations occur, and data resides. The CPU and the GPU are examples of devices. A PyTorch tensor resides in a device, and its operations are executed on the same device.

Let's see how this works in action. Assuming that you installed a GPU-compatible version of PyTorch as explained in section 2.1.3, Installing PyTorch, we can double-check that our runtime indeed supports GPU computing via the following code:

print(torch.cuda.is_available())

The result is:

True

Now, suppose we have two tensors that we can add as follows -- this computation will be carried out on the CPU by default:

This outputs:

We can now use the . to () method[1] to transfer these tensors onto a GPU and perform the addition there:

The output is as follows:

Notice that the resulting tensor now includes the device information, device='cuda:0', which means that the tensors reside on the first GPU. If your machine hosts multiple GPUs, you have the option to specify which GPU you'd like to transfer the tensors to. You can do this by indicating the device ID in the transfer command. For instance, you can use .to("cuda:0"), .to("cuda:1"), and so on.

However, it is important to note that all tensors must be on the same device. Otherwise, the computation will fail, as shown below, where one tensor resides on the CPU and the other on the GPU:

This results in the following:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

In this section, we learned that GPU computations on PyTorch are relatively straightforward. All we have to do is transfer the tensors onto the same GPU device, and PyTorch will handle the rest. Equipped with this information, we can now train the neural network from the previous section on a GPU.

\section*{A.9.2 Single-GPU training}

Now that we are familiar with transferring tensors to the GPU, we can modify the training loop from section 2.7, A typical training loop, to run on a GPU. This requires only changing three lines of code, as shown in code listing A. 11 below.

\section*{Listing A.11 A training loop on a GPU}

print(f"Epoch: \{epoch+1:03d\}/\{num_epochs:03d\}"

f" | Batch \{batch_idx:03d\}/\{len(train_loader):03d\}"

f" | Train/Val Loss: \{loss:.2f\}")

model.eval()

\# Optional model evaluation

\#A Define a device variable that defaults to a GPU.

\#B Transfer the model onto the GPU.

\#C Transfer the data onto the GPU.

Running the above code will output the following, similar to the results obtained on the CPU previously in section 2.7:

We can also use .to("cuda") instead of device = torch.device("cuda"). As we saw in section 2.9.1, transferring a tensor to "cuda" instead of torch.device("cuda") works as well and is shorter. We can also modify the statement to the following, which will make the same code executable on a CPU if a GPU is not available, which is usually considered best practice when sharing PyTorch code:

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

In the case of the modified training loop above, we probably won't see a speed-up because of the memory transfer cost from CPU to GPU. However, we can expect a significant speedup when training deep neural networks, especially large language models.

As we saw in this section, training a model on a single GPU in PyTorch is relatively easy. Next, let's introduce another concept: training models on multiple GPUs.

\section*{PYTORCH ON MACOS}

On an Apple Mac with an Apple Silicon chip (like the M1, M2, M3, or newer models) instead of a computer with an Nvidia GPU, you can change

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
to

device = torch.device("mps" if torch.backends.mps.is_available() else "cpu")

to take advantage of this chip.

\section*{EXERCISE A. 4}

Compare the runtime of matrix multiplication on a CPU to a GPU. At what matrix size do you begin to see the matrix multiplication on the GPU being faster than on the CPU? Hint: I recommend using the \%timeit command in Jupyter to compare the runtime. For example, given matrices a and b, run the command \%timeit a @ b in a new notebook cell.

\section*{A.9.3 Training with multiple GPUs}

In this section, we will briefly go over the concept of distributed training. Distributed training is the concept of dividing the model training across multiple GPUs and machines.

Why do we need this? Even when it is possible to train a model on a single GPU or machine, the process could be exceedingly time-consuming. The training time can be significantly reduced by distributing the training process across multiple machines, each with potentially multiple GPUs. This is particularly crucial in the experimental stages of model development, where numerous training iterations might be necessary to finetune the model parameters and architecture.

MULTI-GPU COMPUTING IS OPTIONAL For this book, it is not required to have access to or use multiple-GPU. This section is included for those who are interested in how multi-GPU computing works in PyTorch.

In this section, we will look at the most basic case of distributed training: PyTorch's DistributedDataParallel (DDP) strategy. DDP enables parallelism by splitting the input data across the available devices and processing these data subsets simultaneously.

How does this work? PyTorch launches a separate process on each GPU, and each process receives and keeps a copy of the model -- these copies will be synchronized during training. To illustrate this, suppose we have two GPUs that we want to use to train a neural network, as shown in figure A.12.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-346.jpg?height=558&width=1230&top_left_y=263&top_left_x=302)

Figure A.12 The model and data transfer in DDP involves two key steps. First, we create a copy of the model on each of the GPUs. Then we divide the input data into unique minibatches that we pass on to each model copy.

Each of the two GPUs will receive a copy of the model. Then, in every training iteration, each model will receive a minibatch (or just batch) from the data loader. We can use a DistributedSampler to ensure that each GPU will receive a different, non-overlapping batch when using DDP.

Since each model copy will see a different sample of the training data, the model copies will return different logits as outputs and compute different gradients during the backward pass. These gradients are then averaged and synchronized during training to update the models. This way, we ensure that the models don't diverge, as illustrated in figure A.13.
![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-346.jpg?height=422&width=1230&top_left_y=1397&top_left_x=332)

Figure A.13 The forward and backward pass in DDP are executed independently on each GPU with its corresponding data subset. Once the forward and backward passes are completed, gradients from each model replica (on each GPU) are synchronized across all GPUs. This ensures that every model replica has the same updated weights.

The benefit of using DDP is the enhanced speed it offers for processing the dataset compared to a single GPU. Barring a minor communication overhead between devices that comes with DDP use, it can theoretically process a training epoch in half the time with two GPUs compared to just one. The time efficiency scales up with the number of GPUs, allowing us to process an epoch eight times faster if we have eight GPUs, and so on.

\begin{abstract}
MULT-GPU COMPUTING IN INTERACTIVE ENVIRONMENTS DDP does not function properly within interactive Python environments like Jupyter notebooks, which don't handle multiprocessing in the same way a standalone Python script does. Therefore, the following code should be executed as a script, not within a notebook interface like Jupyter. This is because DDP needs to spawn multiple processes, and each process should have its own Python interpreter instance.
\end{abstract}

Let's now see how this works in practice. For brevity, we will only focus on the core parts of the previous code that need to be adjusted for DDP training. However, for readers who want to run the code on their own multi-GPU machine or a cloud instance of their choice, it is recommended to use the standalone script provided in this book's GitHub repository at https://github.com/rasbt/LLMs-from-scratch.

First, we will import a few additional submodules, classes, and functions for distributed training PyTorch as shown in code listing A. 13 below.

\title{
Listing A. 12 PyTorch utilities for distributed training
}

import torch.multiprocessing as mp

from torch.utils.data.distributed import DistributedSampler

from torch.nn.parallel import DistributedDataParallel as DDP

from torch.distributed import init_process_group, destroy_process_group

Before we dive deeper into the changes to make the training compatible with DDP, let's briefly go over the rationale and usage for these newly imported utilities that we need alongside the DistributedDataParallel class.

PyTorch's multiprocessing submodule contains functions such as multiprocessing.spawn, which we will use to spawn multiple processes and apply a function to multiple inputs in parallel. We will use it to spawn one training process per GPU.

If we spawn multiple processes for training, we will need a way to divide the dataset among these different processes. For this, we will use the DistributedSampler.

The init_process_group and destroy_process_group are used to initialize and quit the distributed training mods. The init_process_group function should be called at the beginning of the training script to initialize a process group for each process in the distributed setup, and destroy_process_group should be called at the end of the training script to destroy a given process group and release its resources.

The following code in listing A. 13 below illustrates how these new components are used to implement DDP training for the NeuralNetwork model we implemented earlier.

Listing A.13 Model training with DistributedDataParallel strategy

\#A Address of the main node

\#B Any free port on the machine

\#C nccl stands for NVIDIA Collective Communication Library.

\#D rank refers to the index of the GPU we want to use.

\#E world_size is the number of GPUs to use.

\#F Sets the current GPU device on which tensors will be allocated and operations will be performed.

\#G DistibutedSampler takes care of the shuffling now

\#H Enables faster memory transfer when training on GPU

\#I Splits the dataset into distinct, non-overlapping subsets for each process (GPU)

\#J The main function running the model training

\#K rank is the GPU ID

\#L Clean up resource allocation

\#M Launch the main function using multiple processes, where nprocs=world_size means one process per GPU.

Before we run the code from listing A.13, here is a summary of how it works, in addition to the annotations above. We have a __name__ == "__main__" clause at the bottom containing code that is executed when we run the code as a Python script instead of importing it as a module. This code first prints the number of available GPUs using torch.cuda.device_count (), sets a random seed for reproducibility and then spawns new processes using PyTorch's multiprocesses.spawn function. Here, the spawn function launches one process per GPU setting nproces=world_size, where the world size is the number of available GPUs. This spawn function launches the code in the main function we define in the same script with some additional arguments provided via args. Note that the main function has a rank argument that we don't include in the mp.spawn() call. That's because the rank, which refers to the process ID we use as the GPU ID, is already passed automatically.

The main function sets up the distributed environment via ddp_setup -- another function we defined, loads the training and test sets, sets up the model, and carries out the training. Compared to the single-GPU training in section 2.12, we now transfer the model and data to the target device via .to (rank), which we use to refer to the GPU device ID. Also, we wrap the model via DDP, which enables the synchronization of the gradients between the different GPUs during training. After the training finishes and we evaluate the models, we use destroy process group() to cleanly exit the distributed training and free up the allocated resources.

Earlier, we mentioned that each GPU will receive a different subsample of the training data. To ensure this, we set sampler=DistributedSampler(train_ds) in the training loader.

The last function to discuss is ddp_setup. It sets the main node's address and port to allow for communication between the different processes, initializes the process group with the NCCL backend (designed for GPU-to-GPU communication), and sets the rank (process identifier) and world size (total number of processes). Finally, it specifies the GPU device corresponding to the current model training process rank.

\section*{SELECTING AVAILABLE GPUS ON A MULTI-GPU MACHINE}

If you wish to restrict the number of GPUs used for training on a multi-GPU machine, the simplest way is to use the CUDA VISIBLE DEVICES environment variable. To illustrate this, suppose your machine has multiple GPUs, and you only want to use one GPU, for example, the GPU with index 0 . Instead of python some_script.py, you can run the code from the terminal as follows:

CUDA_VISIBLE_DEVICES=0 python some_script.py

Or, if your machine has four GPUs and you only want to use the first and third GPU, you can use

CUDA_VISIBLE_DEVICES=0,2 python some_script.py

Setting CUDA_VISIBLE_DEVICES in this way is a simple and effective way to manage GPU allocation without modifying your PyTorch scripts.

Let's now run this code and see how it works in practice by launching the code as a script from the terminal:

python ch02-DDP-script.py

Note that it should work on both single- and multi-GPU machines. If we run this code on a single GPU, we should see the following output:

PyTorch version: 2.0.1+cu117

CUDA available: True

Number of GPUs available: 1

[GPU0] Epoch: 001/003 | Batchsize 002 | Train/Val Loss: 0.62

[GPU0] Epoch: 001/003 | Batchsize 002 | Train/Val Loss: 0.32

[GPU0] Epoch: 002/003 | Batchsize 002 | Train/Val Loss: 0.11

[GPU0] Epoch: 002/003 | Batchsize 002 | Train/Val Loss: 0.07

[GPU0] Epoch: 003/003 | Batchsize 002 | Train/Val Loss: 0.02

[GPU0] Epoch: 003/003 | Batchsize 002 | Train/Val Loss: 0.03

[GPU0] Training accuracy 1.0

[GPU0] Test accuracy 1.0

The code output looks similar to the one in section 2.9.2, which is a good sanity check.

Now, if we run the same command and code on a machine with two GPUs, we should see the following:

As expected, we can see that some batches are processed on the first GPU (GPUo) and others on the second (GPU1). However, we see duplicated output lines when printing the training and test accuracies. This is because each process (in other words, each GPU) prints the test accuracy independently. Since DDP replicates the model onto each GPU and each process runs independently, if you have a print statement inside your testing loop, each process will execute it, leading to repeated output lines.

If this bothers you, you can fix this using the rank of each process to control your print statements.

if rank \(==0\) : \# only print in the first process

print("Test accuracy: ", accuracy)

This is, in a nutshell, how distributed training via DDP works. If you are interested in additional details, I recommend checking the official API documentation at https://pytorch. org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html.

\section*{ALTERNATIVE PYTORCH APIS FOR MULTI-GPU TRAINING}

If you prefer more straightforward ways to use multiple GPUs in PyTorch, you can also consider add-on APIs like the open-source Fabric library, which I've written about in Accelerating PyTorch Model Training: Using Mixed-Precision and Fully Sharded Data Parallelism https://magazine.sebastianraschka.com/p/accelerating-pytorch-modeltraining.

\section*{A. 10 Summary}
- PyTorch is an open-source library that consists of three core components: a tensor library, automatic differentiation functions, and deep learning utilities.
- PyTorch's tensor library is similar to array libraries like NumPy
- In the context of PyTorch, tensors are array-like data structures to represent scalars, vectors, matrices, and higher-dimensional arrays.
- \(\quad\) PyTorch tensors can be executed on the CPU, but one major advantage of PyTorch's tensor format is its GPU support to accelerate computations.
- The automatic differentiation (autograd) capabilities in PyTorch allow us to conveniently train neural networks using backpropagation without manually deriving gradients.
- \(\quad\) The deep learning utilities in PyTorch provide building blocks for creating custom deep neural networks.
- \(\quad\) PyTorch includes Dataset and DataLoader classes to set up efficient data loading pipelines.
- It's easiest to train models on a CPU or single GPU.
- Using DistributedDataParallel is the simplest way in PyTorch to accelerate the training if multiple GPUs are available.

\section*{A. 11 Further reading}

While this chapter should be sufficient to get you up to speed, in addition, if you are looking for more comprehensive introductions to deep learning, I recommend the following books:
- Machine Learning with PyTorch and Scikit-Learn (2022) by Sebastian Raschka, Hayden Liu, and Vahid Mirjalili. ISBN 978-1801819312
- Deep Learning with PyTorch (2021) by Eli Stevens, Luca Antiga, and Thomas Viehmann. ISBN 978-1617295263

For a more thorough introduction to the concepts of tensors, readers can find a 15 min video tutorial that I recorded:
- Lecture 4.1: Tensors in Deep Learning, https://www.youtube.com/watch? \(\underline{v=J X f D I g r f O B Y}\)

If you want to learn more about model evaluation in machine learning, I recommend my article:
- Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning (2018) by Sebastian Raschka, https://arxiv.org/abs/1811.12808

For readers who are interested in a refresher or gentle introduction to calculus, I've written a chapter on calculus that is freely available on my website:
- Introduction to Calculus by Sebastian Raschka, https://sebastianraschka. com/pdf/supplementary/calculus.pdf

Why does PyTorch not call optimizer.zero_grad() automatically for us in the background? In some instances, it may be desirable to accumulate the gradients, and PyTorch will leave this as an option for us. If you want to learn more about gradient accumulation, please see the following article:
- \(\quad\) Finetuning Large Language Models On A Single GPU Using Gradient Accumulation by Sebastian Raschka, https://sebastianraschka.com/blog/ 2023/llm-grad-accumulation.html

This chapter covered DDP, which is a popular approach for training deep learning models across multiple GPUs. For more advanced use cases where a single model doesn't fit onto the GPU, you may also consider PyTorch's Fully Sharded Data Parallel (FSDP) method, which performs distributed data parallelism and distributes large layers across different GPUs. For more information, see this overview with further links to the API documentation:
- Introducing PyTorch Fully Sharded Data Parallel (FSDP) API, https://pytorch.org/blog/introducing-pytorch-fully-sharded-data-parallelapi/

\section*{A. 12 Exercise answers}

\section*{EXERCISE A.3:}

The network has 2 inputs and 2 outputs. In addition, there are 2 hidden layers with 30 and 20 nodes, respectively. Programmatically, we can calculate the number of parameters as follows:
model = NeuralNetwork(2, 2)

num_params = sum(p.numel() for p in model.parameters() if p.requires_grad)

print("Total number of trainable model parameters:", num_params)

This returns:

752

We can also calculate this manually as follows:
- first hidden layer: 2 inputs times 30 hidden units plus 30 bias units.
- second hidden layer: 30 incoming units times 20 nodes plus 20 bias units.
- output layer: 20 incoming nodes times 2 output nodes plus 2 bias units.

Then, adding all the parameters in each layer results in \(2 \times 30+30+30 \times 20+20+20 \times 2+2\) \(=752\).

\section*{EXERCISE A.4:}

The exact runtime results will be specific to the hardware used for this experiment. In my experiments, I observed significant speed-ups even for small matrix multiplications as the following one when using a Google Colab instance connected to a V100 GPU:

\(a=\) torch. \(\operatorname{rand}(100,200)\)

\(b=\) torch.rand(200, 300)

\%timeit a@b

On the CPU this resulted in:

\(63.8 \mu \mathrm{s} \pm 8.7 \mu \mathrm{s}\) per loop

When executed on a GPU:

a, b = a.to("cuda"), b.to("cuda")

\%timeit a @ b

The result was:

\(13.8 \mu \mathrm{s} \pm 425 \mathrm{~ns}\) per loop

In this case, on a V 100 , the computation was approximately four times faster.

[1] This is the same .to() method we previously used to change a tensor's datatype in section 2.2.2, Tensor data types.

\section*{Appendix B. References and Further Reading}

\section*{Chapter 1}

Custom-built LLMs are able to outperform general-purpose LLMs as a team at Bloomberg showed via a version of GPT pretrained on finance data from scratch. The custom LLM outperformed ChatGPT on financial tasks while maintaining good performance on general LLM benchmarks:
- BloombergGPT: A Large Language Model for Finance (2023) by Wu et al., https://arxiv.org/abs/2303.17564

Existing LLMs can be adapted and finetuned to outperform general LLMs as well, which teams from Google Research and Google DeepMind showed in a medical context:
- Towards Expert-Level Medical Question Answering with Large Language Models (2023) by Singhal et al., https://arxiv.org/abs/2305.09617

The paper that proposed the original transformer architecture:
- Attention Is All You Need (2017) by Vaswani et al., https://arxiv. org/abs/1706.03762

The original encoder-style transformer, called BERT:
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018) by Devlin et al., https://arxiv.org/abs/1810.04805

The paper describing the decoder-style GPT-3 model, which inspired modern LLMs and will be used as a template for implementing an LLM from scratch in this book:
- Language Models are Few-Shot Learners (2020) by Brown et al., https://arxiv.org/abs/2005.14165

The original vision transformer for classifying images, which illustrates that transformer architectures are not only restricted to text inputs:
- An Image is Worth \(16 \times 16\) Words: Transformers for Image Recognition at Scale (2020) by Dosovitskiy et al., https://arxiv.org/abs/2010.11929

Two experimental (but less popular) LLM architectures that serve as examples that not all LLMs need to be based on the transformer architecture:
- RWKV: Reinventing RNNs for the Transformer Era (2023) by Peng et al., https://arxiv.org/abs/2305.13048
- Hyena Hierarchy: Towards Larger Convolutional Language Models (2023) by Poli et al., https://arxiv.org/abs/2302.10866 Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) by Gu and Dao, https://arxiv.org/abs/2312.00752

Meta AI's model is a popular implementation of a GPT-like model that is openly available in contrast to GPT-3 and ChatGPT:
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023) by Touvron et al., https://arxiv.org/abs/2307.092881

For readers interested in additional details about the dataset references in section 1.5, this paper describes the publicly available The Pile dataset curated by Eleuther AI:
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling (2020) by Gao et al., https://arxiv.org/abs/2101.00027.

The following paper provides the reference for InstructGPT for finetuning GPT-3, which was mentioned in section 1.6 and will be discussed in more detail in chapter 7 :
- Training Language Models to Follow Instructions with Human Feedback (2022) by Ouyang et al., https://arxiv.org/abs/2203.02155

\section*{Chapter 2}
- Machine Learning Q and AI (2023) by Sebastian Raschka, https://leanpub.com/machine-learning-q-and-ai
- The following paper provides more in-depth discussions of how how byte pair encoding is used as a tokenization method:

©© Manning Publications Co. To comment go to liveBook

Licensed to lijin lijin <jimlee13894896995@gmail.com>
- Neural Machine Translation of Rare Words with Subword Units (2015) by Sennrich at al., https://arxiv.org/abs/1508.07909

The code for the byte pair encoding tokenizer used to train GPT-2 was open-sourced by OpenAI:
- https://github.com/openai/g.pt-2/blob/master/src/encoder.py

OpenAI provides an interactive web UI to illustrate how the byte pair tokenizer in GPT models works:
- https://platform.openai.com/tokenizer

For readers interested in coding and training a BPE tokenizer from the ground up, Andrej Karpathy's GitHub repository min.bpe offers a minimal and readable implementation:
- A minimal implementation of a BPE tokenizer, https://github. com/karpathy/minbpe

Readers who are interested in studying alternative tokenization schemes that are used by some other popular LLMs can find more information in the SentencePiece and WordPiece papers:
- SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing (2018) by Kudo and

Richardson, https://aclantholog.y.org/D18-2012/
- Fast WordPiece Tokenization (2020) by Song et al., https://arxiv. org/abs/2012.15524

\section*{Chapter 3}

Readers interested in learning more about Bahdanau attention for RNN and language translation can find detailed insights in the following paper:
- Neural Machine Translation by Jointly Learning to Align and Translate (2014) by Bahdanau, Cho, and Bengio, https://arxiv.org/abs/1409.0473

The concept of self-attention as scaled dot-product attention was introduced in the original transformer paper:
- Attention Is All You Need (2017) by Vaswani et al., https://arxiv. org/abs/1706.03762

FlashAttention is a highly efficient implementation of self-attention mechanism, which accelerates the computation process by optimizing memory access patterns. FlashAttention is mathematically the same as the standard self-attention mechanism but optimizes the computational process for efficiency:
- FlashAttention: Fast and Memory-Efficient Exact Attention with IOAwareness (2022) by Dao et al., https://arxiv.org/abs/2205.14135
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (2023) by Dao, https://arxiv.org/abs/2307.08691

PyTorch implements a function for self-attention and causal attention that supports FlashAttention for efficiency. This function is beta and subject to change:
- scaled_dot_product_attention documentation: https://pytorch. org/docs/stable/generated/torch.nn.functional.scaled dot product_attention.html

PyTorch also implements an efficient MultiHeadAttention class based on the scaled_dot_product function:
- MultiHeadAttention documentation: https://pytorch.org/docs/ stable/generated/torch.nn.MultiheadAttention.html

Dropout is a regularization technique used in neural networks to prevent overfitting by randomly dropping units (along with their connections) from the neural network during training:
- Dropout: A Simple Way to Prevent Neural Networks from Overfitting (2014) by Srivastava et al., https://jmlr.orgLpapers/v15/srivastava14a. \(\underline{h t m l}\)

While using the multi-head attention based on scaled-dot product attention remains the most common variant of self-attention in practice, authors found that it's possible to also achieve good performance without the value weight matrix and projection layer:
- Simplifying Transformer Blocks (2023) by He and Hofmann, https://arxiv. org/abs/2311.01906

\section*{Chapter 4}

The layer normalization paper, titled "Layer Normalization," introduces a technique that stabilizes the hidden state dynamics neural networks by normalizing the summed inputs to the neurons within a hidden layer, significantly reducing training time compared to previously published methods:
- Layer Normalization (2016) by Ba, Kiros, and Hinton, https://arxiv. org/abs/1607.06450

Post-LayerNorm, used in the original Transformer model, applies layer normalization after the self-attention and feed forward networks. In contrast, Pre-LayerNorm, as adopted in models like GPT-2 and newer LLMs, applies layer normalization before these components, which can lead to more stable training dynamics and has been shown to improve performance in some cases, as discussed in the following papers:
- On Layer Normalization in the Transformer Architecture (2020) by Xiong et al., https://arxiv.org/abs/2002.04745
- ResiDual: Transformer with Dual Residual Connections (2023) by Tie et al., https://arxiv.org/abs/2304.14802

A popular variant of LayerNorm used in modern LLMs is RMSNorm due to its improved computing efficiency. This variant simplifies the normalization process by normalizing the inputs using only the root mean square of the inputs, without subtracting the mean before squaring. This means it does not center the data before computing the scale. RMSNorm is described in more detail in the following paper:
- Root Mean Square Layer Normalization (2019) by Zhang and Sennrich, https://arxiv.org/abs/1910.07467

The GELU (Gaussian Error Linear Unit) activation function combines the properties of both the classic ReLU activation function and the normal distribution's cumulative distribution function to model layer outputs, allowing for stochastic regularization and non-linearities in deep learning models, as introduced in the following paper:
- Gaussian Error Linear Units (GELUs) (2016) by Hendricks and Gimpel, https://arxiv.org/abs/1606.08415

The GPT-2 paper introduced a series of transformer-based LLMs with varying sizes-124M, \(355 \mathrm{M}, 774 \mathrm{M}\), and 1.5B parameters:
- Language Models are Unsupervised Multitask Learners (2019) by Radford et al., https://d4mucfpksywv.cloudfront.net/better-languagemodels/language models are unsupervised multitask learners.pdf

OpenAI's GPT-3 uses fundamentally the same architecture as GPT-2, except that the largest version (175 billion) is \(100 x\) larger than the largest GPT-2 model and has been trained on much more data. Interested readers can refer to the official GPT-3 paper by OpenAI and the technical overview by Lambda Labs, which calculates that training GPT-3 on a single RTX 8000 consumer GPU would take 665 years:
- Language Models are Few-Shot Learners (2023) by Brown et al., https://arxiv.org/abs/2005.14165

© Manning Publications Co. To comment go to liveBook

Licensed to lijin lijin <jimlee13894896995@gmail.com>
- OpenAI's GPT-3 Language Model: A Technical Overview, https://lambdalabs.com/blog/demystifying-gpt-3

NanoGPT is a code repository with a minimalist yet efficient implementation of a GPT-2 model, similar to the model implemented in this book. While the code in this book is different from nanoGPT, this repository inspired the reorganization of a large GPT Python parent class implementation into smaller submodules:
- NanoGPT, a repository for training medium-sized GPTs, https://github. com/karpathy/nanoGPT

An informative blog post showing that most of the computation in LLMs is spent in the feed forward layers rather than attention layers when the context size is smaller than 32,000 tokens:
- "In the long (context) run" by Harm de Vries, https://www.harmdevries. com/post/context-length/

\section*{Chapter 5}

A video lecture by the author detailing the loss function and applying a log transformation to make it easier to handle for mathematical optimization:
- L8.2 Logistic Regression Loss Function, https://www.youtube.com/watch? \(\underline{v}=\mathrm{GxJeODZvydM}\)

The following two papers detail the dataset, hyperparameter, and architecture details used for pretraining LLMs:
- \(\quad\) Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023) by Biderman et al., https://arxiv.org/abs/2304.01373
- OLMo: Accelerating the Science of Language Models (2024) by Groeneveld et al., https://arxiv.org/abs/2402.00838

The following supplementary code available for this book contains instructions for preparing 60,000 public domain books from Project Gutenberg for LLM training:
- Pretraining GPT on the Project Gutenberg Dataset, https://github. com/rasbt/LLMs-from-scratch/tree/main/ch05/03 bonus_pretraining_ on gutenberg

Chapter 5 discusses the pretraining of LLMs, and Appendix D covers more advanced training functions, such as linear warmup and cosine annealing. The following paper finds that similar techniques can be successfully applied to continue pretraining already pretrained LLMs, along with additional tips and insights:
- Simple and Scalable Strategies to Continually Pre-train Large Language Models (2024) by Ibrahim et al., https://arxiv.org/abs/2403.08763

BloombergGPT is an example of a domain-specific large language model (LLM) created by training on both general and domain-specific text corpora, specifically in the field of finance:
- BloombergGPT: A Large Language Model for Finance (2023) by Wu et al., https://arxiv.org/abs/2303.17564

GaLore is a recent research project that aims to make LLM pretraining more efficient. The required code change boils down to just replacing PyTorch's Adamw optimizer in the training function with the GaLoreAdamw optimizer provided by the galore-torch Python package.
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (2024) by Zhao et al., https://arxiv.org/abs/2403.03507
- GaLore code repository, https://github.com/jiaweizzhao/GaLore

The following papers and resources share openly available, large-scale pretraining datasets for LLMs that consist of hundreds of gigabytes to terabytes of text data:
- Dolma: an Open Corpus of Three Trillion Tokens for LLM Pretraining Research by Soldaini et al. 2024, https://arxiv.org/abs/2402.00159
- The Pile: An 800GB Dataset of Diverse Text for Language Modeling by Gao et al. 2020, https://arxiv.org/abs/2101.00027
- The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only, by Penedo et al. (2023) https://arxiv. org/abs/2306.01116
- RedPajama by Together AI, https://github.com/togethercomputer/ RedPajama-Data
- The FineWeb dataset, which includes over 15 trillion tokens of cleaned and deduplicated English web data sourced from CommonCrawl, https://huggingface.co/datasets/Hug.gingFaceFW/fineweb

The paper that originally introduced top-k sampling:
- Hierarchical Neural Story Generation by Fan et al. (2018), https://arxiv. org/abs/1805.04833

Beam search (not cover in chapter 5) is an alternative decoding algorithm that generates output sequences by keeping only the top-scoring partial sequences at each step to balance efficiency and quality:
- Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models by Vijayakumar et al. (2016), https://arxiv.org/abs/1610.02424

\section*{Chapter 6}

Additional resources that discuss the different types of finetuning:
- Using and Finetuning Pretrained Transformers, https://magazine. sebastianraschka.com/p/using-and-finetuning-pretrained-transformers
- Finetuning Large Language Models, https://magazine.sebastianraschka. com/p/finetuning-large-language-models

Additional experiments, including a comparison of finetuning the first output token versus the last output token, can be found in the supplementary code material on GitHub:
- Additional spam classification experiments, https://github.com/rasbt/ LLMs-from-scratch/tree/main/ch06/02 bonus additional-experiments

For a binary classification task, such as spam classification, it is technically possible to only use a single output node instead of two output nodes, as I discussed in the following article:
- Losses Learned-Optimizing Negative Log-Likelihood and Cross-Entropy in PyTorch, https://sebastianraschka.com/blog/2022/losses-learned-part1. html

You can find additional experiments on finetuning different layers of an LLM in the following article, which shows that finetuning the last transformer block, in addition to the output layer, improves the predictive performance substantially:
- Finetuning Large Language Models, https://magazine.sebastianraschka. com/p/finetuning-large-language-models

Readers can find additional resources and information for dealing with imbalanced classification datasets in the imbalanced-learn documentation:
- Imbalanced-learn user guide, https://imbalanced-learn.org/stable/user guide.html

For readers interested in classifying spam emails rather than spam text messages, the following resource provides a large email spam classification dataset in a convenient CSV format similar to the dataset format used in chapter 6:
- Email spam classification dataset, https://hug.gingface.co/datasets/ TrainingDataPro/email-spam-classification

GPT-2 is a model based on the decoder module of the transformer architecture, and its primary purpose is to generate new text. As an alternative, encoder-based models such as BERT and RoBERTa can be effective for classification tasks:

©( Manning Publications Co. To comment go to liveBook

Licensed to lijin lijin <jimlee13894896995@gmail.com>
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018) by Devlin et al., https://arxiv.org/abs/1810.04805
- RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019) by Liu et al., https://arxiv.org/abs/1907.11692
- Additional Experiments Classifying the Sentiment of 50k IMDB Movie Reviews, https://github.com/rasbt/LLMs-from-scratch/blob/main/ch06/ 03 bonus imdb-classification

Recent papers are showing that the classification performance can be further improved by removing the causal mask during classification-finetuning alongside other modifications:
- Label Supervised LLaMA Finetuning (2023) by Li et al., https://arxiv. org/abs/2310.01208
- LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024) by BehnamGhader et al., https://arxiv.org/abs/2404.05961

\section*{Chapter 7}

The Alpaca dataset for instruction finetuning contains 52 thousand instruction-response pairs and is one of the first and most popular publicly available datasets for instruction finetuning:
- Stanford Alpaca: An Instruction-following Llama Model, https://github. com/tatsu-lab/stanford alpaca

Listed below are additional publicly accessible datasets suitable for instruction finetuning:
- LIMA, https://huggingface.co/datasets/GAIR/lima; one thousand high quality instruction-response pairs; for more information, see the LIMA: Less Is More for Alignment (2023) paper, https://arxiv.org/abs/2305. 11206
- UltraChat, https://huggingface.co/datasets/openchat/ultrachat-sharegpt; a large-scale dataset consisting of 805 thousand instruction-response pairs; for more information, see the Enhancing Chat Language Models by Scaling High-quality Instructional Conversations (2023) paper, https://arxiv.org/abs/2305.14233
- Alpaca GPT4, https://github.com/Instruction-Tuning-with-GPT-4/GPT-4LLM/blob/main/data/alpaca g.pt4 data.json, an Alpaca-like dataset with 52 thousand instruction-response pairs generated with GPT-4 instead of GPT-3.5

Phi-3 is a 3.8B parameter model with an instruction-finetuned variant that is reported to be comparable to much larger proprietary models, such as GPT-3.5:
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone (2024) by Abdin et al., https://arxiv.org/abs/2404.14219

(c) Manning Publications Co. To comment go to liveBook

Licensed to lijin lijin <jimlee13894896995@gmail.com>

Researchers propose a synthetic instruction data generation method that generates 300,000 high-quality instruction-response pairs from an instruction-finetuned Llama-3 model. A pretrained Llama 3 base model finetuned on these instruction examples performs comparably to the original instruction-finetuned Llama-3 model:
- Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024) by Xu et al., https://arxiv.org/abs/2406.08464

Research has shown that not masking the instructions and inputs in instruction finetuning effectively improves performance on various NLP tasks and open-ended generation benchmarks, particularly when trained on datasets with lengthy instructions and brief outputs or when using a small number of training examples:
- Instruction Tuning With Loss Over Instructions (2024) by Shi, https://arxiv.org/abs/2405.14394

Prometheus and PHUDGE are openly available LLMs that match GPT-4 in evaluating longform responses with customizable criteria. We haven't used these in chapter 7 because they are not yet supported by Ollama, and thus cannot be executed efficiently on a laptop.
- \(\quad\) Prometheus: Inducing Finegrained Evaluation Capability in Language Models (2023) by Kim et al., https://arxiv.org/abs/2310.08491
- PHUDGE: Phi-3 as Scalable Judge (2024) by Deshwal and Chawla, https://arxiv.org/abs/2405.08029
- \(\quad\) Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (2024), https://arxiv.org/abs/2405.01535

The results in the following report support the view that large language models primarily acquire factual knowledge during pretraining and that finetuning mainly enhances their efficiency in using this knowledge. Furthermore, this study explores how finetuning large language models with new factual information affects their ability to use preexisting knowledge, revealing that models learn new facts more slowly and their introduction during finetuning increases the model's tendency to generate incorrect information:
- Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?

(2024) by Gekhman, https://arxiv.org/abs/2405.05904

Preference finetuning is an optional step after instruction finetuning to align the LLM more closely with human preferences. The following articles by the author provide more information about this process:
- LLM Training: RLHF and Its Alternatives, https://magazine. sebastianraschka.com/p/llm-training-rlhf-and-its-alternatives
- Tips for LLM Pretraining and Evaluating Reward Models, https://sebastianraschka.com/blog/2024/research-papers-in-march2024.html

\section*{Appendix C. Exercise Solutions}

The complete code examples for the exercises answers can be found in the supplementary GitHub repository at https://github.com/rasbt/LLMs-from-scratch.

\section*{Chapter 2}

\section*{EXERCISE 2.1}

You can obtain the individual token IDs by prompting the encoder with one string at a time:

...

This prints:

[33901]

[86]

\# . . .

You can then use the following code to assemble the original string:

print(tokenizer.decode([33901, 86, 343, 86, 220, 959]))

This returns:

'Akwirw ier'

\section*{EXERCISE 2.2}

The code for the data loader with max_length=2 and stride=2:

dataloader = create_dataloader(raw_text, batch_size=4, max_length=2, stride=2)

It produces batches of the following format:

\[
\begin{aligned}
& \text { tensor([[ 40, 367], } \\
& {[2885,1464] \text {, }} \\
& \text { [1807, 3619], } \\
& [402,271]])
\end{aligned}
\]

The code of the second data loader with max_length=8 and stride=2:

dataloader = create_dataloader(raw_text, batch_size=4, max_length=8, stride=2)

An example batch looks like as follows:

\section*{Chapter 3}

\section*{EXERCISE 3.1}

The correct weight assignment is as follows:

\section*{EXERCISE 3.2}

To achieve an output dimension of 2 , similar to what we had in single-head attention, we need to change the projection dimension d_out to 1 .
d_out = 1

mha = MultiHeadAttentionWrapper(d_in, d_out, block_size, 0.0, num_heads=2)

\section*{EXERCISE 3.3}

The initialization for the smallest GPT-2 model is as follows:

\section*{Chapter 4}

\section*{EXERCISE 4.1}

We can calculate the number of parameters in the feed forward and attention modules as follows:

As we can see, the feed forward module contains approximately twice as many parameters as the attention module:

Total number of parameters in feed forward module: \(4,722,432\)

Total number of parameters in attention module: \(2,360,064\)

\section*{EXERCISE 4.2}

To instantiate the other GPT model sizes, we can modify the configuration dictionary as follows (here shown for GPT-2 XL):

Then, reusing the code from Section 4.6 to calculate the number of parameters and RAM requirements, we find the following:

gpt2-xl:

Total number of parameters: 1,637,792,000

Number of trainable parameters considering weight tying: 1,557,380,800

Total size of the model: 6247.68 MB

\section*{Chapter 5}

\section*{EXERCISE 5.1}

We can print the number of times the token (or word) "pizza" is sampled using the print_sampled_tokens function we defined in this section. Let's start with the code we defined in section 5.3.1.

The "pizza" token is sampled \(0 x\) if the temperature is 0 or 0.1 , and it is sampled \(32 \times\) if the temperature is scaled up to 5 . The estimated probability is \(32 / 1000 \times 100 \%=3.2 \%\).

The actual probability is \(4.3 \%\) and contained in the rescaled softmax probability tensor (scaled_probas[2][6]).

\section*{EXERCISE 5.2}

Top-k sampling and temperature scaling are settings that have to be adjusted based on the LLM and the desired degree of diversity and randomness in the output.

When using relatively small top-k values (e.g., smaller than 10) and the temperature is set below 1, the model's output becomes less random and more deterministic. This setting is useful when we need the generated text to be more predictable, coherent, and closer to the most likely outcomes based on the training data.

Applications for such low k and temperature settings include generating formal documents or reports where clarity and accuracy are most important. Other examples of applications include technical analysis or code generation tasks, where precision is crucial. Also, question answering and educational content require accurate answers where a temperature below 1 is helpful.

On the other hand, larger top-k values (e.g., values in the range of 20 to 40) and temperature values above 1 are useful when using LLMs for brainstorming or generating creative content, such as fiction.

\section*{EXERCISE 5.3}

There are multiple ways to force deterministic behavior with the generate function:

1. Setting to top_k=None and applying no temperature scaling;

2. Setting top_k=1.

\section*{EXERCISE 5.4}

In essence, we have to load the model and optimizer that we saved in the main chapter:

Then, call the train_simple_function with num_epochs=1 to train the model for another epoch.

\section*{EXERCISE 5.5}

We can use the following code to calculate the training and validation set losses of the GPT model:

The resulting losses for the 124 M parameter are as follows:

Training loss: 3.754748503367106

Validation loss: 3.559617757797241

The main observation is that the training and validation set performances are in the same ballpark. This can have multiple explanations.

1. The Verdict was not part of the pretraining dataset when OpenAI trained GPT-2. Hence, the model is not explicitly overfitting to the training set and performs similarly well on The Verdict's training and validation set portions. (The validation set loss is slightly lower than the training set loss, which is unusual in deep learning. However, it's likely due to random noise since the dataset is relatively small. In practice, if there is no overfitting, the training and validation set performances are expected to be roughly identical).

2. The Verdict was part of GPT -2's training dataset. In this case, we can't tell whether the model is overfitting the training data because the validation set would have been used for training as well. To evaluate the degree of overfitting, we'd need a new dataset generated after OpenAI finished training GPT-2 to make sure that it couldn't have been part of the pretraining.

\section*{EXERCISE 5.6}

In the main chapter, we experimented with the smallest GPT-2 model, which has only 124 M parameters. The reason was to keep the resource requirements as low as possible. However, you can easily experiment with larger models with minimal code changes. For example, instead of loading the 1558 M instead of 124 M model weights in chapter 5 , the only 2 lines of code that we have to change are the following:

The updated code is as follows:

\section*{Chapter 6}

\section*{EXERCISE 6.1}

We can pad the inputs to the maximum number of tokens the model supports by setting the max length to max_length \(=1024\) when initializing the datasets:

However, the additional padding results in a substantially worse test accuracy of \(78.33 \%\) (versus the \(95.67 \%\) in the main chapter).

\section*{EXERCISE 6.2}

Instead of finetuning just the final transformer block, we can finetune the entire model by removing the following lines from the code:
for param in model.parameters():

param.requires_grad = False

This modification results in a \(1 \%\) improved test accuracy of \(96.67 \%\) (versus the \(95.67 \%\) in the main chapter).

\section*{EXERCISE 6.3}

Rather than finetuning the last output token, we can finetune the first output token by changing model(input_batch) [:, -1, :] to model(input_batch) [:, 0, :] everywhere in the code.

As expected, since the first token contains less information than the last token, this change results in a substantially worse test accuracy of \(75.00 \%\) (versus the \(95.67 \%\) in the main chapter).

\section*{Chapter 7}

\section*{EXERCISE 7.1}

The Phi-3 prompt format, which is shown in figure 7.4 in chapter 7, looks like as follows for a given example input:

Identify the correct spelling of the following word: 'Occasion' The correct spelling is 'Occasion'. ```

def format_input(entry):
    instruction_text = (
        f"<|user|>\n{entry['instruction']}"
    )
    input_text = f"\n{entry['input']}" if entry["input"] else ""
    return instruction_text + input_text

for i, entry in tqdm(enumerate(test_data), total=len(test_data)):
    input_text = format_input(entry)
    tokenizer=tokenizer
    token_ids = generate(
        model=model,
        idx=text_to_token_ids(input_text, tokenizer).to(device),
        max_new_tokens=256,
        context_size=BASE_CONFIG["context_length"],
        eos_id=50256
    )
    generated_text = token_ids_to_text(token_ids, tokenizer)
    response_text = ( #A
        generated_text[len(input_text):]
        .replace("<|assistant|>:", "")
        .strip()
    )
    test_data[i]["model_response"] = response_text

class InstructionDataset(Dataset):
    def __init__(self, data, tokenizer):
        self.data = data
        self.instruction_lengths = [] #A
        self.encoded_texts = []
        for entry in data:
            instruction_plus_input = format_input(entry)
            response_text = f"\n\n### Response:\n{entry['output']}"
            full_text = instruction_plus_input + response_text
            self.encoded_texts.append(
                tokenizer.encode(full_text)
            )
            instruction_length = len(tokenizer.encode(instruction_plus_input))
            self.instruction_lengths.append(instruction_length) #B
    def __getitem__(self, index): #C
        return self.instruction_lengths[index], self.encoded_texts[index]
    def __len__(self):
        return len(self.data)

def custom_collate_fn(
    batch,
    pad_token_id=50256,
    ignore_index=-100,
    allowed_max_length=None,
    device="cpu"
):
    #A
    batch_max_length = max(len(item)+1 for instruction_length, item in batch)
    inputs_lst, targets_lst = [], []
    for instruction_length, item in batch:

new_item = item.copy()
    new_item += [pad_token_id]
    padded = new_item + [pad_token_id] * (batch_max_length - len(new_item))
    inputs = torch.tensor(padded[:-1])
    targets = torch.tensor(padded[1:])
    mask = targets == pad_token_id
    indices = torch.nonzero(mask).squeeze()
    if indices.numel() > 1:
        targets[indices[1:]] = ignore_index
    targets[:instruction_length-1] = -100
        #B
    if allowed_max_length is not None:
        inputs = inputs[:allowed_max_length]
        targets = targets[:allowed_max_length]
    inputs_lst.append(inputs)
    targets_lst.append(targets)
inputs_tensor = torch.stack(inputs_lst).to(device)
targets_tensor = torch.stack(targets_lst).to(device)
return inputs_tensor, targets_tensor

EXERCISE 7.3 练习 7.3

url = "https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/ch07/01_main-
chapter-code/instruction-data.json"

EXERCISE 7.4 练习 7.4

total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Total trainable parameters before: {total_params:,}")
for param in model.parameters():
    param.requires_grad = False
total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Total trainable parameters after: {total_params:,}")
replace_linear_with_lora(model, rank=16, alpha=16)
total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Total trainable LoRA parameters: {total_params:,}")
model.to(device)

Appendix D. Adding Bells and Whistles to the Training Loop
附录 D.为训练循环添加铃声和口哨声

import torch
from previous_chapters import GPTModel
GPT_CONFIG_124M = {
    "vocab_size": 50257, # Vocabulary size
    "ctx_len": 256, # Shortened context length (orig: 1024)
    "emb_dim": 768, # Embedding dimension
    "n_heads": 12, # Number of attention heads
    "n_layers": 12, # Number of layers
    "drop_rate": 0.1, # Dropout rate
    "qkv_bias": False # Query-key-value bias
}
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
torch.manual_seed(123)
model = GPTModel(GPT_CONFIG_124M)
model.eval()

import os
import urllib.request
file_path = "the-verdict.txt"
url = "https://raw.githubusercontent.com/rasbt/LLMs-from-scratch/main/ch02/01_main-
chapter-code/the-verdict.txt"
if not os.path.exists(file_path):
    with urllib.request.urlopen(url) as response:
        text_data = response.read().decode('utf-8')
    with open(file_path, "w", encoding="utf-8") as file:
        file.write(text_data)
else:
    with open(file_path, "r", encoding="utf-8") as file:
        text_data = file.read()
Next, we load the text_data into the data loaders:
from previous_chapters import create_dataloader_v1
train_ratio = 0.90
split_idx = int(train_ratio * len(text_data))
torch.manual_seed(123)
train_loader = create_dataloader_v1(
    text_data[:split_idx],
    batch_size=2,
    max_length=GPT_CONFIG_124M["ctx_len"],
    stride=GPT_CONFIG_124M["ctx_len"],
    drop_last=True,
    shuffle=True
)
val_loader = create_dataloader_v1(
    text_data[split_idx:],
    batch_size=2,
    max_length=GPT_CONFIG_124M["ctx_len"],
    stride=GPT_CONFIG_124M["ctx_len"],
    drop_last=False,
    shuffle=False
)

D. 1 Learning rate warmup
D. 1 学习率预热

n_epochs = 15
initial_lr = 0.0001
peak_lr = 0.01
warmup_steps = 20

optimizer = torch.optim.AdamW(model.parameters(), weight_decay=0.1)
lr_increment = (peak_lr - initial_lr) / warmup_steps
\#A
global_step = -1
track_lrs = []
for epoch in range(n_epochs): #B
    for input_batch, target_batch in train_loader:
        optimizer.zero_grad()
        global_step += 1
        if global_step < warmup_steps: #C
            lr = initial_lr + global_step * lr_increment
        else:
            lr = peak_lr
        for param_group in optimizer.param_groups:
                                #D
            param_group["lr"] = lr
        track_lrs.append(optimizer.param_groups[0]["lr"])

import matplotlib.pyplot as plt
plt.ylabel("Learning rate")
plt.xlabel("Step")
total_training_steps = len(train_loader) * n_epochs
plt.plot(range(total_training_steps), track_lrs);
plt.show()

D. 2 Cosine decay
双余弦衰减

import math
min_lr = 0.1 * initial_lr
track_lrs = []
lr_increment = (peak_lr - initial_lr) / warmup_steps
global_step = -1
for epoch in range(n_epochs):
    for input_batch, target_batch in train_loader:
        optimizer.zero_grad()
        global_step += 1
        if global_step < warmup_steps:
            lr = initial_lr + global_step * lr_increment
        else:# #B
            progress = ((global_step - warmup_steps) /
                (total_training_steps - warmup_steps))
            lr = min_lr + (peak_lr - min_lr) * 0.5 * (1 + math.cos(math.pi * progress))
        for param_group in optimizer.param_groups:
            param_group["lr"] = lr
        track_lrs.append(optimizer.param_groups[0]["lr"])

plt.ylabel("Learning rate")
plt.xlabel("Step")
plt.plot(range(total_training_steps), track_lrs)
plt.show()

D. 3 Gradient clipping
D.3 梯度裁剪

from previous_chapters import calc_loss_batch
torch.manual_seed(123)
model = GPTModel(GPT_CONFIG_124M)
loss = calc_loss_batch(input_batch, target_batch, model, device)
loss.backward()

def find_highest_gradient(model):
    max_grad = None
    for param in model.parameters():
        if param.grad is not None:
            grad_values = param.grad.data.flatten()
            max_grad_param = grad_values.max()
            if max_grad is None or max_grad_param > max_grad:
                max_grad = max_grad_param
    return max_grad
print(find_highest_gradient(model))

D. 4 The modified training function
修改后的训练函数

from previous_chapters import evaluate_model, generate_and_print_sample
def train_model(model, train_loader, val_loader, optimizer, device, n_epochs,
    eval_freq, eval_iter, start_context, warmup_steps=10,

initial_lr=3e-05, min_lr=1e-6):
train_losses, val_losses, track_tokens_seen, track_lrs = [], [], [], []
tokens_seen, global_step = 0, -1
peak_lr = optimizer.param_groups[0]["lr"] #A
total_training_steps = len(train_loader) * n_epochs #B
lr_increment = (peak_lr - initial_lr) / warmup_steps #C
for epoch in range(n_epochs):
    model.train()
    for input_batch, target_batch in train_loader:
        optimizer.zero_grad()
        global_step += 1
        if global_step < warmup_steps:
            #D
        lr = initial_lr + global_step * lr_increment
    else:
        progress = ((global_step - warmup_steps) /
            (total_training_steps - warmup_steps))
        lr = min_lr + (peak_lr - min_lr) * 0.5 * (
            1 + math.cos(math.pi * progress))
        for param_group in optimizer.param_groups: #E
        param_group["lr"] = lr
    track_lrs.append(lr)
    loss = calc_loss_batch(input_batch, target_batch, model, device)
    loss.backward()
    if global_step > warmup_steps: #F
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
                                #G
    optimizer.step()
    tokens_seen += input_batch.numel()
    if global_step % eval_freq == 0:
        train_loss, val_loss = evaluate_model(
            model, train_loader, val_loader,
            device, eval_iter
        )
        train_losses.append(train_loss)
        val_losses.append(val_loss)
        track_tokens_seen.append(tokens_seen)
        print(f"Ep {epoch+1} (Iter {global_step:06d}): "
            (c) Manning Publications Co. To comment go to liveBook

f"Train loss {train_loss:.3f}, Val loss {val_loss:.3f}")
训练损失 {train_loss:.3f}，验证损失 {val_loss:.3f}

generate_and_print_sample(
    model, train_loader.dataset.tokenizer,
    device, start_context
)

torch.manual_seed(123)
model = GPTModel(GPT_CONFIG_124M)
model.to(device)
peak_lr = 5e-4
optimizer = torch.optim.AdamW(model.parameters(), weight_decay=0.1)
n_epochs = 15
train_losses, val_losses, tokens_seen, lrs = train_model(
    model, train_loader, val_loader, optimizer, device, n_epochs=n_epochs,
    eval_freq=5, eval_iter=1, start_context="Every effort moves you",
    warmup_steps=10, initial_lr=1e-5, min_lr=1e-5
)

Ep 1 (Iter 000000): Train loss 10.934, Val loss 10.939
Ep 1 (Iter 000005): Train loss 8.529, Val loss 8.843

Ep 2 (Iter 000010): Train loss 6.400, Val loss 6.825
Ep 2 (Iter 000015): Train loss 6.116, Val loss 6.861
Every effort moves you,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
...
the irony. She wanted him vindicated--and by me!" He laughed again, and threw back his
head to look up at the sketch of the donkey. "There were days when I
Ep 15 (Iter 000130): Train loss 0.101, Val loss 6.707
Every effort moves you?" "Yes--quite insensible to the irony. She wanted him
vindicated--and by me!" He laughed again, and threw back his head to look up at the
sketch of the donkey. "There were days when I

Appendix Parameter-efficient Finetuning with LoRA
附录 使用 LoRA 进行参数高效的微调

E. 1 Introduction to LoRA
LoRA 简介

E. 2 Preparing the dataset
2.准备数据集

from pathlib import Path
import pandas as pd
from ch06 import
    download_and_unzip_spam_data,
    create_balanced_dataset,
    random_split
)
url = "https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip"
zip_path = "sms_spam_collection.zip"
extracted_path = "sms_spam_collection"
data_file_path = Path(extracted_path) / "SMSSpamCollection.tsv"
download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)
df = pd.read_csv(data_file_path, sep="\t", header=None, names=["Label", "Text"])
balanced_df = create_balanced_dataset(df)
balanced_df["Label"] = balanced_df["Label"].map({"ham": 0, "spam": 1})
train_df, validation_df, test_df = random_split(balanced_df, 0.7, 0.1)
train_df.to_csv("train.csv", index=None)
validation_df.to_csv("validation.csv", index=None)
test_df.to_csv("test.csv", index=None)

Listing E. 2 Instantiating PyTorch datasets
列表 E. 2 实例化 PyTorch 数据集

import torch
from torch.utils.data import Dataset
import tiktoken
from previous_chapters import SpamDataset
tokenizer = tiktoken.get_encoding("gpt2")
train_dataset = SpamDataset("train.csv", max_length=None, tokenizer=tokenizer)
val_dataset = SpamDataset("validation.csv", max_length=train_dataset.max_length,
tokenizer=tokenizer)
test_dataset = SpamDataset("test.csv", max_length=train_dataset.max_length,
tokenizer=tokenizer)

num_workers = 0
batch_size = 8
torch.manual_seed(123)
train_loader = DataLoader(
    dataset=train_dataset,
    batch_size=batch_size,
    shuffle=True,
    num_workers=num_workers,
    drop_last=True,
)
val_loader = DataLoader(
    dataset=val_dataset,
    batch_size=batch_size,
    num_workers=num_workers,
    drop_last=False,
)
test_loader = DataLoader(
    dataset=test_dataset,
    batch_size=batch_size,
    num_workers=num_workers,
    drop_last=False,
)

print("Train loader:")
for input_batch, target_batch in train_loader:
    pass
print("Input batch dimensions:", input_batch.shape)
print("Label batch dimensions", target_batch.shape)

print(f"{len(train_loader)} training batches")
print(f"{len(val_loader)} validation batches")
print(f"{len(test_loader)} test batches")

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-394.jpg?height=33&width=326&top_left_y=816&top_left_x=259)

\section*{E. 3 Initializing the model}

This section repeats the code from chapter 6 to load and prepare the pretrained GPT model. We begin with downloading the model weights and loading them into the GPTModel class:

\section*{Listing E. 4 Loading a pretrained GPT model}

from gpt_download import download_and_load_gpt2

from previous_chapters import GPTModel, load_weights_into_gpt

BASE CONFIG.update(model configs[CHOOSE MODEL])

model_size = CHOOSE_MODEL.split(" ")[-1].lstrip("(").rstrip(")")

settings, params = download_and_load_gpt2(model_size=model_size, models_dir="gpt2")

model = GPTModel(BASE_CONFIG)

load_weights_into_gpt(model, params)

model.eval()

To ensure that the model was loaded corrected, let's double-check that it generates coherent text:

As we can see based on the output below, the model generates coherent text, which is an indicator that the model weights were loaded correctly:

Every effort moves you forward.

The first step is to understand the importance of your work

Next, we prepare the model for classification-finetuning similar to chapter 6 , where we replace the output layer:

Lastly, let's calculate the initial classification accuracy of the not-finetuned model (we expect this to be around \(50 \%\), which means that the model is not able to distinguish between spam and non-spam messages yet reliably):

The initial prediction accuracies are as follows:

Training accuracy: \(46.25 \%\)

Validation accuracy: \(45.00 \%\)

Test accuracy: \(48.75 \%\)

\section*{E. 4 Parameter-efficient finetuning with LoRA}

In this section, we modify and finetune the LLM using LoRA. We begin by initializing a LoRALayer that creates the matrices \(A\) and \(B\), along with the alpha scaling factor and the rank ( \(r\) ) setting.

This layer can accept an input and compute the corresponding output, as illustrated in Figure E.2.

\section*{Outputs}

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-398.jpg?height=677&width=1018&top_left_y=284&top_left_x=526)

Figure E. 2 The LoRA matrices \(A\) and \(B\) are applied to the layer inputs and are involved to compute the model outputs. The inner dimension \(r\) of these matrices serves as a setting that adjusts the number of trainable parameters by varying the sizes of \(A\) and \(B\).

In code, this LoRA layer depicted Figure E. 2 can be implemented as follows:

Listing E. 5 Implementing a LoRA layer

\#A Same initialization that is used for Linear layers in PyTorch

In the preceding code, the rank governs the inner dimension of matrices \(A\) and \(B\). Essentially, this setting determines the number of extra parameters introduced by LoRA, which is used to balance between the adaptability of the model and its efficiency via the number of parameters used.

The other important setting, alpha, functions as a scaling factor for the output from the low-rank adaptation. It primarily dictates the degree to which the output from the adapted layer can impact the original layer's output. This can be seen as a way to regulate the impact of the low-rank adaptation on the layer's output.

The LoRALayer class we have implemented so far enables us to transform the inputs of a layer.

In LoRA, the typical goal is to substitute existing Linear layers, allowing weight updates to be applied directly to the pre-existing pretrained weights, as illustrated in Figure E.3.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-399.jpg?height=630&width=1219&top_left_y=781&top_left_x=312)

Figure E. 3 Illustration of the integration of LoRA into a model layer. The original pretrained weights ( \(W\) ) of a layer are combined with the outputs from LoRA matrices ( \(A\) and \(B\) ), which approximate the weight update matrix ( \(\triangle W\) ). The final output is calculated by adding the output of the adapted layer (using LoRA weights) to the original output.

To integrate the original Linear layer weights, as shown in Figure E.3, we now create a LinearWithLoRA layer. This layer utilizes the previously implemented LoRALayer and is designed to replace existing Linear layers within a neural network, such as the selfattention modules or feed-forward modules in the GPTModel:

\section*{Listing E. 6 A LinearWithLora layer to replace Linear layers}

The preceding code combines a standard Linear layer with the LoRALayer. The forward method computes the output by adding the results from the original linear layer and the LoRA layer.

Since the weight matrix \(B\) (self.B in LoRALayer) is initialized with zero values, the product of matrices \(A\) and \(B\) results in a zero matrix. This ensures that the multiplication does not alter the original weights, as adding zero does not change them.

To apply LoRA to the earlier defined GPTModel, we also introduce a replace_linear_with_lora function. This function will swap all existing Linear layers in the model with the newly created LinearWithLoRA layers:

\#A Replace the Linear layer with LinearWithLoRA

\#B Recursively apply the same function to child modules

We have now implemented all the necessary code to replace the Linear layers in the GPTModel with the newly developed LinearWithLoRA layers for parameter-efficient finetuning. In the subsequent sections, we will apply the LinearWithLoRA upgrade to all Linear layers found in the multi-head attention, feed-forward modules, and the output layer of the GPTModel, as illustrated in Figure E.4.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-401.jpg?height=1176&width=1247&top_left_y=202&top_left_x=296)

Every effort moves you

Figure E. 4 A diagram representing the architecture of the GPT model. It highlights the parts of the model where Linear layers are being upgraded to LinearWithLoRA layers for parameter-efficient finetuning.

Before we apply the LinearWithLoRA layer upgrades depicted in Figure E.4, we first freeze the original model parameters:

After the preceding code, we can see that none of the 124 M model parameters are trainable:

Total trainable parameters before: \(124,441,346\)

Total trainable parameters after: 0

Next, we use the replace_linear_with_lora to replace the Linear layers:

The number of trainable parameters, after adding the LoRA layers, is as follows:

Total trainable LoRA parameters: 2,666,528

As we can see, we reduced the number of trainable parameters by almost 50 x when using LoRA. A rank and alpha of 16 are good default choices, but it is also common to increase the rank parameter, which in turn increases the number of trainable parameters. Alpha is usually chosen to be half, double, or equal to the rank.

Let's verify that the layers have been modified as intended by printing the model architecture:

The output is as follows:

As we can see based on the preceding output, the model now includes the new LinearWithLoRA layers, which themselves consist of the original Linear layers, which we set to non-trainable, and the new LoRA layers, which we will finetune.

However, before we begin finetuning the model, let's calculate the initial classification accuracy:

The resulting accuracy values are as follows:

Training accuracy: \(46.25 \%\)

Validation accuracy: \(45.00 \%\)

Test accuracy: \(48.75 \%\)

If we compare these accuracy values to the initial ones from chapter 6 , we notice they are identical. This occurs because we initialized the LoRA matrix \(B\) with zeros. Consequently, the product of matrices \(A B\) results in a zero matrix. This ensures that the multiplication does not alter the original weights since adding zero does not change them before we start finetuning.

Now, let's move on to the exciting part and finetune the model using the training function from chapter 6. The training takes about 15 minutes on an M3 MacBook Air laptop and less than half a minute on a V100 or A100 GPU:

\section*{Listing E. 7 Finetuning a model with LoRA layers}

The output we see during the training is as follows:

Training completed in 12.10 minutes.

Note that training the model with LoRA takes longer than training it without LoRA in chapter 6, because the LoRA layers introduce an additional computation during the forward pass. However, for larger models, where backpropagation becomes more costly, models typically train faster with LoRA than without it.

As we can see, the model received perfect training and very high validation accuracy. Let's also visualize the loss curves to better see if the training has converged:

The resulting plot is shown in Figure E.5.

![](https://cdn.mathpix.com/cropped/2024_07_13_ba8d9eb54f12ebe396d8g-407.jpg?height=664&width=1155&top_left_y=302&top_left_x=372)

Figure E. 5 The training and validation loss curves over six epochs for a machine learning model. Initially, both training and validation loss decrease sharply, then they level off, indicating the model is converging, which means that it is not expected to improve noticeably with further training.

In addition to evaluating the model based on the loss curves shown in E.5, let's also calculate the accuracies on the full training, validation, and test set (during the training, we approximated the training and validation set accuracies from 5 batches via the eval_iter \(=5\) setting):

The resulting accuracy values are as follows:

The given accuracy shows that the model performs well across training, validation, and test datasets. With a training accuracy of \(100 \%\), the model has perfectly learned the training data. However, the slightly lower validation and test accuracies ( \(96.64 \%\) and \(97.33 \%\), respectively) suggest a small degree of overfitting, as the model does not generalize quite as well on unseen data compared to the training set. Overall, the results are very impressive considering that we finetuned only a relatively small number of model weights (2.7 million LoRA weights instead of the original 124 million model weights).