Tiny Llama-2

(Optional) Uploading the model/data to Google Colab or Kaggle.¶

Please upload your dataset and model to computational platforms if you are using Colab or Kaggle environments.

For Colab users, you can mount your Google Drive files by running the following code snippet:

Necessary Packages, Environment Setups¶

Please set the correct file path based on your environment.

• If you are using Colab, the path may be: /content/drive/MyDrive/xxxxxx
• If you are using Kaggle, the path may be: /kaggle/input/xxxxxx

Load the model checkpoint into either a GPU or CPU (training will be slow on CPU, but decoding will be fair).

As we can see from the statistics, this model is much smaller than Llama-2 but shares the same decoder-only architecture.

😄 You do not need to check complex details! We just present the architecture and number of parameters here.

Task 1: Decoding¶

If you are familar with the usage of model.generate() function in transformer library, please feel free to jump to Task 1 Playground.

💡Tutorials: model.generate() function.¶

Minimal example:

prompt = "Once upon a time, " # Input, prefix of generation

Step 1: Encode raw text using tokenizer model.

tokenized_input = tokenizer.encode(prompt, return_tensors='pt').to(device)

Step 2: Set decoding hyper-parameters. Get the model output.

output_ids = model.generate(tokenized_input, do_sample=True, max_new_tokens=300, temperature=0.6)

Important parameters:

• max_new_tokens: The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt.
• temperature: The value of temperature used to modulate the next token probabilities. Higher temperature -> generate more diverse text. Lower temperature -> generate more deterministic text.
• do_sample: do_sample=False is using greedy decoing strategy. To enable greedy decoding, we also need to set other sampling parameters top_p, temperature as None.
• If you are interested in other decoding algorithms, please refer to this link for setting parameters.

Step 3: Convert model outputs into raw text.

output_text = tokenizer.decode(output_ids[0])

or (when input instances >=1)

output_text = tokenizer.batch_decode(output_ids)

Important parameters:

• Setting skip_special_tokens=True will prevent special tokens, such as <s>, from appearing in the results..

To understand the outputs of each step, let us do a simple generation task step by step! (Note: the base model is only able to produce fluent story text).

Another pipeline example: Sampling decoding with temperature.¶

Task 1 Playground¶

📚 Task 1: Please generate English stories using various prompts and decoding settings. Please feel free to explore any interesting phenomena, such as the impact of different prompts and the effects of various decoding algorithms and parameters. For example, quantify the text properties using linguistic-driven metrics like story length and Type-Token Ratio (TTR). In addition to objective metrics, you are encouraged to discuss your findings based on subjective case studies.

We provide two types of skeleton code: one that takes a single prompt as input and another that can process batched inputs and decoding. Please use the version that best fits your preferences and data types.

What about other languages?¶

Oops! This English language model cannot generate stories in other languages!

Why? Let us evaluate the perplexity of different languages in the next task.

Task 2: Perplexity Evaluation¶

Background¶

The perplexity serves as a key metric for evaluating language models. It quantifies how well a model predicts a sample, with lower perplexity indicating better performance. For a tokenized sequence $X=(x_0,x_1,\dots ,x_t)$, the perplexity is defined mathematically as:

$$\text{Perplexity}(X)=\exp (-\frac{1}{t}\sum _{i=1}^t\log p_{\theta }(x_i|x_{<i}))$$

Here, $p_{\theta }(x_i|x_{<i})$ represents the probability of a token $x_i$ given its preceding tokens, and the formulation incorporates the average log probability across the sequence.

⚠️ Please make sure to run the following cell first to define the evaluation function.

😄 You do not need to check these complex details! Too hard for beginners! However, if you are interested, you can compare the following code with the explanations above to better understand how to implement PPL evaluation using PyTorch.

💡Tutorials: compute_ppl() function.¶

Minimal example:

test_dataset = ["Once upon a time,"]

compute_ppl(
    model=model,
    tokenizer=tokenizer,
    device=device,
    inputs=test_dataset,
    batch_size = 16
)

Important parameters:

• inputs: list of input text, each separate text snippet is one list entry.
• batch_size: the batch size to run evaluations.

Returns:

• perplexity: {"perplexities": [x.x, x.x, ...], "mean_perplexity": x.x} dictionary containing the perplexity scores for the texts in the input list, as well as the mean perplexity. .

Task 2 Playground¶

📚 Task 2: Evaluate the perplexity. Ensure that you evaluate both the English and Chinese test data we provided. You are encouraged to collect more diverse text data and discuss your findings regarding the language understanding capacity of the base model.

Note: If you want to reuse the evaluation codes for JSONL data, please structure the content as follows:

{"text": "one data"}
{"text": "two data."}
...

You may find that the PPL value for Chinese text is significantly higher than that for English text. This is evidence that the base model cannot generate a Chinese story at the end of the last task.

Task 3: Continual Pre-training (in Chinese or in another language you are proficient in)¶

Currently, our base English LM is proficient in English but lacks the capability to generate or comprehend other languages (e.g., Chinese). The objective of this task is to enhance a base English LM by continually pre-training it with text in another language. This process aims to enable the model to understand and generate mini-story in another language.

We have provided 10,000 Chinese training samples. The training process for any language is the same. We have included useful resource links (in Assignment description PDF) to help you create additional data. If you encounter any issues in creating a dataset in another language, please do not hesitate to contact us.

We have implemented data preprocessing and the training pipeline, so you are not required to optimize these components. Instead, focus on tuning the training hyperparameters and observe the changes in model performance.

⚠️ Please make sure to run the following cell first to pre-process data.

😄 You do not need to check the details of whole pipeline construction! Please pay attention to the hyper-parameters of trainer.

Preprocess Data¶

Here, we preprocess (tokenize and group) the text for the subsequent evaluation and pre-training phases.

Load prepared Chinese dataset from Google drive (or local disk).

We tokenize the raw text using Llama-2's tokenizer and group the tokenized text as inputs.

💡Tutorials: TrainingArguments().¶

Important Training Hyper-parameters

• learning_rate: The initial learning rate for optimizer.
• num_train_epochs: Total number of training epochs to perform (if not an integer, will perform the decimal part percents of the last epoch before stopping training).
• *_strategy: The evaluation/saving strategy to adopt during training. Possible values are:
  • "no": No evaluation/saving is done during training.
  • "steps": Evaluation/saving is done (and logged) every eval_steps.
  • "epoch": Evaluation/saving is done at the end of each epoch.
• per_device_train_batch_size: The batch size per GPU/XPU/TPU/MPS/NPU core/CPU for training.
• per_device_eval_batch_size: The batch size per GPU/XPU/TPU/MPS/NPU core/CPU for evaluation.
• save_total_limit: If a value is passed, will limit the total amount of checkpoints.

If you do not understand AdamW optimizer and learning scheduler, you may use default settings.

Optimizer Hyper-parameters

• weight_decay: The weight decay to apply (if not zero) to all layers except all bias and LayerNorm weights in [AdamW] optimizer.
• adam_beta1: The beta1 hyperparameter for the [AdamW] optimizer.
• adam_beta2: The beta2 hyperparameter for the [AdamW] optimizer.

Learning schedule

• lr_scheduler: The scheduler type to use.
• warmup_ratio: Ratio of total training steps used for a linear warmup from 0 to learning_rate.

Explore more parameters here

Task 3 Playground¶

📚 Please just run the following code to do continual pre-training. Please try your best to tune the hyperparameters or collect more data to improve model performance.

Load pre-trained model and try to generate mini-story in another language.

Evaluate the PPL on Chinese text (or another language) again.

You will notice that we actually achieve a much lower PPL after continual pre-training.

The original English base model was pre-trained on 2 million data samples. Considering we are using only 10,000 training samples (0.5% of the original pre-training data), the model can generate a few fluent sentences but may still struggle with long-text generation or common sense of other languages. You can try using more data or training steps depending on your computational resources.