Large language models (LLMs) have rapidly advanced natural language processing capabilities through self-supervised learning on vast text corpora. However, developing these complex neural network models requires extensive data, compute, infrastructure, expertise and time.
In this 2650+ word guide, we provide an in-depth reference on architecting, training and deploying large language models, including:
Contents
- Evolution of LLMs
- Sourcing Training Data
- Selecting Model Architecture
- Specialized Hardware
- Distributed Training
- Tuning Training Parameters
- Monitoring and Troubleshooting
- Evaluating LLMs
- Custom Models
- Architectural Innovations
- The Outlook for LLMs
Let‘s explore the key considerations when developing LLMs!
Evolution of LLMs
LLMs have rapidly increased in scale and capability over the past decade:
| Model | Year | Parameters | Data Scale |
|---|---|---|---|
| BERT | 2018 | 340M | 3.3B words |
| GPT-2 | 2019 | 1.5B | 40 GB |
| T5 | 2020 | 11B | 750 GB |
| Switch Transformer | 2021 | 1.6T | 300 TB |
| PaLM | 2022 | 540B | 780 GB |
| Gopher | 2022 | 280B | 300 TB |
| Wu Dao 2.0 | 2022 | 1.75T | Unknown |
Table 1: Growth of LLMs over time. Adapted from Anthropic and AI Safety Camp.
Architecture innovations and exponential data/hardware scaling underpin these remarkable gains. Next we survey key considerations when developing modern LLMs.
Sourcing Training Data
Training data quantity and quality critically impact model performance. Leading efforts use hundreds of billions of words sourced from diverse corpora like web pages, books, academic papers, and Wikipedia.
Data Collection
Public datasets from resources like Common Crawl and BooksCorpus provide rich sources. Other approaches include:
- Web scraping data from websites
- Accessing databases like news archives
- Licensing proprietary datasets
- Creating new human-labeled data
Domain-specific data (e.g. scientific papers) improves performance on niche tasks. Multimodal data with images and videos also provides grounding.
| Dataset | Contents | Volume |
|---|---|---|
| OpenWebText2 | Web content | 40 GB |
| Pubmed Central | Research papers | 538 GB |
| Wikipedia + Books | Encyclopedic knowledge | 100 GB |
| Common Crawl | Diverse web crawl | Petabytes |
Table 2: Examples of popular LLM training datasets
Data Preprocessing
To prepare raw data for model training, key steps include:
- Cleaning: Fix formatting errors, HTML tags, encoding issues
- Deduplicating: Remove duplicate documents
- Tokenizing: Split text into words/subwords
- Filtering: Reduce noise and sensitivity
This outputs structured, tokenized text ready for ingestion by ML systems. DataSet and Tensorflow-Transform accelerate pipeline development.
Selecting Model Architecture
Most modern LLMs use the Transformer architecture (Fig. 2). The Transformer encodes input text into latent representations that are decoded into predictions.
The Transformer architecture underpins most modern LLMs. Credit:Tensorflow.
Key architectural considerations include:
Depth: Number of layers. More layers learn more complex features but reduce interpretability. 60-100+ layers common for SOTA models.
Width: Dimensionality of layers. Wider layers enable more parameters and capacity. Balance with computational efficiency.
Attention: Parallel attention mechanisms model linguistic relationships and dependencies. Standardly uses 8-16 heads now.
Embeddings: How discrete tokens are numerically represented as continuous input vectors. Choice of static or dynamic embeddings.
Leading LLMs are based on three main Transformer variants:
Encoding Models
Encoder-only. Map input text to contextual latent representations. Includes influential models like BERT and RoBERTa.
Decoder Models
Decoder-only. Generate text from random noise vectors. Includes GPT models.
Two-Way Models
Encoder-decoder. Translation models mapping between input and output sequences like T5 and mT5.
Now we survey training hardware and distributed strategies for scaling models.
Specialized Hardware
Training modern LLMs requires clusters of specialized accelerators like GPUs or TPUs.
GPUs
LLMs are commonly trained on general-purpose GPUs which excel at parallel matrix math for deep learning:
Nvidia‘s A100 GPU provides nearly 10,000 CUDA cores. Credit: Nvidia
For example, 1024 A100 GPUs can provide 9.7 petaFLOPS of mixed precision compute. State dynamics improve memory efficiency.
However, GPUs have high power consumption (e.g. 400W per A100) limiting cluster scalability.
TPUs
Google‘s custom Tensor Processing Units specifically target ML workloads:
Google‘s 4th gen TPUs optimize performance and efficiency. Credit: Google
Benefits include optimizing for low precision (bf16/int8) operations, powerful matrix units and mesh interconnects. TPU v4 packs 4,096 cores into each chip delivering 1,303 TFLOPS at 65W power.
Recommendations
GPUs excel providing flexibility and performance for moderate scales (~1000 chips). TPUs enable extreme efficiency at hyperscale. We recommend a hybrid strategy using GPUs to develop and debug smaller models, then switching to TPU pods to train models at full trillion+ parameter scale.
Now we discuss approaches to distribute training across these accelerators.
Distributed Training
Training LLMs relies on distributing work across many devices.
Strategies
Pipeline parallelism – Different modules run sequentially on each device. Saves memory.
Model parallelism – Model layers split across devices enabling huge width.
Data parallelism – Devices train on shards of data then synchronize updates. Most common.
Hybrid parallelism combines strengths of above approaches.
Infrastructure
Efficient distributed training requires high-speed networking like Infiniband and optimized libraries like Horovod.
Cloud platforms provide managed services for scaling including:
- Google Cloud TPU Pods – Interconnected TPUs
- Amazon EC2 Trn1 Instances – 300 Gbps networking between GPUs
- Microsoft NDv3 Series – 8 Nvidia V100 NVLinks GPUs share 640 GB/s bandwidth
On-premise clusters allow greater control but require extensive engineering.
Now we detail key training parameters when developing models.
Tuning Training Parameters
Optimizing combinations of hyperparameters is key to efficiently developing accurate models:
Batch Size
Number of samples processed per step. Typical values range 32-4096.
Larger batches better utilize hardware accelerators but reduce final accuracy, especially with long sequences. Batch size often determined by hardware memory limits.
Learning Rate
Determines magnitude of updates to model weights when estimating loss function gradients.
Too small – Slow convergence. Too large – instability.
Typical values around 1e-3 to 1e-4. Requires decay schedule for stable optimization.
Mixed Precision
Use lower numerical precision formats like FP16 or BF16 to accelerate computation. Nvidia Tensor Cores provide up to 10x higher TFLOPS for FP16 vs FP32 values.
Must watch for underflow and overflow issues. Enable loss scaling to counter.
We visualize model performance over multiple configurations to determine optimal parameters – often through grid search. With infrastructure set up and parameters tuned, we can now execute training.
Monitoring and Troubleshooting
Careful monitoring helps manage clusters and identify issues:
Utilization – Ensure near 100% accelerator usage for maximum efficiency. Insufficient I/O or network speeds reduce utilization.
Throughput – Track tokens processed per second. Varies widely by model architecture.
Loss and Accuracy – Validation performance determines training progress. Alert on plateaus.
Logs – Errors in individual processes. Hardware issues.
Leading platforms provide monitoring dashboards and alerting. ML management systems like Comet and Neptune facilitate experiment tracking and model versioning.
Common bottlenecks include:
- Long sequences reducing batch size
- Optimizer instability
- Insufficient inter-device bandwidth and buffers
Profiling runs determine whether issues stem from computation vs communication limits.
Evaluating LLMs
After training, rigorous evaluation is vital before deployment:
Fluency – Coherence and grammar of generated text. Requires human review.
Relevance – Match predictions to ground truth targets on test set. Measure across tasks like translation, summarization etc.
Bias – Assess gender, race, toxicity etc. Sensitive issues.
Efficiency – Latency, memory usage etc. on production systems.
Teams may collect additional annotations or fine-tune models to address evaluation gaps. Models are exported to formats like TensorFlow SavedModel and ONNX Runtime for production use at low latency across platforms.
Custom Models
Most organizations license models from providers like Anthropic, Cohere, Google and Microsoft which invest in proprietary training infrastructure.
However, some large enterprises develop custom LLMs tailored to their use cases when sufficient technical resources permit.
Benefits of custom models include:
- Adapting model architecture for specific tasks
- Training on sensitive proprietary data
- Closer product integration
- Avoiding licensing fees
Challenges include:
- Extremely scarce model development talent
- Hundreds of millions in compute infrastructure costs
- Months of engineering time
Before embarking on costly endeavors, validated feasibility and business value. We recommend starting small then incrementally scaling resources.
Architectural Innovations
The past year saw remarkable advances in model architecture:
GShard (Meta) – Sparsely activates small fractions of feedforward network width dynamically based on routing tokens for high efficiency.
GLaM (Google) – Memory-efficient attention using global tokens to propagate information between layers, reducing activations.
Mixture-of-Experts (Google) – Conditional model routing to sparsely update subsets of experts specialized in tasks.
These techniques aim to improve scalability and reduce costs to enable larger models. Automated Neural Architecture Search also accelerates design iterations.
In 2023, advancing module architecture may prove even more critical than brute-force model scaling to drive breakthrough capabilities – especially on multimodal domains.
The Outlook for LLMs
Recent models already exceed many practitioners‘ past expectations. So what‘s next?
Model Size – Beyond 1.75 trillion parameters, size may provide diminishing returns without other advances.iska
Multimodal Models – Models incorporating images, videos and speech data develop deeper understanding of the world.
Specialized Models – Models trained on targeted scientific, technical and professional domains improve capabilities.
Architectural Innovation – Automated and efficient model architectures allow training larger models within constraints.
So while scale will continue growing in 2024, we anticipate even greater progress in model quality and usefulness resulting from novel architectures uniquely designed for navigating and reasoning about multifaceted real-world knowledge.
Conclusion
This guide surveyed the intense process of architecting, training and deploying large language models requiring immense datasets, bleeding-edge hardware, distributed systems, machine learning, and product integration expertise.
We hope you found this 2650+ word reference useful! Let us know if you have any other questions.
