The Evolution of Language Models in Deep Learning

Language modeling is fundamentally an exercise in density estimation over discrete sequences: estimating the joint probability distribution of sequences of symbols (words, subwords, or characters) drawn from a finite vocabulary $\mathcal{V}$.

By applying the probability chain rule, any joint probability over a sequence $W = (w_1, w_2, \dots, w_T)$ can be factorized autoregressively:

$P(W) = P(w_1, w_2, \dots, w_T) = \prod_{t=1}^T P(w_t \mid w_1, w_2, \dots, w_{t-1}) = \prod_{t=1}^T P(w_t \mid w_{<t})$

The history of deep learning in natural language processing is the history of finding increasingly expressive, scalable, and parallelizable parameterizations for the conditional distribution $P(w_t \mid w_{<t})$.

---

1. The Pre-Neural Era: Statistical N-Grams and the Curse of Dimensionality

Before neural networks entered the scene, language modeling was treated as a discrete counting problem governed by the Markov assumption of order $k = n - 1$:

$P(w_t \mid w_1, \dots, w_{t-1}) \approx P(w_t \mid w_{t-n+1}, \dots, w_{t-1}) = \frac{C(w_{t-n+1}, \dots, w_{t-1}, w_t)}{C(w_{t-n+1}, \dots, w_{t-1})}$

Where $C(\cdot)$ represents raw n-gram frequency counts in a text corpus.

The Breakdown of Discrete Counting

While intuitive, count-based n-grams collapse under two fundamental barriers:

1. State Space Explosion: The number of possible conditioning contexts scales exponentially as $|\mathcal{V}|^{n-1}$. For a typical vocabulary of $|\mathcal{V}| = 50,000$ and a modest context of $n=5$, the transition tensor contains $50,000^4 \approx 6.25 \times 10^{18}$ parameters. Counting quickly becomes intractable.
2. Zero Semantic Generalization: Discrete tokens are orthogonal one-hot vectors. If the model observed:
   • “The cat sat on the mat” It learned zero statistical evidence about:
   • “The kitten sat on the rug” Any unseen sequence received zero probability ($P = 0$), forcing decades of heuristic smoothing algorithms (Good-Turing, Kneser-Ney, Katz backoff) that merely patched an intrinsically brittle representation.

---

2. Distributed Representations: Bengio’s Neural LM (2003)

Yoshua Bengio et al. revolutionized the field in their seminal 2003 paper, A Neural Probabilistic Language Model, by proposing two radical ideas:

[!NOTE] Detailed Deep Dive For a full architectural and mathematical breakdown of the 2003 model, see the dedicated note: Bengio’s Neural Probabilistic Language Model (2003).

1. Associate each word in the vocabulary with a continuous, dense distributed feature vector $\mathbf{e} \in \mathbb{R}^d$ ($d \ll |\mathcal{V}|$).
2. Express the conditional probability distribution as a smooth, continuous function of these feature vectors parameterized by a neural network.

Mathematically, the architecture maps context tokens to logits $\mathbf{y} \in \mathbb{R}^{|\mathcal{V}|}$:

$\mathbf{x} = \left[ C(w_{t-n+1}) ; C(w_{t-n+2}) ; \dots ; C(w_{t-1}) \right] \in \mathbb{R}^{(n-1)m}$

$\mathbf{y} = b + W \mathbf{x} + U \tanh(d + H \mathbf{x})$

$P(w_t = i \mid w_{<t}) = \frac{\exp(y_i)}{\sum_{j=1}^{|\mathcal{V}|} \exp(y_j)}$

The Breakthrough

Because the mapping from vector space to probability is continuous, small perturbations in word vectors lead to small changes in output probabilities. If “cat” and “kitten” map to nearby points in $\mathbb{R}^d$, the model naturally transfers predictions across semantically similar contexts without explicitly witnessing every combinatorial variant.

The Remaining Bottlenecks

• Fixed Context Window: Like n-grams, Bengio’s model truncated history to a fixed window $n-1$.
• Softmax Bottleneck: Computing the denominator requires a sum over all $|\mathcal{V}|$ classes for every single token update, creating a steep training overhead.

---

3. Vector Semantics: Word2Vec and GloVe (2013–2014)

A decade later, Tomas Mikolov et al. (2013) demonstrated that dense semantic representations could be learned at massive scale by ditching the full autoregressive language model and training shallow linear classifiers instead:

• Continuous Bag-of-Words (CBOW): Predicts current word $w_t$ given context $\{w_{t-c}, \dots, w_{t+c}\}$.
• Skip-Gram: Predicts context window $\{w_{t-c}, \dots, w_{t+c}\}$ given center word $w_t$.

By introducing Negative Sampling (a simplification of Noise Contrastive Estimation), Mikolov converted an expensive multiclass classification problem over $|\mathcal{V}|$ into a binary logistic regression task:

$\mathcal{L}_{\text{Skip-Gram}} = \log \sigma(\mathbf{v}'^\top_{w_O} \mathbf{v}_{w_I}) + \sum_{k=1}^K \mathbb{E}_{w_{n_k} \sim P_n(w)} \left[ \log \sigma(-\mathbf{v}'^\top_{w_{n_k}} \mathbf{v}_{w_I}) \right]$

Soon after, Pennington et al. (2014) introduced GloVe (Global Vectors), showing that Word2Vec’s skip-gram objective implicitly performed weighted matrix factorization on global log co-occurrence counts:

$J = \sum_{i,j=1}^{|\mathcal{V}|} f(X_{i,j}) \left( \mathbf{w}_i^\top \mathbf{\tilde{w}}_j + b_i + \tilde{b}_j - \log X_{i,j} \right)^2$

The Limitation of Static Embeddings

While Word2Vec and GloVe produced famous geometric relationships like:

$\mathbf{v}_{\text{King}} - \mathbf{v}_{\text{Man}} + \mathbf{v}_{\text{Woman}} \approx \mathbf{v}_{\text{Queen}}$

each word had a single static vector. Polysemous words like “bank” shared the identical vector regardless of whether context implied a river bank or a commercial loan institution.

---

4. Sequential Modeling: RNNs, LSTMs, and Seq2Seq (2014–2016)

To escape fixed-size context windows, NLP embraced Recurrent Neural Networks (RNNs). In an RNN, context is maintained through a recurrent hidden state $\mathbf{h}_t \in \mathbb{R}^d$ updated at each step:

$\mathbf{h}_t = \tanh(W_{hh} \mathbf{h}_{t-1} + W_{xh} \mathbf{x}_t + \mathbf{b}_h)$

$\hat{\mathbf{y}}_t = \text{softmax}(W_{hy} \mathbf{h}_t + \mathbf{b}_y)$

In principle, $\mathbf{h}_t$ can compress an arbitrary sequence history of any length $t$.

     w_1            w_2            w_3                   w_t
      │              │              │                     │
      ▼              ▼              ▼                     ▼
┌───────────┐  ┌───────────┐  ┌───────────┐         ┌───────────┐
│ RNN Cell  │─►│ RNN Cell  │─►│ RNN Cell  │─► ... ─►│ RNN Cell  │
└─────┬─────┘  └─────┬─────┘  └─────┬─────┘         └─────┬─────┘
      │              │              │                     │
      ▼              ▼              ▼                     ▼
     h_1            h_2            h_3                   h_t

The Gradient Pathology: Vanishing & Exploding Gradients

In backpropagation through time (BPTT), the gradient of loss $\mathcal{L}_T$ with respect to state $\mathbf{h}_1$ involves a product of Jacobians:

$\frac{\partial \mathcal{L}_T}{\partial \mathbf{h}_1} = \frac{\partial \mathcal{L}_T}{\partial \mathbf{h}_T} \prod_{j=2}^T \frac{\partial \mathbf{h}_j}{\partial \mathbf{h}_{j-1}} = \frac{\partial \mathcal{L}_T}{\partial \mathbf{h}_T} \prod_{j=2}^T \text{diag}(1 - \mathbf{h}_j^2) W_{hh}^\top$

If the spectral radius (largest eigenvalue) of $W_{hh}$ is less than 1, gradients decay exponentially toward zero over long distances; if greater than 1, they explode to infinity.

Gated Recurrence: LSTM and GRU

Hochreiter & Schmidhuber’s LSTM (1997), along with Cho et al.’s GRU (2014), addressed this with additive cell memory $\mathbf{c}_t$ regulated by gates:

$\mathbf{f}_t = \sigma(W_f [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_f) \quad \text{(Forget Gate)}$

$\mathbf{i}_t = \sigma(W_i [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_i) \quad \text{(Input Gate)}$

$\tilde{\mathbf{c}}_t = \tanh(W_c [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_c)$

$\mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t \quad \text{(Constant Error Carousel)}$

$\mathbf{o}_t = \sigma(W_o [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_o) \quad \text{(Output Gate)}$

$\mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t)$

Because $\frac{\partial \mathbf{c}_t}{\partial \mathbf{c}_{t-1}} = \mathbf{f}_t$, error signals can propagate across hundreds of steps without exponential decay as long as the forget gate remains open ($\mathbf{f}_t \approx 1$).

The Seq2Seq Bottleneck (Sutskever et al., 2014)

The Sequence-to-Sequence (Seq2Seq) framework paired an Encoder LSTM with a Decoder LSTM for translation and conditional generation.

However, compressing an arbitrary-length source sentence into a single terminal vector $\mathbf{h}_T$ created an informational bottleneck: as sentence length exceeded 20–30 words, task performance plummeted.

---

5. The Attention Revolution and the Transformer (2015–2017)

Additive Attention (Bahdanau et al., 2014)

Bahdanau et al. eliminated the fixed-vector bottleneck by allowing the decoder to query all encoder states $\mathbf{h}_i$ dynamically:

$\alpha_{t, i} = \frac{\exp(e_{t, i})}{\sum_{k=1}^{T_x} \exp(e_{t, k})}, \quad e_{t, i} = \mathbf{v}_a^\top \tanh(W_a \mathbf{s}_{t-1} + U_a \mathbf{h}_i)$

$\mathbf{c}_t = \sum_{i=1}^{T_x} \alpha_{t, i} \mathbf{h}_i$

The context vector $\mathbf{c}_t$ is computed on the fly as a weighted sum over the input representations.

The Transformer: “Attention Is All You Need” (Vaswani et al., 2017)

While attention was originally added onto RNN backbones, Vaswani et al. posed a bold question: What if we eliminate recurrence entirely and rely purely on attention?

RNNs process tokens sequentially ($O(T)$ sequential steps), creating a severe serialization barrier on parallel computing architectures like GPUs. Self-attention reduces path length between any two tokens to $O(1)$ and executes in $O(1)$ sequential operations.

Scaled Dot-Product Attention:

  Query (Q)    Key (K)
      │           │
      └───► MatMul ◄───┘
              │
              ▼
         Scale (1/√d_k)
              │
              ▼
         Mask (opt.)
              │
              ▼
           Softmax
              │
              ▼
    Attention Weights (A)    Value (V)
              │                 │
              └──────► MatMul ◄─┘
                          │
                          ▼
                       Output

The core building block is Scaled Dot-Product Attention:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^\top}{\sqrt{d_k}}\right) V$

Where queries $Q = X W_Q$, keys $K = X W_K$, and values $V = X W_V$.

To capture distinct relationship subspaces simultaneously, Multi-Head Attention (MHA) projects inputs into $h$ heads:

$\text{MHA}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h) W^O$

$\text{head}_i = \text{Attention}(Q W_i^Q, K W_i^K, V W_i^V)$

Combined with residual connections, layer normalization, feed-forward networks, and positional encodings, the Transformer became the undisputed foundation for modern language modeling.

---

6. Pretraining and Transfer Learning (2018–2020)

Before 2018, NLP models were typically trained from scratch on specific target tasks. The paradigm shifted toward self-supervised pretraining on raw web-scale text, followed by task-specific fine-tuning.

               Unsupervised Pre-training (Web Corpora)
                                 │
                 ┌───────────────┴───────────────┐
                 ▼                               ▼
       Masked / Bidirectional          Autoregressive / Causal
            (Encoder-only)                  (Decoder-only)
                 │                               │
              BERT, RoBERTa                  GPT-1, GPT-2, GPT-3
                 │                               │
                 ▼                               ▼
        Discriminative Tasks             Generative Completion &
   (GLUE, Classification, NLI)           Few-Shot In-Context Prompting

The Three Structural Branches

Paradigm	Architecture	Training Objective	Representative Models	Primary Strengths
Encoder-only	Bidirectional Transformer	Masked Language Modeling (MLM): predict $[MASK]$ tokens given bidirectional context	BERT, RoBERTa, DeBERTa	Classification, NER, semantic search embeddings
Decoder-only	Causal Masked Transformer	Next-Token Prediction: $P(w_t \mid w_{<t})$	GPT-1/2/3, LLaMA, Mistral	Open-ended text generation, unified in-context reasoning
Encoder-Decoder	Full Seq2Seq Transformer	Denoising span corruption / seq2seq reconstruction	T5, BART	Summarization, translation, controlled transformation

Why Did Decoder-Only Win the Scaling Race?

While encoder-only architectures dominated academic benchmarks initially (GLUE), decoder-only causal models ultimately dominated large-scale production for several architectural reasons:

1. Computational Efficiency in Generation: Autoregressive decoding requires updating state via Key-Value (KV) caching.
2. Task Unification: Any language task (classification, translation, reasoning, coding) can be framed as continuing a string of text.
3. No Mismatch between Pretraining and Inference: Masked models suffer from the artificial discrepancy that $[MASK]$ tokens never appear during inference.

---

7. Scaling Laws and In-Context Learning (2020–2022)

In 2020, Kaplan et al. (OpenAI) published empirical observations establishing that language model cross-entropy loss follows smooth power laws with respect to compute budget ($C$), dataset size ($D$), and parameter count ($N$):

$L(N) \approx \left(\frac{N_c}{N}\right)^{\alpha_N}, \quad L(D) \approx \left(\frac{D_c}{D}\right)^{\alpha_D}, \quad L(C) \approx \left(\frac{C_c}{C}\right)^{\alpha_C}$

Log Loss (L)
   │
   │ \
   │   \
   │     \     L(C) ∝ C^(-α)
   │       \   Predictable power-law scaling across orders of magnitude
   │         \
   └────────────────────── Log Compute (C)

The Chinchilla Correction (Hoffmann et al., 2022)

DeepMind’s Chinchilla paper re-evaluated optimal resource allocation under a compute budget $C \approx 6 N D$. They found that earlier models (like GPT-3 with 175B parameters trained on 300B tokens) were severely undertrained.

Optimal performance requires scaling parameters $N$ and tokens $D$ in equal proportion:

$N \propto C^{0.5}, \quad D \propto C^{0.5}$

This gave birth to “Chinchilla-optimal” open-weights foundations like Meta’s LLaMA series (smaller, denser models trained on trillions of tokens).

In-Context Learning (ICL) and Few-Shot Prompting

With GPT-3 (Brown et al., 2020), a qualitative shift occurred: models of sufficient scale no longer required weight updates or gradient-based fine-tuning to perform novel tasks. By simply conditioning on natural language demonstrations in the prompt:

$\mathcal{P} = \{\text{"Context"} \to \text{"Answer"}, \dots, \text{"New Context"} \to ?\}$

the model performs meta-optimization across its attention layers, inferring the underlying task mapping implicitly.

---

8. Alignment, Post-Training, and Reasoning (2022–Present)

Raw base models trained purely on next-token prediction are fundamentally unaligned: they mimic the distribution of internet text, including factual hallucinations, toxicity, and ramblings.

The modern LLM recipe cleanly splits training into Pre-training and Post-training:

[Raw Web Corpora]
       │
       ▼  Self-Supervised Pre-training (Next-Token Prediction)
[Base Foundation Model]
       │
       ▼  Supervised Fine-Tuning (Instruction & Dialogue Pairs)
[SFT Model]
       │
       ▼  Preference Optimization (RLHF / DPO / PPO)
[Aligned Assistant]
       │
       ▼  Inference-Time Search & Reasoning (CoT, Verifiers, RL)
[Reasoning Model (e.g. o1 / R1)]

1. Supervised Fine-Tuning (SFT)

Base models are fine-tuned on curated multi-turn conversational datasets (e.g., instructions and high-quality responses), transforming a predictive next-word simulator into an instruction-following assistant.

2. Preference Alignment: RLHF and DPO

To optimize for nuanced human values (helpfulness, honesty, harmlessness) that cannot be captured by cross-entropy loss alone:

• RLHF (PPO): Trains a Reward Model $R_\psi(x, y)$ on human pairwise preferences ($y_w \succ y_l$), and updates the policy $\pi_\theta$ using Proximal Policy Optimization with a KL-divergence constraint to prevent policy collapse:

$\max_{\theta} \mathbb{E}_{(x, y) \sim \mathcal{D}_{\pi_\theta}} \left[ R_\psi(x, y) \right] - \beta \mathbb{D}_{\text{KL}}\left(\pi_\theta(y \mid x) \parallel \pi_{\text{ref}}(y \mid x)\right)$

• Direct Preference Optimization (DPO) (Rafailov et al., 2023): Algebraically solves for the optimal policy directly from the preference loss, eliminating the instability of training a separate reward model and running reinforcement learning loops:

$\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l)} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]$

3. Inference-Time Compute Scaling & Reasoning

The latest frontier shifts from scaling pre-training compute to scaling inference-time compute:

• Chain of Thought (CoT): Allowing the model to generate internal scratchpad tokens ($z_1, z_2, \dots, z_k$) before delivering the final answer $y$, factorizing difficult reasoning steps.
• Process Reward Models (PRMs) & Monte Carlo Tree Search (MCTS): Verifying intermediate reasoning steps rather than scoring purely end-outcome correctness.
• Reinforcement Learning on Verifiable Domains: Reinforcing exploration strategies across math, logic, and code generation where reward verification is deterministic.

---

9. Architectural Evolution: Summary Timeline

Era	Core Paradigm	Landmark Models	Primary Formulation	Primary Limiting Factor
~2000	Discrete N-grams	Kneser-Ney, SRILM	$P(w_t \mid w_{t-n+1:t-1}) = \frac{C(\cdot)}{C(\cdot)}$	Exponential sparsity, no semantic generalization
2003	Continuous Embeddings	Bengio NPLM	$y = b + Wx + U\tanh(d + Hx)$	Fixed context window, $O(\|\mathcal{V}\|)$ softmax
2013	Dense Static Vectors	Word2Vec, GloVe	$\log \sigma(\mathbf{v}'^\top_w \mathbf{v}_c)$	Static representations; no contextual polysemy
2014	Recurrent Sequence	LSTM, GRU, Seq2Seq	$\mathbf{h}_t = f(\mathbf{h}_{t-1}, \mathbf{x}_t)$	Sequential dependency $O(T)$, vanishing gradients
2017	Self-Attention	Transformer	$\text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$	Quadratic attention cost $O(T^2)$ in context length
2018	Self-Supervised Pretraining	BERT, GPT-2, T5	$\mathcal{L}_{\text{MLM}}$ / $\mathcal{L}_{\text{CLM}}$	Task transfer mismatch, compute-hungry scaling
2020	Scaling Laws & ICL	GPT-3, Chinchilla, LLaMA	$L \propto C^{-\alpha}$, Prompting	High pretraining capital cost, unaligned outputs
2023+	Post-Training & Reasoning	RLHF, DPO, o1, R1	$P(\text{Final} \mid \text{Thought}) \cdot P(\text{Thought})$	Verification bottlenecks, hallucination in open domains

---

References & Foundational Reading

1. Bengio, Y., et al. (2003). A Neural Probabilistic Language Model. JMLR.
2. Mikolov, T., et al. (2013). Distributed Representations of Words and Phrases and their Compositionality. NeurIPS.
3. Sutskever, I., et al. (2014). Sequence to Sequence Learning with Neural Networks. NeurIPS.
4. Bahdanau, D., et al. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR.
5. Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS.
6. Devlin, J., et al. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL.
7. Radford, A., et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2). OpenAI Technical Report.
8. Brown, T., et al. (2020). Language Models are Few-Shot Learners (GPT-3). NeurIPS.
9. Kaplan, J., et al. (2020). Scaling Laws for Neural Language Models. arXiv:2001.08361.
10. Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models (Chinchilla). NeurIPS.
11. Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback (InstructGPT). NeurIPS.
12. Rafailov, R., et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS.