GPT2

It’s interesting I found out coding GPT2 is much easier than coding the original transformer architecture which designed for machine translation in the beginning. Because we don’t have cross attention, and the encoder decoder architecture

It’s can all go back to the landmark 2017 paper, Attention is all you need.

This paper was revolutionary because it replaced recurrent networks and CNNs entirely with attention mechanisms for transduction sequence modeling

Each block in GPT-2 has:

x -> LayerNorm -> Self-Attention -> Residual Add -> LayerNorm -> MLP -> Residual Add