Machine Learning

llya Sutskever's Top 30 Reading List for Deep Learning

2 min read
348 words
# AI # deep-learning

Around late 2019 to 2020, John Carmack (A legendary programmer, Creator of DOOM) decided to pivot into AGI.

He reached out to llya Sutskever and asked for a reading list to build the foundmental competence of this area.

llya really sent the reading list, and left the famous words:

“If you really learn all of these, you’ll know 90% of what matters today.”

And the reading list became so popular and kinda mysterious in the community.

Phase 1: Foundations

  1. CS231n Course Notes (Karpathy et al.)
  2. AlexNet (2012) (Krizhevsky, Sutskever, Hinton)
  3. Adam Optimizer (2014) (Kingma & Ba)
  4. ResNet (2015) & Identity Mappings (2016) (He et al.)
  5. Dilated Convolutions (2015) (Yu & Koltun)

Phase 2: Attention

  1. The Unreasonable Effectiveness of RNNs (Karpathy, 2015)
  2. Understanding LSTM Networks (Olah, 2015)
  3. RNN Regularization (Zaremba, Sutskever, Vinyals, 2014)
  4. Sequence to Sequence Learning (2014) (Sutskever, Vinyals, Le)
  5. Neural Machine Translation by Jointly Learning to Align and Translate (2014) (Bahdanau et al.)
  6. Order Matters: Sequence to Sequence for Sets (2015) & Pointer Networks (2015) (Vinyals et al.)

Phase 3: The Transformer

  1. Attention Is All You Need (2017) (Vaswani et al.)
  2. The Annotated Transformer (Sasha Rush, 2018)

Phase 4: Generalized Computation

  1. Neural Turing Machines (2014) (Graves, Wayne, Danihelka)
  2. Relation Networks (2017) & Relational RNNs (2018) (Santoro et al.)
  3. Neural Message Passing for Quantum Chemistry / MPNN (2017) (Gilmer et al.)
  4. Variational Lossy Autoencoder (2016) (Chen, Kingma, Salimans, Sutskever et al.)

Phase 5: Scaling

  1. Deep Speech 2 (2015) (Amodei et al.)
  2. GPipe (2018) (Huang et al.)
  3. Scaling Laws for Neural Language Models (2020) (Kaplan et al.)
  4. Language Models are Few-Shot Learners / GPT-3 (2020) (Brown et al.)

Phase 6: Compression

  1. Keeping Neural Networks Simple by Minimizing Description Length (1993) (Hinton & van Camp)
  2. A Tutorial on Minimum Description Length (2004) (Peter Grünwald)
  3. The First Law of Complexodynamics (2011) & The Coffee Automaton (2014) (Scott Aaronson et al.)
  4. Kolmogorov Complexity and Algorithmic Randomness (Shen et al.)
  5. Machine Super Intelligence (2008 PhD Dissertation) (Shane Legg)

Refs

https://aman.ai/primers/ai/top-30-papers/