I had long known that there is a remarkable figure in the field of deep learning named Andrej Karpathy. Last week, I watched his two-hour video on building micrograd and was utterly amazed. What I gained was not so much knowledge, but rather an inspiration about how to learn.
Learning is a problem of planning, not of effort. In my view, the micrograd video primarily demonstrates two points:
- Start from first principles and focus only on the first-order terms of a complex system.
- When an abstract system is difficult to understand, try concentrating on local reasoning with the smallest possible example.
For the first point, through just 100 lines of code for autograd, one can understand the backpropagation algorithm for training neural networks, computational graphs, recursive topological sorting, and gradient descent. In essence, micrograd is like a library such as PyTorch, but stripped of its efficiency optimization modules.
For the second point, the traditional way to learn backpropagation is to write out a bunch of mathematical derivatives of the loss function with respect to the weights, and then expand a series of matrix multiplications via the chain rule. However, in micrograd, Andrej teaches backpropagation by manually computing local gradients. I find this approach—using hand computation for a specific link in a complex system—to be very meaningful.
Through this, I can also glimpse the limitations of using an LLM as a learning mentor. Before I encountered Andrej, it seems that the LLM never pointed me down this path.
Can an ordinary person obtain a mentor like Andrej through an LLM? From the perspective of information theory, the answer is no—or at least, it is quite difficult. Formal proofs are already everywhere, so I won’t try to offer my own modest contribution here.
A questioner cannot describe intuitions they have never experienced.
The language model minimizes cross-entropy loss; for a given prompt, every token it outputs is chosen from the probability distribution along the path with the highest conditional probability. Therefore, what the language model selects for you is often the shortest path in the space, and very likely not the ideal path of your intention.
It is like searching for a small island that holds a gold mine in a vast ocean—but how do you direct the AI to reach that island? If most coordinates point to garbage dumps, how do you gradually find the best constraints? If the coordinates you give are too far off, if self-diagnostic ability is lacking, and if expression is vague, then it becomes even more likely to veer off course.
The LLM we face is a monstrous existence; its interior is a chaotic world of high information entropy, difficult to comprehend. It is as if the LLM can find connections between Shakespeare’s poetry and Shannon’s theory, but because the information lies in high dimensions, it cannot express them in natural language. How do we control such chaos?
I find the description of LLMs by the Zhihu user “Jiang Zijun” particularly interesting. The original text, analyzing through information theory whether someone who does not understand computer science can write expert-level code, goes:
“The LLM has witnessed countless piles of garbage code, harboring all the evil of this world, and has become a twisted Holy Grail that distorts wishes. A muggle who does not understand computer science is like an ordinary person without magic circuits, recklessly attempting to touch the Root and utter their own delusions. The response given by the Holy Grail can only be spiritual contamination containing infinite information entropy, amplifying chaotic wishes a millionfold, and ultimately yielding a writhing mass of flesh—seemingly runnable, yet utterly meaningless.”
But for self-learners, although they may never obtain a mentor like Andrej through an LLM, if you do not have a 90-point mentor, then a 60-point mentor is still twice as good as a 30-point one.