A simple intro to MDL and Generalization of neural network

this article is the reading note of the paper：Keeping Neural Networks Simple by Minimizing the Description Length of the Weights(hinton 1994)

I’m not really a good researcher at this point, so this is for beginner of this area!

the training data contains information about the true regularities in the mapping from input to output. but it also contains sampling error.

there will be accidental regularities just because of the particular training cases that were chosen.

this is the abstract of the paper:

supervised neural network generalize well if there’s much less information in the weights than there is in the output vectors of the training cases

in communication theory, we can set up this as sender-recevier scenerio

if we have the training set: input X = [x1…xN] and the labels: Y =[y1…yN]

Sender needs to transmit the labels to a receiver who only knows about X, using as few bits as possible.

Direct Transmission, sender

神经网络防止过拟合的问题与信息论进行联系。

最小描述长度原则（Minimum Description Length, MDL）

由 Jorma Rissanen 于 1986 年提出