7
$\begingroup$

One of the problems that occur during (artificial) neural network training is called overfitting. The error on the training set is driven to a very small value, but when new data is presented to the network the error is large. The network has memorized the training examples, but it has not learned to generalize to new situations.

The human brain does not overfit when training at some task, but becomes better and better instead. Why? Or conversely, why do artificial neural networks overfit when are trained too much, in contrast with the human brain?

Improve this question
$\endgroup$
9
  • 2
    $\begingroup$ What do you mean with "over fit"? $\endgroup$
    – AliceD ♦
    Commented Jun 4, 2015 at 23:56
  • 4
    $\begingroup$ @AliceD Overfitting is when an artificial neural network captures too many extraneous features of the data during training so that generalization becomes nearly impossible. It's an interesting question, but it's unclear, I agree with both of you. $\endgroup$
    – jonsca
    Commented Jun 5, 2015 at 0:30
  • 6
    $\begingroup$ (parenthetically, I'm not convinced that we don't "overfit" at times, especially in the developing brain, but analogies between ANNs and BNNs are often stretched to the point where such shared features become muddled) $\endgroup$
    – jonsca
    Commented Jun 5, 2015 at 0:32
  • $\begingroup$ @jonsca - aha, interesting stuff, I never heard of this! $\endgroup$
    – AliceD ♦
    Commented Jun 5, 2015 at 0:43
  • $\begingroup$ @jonsca - hopefully OP clarifies the question, I am getting curious :) $\endgroup$
    – AliceD ♦
    Commented Jun 5, 2015 at 11:30

4 Answers 4

Reset to default
14
$\begingroup$

I would say the human brain overfits all the time! Gambling addiction, superstition and anxiety disorders are all examples of overfitting. We are optimized for seeking patterns and avoiding threats. Our brains mess this up all the time! But having said that, one of the major differences between the human brain and a neural network is the amount of information that's being supplied. Neural networks overfit in part because of limited data sets. Our brains have nearly unlimited data sets available, and this provides us at least the opportunity for some self correction.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Do you think that since human brain works on a reinforcement learning basis, this is another factor that outperforms supervised learning in ANN regarding issue of overfitting? $\endgroup$
    – Fraïssé
    Commented Jun 7, 2015 at 6:25
  • $\begingroup$ @Shelly DeForte Well, forget for a moment such complex behaviors. At simple classifications tasks, human do not over fit. Or I am wrong? $\endgroup$
    – emanuele
    Commented Jun 7, 2015 at 14:21
  • $\begingroup$ I'm having trouble thinking of a good side by side comparison. The problem is that we're actually pretty terrible at most things you might use an ANN for because we can't really hold more than a couple of dimensions in our minds at a time. Can you give me an example of a simple classification task that a human outperforms on because the ANN is overfitting and the human is not? $\endgroup$
    – Shelly DeForte
    Commented Jun 7, 2015 at 16:42
  • $\begingroup$ characters recognitions... $\endgroup$
    – emanuele
    Commented Jun 8, 2015 at 12:23
  • 1
    $\begingroup$ I don't think we're better at character recognition because we don't overfit. I think we're better at it because our architecture is immensely better for shape recognition and differentiation. But I admit I don't know how ANNs do character recognition, so I'm not sure what the differences are. $\endgroup$
    – Shelly DeForte
    Commented Jun 8, 2015 at 15:35
3
$\begingroup$

In this fascinating talk by Geoff Hinton, there is the strong implication that the human brain doesn't overfit, because it implements a version of dropout, i.e. of randomly leaving out signals to prevent coadaptation between neurons.

Dropout has been found to be extremely effective in combatting overfitting.

Basically the idea is that the human brain has a stochastic activation of neurons, which is pretty similar to the dropout technique (except that dropout is usually done with the same probability for each neuron).

Improve this answer
$\endgroup$
2
$\begingroup$

Like any statistical learner, the human brain surely experiences overfitting. You can convince people of lots of things that aren't true by showing them biased examples.

That said, don't forget that the tasks by which we judge artificial intelligence are largely defined by what human brains happen to be good at. That's part of what makes us call it "intelligence" instead of just high-performance computing.

In other words, in your character recognition example, don't forget that the character system was developed by humans. Presumably we converged on systems of characters that our neural architectures are good at recognizing and discriminating. And in general, the computer vision tasks that we consider "intelligent" often match tasks that our brains and visual systems have evolved to do well.

So, in a sense, the concepts and tasks that serve as our tests are also inherently "overfit" to our brains.

Improve this answer
$\endgroup$
-2
$\begingroup$

I'm confident we now have a good hypothesis on this topic: dreams are constantly working to reduce the overfitting and improving the generalization. https://www.sciencedirect.com/science/article/pii/S2666389921000945

If this is true, dreaming should be present in much more animals than we currently know.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .