7
$\begingroup$

One of the problems that occur during (artificial) neural network training is called overfitting. The error on the training set is driven to a very small value, but when new data is presented to the network the error is large. The network has memorized the training examples, but it has not learned to generalize to new situations.

The human brain does not overfit when training at some task, but becomes better and better instead. Why? Or conversely, why do artificial neural networks overfit when are trained too much, in contrast with the human brain?

$\endgroup$
  • 1
    $\begingroup$ What do you mean with "over fit"? $\endgroup$ – AliceD Jun 4 '15 at 23:56
  • 4
    $\begingroup$ @AliceD Overfitting is when an artificial neural network captures too many extraneous features of the data during training so that generalization becomes nearly impossible. It's an interesting question, but it's unclear, I agree with both of you. $\endgroup$ – jonsca Jun 5 '15 at 0:30
  • 5
    $\begingroup$ (parenthetically, I'm not convinced that we don't "overfit" at times, especially in the developing brain, but analogies between ANNs and BNNs are often stretched to the point where such shared features become muddled) $\endgroup$ – jonsca Jun 5 '15 at 0:32
  • $\begingroup$ @jonsca - aha, interesting stuff, I never heard of this! $\endgroup$ – AliceD Jun 5 '15 at 0:43
  • $\begingroup$ @jonsca - hopefully OP clarifies the question, I am getting curious :) $\endgroup$ – AliceD Jun 5 '15 at 11:30
12
$\begingroup$

I would say the human brain overfits all the time! Gambling addiction, superstition and anxiety disorders are all examples of overfitting. We are optimized for seeking patterns and avoiding threats. Our brains mess this up all the time! But having said that, one of the major differences between the human brain and a neural network is the amount of information that's being supplied. Neural networks overfit in part because of limited data sets. Our brains have nearly unlimited data sets available, and this provides us at least the opportunity for some self correction.

$\endgroup$
  • $\begingroup$ Do you think that since human brain works on a reinforcement learning basis, this is another factor that outperforms supervised learning in ANN regarding issue of overfitting? $\endgroup$ – Carlos - the Mongoose - Danger Jun 7 '15 at 6:25
  • $\begingroup$ @Shelly DeForte Well, forget for a moment such complex behaviors. At simple classifications tasks, human do not over fit. Or I am wrong? $\endgroup$ – emanuele Jun 7 '15 at 14:21
  • $\begingroup$ I'm having trouble thinking of a good side by side comparison. The problem is that we're actually pretty terrible at most things you might use an ANN for because we can't really hold more than a couple of dimensions in our minds at a time. Can you give me an example of a simple classification task that a human outperforms on because the ANN is overfitting and the human is not? $\endgroup$ – Shelly DeForte Jun 7 '15 at 16:42
  • $\begingroup$ characters recognitions... $\endgroup$ – emanuele Jun 8 '15 at 12:23
  • 1
    $\begingroup$ I don't think we're better at character recognition because we don't overfit. I think we're better at it because our architecture is immensely better for shape recognition and differentiation. But I admit I don't know how ANNs do character recognition, so I'm not sure what the differences are. $\endgroup$ – Shelly DeForte Jun 8 '15 at 15:35
3
$\begingroup$

In this fascinating talk by Geoff Hinton, there is the strong implication that the human brain doesn't overfit, because it implements a version of dropout, i.e. of randomly leaving out signals to prevent coadaptation between neurons.

Dropout has been found to be extremely effective in combatting overfitting.

Basically the idea is that the human brain has a stochastic activation of neurons, which is pretty similar to the dropout technique (except that dropout is usually done with the same probability for each neuron).

$\endgroup$
2
$\begingroup$

Like any statistical learner, the human brain surely experiences overfitting. You can convince people of lots of things that aren't true by showing them biased examples.

That said, don't forget that the tasks by which we judge artificial intelligence are largely defined by what human brains happen to be good at. That's part of what makes us call it "intelligence" instead of just high-performance computing.

In other words, in your character recognition example, don't forget that the character system was developed by humans. Presumably we converged on systems of characters that our neural architectures are good at recognizing and discriminating. And in general, the computer vision tasks that we consider "intelligent" often match tasks that our brains and visual systems have evolved to do well.

So, in a sense, the concepts and tasks that serve as our tests are also inherently "overfit" to our brains.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.