If natural selection favors a certain behavior that is learnt, then wouldn’t that behavior be found more often among a population (as more offspring will learn that behavior from their parents or their peers)? Since the behavior is not encoded in the genes, it is not heritable, but it is transmitted through other means and will thus accumulate in the population. This, to me, seems a lot like the effects of evolution, yet this is not a heritable trait. So my question is why does evolution concern only heritable traits and not those that can be reproduced by learning?
There is a lot to say to address this question fully. I will be brief and a little messy to try to introduce you to some concepts and that will hopefully push you to learn more by yourself.
Heritability in genetics
In genetics, heritability is a statistic that describe the fraction of the phenotypic variation that which is caused by underlying genetic variation. Please have a look at this post to understand the concept of heritability as used in genetics. Note that there are two such statistics, named heritability in the broad sense $h_B^2$ and heritability in the narrow sense $h_N^2$.
Basics of non-genetic inheritance
Here, I will refer to "heritability in the broad" as the passation of trait value (such as behaviour for example) and environment characteristic to another organism. One must not confound what I call here "heritability in the broad" and heritability in the broad sense as defined in genetics ($h_B^2$).
Sure, a lot of things can be inherited from one organism to another. Any such heritable (in the broad sense) trait is susceptible to show interesting variation in the frequency at which trait values are found in a population. If, in addition to that, this trait is susceptible to spontaneous changes (the equivalent of mutations in genetics), then this trait can show interesting evolutionary patterns. If in addition to that, different trait values have different aptitude to replicate (the equivalent of natural selection in genetics), then such trait can evolve in an "even more interesting way".
What can be non-genetically inherited?
Two things can be inherited without the use of any sort of genetic information (I also exclude epigenetic information incl. DNA methylation or protein concentration in the yolk).
- Environment (Ecological inheritance)
See more on Ecological inheritance and memes below.
This is probably the first type of inheritance one would think of when not thinking about biology. In humans, it is not uncommon for offspring to inherit "stuff" from their parents when they die. Similarly, beavers can inherit a dam that has been built by their parents. This type of inheritance in generally called Ecological inheritance.
This type of inheritance might eventually be important to evolutionary processes (see literature on niche construction).
Inheritance of a meme
A meme has been named by the analogy to a gene. A meme is any idea, behaviour, thought or tradition (piece of culture) that can be inherited. The concept of an internet meme is now quite popular, but a meme is more than just an internet meme. While genetics is the study of genes (and much more actually), memetics is the study of memes. Memetics is a very interdisciplinary field of study that attract biologists, philosophers, mathematicians, linguists and archeologists among others. The hypotheses built in the field of memetics rarely offer predictions that can be easily tested, and therefore a good part of the work in this field remains very uncertain.
Analogy to genes
A meme could be a joke for example. If the joke is funny it will replicate easily and enter in new brains. This meme could be said to have high fitness. The joke may come into spontaneous changes (mutations). Be careful with analogies and they may show nothing but similarity. For example the nature of memetic mutation seem to be quite different than from the nature of genetic mutation.
In humans, the examples are very easy to find. As I am writing this answer, I am currently trying to pass on a meme which has been given to me by some other authors. Hopefully I won't mutate this meme too harshly.
Lots of monkeys and apes are transmitting traditions, foraging methods and other memes to their offspring, sometimes through active teaching but often through imitation (or emulation). A classic example is the one reported in Rhythms of Vision in 1975 (I could not find the original peer-reviewed paper). A female monkey, one day had the good idea of washing a sweet potato. This behaviour has since increased in frequency throughout the population so that most monkeys now wash their sweet potatoes.
Pitman and Durban (2011) (and others before them) report that some population of orcas use cooperative hunting while some other population does not. Variation for this trait seems not to be caused by genetic variation.
How does inheritance work?
Inheritance can be "vertical" (parent to offspring) or "horizontal" (between individuals of the same or different species). This holds true for both genetic (consider transduction) and non-genetic inheritance (teaching to offspring or to the neighbors).
In addition to that the passation of a meme can be active (teaching) or passive.