This stuff is so interesting! The instinctive behaviour is also known as a fixed action pattern or FAP which is a collection of well-defined motor patterns that are activated in response to a stimulus, i.e. a group more eleborate reflexes [2]. FAPs allow the brain to concentrate on other things as they don't need require time or energy to be thought about. The theory is they've been selected for because they decrease choice and decision time, which can be particularly important to survival.
There are some cool examples here (http://www.cerebromente.org.br/n09/fastfacts/comportold_i.htm).
the bird is continues to perform parental instinctive behaviour in the absence of a brood of chicks because the similar stimulus of hungry open-mouthed fish is present.
Another article draws comparisons between different animals as to how much of their behaviour is fixed/pre-determined [3]. There can be a correlation between the size of the cerebral cortex, used for thinking and problem solving, and the amount of innate behaviour (i.e. the bigger your cortex the less you need to rely on pre-determined behaviours in certain situations).
These patterns seem to be for the most part genetically determined, however some studies with song-birds in the 70's demonstrated that a learning aspect can modify the innate behviour. Birds which have been reared in an environemnt where they cannot hear other birds of the same species sing still sing their typical song, but when allowed to hear others sing the song becomes more complex.
Investigating the genetic basis behind innate behaviours involve searching for Quantitative Trait Loci (QTLs) or stretches of DNA that contain genes (complex traits require the concerted actions of multiple genes) or are linked to genes that are involved in determining a specifc trait (in this case a behaviour). Classically this involves crossing two animals that differ in the trait you're interested in and looking at the segregation patterns in the offspring to try and pinpoint the specifc region of DNA/collection of genes that control the trait. I think quite a lot of this has been done in insects and mice which have high breeding rates and can be studied easily in the lab.
...at this risk of making this answer ridiculously long I've just included a snippet from another website on behavioural genetics which details the OMIM and some of the traits you can search for[4]:
Online Mendelian Inheritance in Man (OMIM) is a large, searchable, up-to-date database of human genes, genetic traits, and disorders. Each OMIM record contains bibliographic references and a summary of the scientific literature describing what is known about a particular gene, trait, or disorder. The following behavioral traits are included in OMIM. The six-digit number MIM number is used to uniquely identify each record.
Hand skill, relative (handedness): (139900)
Hand clasping pattern: (139800)
Arm folding preference: (107850)
Ears, ability to move: (129100)
Tongue curling, folding, or rolling: (189300)
Musical perfect pitch: (159300)
[2] http://books.google.co.uk/books?id=aQvGXLPjMPAC&pg=PA44&lpg=PA44&dq=fixed+action+patterns+difference+reflex&source=bl&ots=TxaSYeIpTs&sig=RfsmHLWVlNiF8B-dH928-F0I5M8&hl=en&sa=X&ei=Bs1WUZeCFYiZtAbZ_IHYBw&ved=0CDcQ6AEwAA#v=onepage&q=fixed%20action%20patterns%20difference%20reflex&f=false
[3] http://www.grandin.com/references/genetics.html
[4] http://www.ornl.gov/sci/techresources/Human_Genome/elsi/behavior.shtml
