It looks like a simple problem, affecting almost half of the population at the post-pubertal age and of any gender and ethnicity [wikipedia]. What is so complex about finding a cure? Also, if you know some of the most effective treatments, they're welcome.
For dandruff caused mainly from dry scalp, I imagine that there is a delicate balance between moisturizing the scalp and getting oily hair. In my uninformed opinion, this is what shampoos like Head and Shoulders attempt to do.
Taking a more general look at skin health, growing research suggests that the right compilation of microbiota dictates lots of skin health issues. Just like your bowel has an endless supply of mutualistic bacteria, so does your skin. Disruption to this ecosystem (whether genetic, by way of a pathogen, or environmental) could have various negative effects. Dandruff, I imagine, is one of them. It is unfortunately very difficult to understand what a healthy ecosystem looks like, as every person is different both in genetic and environmental factors. This is part of the reason why other dermatological conditions like acne are difficult to treat.
You may be interested in research done through the Human Microbiome Project. Additionally, this article includes some discussion of the fungi mentioned above in the comments and postulates the role of microbiota in determining their success.