Sexual dimorphism evolves, as you suspect, via natural selection, specifically sexual selection (which I view as a form of natural selection). Genetic drift, by definition, is random. Random changes of allele frequencies are extremely unlikely to lead to the consistent evolution of phenotypic differences between males and females.
Sexual selection on the other hand can lead to rapid evolution of secondary sexual characteristics (sexual dimorphism). If you consider natural selection in the broad sense, an organism's traits affect survivorship but do not guarantee reproduction. Individuals that live longer have an increased chance of reproducing more frequently but this does not mean the organism actually will reproduce more often than any other individual. In contrast, sexual selection acts directly on mating success. If a female chooses a male based on his secondary sexual characters, then that male will mate. His fitness (offspring produced) will be increased relative to males that are not chosen and do not mate. This can lead to runaway sexual selection for male traits and even speciation.
You are correct that sexual selection results from sexual conflict. Such conflict can arise because females invest considerable energy into relatively few eggs. Thus, it benefits her to be selective for her mates. If she chooses poorly, her fitness can be decreased quite a bit. On the other hand, males can produce copious quantities of sperm rapidly and with little energy cost. If a male wastes sperm on a genetically-poor female, his fitness is not really reduced. He can quickly mate with other females at little cost.
References
Any evolutionary text. They cover the basics of sexual selection in some detail.
Gavrilets, S. 2000. Rapid evolution of reproductive barriers driven by sexual conflict. Nature 403: 886-889.
Hosken, D.J. and P. Stockley. 2005. Sexual conflict. Current Biology 15: R535-R536.
Lande, R. 1981. Models of speciation by sexual selection on polygenic traits. Proceeedings of the National Academy of Sciences USA 78: 3721-3725.