If I have a brown eye gene which encodes the protein that is responsible for the brown color and have a blue eye gene as well, what is the reason that my eye color is brown? How does one gene maintain dominance over the other?
One version or allele of the gene for eye color may encode instructions to make the brown pigment, while the blue allele makes no pigment. Alternatively, it is possible, logically speaking, that the blue allele blocks the productions of brown pigment, but this is not the case.
Now, brown vs blue is not a case of one gene and two alleles. After producing pigment, it must be transported, deposited and such. Many processes are involved. So, the pigment deposition process may be functional or not, but as above, it is often found that a non-functional version is simply non-functioning and not actively inhibiting the process. There are numerous cases where an aberrant form of a protein, encoded by a variant allele, acts as an active inhibitor of a given biological or biochemical process.
The key to your question as written is blue eye color is a lack of pigmentation.
At the DNA level there are several mechanisms that might be cited.
A mutation (change in the DNA sequence) of a gene may actually render one or both copies of the gene defective. In the case of albino-ism, the skin is pink - no melanin pigment is made at all. This is a case of both copies of a vital gene being defective and an ability lost completely. A "loss of function" recessive trait.
Blue eyes have no retinal pigment - they are a loss of function recessive. Some cases of genetically inherited diabetes (<1%) dwarfism and obesity are examples of this. There are many others - many genetic diseases are like this.
This can be caused by a single mutation in each copy of the gene, and others an entire segment of the chromosome can be lost or disrupted by a new, often nonsensical sequence dropped into the gene region.
Other changes can affect the way the genes turn on and off. - similar sequence changes in the regulatory sequences next to the gene sequence will make the gene behave differently. Even babies of african descent may be born with blonde hair and blue eyes and get the darker pigmentation later. this is genetically caused by a change in they timing of the genes for hair and eye color turning on at other times (blonde hair and blue eyes have little or no pigment, so if the genes are off, then this is what you get). So when genes are active can cause dominant (always on) versus recessive (sometimes off) patterns. which are because one copy of the gene behaves differently than the other copy.
Sometimes a single mutation in one copy of the gene can cause rather interesting effects. A single good gene and a mutated, less effective gene might result in lost of activity because the mixture of the two genes is different between a wholly functional or wholly variant set of genes.
A classic example is the sickle cell Anemia. two copies of sickle cell hemoglobin causes the red blood cells to be stiff and misshapen which in turn causes a painful condition in the carrier. a person with a single copy of the sickle cell gene does not have this problem - is nearly asymptomatic. So we say sickle cell anemia is recessive for the HbS mutation. but in reality there is plenty of the HbS version of hemoglobin (Hb) in the blood cells, just that the mixture of the two does not cause the anemia.
Much rarer is an additive function from a genetic sequence change. The immune system is where you will most often see this. Since the sequences that drive immunity vary widely, some immune sequences are rare - such as inborn immunity to HIV.
Sorry to blockquote, but this note from wikipedia gets to the heart of the question:
Which trait is dominant?
The terms dominant and recessive refer to the interaction of alleles in producing the phenotype of the heterozygote. If there are two alternative phenotypes, by definition the phenotype exhibited by the heterozygote is called "dominant" and the "hidden" phenotype is called "recessive". The key concept of dominance is that the heterozygote is phenotypically identical to one of the two homozygotes. That trait corresponding to the dominant allele may then be called the "dominant" trait.
Dominance is a genotypic relationship between alleles, as manifested in the phenotype. It is unrelated to the nature of the phenotype itself, e.g., whether it is regarded as normal or abnormal, standard or nonstandard, healthy or diseased, stronger or weaker, or more or less extreme. It is also important to distinguish between the "round" gene locus, the "round" allele at that locus, and the "round" phenotype it produces. It is inaccurate to say that "the round gene dominates the wrinkled gene" or that "round peas dominate wrinkled peas."
Recommended reading for an overview:
You can also look to "dominant negative" phenotypes to get a good idea of what "dominance" entails (this was the concept that really locked it in for me in college). One example is Marfan syndrome, where the mutant allele of FBN1 (fibrillin-1) produces a version of the protein that is antagonistic to the protein produced by the "healthy" allele.
Dominant negative and semi-dominant alleles can be seen when the protein forms dimers to function. Activin type receptors form dimers to transduce a signal from an extracellular ligand into the cell. A truncated receptor protein that has the dimerization domain but lacks the intracellular domain necessary for signal transduction will act in a dominant negative fashion over the wildtype allele: Dimers are formed but the signal is not transduced. This has been implicated in hereditary colorectal cancers.
A gene is dominant (lets call it B) when its presence in combination with a recessive allele (heterozygotic Bb) leads to the same phenotype as homozygotic BB. A recessive allele leads only to phenotypic expression when it is present twice (homozygotic bb).
To illustrate this rather abstract definition of dominant and recessive alleles let's look at a striking example, namely a gene mutation responsible for albinism.
One of the causes that can lead to albinism is a mutation in the gene that encodes tyrosinase (TYR). Tyrosinase is used by melanocyte cells to convert the amino acid tyrosine into the pigment melanin that colors the skin, hair, and eyes (Ballantine, 2009). Animals lacking tyrosinase have white skin and hair and red eyes, such as the kangaroo in the picture below (source: Listverse):
One functional gene (B) is enough to allow for pigmentation to occur and hence it is dominant. The other mutated allele (b) produces nonfunctional enzyme, but is recessive as one functional allele produces sufficient enzyme to allow for normal pigmentation in a Bb individual.
In your question you mention blue and brown eyes. AS @Larry_Parnell indicates, in blue eyes the eyes do not contain the pigment melanin and are therefore blue (the blue color is caused by non-pigmented optic effects). Brown eyes are therefore dominant as the melanine colors the eyes such that the blue color is masked (it is still there though). The phenotype of blue eyes is therefore recessive, as one allele for brown eyes is enough to bring pigment into the iris.
Note that other interactions also exist, such as co-dominance (e.g., the ABO blood group system where the genes for A and B antigens are both dominant over O) and incomplete dominance. I refer to the wiki page on dominance here, as the question is not specifically about these interactions.
Below are some insights about the evolution of dominance. It does not directly answers your question but understanding how it is thought to evolve also help understand the mechanism governing dominance relationship between alleles.
There are several hypotheses about the evolution of dominance. It is important first of all, to note that empirical observations show that beneficial alleles tend to be more dominant than detrimental alleles. Among the two main hypotheses to explain the evolution of dominance, one has been formulated by Ronald Fisher and one by Sewall Wright.
According to Fisher's hypothesis, between two equally beneficial alleles, if one is more dominant than the other than it's heterozygote carrier will have higher fitness. In consequence, beneficial alleles evolve to become more dominant while detrimental alleles evolve to be recessive (so that they can hide from selection in heterozygotes).
According to Wright's hypothesis, beneficial alleles are more dominant because of the kinetic of biochemical reactions. The rate of a biochemical reaction is a function of the concentration in the substrates of interest. The function is called "Michaelis-Menten function" after the name of the authors. The Michaelis-Menten function looks like this:
Think about a knock-out mutation. Such mutation will decrease by half the concentration of protein the gene in question produce in heterozygotes. Imagine, the concentration of proteins in the wildtype homozygote was 3 (see above graph). The rate of reaction of this homozygote is therefore about 3. The heterozygote would have a concentration of 1.5 and the rate of the reaction is therefore about 2.5-2.75. Assuming that the rate of this biochemical reaction is directly related to fitness, then a the locus of interest, benefical alleles are necessarily dominant and detrimental alleles are necessarily recessive. Selection for dominance is not involved in Wirght's model.
Is Fisher's or Wright's hypothesis correct?
Current state of the art is to consider Wright's model to be correct and Fisher's model to be wrong. In reality, the truth probably lies somewhere in between these two extremes. Note also that some other alternative explanations may exist in the literature but Wright and Fisher original hypotheses are by far the most considered hypotheses for the evolution of dominance. During my undergrad, I remember a speaker (but I forgot his name, sorry!) who showed that some alleles actually have some domain that are directly responsible for decreasing the expression of the other allele on the sister chromosome suggesting that Fisher's hypothesis might sometime be a good explanation as well.
There are 2 main mechanisms.
Case 1: active/inactive product. A gene with 2 alleles: inactive and active, that codes for a cosmetic difference. In the case you gave (brown over blue eye color), the brown coloring is due to melanin. If you have 1 functioning copy of the gene for melanin in eye color, then this causes melanin to be produced and the brown color overrides whatever would have been there otherwise.
Case 1b: active/inactive paradigm and disease. A common examples are sickle-cell anemia and red-green colorblindness where you need just 1 functioning copy of a gene to avoid negative health effects. These are insidious for inheritance purposes as it can make people asymptomatic carriers of heritable diseases (even worse in the era before genetics and biology as a field of study since there would be families with widespread, unpredictable, unexplainable occurrences of various diseases - often explained as a "curse" by those who didn't know better)
Case 2: the spoiler. This dominance pattern tends to be associated with genetic diseases where a mutation causes a protein to go rogue and damage its host. Having even one copy of such a mutation is a bad thing, and worse because the chance of handing it down to the next generation is roughly a coin flip (if you have 1 copy) and guaranteed (if you have 2 copies).