These three terms are often misused in the literature. Many researchers seem to treat them as synonyms. So, what is the definition of each of these terms and how do they differ from one another?
Both orthologs and paralogs are types of homologs, that is, they denote genes that derive from the same ancestral sequence.
Orthologs are corresponding genes in different lineages and are a result of speciation, whereas paralogs result from a gene duplication. This often has important implications: while orthologs often fulfill the same role, paralogs tend to diverge in their function, so paralogy is a worse indicator of functional analogy than orthology.
This, however, is only the tip of the iceberg, since the situation can be much more complex (see, for example, the hidden paralogy problem).
There is a great article by Fitch on that subject.
First, a note on spelling. Both "ortholog" and "orthologue" are correct, one is the American and the other the British spelling. The same is true for homolog and paralog.
On to the biology. Homology is the blanket term, both ortho- and paralogs are homologs. So, when in doubt use "homologs". However:
The following image, adapted (slightly) from , illustrates the differences:
Part (a) of the diagram above shows a hypothetical evolutionary history of a gene. The ancestral genome had two copies of this gene (A and B) which were paralogs. At some point, the ancestral species split into two daughter species, each of whose genomes contain two copies of the ancestral duplicated gene (A1,A2 and B1,B2).
These genes are all homologous to one another but are they paralogs or orthologs? Since the duplication event that created genes A and B occurred before the speciation event that created species 1 and 2, A genes will be paralogs of B genes and 1 genes will be orthologs of 2 genes:
This however, is a very simple case. What happens when a duplication occurs after a speciation event? In part (b) of the above diagram, the ancestral gene was duplicated only in species 2's lineage. Therefore, in (b):
A common misconception is that paralogous genes are those homologous genes that are in the same genome while orthologous genes are those that are in different genomes. As you can see in the example above, this is absolutely not true. While it can happen that way, ortho- vs paralogy depends exclusively on the evolutionary history of the genes involved. If you do not know whether a particular homology relationship is the result of a gene duplication or a speciation event, then you cannot know if it is a case of paralogy or orthology.
I highly recommend the Jensen article referenced above. I read it when I was first starting to work on comparative genomics and evolution and it is a wonderfully clear and succinct explanation of the terms. Some of the articles referenced therein are also worth a read:
First, the definition: two genes are homologs if they derive from a common ancestor. Generally speaking, if two nucleotide sequences have at least 30% (or greater than 10% amino acid sequence) identity, they are likely to be from a common ancestry, however, they may not be homologous. Note the reverse does not apply: two genes can also be homologs if there is no similarity; this happens every time the drift was long enough (many genes are no longer similar---beyond random identity---after 1 billion years; only highly conserved genes keep some similarity).
In addition to the other answers another diagram and further terms:
(after W. M. Fitch, Trends in Genetics, 16, 5, May 2000, p.228)
I think (1:n) ortholog is a synonym for outparalog.
Klaus D. Grasser: Annual Plant Reviews, Regulation of Transcription in Plants (Volume 29). Wiley-Blackwell, ISBN 1-4051-4528-5, p. 37.