What diagnostic applications, if any, are there in using bioluminescence to detect cancer or tumors (in vivo)?
Not in human but you can use this technique with genetically modified model organisms as described here.
The procedure is quite simple, you express the luciferase enzyme under the control of a specific promoter (specific for your cell type, like cancer) and provide luciferin via intravascular or intraperitoneal injection. The targeted cell type (for example cancer) will light up.
If you want to use that as a diagnostic tool in human you can't as the organism must express an enzyme catalyzing a bioluminescent reaction. Actually with the advance of genome engineering tools you might be able to do that even in humans (e.g. using CRISP/Cas9).
What is the advantage of bioluminescence over fluorescence? I might be wrong but I don't think there are any. I would rather express a fluorophore rather than an enzyme that requires a substrat.