Have you used hhpred to search for homologues? What was your criteria for defining there was no homologues? You could potentially go down to around 30% sequence identity to model.
I would submit the the query sequence to both I-Tasser and Rosetta and see if both of the servers agree on the topology. I-Tasser will always provide 5 models of your query sequence (even if they are rubbish) ranked by a confidence score.
While you wait for the above servers to finish, I would submit the sequence to a number of secondary structure prediction servers to come to a general consensus for the secondary structure of your protein. Some servers you could use are:
If the servers come to a consensus, compare the secondary structure prediction with that of the secondary strucutre of the models, to see if the ab initio modelling concurs with that of the secondary strucutre prediction. This will give you more reason to be confident in your model.
I would then validate your models using:
- Prosa - This calculates a Z-score, and plots where it fits with structures with a similar amino acid length, from the PDB.
- DOPE Score - This is employed in Modeller and measures the "nativeness" of your model, the lower the score the better. But you will need to install Python and have some knowledge of how to use the command line. I can provide the code to Run Dope with modeller.
Finally, are there any constraints you could introduce such a oligomerisation site or a protein-protein interaction site, to help with the modelling?