What I want to know is if polypeptides (those which are not considered as proteins) can also "exhibit" a tertiary structure.
I believe the OP is referring to an old and outdated system of classification, in which all proteins are polypeptides but not all polypeptides are proteins: some books put a lower limit of 50 AA residues, others talked about 60 or 70 AA for a polypeptide being considered a protein. Once again, this system is no longer used (just check Lehninger, "Principles of Biochemistry").
But let's pretend that this system is valid and use it just to answer this question: can a small polypeptide that is not a protein, ranging from 2 to 50 AA lenght, have a tertiary structure?
To answer that, we have to remember that the tertiary structure is not (as some defines) simply the "3D structure of a polypeptide". After all, a single amino acid has a 3D structure, as any molecule indeed. The tertiary structure is the specific 3D format of the polypeptide, but due to interactions of its amino acid side chains.
So, the question is: can a small polypeptide fold over itself and have a tertiary structure? The answer is yes. Apparently, peptides with more than 10 AA can fold: http://www.ncbi.nlm.nih.gov/pubmed/12729764
Your instinct is valid, there are other molecules which can be defined as having tertiary structure.
The most common example is RNA, which can fold into a vast array of conformations. These can be part of ribonucleoprotein complexes like telomerases, spliceosomes, or ribosomes or they can function on their own like tRNAs and many ribozymes such as RNase P. An example of a common tertiary structural motif in RNA is the pseudoknot; a haripin loop in which some of the bases in the loop have base paired to another portion of the RNA molecule.
It is also possible for DNA to possess tertiary structure, though the term is used less often when referring to DNA. Usually DNA tertiary structure describes topological structures formed by supercoiling, protein scaffolding, etc. However, DNA can also form cruciform motifs similar to RNA hairpin loops (though it is less common). If multiple cruciform motifs and other structures were to interact in three dimensions, this could be another form of DNA tertiary structure.
You may also be interested in DNA origami, in which 3D building blocks made from DNA are assembled in desired ways by using sequence complementarity (generally).
I think that polysaccharides and oligosaccharides are also sometimes referred to as having tertiary structures, but I don't know as much about those. I encourage you to look into pectin, amylopectin, branched oligosaccharides, etc in the context of tertiary structures.