'Circular connections', as you refer to them, are super common in the nervous system. This motif usually referred to as feedback connections, and serves a variety of roles.
Connectivity studies can be broadly grouped in to those that look at brain areas, and those that look at individual neurons (usually within a single brain area).
My background is more the latter, so that's what I'll focus on.
Firstly, your question misses an important distinction: inhibitory neurons. So if neuron C was inhibitory, we would have a negative feedback loop, a classical circuit that can be used to produce a homeostatic like signal, keeping a system within certain bounds; a delayed feedback loop can also act as a 'signal terminator', allowing an action to be stopped once it reaches a certain threshold. Lastly, under some circumstance, feedback loops can produce oscillations, which are applicable to a number of tasks.
But more generally, recurrent connectivity is hugely important in the cortex. The majority of inputs to neurons come from neighbouring excitatory neurons (see https://www.ncbi.nlm.nih.gov/pubmed/7638624), and if A connects to B then B is more likely to connect to A than by chance (see http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1054880&tool=pmcentrez&rendertype=abstract for a really nice study on this).
So why? For a variety of reasons. Networks with feedback mechanisms can do more, is the basic answer; from negative feedback for homeostasis, to persistent network states. My personal feeling is that too much of computational neuroscience models a network as an operator - it takes an image, say, and outputs what it thinks the image is. And whilst that is important, it's not what the brain is. We need to maintain a constant internal model of the world around us, rather than going 'that thing is a cat. That thing is a burger'. Perhaps recurrent activity maintains the cortex in a certain state - knowing that the cat is there - and inputs serve to move the state so that we can update our internal models when the sensory environment changes.