Unfortunately, we are all still "confuzzled" by how memory works. We are far from a complete understanding of how memory is stored and recalled. Nonetheless, we do know a little, so read on.
Your understanding of basic neural function is almost correct. First, an individual neuron will signal through its single axon onto the dendrites of many downstream neurons, not the other way around. Second, I am not sure what you mean by "focusing them based on their permutations," but it is true that neural information can undergo many transformations as it propagates through a circuit. Third, if there is a behavioral outcome of the network activity like a muscle response or hormone release, those effects are mediated by nerves communicating with muscles and hormone-releasing cells. I'm not sure if that is what you meant by "focused response mechanism."
Finally, as you have discovered, the analogy of neural circuits to electrical circuits is relatively poor at any reasonably sophisticated level of analysis. My opinion is that biological systems are often poorly served by being framed as engineering problems. Others will disagree with that, but I think understanding a biological system on its own terms makes many things much clearer.
The key thing missing from the electrical circuit analogy turns out to be one of the keys to understanding information storage in neural circuits--the synapse, the site where one neuron communicates with another. The synapse transforms the electrical signal from the upstream neuron into a chemical signal. That chemical signal is then converted back into an electrical signal by the downstream neuron.
The strength of the synapse can be adjusted in a long-term way by changing the level of protein expression--this is called long-term potentiation (LTP) or long-term depression (LTD). LTP and LTD therefore can regulate the ease with which information can flow along a particular path. As a basic example (that should not be taken too seriously), imagine a set of neurons that represents "New York City" and another set of neurons that represents "My Friend John." If you then happen to be in New York City with your friend John, both of those groups of neurons will be active and synapses between these two networks will be strengthened because they are co-active (see Hebbian plasticity). In this way, the idea of NYC and the idea of John are now bound together.
Where are these neurons that represent NYC and John? We are still not totally clear on this, and the question is complicated because there are many different types of memory. For instance, your memory of how to ride a bike (procedural memory) is not treated the same as your memory of what you ate for breakfast (episodic memory). However, a best current answer is that the hippocampus and its associated regions are important for the initial encoding of memories and the neocortex is where longer term memories are stored. There is substantial communication between these two areas so that memories can be effectively adjusted over time.
Update
In response to Jule's comment asking for some resources, I realize it is important to make the point that the Hebbian model I outlined hasn't been definitively shown. Like with all aspects of neuroscience, there is a lot of good work at the molecular and cellular level and good work at the behavioral level, but the causal link between the two is not so clear. Nonetheless, Hebb's idea is still the mainstream working model for how memory works. Some reading might include:
1) Neves, G., Cooke, S.F., Bliss, T.V.P., 2008. Synaptic plasticity, memory and the hippocampus: a neural network approach to causality. Nature Reviews Neuroscience 9, 65–75.
A review on hippocampal memory and its relation to LTP/LTD and Hebbian theory. Notes the general difficulty of proving the theory and some ways for experiments to move forward.
2) Lisman, J., Grace, A.A., Duzel, E., 2011. A neoHebbian framework for episodic memory; role of dopamine-dependent late LTP. Trends in Neurosciences 34, 536–547.
A review proposing an elaboration of the Hebbian model that includes neuromodulatory influence on plasticity and memory process.
3) Johansen, J.P., Cain, C.K., Ostroff, L.E., LeDoux, J.E., 2011. Molecular Mechanisms of Fear Learning and Memory. Cell 147, 509–524.. An excellent review on fear learning and memory with an extensive section on Hebbian theory.
4) Liu, X., Ramirez, S., Pang, P.T., Puryear, C.B., Govindarajan, A., Deisseroth, K., Tonegawa, S., 2012. Optogenetic stimulation of a hippocampal engram activates fear memory recall. Nature. A research article which is perhaps a realization of some of suggestions in the Neves et al review. They use light to reactivate a fear memory. This suggests that activation of the hippocampal network that was active during memory formation is sufficient to elicit the memory.
