What is the energy penalty of maintaining rarely used synapse?
I'm wondering if forgetting is cheaper than remembering unnecessary details.
Short answer is yes, of course. Maintaining a healthy synapse is very energy-consuming process.
Let me first note that forgetting as we know it is more complicated than just pruning of synapses. It might involve rearrangement of proteins in pre/post-synaptic membrane and near it, local de-regulation of local protein synthesis, decrease in synaptic mRNA content and epigenetic change of transcription.
Secondly, average energy spending on a synapse, if it is not being used too often, includes mainly maintaining of membrane potential at resting -80mV level. Cellular membrane is a leaky barrier, so that without ion pumps, such as $K^+/Na^+$-ATPase, potential will quickly go to zero. Maintaining and recycling neurotransmitter pool also require energy.
Action potentials itself are passive processes, its generating does not require energy. ATPase will have to maintain potential at rest and keep it from fluctuating.
You can read this review for more: Synaptic Energy Use and Supply (Neuron, 2012). It generally describes where energy comes from and how it is being used.