To expand on my comments:
Just like you mentioned in your question, carbohydrates (mostly glucose) is used as the primary source of energy when available to satisfy metabolic needs in human cells, however lipids and even proteins will be used if required, in order to produce energy. Since all these biomolecules (carbs, lipids and proteins) can actually be used to sustain the metabolic activity of cells, that means all of them store energy to a greater or lesser extent (which we can quantify in a practical way as food calories). Other things that we eat or drink such as vitamins, cofactors, water, etc. are not accounted for in the calorie count because they can't be broken down to produce energy for metabolism.
Dietary carbohydrates are easily digested into smaller blocks (mostly glucose) and absorbed directly into the bloodstream, afterwards glucose is taken rapidly from the bloodstream by human cells and processed via glycolysis and lactic acid fermentation or the citric acid cycle (all these are metabolic pathways for glucose in humans). Each gram consumed of carbs yields approximately 4 calories of energy.
Lipids on the other hand undergo a more complex process of usage after their slower digestion compared to carbs since after their absorption they have to be transported in special molecules across the lymph, then into the bloodstream and then they can be taken by various kinds of cells. Considering just this, we can start seeing why lipids aren't a fast source of energy (they take long to be available as potential energy sources). Furthermore, the main destination of dietary lipids is the adipose tissue which constitutes a special tissue in which fatty acids can be stored for latter usage. Fatty acids not being used immediately to provide metabolic energy doesn't mean they lose their stored energy (doesn't mean they have no caloric intake value), its just "quiescent" for when its needed (when exercising or when fasting for example).
Each gram of consumed fat yields approximately 9 calories of energy, this quantity is higher compared to that of glucose because the metabolic pathway that breaks down fatty acids (beta-oxidation) in combination with the citric acid cycle produces more metabolic energy (ATP) compared to glycolysis in combination with the same citric acid cycle. However such amounts of energy are not usually required in rest.
It is important to mention that glucose can also be stored in the form of glycogen or even fat since it will be converted to these molecules if there is more glucose available compared to what the cell needs in a given moment. This glucose converted to glycogen or fat also has to be accounted for in the calorie count because its still a potential energy source (just not used right now).
References:
Dietary Carbohydrates - National Agricultural Library
Dietary Lipids - National Agricultural Library
Insights into digestion and absorption of major nutrients in humans - Goodman, B. E.
Food calories per gram of biomolecule - National Agricultural Library
