There are differences in metabolism, and the "calories in" part is the one which differs more than the "calories out" part.
People do lose different amount of calories for the same action, but for mostly mechanical reasons. If a 50 kg person walks up a mountain at 3 km/h, and a 100 kg person walks up the same mountain at the same speed, the 100 kg person will expend more energy, because they did more work. And then there are slight differences in how we do our movement - some people's gait is more efficient than others', and uses less energy. But this is not really interesting for the explanation of having it easy/hard to lose weight.
The point is that the amount of food we intake is determined by our hunger, and our hunger is determined by hormones. Then, the amount of fat we store is also determined by hormones. If your body "thinks" you should be putting on weight, and you go on a calorie-reduced diet, it won't just start complementing the missing energy from the fat stored in your adipose tissue. Rather, it will continue diverting some of your food intake to make more fat storage, all the while rationing the energy available for other activities: making you feel tired so you don't move much, shutting down some luxurious, energy-hungry processes in the brain, mostly the ones associated with higher mental functioning, and making you feel hungry and cold so you will be more motivated to seek out food and eat. The hormones which regulate these processes are many, most of them are poorly understood, some are probably not yet discovered (orexin, one of the main hunger regulating compounds, was only discovered sometime in the last 5 years I think), and we really don't have a complete picture of everything that is involved. It is a complex mesh of finely balanced homeostatic systems which interact with each other and the environment.
The people who say that it is as simple as "calories in, calories out" are right. They are right in the sense that, if you just observe a human being and measure their energy household, you will be able to document the amounts of energy consumed, expended and stored, and the energy consumed will equal the energy consumed plus the energy stored.
But to derive from this that we are talking about a passive overflow system and, if we reduce intake, storage is also reduced, is a big leap of faith and has been proven wrong again and again. Living systems are actively reacting to any influences from outside, by mechanisms much more intricate than simple mechanic rules. Mechanics tells us that if I push you, you will fall over; experience tells us that if I push you, you will brace yourself so you don't fall over, and then try to stop me from pushing you by using words, force, or whatever else you have at your disposal. Our physiological systems are the same way. They like the situation they have chosen to be in as much as you like standing upright. If somebody comes from outside and tries to forcibly change their status quo, they do whatever they can to 1) resist and retain the status quo, and 2) stop the outside force if possible.
As for the difference in metabolism: the hormone systems of different people are differently "tuned". For example, eating pure glucose causes insulin production to spike, and excess insulin causes glucose to be removed from the bloodstream, plus a handful of other effects. But the size of the insulin spike is different in different people. Also, the extent to which cells react to elevated insulin is different. When the reaction becomes dangerously weak at even high insulin levels, we say that it's a pathological variation, and call it "metabolic syndrome". If goes really towards zero, it's called diabetes type two. And this is an example about just one of the hormones connected to metabolism and the distribution and use of energy in the body.
On a simpler level, there is no difference in metabolism where physical energy calculation is concerned. Expending one molecule of ATP will let the same number of Na+ ions to move the same distance along a nerve dendrite, across humans. But this level is not really interesting when it comes to gaining or losing weight.
