Well, the answer which explains the difference between calcium and potassium is quite simple. Do we agree that charge separation (between the two sides of the membrane) is the thing which creates the potential? meaning, if the charge concentrations at the two sides were the same, then the membrane potential was zero. right?
So here is the thing. You have much more Ca outside the cell than inside and you have much more potassium inside than outside. So now, if you add Ca outside, you increase the difference between the two sides (the separation of Calcium ions) so that Ca will be in a greater state of unrest, the system will be less stable and the potential will be higher. On the other hand, when you add K outside, you narrow the gap between its two concentrations on the two sides (decrease its gradient), so now it'll experience less tension, less unrest, hence contribute less potential to the membrane.
Remember that in our cells, the ion pumps are the ones which serve as the battery that keeps the potential fixed. When do you think the pumps will have to work harder and to do more work (voltage times charge is work), when you add more K outside, or when it's the Ca which is being added?