Why are we not able to slowly and smoothly look from side to side, or up and down in a single and smooth transition, given that we are able to do this if our eyes are focused on a moving object?
Smooth pursuit is driven by retinal slip, which is determined by external input registered peripherally in the retina.
The smooth pursuit system is a system designed to minimize retinal slip, i.e., the movement of an image across the retina. Saccades are rapid eye movements meant to fixate quickly on a new target (Purves et al., 2001).
Both movements are readily apparent when sitting in a train and staring outside; the eyes focus on a distant object and pursue that object. Once it exits the field of view, the eyes target a new object via a saccade and smooth pursuit is again initiated (the train window phenomenon).
Saccades can be triggered both involuntary (due to a sudden, unexpected stimulus) and voluntarily (Walker et al., 2000).
The smooth pursuit system cannot be voluntarily controlled, barred in some highly trained observers (Purves et al., 2001). Instead, it is guided by eye movements dependent on the movement (slip) of the fixated target on the fovea (Thier & Uwe, 2005). Hence, this system is controlled by external stimuli, not by internal ones. Retinal slip can of course be voluntarily manipulated by moving the eyes away from the target, for example by moving the head. The smooth pursuit system detects this by registering retinal slip, and keeps the eyes fixated on the target, through smooth pursuit which is involuntary.
Your eyes are attempting to see; not merely move. The muscles that move your eyes are voluntary. You control them, but think about how you usually command them. Whether sudden and reactive or careful and deliberate, you think about where you want to be looking and make your eyes look there. There is no functional purpose in looking without seeing, and seeing involves tracking specific features, focusing. It requires quick movement between multiple points of focus to gain information efficiently.
Compare this to the movement of your fingers. That movement can be smooth and fluid even though multiple muscle groups have to work together to make it happen. When you command the movement of your finger, you have a body-map that gives you a detailed idea of where your finger is, and how it is connected to you. You can be aware of its location with reasonable precision even if you can't see it. When it is necessary to move it with smooth, fluid motion, your muscle movement is guided by your internal body-map, and usually your eyes, confirming the motion. In contrast, you have very little concept of the actual position of your eyes within their socket, probably because it's not generally very useful information.
Positioning your eyes precisely, independent of what they are looking at is actually counter-productive. Doing so costs you the ability to see, as you can demonstrate to yourself by crossing your eyes, which, incidentally, is probably your best chance at achieving the sort of smooth movement you are referring to.
Probably the most significant reason you can't move your eyes smoothly is what you have practiced consistently for most of your life. With your motor muscles, countless various activities have required some degree of precise, smooth movement. During those very activities, what are your eyes doing? They are only moving smoothly if they are focused on a single point in motion - either a point on your body, or a point on an object which you are moving. Otherwise, they are bouncing from point to point, gathering information to measure whether your muscles need to adjust to accomplish the smooth movement you are attempting.
When you were born, your motor muscles were not proficient with smooth action either. Think about how a newborn baby jerks their limbs around, sometimes even clobbering themselves with their own hands, and then getting upset about being hit. As a toddler grows toward primary school age, they gradually learn to control muscle movement, but still have trouble when a situation demands very precise movement. Practice, for years, is what ultimately brings graceful movement. But no one ever naturally practices graceful eye motions (not independent of focusing on an object), because such movements are not useful for seeing.
Body Map Concept: This concept is something you can demonstrate to yourself by moving around in the dark or by performing a task that you cannot see directly, like writing or typing, with your hands hidden from view. Here is a fascinating RadioLab discussion on phantom limbs which relates closely to this concept.
Motor and Eye movement: The Wikipedia article on Eye-Hand Coordination provides some context around the way eye and motor movement correlate.
Childhood development: Smooth pursuit is an acquired skill early in life (Van Hofsten & Rosander, 1997), which I can back up with personal observations of my 3 boys growing up.
Smooth Eye Movement: I have verified every statement I've made on this topic via personal experiment. I can move my eyes between more and less cross-eyed quite smoothly, which demonstrates that the muscles in the eye are entirely capable of smooth movement on command. It is clear as I do this, however, that not being able to focus on objects while crossing my eyes is an important part of the process. Otherwise, the instinct to focus on something and see it clearly overcomes my ability to control the movement. I have also learned to move one eye at a time from left to right - again, beginning in a cross-eyed state. This is not nearly as smooth, but the practice contributes to my understanding of the nature of voluntary movement of the eyes. Observation of the blind also contributes to conclusions in this area. A person who has been completely blind for a while will often exhibit the kind of smooth motion you describe (in addition to jerking movements), as does a close friend of mine who sees nothing but complete blackness after his optical nerve was destroyed by a bullet.