In this paper the authors measured decrease of arterial blood oxygen saturation in healthy and patients with sleep apnae (this latter part is not really relevant only the measurements in healthy individuals).
They found that fall of oxygen saturation depends on the initial saturation rate, but is not affected by the breath-holding time. As you can see on the graph at high initial saturation the fall rate per second is quite low. Here is the info in a table format:
At high initial saturation only mean 0.16% saturation drop occurs per second, thus in average a healthy individual looses 1% oxygen saturation approx every 5 seconds (I'd also like to note that the urge to breathe does not come from the lack of oxygen when holding breath, but from the accumulation of CO2). According to wikipedia normal oxygen saturation is between 100-95%, and only below 90% is considered hypoxemia /hypoxia (a medical condition where blood oxygen level is low). So if we look at the data in the table, it is safe to say that it takes approx 20s until the blood saturation level drops from 97% to 93% and another 5-6 seconds until it drops to 90% (with approx 0.5% saturation loss per sec at 93% initial saturation).
People who are conditioned by exercise have larger vital-capacity (they can breathe more air in for a single breath), thus have means to acquire more oxygen can compensate for the excess use of oxygen. During heavy sport activity like swimming it is unlikely that anybody will hold his/her breath for several tens of seconds.