Remember that glycolysis yields 2 NADH as well as 2 net ATP. This NADH can be used to a terminal electron acceptor to produce an end product with a net gain of ATP. End-products of fermentation can include lactate, acetate, butyrate, propionate and ethanol, all of which generate different amounts of additional ATP.
The exact pathways involved vary according to the organism, but anywhere between 1 (or even 0.5) and 4 additional ATPs can be generated, even without the involvement of oxygen. In the case of aerobic respiration of course, the NADH feeds into the electron transport chain and yields ATP.
EDIT: It's perhaps worth noting that the 2 ATP is gained. See @SatwikPasani's answer for the math.