To understand this, you have to understand how an agarose gel is able to effect the size separation of DNA in the first place.
The reason agarose gels work to separate DNA based on size is that the electric field produced the electrophoresis machine is imparting force on the DNA. The effect of the increasing charge for DNAs with increasing nucleotides is offset almost exactly by the increasing mass of those larger DNA molecules, so in the absence of the gel, all pieces of DNA would migrate through the apparatus at a constant rate.
The reason they don't is because of the mesh framework of the agarose gel. This is like a dense forest or a large number of sieves with random sized holes. There's small holes (of varying size) and in order for large pieces of DNA to work its way through the gel, it has to weave through these pores. Smaller pieces of DNA can go through smaller pores, so they take a faster, more direct path, while larger pieces have to make detours, and even when they find a pore they can fit through have to spend a bunch of time flopping around before they can stuff themselves through the pore.
So it's not strictly speaking "size" or "compactness" by which the agarose gel separates molecules, but instead it's the ability of the piece of DNA to navigate through the mesh of agarose molecules. While relaxed, circular DNA is more compact than an extended linear DNA of the same nt length, that doesn't necessarily mean that it has an easier time working its way through the network.
For example, a linear piece of DNA can orient itself so it goes "end first" through the mesh. Once the end of the DNA goes through a pore, it's relatively easy to pull the rest of the DNA through the same pore, as compared to trying to bend the DNA and pull it "sideways" through the pore - you have to remember that on molecular scales, DNA is rather stiff. You can't kink it in half very easily, so a piece of DNA that has to go through the pore "sideways" takes much more space than one that can be threaded through end-wise, even if you bend it as much as possible into a hairpin.
With a relaxed circular DNA, there's no end, so it's always going through the pores "sideways" so it's limited to the larger pores. In addition, it's large and floppy, so it takes quite a while to go through the pores that are there. When you add supercoiling, you don't fix the issue with going through pores "sideways", but you do limit the flopping around it does, so once you do find a pore of a usable size, you get the rest of the DNA through faster. In fact, with enough supercoiling you reduce the time spent flopping around enough that it goes through the agarose even faster than the linear DNA, which - while it can fit through smaller pores - still spends a bunch of time flopping around instead of going through the pore that it's started to go through.