There isn't any particular structure there: you have the wall of the aorta/adventitia, and if you have an explanted heart there is a space and then the auricle of the left atrium on one side and the right atrium on the other. These would all be contained within the pericardium.
Where the aorta is most "touching" the left atrium is where the pulmonary veins come in: I think this picture from Gray is most helpful.
Figure 494. Henry Gray (1825–1861). Anatomy of the Human Body. 1918.
There really isn't much to distinguish these veins from the non-auricle part of the atrium, similar to the vena cava on the right side. If you were to cut along the veins eventually you would just open up into the atrium.
The Visible Heart Lab is another good reference http://www.vhlab.umn.edu/atlas/aorta for cardiac anatomy.