Xylem is just an array of emptied dead cells with no inherent directional structure. It will/would pass fluid equally well in either direction. Likewise, phloem is also just a series of tubes, but live ones that are sustained by an adjacent cell. It can and does pass fluids in either direction.
Functionally, flow in xylem is only in one direction primarily because roots are at one end of the tree and evaporation/transpiration is at the other. Water also gets unloaded throughout the tree by osmosis. Tracking the detailed osmotic flow might show water and minerals moving downward locally, there is no significance to this (largely academic) exception.
Functionally, the phloem is actively loaded with the products of photosynthesis. Consequent osmosis generates pressure in the phloem that affects transport away from the source leaf. Importantly, it is via this pressure that nutrients and signalling molecules are supplied to the apical meristem. This is how a new shoot is nourished to grow from buds. This is how flowers are produced. After a new shoot has extended, new foliage will load the associated phloem and 'stuff' will flow the opposite direction. Hence, the phloem is emphasized to be multi-directional.