In the image the circular molecule is a restricted vector and the linear red molecule is a DNA insert. I found this protocol in my lessons notes, but I don't understand how it is possible that ligase catalyzes a reaction between 5'-OH and 5'-P. This reaction should form as product a 5'-5' bridge between the 2 nucleotides of the insert and the vector. Usually Ligase repairs nicks between 5'-P and 3'-OH thus creating 5'-3' bonds between nucleotides that are normal in nucleic acids, so my question is how is it possible for the DNA to be functional even though it has a 5'-5' bond?
DNA ligase catalyzes the formation of a bond between 3’-OH and 5’-PO4 of its nucleic acid substrate. (See Lehnman, 1974) The ligation bond will be formed when the proper ends are close together, allowing the ligase to act. Lets redraw your diagram and add the 5’ and 3’ designations to it. I think this may help clear out some points.
Regarding the alkaline phosphatase treatment with ligation, I think Dale and Greenaway said it so much better than I can. Quoting (Dale and Greenaway, 1985):
[The strategy] is to treat the cleaved vector DNA with alkaline phosphatase that removes the 5’-phosphate group from the DNA. Since the presence of a terminal 5’-phosphate is necessary for the action of DNA ligase, a dephosphorylated DNA strand will not be able to participate in a ligation reaction. Since this will apply to both ends of the linearized vector, the ends cannot be joined together again, i e , parental vector regeneration is impossible. This results in a very low frequency of transformation by the vector (since linear DNA transforms very poorly). However, the vector 3’-hydroxyl ends are available for joining (by DNA ligase) to the 5’-phosphate termini of any DNA fragments to be cloned. This will result in a circular recombinant molecules with a single-stranded nick at each end of the vector portion. These molecules usually transform as efficiently as the fully ligated product, and the nicks are repaired in vivo after transformation. Thus the phosphatase treatment of the vector molecules increases the proportion of hybrid DNA molecules in the population of transformed cells, and also prevents the formation of oligomeric forms of the vector.
Dale, J.W. and Greenaway, P.J. (1985) “The use of alkaline phosphatase to prevent vector regeneration” Methods Mol. Biol. 2:231-236. Link to paper
Lehnman, I.R. (1974) “DNA Ligase: Structure, Mechanism, and Function” Science 186:790-797. Link to paper