At an approximation the active sites of enzymes can be considered as having two aspects. The first relates to the catalysis — in this case the breaking of the glycosidic linkage. The second relates to binding the substrate. This review of the α-amylases by MacGregor et al. shows that there is a range of a-amylases, differing in this latter respect — their substrate specificity. In general there are binding sites for a varying numbers of glucose residues at either side of the bond being cleaved. This is shown in Fig. 3 of that review:
The important difference in the structure of glycogen and starch (amylopectin) — seldom mentioned in general biochemical or biology texts — is their patten of branching:
As this previous answer of mine to a different question explains, this results in a globular structure for glycogen granules in which only the ends of the chains are accessible. (The image below, from Protopedia, illustrates this better, especially if you imagine it in three dimensions.)
Hence the substrate-binding site of α-amylase does not have access to the residues that need to bind for it to perform hydroysis of glycogen, and, indeed, the enzyme that breaks down glycogen — glycogen phosphorylase — is specific for these free ends.
The α-amylases that can hydrolyse both α-1,4 and α-1,6 glycosidic links are quite few compared with those with specificity to one or the other type of linkage (see Table 2 of the MacGregor review, if you can obtain access to it). The impression obtained from following up two of the examples there is that the enzymes involved can exist in alternative conformations, the correct one of which is triggered by the substrate. An example is the glycogen debranching enzyme, the studies of which in Sulfolobus solfataricus and Candida glabrata can be read freely on-line. Although somewhat less directly relevant, the example of a Thermoactinomyces vulgaris neopullulanase is another variation on this theme.