Viruses evolve alongside their hosts. If you swapped a human chromosome into a chimp cell for the matching chimp chromosome, there would be a lot of genetic similarities but you would also surface a bunch of incompatibilities because the genes on each chromosome have been evolving independently from the rest of the genome.
Viruses interact with their host genome at many stages. First, they need a way to identify their target hosts (cells) in the environment and gain access to the cells. They do this by recognizing markers on the cell surface. For something like HIV, this identification is far more specific than identifying just humans even: they are identifying particular immune cells.
Once inside, viruses interact with the host to express viral genes, produce and assemble new virions, and get transmitted to subsequent hosts. If an incompatibility occurs at any of these stages, the virus won't be able to replicate.
However, because there is a lot of homology across related species, it is feasible for a virus to infect more than one host. If a virus evolves in a context where it commonly infects multiple hosts, there can be selective pressures for it to maintain those abilities across all the hosts. Alternatively, selective pressures might cause the virus to evolve into separate lineages, one infecting ducks and another infecting sparrows, for example.
For a virus that is specific to a certain host (or range of hosts), it is likely that it will have difficulty in another host outside the context in which it evolved. However, there is also variability in viral genomes, and if many viral copies are exposed to a novel host, there is chance that some of them can survive in that host. This will select for those particular viral genes, and subsequent viral generations will be further selected for traits that help replicate in that host.
Immune reactions could also play a role, but in most cases it is the specialization of the viruses themselves that hold the key to host specificity.
Ahlquist, P., Noueiry, A. O., Lee, W. M., Kushner, D. B., & Dye, B. T. (2003). Host factors in positive-strand RNA virus genome replication. Journal of virology, 77(15), 8181-8186.
Hao, L., Sakurai, A., Watanabe, T., Sorensen, E., Nidom, C. A., Newton, M. A., ... & Kawaoka, Y. (2008). Drosophila RNAi screen identifies host genes important for influenza virus replication. Nature, 454(7206), 890.
Schild, G. C., Oxford, J. S., De Jong, J. C., & Webster, R. G. (1983). Evidence for host-cell selection of influenza virus antigenic variants. Nature, 303(5919), 706.