The majority of primates have really thick nails, though not as broad or flat as humans'. However, marmosets have 18 claws, and only their big toes have nails. What differences in evolutionary pressures caused this? And how does the structural relationship between marmosets' germinal matrices and their distal phalanxes differ from other primates, so they can grow claws?
Since marmosets and tamarins are considered the most primitive primates, we can't assume that evolutionary pressure led to the development of claws. Rather, it's accepted that claws disappeared as primates adapted to an arboreal lifestyle. So the question is really "Why did marmosets not completely lose their claws?"
The clues to the answer lie in their unusually large incisors and disgestive system. The marmoset occupies an evolutionary niche where they rely heavily (20-70% of total diet) on exudates (sap, resin, etc.) from trees. Their claws, and incisors allow them to claw/chew holes in the bark to get at the exudates, while a specially adapted large intestine allows them to digest this food. The advantage of exudates is they are not seasonal, offering the marmoset a relatively stable year-round food supply.
Other primates enjoyed the evolutionary benefits of tool use by losing their claws and gaining opposable thumbs (which marmosets don't have). I've come across no papers that describe a different phlange structure for marmosets and other primates, and there doesn't necessarily need to be one.
Most of this information came from the University of Wisconsin's Primate Research Center:
Edited to add (since I can't comment): One way to deduce how different features evolved is to look at phylogenetic trees. There are many out there, but I pulled this one from J.R. Roede et al. Redox Biology 1 (2013) 387-393.
If you look at this simple tree, you'll see that marmosets diverged from humans on a time frame where it's possible they kept their claws instead of reacquiring this trait (it would be unusual, but not impossible, for this to happen). One thing you can do is dive deeper into the phylogeny, square this up with how primate hands evolved, and see if the split of marmosets from the rest of the primates happened when nails began to evolve.
Ill try and research this later and add references, it's a bit of a draft version while im on laptop in the car, A good place to think but not to write and surf from!
It's probably best viewed as a vestigial trait, made redundant by monkey's prehensile grasp, and related to their strong social and hierarchical behaviour. Indeed the flat nails seem to be a progressive arrival transitioning from lemures/aye-ayes to larger prehensile primates.
Humans have the most vestigial nails, to protect the fingers and for fine fingertip manipulation, and earlier mammals and non-prehensile herbivores find advantage of pointy nails for grasping bark and defense, to decorticate husks rinds, to dig, and other tasks.
We can imagine if nails had become reduced vestigial tools in a different way to what they did, imagine that they had tapered into bulbous tips
We can consider other alternatives that could have evolved, but the flat shape seems most fitting for a finger protection and versatility.
Also consider that large monkeys are belligerent, for example baboons which are very fierce, and monkeys that have sharp teeth which they can use in combat. If they had claws to do demonstrative fighting, it may be less beneficial for the group, and nails would generally discourage the mostly demonstrative social competition which is found in most social and rutting creatures. It demonstrates the contrast of vestigial carnivorous traits of some primates compared to the herbivorous organs of the largest ones.
With regards to the structure, perhaps it's best to check other mammals fingernails to see that it's only a small proportional and dimensional change.
Conical nails require a different shaped matrix which doesn't fit on top of fingertips for grooming and find dexterity, to alight with the bone, the nail matrix surrounds the top of the finger bone, so it's easy to broaden and thin out.
It's matrix cells that generate keratin which is not structurally affixed to the bone, which means it can regenerate efficiently, while it has structural weakness for lifting bodyweight. For carnivores, the stress on the nail is reduced by the cutting edge. The structural difference is not very pronounced. web resources for primate nail anatomy aren't abundant but you can find trees of primate hand anatomy.