You are asking two different, but related questions
Does breast size depend solely on fat amounts?
First, breasts are made up of skin, blood vessels, ducts, fibrous tissue, and fat. There is variation between individuals and within an individual the relative composition changes with age, nutrition, and hormonal status. The size of each of those elements will determine the overall size of the breasts.
What makes breasts firm or soft given the same size?
We don't know, but it's not fat. You can determine the relative amount of fat in breast tissue (fat is black, or radiolucent, other structures are white, or radio-opaque) using mammography. There is even a term for the ratio of non-fatty structures to fat (mammographic density) but physical findings by trained examiners regarding the firmness or softness of breasts do not correlate with the relative composition of a breast. This is tricky because breast density as it is defined in mammography is relative amount of non-fatty structures compared to fat, but you can't predict the mammographic breast density with physical examination of the breast.