No. Making of cloth is only perhaps 30,000 to 40,000 year old (History of clothing and textiles), and body coverings of any form, e.g. animal skins, only around 100,000 years old (based on body lice).
Females being smaller than males is much, much older than this. Chimpanzees, our closest relatives, similarly have smaller females (and don't dress in cloth); their line and our line separated around 4 million years ago. Our next closest relatives, the gorillas, have an even greater difference between the sexes, the females being half the size of the males. They separated from our line perhaps 6 to 8 million years ago. Further back, most all our anthropoid primate relatives also show smaller female size (but the most distant primates, like lemurs, do not). For more information see Sexual dimorphism in non-human primates.