Both the Forewing as well as the Hindwing of Butterflies are made of thin chitin structures which are pretty thin and sensitive. If you touch the wing with to much force, it may break.
Then the upper side of the butterfly wing is covered with small scales - what you called the "dust". This can be seen in this figure (from here):
and also in this electron microscopy image (from the Wikipedia):
The scales have a number of functions, they are responsible for the butterflies color and pattern (which are important for finding mating partners as well as for camouflage) and they also seem to play a role in insulation and thermo regulation (see here for more details).
The scales play also a role in making the wings stronger and help improving the flight characteristics. When flying upwards the get pressed to the wing and act aerodynamically, what the wings are in downward motion, they help stabilizing the flight. See here for more details: "Butterfly wing scales : pigmentation and structural properties".
Some of the scales get lost during flying and collecting nectar, so taken together, loss of a big number of them will influence the flight characteristics of a butterfly. However, I think it is much more likely that such a rude treatment will damage the wings, which is obviously more harmful. So light touches will not do harm, more severe will.