Yes it can, because exposure alone can cause mutations in your genes which is usually the main cause of cancer.
For example, when we get exposed to UV light for longer periods of time, we can get skin cancer/melanoma. Here , of course no traces of these rays are present in our body, its just all due to prolonged exposure.
Now to the fact that why does it even happen , according to wikipedia is
Cancer is a stochastic effect of radiation, meaning that the probability of occurrence increases with effective radiation dose, but the severity of the cancer is independent of dose. The speed at which cancer advances, the prognosis, the degree of pain, and every other feature of the disease are not functions of the radiation dose to which the person is exposed.
So since once induced, it just keeps on increasing even though you are not exposed to that particular rays anymore, you most probably are going to have it. This is the reason why cancer is so dangerous, because once it is there you can't stop and even can't make any precise stats about it until a certain stage.
For more details , visit https://en.wikipedia.org/wiki/Radiation-induced_cancer#Mechanism
I hope this answer helps you :)