Sheikh `Abd al-Wahhâb al-Turayrî, former professor at al-Imâm University in Riyadh
Semen is pure, so it is alright if it touches the hands or the clothes.
The evidence for this is the fact that `A’ishah simply scratched dried semen off of the Prophet’s garment, and then the Prophet (peace be upon him) prayed wearing that garment.
And Allah knows best.
Source: Islam Today