The opening duaa' is Sunnah according to the majority of scholars, and whenever the worshipper catches up with his imam in the first or second rakah, he should recite the opening duaa', so long as he is not worried that the imam will bow (too soon). But if he is worried that the imam will bow, then he should recite al-Faatihah only, because reciting it is obligatory.
An-Nawawi (may Allah have mercy on him) said: If one who joins the prayer late starts to pray, then the imam says Ameen immediately after he has started to pray, then he should say Ameen and then recite the opening duaa', because the saying ameen is brief. If the one who joins the prayer late catches up with the imam in the final tashahhud, he should say takbeer and sit down, then if the imam says the salaam as soon as he has sat down, then he should stand up (to make up what he has missed of the prayer) and not say the opening duaa', because he has missed the place for it.
End quote from al-Majmoo, 3/275
He also said: If he catches up with him when he is still standing, and he knows that he will be able to recite the opening duaa' and seek refuge with Allah and recite al-Faatihah, then he must do so. This was stated by ash-Shaafai in al-Umm, and also by our companions. â¦ If he knows that he will only be able to recite part of the opening duaa' and seek refuge with Allah and recite al-Faatihah, and he will not be able to do all of it, he should do what he is able to do. This was stated in al-Umm.
End quote from al-Majmoo, 3/276
Shaykh Abd al-Azeez (may Allah have mercy on him) said:
If the latecomer joins the prayer when the imam is reaching the end of his recitation, before bowing, should he start his prayer with the opening duaa', or should he join the imam and remain silent?
If the latecomer arrives when the imam is bowing, he should bow with him, and not recite the opening duaa' or recite any Qur'an; rather he should say takbeer and bow. But if he comes when the imam is still standing and there is plenty of time, then he should recite the opening duaa' and al-Faatihah. This is what is prescribed for him: to recite the opening duaa' and then al-Faatihah, even in the prayers in which the imam recites out loud. If there is a pause where the imam falls silent, he should recite it at that time; otherwise he should recite it to himself, then after that he should listen attentively to the imam. But if he comes late, when the imam is bowing, then he should say takbeer and bow, and the requirement to recite al-Faatihah is waived in this case, because he is excused.
End quote from Majmoo al-Fataawa, 30/150
And Allah knows best.
Reproduced from Islam QA