If a person forgets and says when bowing Subhaana rabbiy al-ala by mistake, or he says when prostrating, Subhaana rabbiy al-azeem, it is one of two scenarios:
(i) He remembers that he did not say the dhikr in the right place, in which case he should say Subhaan rabbiy al-azeem before rising from bowing, and he should say when prostrating Subhaana rabbiy al-ala before he sits up.
In this case he does not have to do the prostration of forgetfulness, because he did not omit something that is obligatory; rather it is mustahabb for him to do the prostration of forgetfulness, because he did not say the dhikr in the right place.
(ii) He does not remember that he did not say the dhikr in the right place until after rising from bowing or prostration.
In this case he has to do the prostration of forgetfulness because he omitted something obligatory.
In this case the prostration should come before the salaam.
We have previously discussed this issue in the answer to question no. 7743
Shaykh Ibn Uthaymeen (may Allah have mercy on him) said: If a person says a prescribed phrase in the wrong place, it is prescribed for him to do the prostration of forgetfulness, such as if he says Subhaana rabbiy al-ala whilst bowing, then he remembers and says Subhaana rabbiy al-azeem In this case, he said a phrase that is prescribed, namely Subhaana rabbiy al-ala, but it is prescribed to say Subhaana rabbiy al-ala when prostrating. So if he says it when bowing, we say: You said something that is prescribed in the wrong place, so doing the prostration (of forgetfulness) is Sunnah in your case.
End quote from ash-Sharh al-Mumti, 3/359
Shaykh Ibn Jibreen (may Allah have mercy on him) said: If he says a prescribed phrase in the wrong place by mistake, it does not invalidate the prayerâ¦ so if he recites Qur'aan when he is sitting or recites the Tashahhud when he is standing, or he says Subhaana rabbiy al-ala whilst bowing or Subhaana rabbiy al-azeem whilst prostrating, i.e., he does something that is Sunnah in the wrong place even though it is prescribed, then it is Sunnah for him to do the prostration of forgetfulness, but it is not obligatory, because these phrases all come under the heading of adhkaar al-salaat (dhikrs recited during the prayer), so the prayer is not invalidated by doing so. End quote from Sharh Akhdar al-Mukhtasaraat
If the worshipper was praying behind an imam and did any of the things mentioned above, then he should do the prostration of forgetfulness at the end of his prayer, if he joined the prayer late. If he offered the entire prayer from the beginning with the imam, then he should say the salaam with the imam and he does not have to do the prostration of forgetfulness.
Shaykh Ibn Uthaymeen (may Allah have mercy on him) said: If one who is praying behind an imam makes a mistake in his prayer, and he did not join the prayer late, i.e., he did all the rakahs with the imam, such as if he forgot to say Subhaana rabbiy al-azeem when bowing, he does not have to do the prostration of forgetfulness, because the imam has borne it for him. But if we assume that the one who prayed behind the imam made a mistake that caused one of the rakahs to be invalidated, such as if he forgot to recite al-Faatihah, then in this case, when the imam says the salaam, he has to stand up and do the rakah that was invalidated because of his mistake, then he should recite the tashahhud, say the salaam and do the prostration of forgetfulness after the salaam.
But if the one who prays behind the imam makes a mistake in his prayer and he joined the prayer late, then he should do the prostration of forgetfulness, whether his mistake came when he was praying with the imam or after he stood up to complete what he had missed, because if he prostrates he will not be going against his imam, because his imam will have already finished his prayer. End quote.
See: Risaalah fi Ahkaam Sujood as-Sahw, by Shaykh Ibn Uthaymeen (may Allah have mercy on him). See also the answer to question no. 35909.
And Allah knows best.
Reproduced from Islam QA