He has stopped play, but play is restarted by the fouled team taking the DFK quickly, which they are in their rights to do, (they can also choose to ask for a ceremonial FK (wall).
According to law 12:
- Delaying the restart of play to show a card - Once the referee has decided to caution or send off a player, play must not be
restarted until the sanction has been administered.
This means that if the ball is in play, play continues, and he comes back to given the caution/sending off after play stops. (You see this all the time)
It means in this situation, that play can continue by the quick FK, if the fouled team elect to, if the ref never intended to give a card.