He's got control now and u let him. Fall back.
Do not be affected, do not respond. Some people just want to see if they can get a rise, and if you respond, they get rewarded. If not, you heap burning coals on their head.
If you don't deal with games, people will respect you and know that any behavior is towards you will not be tolerated.