The reality is most players, coaches and spectators actual knowledge of the laws is poor - (my personal belief is that it is more often that not because of the rubbish they hear from tv commentators.)
At least, that's what i tell myself if i am running a line. I am lucky that my mental attitude is such that the rubbish that is said from behind me is water off my back; i know that many other refs struggle with this.
In the middle, it's easy to manage the game/players via a quiet word and then stepped approach, and i mentally block out the spectators tbh.
The people that get to me piss me off is the parents in youth games. (and one player, actually, who used the "C" word* against me last season). That tends to make me angry, and stop enjoying what i am doing primarily for fun (& fitness).
Parents, Your kids may or may not be football gods in 10 years, but right now they need to know that their success will come from hard work, and trying to blame the ref for their mistakes, will not help their development. Also, stop trying to live your dreams through your children. No one will ask them if they won a particular game or not when they were 13; it's not relevant to their future success.
* Cheat, not c*nt.