What can and should be the limitations to free speech?

The US constitution has certain categories of speech that are not protected by the first amendment. Things like incitement to imminent violence and defamation are exceptions to the first amendment—but so-called “hate speech” generally is not. Is there any way to find consistency in these exceptions? Why should neo-nazis have the right to spew their racist theories while it’s illegal to damage someone’s reputation? If there are exceptions to free speech, should it really only be a group of nine judges elected for life who can decide what those exceptions should be and how they are to be applied? What does it mean for our future if students on universities cannot be allowed to protest genocide?