People say weed just "isn't for everyone". Why?

I've smoked several times and I don't get the big deal. It's actually really painful for me. I get angry and annoyed and frustrated and restless. I want to know the medical reason why. I know my brain is rejecting it because it doesn't recognize it, but why do some people have such a high tolerance and why does it hurt so much.