Why do people act deeply offended when I take their picture with my small pocket camera that I use to snap shots throughout the day?

Photography is a favorite hobby of mine & I always take pictures throughout the day with a small camera that I carry around in my pocket. (not one of those major ones, but a cheap small ones). FOR EXAMPLE , just yesterday I had hit a gas line accidently while digging up a tree stump and was taking pictures of the scene when the police and fire men showed up and they were all standing around looking at the area I cut and I was taking pictures and included them in one of the main shots and after I looked at the pictures and they were looking at me with just pure hatred nearly (or like I was deserving to be beaten to a pulp for my rudeness and audacity), but why? I know above is an extreme case, and I guess they maybe thought that I was like mocking them maybe ? (since I had cut a gas line and they had all run out there and I was doing it for a laugh maybe? or I don't know)...but even in normal cases...where I just am taking pictures out in public, AND NOT EVEN REALLY TAKING SOMEONES PICTURE, (they just happen to be in the scene I am taking a picture of)...they will look at me with a hateful look too, like I was just unbelievably rude)