Relationship question?

My boyfriend and I live together, so we see each other often. He always makes a huge deal if I want to go hang with my friends, or do something without him. I suggested that we go to Michigan to gamble (because in Wisconsin you have to be 21, and I’m only19, he’s 22.) he always declines and tells me to save my money. I’ve gone gambling before with my parents, and I did fine. I know my limit. But now he wants to go gambling with his friends, but I’ll be babysitting so I’ll be gone when he leaves, and he’ll be gone once he comes back. I don’t have an issue with him hanging out with his friends, but it just sucks that he wants to go gambling without me. Is it wrong of me to be upset about this?