Can roommate enter room without permission?

I have a verbal agreement with my roommate for renting a master bedroom with private bath. I pay my rent on time as agreed upon. Since I have started renting the room, not only has he increased the rent by $120 a month, he has also moved in his girlfriend, her daughter and constantly has a houseful of people. There is only one other bathroom in the house and he has told me that any time they feel the need to, whoever wants to, will come into my room to use my bathroom. I was under the impression that if I am paying rent on a room, that no one is allowed to come into my room without further notice. I keep my door closed at all times and it never happens when I am home. Only when I am gone. Please help as I am feeling very invaded by my roommate.