Is it a red flag if he doesn't want to hang out at my place?

I'm 27yr. old male. I think he maybe he has known someone you live with or he just isn't a people person.