Well the idea is to build a community right, so 3-4 to start off I think is a manageable amount of people, as for the quick joiners/quitters there's way to incorporate them into an overall story such as a one off encounter or like a random boss. there's way to let random pop in's to feel...
a space for people to set up games of dungeons and dragons.
like there'd be maybe 3-4 going on a week during scheduled times, and in the down spots a place for people to refine their plots or characters and stuff