In a small town like Raveloe, there is always a place (aside from church) where people meet for fun and to share the gossip of town.
The Rainbow Inn was a form of pub (public house) where celebrations were held, and where people went to drink, gossip, and just meet up.
In Silas's mind, the Rainbow was an expensive resort where people went to spend money unnecessarily. However, we know that it may be just a regular meeting convention center where people divided themselves according to class. Hence, the poor and the not-poor would not be likely to share together.
We know that whenever a big celebration takes place, such as Ms. Osgood's birthday, it would be celebrated in the Rainbow Inn, which shows that some people in Raveloe used the place as a symbol of status, among other things, for others to see how much they can offer society.