Depending on which country you mean, the answer could vary a bit, but the theme is the same. Cricket, polo, tennis and golf are commonly played sports among the elite in English speaking countries, and among the Ivy League universities in the United States. Golf in the US in particular was played at clubs that, since they were private, could exclude people on the basis of race, religion or class.
For people in poverty, Football (soccer) is still very common in the greater Colonial World in Africa, the Middle East and Latin America. Baseball in the US was often stickball, because all you needed was a stick and a ball of twine, with coats or shirts for bases. You could play it in the streets of major cities or in the rural countryside of Kansas or Texas. This is how it became the national pastime in America.
Womens' sports in the US came much later on an organized level, really only coming into their own in the 1970s and 80s, after Supreme Court rulings and government regulations opened up women's sports at the public school and university levels.
Some sports are exclusive to the wealthy for the obvious reason that they involve great expense. Sports such as sailing, polo, and horseracing began with gentlemen and continue to be sports of the well-to-do. So also did golf and tennis as most of the time a person had to belong to a club in order to play. Like most sports, they began with what possessions people had. Since the wealthy had their sailboats and yachts, and so many men are competitive by nature, racing one another on the water and on the land with their horses naturally evolved into a sport just as rodeos evolved from cowboys competing with one another at get-togethers, etc.
Other sports, such as baseball and swimming could be participated in by the middle and lower classes since all one needed was an empty lot or the pool at the local Y.M.C.A., begun in the late 1800s. Here is where, of course, Naismith created his game of basketball. Once golf courses were opened to the public, golf became a game available to the middle-class as did tennis which could be played at a municipal facility. With the cost of many things being reduced by mass production, other sports have extended themselves into the middle class; sports such as archery, skiing, water skiing, etc. are now participated in by middle-America.