Absolutely. They were definitely not talking about anyone other than white males. In fact, most of them were not really even talking about all white males over the age of 21.
The era in which the Constitution was written was much more conservative than today. Essentially no one believed that women were intelligent enough and stable enough to vote. We know that John Adams pretty much laughed his wife off when she asked for women's rights Essentially no one believed that non-whites were equal to whites. Furthermore, most people believed that only white men with some money ought to be allowed to vote. There were property requirements for voting in the various states in the time right after the Constitution was written.
So, the men at the Constitutional Convention certainly were talking only about white men (and only white men with property) when they talked about citizens voting.