The key word is "may." It is not clear that Calpurnia's ancestors were slaves, but it is a reasonable assumption. There are several details in the chapter that suggest a background of slavery.
First, when the children go to church, Scout describes the history of the church as a narrator. She says that the church was built by freed slaves. In other words, freed slaves used the the first fruit of their profits to build a church. Here is what the text says:
First Purchase African M.E. Church was in the Quarters outside the southern town limits, across the old sawmill tracks. It was an ancient paint-peeled frame building, the only church in Maycomb with a steeple and bell, called First Purchase because it was paid for from the first earnings of freed slaves.
This is the most direct evidence. Indirect evidence comes from the antagonism of Lula. She does not appreciate Calpurnia bringing white children. The poverty of the church as well might point to an oppressed background.
To Kill A Mockingbird