Frederick Douglass would not have known for sure how old he was when he was sold because he would not have had any education and because his age would not have been emphasized.
We always now how old we are because our society emphasizes it and because we are educated enough to always be aware of what year it is. By contrast, slaves would not have had either of these things be true. Slave masters would not have made big deals about their slaves' birthdays. The slaves would not have had calendars. So even the parents would not have known exactly when their kids were born.
Even my grandparents (from the Philippines) did not know their birthdates for sure. They were told that the were born around some particular time on the Church calendar, but not the exact date. They were not told the year because that wasn't really seen as all that important.
So when your society doesn't really think your age is important, and you and your family are not educated, how are you going to know how old you are?