Frederick Douglass would not have known for sure how old he was when he was sold because he would not have had any education and because his age would not have been emphasized.
We always now how old we are because our society emphasizes it and because we are educated enough to always be aware of what year it is. By contrast, slaves would not have had either of these things be true. Slave masters would not have made big deals about their slaves' birthdays. The slaves would not have had calendars. So even the parents would not have known exactly when their kids were born.
Even my grandparents (from the Philippines) did not know their birthdates for sure. They were told that the were born around some particular time on the Church calendar, but not the exact date. They were not told the year because that wasn't really seen as all that important.
So when your society doesn't really think your age is important, and you and your family are not educated, how are you going to know how old you are?
During Fredrick Douglas' time he was registered as property and the only place the documentation was maintained was in the owner's records. Birthdays often went uncelebrated by slaves. Their days were harsh and the event of a child's birth was not celebrated with joy. Having a child meant that the child was also going to be a servant to a master with no freedom.
Time was measured by the slaves but in a different way. Events such as weddings were celebrated, but often ended sadly when a partner was sold away.