In his narrative, Douglass says that silent slaves made overseers and masters uncomfortable because a silent slave is a thinking slave. The overseers and masters did not want slaves to have active minds because then they might consider the hardship and injustice of their situation and make plans to revolt. Similarly, Douglass says that masters and overseers wanted slaves to be consumed by mindless leisure activities during periods of non-work (i.e. Christmas holidays) so that they wouldn't gain the desire to want for more.
I think it's because the slaveowner would not know what that "silent slave" was thinking. Was he or she plotting against him?