The intended audience of Douglass's speech is white people, particularly those with power.
As he opens his fourth of July speech, he states that he is nervous, despite his experience at speaking. Why this is so comes clear as he gives the kind of Independence Day speech that few who are celebratory want to hear: he differentiates between the freedom being celebrated on this day and the slavery Black people still suffered under in the United States in the 1850s.
He specifically calls this day "your" holiday; in other words, a holiday that can only be enjoyed by those who are not enslaved. He asks why he, a Black man, who only through good fortune made it to freedom, is asked to speak on this holiday:
Do you mean, citizens, to mock me, by asking me to speak to-day?
Douglass uses the speech to make a plea to white people for freeing the slaves. He appeals specifically to the patriotic feelings stirred by the remembrance the fourth of July brings to whites of their courageous stand against British tyranny. He states that the same joy white people feel over the freedom they have achieved would be felt by Black people if they, too, could be freed.
Douglass knows he has to appeal to the emotions of white people to try to motivate them to support abolition more fully than many of them previously have.