In his autobiographical novel, Black Boy, Richard Wright works at a department store in the town he lives. When white people walk in front of the store where Richard and another young man work, Richard looks at them, meeting their eyes with his. After they depart, his friend warns him not to look people in the eye, but to lower his eyes and smile. He explains that the white people do not like them to confront them visually; instead, they must look down and smile and act happy.
Paul Laurence Dunbar's metaphor of "wearing the mask" is an expression of just this pretence of which Wright's friend speaks. Blacks must pretend to be satisfied with their lower status in society.
WE wear the mask that grins and lies,
It hides our cheeks and shades our eyes,—
This debt we pay to human guile;
With torn and bleeding hearts we smile,
And mouth with myriad subtleties.
But, they protest with "subtleties," acts in which they obtain some slight revenge upon their oppressors. Another example from Black Boy is a fight that Richard and another boy are supposed to have for money. But, Richard explains that the white men who pay them to fight just want them to beat each other for their enjoyment; they are merely exploiting them. So, the two young men refuse to fight, a "subtlety" of protest.
I agree with the previous post. Dunbar is trying to convey the idea that African Americans of his day could not show whites their true faces (and perhaps that they did not want to). This was a time when African Americans were seen very much as inferior to whites and were expected to be deferential to whites, particularly in the South. It would not have done for an African American to show their true face, particularly if that face were proud or defiant.
I think Dunbar is also saying that African Americans do not want to let whites see their real faces. I think he is saying that they do not want to let whites see that they are hurt by racist attitudes.
He's trying to tell us that those in his community don't always disclose what they really think, feel, or believe. Dunbar was speaking of formerly enslaved African people and was suggesting that "the mask the grins and lies" is not the true face of the people. Rather, it is a protective mask against the turmoils and dangers of living in such an oppressive and racially charged environment.