The title's taken from a poem by Derek Walcott. The relevant words are as follows:

You in the castle of your skin.

I the swineherd.

Walcott's poem invokes the traditional romantic trope of the noble beauty admired from afar by her humble would-be lover. The implication is that the lady isn't just separated from the swineherd by class but by skin.

What's significant about Lamming's slight reworking of the expression is that it is the narrator who's protected by the "castle" of his skin. For centuries in the narrator's Caribbean homeland black skin was on the wrong end of oppression, racism, and exploitation. Yet his situation is somewhat different, more nuanced. In his alternation between indigenous and colonialist culture, G faces two ways at once. G is part of that first generation in the West Indies for whom black skin is a source of strength, not weakness.

That being so, his black skin is no longer a sign of vulnerability to oppression, but a symbol of strength, fortitude, and impregnability, the kind of characteristics we'd normally associate with a castle.