The primary context of the quote is how Stephen recognizes the spiritually limited condition of his being in the world. For Stephen, his visits to brothels and embrace of the sexual appetites have resulted in a particularly intense notion of shame. The sermonizing to which Stephen had been endured helped to initiate this condition of shame in which the quote settles. Yet, the ultimate shame is that Stephen reflects how Emma would view his actions. Hearing the laughter of a random girl reminds him of Emma. The shame he feels is an excessive indulgence that offends both the divine and Emma, the human apotheosis of his love.
It is in this context in which Stephen views himself as "mad" for having done what he did. The "cold sweat" that emerges is a reflection of the shame he felt, a reaction that is enhanced by the "sordid details of his orgies," memories that "stank under his very nostrils." "The agony of shame" is the context in which Stephen views himself as "mad."