À rebours, its title being translated into English as "Against the Grain" or "Against Nature," is a product of its time. It was the Fin de siècle (end of the 19th century), marked by the symbolic aesthetic movement with which interesting artists are often associated, namely, Stéphane Mallarmé, Verlaine, Baudelaire, Moréas, and other (very eccentric) luminaries reacting against the styles of Naturalism and Realism. The way that they eccentrically depict richly "decorated" and highly symbolic scenes earned them the name "decadents" (usually pronounced in its original French).
Here is an excerpt from chapter 1 of the novel that points at decadent scenarios:
He won a great reputation as an eccentric [...] He had organized a funeral feast in celebration of the most unmentionable of minor personal calamities. The dining-room was hung with black and looked out on a strangely metamorphosed garden, the walks being strewn with charcoal,[...] and the ordinary shrubs superseded by cypresses and pines. The dinner itself was served on a black cloth, decorated with baskets of violets and scabiosae and illuminated by candelabra in which tall tapers flared.
This being said, Jori-Karl Huysmans (pronounced "yoreekarle hu-mahn") is also a huge representative of the decadent period; his novel marks him as a symbolic aesthete, as well. His writing reacted against the tendency of Naturalist authors to assume that the only way to present "real" life is by focusing on the ugly aspects of it.
Back to À rebours. Two salient symbols that are consistent and very strong in the story are the following:
1) Death espoused to excess: Jean des Esseintes's failing health condition
A young dandy of aristocratic background, des Esseintes is so eccentric that his tastes become awkward. He decides to retire and explore every decadent and aesthetic activity. The problem is that as he does his exploits (of a shady and unclear nature that leaves everything to the imagination), his senses become over-indulged by over-stimulation. Interestingly, the more he does, the worse his condition gets. The novel does not show this as a consequential thing, but throws it out there as a karmic coincidence. This is symbolic of the dangers of living by sensations and the danger of spoiling the soul through engaging in extreme behaviors.
His health is also symbolic of the sins of the rich which, in the end, are just as bad and detrimental as those of the poor and the debauched. His weak constitution indicates inbreeding, isolation, perverse tendencies, and an overall creepiness that can only be associated with terrible secrets upon which the novel touches, but does not reveal in their entirety. Des Esseintes's health breaks down to such a point that it reflects the karma coming back to get him to pay for all the things that he has done.
Does this not remind us a bit of Oscar Wilde's The Picture of Dorian Gray? It should. Dorian was written shortly after Oscar became obsessed with the novel À rebours.
2) The famous bejeweled turtle
This tortoise in chapter IV, which represents des Esseintes's insistence on making all things extreme, decadent, and with splendor, is literally made to live unnaturally for the sake of aesthetic beauty. Des Esseintes has apparently lost touch with reality and decides to make his "living companion" into an object:
Accordingly he resolved to have his turtle's back glazed over with gold [...] At first, Des Esseintes was enchanted with the effect; but he soon came to the conclusion that this gigantic jewel was only half finished, that it would not be really complete and perfect till it was encrusted with precious stones.
The turtle dies a slow death caused by the weight and the chemicals put on her shell. Des Esseintes touches it and notices that it is dead. This irony shows that no gold or jewel can replace time, health, or life. This is further symbolic of the shallow and apathetic nature of an obsessed man who, like the tortoise, will succumb to his own extremes.
Therefore the most symbolic thing in the novel is that fate plays out by allowing des Esseintes to engage in every imaginable act of debauchery and excess and then subtlety noting that death lurks quite closely, until it finally catches up with him.