In reality, the book to which Wilde actually refered is Joris-Karl Huysmans' novel A Rebours, translated into English as Against the...
Chapter X of Oscar Wilde's The Picture of Dorian Gray makes mention of the "yellow book" that was given to Dorian by his main tempter, Lord Henry Wooton.
In reality, the book to which Wilde actually refered is Joris-Karl Huysmans' novel A Rebours, translated into English as Against the Grain. This novel, without an actual central plot, focuses on its main character: an eccentric, rich, young Parisian named Jean Des Esseintes, whose convoluted family history and doubtful mental sanity gradually lead him to live a Hedonistic lifestyle that, ultimately, consumes him and his soul due to his desperate search for pleasure an "essence" of something that he could not find. The description of the main character in A Rebours points to two facts: a) that he is capable of committing just about any act of debauchery he pleases including the (at the time) "unmentionable crime" of sodomy and, b) that the search for the aesthetic ideal of beauty knows no limit, not even moral limits.
On to The Picture of Dorian Gray, notice how this "yellow" bound book (the motif of the color yellow permeates the novel) is described:
It was a novel without a plot, and with only one character, being, indeed, simply a psychological study of a certain young Parisian, who spent his life trying to realize in the nineteenth century all the passions and modes of thought that belonged to every century except his own..
It is clear that Wilde molded Dorian Gray after Des Esseintes. It is also clear that the debauched life of Des Esseintes resonates with Dorian's own personal, hidden, secret wants and needs. He slowly discovers, through Des Esseintes, that he too wants to spend his life trying to realize all the passions and modes of thoughts possible. Seeking more while wanting more. It is precisely what Dorian desires and the main reason why the book enthralls him and, based on its main topics "poisons" him.
...never, indeed, had any cause to know—that somewhat grotesque dread of mirrors, and polished metal surfaces, and still water, which came upon the young Parisian so early in his life, and was occasioned by the sudden decay of a beauty that had once, apparently, been so remarkable.
The poison from the text comes from Dorian's obvious sadistic character flaw; he revels in the misfortune of a main character who is so much like himself, but fails as a result of his eventual physical decay. Dorian basically reinforces the idea that he is untouchable, more fortunate, and worthy of much more praise than this man in the book that enthralls him so.