There are two striking instances of the use of the phrase, 'black flower' in this novel. The first is in chapter 1, when the narrator refers to the prison where Hester has been incarcerated, as 'the black flower of civilised society'. Later, in chapter 14, when Chillingworth and Hester meet each other alone by chance, and she pleads with him to abandon his harrassment of Dimmesdale, he refuses to do so, saying 'Let the black flower blossom as it may'.
In both cases, the black flower's most obvious function is as a symbol of retributive punishment - both that which is inflicted publicly by society and privately by individuals. In either example, this symbol appears in a wholly negative light. The prison is a blot upon society, and appears wholly bleak and depressing. Chillingworth, in his hounding of Dimmesdale, often becomes the very personification of vengeance and it renders him a dark, bitter, twisted figure, with virtually no redeeming qualities.
It should be noted however, that in his private meeting with Hester in Chapter 14, although he still appears overwhelmingly grim, Chillingworth also expresses recognition that Hester is in herself not necessarily evil, although her actions have been.
Ye that have wronged me are not sinful, save in a kind of typical illusion; neither am I fiend-like, who have snatched a fiend's office from his hands. It is our fate.
Chillingworth, then, does not condemn Hester outright here, but he is unable to see past her sin, the consequences of which, in his view, can never be eradicated. In a fatalistic way, he succumbs to wholly negative emotions. He cannot allow himself to deviate from the role of vengeance that he has set himself, and this is his tragedy as it makes him so bitter and unloving and ultimately consumes him, as well as inflicting misery on others.