The significance surrounding Father Flynn's nails might tie into the larger symbolism of filth and cleanliness in the drama. In Act III. Father Flynn talks to the boys about hygiene. He relays the story of Timmy Mathisson, whose nails were the source of filth. This filth ended up killing him. Timmy's nails carried filth and impurities that led to his own demise. With this parable, one sees how Father Flynn's nails are symbolic. What filth does Father Flynn carry under his nails? Is this the type of filth that he passes on to others or ingests himself?
The larger issue that comes from the nails is the issue of doubt, itself. What is carried underneath all of the characters' "nails" can be a form of filth. Sister Aloysius lies to uphold what she knows as right. Mrs. Muller carries a similar type of filth in recognizing what society might say about her son and thus leaving him in the presence of someone who might be a child molester. Sister James carries the filth under her nails of having lost her joy of teaching and not being able to withstand the rule of Sister Aloysius. In the end, the symbolism of nails is reflective of what "filth" or even what "doubt" individuals carry with them no matter where they go and with whom they interact.