There's plenty to say about the symbolism of Father Flynn's nails in Doubt. We might want to focus on Flynn's speech in the gym. The location could give us a hint that when Flynn is talking about nails, he might also be alluding to something else, something perhaps more physical.
It's also interesting to note that nails come up in connection to foul shots. What is a foul shot? If you're trying to score in basketball and a player hits you or touches you in a way that’s against the rules, you get to go to the free-throw line and shoot two (sometimes three) shots. We might see the transgression in basketball as a symbol of Flynn's own possible transgressions.
When talking about nails specifically, Flynn does seem to allow for a little transgression. He says,
They’re long, I like them a little long, but look at how clean they are. That makes it okay.
Here, we note how doing one thing (keeping your nails clean) can make doing another thing (growing them a little long) permissible. What does this symbolize? This might cause us to wonder how Flynn could use his supposed purity as a priest to justify illicit behavior like pedophilia.
Flynn castigates one student for having "filthy paws." He says,
You try to talk to a girl with those filthy paws, Mr. Conroy, she's gonna take off like she's being chased by the Red Chinese!
We could also say that Flynn's nails symbolize his identification with the boys. He uses juvenile-like language ("filthy paws"), and he makes fun of them as if he's their friend. Knowing what we do, we could be suspicious about how Flynn's friendly demeanor might symbolize something deviant.
Later on, when he’s ordered to cut his nails, is he not also being ordered to cut out his predatory behavior?