"For Anne Gregory" actually has two speakers, and the poem is framed as a dialogue.  It doesn't matter significantly who the speaker of the first and second stanzas is--it could be Anne Gregory's lover, a friend, or a relative.

The subject of the poem is what's compelling--whether or not beauty is perceived internally or externally.  The speaker in the first stanza, for example, tells Anne Gregory that her "great honey-coloured" hair essentially guarantees that a young man will never love "you for yourself alone."  In other words, the beauty of the hair is so distracting that the prospective lover will look no farther--certainly not into the personality or soul of Anne Gregory.

Gregory's response in the secton stanza, not unexpected for someone who has a sense of self-worth, is that she can dye her hair any color--"brown, or black, or carrot"--and men will no longer be dazzled by her "yellow hair" and love her inner being rather than something as superficial as beautiful hair.

Her opponent in this discussion, in stanza three, does what a good logical person would do--he or she appeals to authority by saying that a religious sage "had found a text to prove that only God" could love her for herself "and not your yellow hair."

If we take this dialogue as one between lovers, then the comment at the end is the ultimate statement on Anne Gregory's beautiful hair, which is so beautiful that only God has the ability to ignore its beauty and look inside the individual.  Men, without God's strength, simply can't look beyond such beauty.