Though there is significant contradiction in some of the images and symbols used in the poem, the overwhelming religious feeling is one of Christian origin, and in some ways focused on an Old Testament style god that exacts vengeance on sinners through painful and powerful retribution for past actions.
When the mariner kills the albatross, he feels shame and guilt for his actions, but this is only magnified through the tortures he faces of starvation and thirst and the visitation of dark and forbidding spirits from another world.
Some consider his sin of killing the albatross to be akin to Adam's original sin which gets him cast out of the garden and the mariner cast out of the calm and regular world of the seafarer. Only through repentance and time can he return to what he knew before.
There are also specific symbols that some point to such as the rain being a form of baptism that washes away his sin and allows him to return to the presence of god.