I think the best place to start would actually be at the end of the poem, which concludes the tale and also tells us of the impact of the Ancient Mariner's story on his audience, the Wedding Guest. We are told that the Ancient Mariner is plagued by the guilt of his actions in killing the albatross and is compelled by this guilt to travel from land to land, where he will find individuals whom he knows he must share his tale with:
Since then, at an uncertain hour,
That agony returns:
And till my ghastly tale is told,
This heart within me burns.
I pass, like night, from land to land;
I have strange power of speech;
That moment that his face I see,
I know the man that must hear me:
To him my tale I teach.
This clearly conveys the huge, massive sense of guilt that the Mariner feels as he is left to wander the world, sharing his tale with a carefully selected audience.
If you want other quotes, you will want to examine the part when the Mariner actually kills the albatross - plenty to look for there.