The answer to this question can be found at the very end of Part I and is reinforced at the beginning of Part II. At the end of Part I of this excellent ballad, the Ancient Mariner tells the Wedding Guest that he "shot the albatross," which was the catalyst that triggered off his subsequent disasters. Consider the following stanzas from the beginning of Part II and the way that they immediately indicate that something has gone very wrong indeed:
And I had done a hellish thing,
And it would work 'em woe:
For all averred, I had killed the bird
That made the breeze to blow.
Ah wretch! said they, the bird to slay,
That made the breeze to blow!
It is therefore the Mariner's act of unthinkingly killing the albatross that causes the deadly predicament that both he and his fellow sailors face. It is this act that has so offended the "polar spirit" that nature itself, personified in this form, takes his revenge on not just the Mariner, but all of his fellow sailors as well.