Well, the crew certainly suffer many "difficulties" as you euphemistically put it after the Mariner kills the fated albatross with his crossbow. This causes the Polar Spirit, which symbolises the spirit of nature itself, to pursue the ship and suffering and punishment is visited upon the crew. Firstly, the breeze vanishes, effectively stranding the ship in the ocean without any hope of freedom or movement. Secondly, nature itself rebels against the crew, as the Mariner talks of how the "very deep did rot" and how "slimy things" crawled upon "the slimy sea."
Desperately thirsty, the crew see a ship draw near that only contains two passengers. These two specters, Death and Life-in-Death, play dice for the crew. All are one by Death and die except for the Mariner, who is won by Life-in-Death:
Four times fifty living men,
(And I heard nor sigh nor groan)
With heavy thump, a lifeless lump,
They dropped down one by one.
Thus it is that the rest of the crew meet their end after a time of immense trial and suffering thanks to the Mariner's action in killing the albatross.