Why does Gawain decline the Green Knight`s invitation to celebrate the new year together at the end of Sir Gawain and the Green Knight?
The challenge presented in Sir Gawain and the Green Knight is one of honor. When the Green Knight entered King Arthur's dining hall, he threw down the gauntlet, challenging the bravest among them to take an axe to his head in return for the same treatment a year from then. It's true Gawain was not the first to speak; however, he was valiant enough to be the first after his king had spoken.
When Gawain ended up in what turned out to be the Green Knight's home, Gawain was faithful to the bargain the two men struck--that each would faithfully share whatever they had received during the day. That is, Gawain was faithful in all but one detail--he did not give back the scarf he had received, a scarf (baldric) which had special powers. This was a moment of weakness for Gawain, for he put his hope in the magic cloth, knowing he was likely to lose his head without it.
The Green Knight did know about the withheld item, and the next day when the axe fell, it was just a nick. Gawain was spared because the Green knight found him to be an honorable man. Gawain, on the other hand, was a knight; and he knew what he had done was a display of great cowardice according to the knight's code of honor. (Which is why he went on to wear the baldric always, as a reminder that he must never again fall to such temptation.)
When he was invited to return, he was too ashamed of his dishonorable action to ever want to face the place or the people of his shame and dishonor again. Obviously no one else thought it shameful of him to want to save his own life, as the invitation was given and his fellow knights treated him well; however, Gawain set a high standard for himself and simply did not want the reminder and refused the invitation.