Here is a quick explanation of the symbolism behind each monster. For a quick answer, I would say there is great deal of connection between Grendel and his mother, as they are both part of the classic story of hero versus evil. The dragon almost tells a different story entirely, the story of a great man meeting his unavoidable end.
According the the writer and theorist, John Gardner, who wrote an alternative version of this story which focuses on Grendel, humans create monsters because of something dark and lurking we see in ourselves, something horrible yet seductive. Grendel is perhaps the perfect example of this idea. He is descended from Cain, one of the first humans in the Bible. Cain becomes the first murderer when he kills his own brother, Abel. This reinforces the notion that Grendel represents a darker side of our human nature. We see this by the explanation of his hatred for hearing sounds of joy in the hall. He is jealous of other's happiness, a common human weakness. He represents the same human greed, anger, and jealousy that we fear in ourselves.
Joseph Campbell, an important scholar, explains that all heroes are essentially the same at the core. They all undergo the same structure of story; it is a tale that is part of our human nature. A crucial piece of this structure is what is often called "the belly of the whale." In this portion of the hero's journey, he or she goes to an extremely low point to face the greatest challenge. When the hero goes into the belly of the whale (sometimes the underworld, sometimes an actually whale's belly, sometimes death) he/she returns a true hero and his task is complete.
This story is somewhat unusual in that it continues after his success here, but after defeating Grendel's mother, it is unquestionable that he has now proven himself as the hero of the story. The battle with Grendel's mother represents the key moment where our hero goes to a dark place alone and emerges as the glorious victor. He wins glory, fame, and eventually even a kingdom. Grendel's mother represents the sinister challenge between victory and defeat.
It is a well understood symbol in both literature and religion that both serpents and dragons represent mortality and death. The perfect example of this comes in the garden of Eden, when it is the serpent that convinces Eve and Adam to eat the fruit and thereby end the immortality of humankind. By the time our hero encounters the Dragon he is an old man. Despite his amazing strength, he is still a mortal, and he still must face death. Even he cannot survive his encounter with the dragon. This shows that no amount of strength can help a person escape death.