I believe that none of Hrothgar's men challenge Grendel because they are not brave enough. After all, Grendel is a hellish monster and most people would rather just get out of the way (move out of the mead hall) rather than have to face such a thing. Remember, this monster killed 30 men in one attack. That would be a pretty scary thing to try to challenge.
This is why Beowulf is such a hero. He is able to defeat a monster so fearsome that no one else is willing to challenge it.
If Grendel were something that just anyone could challenge, Beowulf would not really have been needed.
One of the reasons that none of Hrothgar's men challenge Grendel after Beowulf comes on the scene is that it is now Beowulf's destiny to 'face the monster.' It is almost as if the warriors know this, know that Beowulf will proudly step up to the plate to fulfil his destiny and respecting him for that, draw back. He has been called upon to ride to the rescue with courage and drive and the determination to succeed. The monster is hideously bloodthirsty, vengeful, vindictive and cruel - so Hrothgar's men are probably relieved to support Beowulf with their loyalty and moral support, but not to actually step into the fray. The story would not have lasted the test of time perhaps, if the action had been 'diluted' too much with lesser battles involving more characters.
Hrothgar's men never really have much of a chance to challenge Grendal. Grendal is very sneaky and hides in wait for the men. He is also stronger and more skilled at murder than normal mortal man. When the men become so frightened of Grendel that they feel safety would be in numbers, the people all move into the hall to stay together.
Grendel is still able to attack them and kill them. Fear has made them frozen because they have become powerless to stop Grendal. Some men may have tried to kill Grendal but
"how no warrior,
no matter how brave,
could kill Grendel."
none of Hrothgar men was able to challenge Grendel because Grendel is a devouring monster and as such hostile to people and an enemy to the human world, grendel is one of those monsters that is groomed naturally to be large and strong who do not require physical weapons to be struck down in the battle field. this could be seen in part one when the narrator mentioned thus: " but Beowulf, unarmed, wrestles with the monster, and tears his arm from the shoulder."
Hrothgar ,men lack the following qualities that a hero like Beowulf should portray in the battle field:
bravery, courage,determination, selflessness, sacrifice, and above all, the humility to stand with the people and not on top of the people. Hrothgar men could not have afforded to sacrifice their lives for the security of their nation and even if they did, they would have been wasted because it is not in them to do the fight. but we see Beowulf in part one readily and willingly sailing to Denmark to offer a help to the long ravaged state.
Hrothgar men lacked the necessary skills required of a hero like Beowulf to confront Grendel.
Beowulf is the Hero that sacrifices without hesitating. he is a genius.