There are several reasons why Beowulf wants to fight Grendel without the use of a sword. His famous "boasts" of bravery in battle is one reason-he wants people to talk about it like they have talked about his other conquests. For instance, he loves the fact that Wiglaf has talked about his fight with the dragon under water and the fact that his speech backfires on him. He also wants to prove it to himself that he can do it. I think there is part of him that wants to fight Grendel fairly. Grendel has no weapons, so he wants to meet him on his terms, as seen below:
"Of force in fight no feebler I count me,
in grim war-deeds, than Grendel deems him.
Not with the sword, then, to sleep of death
his life will I give, though it lie in my power.
No skill is his to strike against me,
my shield to hew though he hardy be,
bold in battle; we both, this night,
shall spurn the sword, if he seek me here,
unweaponed, for war. Let wisest God,
sacred Lord, on which side soever
doom decree as he deemeth right."
He wants to battle Grendel "unweaponed" because Grendel cannot help but "spurn the sword." The final reason is illustrated in the last line of this passage: "Let wisest God, sacred Lord, on which side soever doom decree as he deemeth right." Beowulf trusts on God to help the righteous winner win.