This is an interesting question, as the answer clearly has to be both yes and no. Beowulf is the story of a great Anglo-Saxon hero who does battle, on behalf of a besieged king and town, against a marauding beast named Grendel. The very night he and his men arrive, Beowulf determines to fight his opponent in the same fashion Grendel has been fighting--barehanded and weaponless. Grendel's reign of terror has gone unchecked for years, and he obviously has no sense that he is about to meet his match in Beowulf. In that way, he is fearless when he enters the mead-hall. Another reason he is probably fearless even once he realizes Beowulf is his enemy is that he has wrought a charm (or spell) which protects him from all blades (swords). That is not particularly helpful to him since Beowulf has committed to weaponless combat; however, Grendel does not know that at the beginning of their battle.
Once he is locked in a death-grip with Beowulf, however, Grendel must certainly have experienced some fear. This is a man who has the strength of thirty men in one arm, and Grendel had to know he had met his match in Beowulf. Certainly that generated fear even in this fierce monster.
An interesting story written from Grendel's perspective is Grendel by John Gardner. I hesitate to recommend it, for it contains some quite offensive language; however, I'm certain you can find an excerpt if you'd like to read something more about it. I've attached a helpful eNotes summary, below, as well. Again, thanks for an interesting question.