Beowulf, as an epic, is distinguished from Greek epics like the Iliad or Odyssey chiefly in the scope of the action, the role of the gods in the story, and the hero's relationship to society.
First, the world of Beowulf is much more constrained than that of the Homeric epics. Whereas Odysseus travels throughout the Greek world, the action in Beowulf is concentrated in Hrothgar's kingdom. This gives Beowulf a more insular, even intimate, tone. This is true also of the fighting; combat in Beowulf is chiefly between the hero and a monster; there is nothing like the epic battle sequences in the Iliad.
The gods play a central role in the Homeric epics. In the Iliad, the Trojan War is fought as much between the gods as it is the Greeks and Trojans; there are direct relationships between gods and men, and the gods influence the outcome of the war in many ways. Beowulf is very different on this point. The poem's later Christianization aside, Beowulf acts independently of any god; his strength comes from himself, and his heroism is personal. Beowulf's defeat of Grendel and his mother are the result of his prowess as a warrior, not because of any intervention by the gods.
Finally, Beowulf—the character and the poem—represent a society very different from that of the Greeks. Beowulf fights for glory: his aim, in a sense, is to become the hero of a great poem. The society he belongs to is much more fragile and fleeting than that of the Greeks. The sense at the end of Beowulf is that, with the death of the great hero, his people will be swept away by their many enemies. This gives Beowulf a tragic element that Homer lacks.