In the novel Grendel, would "firelight flickering on the walls" be considered the use of kenning or alliteration?
"Firelight flickering on the walls," would certainly be alliteration as opposed to a kenning.
You can check the enotes Guide to Literary Terms in the Literature Guides section to get these definitions. A kenning, in short, is a compound metaphor that renames. A writer puts two words together to rename something--bonehouse for body or whales' home for the sea. It can be a little more complicated that that, but that's, for the most part, what a kenning is. It was popular among Anglo-Saxon poets but not prevalent in other literary periods. So much so, that the use of kennings in literature not from the Anglo-Saxon period can often be interpreted as an allusion to Anglo-Saxon literature. As you can see, "firelight flickering on the walls" does not rename.
What the phrase does do is use the sound device alliteration, which is usually taught as the repetition of initial consonant sounds in neighboring stressed/accented words: in this case the "f" sound in firelight and flickering. The device creates unity, of course (when a reader hears the second "f" sound he/she is reminded of the first "f" sound), and is therefore pleasant to our ears, but in Old English poetry it served another purpose as well. Since Anglo-Saxon poetry was oral literature, alliteration helped reciters memorize the lines. Once a presenter remembers the "f" in firelight, for instance, he would be aware that there's most likely at least one other word in the line that also starts with "f".
John Gardner in Grendel, of course, is writing as an Old English narrator, the monster Grendel. Thus, Grendel throws in the occasional line using alliteration.