It's an interesting question, and I think it would be a mistake to view the answer in terms of an either-or choice. Keep in mind, Odysseus's return homeward is a journey that unravels across an extraordinary span of time, and this introduces an element of variability into Odysseus's psychology. He's...
It's an interesting question, and I think it would be a mistake to view the answer in terms of an either-or choice. Keep in mind, Odysseus's return homeward is a journey that unravels across an extraordinary span of time, and this introduces an element of variability into Odysseus's psychology. He's not possessed of a single state of mind across the entire story, through which we can simplify his motivations and intentions. Things are more complicated than that, and in fact, these two motivations (the desire for glory and the desire to return home) tend to be weighed in conversation with one another. They are both important, and at different points in different circumstances, one can outweigh the other.
I would suggest that, ultimately, all told, his main desire is to return to Ithaca and his family—that's what is ultimately driving him. That being said, what we have to understand is that for Greek heroes, they are largely defined in terms of their achievement, and Odysseus is no different. The desire for personal glory is more than a desire, it's a large part of his psychological makeup. Greek heroes are larger than life, superhuman figures, and Odysseus is very much understood as heroic within that Greek context.
So regardless of his larger goals, when the opportunity for personal glory arises, he aims to seize it. That's part of who he is, even when the action is destructive to his larger purpose. We see this in his encounter with Polyphemus, for example, or also when he decides he to listen to the song of the Sirens. This does not make his desire for home any less significant (especially taken as a long-term motivator), but when the opportunity for personal glory or achievement arises, his heroic character tends to rise to the forefront. I might even suggest that this is a tension which carries across the narrative, and is central to his own characterization.
On the other hand, it should be noted that Odysseus's desire to return home is itself not entirely free of fluctuation. We see moments of profound homesickness and longing (one of the most notable examples of this is when we first meet him, on the island of Calypso. One can also point towards his landing at Ithaca, and the despair he feels when he thinks he has come to yet another strange land). Yet at the same time, be aware that even here, he does lapse on occasion. An episode that tends to stand out lies is in his time spent with Circe, which he describes in the following manner:
"So she enticed / and won our battle-hardened spirits over. / And there we sat at ease, / day in, day out, til a year had run its course, / feasting on sides of meat and drafts of heady wine . . . " (The Odyssey, trans. Robert Fagles, Penguin Classics, New York: 1996, paperback ed., p. 245)
From this perspective, I don't think its a case of one or the other (though if I were to choose, I'd be inclined to side with homesickness and the desire to return home). Still, I'd rather view it in terms of a continuum, with Odysseus fluctuating between those poles, throughout the course of this epic poem.