The speaker, the fictional Ulysses, is speaking to an implied audience in Tennyson's poem named after him. The poem is a dramatic monologue, although the silent listener is an implied audience--so there is more than one silent listener.
Ulysses says "You and I are old." The "you," here, is presumably plural. He may be addressing his former followers who returned home with him after his adventures, although this would alter the story as written by Homer in The Odyssey."
Ulysses also addresses his audience as "my friends." Thus, the reader knows he is addressing more than one person. The poetic dramatic monologue form was new at the time Tennyson wrote the poem, so the convention of only addressing one silent listener was not well established.
In short, then, Ulysses is talking a group of followers, old men as he is, into going on an adventure with him.