The word "lisp" used as a noun usually denotes a speech imperfection in which sibilant /s/ or /z/ is mispronounced, sounding more like a /th/ phoneme. However, when used as the verb "to lisp" it also denotes childish speech, especially in early literature in which it is a standard convention of a toddler's or young child's imperfect speech.
The presence of Poe's use of "lisp" in the first stanza of "Romance" reinforces his description of himself as a young child. Since Poe is speaking of himself as a child, it makes sense that a "paroquet" (parakeet) was a familiar friend and "Taught me my alphabet to say —". Paroquets are small parrots and thus are attributed with the skill of mimicking speech. This image of a boy and bird and time to learn the alphabet stands in sharp contrast to the primary thematic image of a grown up who now says: "That little time with lyre and rhyme / To while away — forbidden things!"