Literary conventions are abundant in any complex, long poem, including Tennyson's "The Lady of Shalott." In fact, if you really wanted to list every convention in the poem, the list would be pages long. Literary conventions include about everything a poet does.
The poem alludes to Lancelot and Camelot, and these allusions represent a romantic tendency to idealize the past and see it as a fertile topic for poetry. It also uses knights, a woman stranded in a tower in need of rescue, and a character portrayed as an artist. The Lady uses a loom to weave, which is another allusion, this time to classical myths about the fates weaving the destinies of humans. It uses symbols, iambic tetrameter, rhyme: even the use of stanzas is a literary convention. Again, the list is lengthy.
Comparing literary conventions to conventions in the visual arts is difficult, however. At least it is for me. They are two different art forms, and I'm not an expert on paintings, so more similarities may exist than I can give you. I'll give you what I know, though.
In the poem, symbols of the tower and of Shalott are used to demonstrate contrasts between Shalott and Camelot. These symbols dominate the poem, by revealing the Lady's curse of experiencing life only as art. She sees life only through the magical mirror, and this becomes unsatisfying. Camelot is life, but it only passes by her. She is isolated. She cannot experience love and cannot be with Lancelot.
The painting uses symbols as well: the Lady is in white (she is pure, virginal, she hasn't lived), her hand holds a chain (she is cursed by fate), candles (of life) burn on her boat. She is isolated in the boat, as she is in the poem. She is a beautiful, suffering maiden in the painting.
These are all conventions. Both works are romantic in nature, both look backward to an idealized past, both take the reader/viewer back to Camelot.