A literary convention is the pieces within a text that define it as part of a genre. For example, Romantic poets tended to write about the importance of nature, the individual, and imagination. A poem which includes these concepts and ideas could be considered a part of the Romantic movement.
That said, Edgar Allan Poe was an American Gothic writer. Stemming from the American Romantic movement, Poe's work depicted death, horrific images, insanity, and morbid passages.
In regards to "The Tell-Tale Heart," the literary conventions of the Gothic genre are very apparent. The narrator defines himself as being completely sane. Readers, on the other hand, tend to define him as insane (given his murder of an old man based upon a filmed over eye). The story involves insanity, death, horror, and morbid ideas. Therefore, the conventions of the American Gothic literature are readily seen within the text.