James Joyce's brother Stanislaus wrote in his memoirs that his relationship with an older woman was the inspiration for this story. The eNotes study guide tells us that James Duffy was very much like Stanilaus:

James Duffy does share aspects of Stanislaus’ character (the collecting of little quotations) as well of those of the author (the penchant for Nietzsche and the distrust of middle-class intellectuals).

In his book James Joyce A to Z, A. Nicholas notes that Joyce used an almost word-for-word quotation from his brother's diary as a journal entry for Duffy:

Love between man and man is impossible because there must not be sexual intercourse and friendship between man and woman is impossible because there must be sexual intercourse.

Stanislaus Joyce wrote a book entitled My Brother's Keeper: James Joyce's Early Years, in which he discussed life with his brother.