There is really little about the definition of a "protagonist" within the literary context that suggests that Washington Irving's protagonists are particularly unconventional. Certainly, the most commonly-understood definition or meaning of the word suggests a more positive image than one finds in Tom Walker, the protagonist in Irving's short story The Devil and Tom Walker. "Protagonist," however, merely, in general, refers to the main character in a story, irrespective of that individual's merits as a human being. Such is the case with Tom Walker. Irving's narrative provides ample evidence of the improbability of his main character emerging as anything remotely resembling heroic. As the following two quotes from the story demonstrate, Tom Walker is a seriously down-on-his-luck character who lives a bitter and economically-destitute existence:

"He had a wife as miserly as himself; they were so miserly that they even conspired to cheat each other."

"Tom's wife was a tall termagant, fierce of temper, loud of tongue, and strong of arm. Her voice was often heard in wordy warfare with her husband; and his face sometimes showed signs that their conflicts were not confined to words."

Taking into account the early-19th century period in which Washington Irving lived and worked, the suggestion that Tom Walker was the victim of domestic abuse evident in the second of the above two quotes does serve to further diminish his stature, mentally and emotionally, if not physically. 

Do Irving's descriptions of Tom Walker suggest the latter is unconventional? To the extent that the common conception of a protagonist is of a human of considerably greater stature, then the answer is 'yes.' Tom is a pathetic figure who casts his lot with Satan and whose marriage can most charitably be described as awful, so, yes, he is unconventional. If one subscribes, however, to the notion of a protagonist as merely the main character in a story, then the answer is 'no.' Tom Walker is the main character in the story that includes his name in the title. In that sense, he is a conventional protagonist.