The fact to "The Scarlet Letter" is that while working in a customs house in Salem, Hawthorne discovered a scarlet letter A upstairs. This ornately embroidered letter was of such ancient artistry that even if the threads were removed to be studied, no one would be able to replace them:
But the object that most drew my attention, in the mysterious package, was a certain affair of fine red cloth, much worn and faded. There were traces about it of gold embroidery, which, however, was greatly frayed and defaced; so that none, or very little of the glitter was left. It had been wrought,...with wonderful skill of needlework; and the stich...gives evidence of a now forgotten art, now to be recovered even by the process of picking out the threads....This rag of scarlet cloth...on careful examination, assumed the shape of a letter. It was the capital A....While thus perplexed...I happened to place it on my breast. it seemed to me...that I experienced a sensation not altogether physical, yet almost so, as ofburning heat; and as if the letter were not of red cloth, but red-hot iron. I shuddered, and involuntarily let it fall upon the floor....
Prying further into the manuscript, Hawthorne finds the record of other doings and sufferings of "this singular woman," for most of which the reader is referred to the story entitled "The Scarlet Letter";
and it should be borne carefully in mind, that the main facts of the story are authorised and authenticated by the document of Mr. Surveyor Pue. The original papers, together with the scarlet letter itself--a most curious relic--are still in my possession."
If the imaginative faculty refused to act at such an hour, it might well be deemed a hopeless case. Moonlight, in a familiar room, falling so white upon the carpet , and showing all its figures so distinctly--...is a medium the most suitable for a romance-writer to get acquainted with his illusive guests.
Thus, Hawthorne constructed his narrative around cloth remnants and the bits of manuscript strewn upon the floor.