In today's society, for me, it would depend primarily on who accused me. Coming from a strongly religious background, if my husband or parents accused me of this, I would take it very seriously and go about disproving the statement because they are people I care about. If one of my kids approached me with the rumor, we'd have an intelligent and rational discussion as to why it wasn't true...and why someone might be tempted to make up such a thing. If my pastor expressed a concern, we would have a discussion, but she knows me rather well, and I feel I could depend on her support...as well as that of my close friends. I am such an open book that I think most who know me would chuckle at the idea.
If my family (and especially my husband) was supportive and believed me, I think I would probably do my best to ignore it. It is hard to fight rumors and innuendo. If people are going to lie, what can you do but live a life that speaks more to who you are than a rumor does?
However, if I lived during the Puritan era, when women were so harshly judged (if you kissed your husband in public, didn't go to church every Sunday, laughed during service, or God forbid, were accused of witchcraft without a second thought), my response would be, I'm sure, very different. There would be great fear. I would also (I hope) have enough support from my husband that we might sell the cow, pack up the house and kids, and move—joining a different kind of church as well!