This is a good question. Moreover, this is becoming a more common question of late, because of the book by Dan Brown, Da Vinci Code. The hollywood block buster also has a role to play. The premise of the book is that Jesus had a family and was married to Mary.
This is pure fiction. There is no evidence of this and tradition does not support this as well. Bart Erhman, a professor at the University of North Carolina has even written a book on this topic, because so many people have been misled by Dan Brown. No serious scholar believes that Jesus was married. In addition, the best available sources do not state that Jesus was married as well. In light of this, this question is something that is purely modern and cannot be sustain with any serious thought.