How does modern art challenge realism and naturalism?
Modern artists challenge realism. I went over to the MoMA and just randomly pulled their current exhibitions, and I came up with Alexander Calder and Sigalit Landau, but I think you could use most any modern artists. Modern art tries to get people to see things out of context or in a new way. They don't hold themselves to representing either the real world (realism) or the pessimistic side of reality (naturalism). In the two examples I've linked to, the human form is shown in a way that you would never see it in real life. In Calder's case, he represents Josephine Baker, a very attractive woman, in wire as a series of three spirals focusing on her breasts and genitals. Certainly, he does have a point because Baker sexual appeal is incredible, even in a day when African American women weren't supposed to be seen as desirable in the mainstream. However, he makes no attempt to show her as she really is. And the second modern artist I randomly pulled from the MoMA, Sigalit Landau, shows the human form floating in a spiral of watermelons. This defies reality. However, most modern artists consider reality to fail at actually revealing anything important. The emphasize and reinterpret to get people to see something in a new way.
This is a broad question, so I'm not sure if you're speaking in reference to literature or 'art' as in paintings, etc. Realism and naturalism in literature involve authors writing literature about real-life events and happenings in the worlds of ordinary people and also in nature (and man's connection to nature). 'Modern art,' which again, is hard to define, would, in a very general sense, be contradictory to realism in that it perhaps is not old-fashioned or 'real' for everyday life, perhaps, and that it is too abstract to be natural.