I think that when it was first developed, the dream was seen as an example of innocence. Holden sought to be with children because they did not represent the phoniness and duplicity of adults, or the adult world that Holden encounters. Additionally, I think that the children and the desire to be with them is an extension of Holden's anti- establishment position. Given the fact how Holden sees himself, "catching" these children off the edge of the cliff, he sees his own role as one that protects the children from the corruptibility of maturation and preserving their own state of being in the world. I do not think that there is much in his vision which was intended to be inappropriate or perverse. Yet, as our society has changed, and as there has been greater revelation about the inappropriate contact between many adults and children in different forms and the sexual abuse that has resulted from this, seeing his vision in this light does create a feeling of uneasiness. Consider, "Thousands of little kids, and nobody's around—nobody big, I mean—except me" as a line that indicates Holden's relationship to these children. This helps to conjure images of individuals who use their power and control over children to ensure that their own needs can be satisfied. The lack of institutional checks or controls in this statement, when seen in the modern context of adult/ child victimization, is startling. The fact that this vision is all that Holden "wants to be" is also something that the modern vision would see as inappropriate as it reflects a dangerous blurring of lines between maturation and the socially acceptable way in which adults need to associate with children. At the time, I think that the vision was seen in one way. Yet, the passage of time and the appropriation of different social values and norms have resulted in this dream being seen in a different light.
Akannan's answer is excellent. Just to add, there is no suggestion of a sexual aspect of Holden's catcher in the rye fantasy. It is true, as Akannan says, that seen through relevatively modern awareness of the presence of paedophilia in society, a perverse slant might be given to it. However, I would argue that this is unjustified on the basis of the book alone.
In fact, it is arguable, that Holden's fantasy of catching the children in the rye is a wish to be freed of sexual desire. The phrase "catcher in the rye" is Holden's misunderstanding of Robert Burns poem "Comin Thro the Rye" which is sexual in that it is about lovers kissing in the rye.
However, the fact that Holden mishears the poem's title and gives it an asexual interpretation, it can be argued, that it suggests that Holden's fantasy seeks to escape the sexuality associated with maturation.