Since Rousseau didn't publish his first work, New Heloise, until 1760 and Goethe finished what is called the Urfaust in 1775, following The Sorrows of Young Werther, it must be stated that any connection between the two would not correctly reflect influence. In other words, Rousseau cannot be assumed to have influenced Goethe's work even though Goethe only completed and published Faust Part I in 1808, and this he only did at friends urgent promptings. The reason is that, having begun Romanticism with Werther, Goethe soon renounced Romanticism, after having written the Urfaust, as a result of seeing the end of a young woman's suicide; as she was pulled out of a frozen river, Goethe saw a copy of Werther in her pocket. Thus he abandoned the Romanticist Faust until urged to complete it, which he did in a neo-classical style, not a Romantic style. Yet it may be possible to say that the two connect because Goethe countered the tenets of Romanticism in Faust Part I.
Having said this, some commonalities between Faust and Rousseau's ideas relate to (1) emotionalism and (2) nature.
1. At first, Faust tells Mephistopheles that he can't be tempted by human emotions. Then he switches to the position that he has exhausted academic study therefore will immerse himself in the passions that make people so earnestly moved and motivated. As a result of this switch, Faust demands Mephisto procure Margarete for him.
One of Rousseaus's prominent ideas is that people are good in their innate emotions and passions and that it is civilizing forces of society and uniformizing forces in education that corrupt these emotions and passions. Thus there is a direct connection between Rousseau's ideas and the sensations Faust pursues in Part I. In addition, this connection is dramatized by Margarete/Gretchen's ravings in the dungeon and her exchange with Faust there. The connection may not be viewed as a supportive one, however. Goethe is calling this sort of emotionalism and passion into question, especially as pursued by Faust (less so that natural emotion and passion demonstrated by Gretchen).
2. Another connection is in "Gloomy Day" in which Faust is in a nature setting after his seduction of Gretchen. Mephisto has orchestrated events so that Faust slays Gretchen's brother Valentine. As a result, Faust has to flee to escape arrest. Mephisto deliberately acted to leave Gretchen abandoned, pregnant and unmarried. Thus Faust is living contentedly in nature, knowing nothing of the harm and tragedy that is unfolding in the city--from Gretchen's pregnancy to her mother's death to, eventually, her baby's death. Here Goethe seems to be suggesting that while nature is restorative and peaceful, it is an escape from the realities of human life, which of necessity includes interactions and community.
One of Rousseau's ideas is that nature is good and simple while city life is corrupt and artificial. He believed that people should leave the towns and leave behind the laws and institutions and return to a state of living peacefully in nature. His idea that nature is kindly and good had a powerful effect on European and English Romanticism as is evidenced by its emphasis on the pastoral lifestyle. This idea connects to Faust because Faust is sent by Mephisto to nature to escape the institutions and laws of towns and to regenerate from his ordeal with Valentine. Yet the connection fails when Faust learns of Gretchen's plight.