would Bilbo try to to rescue the dwarves if they were still inside the mountains in chapter 6? If so, what would his motivations be?
I believe if he had decided to save them he would have waited for some time, rather than charging back him. He would have figured with the help of Gandalf it only would have been a matter of time. However, he would have eventually found some way to save them or at least try by going invisible and checking for any distractions he could create. Later on when caught by the Wood Elves he spends days eating scraps and sleeping in corners with his ring on in the hopes of finding a way out for the dwarves. Whether their camaraderie had grown since that point I can't say, but Bilbo does have an innate sense of duty and would have felt an immeasurable amount of guilt at leaving behind his new friends, who were once guests in his home, and were counting on him. His motivations would not be selfish in saving them, he could survive well enough on his own albeit not comfortably, rather, by this point, he did feel like a part of the group and would have taken it very personally if he had not gone back to help them.
If the dwarves were in a position to rescue Bilbo, however, they would have gone back in, but for entirely other reasons. Without a burglar they would have had an even smaller chance at reclaiming their horde and the dwarves cared much more about their horde than their burglar. They would have attempted some sort of rescue out of sheer greed rather than for any personal reasons as by this point he had yet to truly wow them with his burglar-ing ability.