Bilbo gains self-confidence mainly through successes. Each time he manages to steal something or otherwise make a solid contributing to the group's endeavours, he becomes more confident in his own abilities. A good example of an episode in which this happens is the one with the trolls; another is his going into Smaug's lair.
Two other factors contribute to his confidence. The first is his acquisition of the Ring. The ability to turn invisible allows him to succeed and having it available to him as a last resort gives him confidence.
When Bilbo is worried about his comanions and motivated by friendship and a sense of responsibility is when he is most confident.