In his soliloquy at the beginning of Act 3, Scene 1, Banquo expresses the fear that Macbeth has done something terrible in order to make the prophecy come true. He says, "Thou hast it now -- king, Cawdor, Glamis, all/ As the Weird Women promised, and I fear/ Thou played'st most foully for 't" (1.3.1-3). Suddenly, Macbeth has everything he was told he would have, and Banquo believes him to have been capable of murder in order to secure them. 

Given the fact that Macbeth did resort to regicide in order to become king, Banquo's fear seems to be extremely justified. Further, in that soliloquy, he wonders whether or not what the Weird Sisters told him would come true as well, and he is right to connect these two because Macbeth certainly does. Once Macbeth realizes that he has committed a grievous sin that will ultimately benefit Banquo's descendants and not his own, he resolves to have Banquo and Fleance murdered as well. If anything, then, I'd argue that Banquo was not quite concerned enough! He was right that Macbeth played "most foully" to get what he wanted, but he failed to realize that there might be more for him to want.