If you choose to look at this scene from Macbeth's perspective, Macbeth's fear of Banquo is justified.
Examine the lines Banquo uses at the start of the scene:
Thou hast it now: king, Cawdor, Glamis, all,
As the weird women promised, and, I fear,
Thou play'dst most foully for't: yet it was said
It should not stand in thy posterity,
But that myself should be the root and father
Of many kings. If there come truth from them—
As upon thee, Macbeth, their speeches shine—
Why, by the verities on thee made good,
May they not be my oracles as well... (III.i.1-9)
Although in the script Shakespeare does not designate these lines as a character aside, Banquo is alone. In a sense, he is whispering this to himself; the lines can be heard by the audience, but they are not heard by the other characters.
At the end of lines two and three, Banquo implies (to the audience) that he suspects Macbeth killed King Duncan. Then at the end, Banquo seems sure that if the witches' prophesies came true for Macbeth should they not come true for Banquo, too.
It is interesting that Shakespeare chose not to reveal Banquo's thoughts in an aside. He may have done this to show that Banquo is much more transparent character than Macbeth.
When you are examining quotes by Shakespeare... ask yourself a few questions. What does this quote say? (Here, you rewrite Shakespeare's language in today's language.) What does the quote mean? (Here, you analyze the quote.) Why does this quote matter? (Here, you think about why the quote is important in the play.)