Brutus does not want to swear an oath because this would lessen the nobility of their endeavor.  As honorable and honest men—good Romans all, they are pledging themselves to commit an honorable deed by killing Caesar; an oath would suggest that they lack nobility, honest, and courage. He says “Not an oath. / If not the face of men, /The sufferance of or souls, the time’s abuse,/ If these motives weak, break off betimes, And every man hence to his idle bed.” If their cause is not just in itself, then they shouldn’t kill Caesar to begin with. And then he adds, “What other bond / than secret Romans that have spoke the word and will not palter” (2.1.125-137). We can infer that Casca is a bit "dull," because Cassius tells us he is back in Act 1 and we see him only agreeing in Act 2.1 (52-55). We can infer that Cassius is a very clever politician because he realizes the danger Antony presents and wants to kill him too 9169-174). We know Brutus is noble but not a savvy politician because he refuses to kill Antony, and this decision results in great trouble as the play proceeds.

They don't need an oath because they're already in it together.  Brutus is saying that this is an important enough discussion without an oath.  They are already tied together to this plan simply because they are here talking about it.  It would be virtually impossible for any of them to nobly back out at this point.  Brutus is saying that an oath is just words; their actions are what will truely bind them together.

This is the first scene where you really see how important Brutus is to the rest of them.  Every decision that needs to be made is made by Brutus, regardless of everyone else's feelings about it.  Two of those decisions turn out very badly for the conspirators, particulary not killing Anthony.