There is a sense in which this part of the play shows how naive and innocent Brutus is. He is presented as a man who is manipulated by Cassius and the other conspirators and used, and this is just another piece of evidence that can be used to indicate that goodness and sense of honour that is so easily subverted by others. His belief that an oath is unnecessary indicates his belief in himself and in others.
Brutus is against the idea of an oath because it solidifies that they need to take an oath to stay true to one another. He believes that if they are doing the right thing for the right reasons, there is no need to pledge loyalty. Since he is the only one of the conspirators who has acted against Caesar for the love of Rome, he is true to his purpose. He does not realize that the other conspirators have other reasons (greed, ambition, jealousy, etc.) for wanting Caesar dead.
I think that Brutus does this because he is something of an idealist. Brutus wants to believe that their plot against Caesar is not about taking power for themselves. He wants to believe that it is about doing what is best for Rome. He does not want them to swear because he thinks that they will be bound together simply by the nobility of their cause. He doesn't want them to be bound because they have sworn an oath -- that's not a noble/ideal enough reason. He wants them to be dedicated to their plot because it is morally right.