The first time Sir Andrew confronts Cesario is when he challenges Viola as Cesario to a duel in Act 3, Scene 4. Sir Andrew was brought by Sir Toby into Olivia's household as a hopeful suitor to Olivia, but now Sir Andrew has noticed that Olivia is giving Cesario, Duke Orsino's page, far more attention than she is giving Sir Andrew. At Fabian's and Sir Toby's encouragement, Sir Andrew decides to challenge Cesario in the hopes that it will attract Olivia's attention. However, little does he know that Cesario is really a woman named Viola who of course was never educated in sword fighting. In addition, Sir Andrew is actually a coward, so when he is egged on by Sir Toby and Fabian, and he confronts Viola as Cesario, Sir Andrew is actually just as terrified as Viola. Since both are terrified, both try and pacify the situation. Sir Andrew even offers Viola a horse should she not kill him. When they are interrupted by Antonio, they both very gladly put away their swords.
In contrast with his twin sister Viola, Sebastian of course being a man was raised to learn fighting skills. In addition, he seems to have a much more aggressive nature than his sister. Hence, when in the very next scene Sir Andrew encounters whom he thinks is Cesario but is actually Sebastian, their encounter becomes much more violent. Sir Andrew, apparently still angered by Cesario, strikes whom he thinks is Cesario, and unlike Viola, Sebastian of course very readily strikes him back multiple times. Sebastian is so ready to continue the fight with Sir Andrew that Sir Toby must hold him back. Eventually Sebastian and Sir Toby challenge each other to a duel, which is broken up by Olivia. Hence, we see that Sebastian is far more aggressive with Sir Andrew than Viola, and one reason is of course that he is a man while Viola is really a woman.