Cassius, wisely, suggests that they stay at Sardis and rest and recoup while Antony's army uses their time, energy, and supplies to march to Sardis to fight. In addition to being well rested, their armies would occupy the high ground, a huge military advantage in battle.
As far as agreeing with Brutus at this point, probably for the same reason he's agreed throughout the play. Cassius recognizes the importance of Brutus's support and can't do anything to make Brutus upset. If Cassius has any chance to be a leader in a new regime, he will have to be a good buddy of Brutus. That's why Brutus is allowed to make bad decision after bad decision; these conspirators need him and his influence on the commoners just in case they win.