For Captain Vere, order is everything. If the Indomitable is to achieve a successful voyage, good order among the crew must always prevail. And as Vere is captain of the ship, that onerous responsibility falls upon his shoulders. He loved Billy dearly, but when it comes to doing his duty as captain, Vere's feelings simply don't enter into the equation. If Captain Vere were to show lenience to Billy, then he'd be setting a very dangerous precedent indeed. Sailors throughout the British navy would see it as a green light to challenge authority, and that's not something that Vere, or any other ship's captain, should ever be prepared to tolerate.
In the above quotation, what Vere is saying is that custom or habit, not emotion, governs our behavior. This is especially true of professionals involved in dangerous occupations such as sailors, who need to adhere to the "measured forms" of the Mutiny Act if they're to maintain good order on the high seas.