The best way that Irving describes how Brom wants to deal with Ichabod is when he quotes him for saying that he would "double the schoolmaster up, and lay him on a shelf on his own schoolhouse." So, yes, Brom would like to engage Ichabod in a physical fight because he knows that he would win in such a competition. That being said, though, Brom is intimidated by Ichabod's social status. Ichabod is a leader in the community and beloved by every woman in town. Ichabod's leadership and status was right up there with the parson! If Brom did anything too tremendously harmful to Ichabod, then Brom would look like a bully or even a criminal. Plus, the characterization of Brom describes him as one who never wanted to incur true bodily harm to anyone because he is more of a prankster than a bully. Irving says that Brom "had more mischief than ill-will in his composition." Therefore, Brom wasn't a brute by nature and decides to counter Ichabod's rivalry with practical jokes rather than with violence.