We have to determine a and b given that for x*y = xy + 2ax + by, the * operator is commutative.
x*y = y*x
=> xy + 2ax + by = yx + 2ay + bx
=> 2ax + by = 2ay + bx
equating the coefficients of x and y we get
2a = b
If the numbers a and b are related as 2a = b, then * is commutative