We'll start from the given constraint X*Y=Y*X. We know that the product of two matrices is not commutative.
Since X is not equal to Y, then, X*Y=Y*X if and only if Y = X^-1 (Y is the inverse of the matrix X).
We know that X*X^-1 = I, where I is the identity matrix.
Since the square matrix X has the inverse X^-1, then the determinant of the matrix X is different from zero value.
det X>0 or det X <0
det (X^2 + Y^2) = det X^2 + det Y^2
If det X<0, then det X^2 > 0
Since Y = X^-1, then det Y = det X^-1 => det Y^2>0
Therefore, the given inequality det (X^2 + Y^2) >= 0.