Orthogonality refers to the independence of objects/variables - i.e. they are completely independent - there is no part of one variable in the other. This is the same as the geometric meaning, if lines are perfectly perpendicular, then the vector representation of both lines are completely independent - there is no directional component of one line in the other.
For experimental design, a hypothesis, or model is assumed. If the result, R, is a function of two variables (x and y) and they may be dependent on each other (i.e. food availability and weight) then R(x,y). But, if the variables are independent (weight and favourite colour) then R(x,y)=R(x)*R(y). That is, the mathematics for both variables can be ascertained, described and manipulated, independently.
Getting even deeper into the maths, an orthogonal set of variables will follow `X^T X =I`
in linear algebra terms.