The main reason for this rule is to prevent the Question Period from devolving into a period filled with arguments and debates. It is difficult enough to prevent this even with the rules as they are and it would be much more difficult if members of the House were to speak directly to one another.
Question Period is supposed to be a time for asking specific questions, not a time for political debate. This is one reason for the very short time limit placed on questions. If speakers were to address one another, this could well change. As speakers addressed one another, tempers would inevitably rise. People on opposite sides would start to talk over one another. The whole process would become more chaotic than it already is. We know that some people already feel that Question Period lacks civility and decorum. Without this rule it would have even less civility and would lose much of its usefulness.