The connection between the rules of engagement and the idea of limited war can be seen more clearly in the rules of engagement for the air war in Vietnam. The rules of engagement that governed infantry operations were more concerned with winning the "hearts of minds" of the Vietnamese than with keeping the war limited.
In the air, there were rules of engagement meant to prevent the war from spreading. This included restrictions on what sorts of targets could be bombed and on the circumstances in which American pilots could attack enemy aircraft. The point of these restrictions was to avoid coming into direct conflict with China and/or the Soviet Union.
With regard to the infantry, the rules of engagement were meant more to prevent civilian casualties. This is why there were different rules of engagement for "free fire zones" than for other places. This, of course, made it very difficult for American soldiers in places other than the free fire zones to protect themselves given that it was very difficult to distinguish between civilians and combatants.