One factor you should keep in mind when speaking about the Industrial Revolution and the Industrial Age is that industrialization brought with it the advent of class consciousness. In a very real way, members of the middle and upper classes tended to view their own class identity in contrast to that of the working class. Indeed, consider a theme such as the Cult of Domesticity, which insisted that women within the privileged classes should not work for a living, and certainly not within the confines of manual labor (which was regarded as strictly within the purview of the working classes).
To members of the privileged classes, workers were often perceived as unruly and unrefined. Furthermore, to be impoverished was often viewed with a kind of moral indignation. Because poverty was often viewed as a temporary state (and a state which, therefore, anyone could potentially overcome), for members of the more privileged classes, the failure to overcome poverty and advance upwards through society was often perceived as a personal failure. Thus, when viewed collectively, members of the more privileged classes could be expected to hold the working classes in disdain, while rejecting the notion of having anything in common with their number.