I do not believe that this would be a problem. We might get more divisive elections, but our Congress would be less divided and more able to work together.
Our politically homogeneous districts lead to very extreme candidates getting elected (even if the elections are pretty calm). If your district is all Republican, the most extreme Republican is likely to win. If the district has lots of Democrats and Independents, a moderate will be more likely to win.
If you have moderates winning (even if the elections are contentious) they will be more able to cooperate with other moderates from both parties. This would lead to a better Congress that could actually govern from the center instead of fighting between polar opposites.
So the elections might be tougher and nastier, but the people elected would be more likely to work together well and we'd all be better off.
Please refer to the link for a similar argument, made in more detail.