Opposition in the late nineteenth century was a result of the fact that the immigrants who came to the U.S. during this time were largely from Eastern and Southern Europe, namely Poles, Russians, Jews, Italians, Greeks, etc. These groups did not readily assimilate into U.S. culture; instead they tended to retain their old world customs and even language. They primarily settled in big cities and lived in neighborhoods with people of their own background. They also tended to be fiercely Roman Catholic, at a time when anti-Catholicism was still extant in the U.S.
Opposition after World War I (the twenties) was partly due to the horror of the war, and new ideas about genetic purity, such as Madison Grant's Passing of a Great Race, which claimed that the great race of Nordics from Northern Europe was being threatened by the Latin and Slavic peoples of Eastern Europe. Also popular at the time was The Decline of the West by Oswald Spengler, who argued that European civilization had entered an inevitable state of decline, and would be superseded by a yellow race. There also was a very popular false science known as Eugenics, which held that human race could be controlled by controlling humanity.
These factors, and continued anarchism in Europe led many Americans to conclude that all people of eastern European ancestry were potential anarchists. The end result was the Emergency Immigration Act of 1921 which severely limited immigration from all areas except the Americas, and completely excluded people of Asian ancestry. This last was a blunt insult to the Japanese, and did not help matters when war clouds began gathering in the Pacific.