The first major waves of immigration came to the United States in the 1830s and 40s, and involved hundreds of thousands of people from Ireland and Germany. They came mostly to northern cities with the goal of migrating west for land, but some ended up in southern ports as well.
Like many first generation immigrants, they were easy taregts of discrimination. The Germans with their broken or non-existent English and heavy accent (not to mention their German names), were viewed as a threat to American culture, and as mostly unwanted competitors for the job and economic opportunities American citizens wanted for themselves. Education in America was quite limited, and intolerance was high.
The Irish too, with their thick Irish brogue, were widely looked down upon, economically segregated into their own neighborhoods in major cities, and were given the lowest paying jobs that no one else wanted, especially in the textile mills and iron foundries, or building railroads. Too make it more difficult for them, they were Catholic, and anti-Catholic sentiment rose sharply at this time.
A political party was even formed in response to these two groups of immigrants, believe it or not, called the Know Nothing Party. They didn't mean the name the way it sounds now, but they elected anti-immigrant candidates, tried to pass restrictive legislation and wanted to permanently limit immigrants' right to vote, even after they had become citizens.