The line means that Uncle Jack was trying to irritate Miss Maudie.
According to historians, the term "get your goat" originated in the United States in 1904. The phrase didn't catch on in the UK until 1924, however.
Although the phrase meant "to irritate or annoy someone," as it does now, the story behind its origin is a little more colorful in nature.
Accordingly, goats used to be placed in the stall with high-strung, nervous racehorses. Since goats were docile creatures, they were said to exert a calming influence on the horses. So, "to get someone's goat" referred to the practice of stealing the hapless animals prior to an important race. The idea was to deprive the nervous racehorse of its calming companion so as to spoil its chances of winning the race.
In the novel, Scout relates that Uncle Jack has been trying "to get Miss Maudie's goat" for forty years. Every Christmas, he would yell across the street for Miss Maudie to come marry him. We get the idea that Uncle Jack isn't entirely serious about his proposal. Even Scout and Jem think that it's a strange way for a man to ask for a woman's hand in marriage.
The text also tells us what Uncle Jack already knows: he is the last person Miss Maudie would ever think of marrying. However, since he's the first person she always thinks of teasing, Uncle Jack decides to give as good as he gets, so to speak.
"We saw Uncle Jack every Christmas, and every Christmas he yelled across the street for Miss Maudie to come marry him. Miss Maudie would yell back, 'Call a little louder, Jack Finch, and they'll hear you at the post office, I haven't heard you yet!' . . . He said he was trying to get Miss Maudie's goat, [and] that he had been trying unsuccessfully for forty years. . ." (43-44).
This passage shows Uncle Jack teasing Miss Maudie. He explains the situation to the kids using an idiom--a figure of speech that uses words in their non-literal sense, but also in an unpredictable way. To "get someone's goat" probably originates in the early 1900s in America when slang became a popular way to speak. "Get your goad" might also be the original form, which means to get irritated. As people mispronounced phrases, heard things incorrectly, or used slang, the phraseology changed to "get your goat".
Children like Scout are very literal as they are learning to speak, read, and write. Sarcasm, and figures of speech such as idioms, can therefore be confusing. Just think about what kids might think is literally happening with other idioms such as "Cat got your tongue?" or "He can't see past the end of his nose," or "It's raining cats and dogs." Fortunately, Miss Maudie doesn't really have a goat for Uncle Jack to steal, he was simply trying to annoy her and catch her off guard. When he says he has been unsuccessful at it for forty years, that means she always has a good response to volley back at him.