Science thus far does not have a definitive answer to this question, but it is a fascinating and important one that psychologists, educators, linguists, and neuroscientists have been and will continue to look into. For the moment, most of what we have is speculation, which is that the human brain has a language acquisition "window" that narrows more and more as we get older. It never shuts completely, assuming no neurological pathology, but it is clear, as you note, that it is far more difficult for adults to acquire a new language. There could be many reasons for this.
One possible reason is that language is a matter of connections amongst neurons, just as every other kind of knowledge acquisition is, and once those first pathways are set down, they become the path of least resistance. Some evidence that supports this is that when children are bilingual at an early age, they find it far more easy to acquire additional languages later on in life. This suggests that they have acquired the skill of "switching tracks" in their brains. If you do not have one path of least resistance in your brain, but two, that makes you less likely to be stuck on one track later on. This might be a kind of code-switching skill at a very early age that makes us better at code-switching later on.
Another aspect of language acquisition is social and emotional. We learn our "mother tongue" from our mothers, more than from anyone else. Mothers are good at this, emphasizing new words for their children, holding them on their laps while they talk to them, explaining processes as they prepare food, or just naming objects on a walk. When babies respond, first with babbling and then with words, mothers are wildly enthusiastic. The social and emotional aspects of learning are very powerful. We have the sense very early on that communication is a loving and pleasurable activity. And we also know that children who are deprived of this aspect of language acquisition do not fare well later on without a great deal of time and support. The film Nell provides some fascinating and moving glimpses into this problem.
Contrast the previous situation with the acquisition of a new language as an adult. You are far more likely to get a great deal of negative reinforcement in this endeavor. We all have seen how people respond unkindly to those amongst us who do not speak perfect English. And classroom learning might provide you with an "A," but it will not provide you with the social and emotional support you received when you learned your first language. No one is there to encourage our babblings or to hold us on a lap or to applaud enthusiastically when we say our first sentence. Negative reinforcements never work nearly as well as positive ones.
The language window may also get narrowed simply because we do not have the context, focus, or time that we have put into learning our first language. We are usually, as adults, learning words in a kind of vacuum, not in their natural contexts. We are not walking through the woods learning the word for a tree or a flower. And we are distracted by our other obligations, too. An infant's language acquisition is full-time. An infant can focus on this and not have to worry about making dinner or paying the bills. These, too, might very well help explain the difficulty of adult language acquisition.
Scientists of all sorts are interested in this question because if we understand this, we are gaining insight into the mechanisms of the brain and its interaction with the environment. There are implications of all sorts, for education, for psychology, for linguistics, just to name a few disciplines. It is a great question!