Do not worry about your difficulties in Mathematics. I can assure you mine are still greater.
How can Albert Einstein have said this if, in fact, he was a physicist, who dealt with mathematical formulas all day?
Have you ever heard the saying to the effect that the more you learn about a subject, the more confused you are? I think that is what is going on here.
Of course Einstein would have been great at relatively low levels of math (once he got past failing math in grade school). But that's not the kind of math that he needed to do. He was thinking about issues of math way up in levels that few of us will ever understand.
So it makes sense to me that he would have trouble. He was dealing with things that are beyond the comprehension of all but a few people.
Follow the link for a couple of other idea about what he meant.
The simple answer to your question is that Einstein was referring to his struggles at learning math when he was younger. Of course, he eventually mastered the subject on a very high level and was able to use it as a tool in his work as a physicist.
The deeper answer is that mathematics always remained a difficult subject for Einstein because he recognized how profound and difficult it truly is. When a scholar truly recognizes the complexity of his chosen field of study, it becomes harder, not easier.
For example, when you first learned about the Civil War in 4th or 5th grade, you were probably taught that it was all about slavery. This makes it easy to understand the conflict about the North and South. The problem is that it is a very imcomplete picture of the truth. When a scholar studies the Civil War, he realizes that the truth is much more complex--and therefore, much more difficult to understand.
I do not know, when or why Einstein said the above words attributed to him. But assuming Einstein spoke these words after he became a great mathematician, the meaning conveyed to me is that just because mathematics is difficult to work with, it does not mean that you cannot learn or use mathematics effectively. It points to the fact that even a great mathematicians has to work hard at mathematics to produce something worthwhile using it.
This interpretation is in line with general principle that it is not desirable to give up on a worthwhile task just because it is difficult to do, and that the joy of achieving success in such tasks is worth the effort and difficulty in performing those tasks.