Berkeley was an empirical philosopher, meaning that he believed all knowledge is derived from direct observation and experience. However, a number of previous philosophers, such as Descartes, Leibniz, Spinoza, and to a lesser degree, Kant, believed in rationalism. Rationalism meant that knowledge could be reached through reason and deduction alone, sans experience. Abstract thoughts through numbers is usually considered a prime example of the way that humans can have knowledge without experience (e.g. 2+2 = 4 is true regardless of our experiences, and as long as we can conceive what 2 is and what 4 is, we can reach this conclusion with logic alone).
In this section of A Treatise..., Berkeley points out that arithmetic did not begin with abstract thought, and suggests that historically, humans kept numbers (say, the amount of cattle in a herd) through the use of check marks or strokes, or some other such counting device. He observes that as humans needed larger and more complex ways of counting, we invented numbers to represent numerical ideas in the same way that humans use words to represent linguistic thoughts. Therefore, while we may think in abstract terms when dealing with numbers and arithmetic, Berkeley makes the point that mathematical theories are actually not "distinct from particular numerable Things," except in the sense that we have given numbers names to represent the very things that they count (para. CXXII).
In the next paragraph (CXXIII), Berkeley disparages some of the more abstract and theoretical approaches to mathematics due to the fact that they are simply not based in reality. He provides the example of an object that can be divided infinitely. Imagine, for instance, a wooden rod that can be broken into two equal pieces. These equal pieces can each be broken into halves, creating fourths, then eighths, then sixteenths, and this process can occur infinitely. While Berkeley acknowledges that this is an amusing thought, it is also impossible to actually do, and if we simply did away with silly thoughts like this, things would be much simpler for mathematics as a field.