I would not agree that there is "no evidence" that Jefferson reduced the size of the military. For example, I quote from the link below:
With the expert help of Albert Gallatin, secretary of the treasury, and James Madison, secretary of state, he greatly reduced the army, the navy, and the foreign diplomatic corps.
I also quote from America: A Narrative History by Tindall and Shi:
The basic formula was simple: cut back expenses on the military. A standing army menaced a free soceity anyway. It should therefore be kept to a minimum... The navy ... ought to be reduced further..." (pp. 376-7 in the 5th edition of the book).
I further quote from The American Pageant by Bailey, Kennedy, and Cohen.
Jefferson reduced the military establishment to a mere police force of 2,500 officers and men. ... Accordingly, Jefferson shrank the navy to a peacetime footing.
These constitute, in my mind, evidence for my contention that Jefferson shrank the military.