use t = ln(x) so dt = 1/x dx    x = e^t, so e^t dt = dx so

integral(cos^2(ln x) dx) = integral(cos^2(t) e^t dt)

and cos^2(t) = 1/2cos(2t) + 1/2

integral(cos^2(ln(x)) dx) = 1/2 integral((cos(2t) + 1)e^t dt)

= 1/2 integral(e^t cos(2t) dt) + 1/2 integral(e^t dt)

Integrate(e^t cos(2t) dt) by parts using

u = e^t,  du = e^t dt

dv =cos(2t) dt, v = sin(2t)/2

integral(cos(2t) e^t dt) = e^t sin(2t)/2 - integral(e^t sin(2t)/2  dt) =

= e^t sin(2t)/2 - 1/2 integrate(e^t sin(2t) dt)

integrate by parts again,

u = e^t,  du = e^t dt  and

dv = sin(2t) dt,  v = -1/2 cos(2t)

to get

= e^t sin(2t)/2 - 1/2 (-1/2 e^t cos(2t) - integral(-1/2 cos(2t) e^t dt))

integral(cos(2t) e^t dt) = e^t sin(2t)/2 + 1/4 e^t cos(2t) - 1/4 integral(cos(2t) e^t) dt) so

Add 1/4 integral(cos(2t) e^t) dt) to both sides

5/4 integral(cos(2t) e^t dt) = e^t sin(2t)/2 + 1/4 e^t cos(2t))

Multiply both sides by 4/5 to get

integral(cos(2t) e^t dt) = 1/5e^t(2 sin(2t) + cos(2t))

integral(cos^2(ln(x)) dx) = 1/2 integral(e^t cos(2t) dt) + 1/2 integral(e^t dt)

= 1/2 (1/5 e^t(2 sin(2t) + cos(2t))) + 1/2 e^t + C

Substituting t = ln(x) we get

integral(cos^2(ln(x) dx) = 1/10 x (2 sin(2 ln(x)) + cos(2 ln(x)) + 1/2x + C

= 1/10  x (2 sin(2 ln(x)) + cos(2 ln(x)) + x/2) + C