You should remember that `sec^2 x = 1/(cos^2 x) = 1 + tan^2 x` , hence, you may write `tan^2 x` in terms of `sec^2 x` such that:

`tan^2 x = sec^2 x - 1`

You need to substitute `sec^2 x - 1` for `tan^2 x` such that:

`int sec^3 x tan^2 x dx = int sec^3 x(sec^2 x - 1) dx`

Using the property of linearity yields:

`int sec^3 x tan^2 x dx = int sec^5 x dx- int sec^3 x dx`

You should use the following formula such that:

`int sec^k x dx = (sin x sec^(k-1) x)/(k-1) + (k-2)/(k-1) int sec^(k-2) xdx`

Considering k = 5 yields:

`int sec^5 x dx = (sin x sec^4 x)/4 + 3/4 int sec^3 x dx`

`int sec^3 x dx = (sin x sec^2 x)/4 + 1/2 int sec x dx`

`int sec^5 x dx = (sin x sec^4 x)/4 + 3/4((sin x sec^2 x)/4 + 1/2 int sec x dx)`

`int sec x dx = log(tan x + sec x) + c`

`int sec^5 x dx = (sin x sec^4 x)/4 + 3/4((sin x sec^2 x)/4 + 1/2 log(tan x + sec x))`

`int sec^3 x tan^2 x dx = (sin x sec^4 x)/4 + 3/4((sin x sec^2 x)/4 + 1/2 log(tan x + sec x)) - (sin x sec^2 x)/4- 1/2log(tan x + sec x) + c`

**Hence, evaluating the given integral yields `int sec^3 x tan^2 x dx = (sin x sec^4 x)/4 + 3/4((sin x sec^2 x)/4 + 1/2 log(tan x + sec x)) - (sin x sec^2 x)/4 - 1/2log(tan x + sec x) + c.` **