a. Denote (ax)/(x^2+b) = g(x), then h(x) = arcsin(g(x)).
We know that (arcsinx)' = 1/sqrt(1-x^2). So by the formula for the derivative of the composition
h'(x) = g'(x)/sqrt[1-(g(x))^2] =
[a*(x^2+b - 2x^2)/(x^2+b)^2] / [sqrt(((x^2+b)^2-a^2x^2)/(x^2+b)^2)] =
[a*(b-x^2)/(x^2+b)^2] / [sqrt((x^2+b)^2-a^2x^2) / |x^2+b|] =
a*(b-x^2)/sqrt((x^2+b)^2-a^2x^2) * (1/|x^2+b|).
Here we used that sqrt(u^2) = |u| and that |u|/u^2 = 1/|u|.
So the derivative really has a form we had to prove.
b. (x^2+b)^2 - a^2x^2 = (x^2)^2 - 2x^2*(a^2/2 - b) + b^2.
If (a^2/2 - b) = b this expression becames the full square,
(x^2 - b)^2 and sqrt of it is equal to |b - x^2|.
(a^2/2 - b) = b means that a^2 = 4b and a^2/b = 4. So b=a^2/4>=0
(if b=0 then a=0 and h(x)=0 for all x, so we can consider only b>0).
Then h'(x) = [a/|x^2+b|]*[(b-x^2)/|b-x^2|]. The second multiplier is +-1,
let's consider where + and where -. b>0 so |x^2+b| = x^2+b.
b-x^2 < 0 when -sqrt(b) < x < sqrt(b) or -a/2 < x < a/2.
At this segment h'(x) = -a/(x^2+b).
For x outside of this segment h'(x) = a/(x^2+b).
So the answer is:
h'(x) = a/(x^2+b) when x in (-∞, -sqrt(b)) U (sqrt(b), +∞)
h'(x) = -a/(x^2+b) when x in (-sqrt(b), sqrt(b)).
3. We have to prove that |g(x)| <= 1 for arcsin to have sense.
For a=0 it is trivial and for a>0 b>0. And b=a^2/4.
Consider 1-|g(x)| = (x^2+b-|ax|)/(x^2+b) =
[|x|^2 - 2(|a|/2)|x| + (|a|/2)^2]/(x^2+b).
The denominator >0 for all x and the numerator is equal to
(|x| - |a|/2)^2 >= 0 for all x.
So we proved that 1-|g(x)| >= 0 for all x or |g(x)| <= 1.
Please ask if anything is unclear.