Please look at the picture and note that the point S is on AP, not AB, which would be much simpler.
To minimize the area of PQRS, let's maximize the area of its complement, which consists of the non-overlapped triangles ABP, PCQ, QDR and SAR. The only one which area is not obvious is SAR.
Its area is `1/2 AR * AS * sin (SAR) = 1/2 (4-k) * AS * 4/sqrt(16+k^2) = AS * (2(4-k))/sqrt(16+k^2).`
To find AS, use the proportion `( AS ) / (SP) = k / (4-k) ` and `AS + SP = sqrt(16+k^2), ` which gives that `AS = k / 4 sqrt(16+k^2).`
This way, the area of the complement is `C(k) = 2k + k(4 - k) + (k(4-k)) / 2 = -3/2 k^2 + 8k. ` The maximum of this quadratic function is attained at k = 8/3, which is in the interval (0,4).