My approach is as follows:

$y^2=a^2+b^2+2[{\sin }^2(\theta ){\cos }^2(\theta )(a^4+b^4)+a^2{b}^2({\sin }^4\theta +{\cos }^4\theta {)]}^{1/2}$

Now use ${\cos }^4\theta ={\cos }^2\theta (1-{\sin }^2\theta )$, ditto for sin, and again use ${\sin }^2\theta +{\cos }^2\theta =1$ to get

$y^2=a^2+b^2+2[a^2{b}^2+{\sin }^2(\theta ){\cos }^2(\theta )(a^4+b^4-2a^2{b}^2{)]}^{1/2}$

But $\sin (2\theta )=2\sin \theta \cos \theta$ so substitute and take roots:

$y=[a^2+b^2+[4a^2{b}^2+{\sin }^2(2\theta )(a^4+b^4-2a^2{b}^2{)]}^{1/2}{]}^{1/2}$

Now we have two approaches to finding the stationary points:

(i)the squareroot function is monotonically increasing, so assuming $a$ and $b$ are not both zero, $y$ is stationary when the argument of the outer squareroot is stationary. But $a$ and $b$ are constants so we are looking for inner root to be stationary. The same argument tells us that the argument of the inner root must be stationary, and because $a$, $b$ are constants, we must have ${\sin }^2\theta$ stationary, i.e. at 0, pi/4 etc. You should also be able to argue for the 0, pi/2 etc being minima, the others maxima.
(ii) Use calculus. But this time it is easy, if we assume $a$, $b$ not both zero, because we can cancel lots of stuff as we work through the differential (make sure you understand why):

$0=\mathrm{dy}/d\theta =d/d\theta [a^2+b^2+[4a^2{b}^2+{\sin }^2(2\theta )(a^4+b^4-2a^2{b}^2{)]}^{1/2}{]}^{1/2}$

$0=d/d\theta [a^2+b^2+[4a^2{b}^2+{\sin }^2(2\theta )(a^4+b^4-2a^2{b}^2{)]}^{1/2}]$

$0=d/d\theta [4a^2{b}^2+{\sin }^2(2\theta )(a^4+b^4-2a^2{b}^2{)]}^{1/2}$

$0=d/d\theta [4a^2{b}^2+{\sin }^2(2\theta )(a^4+b^4-2a^2{b}^2)]$

$0=d/d\theta [{\sin }^2(2\theta )]$