(1) If the area of the ellipse equals the area of the annulus then $\pi ab = \pi b^2 - \pi a^2$ and so $ab = b^2 - a^2$. Then, dividing by $a^2$, $$b/a = (b/a)^2 -1.$$ The ratio we want to find is $b/a$, the ratio of the longer to the shorter axis of the ellipse. So let $b/a = x$ then $$x^2 - x - 1 = 0.$$ Using quadratic formula: $$x = {1\pm \sqrt 5 \over 2}$$ We choose the positive root knowing that the ratio $b/a$ is positive so this ratio is equal to the golden ratio.
(2) Note that $R$ appears itself in the nested root. Therefore we can say $$R = \sqrt (1 + R)$$ and so $$R^2 - R - 1 = 0.$$ We have a quadratic of the same form as above. Hence we find $R$ to be the golden ratio.