Congratulations Tom Davie, Mike Gray and Ella Ryan of Madras
College for your excellent team work on this problem. This is their
solution; Tom wrote up the first part, Mike solved the equation and
showed that there are two possible circles, and Ella described the
construction of the smallest circle. Work like this is a real
pleasure to read.
Let the radius of the small circle be $r$ and the radius of the
larger circles be $R$. From the two triangles in the diagram
formulae for $h$ can be found: $$h = r + \sqrt{(R + r)^2  (R 
r)^2} = r + 2\sqrt{rR}$$ and $$h = R + R\sqrt 2.$$ Hence
\begin{eqnarray} \\ (1 + \sqrt 2)R  r
&=& 2\sqrt{rR}\\ (1 + \sqrt 2)^2 R^2  2(1 + \sqrt 2)rR +
r^2 &=& 4{rR}\\ (3 + 2\sqrt 2)R^2  2((1 + \sqrt 2) + 4)rR
+ r^2 &=& 0 \\ (3 + 2\sqrt 2)R^2  2(3 + 2\sqrt 2)rR + r^2
&=& 0. \end{eqnarray}
This is a quadratic equation giving $r$ in terms of $R$. Using the
quadratic formula: $$r = {1 \over 2}\big((3 + \sqrt 2)R \pm
\sqrt{(2(3 + \sqrt 2)R)^2 4 \times 1 \times (3 +\sqrt 2)R^2}\big
)$$ Simplifying this expression gives: $$r = R(3 + \sqrt 2 \pm
2\sqrt{2 + \sqrt2)})$$ The two solutions give radii, $r_1$ and
$r_2$, of two circles, $C_1$ and $C_2$, which touch both circles of
radius $R$ and also touch the tangent lines as shown in the
diagram.
$$r_1 = R(3 + \sqrt 2  2\sqrt{2 + \sqrt2)})$$ $$r_2 = R(3 + \sqrt
2 + 2\sqrt{2 + \sqrt2)}).$$
Constructing the small circle. The
small circle has radius $r$ (given in terms of $R$). $$r = R(3 +
\sqrt 2  2\sqrt{2 + \sqrt2)})$$ To construct it you will need a
pencil, compasses and straight edge.
 Set your compasses to the size of the radius of the larger
circles (length $R$) ).

Find $\sqrt 2 R$:
 take 2 lengths of $R$
 construct the perpendicular bisector of this line
 mark a length $R$ on the bisector
 the resulting diagonal has length $\sqrt 2 R$
 Find $3R$ :
 Find$(\sqrt{2 + \sqrt 2})R$
Ella's method uses the intersecting chord theorem: $x^2 = PA\times PB$
 draw line $AB$ of length$R +2R + \sqrt 2 R$
 mark the point$P$ such that$PA = R$
 draw the circle on $AB$ as diameter
 draw the chord perpendicular to $AB$ through $P$
 if this chord has length $2x$ then, by the intersecting chord
theorem, $$x^2 = R \times (2R + \sqrt2 R)$$ hence $$x = \sqrt{(2 +
\sqrt2)R}.$$
 Find $2(\sqrt {2 + \sqrt2})R$
 Find $r$
 subtract$2(\sqrt {2 + \sqrt 2})R$ from$\sqrt 2 R + 3R$
 Find the centre, $S$ of the small circle
 set the compass to length to$r$
 mark off the length$r$ from the origin on both axes, giving the
points where the circle touches the axes
 draw two arcs of radius $r$from these points, then the point
where the arcs intersect is the centre$S$ of the small circle
 finally draw the circle centre $S$ radius $r$
APPENDIX 1  DRAWING PERPENDICULAR LINES AND FINDING THE
MIDPOINT OF A LINE
 set your compass to any length, retaining it throughout
 draw arcs above and below the line from each end of the
line
 where the arcs intersect are two points on the perpendicular
line
 where the perpendicular line and the original line cross is the
midpoint.