The solution here depends on using Pythagoras theorem in 3 dimensions (actually using the theorem for a right angled triangle on the floor and then a second time for a vertical right angled triangle). The centre of the large ball (of radius $5$ cm) is at the centre of the box and, if you think of the straight line from one corner of the box to the centre of the box, it goes through the centre of a small `packing' ball.

This solution came from Christopher, Bishop's Stortford College.

(i)

This involves using Pythagoras' Theorem in three-dimensions. Let $r$ be the radius of the ball. When the ball rolls into the corner of a room, its centre is a distance $r$ from the sides and floor, therefore the horizontal distance to the corner is given by: $$x^2 = r^2 + r^2 = 2r^2$$ therefore $x = r\sqrt{2}.$ The ball's centre is at height $r$, therefore the distance $d$ to the corner is given by: $$d^2 = r^2 + (r\sqrt{2})^2 = r^2 + 2r^2 = 3r^2$$ therefore $d = r\sqrt{3}$. So for a ball rolled into the corner of a room the distance of the centre of the ball from the corner is $r\sqrt{3}$ where $r$ is its radius.

(i)

Let $R$ be the radius of the big ball.

Let $r$ be the radius of the small balls.

Let $x$ be the diagonal (corner to opposite corner, i.e. the longest dimension) of the box.

Let a be the length of the sides of the cube.

By Pythagoras's Theorem in three dimensions again, $x = a\sqrt{3}.$ The small balls fit into the corners and just touch the big ball so the diagonal of the box is equal to the diameter of the big ball plus twice the diameter of the small ball plus twice the small distance between the small ball and the corner (=the answer to part (i) minus the radius of the small ball) therefore $$x = a\sqrt{3} =2R + 4r + 2r\sqrt{3} - 2r$$ Rearranging this expression gives: $$r = (a\sqrt{3} - 2R) / (2 + 2\sqrt{3})$$ Substituting $R$ = 5 cm, $a$ = 12 cm and calculating the radius of the small ball to three significant figures we get: $$r = (12\sqrt{3} - 10) / (2 + 2\sqrt{3}) = 1.97 {\rm cm.}$$