Suppose that the triangle and the square are placed so that the area of the overlap is as large as possible.

The area of the square is $36{cm}^2$.

The area of overlap is $2\over 3$ of this, namely $24{cm}^2$.

This is $60$% of the area of the triangle.

So the area of the triangle is $\frac{10}{6}\times 24 = 40{cm}^2$.

*This problem is taken from the UKMT Mathematical Challenges.*