It might be a good idea to work systematically:
Start by cutting out a 1cm square from each corner - what would the volume be?
Then a 2cm square, then a 3cm square, and so on...

The side length of the square doesn't necessarily have to be a whole number...

Perhaps it would help to use a spreadsheet to perform and record your calculations.