Solutions to aspects of this problem have been submitted by Coope, Wern and Adam. I have used their solutions as the basis of what follows. Many thanks to each of you for your efforts and well done.

Since each row must add up to a prime number and the lowest number possible on any row, column or diagonal is 1+2+3 = 6, you can use the restriction that all totals of rows and columns must be odd as all primes greater than 2 are odd.

Using this information let us first consider the case for making horizontal and vertical sets prime. The restriction of 5 odd numbers means that to satisfy the conditions given above the following formation must be present:
1. One row contains 3 odd numbers, and the remaining 2 contain one odd number each.
2. One column contains 3 odd numbers, and the remaining 2 contain one odd number each.

There are 9 possible arrangements for the odd numbers. 4 are rotations of a T shape, 4 are rotations of an L shape and there's one that's a cross shape.  

These are the only configurations of odd numbers that will allow for the horizontal and vertical sets to possibly sum to a prime. However, out of all of these, the only possible arrangement that could satisfy the diagonal property is the 'cross' shape, as all the others have an even number of odd numbers on at least one diagonal. 

It's important to note that swapping a pair of rows or a pair of columns doesn't affect the row and column sums (you should check this is true). However, these swaps do affect the diagonal sums. To make it easier to be systematic, let's just consider squares of the following form, with all the odd numbers in the third row or the third column (as all other squares can be changed into this form by row and column swaps): 

We know we can swap rows and columns to get our square into this form without affecting the row sums and the column sums. Let's not worry about the diagonal sums for the moment.

It turns out there are 16 possible squares of this form, which are listed in this table according to which odd number is in the bottom right corner. Intriguingly, there are no squares with 5 in the bottom right corner.

Now let's see if any of these also satisfy the property that their diagonals also sum to a prime. We know the only possible arrangement that will satisfy this is the 'cross', with all the even numbers at the corners. Let's try the first one, in this case ending up with 1 in the centre:
 

The final square is of the 'cross' form, but its diagonals sum to 9 and 13, so it doesn't satisfy the diagonal property as well as the row and the column property. 

If this process is repeated for all the others, it turns out none of them have the diagonal property as well as the row and the column property, so it's not possible to have all the rows, columns and diagonals summing to a prime using the rules we have.

The following table shows all the above solutions in the 'cross' arrangement, and it's easily verified that in each one at least one of each of the diagonals doesn't add to a prime.