This is Christopher's solution: Let $P_1$ be a plane through the points $A(2,1,0),B(1,1,1)$ and $C(1,7,3)$ and $P_2$ be a plane through the points $A,B$ and $V(x,y,z)$. Find all the points $V(x,y,z)$ such that the two planes are perpendicular.

The general equation of a plane in 3 dimensional Cartesian space is:

${\displaystyle \mathrm{ax}+\mathrm{by}+\mathrm{cz}+d=0}$

This equation will be divided through by $a$, and slightly rearranged, to make the following process somewhat easier:

${\displaystyle x+\mathrm{py}+\mathrm{qz}=r}$

Where: $p=b/a,\hspace{0.5em}q=c/a$, and $r=-d/a$.

By substituting the $x,y$, and $z$ values from the points $A,B$, and $C$ into the general equation, it is possible to obtain the three equations below involving $p,q$, and $r$ that can be solved simultaneously to derive the formula of the plane $P_1$.

${\displaystyle \begin{array}{cccc}\multicolumn{1}{c}{2+p}& =& r& \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(1)\\ \multicolumn{1}{c}{1+p+q}& =& r& \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(2)\\ \multicolumn{1}{c}{1+7p+3q}& =& r& \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(3)\end{array}}$

Subtracting equation (2) from (1):

${\displaystyle 1-q=0\hspace{0.5em}\mathrm{so}\hspace{0.5em}q=1}$

Substituting this, and the relation in (1) into (3) gives:

${\displaystyle \begin{array}{ccc}\multicolumn{1}{c}{1+7p+3}& =& 2+p\\ \multicolumn{1}{c}{6p}& =& -2\\ \multicolumn{1}{c}{p}& =& -1/3\end{array}}$

Finally, substituting this into (1) gives:

${\displaystyle 2-1/3=r\hspace{0.5em}\mathrm{so}\hspace{0.5em}r=5/3.}$

Therefore, the equation of the plane $P_1$ is:

${\displaystyle x-1/3y+z=5/3.}$

Multiplying throughout by 3 gives:

${\displaystyle 3x-y+3z=5.}$

Rearranging for y gives:

${\displaystyle y=3x+3z-5}$

This implies that for every unit increase in the $x$ or $z$ direction, there is a corresponding increase of 3 in the $y$ direction. A normal plane would therefore have the property that for an increase of 3 in the $x$ or $z$ direction, there would be a corresponding increase of $-1$ in the $y$ direction. This implies that the vector $(3,-1,3)$ describes a translation such that for any point on the plane $P_2$ (which is normal to $P_1$), the resulting point will also lie on $P_2$.

The translation is applied to point $A(2,1,0)$, giving the point $D(5,0,3)$. Using the points $A,B$, and $D$, which all lie on the plane $P_2$, the formula for $P_2$ can be determined:

${\displaystyle \begin{array}{cccc}\multicolumn{1}{c}{2+p}& =& r& \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(4)\\ \multicolumn{1}{c}{1+p+q}& =& r& \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(5)\\ \multicolumn{1}{c}{5+3q}& =& r.& \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(6)\end{array}}$

From (4) - (5)

${\displaystyle 1-q=0\hspace{0.5em}\mathrm{so}\hspace{0.5em}q=1.}$

Substituting this into (6) gives:

${\displaystyle 5+3=r\hspace{0.5em}\mathrm{so}\hspace{0.5em}r=8.}$

Substituting this into (2)gives:

${\displaystyle 1+p+1=8\hspace{0.5em}\mathrm{so}\hspace{0.5em}p=6.}$

Therefore the equation of the plane $P_2$ is:

${\displaystyle x+6y+z=8.}$

or:

${\displaystyle x+6y+z-8=0.}$

The point $V(x,y,z)$ lies on this plane; therefore, all the points $V(x,y,z)$ such that the two planes $P_1$ and $P_2$ are perpendicular are those that obey the equation above.

This is Tim's method:

Let $a\times b$ denote the cross product of $a$ and $b$. Let $v_1=\mathrm{AC}\times \mathrm{AB}$ (which is perpendicular to $P_1$). Let $v_2=\mathrm{AB}\times \mathrm{AV}$ (which is perpendicular to $P_2$).

As $P_1$ and $P_2$ are perpendicular, $v_1$ and $v_2$ are perpendicular as the angle between the planes is the same as that between their perpendiculars. Thus, the scalar product $v_1.v_2=0$.

${\displaystyle \begin{array}{ccccc}\multicolumn{1}{c}{\mathrm{AC}}& =& (1,7,3)-(2,1,0)& =& (-1,6,3)\\ \multicolumn{1}{c}{\mathrm{AB}}& =& (1,1,1)-(2,1,0)& =& (-1,0,1)\\ \multicolumn{1}{c}{\mathrm{AV}}& =& (x,y,z)-(2,1,0)& =& (x-2,y-1,z).\end{array}}$

${\displaystyle v_1=\mathrm{AC}\times \mathrm{AB}=(-1,6,3)\times (-1,0,1)=6i-2j+6k}$

(which I evaluated using the determinant method)

${\displaystyle v_2=\mathrm{AB}\times \mathrm{AV}=(-1,0,1)\times (x-2,y-1,z)=(1-y)i+(x+z-2)j+(1-y)k}$

(again, using determinant method)

${\displaystyle v_1.v_2=6-6y-2x-2z+4+6-6y=0.}$

Therefore $2x+12y+2z=16$ (rearranging) or, $x+6y+z=8$ (dividing by 2). This is the equation of the plane containing all $V(x,y,z)$ (the plane $P_2$)

Note: $(6,-2,6).(1,6,1)=6-12+6=0$ i.e. the two planes are perpendicular (as required).

Also, $A$ and $B$ are in $P_2$ (as required) since $(2,1,0)$ and $(1,1,1)$ are solutions of the equation.