Shakil wants to remove numbers from the set $\{1,2,3,..., 16\}$ so that no two remaining numbers add to make a perfect square. What is the smallest number of numbers that he needs to remove?