It is assumed that the planets' densities are uniform so that the planets act gravitationally as if all their masses were concentrated at their centers and the planets act like mass points.
The additive constant in the potential energy is chosen so that the potential energy of the system is zero when the two planets are infinitely far apart. Since the gravitational potential energy decreases as the planets move closer, the potential energy is therefore negative at finite planet separations. The same is true for the total energy if the planets start from rest at some finite separation.