the crucial piece of information is that noise stays even when you
have opened up the VCO loop.

looking at your setup, it appears to me that the noise can come from
two sources, a) the VCO and b) the PLL. evidentaly, it is not from the
VCO as your experiments prove. it is probably from the PLL circuitry.
this is quite possible. the PLLs involve a lot of digital, noisy
switching that can generate these birdies and spurs. however, these
should stay well inside the PLL block. in your case, they are getting
coupled back to your output.

there are a number of cures for this. all of them will work at a
better buffer between the vco and the following PLL. the simplest
solution is use a grounded gate FET amplifier between the PLL input
and the VCO output. be careful though, such a configuration is almost
gaurenteed to self oscillate. but that is easily taken care of. what
you do is this .. solder the FET upside down with its legs sticking
up. solder the gate to the ground with as small a lead as you can.
then, using a thin copper sheet or an unetched pcb, make a sheild that
is soldered vertically over the FET (with a cutaway to allow the FET
body). keep the source and drain leads on opposite sides of the
sheild. bias the FET for nominal current at about half the Idss.

if you have a copy of EMRFD around, you can check this design in the
chapter on oscillators.

- farhan