Thanks to all for the constructive and informative discussion. I wish I
could say that it has helped to solve my problem but I'm still wrestling
with it. The low frequency circuit analysis by Spice, Bode and Nyquist plot
are beyond my capacity but I have had some excellent help from Dave,
off-list, who eye-balled the DX-394 schematic and my mod. We identified
decoupling networks and what I assume to be AGC 'delay' networks and tackled
it on the basis that reducing the time constants of the larger ones should
reduce the low frequency phase shift that could be contributing to the
problem. The opposite occurred - increasing the 'delayed AGC' time constant
reduced the instability and effectively slowed the attack, especially on the
RF front-end. More or less the same effect was obtained by slowing the
attack in the attack network that affects all stages.

I believe 'delayed AGC' means a slower or delayed attack at the RF stages;
in the DX-394, there is a R-C network adding maybe 15 ms to the attack on
the AGC line affecting both the 1st mixer and the drain-source current of
the RF preamp and a second network adding maybe 10 ms on top of this
affecting the AGC gate of the RF preamp. I've doubled that first time
constant and doubled what I would like in my attack and release networks in
order to get the fastest stable speeds which I guess would be on the order
of 20-40 ms attack and 50-80 ms release.

My mod has a FET amp/buffer at IF driving the heck out of the diodes so that
should be fairly linear. I have adjustable gain at the output of the
detector/attack filter. I don't think BFO interference is an issue; I don't
notice any great difference in stability with it on or off - there are
separate envelope and product detectors.

I'd welcome any more input. The base DX-394 schematic is at
http://www.monitor.co.uk/radio-mods/dx-394/dx-394.htm and I'd be happy to
send anyone the schematic of my mod.

73, Tom