My guess is that both would work, but that both would have problems.
We know that C-QUAM works and works quite well for MF ground wave reception. At VHF, the channel widths are such that there would be less concern about curtailing the PM sidebands, and it might be possible to extend the difference channel to a full 15 kHz bandwidth, rather than the 7.5 kHz used at MF.
So far, so good. But C-QUAM is probably susceptible to multipath reception. The C-QUAM decoder ICs, such as the MC13020, include a co-channel carrier detector that inhibits stereo operation when co-channel interference is above a specific low level. With MF reception, this is more likely to happen outside of the primary reception zone, where there is more likely to be interference from another skywave or the desired transmitter’s own skywave. I suspect, but I don’t know for sure, that at VHF, multipath would be seen by the decoder as co-channel interference and inhibit stereo operation. If so, as multipath could occur anywhere in the primary service area, VHF C-QUAM might suffer from a higher incidence, possibly or probably an unacceptably higher incidence, of switching to mono than would MF C-QUAM in its ground wave primary service area.
Noise limiters were regarded as highly desirable, if not essential for VHF AM, and I think that most UK 405-line TV receivers incorporated such. The conventional diode type that cutoff on fast rise-time pulses could also be thought of as a frequency division device, and required an input bandwidth of around 100 kHz or so. It may be recalled that the UK VHF modulation argument was to some extent three-way, namely FM, narrowband AM and wideband AM, the last-mentioned enabling the use of receiver noise limiters. So for VHF C-QUAM, decoders would have required wide “audio” bandwidths. That should not have been too difficult; one may easily imagine a 10.7 MHz, wideband version of the MC13020. Perhaps IC-based noise limiters would have been used, as well. I think that FM stereo noise blankers such as the Philips TDA1001 and Toko KB4423 would have worked; these could also have been used for AM TV sound although I doubt that they were.
Something else to ponder though is that C-QUAM was developed in the integrated circuit age, and so included decoding features, such as pilot tone cycle counts, that were easily implemented in IC form, but which would have been more difficult to do, at least for economical mass production, in discrete solid sate or valve form. That is not to so that the basic decoding could not have been done well in the pre-IC age. With valve technology, for example one might look for inspiration in the circuitry of the GE YRS-1 SSB adaptor of the late 1940s, which used the Norgaard phasing technique for sideband separation, and the Hazeltine DC quadricorrelator of the 1950s, developed for NTSC colour TV subcarrier recovery and decoding control.
Nevertheless, stereo AM systems for MF were developed in the late 1950s and early 1960s, and these were of course intended to work with relatively simple valve decoders. RCA did a lot of work in this field, and concluded that with the technology of the time, an AM-FM system, with the difference channel frequency modulated on to the AM carrier, was the best choice. This is recorded in detail in RCA Review for 1960 September, p.299ff, and available here: http://www.americanradiohistory.com/RCA ... ue_Key.htm
. The BBC made brief mention of the MF stereo AM proposals of the period in its Monograph #27 of 1960 April, “A Summary of the Present Position of Stereophonic Broadcasting”, available here: http://downloads.bbc.co.uk/rd/pubs/arch ... aph_29.pdf
Had VHF-AM been a “going concern” in the UK back when FM stereo was being debated in the USA, then I imagine that the proposed MF AM stereo systems would have been the starting point for the development of a VHF AM stereo system. I think that MF AM stereo as well as FM stereo was within the original ambit of the US NSRC (National Stereo Radio Committee), which advised the FCC, but was dropped off, perhaps because of lack of any real commercial interest at the time.
That an AM-AM subcarrier system could work is evidenced by the French bilingual TV sound transmission system of the later 1950s. What works for bilingual sound has a good chance of working for stereo, although the converse is not necessarily true. It was conceived as a TDM system, but a TDM system that transmits just the first set of sampling frequency sidebands is the same as an FDM system whose subcarrier frequency is the same as the TDM sampling frequency. In this case the sampling frequency/subcarrier frequency was 20 475 Hz, the line frequency of the 819-line TV system. And the audio bandwidth was 9 kHz. There was no pilot tone though, as the line frequency reference was obtained from the line timebase (or from separated sync from the demodulated video signal). The availability of an “external” reference allowed very simple TDM decoders. One of the sound IF stages was simply switched at on and off at line frequency, with one or other channel chosen by reversing the relative mark-space positioning. I imagine that a 9 kHz low-pass filter in the audio channel may have been desirable. Possibly a diode noise limiter could have been used ahead of this, although whether subcarrier artefacts would have gotten in the way, so as to speak, I don’t know.
For non-TV applications though, a pilot tone (or residual subcarrier) would have been needed. As this could be recovered only after demodulation, then the simple pre-demodulation decoding system described above could not be used. Instead, normal pilot-tone decoding would be required. Then I think that noise-limiting would have become a problem, at least in the pre-IC age. Noise-blanking circuitry inserted between the demodulator and decoder that also provided for pilot-tone continuity might have required a non-trivial valve count, but I haven’t really thought that one through. It might have helped to keep the subcarrier frequency as low as possible, on the assumption that the effect of impulsive noise increased somewhat with frequency. In that case the FM stereo polar modulation system, with 32 Hz subcarrier, only partially suppressed, might have been a better choice than the Zenith-GE pilot tone system.
Probably some attention would have had to be paid to AM diode demodulator performance, to minimize non-linearities that resulted in cross-modulation and intermodulation involving the subcarrier. In fact it seems likely that the better class VHF-AM receivers, had they existed, would have had AFC obtained from a limiter-discriminator combination, in which case the limiter grid could well have been used for AM demodulation. That this was satisfactory, despite the conventional wisdom about grid-leak demodulation, was evidenced by the fact that it was used in the Ambassador-BBC VHF-AM/FM comparator receiver, which was designed for high-quality reception.
So, no easy answer, but I think we can say with reasonable certainty that had VHF-AM stereo been required circa 1960, then both subcarrier and AM-PM systems would have been considered, AM-PM covering both the QUAM and AM-FM varieties.