I have a band-reject (notch filter) duplexer with low side at 462 MHz (transmitter) and high side at 467 MHz (receiver).
I would like to learn on how to tune a duplexer, so I’ve watched many tutorials on the use of a VNA. What I am having a hard time understanding is why S21 is connected to the antenna port and not to the opposite port of S11 (Low or High). I tried both ways and it seems that if I place S21 on the opposite port I get a complete picture of both notch filters and insertion loss. Whereas if I place S21 on the antenna port, I only see one notch filter. Am I missing something here?
For a two port VNA you would usually connect the S21 (source) port to the antenna side of the duplexer, connect the measure port of the VNA to the side of the duplexer you will be tweaking and place a 50 ohm load on the unused port. When done with one side of the duplexer move the VNA to the other side and put the 50 ohm load on the now unused port.
Many flatpack type notch only duplexers are rated around 75dB isolation max which is not that good but it can be at the dynamic range limits of a Nano VNA or similar. You want the VNA measurable dynamic range to be at least 10dB better than what you will be measuring and it will probably take a good bench top HP/Agilent or similar unit to make reliable measurements on a good duplexer.
Also, connecting a VNA across both output ports of a duplexer, hopefully with a load on the antenna port, will let you see both sets of notches but at much reduced resolution. The max notch depth is very narrow and you want as much visibility and frequency accuracy on your VNA screen to adjust them to the precise needed frequencies and I like to keep the span less than 2MHz wide and usually about 1MHz wide when making the final adjustments. For a 5MHz spread you need 6 or 7MHz span to see both sets and you loose sight of the fine details of the notches. Once you get a set of notches in the ball park narrow up the span so you can see exactly what you are doing. And don't forget to check insertion loss from antenna port to each output before calling it a day.
Even with all that I have used some of my lower end two port Chinese antenna analyzers for a basic sanity check on a duplexer and to rough them in before fine tuning with a better instrument.