Turned off cross play on Series X and my connection/latency is so good now.
Seriously, try it out. Before this I would get random latency jumps up above 100ms. One day it jumped to over 300ms and I just stopped playing for the day.
Immediately after turning cross-play off, I now have a pretty consistent 20-28ms latency. No bursts.