I basically only update when (a beta of) Xcode tells me it won't run on my current version. Usually that's the point when either all bugs have been fixed or they will not be fixed before the new version.

Yea, Xcode is so annoying with this. What magical features does it use under the hood to not allow basic functionality and iOS support that seems completely unrelated to the macOS version? I have zero incentive to update macOS except Xcode telling me it needs a new version that only runs on the new one.

Xcode still needs to compile for macOS of course. It's not just for iOS development. If you want to compile for the next version of macOS you need to be on the next version.

But why? You don't need to be running the latest version of the Linux kernel to compile binaries for it. You don't need to run Windows 10 to compile programs for it. Why does Apple's compiler need to run on the same system its build target is for?

Probably also to run the actual application. But a Linux kernel is a different beast than macOS versions. macOS versions are pretty stale in terms of features to change abruptly with a new release.

But I think if you keep compiling for older versions you should be able to stay on an older version for a while without newer versions of the OS refusing to run it.

It's just that sometimes new features are introduced that require you to change something in your application because there's a new or deprecated framework. Apple likes to break things to not drag a lot of legacy around.