A common ABI doesn't save anything as we still need to build for ARM, and x86 (MIPS, RISCV are also out there and may be important to you). Those processors all have different generations, it might be worth having a build for each variant of your CPUs. Once you take care of that different ABIs are just a trivial extension. RPM and .deb have been able to handle this for years.