The software doesn't suddenly become incompatible with CPU-specific optimisations (or many other compiler flags that change its output), but if you do so, you won't be able to reproduce the distribution binaries. Distributions don't enable CPU-specific optimisations anyway, since they want to be usable on more than one CPU model.