I addressed this elsewhere (https://news.ycombinator.com/item?id=16072690) but to quickly recap: that's not the hard problem, and hardened SAML IdPs that have the option of exploiting this turn out to have per-tenant keys anyway so that they can get cryptographic binding instead of counting on audience restrictions being checked.

Additionally, your TLS terminating stack is much better hardened than median in-app crypto code.