"Changing out the 3d driver to be pluggable in a backend-agnostic way will be extremely hard/annoying to code around."

I hear that the Gallium3D project has done a substantial part of that work already:

https://en.wikipedia.org/wiki/Gallium3D