It seems like there could be some sort of cross origin cache header that could be set? That way by default you couldn't store in a global cache but server operators would be able to mark certain resources that could be.
Server operators would just set that header on everything. There's no downside for them. The privacy issue only impacts users and it would still exist even if only a subset of libraries were shared.
