This means you're using SLAAC instead of DHCPv6, right? In that case, your servers should have a fixed IPv6 address derived from their MAC address. Even if they also have privacy addresses (and prefer to use them for outbound connections), they should still have the non-privacy address which you could use for inbound connections.
Or is it the problem that, when your router reboots, the IPv6 prefix changes? That is a harder problem (and would be the same in IPv4).
This means you're using SLAAC instead of DHCPv6, right? In that case, your servers should have a fixed IPv6 address derived from their MAC address. Even if they also have privacy addresses (and prefer to use them for outbound connections), they should still have the non-privacy address which you could use for inbound connections.
Or is it the problem that, when your router reboots, the IPv6 prefix changes? That is a harder problem (and would be the same in IPv4).