v6 actually has longer headers on average (40B instead of 20B) but is generally more efficient to process because it's a fixed header length.

ON the other hand, theoretically IPv4 header length could vary depending on the presence of IP options, so you'd need to compute the offset to continue processing the packet.

(I say theoretically because conventional wisdom is that IP options are unreliable in the face of middleboxes, so they're mostly unused. But compliant IPv4 processors have to calculate the header size anyways...)

And no CGNAT which should make a bigger difference. Xbox live supported IPv6 for years because gamers really care about latency.