I agree. I think you may have misunderstood my comment. The wire representation should be read into memory via any byte order manipulations that are needed.
Why "every time you read the variable" then? You'd have a packed-binary-struct and a memory-representation-struct and conversions between them to use when reading or writing. I say "would", but I actually already do this. I have typedefs of the kind
typedef union {
uint8_t bytes[sizeof(uint64_t)];
uint64_t native; /* alignment hint */
} uint64le_t;
which I use in structs of my on-disk data structures.
It would, in the worst case, not be worse than what the original poster is suggesting.
And at least in that case, the code would be more readable.
But, as the compiler knows more about what's going on (it's not just parsing and compiling a general expression with ORs and shifts) then it could well be faster (e.g. if there were a CPU instruction to do this, then it could be used, etc.).