I suppose emphasis can turn into exaggeration, then literally is indeed a bad fit. But it can be emphasizing something without exaggeration too.

I.e. "literally days ago". Which indeed can mean a few days ago, but it's emphasizing it. I don't think it's a wrong usage.