Your best bet is to find some kind of East Asian brand(Uniqlo or H&M for example but they are fast fashion tier places so I'm not actually recommending those specifically) where smalls are actually small and fit nicely on smaller frames.
I actually do have a bunch of shirts from Uniqlo. The problem, however, is while they mostly fit nicely (i.e., they aren't designed for fat people), they're not quite long enough for me, so they frequently become untucked. Their idea of a "medium" guy is a guy who's 5'7 at the most, I'm guessing, and I'm over 6'. This isn't that much of a problem, though, compared to the long-sleeve shirts: on those the sleeves are an inch or two too short, because again they're not thinking a "medium" is a guy who's over 6' with long arms. I have gotten pretty lucky with some of the short-sleeve shirts though.
Have you looked into Everlane? They're quite more expensive but there's a range of sizing that they provide with exact measurements and from what I read their clothes are designed for more European/Asian bodytypes instead of Americans. I also like the philosophy behind their company/brand which you can read on their website.
