You could argue that, in a sense, Google already punishes sites with annoying features like in-page popups and mandatory Facebook logins, in the same way it punishes all sites with annoying features - if you load a page and then hit the back button within a few seconds, Google knows the site it sent you to probably wasn't very useful and that will affect the page's rating. Add that to myriad other signals they use to detect whether you got value out of the site or not and I'd say they do a good job of punishing sites that people genuinely don't like.

What's probable though is is that although we don't like those things, other people aren't so bothered by them and still find utility in sites that use them, so Google doesn't punish them too much.