It's not so much a rule as it is that Boehner needs to keep a majority of his party behind him in order to remain speaker. If he goes against them, they just vote for someone else. Everyone assumes that there won't be any defectors from either party in that scenario, so whoever gets the majority of the Republicans is the Speaker.

A majority of the Republicans is hardly a small fraction in the grand scheme of ways things can get blocked in Congress. Committee chairs block things essentially on their own all the time.