That's because bikers (even well behaved ones) are annoying to deal with. Even if you are following all the rules, I still can have a hard time getting around you if the street is busy. Pretty frustrating dealing with a bicyclist climbing a hill at 5mph on a busy main road when you don't have quite enough room to safely get around them.
That's not even counting the bicyclists who blow through stop lights, stop signs, don't stop for pedestrians at crosswalks, ride on busy sidewalks.
In my honest opinion, I don't think bicycles belong on roads (at least in city areas).
I'm a very data driven person. Here's an experiment I did that ended up changing some of my driving habits.
The important data to me is my total driving time from A to B. So, I measured my total driving time to and from work. I also noted the number of bicycles I had to pass on that journey. If the cyclists were slowing down my total commute time, then there should be a trend in average driving times with 0, 1, or 2 cyclists encountered. No such correlation exists in my data. On average, red-lights and other traffic have more impact on my commute time than do delays caused by cyclists.
My conclusion: my emotional response to a cyclist was not proportional to any measurable delay. Now I wait calmly for a safe place to pass, secure in the knowledge that, on average, it doesn't delay me at all.
(obviously your commute time could be affected differently than mine, but I encourage you to look at the actual delay, and see if your frustration is warranted)
The slowdown effect is lower on the list of concerns than the knowledge that a slight mistake on the part of either parties probably results in the death or serious injury of the cyclist. I just find driving around cyclists stressful.
The closest comparison I can imagine would be if there were no sidewalks in cities and you had to deal with pedestrians on the side of every road as you were driving.
That's not even counting the bicyclists who blow through stop lights, stop signs, don't stop for pedestrians at crosswalks, ride on busy sidewalks.
In my honest opinion, I don't think bicycles belong on roads (at least in city areas).