A hobbyist small UAV builder can indeed cause a lot of damage if he has evil intent. For instance, a small UAV with half a pound of high-explosive could glide over a "secure area" and try to hit a high-value civilian target. Much easier than detonating a car-bomb or trying to take someone out with a sniper rifle. Sharpshooters will have to scan the skies too (if they're not doing it already).

Fortunately, it's not exactly easy to buy IMU's, otherwise we would have heard of guided-missile hobbyists doing evil things. Also fortunately, GPS shuts down above a certain altitude and speed (not applicable to GPS units belonging to the U.S. military).

There are so many more people that know how to drive a car that this kind of threat is still nothing to worry about.

Lots of car bombers don't know there is a bomb in the car before it is remotely detonated.

Also, snipers are really, really effective, and again, are much more common.

Snipers are effective, but they can be detected using laser technology, and they can be neutralized by sharpshooters. What kind of counter-measure could one use against a UAV-bomb?