The difference is in being abusive or not, just like freedom is not absolute or in one direction; i.e. in the example of your freedom ending where someone else's freedom begins.