LSD is actually a fairly popular party drug. At certain types of electronic music events it tends to be more popular than MDMA, even. With psychedelics, the environment has a profound effect on the experience. As such, people using it as a party drug may be seeking (and receiving) a different experience than a user taking it and going on a stroll through the woods.
I've actually known quite a lot of people who have used LSD to that end. The Mushroom, however, is a different, more spiritual journey that is very difficult to evade.