Even without dust in the air, you have things like Rayleigh scattering. But yes, we only "see" the photons that were deflected from their original path.