Stone is a brittle material that chips and shatters if looked at the wrong way. Stonecutters and masons were highly skilled professionals. What you're looking for are the masses of porters and carriers, unskilled workers moving construction materials.