Only completely ignorant people[1] would have called you paranoid. Any device left to the mercy of such vulnerable software cannot be trusted. Even if societies pushed back against this surveillance machine and there were someday open, transparent modes of government -- even then -- still only fools would mock others who act out of caution and practical wisdom. RATting alone is reason enough, nevermind basic hacks that lead to total compromise. Webcams and mics have been easy targets since they've first existed in the mainstream (mid 90s). Further, if one's rightly concerned, unplug XBOX One and such devices when they're not in use. Built-in mics are the truly annoying part to counteract.

[1] only the vast majority of people (both computer savvy and illiterate)