Peer review doesn't tell you if the data is valid or not. they published their methodology and anyone is free to repeat the study.
Peer review just checks for obvious errors in study design, asks for more info if needed, and decides whether the paper is a good fit for the journal.
Watson and Crick's paper describing the structure of DNA wasn't peer reviewed. if you think they're wrong, try it for yourself and publish the results.
When a few groups all get the same result then you can be confident about the claims made. until then, it's just kind of interesting to think about, which is fine.
> A.J.K.P. and S.W.C. are co-founders and co-directors of Circadian Health Innovations PTY LTD
I do agree that this paper alone should not be used to help sell a product. But it looks like this paper just confirms previous findings using more rigorous methodology (see background):
"Light at night causes circadian disruption, (21–23) and is therefore a potential determinant of cardiovascular disease risk. Higher risks for coronary artery disease (24) and stroke (25) have been observed in people living in urban environments with brighter outdoor night light, as measured by satellite. Brighter night light has been cross-sectionally related to atherosclerosis, (26,27) obesity, hypertension, and diabetes (28) in small but well-characterized cohorts, using bedroom (26,27) and wrist-worn (28) light sensors. Moreover, experimental exposure to night light elevates heart rate and alters sympathovagal balance. (29) However, current evidence linking night light with cardiovascular risk is mostly within small cohorts, or relies on geospatial-level measurements of outdoor lighting, rather than measures of personal light exposure. (30,31)"
> Peer review doesn't tell you if the data is valid or not.
Sure but nobody claimed that.
> Watson and Crick's paper describing the structure of DNA wasn't peer reviewed.
I'd point out that outliers exist but that was before peer review become so popular.
Right now there's a good correlation between competency and peer review.
> if you think they're wrong, try it for yourself and publish the results.
Watson and Crick or the article?
For a balanced discussion of the article, it's reasonable to point out a lack of peer review to give context to what stage this is at. If "try it yourself" is the bar then I guess nobody comments? That doesn't seem like a good way to learn anything.
>> Watson and Crick's paper describing the structure of DNA wasn't peer reviewed.
> I'd point out that outliers exist but that was before peer review become so popular
what outlier? I just picked a famous example, there are almost infinitely many examples to choose from...
>> if you think they're wrong, try it for yourself and publish the results.
> Watson and Crick or the article?
yes
> For a balanced discussion of the article, it's reasonable to point out a lack of peer review to give context to what stage this is at.
The first thing that the pre-print says, in bold at the top of the page, is that this is a non-peer-reviewed article and shouldn't be used for clinical practice. so commenting "it's not peer reviewed" doesn't add anything
> If "try it yourself" is the bar then I guess nobody comments? That doesn't seem like a good way to learn anything.
"try it yourself" is the bar for determining the validity of the results. A comment section is not going to be able to determine the validity. My whole point is that it's worth discussing the article without waiting for a final peer-reviewed version of it. If you disagree with the results, you can point out a perceived flaw in the study or find papers which contradict the results so we can discuss something concrete
> what outlier? I just picked a famous example, there are almost infinitely many examples to choose from...
And there's even more almost infinitely many examples that say peer review is a strong signal.
> yes
If you were including the former, you were making a very rude argument by implying that anyone that values peer review is rendered invalid by that example.
> My whole point is that it's worth discussing the article without waiting for a final peer-reviewed version of it.
Telling people to shut up about peer review is bad for discussion.
I replied to the comment "Preprint not peer reviewed." which added nothing and arguably shuts down discussion.
My whole point is that it's ok to find research interesting and discuss it even though it's not peer reviewed yet.
> If you were including the former, you were making a very rude argument by implying that anyone that values peer review is rendered invalid by that example
No, I'm pointing out that not being peer reviewed is not automatically disqualifying and that the real way that or prove/disprove science is by replication attempts, not through peer review.
> And there's even more almost infinitely many examples that say peer review is a strong signal.
So you say, but if you think about it all papers in the ongoing replication crisis are peer reviewed. I know several peer reviewed papers which have inaccurate results, and in my experience having been on both sides of the peer review process I can tell you that it's pretty flawed since very few scientists are willing to invest a lot of their own time to do meticulous unpaid review of other people's work. Meanwhile, science progressed fine before peer review became standard in the 1970's.
> Telling people to shut up about peer review is bad for discussion.
I'll keep that in mind for the future but doesn't apply to anything I said. Maybe you should take a few minutes to read what I actually wrote instead of reacting emotionally
This creates a perfect setup for manipulation - the high barriers to entry for proper equipment, organization, and funding needed to produce quality reproductions mean that if someone posts fake content that mimics scientific paper formatting and includes all the right academic signals, most people will accept it as legitimate without question.
it's the opposite. no doctor or insurance is going to change their policy based on a single result, you wait until a few separate groups have replicated the results. In the scientific community there is a reward (being cited) for pointing out that someone else's finding was wrong. The more important the initial finding, the more citations (and attention) you will get if you're the first one to correct the record. There's also a reward for building on an initial finding and going further, but you can only do that if the original finding stands. So one way or another the truth will come out.
Publishing a pre-print is only valuable if your result hold long term. You're just stating that you did it first.
The Beth Israel Deaconess Medical Center ran a trial on parachute use to prevent death and major trauma when jumping from aircraft, and gave half the people jumping a placebo (empty backpack) instead of a parachute. No joke.
OK, I read the article, wondered a bit, then started to read the responses (first aligning with the one "April Fools' in December?"). I mean, what's the point of asking people to jump from airplanes on the ground, write a study on it, and waste everybody's time
But then, reading other responses, and taking into account the name of the journal, and the fact that it did get published, I realized it's a very good example of a flawed study, where everybody see a flaw because it's evident, whereas in other studies a flaw of the same magnitude might be quite difficult to see (I could give quite a few examples from my field).
> Parachutes are routinely used to prevent death or major traumatic injury among individuals jumping from aircraft. However, evidence supporting the efficacy of parachutes is weak and guideline recommendations for their use are principally based on biological plausibility and expert opinion.
Results :
> Parachute use did not significantly reduce death or major injury (0% for parachute v 0% for control; P>0.9). This finding was consistent across multiple subgroups.
Methodology might have an influence on the result:
> Compared with individuals screened but not enrolled, participants included in the study were on aircraft at significantly lower altitude (mean of 0.6 m for participants v mean of 9146 m for non-participants; P<0.001) and lower velocity (mean of 0 km/h v mean of 800 km/h; P<0.001).
Conclusions
> Parachute use did not reduce death or major traumatic injury when jumping from aircraft in the first randomized evaluation of this intervention.
Implications
> Should our results be reproduced in future studies, the end of routine parachute use during jumps from aircraft could save the global economy billions of dollars spent annually to prevent injuries related to gravitational challenge.
Limitations
> However, the trial was only able to enroll participants on small stationary aircraft on the ground, suggesting cautious extrapolation to high altitude jumps.
I can already see the YouTube thumbnails: THIS IS THE STUDY PARACHUTE MAKERS DON'T WANT YOU TO SEE
We need to make it a lot cooler for researchers to not do original research all the time but to do replication and peer review. We should also demand publication of failures -- it's ok to fail, but only if you publish about the failure.
The whole publish-or-perish culture is a disaster that incentivizes cheating.
It should be considered just as valuable to have a few grad students working on replication as on original research, and that should not hurt the students' prospects.
an active, public pre-review discussion. we nurture critical thinking before the nudge.
some charismatic, intelligent dude or dudette or couple with a curious but rather uneducated mind inviting grad students and scientists to discuss studies and pre-prints and so on, constantly babbling scientifically more or less correct nonsense but getting semantically corrected by the guests or something ... ... so viewers can feel "smartypants" and or relate and learn.
I'm sure there's podcasts but I don't listen to podcasts.
> Also: ”A.J.K.P. and S.W.C. are co-founders and co-directors of Circadian Health Innovations PTY LTD.”
The other edge of the "not impartial" sword: these are people who are highly familiar, likely experts, in the related field. Who else is more qualified to conduct such a study?
> Who else is more qualified to conduct such a study?
Whoever does not have a monetary interest in the studied subject is the one who is more qualified. Same with smoking related research, same with children's toy paint related research. But the third edge is: These are phases of fraud that societies go through. They will happen, some will think they are legit, some won't, they disappear into some third world country after taking their sum. It always works like that. And that's predictive science right there.
> Whoever does not have a monetary interest in the studied subject is the one who is more qualified.
I’m not really understanding the monetary interest angle. The monetary interest in this subject is the curtain and window-blind manufacturers.
The researches here have a company who sells body worn light sensors. And if their light sensor is good maybe other researchers interested in verifying this phenomena will purchase a few units from them. But they are far from the only company making light sensors. In fact if i want to verify this study i would buy sensors from anyone but them. For the sake of having independent proof free from biases.
> they disappear into some third world country after taking their sum
What sum, please tell me? Where do you see the riches here?
You don’t have to pretend anything. Just go on and explain where you see the monetary incentives with this research and researchers. (If you see them.)
However, there is likely a correlation between nighttime light exposure and living in a city, on a lower floor, next to a street with medium or heavy traffic.
There is likely also a significant correlation between living in such conditions and exposure to pollution, especially inhaled particulate matter.
Such a correlation also holds elsewhere: living in a high-rise means no streetlights outside your window, and also yields cleaner air than living at or near street level; living in the countryside means practically no light pollution, but also much less traffic and thus much less particulates in the air.
Inhaled particulate matter is a leading cause of cardiovascular disease.
I've skimmed lots of studies about the relationship between light and health in the past.
In both humans and other animals, our internal sleep-wake clocks are synchronized to the day-night cycle through light exposure(and indeed, without it aren't even 24h-long), and have broad implications for mood, wakefulness, metabolism, and general health. I well might have seen something about cardiovascular problems specifically, though I don't remember.
Also: ”A.J.K.P. and S.W.C. are co-founders and co-directors of Circadian Health Innovations PTY LTD.”
Lemme guess, looking for funding.