The ACM.org website published the work of a team at Carnegie Mellon (#CMU) which was said to include source code. Then the code was omitted from the attached ZIP file, which only contained another copy of the paper. I asked the lead researcher (a prof) for the code and was ignored. Also asked the other researchers (apparently students), who also ignored the request. The code would have made it possible to reproduce the research and verify it. ACM ~~also ignored my request and~~ also neglected to fix the misinfo (the claim on the page that source code is available). Correction: ACM replied and tried to find the missing code but then just gave up.

It seems like this should taint the research in some way. Why don’t they want people reproducing the research? If the idea is that scientific research is “peer reviewed” for integrity, it seems like a façade if reviewers don’t have a voice. Or is there some kind of 3rd party who would call this out?

Are You Eating a Credit Card Every Week?

No, but it is an interesting example of how misinformation can be created and spread, even without malicious intent. Video is a couple years old, but it still holds up.

cross-posted from: https://beehaw.org/post/15160546 | ghost archive | Excerpts:

... findings with null or negative results — those that fail to find a relationship between variables or groups, or that go against the preconceived hypothesis — gather dust in favour of studies with positive or significant findings. A 2022 survey of scientists in France, for instance, found that 75% were willing to publish null results they had produced, but only 12.5% were able to do so2. Over time, this bias in publications distorts the scientific record, and a focus on significant results can encourage researchers to selectively report their data or exaggerate the statistical importance of their findings. It also wastes time and money, because researchers might duplicate studies that had already been conducted but not published. Some evidence suggests that the problem is getting worse, with fewer negative results seeing the light of day3 over time.

At the crux of both academic misconduct and publication bias is the same ‘publish or perish’ culture, perpetuated by academic institutions, research funders, scholarly journals and scientists themselves, that rewards researchers when they publish findings in prestigious venues, Scheel says.

But these academic gatekeepers have biases, say some critics, who argue that funders and top-tier journals often crave novelty and attention-grabbing findings. Journal editors worry that pages full of null results will attract fewer readers, says Simine Vazire, a psychologist at the University of Melbourne in Australia and editor of the journal Psychological Science.

One of the most significant changes to come out of the replication crisis is the expansion of preregistration (see ‘Registrations on the rise’), in which researchers must state their hypothesis and the outcomes they intend to measure in a public database at the outset of their study (this is already the norm in clinical trials). ... Preliminary data look promising: when Scheel and her colleagues compared the results of 71 registered reports with a random sample of 152 standard psychology manuscripts, they found that 44% of the registered reports had positive results, compared with 96% of the standard publications^7^ (see ‘Intent to publish’). And Nosek and his colleagues found that reviewers scored psychology and neuroscience registered reports higher on metrics of research rigour and quality compared with papers published under the standard model^8^.