tag:blogger.com,1999:blog-738485353871431380.post2007479208379504516..comments2024-03-23T13:15:48.445-07:00Comments on Crystal Prison Zone: Curiously Strong effectsJoehttp://www.blogger.com/profile/10825531253125205466noreply@blogger.comBlogger16125tag:blogger.com,1999:blog-738485353871431380.post-34834491363212089922017-05-20T08:12:39.762-07:002017-05-20T08:12:39.762-07:00An interesting problem, but one that could be elim...An interesting problem, but one that could be eliminated by tightening standards for publication. Sample size of submitted manuscripts should be large enough to support a split-half test of the hypothesis. If the finding of the first half is replicated by the second, then the manuscript, if otherwise acceptable, can be published. This will increase the cost of conducting research and likely delay the publication of findings. But what do we really have to lose? Replicability and replication is at the heart of empirical science. We face a replication crisis that is causing erosion of public trust in the scientific enterprise. See: A manifesto for reproducible science<br />Marcus R. Munafò, Brian A. Nosek, Dorothy V. M. Bishop, Katherine S. Button, Christopher D. Chambers, Nathalie Percie du Sert, Uri Simonsohn, Eric-Jan Wagenmakers, Jennifer J. Ware & John P. A. Ioannidis<br />Nature Human Behaviour 1, Article number: 0021 (2017)<br />doi:10.1038/s41562-016-0021John H Noble Jrhttps://www.blogger.com/profile/17989392954317389331noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-59625056612441469492017-05-16T14:57:59.501-07:002017-05-16T14:57:59.501-07:00Interesting post. I agree with those comments poin...Interesting post. I agree with those comments pointing the finger at study design, only to add that failing to take into account the underlying structure of the data can generate unbelievably-high effect sizes. Yule pointed this out almost a hundred years ago with regards to nonsense correlations in time-series (https://www.jstor.org/stable/2341482?seq=1#page_scan_tab_contents). Dan Hruschka and I have a commentary coming out soon on a study that looked at the effect of the ebola outbreak on voting intentions using time-series data, without accounting for autocorrelation (i.e non-independence) of data points. This, combined with the fact that the original paper used smoothed data instead of raw data, greatly inflated the estimates of effect-size.<br /> (http://leotiokhin.com/assets/uploads/Tiokhin_Hruschka_Ebola_Elections_2016_Merged.pdf)Anonymoushttps://www.blogger.com/profile/17374347841980277237noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-64397786017471036552017-05-16T11:17:52.081-07:002017-05-16T11:17:52.081-07:00This is a great situation where Simonsohn's &q...This is a great situation where Simonsohn's "small telescopes" approach (http://datacolada.org/wp-content/uploads/2015/05/Small-Telescopes-Published.pdf) is useful - you can adjust your sample size so that you can draw conclusions about whether the original study was adequately powered to detect any effect that could be there. <br />-SimineUnknownhttps://www.blogger.com/profile/04212003102411142870noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-4407023727918095722017-05-15T18:41:22.127-07:002017-05-15T18:41:22.127-07:00Surely, replication is the only way for science to...Surely, replication is the only way for science to proceed, irrespective of the size of the effect. There are three things we (should) want to know: (1) is the finding reliable? (i.e., can it be replicated in an independent sample of subjects); (2) how big is it?; and (3) how general is it? P values don't address these questions; only doing the hard work does ...<br /><br />Geoff Hammondhttps://www.blogger.com/profile/06787242437033453489noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-8049240389160056302017-05-15T15:12:59.118-07:002017-05-15T15:12:59.118-07:00It might be constructive to transform the effect s...It might be constructive to transform the effect sizes into the r-metric, where we have more benchmarks. A d of 2.5 is going to be equivalent to a correlation .78. That's a correlation that exceeds the average reliability of most measures used in the social sciences (.75) and would be correlation used to make the argument that the IV and DV are the same thing. That might make a more compelling argument.<br />Anonymoushttps://www.blogger.com/profile/10671573769384037110noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-56812934334732275312017-05-15T14:57:29.234-07:002017-05-15T14:57:29.234-07:00To address “maybe the effect is overestimated beca...To address “maybe the effect is overestimated because of demand effects,” you could see if the curiously strong effect correlates with the Perceived Awareness of the Research Hypothesis scale: https://osf.io/preprints/psyarxiv/m2jgn/Mark Rubinhttps://www.newcastle.edu.au/profile/mark-rubinnoreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-45382980163087608252017-05-15T14:32:46.796-07:002017-05-15T14:32:46.796-07:00This is a really important point to keep emphasisi...This is a really important point to keep emphasising the research community. <br /><br />There are some things worth checking if you see large d or r values. My initial thought on seeing such large standardised effects (usually a point estimate) is that the sample size is small and the effect size is inevitable. My second thought is to wonder how they have computed d or r - it is quite easy to use an incorrect conversion formula or some other approach that inflates d (e.g., computing d from t with a within-subject design). Third, there are artefacts of the design that distort d or r and combined with choice of computational approach produce large d or r values. Ceiling and floor effects are an example as they can dramatically shrink the sample variance as can ecological correlations. Finally, there are study designs that produce large effects by increasing the strength of a manipulation such as extreme group designs.<br /><br />In most cases it really helps to get a measure of effects size that is unstandardized to give more context (or equivalently a raw data plot).thomhttps://www.blogger.com/profile/00392478801981388165noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-12050111284177813702017-05-15T13:29:34.762-07:002017-05-15T13:29:34.762-07:00Fair point!Fair point!Joehttps://www.blogger.com/profile/10825531253125205466noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-8932704824770856562017-05-15T12:36:55.466-07:002017-05-15T12:36:55.466-07:00Then it's no longer a 'curiously large eff...Then it's no longer a 'curiously large effect' to worry aboutProfessor Keith R Lawshttps://www.blogger.com/profile/11760248140027990471noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-55732793497170389472017-05-15T12:25:25.028-07:002017-05-15T12:25:25.028-07:00It sounds like you and Dr. LeBel both see value in...It sounds like you and Dr. LeBel both see value in a replication attempt. The thought had occurred to me, but it seems a shame to do all the set-up only to collect N = 10. I'm also concerned about the possibility of a fighting retreat: "Well, no, so it's not d = 2.5, but maybe it's d = 0.5, which your replication wouldn't detect." Perhaps I'm overthinking it and a little replication would go a long way. Joehttps://www.blogger.com/profile/10825531253125205466noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-62352114609059687992017-05-15T12:20:19.345-07:002017-05-15T12:20:19.345-07:00I guess you could calculate the sample size needed...I guess you could calculate the sample size needed for a 'curiously large effect. it would be wonderfully small - if d=2.5, then you would probably need only 5 per group (to get 95% power). Why not then get multiple labs to replicate using variations around these sample sizes and 'bobs your uncle' - see how many replicate... then pool them all together in a meta-analysis ...add a few moderator variables ...ready for publicationProfessor Keith R Lawshttps://www.blogger.com/profile/11760248140027990471noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-89737044238725437072017-05-15T10:48:05.385-07:002017-05-15T10:48:05.385-07:00One might at least expect authors to point out the...One might at least expect authors to point out their curiously strong effects and offer their thoughts on the matter.Daniel Ozernoreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-38607193096717504312017-05-15T10:33:54.271-07:002017-05-15T10:33:54.271-07:00Good post. BTW, a curiously strong effect (spotted...Good post. BTW, a curiously strong effect (spotted by Richard Morey) played a role in a recent Psych Science retraction. Also, as others have noted, when power is low an effect will be significant only if it exaggerates true effect size. Be deeply skeptical of low-powered, single-experiment studies with surprising results.<br />SteveAnonymoushttps://www.blogger.com/profile/07485890516098106807noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-3137263308566452132017-05-15T10:05:44.839-07:002017-05-15T10:05:44.839-07:00Interesting and valuable blog post! A few quick co...Interesting and valuable blog post! A few quick comments on specific points:<br /><br /><br />>>>The only solution I can see, barring some corroborating evidence that leads to retraction, is to try to replicate the curiously strong effect. Unfortunately, that takes time and expense, especially considering how replications are often expected to collect substantially more data than original studies. <br /><br />Yes! Independent corroboration via strong falsification attempts via replicability tests **IS** the only fail-proof way to increase one's belief confidence in a curiously strong, or any, published effect. Falsifiability is simply **NOT** optional for scientific progress to be possible: https://osf.io/preprints/psyarxiv/dv94b/ <br /><br /><br />>>>>Even after the failure to replicate, one has to spend another 3 or 5 years arguing about why the effect was found in the original study but not in the replication. <br /><br />No, this is not necessary. Our job isn't to figure out **why** certain published findings are false (there could be a thousand different reasons why original researchers got it wrong). Our job is to better understand reality by building upon each others' findings, which are assumed to be (in principle) replicable according to the specified conditions. If independent labs, in good faith, cannot demonstrate to themselves that a published effect is replicable, then they must simply move onto investigating other effects that are indeed replicable in the hope of increasing our understanding of the world.Etienne LeBelhttps://www.blogger.com/profile/02395031595701212100noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-89068416670029185952017-05-15T09:52:42.642-07:002017-05-15T09:52:42.642-07:00Sadly, I suspect that fabrication will very often ...Sadly, I suspect that fabrication will very often turn out to be the most parsimonious explanation. Fraudulent researchers tend to be rather incompetent, including in their understanding of what a reasonable effect size might be.Nick Brownhttps://www.blogger.com/profile/18266307287741345798noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-80155882648602323672017-05-15T07:29:08.295-07:002017-05-15T07:29:08.295-07:00Great post. This is one advantage that I see of re...Great post. This is one advantage that I see of results-blind reviewing. In my experience, many of these curiously strong effects happen with very small samples. If we were to evaluate the study without knowing the results, we would likely say it had too little power/precision to produce meaningful results. Focusing on the design takes the pressure off of having to address what produced those results.<br />Of course this only applies to cases where we'd be likely to think the design is weak.<br />-SimineAnonymoushttps://www.blogger.com/profile/11896349540651366259noreply@blogger.com