tag:blogger.com,1999:blog-738485353871431380.post5807588667880485709..comments2023-08-14T03:30:09.330-07:00Comments on Crystal Prison Zone: Bayes Factor: Asking the Right QuestionsJoehttp://www.blogger.com/profile/10825531253125205466noreply@blogger.comBlogger13125tag:blogger.com,1999:blog-738485353871431380.post-29338129124667874232015-04-22T07:27:09.758-07:002015-04-22T07:27:09.758-07:00That's OK. But in general, it's a good ide...That's OK. But in general, it's a good idea to read a paper before commenting on it!David Colquhounhttp://www.dcscience.net/noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-14813230037636201132015-04-21T10:58:08.601-07:002015-04-21T10:58:08.601-07:00You cut the statement off a bit too early there. T...You cut the statement off a bit too early there. They are only problematic in the same sense that the priors needed in Bayesian analysis are problematic - they also require additional specification that many psychologists don't do. <br /><br />I have not yet seen your FDR analysis itself and did not wish to impugn those results, as I have no doubt that you have done extensive work supporting its effectiveness. I also did not intend to make this a long discussion and was stating an initial impression of mine that was intended to be considered a throw-away statement to be considered and then cast aside. I apologize for my mistakes and any over-reaching that was carried out in my comment.Jonathan Thielenoreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-86311921636712161882015-04-21T00:22:07.046-07:002015-04-21T00:22:07.046-07:00You say
"My take on calculating the "fa...You say <br />"My take on calculating the "false discovery rate" is that it also requires a specification of an alternative model in order to calculate it correctly which, unless I'm mistaken, is problematic"<br /><br />Perhaps you would be so kind as to point out what you think is wrong with the results on FDR that I got from simulated t tests?<br /><br />And perhaps you could also point out the mistakes being made by Valen Johnson in his approach to the problem via uniformly most-powerful Bayesian tests. That approach gives similar results to mine.David Colquhounhttp://www.dcscience.net/noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-36179895464840048712015-04-21T00:14:34.590-07:002015-04-21T00:14:34.590-07:00Richard Morey
If it's the case that the point ...Richard Morey<br />If it's the case that the point null is nonsense, then several generations of statisticians who have taught null hypothesis testing (including RA Fisher) have been misleading us badly. Is that what you're saying?<br /><br />In my opinion what scientists want is to be able to say whether or not an observed difference is consistent with random chance, or whether there is evidence for a real effect. Of course, in the latter case you would estimate the effect size and decide whether or not it was big enough to matter in practice. <br /><br />The problem with P values is that they don't tell you what you want to know in practice. What you want to know is what your chances are of being wrong if you claim there's a real effect, i.e. the false discovery rate. The lack of a totally unambiguous way of calculating the FDR is a problem, but it's possible to put lower limits on the FDR, and that limit is high enough that it's important to change, at least, the words that are used to describe P values. My suggestions are at http://rsos.royalsocietypublishing.org/content/1/3/140216#comment-1889100957 Steven Goodman has made similar suggestions.<br /><br />Perhaps, if you are still sceptical, you could point out to me what's wrong with the simulated t tests in my paper. They mimic exactly what's done usually in practice. David Colquhounhttp://www.dcscience.net/noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-14665884506453750942015-04-20T15:13:07.173-07:002015-04-20T15:13:07.173-07:00"The conventional null hypothesis is that the..."The conventional null hypothesis is that the difference between means is zero, and the alternative hypothesis is that it's not zero. They're precisely what we want to test."<br /><br />What sort of hypothesis is "not zero"? It makes no predictions, and is completely unfalsifiable. It is unscientific. How do you test it? We want -- need, actually -- hypotheses that make connections with the data. Anything else is unacceptable. You can call that subjective if you like, but it is really just basic a scientific desiderata. Any scientist who says "I think that effect is not zero" is certainly in no risk of being proven wrong; that's because the hypothesis is a unconstrained nonsense.<br /><br />Once one accepts the fact that any hypothesis must make predictions, then the question becomes how to make those predictions. That's what a prior does.Richard Moreyhttps://www.blogger.com/profile/11319149283079163004noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-87483885825584096952015-04-20T15:05:13.673-07:002015-04-20T15:05:13.673-07:00[Again posting on behalf of Dr. Calquhoun.]
David...[Again posting on behalf of Dr. Calquhoun.]<br /><br />David Calquhoun says:<br />My thinking on this topic started out on my blog, See <br />http://www.dcscience.net/2014/03/10/on-the-hazards-of-significance-testing-part-1-screening/<br /><br />and<br /><br />http://www.dcscience.net/2014/03/24/on-the-hazards-of-significance-testing-part-2-the-false-discovery-rate-or-how-not-to-make-a-fool-of-yourself-with-p-values/<br /><br />Eventually, after 4 months on arXiv it evolved into a proper paper<br />http://rsos.royalsocietypublishing.org/content/1/3/140216<br /> <br />and lastly a simplified version on Youtube <br />https://www.youtube.com/watch?v=tRZMD1cYX_cJoehttps://www.blogger.com/profile/10825531253125205466noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-8858308281484544712015-04-20T14:48:28.391-07:002015-04-20T14:48:28.391-07:00I have to say that I am not familiar with the &quo...I have to say that I am not familiar with the "Objective Bayesian" school of thought, but it has always seemed to me a contradiction in terms. Placing expectations on effect sizes will always require some judicious but subjective decision-making, as I understand it. So on the face of it, a method of inference that is wholly without subjectivity in the allocation of priors seems impossible -- the inferential equivalent of perpetual motion.<br /><br />That said, I'll do my best to read the cited literature, but it seems there's always some form of lurking, unstated assumption or prior (e.g. as above, "every reasonable prevalence") or equivocation (e.g. also as above, "at least 26%") that has been chosen to make the machinery work.<br /><br />I lean toward subjective Bayes because 1) the assumptions, models, etc are all stated plainly so that limitations are obvious and 2) it's not difficult to make fair and reasonable subjective decisions that are broadly appropriate. So while there cannot always be objective justification, I don't think that makes the priors weird, and certainly does not prevent -subjective- justification.Joehttps://www.blogger.com/profile/10825531253125205466noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-70219428474258432092015-04-20T14:48:17.843-07:002015-04-20T14:48:17.843-07:00David, This is inaccurate. Can you provide a citat...David, This is inaccurate. Can you provide a citation or some argument for the claim that "depends entirely on the subjective guess of the shape of prior distributions?" It is true that if one chooses absolutely ridiculous, indefensible priors, then one can get ridiculous answers (I call these "Rush Limbaugh priors" because Rush is so fixated on his answers that data cannot shift his beliefs!). Yet, this argument carries no credibility as researchers need to justify the reasonableness of their priors. They can be neither too thin nor too fat to be justifiable. There will be some subjectivity on this determination, but in my experience and in my writings, this subjectivity has minimal impact. See for example http://pcl.missouri.edu/sites/default/files/p_8.pdf<br /><br />Maybe we are talking past each other. I have not gotten into FDR too much. It seems to be marginalizing over things I am uninterested in. Maybe a some blog posts about why FDR is so important and how subjectivity makes it unattainable are in order. You are welcome to guest write on my blog, Invariances, http://jeffrouder.blogspot.com if you dont have your own. Jeff Rouderhttps://www.blogger.com/profile/12042232118911308833noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-26586226669801281672015-04-20T14:37:52.683-07:002015-04-20T14:37:52.683-07:00[Posting on behalf of Dr. Colquhoun due to problem...[Posting on behalf of Dr. Colquhoun due to problems with the comment software. -- Joe]<br /><br />David Calquhoun says:<br />On the contrary, it's subjective Bayesians who put themselves ahead of the data. They give a answer that depends entirely on subjective guess of the shape of prior distributions. Or, still less usefully, they give a whole range of answers.<br /><br />The conventional null hypothesis is that the difference between means is zero, and the alternative hypothesis is that it's not zero. They're precisely what we want to test. Within that framework, every reasonable prevalence of real effects (i.e. 0.5 or less) gives a false discovery rate of at least 26%. <br /><br />DavidJoehttps://www.blogger.com/profile/10825531253125205466noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-24181408954413527842015-04-20T13:22:18.896-07:002015-04-20T13:22:18.896-07:00My take on calculating the "false discovery r...My take on calculating the "false discovery rate" is that it also requires a specification of an alternative model in order to calculate it correctly which, unless I'm mistaken, is problematic for the same reason that the "subjective Bayesian who feels free to postulate weird sorts of prior for which there is no objective justification" is problematic. You're still specifying something that isn't necessarily objectively justifiable but go on as if it does because it "makes sense."<br /><br />That aside, focusing on false discovery rate itself misses the point of research because it removes the researcher from the analysis and then places them in the spotlight for making such a grand discovery instead of putting them behind the spotlight so that they can highlight the interesting parts of the data. And, last I checked, (good) science is supposed to be concerned with what the data say and not so much with the people who generated it. <br /><br />Of course there will be some bias, some false discovery, and some parts of the data that were overlooked because of some weird prior specifications, but being so preoccupied with their existences that you obsess over removing them completely doesn't produce useful results. Acknowledging that your first estimates are off for those reasons or that your methods didn't consider some alternative and then re-calibrating and correcting for those errors in the next iteration does.Jonathan Thielenoreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-19022078593559180982015-04-20T12:57:59.169-07:002015-04-20T12:57:59.169-07:00No comment, just subscribing to this thread. Promi...No comment, just subscribing to this thread. Promises to be interesting.Antiquated Toryhttps://www.blogger.com/profile/08727187247287268014noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-42994874599632949542015-04-20T12:35:21.175-07:002015-04-20T12:35:21.175-07:00David, Is it not true that most objective justific...David, Is it not true that most objective justifications are based on frequentist-ispired large-sample properties? What possible Bayesian justification could there be? As a subjectivist who follows Lindley and Goldstein, I think getting different answers across different analysts who specify different priors is no problem at all. Priors are part of the model because they affect the predicted distribution of data (that is, I view models as statements about data). We always get different answers when we posit different models, and this is true of frequentist and objective Bayesians, so I don't see how it is a proper critique of subjective Bayesians. Likewise, the inference from a model is conditional on the reasonableness of the model specification, so priors need to be defended of course, but that is a matter of rhetoric and context, and not a matter of long-run property of resulting posteriors. Jeff Rouderhttps://www.blogger.com/profile/12042232118911308833noreply@blogger.comtag:blogger.com,1999:blog-738485353871431380.post-71312289103671235042015-04-20T12:17:31.162-07:002015-04-20T12:17:31.162-07:00The problem with this post is that it still doesn&...The problem with this post is that it still doesn't answer the question that most experimenters want to ask, namely what is the chance that I'll be wrong if I claim that there is a real effect. In other words what experimenters want to know is the false discovery rate (a surprisingly large number of them make the mistake of thinking that's what the P value gives).<br /><br />On the basis of simulated t tests, I maintain that if you observe P = 0.047 in a single test, the false discovery rate is at least 30%. See http://rsos.royalsocietypublishing.org/content/1/3/140216<br /><br /> This agrees quite well with the results of Sellke & Berger, and of Valen Johnson.<br /><br />It's true that this result depends on assumption of a point null hypothesis. That seems to be the only sort of null hypothesis that makes much sense. We want to see whether or not our results are consistent with both groups being given exactly the same treatment. Of course you can get different answers if you are a subjective Bayesian who feels free to postulate weird sorts of prior for which there is no objective justification.<br />David Colquhounhttps://www.blogger.com/profile/07610935223901133825noreply@blogger.com