This post delivers an in-depth analysis of the significance of the statistical term Significance Level (in response to an article in Significance Journal).
In a focus article that has appeared in Significance magazine (October, 2013), the author Mark Kelly delivers an excellent review of what “luminaries have to say” regarding the proper significance level to use in statistical hypothesis testing. The author thence concludes:
“No one therefore has come up with an objective statistically based reasoning behind choosing the now ubiquitous 5% level, although there are objective reasons for levels above and below it. And no one is forcing us to choose 5% either.”
In a response article, sent to the editor of Significance, Julian Champkin, I have made the point that, unlike the claim made in the original article, there is an obvious method to determine objectively the optimal statistical significance level. While the editor accepted my article, he declined to include the detailed numerical example therein since “Your illustration, though, is a little too technical for some of our readers – we have many who are not statisticians, and we try to keep heavy maths to a minimum in the magazine.”
In a further (unanswered) e-mail to the editor, I have suggested a solution to the editor’s concern and stated that “Personally I feel that there are many practitioners out there who could benefit from this simple practical example and get aware that engineering considerations are part and parcel of hypothesis testing in an engineering environment. I often feel that these engineers are somewhat neglected in the statistics literature in favor of pure science.”
Based on my own experience of over thirty years of academic teaching to industrial engineering undergraduates, I feel that it is important that individuals working in an engineering environment understand that the view point expressed in Kelly’s article in the Significance magazine, which is quite prevalent, is not accurate in all circumstances.
With this in mind, the originally submitted article, titled:
“What is the significance of the significance level?” “It’s the error costs, stupid!”
is linked below: