Oh no! No significant results, your project is ruined. Ruth Hand from the Mathematics Resources Centre (MASH) offers a fresh perspective and maybe it's not so bad after all.
POV: you are an undergraduate final year with a non-significant stats test
After months of data collection and weeks of cleaning and tidying your dataset, you run the analysis. You excitedly scan the results for that all-important p-value. It is 0.642.
It is hard not to take this personally when your lit review suggested otherwise and you have already drafted your discussion section with significant results in mind. Before you panic (or start inventing new data points!), let’s look at what a non-significant result actually means and how to write a great dissertation without needing a significant hypothesis test.
There are no marks for statistical significance
You should check your own mark scheme but expect to see phrases like ‘appropriate and accurate use of statistics’. Take this alongside the fact that all dissertations are an exercise in critical writing, and we see that performing a test and critiquing it is the gold standard, whichever side of 0.05 you are on.
The power and the glory
You can spot anything with a strong enough microscope. A non-significant result does not mean there isn't an effect, it means you haven't found any evidence of one with the dataset you collected. This could mean:
- your sample size wasn't large enough to detect an effect as small as the one that exists
- the true effect is smaller than you expected it to be
- your measurements had greater variability than you expected
- your population is different from that of previous research (and therefore the effect size might also differ)
Explicitly discussing statistical power shows maturity, so you've got an opportunity to exploit your underpowered test as a discussion point. If you did an a priori sample-sizing calculation, then compare your predicted effect size to the one you observed in your sample. If you didn’t do a power calculation, you can now work backwards to calculate the minimum effect size that could have been detected.
What is so great about 0.05 anyway?
This is slightly controversial, as many disciplines still love the black-and-white cut-off of α = 0.05; however, there is a growing trend towards more nuanced interpretations. In the first instance, it’s worth considering how far off you are: hands up if you can't really see why p = 0.049 is meaningfully different from p = 0.051 (spoiler, it isn't), noise in the data or error in the model is likely to dwarf any meaningful difference between those two numbers.
Rather than still sounding like someone who wishes they were below 0.05 (avoid saying “almost significant”), it’s better to report confidence intervals alongside your p-values, giving you scope for discussion. In many situations, the effect size is more interesting than the p-value anyway. Depending on your subject area, you might look to clinical or practical significance, rather than just statistical significance.
None of this means p-values are useless, but they are rarely the whole story, so give yourself the opportunity to widen the discussion.
This is not a drill (or, maybe it is)
Most researchers like to have some solid evidence before doing a formal hypothesis test, but undergrads don’t usually have the time or the money for a pilot study. That doesn’t automatically mean inferential stats are off the table, but it does mean a fully confirmatory analysis might not be the most sensible choice.
Exploratory analysis tends to be wider in scope than inferential testing and can be a good option when you’re not sure what effect size to expect or when your sample is small. Exploratory work might look at:
- patterns in subscales
- correlations
- trends that didn’t reach significance but suggest future directions
- estimation of effect sizes
In essence, you’re generating hypotheses, not testing them.
Suddenly switching analysis plans is drastic, an exploratory analysis is not a substitute, it’s a totally different style of analysis with its own pros and cons. Talk it through with your supervisor, they may well agree that there are benefits to exploring and discussing your data, rather than just reporting the dead-end result of a non-significant hypothesis test.
Do not change the plan without telling your supervisor
This is important enough to say more than once: do not rewrite your research questions or analysis plan without discussing it with your supervisor first!
Changing direction can be completely reasonable, but it needs to be agreed. Some supervisors are more intimidating than others, but they always have the best interests of your project in mind.
Respond
