Introduction to statistical testing; the sign test. Probability and significance, the use of statistical tables and critical values, type I and type II errors, choosing a statistical test.

What this dot point is asking

AQA wants you to explain probability and significance, the sign test, critical values, type I and type II errors, and how to choose a statistical test. The exam skill is to calculate and interpret the sign test, to apply the critical-value rule correctly, and to choose the right test from the design, the aim and the level of measurement.

Probability and significance

Inferential statistics let researchers decide whether a result is large enough to be unlikely to have arisen by chance. The standard significance level in psychology is $p < 0.05$, meaning there is a 5% or smaller probability that the result occurred by chance alone. If a test shows the result is significant at this level, the researcher rejects the null hypothesis (the prediction of no effect or no relationship) and accepts the alternative hypothesis. A stricter level such as $p < 0.01$ is used where being wrong is costly (for example in drug trials), and a more lenient level such as $p < 0.10$ is rarely used because it raises the risk of a false positive. The decision is made by comparing a calculated value (worked out from the data using the chosen test) against a critical value (read from a statistical table using the number of participants, whether the hypothesis is one- or two-tailed, and the significance level).

The sign test, errors and choosing a test

The sign test is the only test you must be able to calculate. It is used when the study is a test of difference, uses a related (repeated measures or matched pairs) design, and produces nominal data (data in categories, such as "improved" or "got worse"). To calculate it, you record the direction of change for each participant as a plus or a minus, discard any participants who showed no change, count the number of pluses and minuses, and take the calculated value $S$ as the total of the less frequent sign. The value of $N$ is the number of participants after discarding the no-change cases. You then compare $S$ with the critical value from the sign-test table: for the sign test the result is significant only if the calculated value is equal to or less than the critical value (this direction of the rule must be checked carefully for every test). When choosing a test, three questions decide it: is the study looking for a difference or a correlation, is the design related or unrelated, and what is the level of measurement (nominal, ordinal, or interval). The sign test answers difference, related, nominal. Errors arise because the decision is probabilistic: a type I error wrongly rejects a true null (a false positive, more likely with a lenient level), while a type II error wrongly retains a false null (a false negative, more likely with a strict level), and $p < 0.05$ is the conventional balance between the two risks.