Statistics: Foundations & Descriptive Statistics

Descriptive statistics organize and summarize observed data through numerical measures and visual displays. Inferential statistics go further by using sample data to draw conclusions about a population. This note begins with descriptive foundations and then introduces the probability, sampling, and hypothesis-testing ideas needed for later statistical tests.

Populations, Samples, Parameters and Statistics

A population is the complete set of individuals, objects, or measurements of interest in a study. A sample is a subset drawn from that population and used to study the larger group. A numerical characteristic of a population is a parameter, and the corresponding value calculated from a sample is a statistic. By convention, Greek letters denote parameters and Roman letters denote statistics.

A statistic rarely equals its parameter exactly. The difference between them is sampling error: chance variation that arises because only part of the population was observed. The later sections on sampling distributions and hypothesis testing build on this idea.

A variable is a characteristic that can take different values across people, objects, occasions, or observations, and data are the recorded values of one or more variables. The form of a variable determines how it can be summarized and which statistical procedures are sensible. Two distinctions are especially useful: its scale of measurement and its type.

Scales of Measurement

Stevens described four scales of measurement: nominal, ordinal, interval, and ratio. They differ in the information carried by the numbers and therefore in the mathematical operations that are meaningful. The scales are cumulative: each keeps the properties of the scale below it and adds one more. From least to most informative, the order is nominal → ordinal → interval → ratio.

Because the scales are cumulative, a statistic suited to a lower scale remains usable at a higher one. A percentage or mode can describe nominal data, the median needs at least ordinal data, and the mean and standard deviation need at least interval data.

Types of Variables

A categorical (qualitative) variable places observations into categories. A quantitative variable records a numerical amount, and quantitative variables are further divided into discrete and continuous variables.

A discrete variable is often a count, whereas a continuous variable is measured along a scale. This distinction is separate from the four levels of measurement above. Continuous measurements are always recorded to some finite precision, so a reaction time recorded as 412 ms stands for the interval from 411.5 to 412.5 ms.

A frequency distribution shows how often each value or interval occurs. It can be presented as a table or graph and is often the quickest way to see concentration, spread, skew, gaps, and possible outliers.

When a variable has many possible values, scores are grouped into class intervals of equal width. A grouped frequency table reports the frequency (f) of each interval and may add the relative frequency (the proportion or percentage in each interval) and the cumulative frequency (the number of scores up to the top of each interval). Cumulative frequencies are the basis for medians, percentiles, and percentile ranks.

Choosing a Display

A scatterplot serves a different purpose: it shows the relationship between two quantitative variables and is covered in the Correlation & Regression note. The box plot is explained with quartiles under Measures of Variability.

Measures of central tendency identify a single value that best represents the center or typical score in a distribution. The three primary measures are the mean, median, and mode. The appropriate choice depends on the scale of measurement, the shape of the distribution, and the presence of extreme scores.

The Mean

The arithmetic mean is the sum of all scores divided by the number of scores. It is the most commonly used measure of central tendency for interval and ratio data. The mean incorporates every score in the dataset, making it sensitive to the exact value of each observation. This sensitivity is both a strength and a weakness: the mean provides a mathematically precise center, but it can be distorted by extreme values.

The mean is the balance point of a distribution—if scores were weights on a board, the mean would be where the board balances. Formally, the deviations of scores from the mean always sum to zero: Σ(X − M) = 0. The mean is also the value about which the sum of squared deviations is smallest, which links it directly to variance and standard deviation.

Because every score contributes, a few extremely high or low scores can pull the mean away from where most scores lie. The scores 4, 5, 5, 6, and 30 have a mean of 10, although four of the five scores are 6 or lower.

The Median

The median is the middle score when all values are arranged in order from lowest to highest. Half of the scores fall below it and half above it, so the median is also the 50th percentile and the second quartile (Q2). For an odd number of scores, the median is the actual middle value. For an even number of scores, it is the average of the two middle values.

The median is appropriate for ordinal, interval, and ratio data. Unlike the mean, it is not affected by the exact values of extreme scores—only by their position in the ordered array. This makes the median a robust measure of central tendency for skewed distributions or data with outliers. The median of 4, 5, 5, 6, and 30 is 5, and it would remain 5 even if 30 became 300.

The Mode

The mode is the most frequently occurring score in a distribution. It is the only measure of central tendency appropriate for nominal data, though it can be used with any scale of measurement. It is useful for describing categorical data, such as the most common diagnosis or the most popular choice.

A distribution with one peak is unimodal, one with two peaks is bimodal, and one with more than two is multimodal. Two clear peaks often indicate two distinct subgroups in the data. If every score occurs equally often, as in 1, 2, 3, 4, 5, there is no mode. For continuous data, the mode may be uninformative because exact scores rarely repeat. It is also the least stable of the three measures across samples, because a small change in the data can shift which value is most frequent.

Choosing a Measure

For roughly normal data, the mean also varies least from sample to sample, which is one reason it underlies most inferential procedures.

While central tendency describes the typical score, variability describes how spread out or dispersed the scores are around that center. Two distributions can have identical means but very different spreads. High variability means that scores are widely scattered, whereas low variability means that they cluster tightly. Variability also matters beyond description because it affects the precision of estimates and the power of statistical tests.

Range

The range is the simplest measure of variability: the highest score minus the lowest score. For the scores 5, 8, 12, 15, and 23, the range is 23 − 5 = 18. The range gives a quick sense of total spread, but it uses only two values and ignores all the others. A single extreme score can therefore make it misleading, and it is the least stable measure of variability across samples.

Quartiles, Interquartile Range and Quartile Deviation

A percentile is the score at or below which a given percentage of the distribution falls. Quartiles divide an ordered distribution into four equal parts: Q1 is the 25th percentile, Q2 is the median, and Q3 is the 75th percentile. Deciles divide it into ten equal parts.

The IQR covers the middle 50% of scores, so extreme scores at either end do not affect it. This makes it the natural measure of spread to report with the median. A box plot displays these quantities together: the box runs from Q1 to Q3, a line inside it marks the median, the whiskers extend toward the lowest and highest scores that are not flagged as outliers, and separately plotted points mark potential outliers.

Average Deviation

The average deviation, also called mean absolute deviation (MAD), is the average of the absolute differences between each score and the mean. Absolute values are needed because deviations from the mean always sum to zero; ignoring their signs lets every score contribute to the spread. Unlike variance, average deviation does not square the deviations, so it stays in the original units and is easy to interpret. However, because absolute values are algebraically inconvenient, it is rarely used in inferential statistics and does not appear in the formulas for t-tests or ANOVA.

Variance and Standard Deviation

Variance is the average of the squared deviations from the mean. Squaring the deviations serves two purposes: it removes negative signs, and it gives more weight to large deviations, which makes variance sensitive to outliers. The standard deviation (SD) is the square root of variance. Taking the square root returns the measure to the original units of the data, so the SD is the measure normally used to describe spread.

Population variance divides by N, whereas sample variance divides by n − 1. This correction, called Bessel's correction, makes the sample variance an unbiased estimator of the population variance. Without it, sample variance would systematically underestimate population variance.

A small SD means scores cluster tightly around the mean, and a large SD means they are widely dispersed. The SD equals 0 only when all scores are identical. Consider two classes that both average 70. In Class A, SD = 3, and most students score between 67 and 73. In Class B, SD = 15, and scores are scattered much more widely. The means are the same; only the SD reveals the difference in consistency.

Variance is expressed in squared units, such as cm² or seconds², which makes it difficult to interpret directly. It remains central to statistical theory because sums of squares can be partitioned into components, as in ANOVA.

Coefficient of Variation

Sometimes variability must be compared between datasets measured in different units or with very different means. Comparing the raw SDs of height in centimetres and weight in kilograms is not meaningful because the units differ. The coefficient of variation (CV) expresses the SD as a percentage of the mean, which gives a unit-free measure of relative spread.

Reaction times with M = 400 ms and s = 60 ms have CV = 15%, whereas heights with M = 160 cm and s = 8 cm have CV = 5%. Relative to their means, the reaction times are three times as variable. Likewise, a dataset with M = 100 and s = 10 has the same relative spread (CV = 10%) as one with M = 50 and s = 5.

Comparing the Measures

Transforming Every Score

Adding a constant to every score slides the whole distribution along the number line without changing the distances between scores. Multiplying every score by a constant stretches or shrinks those distances. Each change affects the summary statistics in a predictable way.

If a teacher adds five grace marks to every student's score, the mean rises by exactly five, but the SD, the rank order of students, and the skewness of the distribution stay the same. The CV does change, because the SD stays fixed while the mean rises. Multiplying by a negative constant reverses the order of the scores and the direction of any skew.

The shape of a distribution affects how its center should be interpreted. Shape is described mainly by symmetry, measured as skewness, and by tail weight, measured as kurtosis. Both are usually judged against the normal distribution.

Skewness

A symmetric unimodal distribution has mean, median, and mode at the same central location. In a skewed distribution, most scores cluster toward one end and a few scores stretch into a long tail at the other. Positive (right) skew has its long tail toward high scores: most scores are low, with a few very high ones, as with income or reaction times. Negative (left) skew has its long tail toward low scores: most scores are high, with a few very low ones, as on an easy test.

The mean is more sensitive to the long tail than the median, while the mode stays at the peak. For the conventional unimodal shapes used in introductory statistics questions, the usual ordering is:

• Positive (right) skew: Mode < Median < Mean
• Negative (left) skew: Mean < Median < Mode

These are useful textbook patterns rather than identities that every possible dataset must obey. For moderately skewed distributions, Karl Pearson's empirical relation links the three measures: Mode ≈ 3 Median − 2 Mean.

The Normal Distribution

The normal distribution is a theoretical, symmetric, unimodal probability distribution defined by its mean and standard deviation. In a perfectly normal distribution, mean = median = mode. Its tails are asymptotic: they approach the horizontal axis but do not meet it. Its skewness is 0, and its kurtosis is the reference point for describing other distributions. Because the curve is symmetric, all of its odd-order central moments are zero; the second central moment is the variance.

The total area under the curve represents all scores, and the area between two points gives the proportion of scores in that range. Approximately 68% of scores lie within ±1 SD of the mean, 95% within ±2 SD, and 99.7% within ±3 SD. Half of each band lies on either side of the mean, so about 34% of scores fall between the mean and +1 SD.

Moving outward from the mean on either side, successive 1-SD bands contain about 34.13%, 13.59%, and 2.14% of scores, and about 0.13% lie beyond 3 SD. The interval ±1.96 SD contains almost exactly 95% of scores, a value used for confidence intervals and two-tailed tests.

The normal curve also fixes the relationships among the spread measures. The average (mean) deviation is about 0.80σ, roughly 4/5 SD, and the quartile deviation is about 0.6745σ, roughly 2/3 SD, so QD < AD < SD. These ratios hold for the normal curve, not for every distribution.

Kurtosis

Kurtosis is most usefully interpreted through tail weight. A normal distribution is mesokurtic. Under the ordinary kurtosis convention, normal kurtosis = 3; under the excess-kurtosis convention, normal = 0. Leptokurtic distributions have positive excess kurtosis and heavier tails than the normal distribution, whereas platykurtic distributions have negative excess kurtosis and lighter tails.

Checking Normality

Normality can be examined graphically and with formal tests. A Q-Q plot compares observed quantiles with those expected under a normal distribution; points lying reasonably close to a straight reference line support approximate normality. A histogram gives a rougher visual check. Formal procedures include the Shapiro-Wilk test, Kolmogorov-Smirnov-type tests, and the Jarque-Bera test, which is based on sample skewness and kurtosis. Graphical checks and tests are best interpreted together because formal tests are sensitive to sample size. A scatterplot does not test univariate normality, because it displays the relationship between two variables.

A z-score expresses a raw score in standard-deviation units. It is calculated by subtracting the mean from the raw score and dividing by the standard deviation. A positive z-score lies above the mean, a negative z-score lies below it, and z = 0 lies at the mean.

If a test has M = 70 and SD = 10, a score of 85 has z = (85 − 70) / 10 = +1.5, so the score lies 1.5 SD above the mean. Because z-scores place different tests on a common scale, they also allow scores from different distributions to be compared.

Converting scores to z-scores gives them a mean of 0 and an SD of 1, but it does not change the shape of the distribution or the order of the scores. A skewed set of raw scores therefore produces an equally skewed set of z-scores. The standard normal distribution is the normal distribution with mean 0 and SD 1, so z-scores from a normal distribution can be read directly against normal-curve areas.

Other Standard Scores

A z-score can be transformed to another standard-score scale using: new score = desired mean + (desired SD × z). These linear transformations remove the negative signs and decimals of z-scores while preserving each person's relative position. For example, z = +2 corresponds to T = 70 and a deviation IQ of 130.

Stanines ("standard nine") divide the normal distribution into nine bands containing about 4, 7, 12, 17, 20, 17, 12, 7, and 4 percent of scores. Some sources describe score scales with nonstandard parameters, such as a "T-score" with M = 100 and SD = 20. A conversion always uses the mean and SD specified for the scale in question.

Percentile ranks, normalized standard scores, and the use of norm groups in test interpretation are covered in the Psychometrics note.

Probability provides the numerical language for uncertainty. Values range from 0 (impossible) to 1 (certain), and probabilities can also be expressed as percentages. When all outcomes are equally likely, the probability of an event is found by counting outcomes. The probability of drawing one specified card from a 52-card deck is therefore 1/52.

Two rules combine the probabilities of separate events. The addition rule gives the probability that at least one of two events occurs, and the multiplication rule gives the probability that both occur.

The total area under a probability distribution is 1. For the normal distribution, the area between two z-scores gives the probability that a randomly selected score falls in that range, which is why normal-curve areas can be read as probabilities.

Inferential statistics depend on the idea that a statistic varies from sample to sample. A sampling distribution is the distribution of a statistic, such as the sample mean, across all possible samples of a given size from the same population.

Standard Error

The standard error of the mean is the standard deviation of the sampling distribution of means. It describes how much sample means are expected to vary from sample to sample. Because the population SD is usually unknown, the standard error is normally estimated from the sample SD.

If σ = 15 and n = 36, SE = 15 / 6 = 2.5. As n increases, SE decreases, so larger samples generally estimate the population mean more precisely.

Confidence Intervals

A confidence interval uses the standard error to express an estimate as a range of plausible values for the parameter. When the sampling distribution is normal, a 95% confidence interval for the mean is approximately M ± 1.96 SE. If M = 104 and SE = 2.5, the interval runs from about 99.1 to 108.9. The 95% describes the method: across many repeated samples, about 95% of intervals constructed this way would contain μ. A narrower interval indicates a more precise estimate. When σ is unknown and the sample is small, a critical t-value replaces 1.96.

Central Limit Theorem

The Central Limit Theorem states that, under standard conditions, the sampling distribution of the sample mean becomes increasingly close to normal as sample size grows, even when the population itself is not normal. The required sample size depends on the population shape; n ≈ 30 is a common classroom and exam rule of thumb, not a universal guarantee.

For samples of a fixed size n, the sampling distribution of the mean has three properties:

• its mean equals the population mean, μ;
• its standard deviation is the standard error, σ/√n;
• its shape approaches the normal distribution as n increases.

Sample means close to μ are therefore more frequent than sample means far from it. The CLT concerns this distribution of sample means, not the shape of the raw-score distribution. Approximate normality of the sampling distribution helps justify normal-based inference for sample means when its conditions are reasonably met.

Bootstrap Resampling

The bootstrap estimates sampling variability by repeatedly drawing new samples with replacement from the observed sample and recalculating the statistic of interest. Each bootstrap sample is usually the same size as the original, and a large number of them are drawn. The resulting bootstrap distribution can be used to estimate a standard error or confidence interval without assuming a particular population distribution. The method applies to many statistics and to metric as well as non-metric data. It is a resampling method rather than simply another rank-based non-parametric test.

Hypothesis testing compares the observed data with what would be expected under a null hypothesis. The procedure follows a fixed sequence of decisions.

A statement such as "there is no difference between male and female incomes in the population" is a typical null hypothesis. The wording of directional and non-directional research hypotheses is covered in the Research Foundations note.

p-Values and the Decision Rule

A p-value is the probability, assuming H₀ and the statistical model are true, of obtaining the observed result or a result at least as extreme. Thus p = .03 does not mean there is a 3% probability that H₀ is true.

If p ≤ α, the result is statistically significant and H₀ is rejected. If p > α, we fail to reject H₀. A non-significant result does not by itself prove that there is no effect. Rejecting H₀ does not prove H₁ either: when H₀ is true, a significant result still occurs with probability α, so the risk of a wrong rejection is small but never zero. Statistical significance also does not indicate how large or important an effect is; effect sizes, covered in the t-tests & ANOVA note, describe that.

Critical Regions and One- vs Two-Tailed Tests

The critical region, or rejection region, is the set of test-statistic values extreme enough to reject H₀ at the chosen α, and its boundary is the critical value. A two-tailed test divides α between both tails and can detect a difference in either direction. A one-tailed test places all of α in one tail, so it needs a less extreme value to reach significance but cannot detect an effect in the opposite direction. A one-tailed test is justified only when the direction is predicted before the data are collected.

Type I and Type II Errors

A Type I error occurs when a true H₀ is rejected; its probability is α. A Type II error occurs when a false H₀ is not rejected; its probability is β. Power = 1 − β, the probability of rejecting H₀ when H₀ is false.

Power and the α–β Trade-off

With other features fixed, making α more stringent tends to increase β. Moving from α = .05 to α = .01, for example, lowers the risk of a Type I error but raises the risk of a Type II error. At a fixed α, increasing sample size generally lowers β and increases power; the nominal α remains at the chosen level. Power increases with:

• larger samples;
• larger true effects;
• lower unexplained variability;
• a less stringent α, other things equal;
• a one-tailed test, when the effect lies in the predicted direction.

The topics below are ranked by how often they appear in previous-year questions linked to this note; the darkest tiles are the most frequent.