8.1 Formulas you must be able to interpret
| Quantity | Formula and required context |
|---|---|
| Observed mean | Sum of observed values / valid n; state units and population |
| Weighted mean/rate | Sum of group weight times group value; state what defines weights |
| Sample variance | Sum((x-mean)^2)/(n-1); units squared |
| Population-style variance | Sum((x-mean)^2)/n; use the requested convention |
| SD | Square root of variance; original units |
| Range | Maximum-minimum; distinguish width from endpoints |
| IQR | Q3-Q1; state quartile convention |
| Review fences | Q1-1.5IQR and Q3+1.5IQR; not deletion rules |
| Z-score | (x-reference mean)/reference SD; reference SD must be positive |
| Min-max | (x-training min)/(training max-training min) |
| Log transform | ln(1+x), used here for nonnegative counts |
| Euclidean distance | Square root of sum of squared coordinate differences |
| Response rate | Replies/invited; use group-specific invited denominators |
| Missing fraction | Missing/eligible; distinguish cells from absent people |
| Percentage-point gap | 100*(pA-pB) when p values are proportions |
| Relative percent difference | 100*(pA-pB)/pB; requires a stated nonzero reference |
| Classification error | Wrong labels/evaluated cases |
| Accuracy | Correct/evaluated = 1-error under the same counted cases |
| Sensitivity mean | [Observed sum + missing count*assumed missing mean]/eligible N |
8.2 Read the pandas operations in plain English
| Code pattern | Meaning and common trap |
|---|---|
df.shape, len(df) |
Table dimensions or number of rows; not necessarily people |
s.nunique(), s.is_unique |
Distinct values or uniqueness; inspect missing keys too |
s.count() versus s.size |
Nonmissing values versus entries |
s.isna() / s.notna() |
Missingness mask; an empty string may need separate normalisation |
s.eq('') |
Exact blank strings only |
s.value_counts() |
Category frequencies; defaults may omit missing categories |
s.mode() |
Most frequent value(s); can return several |
s.mean(), s.median() |
Summaries generally skip missing values; report valid n |
s.std(ddof=1) |
Sample SD; use 0 for the W4 population-style scaler |
s.quantile(.25) |
A quartile under the software's stated convention |
groupby(...).agg(...) |
Split into groups and summarise; specify the row unit first |
merge(..., validate=...) |
Join with a cardinality check; also check unmatched keys |
to_numeric(errors='coerce') |
Parse numbers; unparsed entries become missing |
to_datetime(format=...) |
Parse using the source's date convention |
fillna(saved_value) |
Fill with a specified replacement; does not recover truth |
between(a,b) |
Check a numeric range; cannot establish source accuracy |
assert condition |
Stop if the condition fails; only checks what you wrote |
8.3 Python ingredients appearing in your labs
range(12) supplies integers 0 through 11. In f'{101+i:05d}', 05d means an integer padded to five digits, so the first ID is'00101'. A list comprehension repeats an expression for each item. rng.integers(55,91,12) draws 12 integers from 55 through 90: the upper bound is excluded.
loc selects by label or Boolean condition; iloc selects by integer position. base.iloc[[1]] returns a one-row table, while a single index can return a row-like Series. copy(deep=True) creates an independent copy for the scalar-column records used in the lab. concat stacks tables; it does not deduplicate them. ignore_index=True rebuilds row index labels, not student IDs.
np.nan and pd.NA represent missing values in different dtype contexts. Test missingness with isna; equality with NaN is not a reliable missing-value test. In the original lab code, np.select assigns the first matching status rule. Checking original blanks before parser failures preserves their different provenance.
An f-string formats output. A seed makes the random construction reproducible given the same algorithm and sequence of draws. The seed is not a model parameter and does not establish that a case is representative. A case code identifies the generated version without being the student's exam result.