Choose actions from evidence
Q3–Q4 · 25 points
The lecture distinguishes copies, corrections and repeated events. The evidence packet supports removing R13, retaining R14 over R02, and flagging an invalid age value.
Week 3: repeated records, PDF pp. 16–20
Your task
Write your decision plan and predict counts before running. Compare flag_value with drop_student, then restore the final policy.
Evidence to include
Cite record IDs, source verification, final row and missing-score counts, and which usable score is lost under row deletion.
Notebook questions and code
Read here; run in your downloaded notebook. Start with its setup cells and run in order. Use your own ID and outputs where requested.
Download Lab 3 notebook2 · Choose actions before running the cleaning
Evidence packet from the fictional source owner:
- R13 is a confirmed extra export copy of R01.
- R14 is a verified correction of R02 for the same entry exam; it supersedes R02. This decision is based on verification, not just its date.
- BIO and outside spaces can safely be standardized to Biology. Bioengineering is a separate programme.
- The intended value for age 220 is unknown. The older student's age is verified correct.
- The entry “seventy” is not verified against the exam record. Flag the unparsed value and preserve the raw text; do not guess.
- No source measurement is available for the originally blank score.
Predict before executing: use the evidence to decide which rows to remove, which values to flag, and which values to retain.
Your explanation
Q3 · Your decision plan (12 points). Complete the table. “It looks wrong” is not enough evidence.
| Record or field | Keep / standardize / remove copy / use correction / flag value | Evidence and reason | Risk of an alternative action |
|---|---|---|---|
| R13 | … | … | … |
| R02 and R14 | … | … | … |
| age 220 | … | … | … |
| verified older age | … | … | … |
| Bioengineering | … | … | … |
| “seventy” | … | … | … |
Predict final row count and number of missing entry scores: …
Write here: …
Decision controls — change only the quoted choices
These settings let you inspect consequences without writing an algorithm. First choose your preferred policy. You may then experiment; return to a defensible policy before exporting.
COPY_POLICY:"remove_confirmed"or"keep_all"VERSION_POLICY:"verified_correction"or"keep_both"INVALID_AGE_POLICY:"flag_value"or"drop_student"
The code keeps the original export unchanged. It never fills a missing entry score.
Python · cell 11 — read here, run in your notebook
COPY_POLICY = "remove_confirmed"
VERSION_POLICY = "verified_correction"
INVALID_AGE_POLICY = "flag_value"
assert COPY_POLICY in ["remove_confirmed", "keep_all"]
assert VERSION_POLICY in ["verified_correction", "keep_both"]
assert INVALID_AGE_POLICY in ["flag_value", "drop_student"]
clean = raw.copy(deep=True)
log = []
def record(issue, affected, unit, action, evidence):
log.append(dict(issue=issue, affected=int(affected), unit=unit, action=action, evidence=evidence))
if COPY_POLICY == "remove_confirmed":
clean = clean[clean.record_id != "R13"].copy()
record("Extra export copy", 1, "row removed", "Remove R13", "Source confirms copy of R01")
if VERSION_POLICY == "verified_correction":
clean = clean[clean.record_id != "R02"].copy()
record("Superseded exam version", 1, "row removed", "Remove R02; retain R14", "Source verifies R14 correction")
clean["programme_raw"] = clean.programme
clean["programme"] = clean.programme.str.strip().replace({"biology":"Biology", "BIO":"Biology"})
record("Programme labels", (clean.programme != clean.programme_raw).sum(), "cells changed", "Apply approved mapping", "Dictionary; Bioengineering remains separate")
clean["entry_score_raw"] = clean.entry_score
score_text = clean.entry_score.str.strip().str.removesuffix("%")
clean["entry_score"] = pd.to_numeric(score_text, errors="coerce")
clean["score_status"] = np.select(
[clean.entry_score_raw.eq(""), clean.entry_score.isna()],
["originally_missing", "unparsed_needs_source"], default="observed")
record("Percentage representation", clean.entry_score_raw.str.endswith("%").sum(), "cells converted", "Store numeric points", "Dictionary specifies 0–100 points")
record("Unparsed score", clean.score_status.eq("unparsed_needs_source").sum(), "values flagged", "Keep missing; retain raw text", "No verified exam value")
record("Original blank score", clean.score_status.eq("originally_missing").sum(), "values retained missing", "No imputation", "No source measurement")
clean["age_raw"] = clean.age
clean["age"] = pd.to_numeric(clean.age, errors="coerce")
invalid_age = ~clean.age.between(16,100)
record("Invalid age", invalid_age.sum(), "values flagged" if INVALID_AGE_POLICY=="flag_value" else "rows removed", INVALID_AGE_POLICY, "Dictionary; correct age unknown")
if INVALID_AGE_POLICY == "flag_value":
clean.loc[invalid_age, "age"] = np.nan
else:
clean = clean.loc[~invalid_age].copy()
clean["age_flag"] = clean.age.isna()
clean["updated_at"] = pd.to_datetime(clean.updated_at, format="%d/%m/%Y")
clean["phone"] = clean.phone.replace("", pd.NA)
clean = clean.sort_values("student_id").reset_index(drop=True)
cleaning_log = pd.DataFrame(log)
display(clean[["record_id","student_id","programme","age","entry_score","score_status"]])
display(cleaning_log)
print("Rows before/after:", len(raw), len(clean))
print("Scores missing after cleaning:", clean.entry_score.isna().sum())Your explanation
Q4 · Interpret the result and challenge the shortcut (13 points).
- Did the results match your prediction? Explain any revision to your thinking.
- Cite the record IDs removed and distinguish a row removed from a value flagged.
- A colleague says “keep the latest row for every ID.” In this case, why is the retained version justified? If R14 were just a later upload with no verification, what should happen instead?
- Two rows are for the same student in weeks 1 and 2. Should one be removed? State an appropriate key and row definition.
- Try
"drop_student"once: how does the row count change, and which usable score is lost? Return to your final policy and rerun the cleaning cell.
Write here: …
Saved only in this browser. This note doesn’t fill in the downloaded notebook; “reviewed” is a personal checklist, not a grade.