{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "592c0b28",
   "metadata": {},
   "source": [
    "# Lab 3 — Data Cleaning and Quality\n",
    "## Think first. Inspect evidence. Explain your decision.\n",
    "**Data Science Fundamentals · New Uzbekistan University · Mohammad Sabokrou**\n",
    "\n",
    "**Time:** approximately 110 minutes + 10 minutes to check and submit.  \n",
    "**Goal:** decide what data can support, not demonstrate programming skill.\n",
    "\n",
    "All code is provided. Edit only your identity, the three decision settings, and written answers. No formulas to memorize or derive. Run cells in order using ▶ or Shift+Enter. In Colab, double-click a text cell to answer; use **File → Download → Download .ipynb** to save your completed notebook.\n",
    "\n",
    "Your student ID generates a reproducible version: scores, missing responses, review cases, and challenge details vary. Use your own numbers and record IDs. A generic answer without evidence is incomplete. Discuss ideas, but explain your own reasoning.\n",
    "\n",
    "**Learning outcomes:** diagnose quality problems; justify safe cleaning; distinguish missingness mechanisms; interpret sensitivity and outlier checks; report a result with its denominator, uncertainty, and audit trail.\n",
    "\n",
    "| Part | Focus | Minutes | Points |\n",
    "|---|---|---:|---:|\n",
    "| 1 | Purpose, dictionary, inventory | 15 | 15 |\n",
    "| 2 | Evidence-based cleaning decisions | 25 | 25 |\n",
    "| 3 | Missingness and who remains | 20 | 20 |\n",
    "| 4 | Missing-data choices and sensitivity | 15 | 15 |\n",
    "| 5 | Unusual values and consequences | 15 | 10 |\n",
    "| 6 | Audit, reporting, changed-case challenge | 20 | 15 |\n",
    "\n",
    "Points reward reasoning, relevant evidence, and limitations. Code elegance and writing formulas receive no credit. Keep missing values unfilled in your final registry. The temporary imputation example is for comparison only."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f17b023",
   "metadata": {},
   "source": [
    "## Setup — your version\n",
    "Enter your real student ID as **text inside quotation marks**, preserving leading zeros. Enter your name and class group. The ID check verifies format only; it does not verify registration. The notebook starts without outputs. Replace the demonstration ID with yours, then run the cells in order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60c9ff48",
   "metadata": {},
   "outputs": [],
   "source": [
    "STUDENT_ID = \"DEMO123\"  # replace with your university student ID\n",
    "STUDENT_NAME = \"Your name\"\n",
    "CLASS_GROUP = \"Your class group\"\n",
    "\n",
    "import hashlib, re, io, zipfile\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython.display import display, FileLink\n",
    "\n",
    "assert re.fullmatch(r\"[A-Za-z0-9_-]{3,30}\", STUDENT_ID), \"Use 3–30 letters, digits, _ or -; preserve leading zeros.\"\n",
    "seed = int(hashlib.sha256(STUDENT_ID.encode()).hexdigest()[:8], 16)\n",
    "rng = np.random.default_rng(seed)\n",
    "print(\"Student:\", STUDENT_ID, \"| Version:\", f\"{seed:08x}\")\n",
    "if STUDENT_ID == \"DEMO123\":\n",
    "    print(\"DEMONSTRATION ONLY — enter your own ID before submitting.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a352c36",
   "metadata": {},
   "source": [
    "## 1 · What does “clean” mean for this task?\n",
    "You are preparing a **current registry of 12 fictional students**. The question is: *What entry scores are observed, and how complete is our information about this cohort?* These are synthetic teaching data, not university records.\n",
    "\n",
    "**One final row = one current student.** A separate weekly-results table would have a different row definition.\n",
    "\n",
    "| Field | Meaning and rule |\n",
    "|---|---|\n",
    "| student_id | Text identifier, unique in the final registry; preserve leading zeros |\n",
    "| programme | Biology, Bioengineering, or Economics; the source confirms BIO means Biology |\n",
    "| group | Day or Evening; known for every student |\n",
    "| age | Years; dictionary accepts 16–100 for this fictional cohort, but validity does not prove accuracy |\n",
    "| entry_score | Points out of 100. Here “80%” means 80 points; plain numbers already mean points |\n",
    "| phone | Contact number; not required for score summaries |\n",
    "| updated_at | Source revision date, explicitly day/month/year; newest does not automatically mean authoritative |\n",
    "\n",
    "The expected roster contains exactly IDs 00101–00112. Blank strings represent missing values. Source notes will be supplied in Part 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4f6d675",
   "metadata": {},
   "outputs": [],
   "source": [
    "ids = [f\"{101+i:05d}\" for i in range(12)]\n",
    "scores = rng.integers(55, 91, 12)\n",
    "older_age = int(rng.choice([43, 47, 52]))\n",
    "base = pd.DataFrame({\n",
    "    \"student_id\": ids,\n",
    "    \"programme\": [\" biology \", \"BIO\", \"Biology\", \"Bioengineering\", \"Economics\", \"Economics\"]*2,\n",
    "    \"group\": [\"Day\"]*8 + [\"Evening\"]*4,\n",
    "    \"age\": [\"19\", \"20\", \"220\", \"21\", \"19\", \"22\", \"20\", \"21\", str(older_age), \"22\", \"21\", \"20\"],\n",
    "    \"entry_score\": [str(x) for x in scores],\n",
    "    \"phone\": [f\"555-{i:04d}\" for i in range(12)],\n",
    "    \"updated_at\": [\"03/04/2026\"]*12,\n",
    "})\n",
    "base.loc[3, \"entry_score\"] += \"%\"\n",
    "base.loc[5, \"entry_score\"] = \"\"\n",
    "base.loc[9, \"entry_score\"] = \"seventy\"\n",
    "base.loc[10, \"phone\"] = \"\"\n",
    "correction = base.iloc[[1]].copy()\n",
    "correction[\"entry_score\"] = str(min(int(scores[1])+7, 100))\n",
    "correction[\"updated_at\"] = \"05/04/2026\"\n",
    "raw = pd.concat([base, base.iloc[[0]], correction], ignore_index=True)\n",
    "raw.insert(0, \"record_id\", [f\"R{i+1:02d}\" for i in range(len(raw))])\n",
    "raw_snapshot = raw.copy(deep=True)\n",
    "display(raw)\n",
    "print(\"Rows:\", len(raw), \"| Distinct students:\", raw.student_id.nunique())\n",
    "print(\"Expected roster IDs absent:\", sorted(set(ids)-set(raw.student_id)))\n",
    "print(\"Blank cells by field:\")\n",
    "display(raw.eq(\"\").sum().rename(\"blank_cells\").to_frame())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0a8fcb0",
   "metadata": {},
   "source": [
    "### Your explanation\n",
    "**Q1 · Quality depends on purpose (8 points).**\n",
    "1. Identify one issue for **each** of completeness, validity, consistency, uniqueness, accuracy, and timeliness. For dimensions not diagnosable here, state a concrete check or external evidence needed; do not invent an error.\n",
    "2. Which missing field matters for urgent phone calls but not for the entry-score summary? Why should its blank not cause the whole row to be deleted?\n",
    "3. Give the row count, distinct-student count, and absent-roster count in your version. Why can these three quantities differ?\n",
    "\n",
    "**Write here:** …"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08e45b5c",
   "metadata": {},
   "source": [
    "### Your explanation\n",
    "**Q2 · Meaning before conversion (7 points).**\n",
    "1. In this dictionary, what number should replace the percentage text? Why would automatically dividing it by 100 be wrong here?\n",
    "2. What happens if identifiers become integers? Interpret 03/04/2026 using the documented source convention.\n",
    "3. Explain why “seventy” is not the same situation as an originally blank score. Does a parser failure prove the intended value was zero?\n",
    "\n",
    "**Write here:** …"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b367fa1c",
   "metadata": {},
   "source": [
    "## 2 · Choose actions before running the cleaning\n",
    "**Evidence packet from the fictional source owner:**\n",
    "\n",
    "- R13 is a confirmed extra export copy of R01.\n",
    "- R14 is a verified correction of R02 for the same entry exam; it supersedes R02. This decision is based on verification, not just its date.\n",
    "- BIO and outside spaces can safely be standardized to Biology. Bioengineering is a separate programme.\n",
    "- The intended value for age 220 is unknown. The older student's age is verified correct.\n",
    "- The entry “seventy” is not verified against the exam record. Flag the unparsed value and preserve the raw text; do not guess.\n",
    "- No source measurement is available for the originally blank score.\n",
    "\n",
    "**Predict before executing:** use the evidence to decide which rows to remove, which values to flag, and which values to retain."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87f9c339",
   "metadata": {},
   "source": [
    "### Your explanation\n",
    "**Q3 · Your decision plan (12 points).** Complete the table. “It looks wrong” is not enough evidence.\n",
    "\n",
    "| Record or field | Keep / standardize / remove copy / use correction / flag value | Evidence and reason | Risk of an alternative action |\n",
    "|---|---|---|---|\n",
    "| R13 | … | … | … |\n",
    "| R02 and R14 | … | … | … |\n",
    "| age 220 | … | … | … |\n",
    "| verified older age | … | … | … |\n",
    "| Bioengineering | … | … | … |\n",
    "| “seventy” | … | … | … |\n",
    "\n",
    "Predict final row count and number of missing entry scores: **…**\n",
    "\n",
    "**Write here:** …"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8957abb",
   "metadata": {},
   "source": [
    "### Decision controls — change only the quoted choices\n",
    "These settings let you inspect consequences without writing an algorithm. First choose your preferred policy. You may then experiment; return to a defensible policy before exporting.\n",
    "\n",
    "- `COPY_POLICY`: `\"remove_confirmed\"` or `\"keep_all\"`\n",
    "- `VERSION_POLICY`: `\"verified_correction\"` or `\"keep_both\"`\n",
    "- `INVALID_AGE_POLICY`: `\"flag_value\"` or `\"drop_student\"`\n",
    "\n",
    "The code keeps the original export unchanged. It never fills a missing entry score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2402b9dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "COPY_POLICY = \"remove_confirmed\"\n",
    "VERSION_POLICY = \"verified_correction\"\n",
    "INVALID_AGE_POLICY = \"flag_value\"\n",
    "\n",
    "assert COPY_POLICY in [\"remove_confirmed\", \"keep_all\"]\n",
    "assert VERSION_POLICY in [\"verified_correction\", \"keep_both\"]\n",
    "assert INVALID_AGE_POLICY in [\"flag_value\", \"drop_student\"]\n",
    "clean = raw.copy(deep=True)\n",
    "log = []\n",
    "def record(issue, affected, unit, action, evidence):\n",
    "    log.append(dict(issue=issue, affected=int(affected), unit=unit, action=action, evidence=evidence))\n",
    "if COPY_POLICY == \"remove_confirmed\":\n",
    "    clean = clean[clean.record_id != \"R13\"].copy()\n",
    "    record(\"Extra export copy\", 1, \"row removed\", \"Remove R13\", \"Source confirms copy of R01\")\n",
    "if VERSION_POLICY == \"verified_correction\":\n",
    "    clean = clean[clean.record_id != \"R02\"].copy()\n",
    "    record(\"Superseded exam version\", 1, \"row removed\", \"Remove R02; retain R14\", \"Source verifies R14 correction\")\n",
    "clean[\"programme_raw\"] = clean.programme\n",
    "clean[\"programme\"] = clean.programme.str.strip().replace({\"biology\":\"Biology\", \"BIO\":\"Biology\"})\n",
    "record(\"Programme labels\", (clean.programme != clean.programme_raw).sum(), \"cells changed\", \"Apply approved mapping\", \"Dictionary; Bioengineering remains separate\")\n",
    "clean[\"entry_score_raw\"] = clean.entry_score\n",
    "score_text = clean.entry_score.str.strip().str.removesuffix(\"%\")\n",
    "clean[\"entry_score\"] = pd.to_numeric(score_text, errors=\"coerce\")\n",
    "clean[\"score_status\"] = np.select(\n",
    "    [clean.entry_score_raw.eq(\"\"), clean.entry_score.isna()],\n",
    "    [\"originally_missing\", \"unparsed_needs_source\"], default=\"observed\")\n",
    "record(\"Percentage representation\", clean.entry_score_raw.str.endswith(\"%\").sum(), \"cells converted\", \"Store numeric points\", \"Dictionary specifies 0–100 points\")\n",
    "record(\"Unparsed score\", clean.score_status.eq(\"unparsed_needs_source\").sum(), \"values flagged\", \"Keep missing; retain raw text\", \"No verified exam value\")\n",
    "record(\"Original blank score\", clean.score_status.eq(\"originally_missing\").sum(), \"values retained missing\", \"No imputation\", \"No source measurement\")\n",
    "clean[\"age_raw\"] = clean.age\n",
    "clean[\"age\"] = pd.to_numeric(clean.age, errors=\"coerce\")\n",
    "invalid_age = ~clean.age.between(16,100)\n",
    "record(\"Invalid age\", invalid_age.sum(), \"values flagged\" if INVALID_AGE_POLICY==\"flag_value\" else \"rows removed\", INVALID_AGE_POLICY, \"Dictionary; correct age unknown\")\n",
    "if INVALID_AGE_POLICY == \"flag_value\":\n",
    "    clean.loc[invalid_age, \"age\"] = np.nan\n",
    "else:\n",
    "    clean = clean.loc[~invalid_age].copy()\n",
    "clean[\"age_flag\"] = clean.age.isna()\n",
    "clean[\"updated_at\"] = pd.to_datetime(clean.updated_at, format=\"%d/%m/%Y\")\n",
    "clean[\"phone\"] = clean.phone.replace(\"\", pd.NA)\n",
    "clean = clean.sort_values(\"student_id\").reset_index(drop=True)\n",
    "cleaning_log = pd.DataFrame(log)\n",
    "display(clean[[\"record_id\",\"student_id\",\"programme\",\"age\",\"entry_score\",\"score_status\"]])\n",
    "display(cleaning_log)\n",
    "print(\"Rows before/after:\", len(raw), len(clean))\n",
    "print(\"Scores missing after cleaning:\", clean.entry_score.isna().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06878bf9",
   "metadata": {},
   "source": [
    "### Your explanation\n",
    "**Q4 · Interpret the result and challenge the shortcut (13 points).**\n",
    "1. Did the results match your prediction? Explain any revision to your thinking.\n",
    "2. Cite the record IDs removed and distinguish a **row removed** from a **value flagged**.\n",
    "3. A colleague says “keep the latest row for every ID.” In this case, why is the retained version justified? If R14 were just a later upload with no verification, what should happen instead?\n",
    "4. Two rows are for the same student in weeks 1 and 2. Should one be removed? State an appropriate key and row definition.\n",
    "5. Try `\"drop_student\"` once: how does the row count change, and which usable score is lost? Return to your final policy and rerun the cleaning cell.\n",
    "\n",
    "**Write here:** …"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c72617d6",
   "metadata": {},
   "source": [
    "## 3 · Missingness changes who we see\n",
    "This is a **separate fictional survey**, not the entry-score registry. Everyone in the roster was invited. Satisfaction ranges from 1 to 5. Group is known for everyone. No answer means unknown, not zero.\n",
    "\n",
    "Before running: if Evening students answer less often, will the respondent mix necessarily match the invited mix? Write a prediction in Q5 first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffb2de4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "survey_rng = np.random.default_rng(seed + 1)\n",
    "n_day, n_evening = 20, 10\n",
    "reply_day = int(survey_rng.integers(15,19))\n",
    "reply_evening = int(survey_rng.integers(3,6))\n",
    "survey = pd.DataFrame({\"group\": [\"Day\"]*n_day+[\"Evening\"]*n_evening,\n",
    "    \"rating\": [4.0]*reply_day+[np.nan]*(n_day-reply_day)+[2.0]*reply_evening+[np.nan]*(n_evening-reply_evening)})\n",
    "summary = survey.groupby(\"group\").rating.agg(invited=\"size\", replies=\"count\", observed_mean=\"mean\")\n",
    "summary[\"missing\"] = summary.invited-summary.replies\n",
    "summary[\"response_percent\"] = (100*summary.replies/summary.invited).round(1)\n",
    "summary[\"share_of_invited_percent\"] = (100*summary.invited/len(survey)).round(1)\n",
    "summary[\"share_of_replies_percent\"] = (100*summary.replies/summary.replies.sum()).round(1)\n",
    "display(summary)\n",
    "print(\"Pooled observed mean:\", round(survey.rating.mean(),3))\n",
    "summary.response_percent.plot.bar(color=[\"#238b8d\", \"#dd9050\"], ylim=(0,100), ylabel=\"Responses / invited (%)\", rot=0)\n",
    "plt.title(\"Who answered in your version?\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92967fb1",
   "metadata": {},
   "source": [
    "### Your explanation\n",
    "**Q5 · Denominators and representation (10 points).**\n",
    "- Prediction before running: **…**\n",
    "- Report the missing fraction for each group using **counts and denominators**.\n",
    "- Compare Evening's share of invitations with its share of replies. Who becomes less represented after dropping missing ratings?\n",
    "- Write one accurate sentence reporting the pooled observed mean and valid n. Explain why it does not establish the all-student mean.\n",
    "- Explain the difference between **0 hours of lab use**, **unknown hours**, and **not enrolled in the lab module**. Which students belong in the target population when summarizing hours for enrolled students?\n",
    "\n",
    "**Write here:** …"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c5040cb",
   "metadata": {},
   "source": [
    "### Three possible stories for the same survey\n",
    "**Story A:** after everyone answers, a technical process independently loses each answer with equal probability, unrelated to group or rating.\n",
    "\n",
    "**Story B:** Evening students have less time to answer. Within each group, reply probability does not depend on satisfaction.\n",
    "\n",
    "**Story C:** within the Evening group, dissatisfied students are less likely to answer than satisfied students, even after accounting for known group.\n",
    "\n",
    "These are hypothetical mechanisms; the observed response counts do not tell us which story actually occurred."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1a8c654",
   "metadata": {},
   "source": [
    "### Your explanation\n",
    "**Q6 · Explain mechanisms, not just labels (10 points).**\n",
    "1. Match A, B, C to MCAR, MAR, MNAR and justify each in an ordinary sentence.\n",
    "2. Does the group difference in your table prove MAR? State exactly what must be true **within each group** for Story B.\n",
    "3. Propose one practical follow-up that could help investigate nonresponse. Explain what would remain uncertain.\n",
    "4. If both groups had identical response rates, would this guarantee an unbiased comparison? Give a counterexample.\n",
    "\n",
    "**Write here:** …"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e42450b",
   "metadata": {},
   "source": [
    "## 4 · Four responses to missing values\n",
    "Decide what each situation permits. Recovering a verified value is different from estimating it.\n",
    "\n",
    "| Situation | Your choice: recover / keep missing / exclude for stated analysis / impute as a model assumption | Reason and what to report |\n",
    "|---|---|---|\n",
    "| A verified exam sheet for the exact student and exam contains the missing score | … | … |\n",
    "| Programme is known; phone is blank; the question is programme counts | … | … |\n",
    "| You explicitly report scores only for students with recorded scores | … | … |\n",
    "| A separate demonstration fills blanks with the observed mean | … | … |\n",
    "\n",
    "**Q7a (5 points):** complete this table. Explain why the fourth row does not create newly observed evidence. Keep your final registry unfilled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f6a9c19",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Separate demonstration only: neither clean nor survey is overwritten.\n",
    "example = pd.Series([2.0,4.0,np.nan,np.nan,np.nan])\n",
    "filled_demo = example.fillna(example.mean())\n",
    "display(pd.DataFrame({\n",
    "    \"treatment\":[\"Observed only\", \"Mean-filled demonstration\"],\n",
    "    \"mean\":[example.mean(), filled_demo.mean()],\n",
    "    \"sample_SD\":[example.std(), filled_demo.std()],\n",
    "    \"numeric_cells\":[example.count(), filled_demo.count()],\n",
    "    \"genuinely_observed_answers\":[2,2]\n",
    "}).round(3))\n",
    "# Hypothetical values for missing responses: sensitivity, NOT a cleaned dataset.\n",
    "observed_total = survey.rating.sum()\n",
    "missing_n = int(survey.rating.isna().sum())\n",
    "sensitivity = pd.DataFrame({\"assumed_mean_of_missing\":[1.,2.,3.,4.,5.]})\n",
    "sensitivity[\"hypothetical_all_student_mean\"] = (observed_total + missing_n*sensitivity.assumed_mean_of_missing)/len(survey)\n",
    "sensitivity[\"meets_target_3_5\"] = sensitivity.hypothetical_all_student_mean >= 3.5\n",
    "display(sensitivity.round(3))\n",
    "print(\"Target: all-student satisfaction mean at least 3.5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "984ede37",
   "metadata": {},
   "source": [
    "### Your explanation\n",
    "**Q7b · What can we claim? (10 points).**\n",
    "1. In the imputation demonstration, what changes and what stays the same? Why is “we now have five observed answers” false?\n",
    "2. Cite two rows from your sensitivity table that lead to different target decisions. Are these recovered answers, or assumptions?\n",
    "3. State the smallest and largest possible all-student mean on the 1–5 scale using the table. Is “the course definitely meets the target” defensible?\n",
    "4. Write a cautious alternative conclusion and suggest what information would strengthen it. No formula derivation is needed.\n",
    "\n",
    "**Write here:** …"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12cffe63",
   "metadata": {},
   "source": [
    "## 5 · Flagged does not mean wrong\n",
    "The code computes an IQR review rule for you. **You only need to use the output:** a flag marks a value worth investigating; it is not an instruction to delete.\n",
    "\n",
    "This review table uses the original 12 students, before correcting the invalid age. The dictionary and source evidence remain essential."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ec305e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "ages = pd.to_numeric(base.age)\n",
    "q1, q3 = ages.quantile([0.25,0.75])\n",
    "low, high = q1-1.5*(q3-q1), q3+1.5*(q3-q1)\n",
    "review = pd.DataFrame({\"student_id\":base.student_id, \"age\":ages,\n",
    "    \"flagged_for_review\": (ages<low)|(ages>high)})\n",
    "print(\"Computed review limits:\", round(low,2), \"to\", round(high,2), \"years\")\n",
    "display(review[review.flagged_for_review])\n",
    "# A separate fully verified cohort illustrates the consequences of deleting older students.\n",
    "older_n = int(rng.integers(4,8))\n",
    "older_withdrawals = older_n-1\n",
    "other_n, other_withdrawals = 100-older_n, 6\n",
    "impact = pd.DataFrame({\"population\":[\"All enrolled students\",\"After excluding verified older students\"],\n",
    "    \"students\":[100,other_n], \"withdrawals\":[older_withdrawals+other_withdrawals,other_withdrawals]})\n",
    "impact[\"withdrawal_percent\"] = 100*impact.withdrawals/impact.students\n",
    "display(impact.round(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffb4a865",
   "metadata": {},
   "source": [
    "### Your explanation\n",
    "**Q8 · Same flag, different actions (10 points).**\n",
    "1. Compare age 220 and your verified older age. Use the output and source evidence to justify different actions.\n",
    "2. Does an age inside the review limits guarantee accuracy? Give a plausible recording error that could pass.\n",
    "3. Quote the two withdrawal rates and their denominators. Did deleting valid older students improve the underlying outcome, or change who was counted?\n",
    "4. If the intended population is all enrolled students, which rate should lead your report? What useful comparison could you still show?\n",
    "\n",
    "**Write here:** …"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a64238b",
   "metadata": {},
   "source": [
    "## 6 · Audit, report, and transfer your understanding\n",
    "Run the checks after returning to your final cleaning policy. A failed check is useful feedback: inspect your decision settings and rerun Part 2, then this cell. You do not need to rewrite the checks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c9e1823",
   "metadata": {},
   "outputs": [],
   "source": [
    "checks = {\n",
    "    \"One row per student\": clean.student_id.is_unique,\n",
    "    \"All expected students retained\": set(clean.student_id)==set(ids),\n",
    "    \"IDs remain five-character text\": clean.student_id.map(lambda x:isinstance(x,str) and len(x)==5).all(),\n",
    "    \"Only approved programmes\": clean.programme.isin([\"Biology\",\"Bioengineering\",\"Economics\"]).all(),\n",
    "    \"Observed scores within 0–100\": clean.entry_score.dropna().between(0,100).all(),\n",
    "    \"Observed ages within dictionary range\": clean.age.dropna().between(16,100).all(),\n",
    "    \"Unresolved scores remain missing\": clean.loc[clean.score_status!=\"observed\",\"entry_score\"].isna().all(),\n",
    "    \"Raw export unchanged\": raw.equals(raw_snapshot),\n",
    "    \"Row removals reconciled\": len(raw)-len(clean)==sum(x[\"affected\"] for x in log if x[\"unit\"] in [\"row removed\",\"rows removed\"]),\n",
    "}\n",
    "display(pd.DataFrame({\"check\":checks.keys(),\"passed\":checks.values()}))\n",
    "print(\"Final rows:\",len(clean),\"| Observed scores:\",clean.entry_score.count(),\"| Missing scores:\",clean.entry_score.isna().sum())\n",
    "print(\"Observed entry-score mean:\",round(clean.entry_score.mean(),2),\"points out of 100\")\n",
    "assert all(checks.values()), \"Review failed checks and your policies; rerun Part 2 and then this cell.\"\n",
    "print(\"Structural checks passed. Accuracy still requires source evidence.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c62b321",
   "metadata": {},
   "source": [
    "### Your explanation\n",
    "**Q9 · Final report and audit (8 points).**\n",
    "Write 100–150 words for a reader who has not opened your notebook. Include:\n",
    "- your version code; target population; raw/final row counts;\n",
    "- key actions with evidence, distinguishing row removal from value flagging;\n",
    "- observed entry-score mean, its unit, valid n, and missing count;\n",
    "- one limitation about missingness and one about accuracy.\n",
    "\n",
    "Then cite **two entries** in the generated cleaning log. Explain why passing every check does not prove that every stored score matches the actual exam.\n",
    "\n",
    "**Write here:** …"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25ecc082",
   "metadata": {},
   "outputs": [],
   "source": [
    "challenge = [\n",
    "    \"An analyst replaces every missing phone with 0 and deletes students with missing scores before counting programmes. Evaluate BOTH decisions for the programme-count question.\",\n",
    "    \"A new weekly-assessment export has two rows per student, one per week. An analyst reuses the current-registry rule and keeps only the latest. Explain what is lost and define the correct key.\",\n",
    "    \"An analyst merges Bioengineering into Biology because both start with Bio, then reports that all validity checks pass. Explain the semantic error and why a check can miss it.\"\n",
    "][seed % 3]\n",
    "print(\"YOUR CHANGED-CASE CHALLENGE:\", challenge)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a274db2",
   "metadata": {},
   "source": [
    "### Your explanation\n",
    "**Q10 · Independent transfer (7 points).** Answer your displayed challenge in 60–100 words. Give (a) the problem, (b) a better action and supporting evidence needed, and (c) a consequence for the conclusion. Finally name one answer in this lab you changed after inspecting evidence, or explain why you retained your initial answer.\n",
    "\n",
    "**Write here:** …"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62e0e12f",
   "metadata": {},
   "source": [
    "## Export and submission\n",
    "Run the cell below to create a ZIP with your **cleaned registry CSV**, **cleaning log CSV**, and original synthetic export. The ZIP does **not** contain the notebook: download your completed `.ipynb` separately with all answers and outputs.\n",
    "\n",
    "**Submit:** completed notebook + generated ZIP through the Lab 3 submission route specified by your instructor/course page. Do not replace an existing course submission subject or deadline with a guessed one.\n",
    "\n",
    "Before submitting:\n",
    "- [ ] My own student ID, name, and class group are entered.\n",
    "- [ ] I reran all cells after selecting my final policy; checks pass.\n",
    "- [ ] Q1–Q10 and the four-responses table have my explanations and relevant numbers.\n",
    "- [ ] Unknown values remain missing in the final registry.\n",
    "- [ ] My downloaded notebook contains outputs and written answers; the ZIP opens.\n",
    "\n",
    "**Lecture alignment:** Week 3 *Data Cleaning and Quality*, student slides dated 1 October 2026: quality/dictionary (6–15); duplicates (16–20); missingness and responses (23–39); unusual values (40–44); audit/assertions/reporting (46–55). This lab implements the lecture instruction to keep missing values unfilled. It uses built-in synthetic data so no external dataset download is needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31186ca2",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert all(checks.values()), \"Complete the checks before exporting.\"\n",
    "assert STUDENT_ID != \"DEMO123\", \"Enter your own ID, then rerun all cells before exporting.\"\n",
    "assert STUDENT_NAME != \"Your name\" and CLASS_GROUP != \"Your class group\", \"Complete your name and class group.\"\n",
    "zip_name = f\"{STUDENT_ID}_Lab3_Data_and_Log.zip\"\n",
    "with zipfile.ZipFile(zip_name, \"w\", zipfile.ZIP_DEFLATED) as z:\n",
    "    z.writestr(f\"{STUDENT_ID}_Lab3_Cleaned_Registry.csv\", clean.to_csv(index=False))\n",
    "    z.writestr(f\"{STUDENT_ID}_Lab3_Cleaning_Log.csv\", cleaning_log.to_csv(index=False))\n",
    "    z.writestr(f\"{STUDENT_ID}_Lab3_Raw_Export.csv\", raw.to_csv(index=False))\n",
    "    z.writestr(\"VERSION.txt\", f\"Student: {STUDENT_ID}\\nName: {STUDENT_NAME}\\nGroup: {CLASS_GROUP}\\nVersion: {seed:08x}\\nSynthetic teaching data. Missing values intentionally unfilled.\\n\")\n",
    "print(\"Created\", zip_name, \"— also download your completed notebook separately.\")\n",
    "try:\n",
    "    from google.colab import files\n",
    "    files.download(zip_name)\n",
    "except ImportError:\n",
    "    display(FileLink(zip_name))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}