Data Cleaning and Quality
Make supported cleaning choices and explain what remains uncertain.
This notebook generates a fictional 12-student registry, a separate 30-person survey and a separate verified withdrawal example. They are different populations. The study copy resets identity fields to placeholders and clears outputs; enter your own ID, name and class before running.
Start here: notebook introduction and setup
Lab 3 — Data Cleaning and Quality
Think first. Inspect evidence. Explain your decision.
Data Science Fundamentals · New Uzbekistan University · Mohammad Sabokrou
Time: approximately 110 minutes + 10 minutes to check and submit.
Goal: decide what data can support, not demonstrate programming skill.
All code is provided. Edit only your identity, the three decision settings, and written answers. No formulas to memorize or derive. Run cells in order using ▶ or Shift+Enter. In Colab, double-click a text cell to answer; use File → Download → Download .ipynb to save your completed notebook.
Your student ID generates a reproducible version: scores, missing responses, review cases, and challenge details vary. Use your own numbers and record IDs. A generic answer without evidence is incomplete. Discuss ideas, but explain your own reasoning.
Learning outcomes: diagnose quality problems; justify safe cleaning; distinguish missingness mechanisms; interpret sensitivity and outlier checks; report a result with its denominator, uncertainty, and audit trail.
| Part | Focus | Minutes | Points |
|---|---|---|---|
| 1 | Purpose, dictionary, inventory | 15 | 15 |
| 2 | Evidence-based cleaning decisions | 25 | 25 |
| 3 | Missingness and who remains | 20 | 20 |
| 4 | Missing-data choices and sensitivity | 15 | 15 |
| 5 | Unusual values and consequences | 15 | 10 |
| 6 | Audit, reporting, changed-case challenge | 20 | 15 |
Points reward reasoning, relevant evidence, and limitations. Code elegance and writing formulas receive no credit. Keep missing values unfilled in your final registry. The temporary imputation example is for comparison only.
Setup — your version
Enter your real student ID as text inside quotation marks, preserving leading zeros. Enter your name and class group. The ID check verifies format only; it does not verify registration. The notebook starts without outputs. Replace the demonstration ID with yours, then run the cells in order.
Python · cell 3 — read here, run in your notebook
STUDENT_ID = "DEMO123" # replace with your university student ID
STUDENT_NAME = "Your name"
CLASS_GROUP = "Your class group"
import hashlib, re, io, zipfile
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import display, FileLink
assert re.fullmatch(r"[A-Za-z0-9_-]{3,30}", STUDENT_ID), "Use 3–30 letters, digits, _ or -; preserve leading zeros."
seed = int(hashlib.sha256(STUDENT_ID.encode()).hexdigest()[:8], 16)
rng = np.random.default_rng(seed)
print("Student:", STUDENT_ID, "| Version:", f"{seed:08x}")
if STUDENT_ID == "DEMO123":
print("DEMONSTRATION ONLY — enter your own ID before submitting.")Lecture-to-lab connections 6 sections
Purpose, dictionary and inventory →
The six quality dimensions are applied to a concrete task. The dictionary decides the score scale, date order and allowed programme meanings.
Week 3: dictionary and quality, PDF pp. 6–15
Choose actions from evidence →
The lecture distinguishes copies, corrections and repeated events. The evidence packet supports removing R13, retaining R14 over R02, and flagging an invalid age value.
Week 3: repeated records, PDF pp. 16–20
Who answered the survey? →
The lecture connects nonresponse with representation. The survey separates group invitation shares from reply shares and asks what MCAR/MAR/MNAR mean.
Week 3: missingness and response mechanisms, PDF pp. 23–39
Missing-value choices and sensitivity →
The lecture separates recovered observations from assumed replacements. The mean-fill demonstration and sensitivity table show different consequences of assumptions.
Week 3: missing-data responses and sensitivity, PDF pp. 23–39
Flags, validity and who is counted →
An IQR flag describes unusualness, while the dictionary and source establish different evidence. Deleting verified older students changes the population behind a rate.
Week 3: unusual values, PDF pp. 40–44
Audit, report and transfer →
The lecture's audit trail becomes a final report. Structural checks verify stated rules, while source accuracy and broader claims still need separate evidence.
Week 3: assertions, logs and reporting, PDF pp. 46–55
Before submitting
Complete Q1–Q10 and the Q7a table, return to a defensible final policy, rerun and pass checks, and keep unresolved scores missing. Submit the completed notebook plus its generated ZIP. The ZIP includes registry/log/raw CSV files but does not include the notebook.
Notebook submission instructions
Export and submission
Run the cell below to create a ZIP with your cleaned registry CSV, cleaning log CSV, and original synthetic export. The ZIP does not contain the notebook: download your completed .ipynb separately with all answers and outputs.
Submit: completed notebook + generated ZIP through the Lab 3 submission route specified by your instructor/course page. Do not replace an existing course submission subject or deadline with a guessed one.
Before submitting:
- My own student ID, name, and class group are entered.
- I reran all cells after selecting my final policy; checks pass.
- Q1–Q10 and the four-responses table have my explanations and relevant numbers.
- Unknown values remain missing in the final registry.
- My downloaded notebook contains outputs and written answers; the ZIP opens.
Lecture alignment: Week 3 Data Cleaning and Quality, student slides dated 1 October 2026: quality/dictionary (6–15); duplicates (16–20); missingness and responses (23–39); unusual values (40–44); audit/assertions/reporting (46–55). This lab implements the lecture instruction to keep missing values unfilled. It uses built-in synthetic data so no external dataset download is needed.
Python · cell 30 — read here, run in your notebook
assert all(checks.values()), "Complete the checks before exporting."
assert STUDENT_ID != "DEMO123", "Enter your own ID, then rerun all cells before exporting."
assert STUDENT_NAME != "Your name" and CLASS_GROUP != "Your class group", "Complete your name and class group."
zip_name = f"{STUDENT_ID}_Lab3_Data_and_Log.zip"
with zipfile.ZipFile(zip_name, "w", zipfile.ZIP_DEFLATED) as z:
z.writestr(f"{STUDENT_ID}_Lab3_Cleaned_Registry.csv", clean.to_csv(index=False))
z.writestr(f"{STUDENT_ID}_Lab3_Cleaning_Log.csv", cleaning_log.to_csv(index=False))
z.writestr(f"{STUDENT_ID}_Lab3_Raw_Export.csv", raw.to_csv(index=False))
z.writestr("VERSION.txt", f"Student: {STUDENT_ID}\nName: {STUDENT_NAME}\nGroup: {CLASS_GROUP}\nVersion: {seed:08x}\nSynthetic teaching data. Missing values intentionally unfilled.\n")
print("Created", zip_name, "— also download your completed notebook separately.")
try:
from google.colab import files
files.download(zip_name)
except ImportError:
display(FileLink(zip_name))Source: Copy of Lab_03_Data_Cleaning_and_Quality_Student(1).ipynb. The download is a study copy; the original upload is unchanged.