← Data Science learning path0%
03
Week 3 · 56 slides + Lab 3
Data Cleaning and Quality
Data quality dimensions, cleaning decisions, missing-data mechanisms and audit trails.
Not started yet
Start Quiz mastery0%
What you’ll take away
- Good for which purpose?
- Count first. Change with evidence. Keep a log.
- Meaning first; conversion second.
- Ask why the ID repeats.
- Zero is known. Missing is unknown. Not applicable is outside this question.
- Explain the mechanism, not just the label.
- Name the denominator every time.
- A replacement is not a new observation.
- If the conclusion changes, say which assumption matters.
- Investigate the value; respect the population.
- Remove confirmed extra rows; keep useful student records.
Lecture topics
Quality means fitness for the task; Inventory before editing; Meaning comes before conversion; Repeated records need different actions; Zero, missing and not applicable; MCAR, MAR and MNAR are mechanisms; Calculate representation explicitly; Four responses to missing data; Sensitivity analysis and bounds; Outliers, invalid values and the population; Your earlier registry: the fixed outcomes
Lessons
- Quality means fitness for the taskA field can be important for one task and irrelevant to another. Judge whether the needed information is complete, sensible and trustworthy.7 min
- Inventory before editingStart by counting and recording what is there. Then make a cleaning plan that explains each change and its evidence.6 min
- Meaning comes before conversionChanging text to a number is not enough. You must know the unit, date convention and approved category meanings.7 min
- Repeated records need different actionsThe same ID can appear more than once for several reasons. The correct action depends on what the rows mean.6 min
- Zero, missing and not applicableThese three meanings are different. Replacing them all with zero changes the question your numbers answer.6 min
- MCAR, MAR and MNAR are mechanismsMCAR, MAR and MNAR describe how missingness happens. The observed counts alone do not prove which explanation is true.7 min
- Calculate representation explicitlyA group can make up a smaller share of replies than of invitations. That changes the weights in the pooled observed mean.6 min
- Four responses to missing dataRecovering a source measurement and guessing a replacement are different actions. Choose the action that fits the analysis.6 min
- Sensitivity analysis and boundsSensitivity analysis asks “What if the missing values were different?” Bounds show the smallest and largest results allowed by the known scale.6 min
- Outliers, invalid values and the populationA value can be statistically unusual and still correct. A value inside normal-looking limits can still be a recording error.6 min
- Your earlier registry: the fixed outcomesThe final registry should have one current record per expected student while keeping unresolved values honestly missing.7 min
Practise this week · Lab 3: Data Cleaning and Quality6 guided sections · Make supported cleaning choices and explain what remains uncertain.
Key terms
- Completeness
- Needed data are present.
- Validity
- Values follow rules.
- Accuracy
- Values match the true source.
- Audit log
- A record of changes and reasons.
- Roster
- The list of expected people or records.
- Parsing
- Reading text into a structured value.
- Coercion
- Replacing an unparseable value with a missing marker.
- Copy
- An extra record of the same thing.
- Correction
- A verified replacement.
- Repeated observation
- A new week, course or event.
- Zero
- Measured absence of an amount.
- Missing
- Amount unknown.
- Not applicable
- The measurement does not belong to this person or situation.
- MCAR
- Missingness unrelated to the values.
- MAR
- After using known information, missingness no longer depends on the missing value.
- MNAR
- That dependence remains.
- Response rate
- Replies divided by that group's invitations.
- Reply share
- That group's replies divided by all replies.
- Recover
- Obtain the verified observation.
- Impute
- Fill using an assumption or model.
- Exclude for analysis
- Leave out a measurement for a clearly limited calculation.
- Sensitivity scenario
- A stated assumption to explore.
- Bound
- An extreme possible value given the available information.
- Outlier flag
- A value worth checking.
- Invalid value
- Breaks a documented rule.
- Row removed
- The entire record leaves the working table.
- Value flagged
- One field is marked while the rest of the row can stay.
Practice this topic66 questions · earn XP and build your streak