Three leakage routes
Leakage lets a model use information it should not have. It can make test results look better than real future performance.
Future leakage: using information unavailable at the decision time, such as W5 logins in a W4 prediction. Target leakage: putting the answer or an answer-revealing field into the predictors. Test leakage: letting held-out examples influence preprocessing, feature choice or model selection.
Example: training values [2,4] have mean 3. Adding held-out [100] gives 35.333. Only 3 may be saved as the training mean. The problem exists even without reading test labels because test features have influenced fitting.
Do not put the same student's registrations on both sides when evaluating performance on new students. Such a split can share person-specific information. For a future-time prediction task, design the split to respect time too. The split must mimic the kind of generalisation claimed.
Sources: W4 PDF pp. 2-42; L4 Tasks 1-7. All fixed worked values here were recalculated from the supplied examples. These toy rules are teaching devices, not deployable student-support recommendations.
The training rows supply every replacement and scaling value. Filling a gap lets the model run, but does not recover the true measurement.
The code should carry out the exact stated rules, including missing-value handling, feature order and tie-breaking. Explain these decisions rather than memorising lines.
Practice problems
Supplementary study questions on the lecture concepts — not verbatim exam questions. Each answer counts toward spaced review.
1. Conceptual understanding
Definitions and distinctions.
Fitting a scaler on training plus test features is what?
What split is appropriate when claiming performance on new students?
2. Application
Apply the concept to a case or calculation.
Which feature leaks future information into an end-of-week-4 prediction?
Which input most directly leaks the answer when predicting withdrawal?
3. Interpretation and limitations
Evaluate assumptions and justify conclusions.
Training [2,4] and test [100] are available. What training mean should be saved?
The same student's records occur in training and test while claiming performance on new students. What is the risk?
Explain it in writing
Self-checked, not marked. Write a few sentences, then compare with the example answer. Your writing is saved as you type.
Show an example answer
The fitted transformation now depends on held-out examples. The test set has influenced the procedure being evaluated, so it is no longer fully separate.
Check: did you answer the question, use evidence or working, and avoid claiming more than the evidence supports?
Show an example answer
Assign each student's records entirely to one split. If the claim also concerns future time, respect that time boundary as well.
Check: did you answer the question, use evidence or working, and avoid claiming more than the evidence supports?