Error, accuracy and model complexity
Error is the share of evaluated cases predicted incorrectly. A strong result on training cases or a tiny test set does not guarantee good future predictions.
Classification error = wrong predicted labels / evaluated cases. Accuracy = 1-error when each evaluated case has one true and one predicted label and all are counted the same way. In the three-row lecture example, one mistake gives error 1/3=33.3% and accuracy 2/3=66.7%.
Training error measures fit to familiar data. Test error measures performance on held-out cases under that sampling and split design. Perfect training performance is insufficient evidence about new students.
An underfit rule is too simple to capture important structure; an overfit rule captures details specific to training data. Increasing flexibility can reduce training error while worsening unseen-case error. This is a common pattern, not a mathematical requirement that every test error must exceed training error in every finite sample. Validation guides model choices; tiny test samples make performance estimates unstable.
Sources: W4 PDF pp. 2-42; L4 Tasks 1-7. All fixed worked values here were recalculated from the supplied examples. These toy rules are teaching devices, not deployable student-support recommendations.
Use all six feature positions to calculate distance to training students. Choose the smallest distance, then take that training student's label.
Practice problems
Supplementary study questions on the lecture concepts — not verbatim exam questions. Each answer counts toward spaced review.
1. Conceptual understanding
Definitions and distinctions.
One mistake among three evaluated cases gives what error?
What does perfect training accuracy establish?
2. Application
Apply the concept to a case or calculation.
There are 3 mistakes among 20 test cases. What is error?
For those same 20 cases with 3 mistakes, what is accuracy?
3. Interpretation and limitations
Evaluate assumptions and justify conclusions.
Training error keeps falling while validation error worsens. What does this suggest?
A model gets 2 of 2 test cases correct. Which statement is justified?
Explain it in writing
Self-checked, not marked. Write a few sentences, then compare with the example answer. Your writing is saved as you type.
Show an example answer
It may fit training-specific details or noise rather than stable patterns. The training data are familiar; held-out cases reveal whether the learned rule carries over.
Check: did you answer the question, use evidence or working, and avoid claiming more than the evidence supports?
Show an example answer
An underfit rule is too simple to capture useful patterns. An overfit rule captures details specific to the training sample. Validation helps assess the trade-off.
Check: did you answer the question, use evidence or working, and avoid claiming more than the evidence supports?