Keep a fair test unseen
Separate choosing a rule from evaluating it and recognise answer leakage and repeated test tuning.
In this lesson
- It starts with a question. A club chooses a rule using all eight labelled cards, then reports the score on those same cards as an unseen test. Is that description accurate?
- The idea you will find. Connect the idea — Training examples influence a rule; held-out examples evaluate the locked rule. Keep test labels hidden while choosing. Repeatedly changing the rule after seeing its test mistakes makes that set part of development, so a fresh final test is needed. Also avoid near-duplicate records crossing the split and report the small sample size.
- How you learn it. By doing: putting things in order.
- Length. 5 short steps. Progress saves on this phone.
Part of Test a simple classifier
Use fictional school sorting cards to trace a threshold rule, reserve unseen examples and count errors. This local model exercise does not train or deploy an AI service.