Count errors and compare tradeoffs
Calculate test accuracy and distinguish overall accuracy from the consequences and distribution of errors.
In this lesson
- It starts with a question. A toy classifier gets six of eight held-out cards right. What is its accuracy on those cards?
- The idea you will find. Connect the idea — Accuracy is correct predictions divided by all tested predictions. It does not show every kind of mistake or guarantee performance elsewhere. A false alarm predicts the positive category when it is absent; a miss fails to predict it when present. Compare both, inspect coverage of relevant groups and explain the limits of a tiny test.
- How you learn it. By doing: building a chart.
- Length. 5 short steps. Progress saves on this phone.
Part of Test a simple classifier
Use fictional school sorting cards to trace a threshold rule, reserve unseen examples and count errors. This local model exercise does not train or deploy an AI service.