Learning outcomes
After studying this lesson, you will be able to:
- Perform initial checks of completeness, types, ranges, and duplicates.
- Distinguish a missing value, zero, and a structurally absent field.
- Explain common missing-data strategies and their limitations.
- Distinguish an erroneous value from a rare but genuine observation.
1. What data quality means
Data quality is fitness for a particular task, not an abstract label of “good” or “bad.” Common dimensions include accuracy, completeness, consistency, timeliness, uniqueness, and conformity to definitions.
A dataset may be adequate for annual energy analysis but unsuitable for hourly forecasting if measurements are too sparse. Quality checks must be tied to the intended use.
2. Initial checks
Inspect row and column counts, field types, units, valid ranges, key uniqueness, duplicate rows, and gaps in timestamps. Compare summaries across sources, periods, and entity groups.
Automated rules help, but each rule needs a domain reason. Negative building floor area is impossible by definition; a negative temperature may be entirely normal.
3. Missing is not zero
A missing value means the value is unknown or was not recorded. Zero is a measured or meaningful value. “Not applicable” is a different state, such as an elevator field for a building without an elevator.
Missingness may result from random sensor failure, non-response, unavailable measurements, or systematic gaps under particular conditions. Sometimes the absence itself carries information.
4. Strategies for missing values
Options include dropping rows or features, mean/median imputation, a separate category, model-based imputation, and a missingness indicator. The choice depends on the proportion, mechanism, and role of the variable.
Mean imputation is simple but reduces variation and can distort relationships. Dropping rows may change the population represented. A missingness indicator is not always useful and may encode collection-process artifacts. Fit imputation values on training data only.
5. Outliers: error or useful signal?
An outlier is an observation that differs substantially from most of the data. It may come from sensor failure, data entry, a rare event, or a genuine extreme operating condition. Difference alone does not prove an error.
Check units, equipment, timestamps, and the original source first. Do not remove rare values merely because they worsen a metric: emergency conditions may be exactly what the model should detect.
6. Practice: design quality controls
For an energy dataset, define checks: values are non-negative; timestamps are unique for each building-hour pair; units are consistent; scale changes do not create impossible jumps; and missingness is measured per sensor.
For each issue, assign an action: correct it from the source, mark it unknown, exclude it with justification, or retain it as a genuine extreme. Record the reason and the impact.