← BIT 304 Data Science and Analytics
Data Cleaning and Exploration
Notes
Establishing data quality
Data cleaning identifies and resolves problems that make analysis unreliable. Begin by checking the meaning, source and expected type of each variable, then inspect missing values, duplicates, inconsistent units and invalid ranges. A data dictionary records definitions and permitted values. Missing does not automatically mean zero: an absent sales amount may indicate an unfinished transaction rather than no sale. Preserve raw data and document transformations so results can be reproduced. Remove duplicate events only after confirming their identity, since two transactions with equal amounts may still represent different legitimate purchases.
Exploring before drawing conclusions
Exploratory data analysis uses summary statistics and visual inspection to understand distributions, relationships and unusual observations. The mean is the arithmetic average; the median is the middle ordered value and is less sensitive to extreme observations. Compare categories and time periods to expose hidden differences. An outlier is an observation unusually far from others, but it may reflect a real event rather than an error. Investigate its context before removal. When preparing predictive models, learn imputation rules from training data only to avoid leaking information from the evaluation dataset.
Practical example
A Nairobi e-commerce startup examines seven order values recorded as KES 500, 700, 700, 900, 1,200, 8,000 and one blank. The six observed values total KES 12,000, giving a mean of 12,000 divided by 6, or KES 2,000. Their median is the average of the third and fourth values, or KES 800. The KES 8,000 purchase pulls the mean upward and needs verification against its receipt. The blank remains missing until investigated. If it is incorrectly replaced with zero, the mean becomes approximately KES 1,714.29, changing the result without establishing what actually happened.
Quiz
1. Why should a missing transaction amount not automatically become zero?
2. What is the best first response to an unusually large genuine-looking order?
3. For values 500, 700, 700, 900, 1200 and 8000, what is the median?
Sign in to submit and securely grade your quiz.