Collecting & Cleaning Data

Data almost never arrives ready to use. Before any real analysis can start, it usually needs to be tracked down from wherever it lives, then cleaned up: missing values filled in or removed, duplicates dropped, and inconsistent formats made consistent.

Where data actually comes from

In practice, data comes from all over: a production database, a third-party API, a CSV export someone emailed you, a form people filled out by hand. Each source tends to bring its own quirks — an API might rate-limit you, a hand-filled form will have typos, an old database export might use a totally different date format than a newer one.

Missing values

A dataset with gaps is the norm, not the exception. A customer record might be missing a phone number; a sensor might have dropped a reading. There's no single correct fix — sometimes you drop the incomplete rows, sometimes you fill the gap with a reasonable estimate (like the average of the column), and sometimes the fact that a value is missing is itself useful information.

ages = [29, 34, None, 41, 25, None, 38]
known_ages = [a for a in ages if a is not None]

print(f"Missing: {len(ages) - len(known_ages)} of {len(ages)}")
print(f"Average of known ages: {sum(known_ages) / len(known_ages):.1f}")

Missing: 2 of 7
Average of known ages: 33.4

Filtering out the None values before computing the average is a small example of a much bigger habit: knowing exactly how much of your data is missing, and making a deliberate choice about what to do with the gaps, rather than letting them silently break a calculation.

Duplicates

The same record can end up in a dataset more than once — a customer who filled out a form twice, a sync job that ran twice by accident. Left in, duplicates quietly inflate counts and skew averages toward whatever got duplicated.

Inconsistent formats

The same real-world value can be written many ways: "NY" vs "New York" vs "new york", or a date as 03/04/2026 in one system and 2026-04-03 in another. Without standardizing these, a program will happily treat them as different values entirely, even though a human reading the data knows they mean the same thing.