Data Validation Rules for Analysts

Data validation rules are the explicit checks that answer one question: does this data meet the conditions my analysis assumes? Ages should be between 0 and 120. Every order should have a customer ID. Status should be one of active, cancelled, pending — nothing else. Without stated rules, "clean data" is a feeling; with them, it is something you can test, prove, and re-run on tomorrow's data. Validation is the step that turns cleaning from a one-off tidy-up into a dependable process.

Validation closes the loop opened in what is data cleaning: cleaning fixes problems, validation confirms none remain, and together they catch issues in every new batch of data. It also reinforces earlier steps like removing duplicate records by checking that keys really are unique.

The main categories of rules

• Range checks — numeric values fall within sensible bounds (age 0–120, quantity ≥ 0, a percentage 0–100).
• Type checks — each column is the expected type (price numeric, date a real datetime).
• Completeness checks — required fields are not missing (every row has an ID).
• Uniqueness checks — key columns have no duplicates (one row per customer ID).
• Allowed-value checks — categorical columns contain only values from a known set (status in a fixed list).

Most real quality problems fall into one of these five. Writing a rule for each expectation is how you make "the data is fine" a testable claim.

Expressing rules in pandas

Each rule is a boolean condition; rows where it is False fail. Here is an illustrative sample with one violation of each kind.

import pandas as pd

df = pd.DataFrame({
    "cust_id": [1, 2, 3, 4, 4],                       # 4 is duplicated
    "age":     [29, 41, 214, 35, 28],                 # 214 is impossible
    "status":  ["active", "cancelled", "pending", "unknown", "active"],  # 'unknown' not allowed
    "spend":   [1200, 980, 550, None, 760],           # missing value
})

allowed_status = {"active", "cancelled", "pending"}

checks = {
    "age_in_range":    (df["age"] >= 0) & (df["age"] <= 120),
    "status_allowed":  df["status"].isin(allowed_status),
    "spend_present":   df["spend"].notna(),
    "cust_id_unique":  ~df["cust_id"].duplicated(keep=False),
}

for name, ok in checks.items():
    print(f"{name}: {(~ok).sum()} failing row(s)")

Expected output:

age_in_range: 1 failing row(s)
status_allowed: 1 failing row(s)
spend_present: 1 failing row(s)
cust_id_unique: 2 failing row(s)

Each rule is one readable line, and the failure counts tell you exactly what is wrong: one impossible age, one disallowed status, one missing spend, and two rows sharing a customer ID. To inspect the offending rows for any rule, filter with the negated condition:

print(df[~checks["age_in_range"]])

   cust_id  age   status   spend
2        3  214  pending   550.0

Now you can see the exact row and decide whether 214 is a typo for 21 or 24, or a value to drop.

Building a reusable validation report

In practice you combine the checks into a single pass so a whole dataset gets a clean bill of health or a precise list of problems:

report = pd.DataFrame({name: ~ok for name, ok in checks.items()})
df_failing = df[report.any(axis=1)]
print("rows with at least one failure:", len(df_failing))

rows with at least one failure: 3

Three of five rows failed at least one rule. This pattern — one boolean column per rule, then any(axis=1) to find all problem rows — scales to dozens of rules and produces an auditable summary you can log every time the pipeline runs.

How analysts use it

Validation runs at two moments. At the start of cleaning, it measures how dirty the raw data is, which guides where to spend effort. At the end, it confirms the cleaning actually worked before anything reaches a report. In production pipelines, the same rules run on every new batch, so a broken upstream export is caught before it corrupts a dashboard rather than after a stakeholder notices a wrong number. The crucial habit is defining a response to each failure: minor issues get flagged and logged, while critical ones — missing primary keys, an empty file — stop the pipeline and alert someone. Silent failure is the outcome validation exists to prevent.

Common mistakes

• No validation at all. Trusting that data "looks fine" means errors surface in front of stakeholders instead of in your checks.
• Rules with no defined response. Detecting a failure but doing nothing about it is only marginally better than not checking. Decide in advance what happens when each rule fails.
• Over-strict rules that block good data. A range that is too narrow rejects legitimate values and cries wolf. Base bounds on real domain knowledge.
• Validating once and never again. Data changes. Rules that run only on the first dataset miss the bad batch that arrives next month. Automate them into the pipeline.

In interviews

Expect questions like "How do you ensure data quality?" or "What checks would you run on a new dataset?" Strong answers enumerate the rule categories — range, type, completeness, uniqueness, allowed values — and show how each becomes a boolean condition in pandas. Mentioning that validation runs on every incoming batch, and that each failure needs a defined response (flag versus stop), signals you think about data quality as an ongoing process, not a one-time cleanup. If given a dataset, demonstrating the check-and-count pattern above is a clean, confident answer.

Where this fits in your learning path

Data validation is the capstone of the data cleaning cluster: it is how you prove every earlier step worked. It builds directly on what is data cleaning and reinforces specific fixes like removing duplicate records through uniqueness checks. Treating validation as a habit rather than an afterthought is a mark of a dependable analyst on the data analyst roadmap.