Pandas: Filtering Rows by Condition

Most analysis questions are really filtering questions in disguise. "Which customers spent over ten thousand?" "How many orders came from Pune last quarter?" Every one starts by narrowing a full table down to the rows that matter. In pandas that narrowing is called boolean filtering, and it is the operation you will reach for more than any other.

The idea is simple and worth internalizing: a comparison on a column produces a column of True and False values, and pandas keeps only the rows marked True. Once you see filtering as "build a mask, apply the mask," every variation below becomes obvious.

The boolean mask

Start with a small illustrative sample DataFrame:

import pandas as pd

df = pd.DataFrame({
    "name": ["Aarti", "Bhaskar", "Chitra", "Devan", "Esha"],
    "city": ["Hyderabad", "Hyderabad", "Bengaluru", "Pune", "Bengaluru"],
    "age": [24, 31, 28, 45, 22],
    "salary": [48000, 65000, 52000, 90000, 47000],
})

A comparison on a column returns a boolean Series, one True/False per row:

mask = df["salary"] > 50000
print(mask)

0    False
1     True
2     True
3     True
4    False
Name: salary, dtype: bool

Pass that mask into the DataFrame and pandas keeps only the True rows:

print(df[df["salary"] > 50000])

      name       city  age  salary
1  Bhaskar  Hyderabad   31   65000
2   Chitra  Bengaluru   28   52000
3    Devan       Pune   45   90000

That is the entire mechanism. Everything else is just building richer masks.

Combining conditions

Real questions usually have more than one condition. Combine masks with & (and) and | (or), and wrap each condition in parentheses:

print(df[(df["salary"] > 50000) & (df["age"] < 35)])

      name       city  age  salary
1  Bhaskar  Hyderabad   31   65000
2   Chitra  Bengaluru   28   52000

Two rules trip up beginners here. First, use the symbols & and |, never the Python words and/or, because pandas needs to compare element by element. Second, the parentheses are mandatory: without them, Python's operator precedence evaluates the comparison in the wrong order and raises an error.

To negate a condition, put ~ in front of it: df[~(df["city"] == "Pune")] returns every row except Pune.

Cleaner filters: isin and between

When a column should match any of several values, isin is far tidier than a chain of |:

print(df[df["city"].isin(["Pune", "Bengaluru"])])

     name       city  age  salary
2  Chitra  Bengaluru   28   52000
3   Devan       Pune   45   90000
4    Esha  Bengaluru   22   47000

For a numeric range, between reads better than two comparisons and is inclusive of both endpoints by default:

print(df[df["salary"].between(47000, 52000)])

     name       city  age  salary
0   Aarti  Hyderabad   24   48000
2  Chitra  Bengaluru   28   52000
4    Esha  Bengaluru   22   47000

The query method

For complex filters, query accepts a plain string and can read more naturally, letting you name columns directly:

print(df.query("age < 35 and salary > 50000"))

      name       city  age  salary
1  Bhaskar  Hyderabad   31   65000
2   Chitra  Bengaluru   28   52000

Inside query you can use the words and, or, not, which some people find clearer than the symbol form. To reference a Python variable inside the string, prefix it with @, as in df.query("salary > @threshold").

Filtering then selecting columns

Filtering often pairs with column selection. Use loc to do both in one step: rows by condition, columns by name:

print(df.loc[df["age"] > 30, ["name", "salary"]])

      name  salary
1  Bhaskar   65000
3    Devan   90000

This loc[condition, columns] form is the workhorse of practical analysis, and it connects directly to the loc/iloc tutorial.

Practical usage

Filtering is the first move in almost every analysis. You narrow to the relevant segment, then count it, average it, or group it. A churn analysis filters to inactive users; a sales report filters to a region and quarter; a data-quality check filters to rows with missing or impossible values. Because a filter is just a saved boolean expression, you can name it, reuse it, and combine several to describe precisely the slice a stakeholder is asking about.

Common mistakes

• Using and/or instead of &/|. The Python keywords raise "truth value of a Series is ambiguous." Always use the symbol operators for element-wise logic.
• Missing parentheses. df[df["a"] > 1 & df["b"] < 2] fails because & binds tighter than the comparisons. Wrap each condition: df[(df["a"] > 1) & (df["b"] < 2)].
• Comparing to NaN with ==. df[df["col"] == None] never matches, because NaN is not equal to anything. Use df[df["col"].isna()] to find missing values.
• Assuming a filter modifies the original. df[df["age"] > 30] returns a new DataFrame; the original is unchanged. Assign the result to a variable, and use .copy() if you plan to edit it, to avoid the SettingWithCopyWarning.

In interviews

Filtering shows up constantly in take-home tasks and live coding. Interviewers hand you a dataset and ask "give me the rows where X and Y," expecting fluent boolean indexing with correct operators and parentheses. A common trap question is why and fails and & works, so be ready to explain element-wise evaluation. Knowing isin, between and query lets you write the cleanest solution, which reads as experience rather than brute force.

Where this fits in your learning path

Filtering is a cornerstone of the data analytics path and follows naturally from precise selection in loc and iloc. Once you can isolate the right rows, the next step is summarizing them by category with groupby, and filtering text columns leads into string methods. Fast, correct filtering is a daily skill in the data analyst role.