1. Introduction
Pandas is a popular Python library for data analysis and manipulation. It provides various methods and functions to work with dataframes, which are two-dimensional tabular data structures with labeled rows and columns.
One of the common tasks that you may encounter when working with dataframes is filtering, which means selecting a subset of data based on some criteria. For example, you may want to filter out rows that have missing values, or columns that have a certain value.
In this tutorial, you will learn how to use two methods for pandas dataframe filtering: the where method and the mask method. These methods allow you to filter data and replace values in a dataframe with ease and flexibility.
You will also learn how to compare the where and mask methods and understand their differences and similarities.
By the end of this tutorial, you will be able to apply the where and mask methods to your own dataframes and perform filtering and replacing values with confidence.
Are you ready to get started? Let’s go!
2. Creating a Sample DataFrame
Before you can apply the where and mask methods to filter data and replace values, you need to have a dataframe to work with. In this section, you will learn how to create a sample dataframe using pandas.
A dataframe is a two-dimensional data structure that consists of rows and columns. Each column represents a variable, and each row represents an observation. You can think of a dataframe as a spreadsheet or a table.
To create a dataframe, you can use the DataFrame constructor from pandas. You can pass a dictionary of lists as the data argument, where the keys are the column names and the values are the lists of data. You can also specify the index argument to label the rows.
For example, you can create a dataframe that contains information about some fruits, such as their name, color, weight, and price. You can use the following code:
import pandas as pd # import pandas library
# create a dictionary of lists
data = {
"name": ["apple", "banana", "cherry", "durian", "elderberry"],
"color": ["red", "yellow", "red", "green", "purple"],
"weight": [150, 120, 80, 1000, 10],
"price": [0.5, 0.4, 0.6, 5, 1]
}
# create a dataframe from the dictionary
df = pd.DataFrame(data, index = ["A", "B", "C", "D", "E"])
# display the dataframe
df
The output of the code is:
| name | color | weight | price | |
|---|---|---|---|---|
| A | apple | red | 150 | 0.5 |
| B | banana | yellow | 120 | 0.4 |
| C | cherry | red | 80 | 0.6 |
| D | durian | green | 1000 | 5 |
| E | elderberry | purple | 10 | 1 |
Congratulations, you have created a sample dataframe using pandas! You can use this dataframe to practice the where and mask methods in the next sections.
3. Filtering Data with the Where Method
Now that you have a sample dataframe, you can start applying the where method to filter data. The where method is one of the methods for pandas dataframe filtering that allows you to select a subset of data based on a condition.
The where method takes a boolean expression as an argument and returns a dataframe with the same shape as the original one, but with the values that do not satisfy the condition replaced by NaN (Not a Number).
For example, if you want to filter the dataframe to select only the fruits that have a weight less than 200 grams, you can use the following code:
# filter the dataframe by weight df.where(df["weight"] < 200)
The output of the code is:
| name | color | weight | price | |
|---|---|---|---|---|
| A | apple | red | 150 | 0.5 |
| B | banana | yellow | 120 | 0.4 |
| C | cherry | red | 80 | 0.6 |
| D | NaN | NaN | NaN | NaN |
| E | elderberry | purple | 10 | 1 |
As you can see, the where method has replaced the values in the row D with NaN, because the weight of the durian is 1000 grams, which is greater than 200 grams.
You can also use multiple conditions with the where method, by using the logical operators & (and), | (or), and ~ (not). For example, if you want to filter the dataframe to select only the fruits that have a weight less than 200 grams and a price less than 1 dollar, you can use the following code:
# filter the dataframe by weight and price df.where((df["weight"] < 200) & (df["price"] < 1))
The output of the code is:
| name | color | weight | price | |
|---|---|---|---|---|
| A | apple | red | 150 | 0.5 |
| B | banana | yellow | 120 | 0.4 |
| C | cherry | red | 80 | 0.6 |
| D | NaN | NaN | NaN | NaN |
| E | NaN | NaN | NaN | NaN |
As you can see, the where method has replaced the values in the rows D and E with NaN, because the price of the durian is 5 dollars, and the price of the elderberry is 1 dollar, which are both greater than or equal to 1 dollar.
The where method is a useful way to filter data and keep the original shape of the dataframe. However, sometimes you may want to remove the rows or columns that have NaN values, instead of keeping them. In that case, you can use the dropna method, which will be explained in the next section.
4. Replacing Values with the Where Method
In the previous section, you learned how to use the where method to filter data and keep the original shape of the dataframe. However, sometimes you may not want to replace the values that do not satisfy the condition with NaN, but with some other value. In that case, you can use the where method with an additional argument: the other argument.
The other argument allows you to specify what value to use instead of NaN when the condition is not met. For example, if you want to filter the dataframe to select only the fruits that have a weight less than 200 grams, and replace the other values with 0, you can use the following code:
# filter the dataframe by weight and replace with 0 df.where(df["weight"] < 200, other = 0)
The output of the code is:
| name | color | weight | price | |
|---|---|---|---|---|
| A | apple | red | 150 | 0.5 |
| B | banana | yellow | 120 | 0.4 |
| C | cherry | red | 80 | 0.6 |
| D | 0 | 0 | 0 | 0 |
| E | elderberry | purple | 10 | 1 |
As you can see, the where method has replaced the values in the row D with 0, instead of NaN.
You can also use the other argument to replace the values with another column or series, as long as they have the same shape as the dataframe. For example, if you want to filter the dataframe to select only the fruits that have a weight less than 200 grams, and replace the other values with the price column, you can use the following code:
# filter the dataframe by weight and replace with price df.where(df["weight"] < 200, other = df["price"])
The output of the code is:
| name | color | weight | price | |
|---|---|---|---|---|
| A | apple | red | 150 | 0.5 |
| B | banana | yellow | 120 | 0.4 |
| C | cherry | red | 80 | 0.6 |
| D | 5 | 5 | 5 | 5 |
| E | elderberry | purple | 10 | 1 |
As you can see, the where method has replaced the values in the row D with the values from the price column.
The where method is a powerful way to filter data and replace values in a dataframe. However, it is not the only method that can do that. In the next section, you will learn about another method for pandas dataframe filtering: the mask method.
5. Filtering Data with the Mask Method
Another method for pandas dataframe filtering that you can use is the mask method. The mask method is similar to the where method, but with one key difference: it does the opposite of the where method.
The mask method takes a boolean expression as an argument and returns a dataframe with the same shape as the original one, but with the values that satisfy the condition replaced by NaN (Not a Number).
For example, if you want to filter the dataframe to select only the fruits that have a weight greater than or equal to 200 grams, you can use the following code:
# filter the dataframe by weight df.mask(df["weight"] < 200)
The output of the code is:
| name | color | weight | price | |
|---|---|---|---|---|
| A | NaN | NaN | NaN | NaN |
| B | NaN | NaN | NaN | NaN |
| C | NaN | NaN | NaN | NaN |
| D | durian | green | 1000 | 5 |
| E | NaN | NaN | NaN | NaN |
As you can see, the mask method has replaced the values in the rows A, B, C, and E with NaN, because the weight of the fruits is less than 200 grams.
You can also use multiple conditions with the mask method, by using the logical operators & (and), | (or), and ~ (not). For example, if you want to filter the dataframe to select only the fruits that have a weight greater than or equal to 200 grams and a price greater than or equal to 1 dollar, you can use the following code:
# filter the dataframe by weight and price df.mask((df["weight"] < 200) | (df["price"] < 1))
The output of the code is:
| name | color | weight | price | |
|---|---|---|---|---|
| A | NaN | NaN | NaN | NaN |
| B | NaN | NaN | NaN | NaN |
| C | NaN | NaN | NaN | NaN |
| D | durian | green | 1000 | 5 |
| E | elderberry | purple | 10 | 1 |
As you can see, the mask method has replaced the values in the rows A, B, and C with NaN, because the weight of the fruits is less than 200 grams, or the price of the fruits is less than 1 dollar.
The mask method is a useful way to filter data and keep the original shape of the dataframe. However, sometimes you may want to remove the rows or columns that have NaN values, instead of keeping them. In that case, you can use the dropna method, which will be explained in the next section.
6. Replacing Values with the Mask Method
The mask method can also be used to replace values in a dataframe, just like the where method. However, the mask method has an advantage over the where method: it can replace values with different values for different columns.
The mask method can take a dictionary or a series as the other argument, where the keys or the index are the column names and the values are the values to use instead of NaN when the condition is met. For example, if you want to filter the dataframe to select only the fruits that have a weight less than 200 grams, and replace the other values with different values for each column, you can use the following code:
# filter the dataframe by weight and replace with different values
df.mask(df["weight"] < 200, other = {"name": "unknown", "color": "black", "weight": -1, "price": 0})
The output of the code is:
| name | color | weight | price | |
|---|---|---|---|---|
| A | NaN | NaN | NaN | NaN |
| B | NaN | NaN | NaN | NaN |
| C | NaN | NaN | NaN | NaN |
| D | unknown | black | -1 | 0 |
| E | NaN | NaN | NaN | NaN |
As you can see, the mask method has replaced the values in the row D with different values for each column, according to the dictionary passed as the other argument.
You can also use the other argument to replace the values with another dataframe, as long as it has the same shape as the original one. For example, if you want to filter the dataframe to select only the fruits that have a weight less than 200 grams, and replace the other values with the values from another dataframe, you can use the following code:
# create another dataframe
data2 = {
"name": ["orange", "pear", "grape", "mango", "blueberry"],
"color": ["orange", "green", "green", "yellow", "blue"],
"weight": [200, 180, 5, 300, 1],
"price": [0.8, 0.7, 0.2, 2, 0.5]
}
df2 = pd.DataFrame(data2, index = ["A", "B", "C", "D", "E"])
# filter the dataframe by weight and replace with another dataframe
df.mask(df["weight"] < 200, other = df2)
The output of the code is:
| name | color | weight | price | |
|---|---|---|---|---|
| A | orange | orange | 200 | 0.8 |
| B | pear | green | 180 | 0.7 |
| C | grape | green | 5 | 0.2 |
| D | durian | green | 1000 | 5 |
| E | blueberry | blue | 1 | 0.5 |
As you can see, the mask method has replaced the values in the rows A, B, C, and E with the values from the other dataframe.
The mask method is a flexible way to filter data and replace values in a dataframe. However, it is not the only way to do that. In the next section, you will learn how to compare the mask method with the where method and understand their differences and similarities.
7. Comparing the Where and Mask Methods
In the previous sections, you learned how to use the where and mask methods to filter data and replace values in a dataframe. You may wonder what are the differences and similarities between these two methods, and when to use one over the other. In this section, you will learn how to compare the where and mask methods and understand their pros and cons.
The main difference between the where and mask methods is that they do the opposite of each other. The where method returns a dataframe with the values that do not satisfy the condition replaced by NaN or another value, while the mask method returns a dataframe with the values that satisfy the condition replaced by NaN or another value.
This means that you can use either method to achieve the same result, as long as you use the opposite condition. For example, if you want to filter the dataframe to select only the fruits that have a weight less than 200 grams, you can use either of the following codes:
# using the where method df.where(df["weight"] < 200) # using the mask method df.mask(df["weight"] >= 200)
Both codes will produce the same output:
| name | color | weight | price | |
|---|---|---|---|---|
| A | apple | red | 150 | 0.5 |
| B | banana | yellow | 120 | 0.4 |
| C | cherry | red | 80 | 0.6 |
| D | NaN | NaN | NaN | NaN |
| E | elderberry | purple | 10 | 1 |
The main similarity between the where and mask methods is that they both allow you to filter data and replace values in a dataframe with ease and flexibility. They both take a boolean expression as an argument and return a dataframe with the same shape as the original one. They both also accept an other argument that allows you to specify what value to use instead of NaN when the condition is not met or met, respectively.
The main advantage of using the where and mask methods is that they preserve the original shape of the dataframe, which can be useful for alignment and concatenation purposes. However, this also means that they may produce a lot of NaN values, which can be problematic for some operations and calculations. In that case, you may want to use the dropna method to remove the rows or columns that have NaN values, or use another method to filter data, such as the loc or iloc methods, which will be explained in the next section.
The main disadvantage of using the where and mask methods is that they can be confusing and error-prone, especially if you use the wrong condition or the wrong method. For example, if you use the where method with the condition df["weight"] >= 200, instead of df["weight"] < 200, you will get the opposite result of what you want. Similarly, if you use the mask method with the condition df["weight"] < 200, instead of df["weight"] >= 200, you will also get the wrong result. Therefore, you need to be careful and check your code before using these methods.
The where and mask methods are powerful tools for pandas dataframe filtering that allow you to select a subset of data based on a condition and replace values in a dataframe. However, they are not the only tools that you can use. In the next section, you will learn about another method for filtering data: the loc method.
8. Conclusion
In this tutorial, you have learned how to use the where and mask methods to filter data and replace values in a pandas dataframe. You have also learned how to compare these two methods and understand their differences and similarities.
Here are some key points to remember:
- The where method returns a dataframe with the values that do not satisfy the condition replaced by NaN or another value.
- The mask method returns a dataframe with the values that satisfy the condition replaced by NaN or another value.
- Both methods preserve the original shape of the dataframe, which can be useful for alignment and concatenation purposes.
- Both methods accept an other argument that allows you to specify what value to use instead of NaN when the condition is not met or met, respectively.
- Both methods can be used to achieve the same result, as long as you use the opposite condition.
- Both methods can be confusing and error-prone, especially if you use the wrong condition or the wrong method.
You can use the where and mask methods to filter data and replace values in a dataframe with ease and flexibility. However, they are not the only methods that you can use. There are other methods for pandas dataframe filtering, such as the loc and iloc methods, which allow you to select a subset of data based on labels or positions. You can learn more about these methods in the pandas documentation.
We hope you enjoyed this tutorial and found it useful. If you have any questions or feedback, please let us know in the comments below. Thank you for reading and happy coding!
