Step 3: Accessing and modifying dataframe rows

1. Introduction

In this blog, you will learn how to access, update, append, and drop dataframe rows using various pandas methods and attributes in Python. Dataframe rows are the horizontal slices of data that contain the values for each column in a dataframe. You can think of them as the records or observations in your dataset.

Accessing dataframe rows is an essential skill for data analysis, as it allows you to select and filter the data you want to work with. Updating dataframe rows is useful for modifying the existing data or adding new information. Appending dataframe rows is helpful for combining data from different sources or expanding your dataset. Dropping dataframe rows is necessary for removing unwanted or irrelevant data or reducing the size of your dataset.

By the end of this blog, you will be able to perform these operations on dataframe rows using pandas, a popular Python library for data manipulation and analysis. You will also see some examples of how to apply these techniques to real-world data. Let’s get started!

2. Accessing dataframe rows

One of the most common tasks in data analysis is accessing dataframe rows. This means selecting one or more rows from a dataframe based on some criteria. For example, you might want to access the first row, the last row, a specific row, or a range of rows. You might also want to access rows that match a certain condition, such as having a specific value or satisfying a logical expression.

Pandas provides several ways to access dataframe rows, depending on your needs and preferences. In this section, you will learn two of the most popular methods: using loc and iloc attributes, and using boolean indexing. These methods allow you to access dataframe rows by label, by position, or by condition, respectively.

Before you start, you will need to import pandas and create a sample dataframe to work with. You can use the following code to do so:

import pandas as pd
# create a sample dataframe with 5 rows and 4 columns
df = pd.DataFrame({'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
                   'age': [25, 30, 35, 40, 45],
                   'gender': ['F', 'M', 'M', 'M', 'F'],
                   'salary': [5000, 6000, 7000, 8000, 9000]})
# display the dataframe
df

The output should look like this:

Now you are ready to access dataframe rows using different methods. Let’s begin with the loc and iloc attributes.

2.1. Using loc and iloc

The loc and iloc attributes are two ways to access dataframe rows by label or by position, respectively. They are both special indexing operators that return a subset of the dataframe based on the specified criteria. You can use them to access a single row, a list of rows, or a slice of rows.

The loc attribute uses the row labels or index values to access the rows. You can pass a single label, a list of labels, or a slice of labels to the loc attribute, and it will return the corresponding rows. For example, you can use the following code to access the first row, the second and third rows, and the first three rows of the dataframe using loc:

# access the first row by label
df.loc[0]
# access the second and third rows by a list of labels
df.loc[[1, 2]]
# access the first three rows by a slice of labels
df.loc[0:2]

The output should look like this:

Note that when using a slice of labels, the loc attribute includes both the start and the end labels in the result. This is different from the usual Python slicing syntax, where the end index is excluded.

The iloc attribute uses the row positions or indices to access the rows. You can pass a single integer, a list of integers, or a slice of integers to the iloc attribute, and it will return the corresponding rows. For example, you can use the following code to access the first row, the second and third rows, and the first three rows of the dataframe using iloc:

# access the first row by position
df.iloc[0]
# access the second and third rows by a list of positions
df.iloc[[1, 2]]
# access the first three rows by a slice of positions
df.iloc[0:3]

The output should look like this:

Note that when using a slice of positions, the iloc attribute follows the usual Python slicing syntax, where the end index is excluded.

Both the loc and iloc attributes can also be used to access specific columns along with the rows, by passing a second argument to the indexing operator. For example, you can use the following code to access the name and salary columns of the first three rows using loc and iloc:

# access the name and salary columns of the first three rows by label using loc
df.loc[0:2, ['name', 'salary']]
# access the name and salary columns of the first three rows by position using iloc
df.iloc[0:3, [0, 3]]

The output should look like this:

As you can see, the loc and iloc attributes are very powerful and flexible ways to access dataframe rows by label or by position. However, sometimes you might want to access rows based on a condition, such as having a certain value or satisfying a logical expression. For that, you can use boolean indexing, which we will cover in the next section.

2.2. Using boolean indexing

Another way to access dataframe rows is by using boolean indexing. This means selecting the rows that satisfy a certain condition or a logical expression. For example, you might want to access the rows that have a specific value in a column, or the rows that meet a certain criteria based on multiple columns. Boolean indexing allows you to filter the dataframe based on the values of the data, rather than the labels or positions of the rows.

To use boolean indexing, you need to create a boolean series that represents the condition or expression you want to apply to the dataframe. A boolean series is a series of True or False values that correspond to each row in the dataframe. You can create a boolean series by using comparison operators, logical operators, or pandas methods on the dataframe or its columns. For example, you can use the following code to create a boolean series that indicates which rows have a salary greater than 7000:

# create a boolean series that indicates which rows have a salary greater than 7000
df['salary'] > 7000

The output should look like this:

0    False
1    False
2    False
3     True
4     True
Name: salary, dtype: bool

Once you have a boolean series, you can pass it to the indexing operator of the dataframe, and it will return only the rows that have a True value in the boolean series. For example, you can use the following code to access the rows that have a salary greater than 7000 using the boolean series:

# access the rows that have a salary greater than 7000 using the boolean series
df[df['salary'] > 7000]

The output should look like this:

You can also combine multiple conditions or expressions using logical operators, such as & (and), | (or), and ~ (not). For example, you can use the following code to access the rows that have a salary greater than 7000 and a gender of F using boolean indexing:

# access the rows that have a salary greater than 7000 and a gender of F using boolean indexing
df[(df['salary'] > 7000) & (df['gender'] == 'F')]

The output should look like this:

Note that when using logical operators, you need to enclose each condition or expression in parentheses, otherwise you will get an error.

Another way to create a boolean series is by using pandas methods that return a boolean value for each row. For example, you can use the isin method to check if a column value is in a given list of values. For example, you can use the following code to access the rows that have a name of Alice or Bob using boolean indexing:

# access the rows that have a name of Alice or Bob using boolean indexing
df[df['name'].isin(['Alice', 'Bob'])]

The output should look like this:

As you can see, boolean indexing is a very useful and flexible way to access dataframe rows based on a condition or a logical expression. It allows you to filter the dataframe based on the values of the data, rather than the labels or positions of the rows. However, sometimes you might want to modify the existing data or add new information to the dataframe. For that, you can use methods to update, append, or drop dataframe rows, which we will cover in the next sections.

3. Updating dataframe rows

Sometimes, you might want to update the existing data or add new information to the dataframe rows. For example, you might want to change the values of a column, add a new column, or apply a function to a column. Updating dataframe rows is useful for modifying the data according to your needs and preferences.

Pandas provides several ways to update dataframe rows, depending on the type and scope of the update you want to perform. In this section, you will learn two of the most common methods: using the assignment operator, and using the update method. These methods allow you to update dataframe rows by assigning new values, or by merging with another dataframe or series.

Before you start, you will need to import pandas and create a sample dataframe to work with. You can use the same dataframe that we used in the previous section, or you can create a new one. You can use the following code to do so:

import pandas as pd
# create a sample dataframe with 5 rows and 4 columns
df = pd.DataFrame({'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
                   'age': [25, 30, 35, 40, 45],
                   'gender': ['F', 'M', 'M', 'M', 'F'],
                   'salary': [5000, 6000, 7000, 8000, 9000]})
# display the dataframe
df

The output should look like this:

Now you are ready to update dataframe rows using different methods. Let’s begin with the assignment operator.

3.1. Using assignment operator

The assignment operator is one of the simplest and most intuitive ways to update dataframe rows. It allows you to assign new values to a column, a row, or a subset of the dataframe. You can use the assignment operator to change the values of an existing column, add a new column, or apply a function to a column. The assignment operator is the equal sign (=), and it works by assigning the value on the right-hand side to the object on the left-hand side.

For example, you can use the assignment operator to change the values of the salary column by multiplying them by 1.1, to give everyone a 10% raise. You can use the following code to do so:

# change the values of the salary column by multiplying them by 1.1
df['salary'] = df['salary'] * 1.1
# display the dataframe
df

The output should look like this:

You can also use the assignment operator to add a new column to the dataframe, by assigning a value or a series to a new column name. For example, you can use the following code to add a new column called bonus, which is 10% of the salary:

# add a new column called bonus, which is 10% of the salary
df['bonus'] = df['salary'] * 0.1
# display the dataframe
df

The output should look like this:

You can also use the assignment operator to apply a function to a column, by passing the column to the function and assigning the result to the column. For example, you can use the following code to apply the upper function to the name column, to make all the names uppercase:

# apply the upper function to the name column
df['name'] = df['name'].apply(upper)
# display the dataframe
df

The output should look like this:

As you can see, the assignment operator is a very simple and convenient way to update dataframe rows by assigning new values, adding new columns, or applying functions. However, sometimes you might want to update the dataframe rows by merging with another dataframe or series, rather than assigning new values. For that, you can use the update method, which we will cover in the next section.

3.2. Using update method

The update method is another way to update dataframe rows by merging with another dataframe or series. It allows you to update the values of the original dataframe with the values of the other dataframe or series, based on the matching index or column names. The update method is useful for updating the dataframe rows with new or missing information from another source.

To use the update method, you need to pass another dataframe or series as an argument to the method, and it will modify the original dataframe in place. You can also specify some optional parameters, such as join, which determines how to handle non-matching indexes or columns, and overwrite, which determines whether to overwrite the existing values or not. For example, you can use the following code to create another dataframe with some updated information, and use the update method to merge it with the original dataframe:

# create another dataframe with some updated information
df2 = pd.DataFrame({'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
                    'age': [26, 31, 36, 41, 46],
                    'gender': ['F', 'M', 'M', 'M', 'F'],
                    'bonus': [600, 700, 800, 900, 1000]})
# display the dataframe
df2

The output should look like this:

# use the update method to merge the original dataframe with the new dataframe
df.update(df2)
# display the dataframe
df

The output should look like this:

As you can see, the update method has updated the age and bonus columns of the original dataframe with the values of the new dataframe, based on the matching index and column names. The salary and gender columns have remained unchanged, as they were not present in the new dataframe. The update method has also converted the age and bonus columns to float type, as they were float type in the new dataframe.

The update method is a very useful and flexible way to update dataframe rows by merging with another dataframe or series, based on the matching index or column names. It allows you to update the dataframe rows with new or missing information from another source. However, sometimes you might want to add new rows to the dataframe, rather than updating the existing rows. For that, you can use methods to append or concatenate dataframe rows, which we will cover in the next sections.

4. Appending dataframe rows

Sometimes, you might want to add new rows to the dataframe, rather than updating the existing rows. For example, you might want to combine data from different sources, or expand your dataset with new observations. Appending dataframe rows is useful for increasing the size and diversity of your dataset.

Pandas provides several ways to append dataframe rows, depending on the source and format of the data you want to add. In this section, you will learn two of the most common methods: using the append method, and using the concat function. These methods allow you to append dataframe rows by adding another dataframe, a series, or a dictionary to the end of the original dataframe.

Before you start, you will need to import pandas and create a sample dataframe to work with. You can use the same dataframe that we used in the previous sections, or you can create a new one. You can use the following code to do so:

import pandas as pd
# create a sample dataframe with 5 rows and 4 columns
df = pd.DataFrame({'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
                   'age': [25, 30, 35, 40, 45],
                   'gender': ['F', 'M', 'M', 'M', 'F'],
                   'salary': [5000, 6000, 7000, 8000, 9000]})
# display the dataframe
df

The output should look like this:

Now you are ready to append dataframe rows using different methods. Let’s begin with the append method.

4.1. Using append method

The append method is one of the simplest and most intuitive ways to append dataframe rows. It allows you to add another dataframe, a series, or a dictionary to the end of the original dataframe. The append method is useful for combining data from different sources or adding new observations to your dataset.

To use the append method, you need to pass another dataframe, a series, or a dictionary as an argument to the method, and it will return a new dataframe with the appended rows. You can also specify some optional parameters, such as ignore_index, which determines whether to ignore the index values of the appended rows or not, and sort, which determines whether to sort the columns or not. For example, you can use the following code to create another dataframe with some new information, and use the append method to add it to the original dataframe:

# create another dataframe with some new information
df3 = pd.DataFrame({'name': ['Frank', 'Grace', 'Helen', 'Ivan', 'Jack'],
                    'age': [50, 55, 60, 65, 70],
                    'gender': ['M', 'F', 'F', 'M', 'M'],
                    'salary': [10000, 11000, 12000, 13000, 14000]})
# display the dataframe
df3

The output should look like this:

# use the append method to add the new dataframe to the original dataframe
df = df.append(df3, ignore_index=True)
# display the dataframe
df

The output should look like this:

As you can see, the append method has added the new dataframe to the end of the original dataframe, and assigned new index values to the appended rows. The bonus column has been filled with NaN values for the new rows, as they were not present in the new dataframe. The append method has also preserved the data types of the columns, as they were compatible between the two dataframes.

The append method is a very simple and convenient way to append dataframe rows by adding another dataframe, a series, or a dictionary to the end of the original dataframe. It allows you to combine data from different sources or add new observations to your dataset. However, sometimes you might want to append dataframe rows along a different axis, such as horizontally or vertically. For that, you can use the concat function, which we will cover in the next section.

4.2. Using concat function

The concat function is another way to append dataframe rows. It allows you to concatenate or join two or more dataframes, series, or dictionaries along a specified axis. The concat function is useful for appending dataframe rows horizontally or vertically, or for combining data with different structures or levels of detail.

To use the concat function, you need to pass a list of dataframes, series, or dictionaries as an argument to the function, and it will return a new dataframe with the concatenated rows. You can also specify some optional parameters, such as axis, which determines whether to concatenate along rows (axis=0) or columns (axis=1), and join, which determines how to handle non-matching indexes or columns. For example, you can use the following code to create another dataframe with some additional information, and use the concat function to add it to the original dataframe along the column axis:

# create another dataframe with some additional information
df4 = pd.DataFrame({'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
                    'city': ['New York', 'Los Angeles', 'Chicago', 'Houston', 'Phoenix'],
                    'hobby': ['reading', 'gaming', 'cooking', 'traveling', 'dancing']})
# display the dataframe
df4

The output should look like this:

# use the concat function to add the new dataframe to the original dataframe along the column axis
df = pd.concat([df, df4], axis=1)
# display the dataframe
df

The output should look like this:

As you can see, the concat function has added the new dataframe to the right of the original dataframe, and matched the rows by the index values. The name column has been duplicated, as it was present in both dataframes. The concat function has also preserved the data types of the columns, as they were compatible between the two dataframes.

The concat function is a very powerful and flexible way to append dataframe rows by concatenating or joining two or more dataframes, series, or dictionaries along a specified axis. It allows you to append dataframe rows horizontally or vertically, or to combine data with different structures or levels of detail. However, sometimes you might want to drop some rows from the dataframe, rather than appending new rows. For that, you can use methods to drop or filter dataframe rows, which we will cover in the next sections.

5. Dropping dataframe rows

Sometimes, you might want to drop some rows from the dataframe, rather than appending new rows. For example, you might want to remove unwanted or irrelevant data, or reduce the size of your dataset. Dropping dataframe rows is useful for cleaning and optimizing your dataset.

Pandas provides several ways to drop dataframe rows, depending on the criteria and logic you want to use. In this section, you will learn two of the most common methods: using the drop method, and using the query method. These methods allow you to drop dataframe rows by index, by label, or by condition.

Before you start, you will need to import pandas and create a sample dataframe to work with. You can use the same dataframe that we used in the previous sections, or you can create a new one. You can use the following code to do so:

import pandas as pd
# create a sample dataframe with 10 rows and 6 columns
df = pd.DataFrame({'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve', 'Frank', 'Grace', 'Helen', 'Ivan', 'Jack'],
                   'age': [26, 31, 36, 41, 46, 50, 55, 60, 65, 70],
                   'gender': ['F', 'M', 'M', 'M', 'F', 'M', 'F', 'F', 'M', 'M'],
                   'salary': [5500, 6600, 7700, 8800, 9900, 10000, 11000, 12000, 13000, 14000],
                   'bonus': [600, 700, 800, 900, 1000, np.nan, np.nan, np.nan, np.nan, np.nan],
                   'city': ['New York', 'Los Angeles', 'Chicago', 'Houston', 'Phoenix', 'New York', 'Los Angeles', 'Chicago', 'Houston', 'Phoenix'],
                   'hobby': ['reading', 'gaming', 'cooking', 'traveling', 'dancing', 'reading', 'gaming', 'cooking', 'traveling', 'dancing']})
# display the dataframe
df

The output should look like this:

Now you are ready to drop dataframe rows using different methods. Let’s begin with the drop method.

5.1. Using drop method

The drop method is one of the simplest and most intuitive ways to drop dataframe rows. It allows you to drop one or more rows from the dataframe by specifying their index or label. The drop method is useful for removing unwanted or irrelevant data, or reducing the size of your dataset.

To use the drop method, you need to pass the index or label of the rows you want to drop as an argument to the method, and it will return a new dataframe without the dropped rows. You can also specify some optional parameters, such as axis, which determines whether to drop rows (axis=0) or columns (axis=1), and inplace, which determines whether to modify the original dataframe or return a new one. For example, you can use the following code to drop the first and the last rows of the dataframe by index using the drop method:

# drop the first and the last rows of the dataframe by index
df = df.drop([0, 9])
# display the dataframe
df

The output should look like this:

As you can see, the drop method has removed the first and the last rows of the dataframe by index, and returned a new dataframe with the remaining rows. The index values of the dataframe have also been updated to reflect the changes. The drop method has also preserved the data types of the columns, as they were not affected by the operation.

The drop method is a very simple and convenient way to drop dataframe rows by index or label. It allows you to remove unwanted or irrelevant data, or reduce the size of your dataset. However, sometimes you might want to drop rows based on a condition, such as having a certain value or satisfying a logical expression. For that, you can use the query method, which we will cover in the next section.

5.2. Using query method

The query method is another way to drop dataframe rows. It allows you to drop rows from the dataframe that do not satisfy a given condition or expression. The query method is useful for filtering or subsetting your dataset based on some criteria.

To use the query method, you need to pass a string that represents the condition or expression you want to apply to the dataframe as an argument to the method, and it will return a new dataframe with the rows that satisfy the condition or expression. You can also specify some optional parameters, such as inplace, which determines whether to modify the original dataframe or return a new one, and engine, which determines the backend parser to use. For example, you can use the following code to drop the rows of the dataframe that have a salary less than 8000 using the query method:

# drop the rows of the dataframe that have a salary less than 8000
df = df.query('salary >= 8000')
# display the dataframe
df

The output should look like this:

As you can see, the query method has removed the rows of the dataframe that have a salary less than 8000, and returned a new dataframe with the remaining rows. The index values of the dataframe have also been updated to reflect the changes. The query method has also preserved the data types of the columns, as they were not affected by the operation.

The query method is a very powerful and flexible way to drop dataframe rows by condition or expression. It allows you to filter or subset your dataset based on some criteria. However, sometimes you might want to update some rows of the dataframe, rather than dropping them. For that, you can use methods to update or modify dataframe rows, which we will cover in the previous sections.

6. Conclusion

In this blog, you have learned how to access, update, append, and drop dataframe rows using various pandas methods and attributes in Python. You have seen how to use the loc and iloc attributes to access dataframe rows by label or by position, and how to use the boolean indexing to access dataframe rows by condition. You have also learned how to use the assignment operator and the update method to update dataframe rows by value or by another dataframe. You have also learned how to use the append method and the concat function to append dataframe rows by adding new rows or by joining two or more dataframes. Finally, you have learned how to use the drop method and the query method to drop dataframe rows by index, by label, or by condition.

By mastering these techniques, you will be able to manipulate and analyze your data more efficiently and effectively. You will also be able to perform various operations on dataframe rows, such as selecting, filtering, modifying, combining, or removing data. These skills are essential for any data analyst or data scientist who works with pandas and Python.

We hope you have enjoyed this blog and learned something new and useful. If you have any questions or feedback, please feel free to leave a comment below. Thank you for reading and happy coding!