14 Working with single tables in dplyr
Data frames are usually the most convenient objects for storing, plotting or analysing data in R. We also need to be able to manipulate data in data frames.
This tutorial will show you how to manipulate data frames using the dplyr
package, part of tidyverse
.
Everything that can be done with dplyr
can be done with base R. The dplyr
solution is usually easier to write and understand, and can be faster.
There are many function in dplyr
, this tutorial focuses on the functions we find most useful.
14.0.1 Load the package
You can load dplyr
with
But is is usually more convenient to load tidyverse
so that you get ggplot2
, dplyr
, readr
and other useful packages with one command.
14.0.2 More feedback tidylog
We can get more feedback on what dplyr
functions have done to the data by loading the tidylog
package.
More than once, tidylog
has helped identified bugs in my code.
14.1 The penguins
dataset
This tutorial will use the penguins
dataset from the palmerpenguins
package. This dataset includes measurements of three species of penguin.
# Load the data
data("penguins", package = "palmerpenguins")
# Show the data
penguins
# A tibble: 344 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Torgersen 39.1 18.7 181 3750
2 Adelie Torgersen 39.5 17.4 186 3800
3 Adelie Torgersen 40.3 18 195 3250
# ℹ 341 more rows
# ℹ 2 more variables: sex <fct>, year <int>
14.2 Selecting columns with select()
You can choose which columns of the data frame you want with select()
.
The first argument is the data, which is supplied by the pipe |>
, the next arguments are the names of the columns you want. The names do not need quote marks.
#select species, bill_length_mm & bill_depth_mm
penguins |>
select(species, bill_length_mm, bill_depth_mm)
select: dropped 5 variables (island, flipper_length_mm, body_mass_g, sex, year)
# A tibble: 344 × 3
species bill_length_mm bill_depth_mm
<fct> <dbl> <dbl>
1 Adelie 39.1 18.7
2 Adelie 39.5 17.4
3 Adelie 40.3 18
# ℹ 341 more rows
This is equivalent to the base R code
#select species, bill_length_mm & bill_depth_mm
penguins[, c("species", "bill_length_mm", "bill_depth_mm")]
# A tibble: 344 × 3
species bill_length_mm bill_depth_mm
<fct> <dbl> <dbl>
1 Adelie 39.1 18.7
2 Adelie 39.5 17.4
3 Adelie 40.3 18
# ℹ 341 more rows
Remember that if you want to use the output of this code in a further analysis, you need to assign it to an object name with <-
.
From the penguins
data, select
species
-
species
andbill_length_mm
- all columns except year
Hint
|>
penguins select(___, ___)
14.2.1 select()
helpers
Sometimes we don’t want to write out the names of all the columns we want to select. We might not even know them all in advance. Fortunately there are some helper functions.
If you want to select()
adjacent columns, you can use the notation first:last
.
#select species to bill_depth_mm
penguins |> select(species:bill_depth_mm)
select: dropped 4 variables (flipper_length_mm, body_mass_g, sex, year)
# A tibble: 344 × 4
species island bill_length_mm bill_depth_mm
<fct> <fct> <dbl> <dbl>
1 Adelie Torgersen 39.1 18.7
2 Adelie Torgersen 39.5 17.4
3 Adelie Torgersen 40.3 18
# ℹ 341 more rows
Sometimes it is easier to remove the columns you don’t want. You can do this by putting a -
in front of the column name.
#select everything but year and sex
penguins |> select(-year, -sex)
select: dropped 2 variables (sex, year)
# A tibble: 344 × 6
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Torgersen 39.1 18.7 181 3750
2 Adelie Torgersen 39.5 17.4 186 3800
3 Adelie Torgersen 40.3 18 195 3250
# ℹ 341 more rows
If there is a pattern to the column names that we want to select (or remove), there are some helper functions. For example, to select columns that start with “bill”, we can use starts_with()
.
#select bill_length_mm & bill_depth_mm"
penguins |> select(starts_with("bill"))
select: dropped 6 variables (species, island, flipper_length_mm, body_mass_g, sex, …)
# A tibble: 344 × 2
bill_length_mm bill_depth_mm
<dbl> <dbl>
1 39.1 18.7
2 39.5 17.4
3 40.3 18
# ℹ 341 more rows
Conversely, if we want to select all columns that end with “mm”, we can use ends_with()
. contains()
is more flexible and matches()
is the most powerful of the helper functions, using regular expressions to identify the columns (see the regular expression tutorial).
Sometimes, you might want to select all the columns of a certain type. For example, to select all the numeric columns we can use the is.numeric
function inside select()
with the helper where()
.
penguins |> select(where(is.numeric)) # No brackets on the function
select: dropped 3 variables (species, island, sex)
# A tibble: 344 × 5
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g year
<dbl> <dbl> <int> <int> <int>
1 39.1 18.7 181 3750 2007
2 39.5 17.4 186 3800 2007
3 40.3 18 195 3250 2007
# ℹ 341 more rows
Other is.*
functions exist, for example, is.character
for text.
You can also select columns by number (1 being the first column), but this is generally a bad idea because it makes the code difficult to understand and if a new column is added, or the column order is changed, the code will break.
Which of these strategies works best is context dependent.
From the penguins
data, select
- all columns except year
- all non-numeric columns
-
species
and columns ending in “mm”
Hint
# hint 1
|>
penguins select(-___)
# hint 2
?where
# hint 3
?ends_with
14.3 Renaming columns with rename
You can use rename()
to rename columns
penguins |> rename(Species = species)
rename: renamed one variable (Species)
# A tibble: 344 × 8
Species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Torgersen 39.1 18.7 181 3750
2 Adelie Torgersen 39.5 17.4 186 3800
3 Adelie Torgersen 40.3 18 195 3250
# ℹ 341 more rows
# ℹ 2 more variables: sex <fct>, year <int>
The syntax is new_name = current_name
.
You can also rename a column when selecting. This is convenient if you are using select()
anyway.
penguins |> select(Species = species)
select: renamed one variable (Species) and dropped 7 variables
# A tibble: 344 × 1
Species
<fct>
1 Adelie
2 Adelie
3 Adelie
# ℹ 341 more rows
14.4 Moving columns with relocate()
Sometimes it is useful to reorder the columns. This is never necessary for data analysis or plotting, but can be needed when making a table for presentation.
penguins |> relocate(island)
relocate: columns reordered (island, species, bill_length_mm, bill_depth_mm, flipper_length_mm, …)
# A tibble: 344 × 8
island species bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Torgersen Adelie 39.1 18.7 181 3750
2 Torgersen Adelie 39.5 17.4 186 3800
3 Torgersen Adelie 40.3 18 195 3250
# ℹ 341 more rows
# ℹ 2 more variables: sex <fct>, year <int>
The default is to move the named column first, the .before
and .after
arguments let you move the column into any position.
14.5 Filtering rows with filter()
Filtering rows that meet some condition is a very common task.
For example, to filter rows of penguins
that have a bill length greater than 40 mm, we can use
penguins |> filter(bill_length_mm > 40)
filter: removed 102 rows (30%), 242 rows remaining
# A tibble: 242 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Torgersen 40.3 18 195 3250
2 Adelie Torgersen 42 20.2 190 4250
3 Adelie Torgersen 41.1 17.6 182 3200
# ℹ 239 more rows
# ℹ 2 more variables: sex <fct>, year <int>
This will filter out each row where the condition is TRUE
.
The base R equivalent of this is
penguins[penguins$bill_length_mm > 40, ]
# A tibble: 244 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Torgersen 40.3 18 195 3250
2 <NA> <NA> NA NA NA NA
3 Adelie Torgersen 42 20.2 190 4250
# ℹ 241 more rows
# ℹ 2 more variables: sex <fct>, year <int>
Generally, filter
makes it easier to understand, especially with more complex criteria.
Other tests include
-
==
exactly equals. Often a bad idea to use with numeric data -
near
safe function for testing equality of numeric data as it has a tolerance for rounding errors.
sqrt(2) ^ 2 == 2 # should be true, but rounding errors
[1] FALSE
sqrt(2) ^ 2 - 2 # the difference
[1] 4.440892e-16
[1] TRUE
-
!=
not equal to -
<
less than -
<=
less than or equal to -
>
greater than -
>=
greater than or equal to -
is.na()
for filtering by missing values. -
between()
for filtering values with a range -
%in%
is used when you want to test if a value is in a vector
filter: removed 124 rows (36%), 220 rows remaining
# A tibble: 220 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Torgersen 39.1 18.7 181 3750
2 Adelie Torgersen 39.5 17.4 186 3800
3 Adelie Torgersen 40.3 18 195 3250
# ℹ 217 more rows
# ℹ 2 more variables: sex <fct>, year <int>
#equivalent to
penguins |>
filter(species == "Adelie" | species == "Chinstrap") # with many alternatives, this gets long
filter: removed 124 rows (36%), 220 rows remaining
# A tibble: 220 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Torgersen 39.1 18.7 181 3750
2 Adelie Torgersen 39.5 17.4 186 3800
3 Adelie Torgersen 40.3 18 195 3250
# ℹ 217 more rows
# ℹ 2 more variables: sex <fct>, year <int>
14.5.1 Filtering on multiple criteria
If we want to filter on multiple criteria, we need to decide whether we want all criteria to be TRUE
(AND in Boolean logic), or for one or more to be TRUE
(OR in Boolean logic).
If we want all criteria to be TRUE
, we can separate them by a comma (or by an &
if you want to be explicit).
penguins |>
filter(bill_length_mm > 40, bill_depth_mm > 18)
filter: removed 263 rows (76%), 81 rows remaining
# A tibble: 81 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Torgersen 42 20.2 190 4250
2 Adelie Torgersen 42.5 20.7 197 4500
3 Adelie Torgersen 46 21.5 194 4200
# ℹ 78 more rows
# ℹ 2 more variables: sex <fct>, year <int>
If we want rows where any of the criteria is TRUE
, we can separate them by a |
.
penguins |>
filter(bill_length_mm > 40 | bill_depth_mm > 18)
filter: removed 53 rows (15%), 291 rows remaining
# A tibble: 291 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Torgersen 39.1 18.7 181 3750
2 Adelie Torgersen 40.3 18 195 3250
3 Adelie Torgersen 36.7 19.3 193 3450
# ℹ 288 more rows
# ℹ 2 more variables: sex <fct>, year <int>
We can negate a criterion by putting !
in front of it. So to filter rows that do not have bills longer than 40 mm we can use
penguins |> filter(!bill_length_mm > 40)
filter: removed 244 rows (71%), 100 rows remaining
# A tibble: 100 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Torgersen 39.1 18.7 181 3750
2 Adelie Torgersen 39.5 17.4 186 3800
3 Adelie Torgersen 36.7 19.3 193 3450
# ℹ 97 more rows
# ℹ 2 more variables: sex <fct>, year <int>
Of course, in this example, we could also use <=
as the test.
14.5.2 Common errors
The commonest error is to use a single =
rather than ==
. Only the latter is a test of equality. If you do this, the error message is quite helpful.
penguins |> filter(species = "Chinstrap")
Error in `.fun()`:
! We detected a named input.
ℹ This usually means that you've used `=` instead of `==`.
ℹ Did you mean `species == "Chinstrap"`?
Another common error is to forget to quote any strings.
penguins |> filter(species == Chinstrap)
Error in `.fun()`:
ℹ In argument: `species == Chinstrap`.
Caused by error:
! object 'Chinstrap' not found
From the penguins
data, filter
- Gentoo penguins
- Gentoo or Adelie penguins
- penguins with a mass greater than or equal to 5000g
- penguins with a bill length between 45 and 50 mm
- Gentoo penguins not from from Dream Island
Hint
#hint 1
|> filter(___ = ___)
penguins
#hint 2
`%in%`
?
#hint 3
|> filter(___ >= ___)
penguins
#hint 4
?between
#hint 5
`!` ?
14.6 Slicing the data with slice()
Sometimes it is useful to extract rows by row number.
penguins |> slice(3:7)
slice: removed 339 rows (99%), 5 rows remaining
# A tibble: 5 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Torgersen 40.3 18 195 3250
2 Adelie Torgersen NA NA NA NA
3 Adelie Torgersen 36.7 19.3 193 3450
4 Adelie Torgersen 39.3 20.6 190 3650
5 Adelie Torgersen 38.9 17.8 181 3625
# ℹ 2 more variables: sex <fct>, year <int>
You can use negative numbers to remove rows. Be careful using slice()
as if the data change, different rows may be returned.
14.7 Distinct rows with distinct()
If there are duplicates in the data, we can remove these with distinct()
. distinct()
with no extra arguments will remove duplicate rows.
penguins |> distinct()
distinct: no rows removed
# A tibble: 344 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Torgersen 39.1 18.7 181 3750
2 Adelie Torgersen 39.5 17.4 186 3800
3 Adelie Torgersen 40.3 18 195 3250
# ℹ 341 more rows
# ℹ 2 more variables: sex <fct>, year <int>
If we are only interested in some of the columns, we can supply the names of these columns.
penguins |> distinct(island)
distinct: removed 341 rows (99%), 3 rows remaining
# A tibble: 3 × 1
island
<fct>
1 Torgersen
2 Biscoe
3 Dream
Other columns will be removed unless the argument .keep_all = TRUE
is used.
From the penguins
data, find distinct values of
- species
- species, island and sex
Hint
|> distinct(___, ___) penguins
14.8 Random rows
Sometimes you want to sample rows at random from a data.frame. This can be done with slice_sample()
. This can either sample a constant n rows or constant fraction of the rows depending on whether the n
or prop
argument is used.
penguins |> slice_sample(n = 10)
slice_sample: removed 334 rows (97%), 10 rows remaining
# A tibble: 10 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Dream 40.6 17.2 187 3475
2 Gentoo Biscoe 49 16.1 216 5550
3 Chinstrap Dream 53.5 19.9 205 4500
4 Gentoo Biscoe 45.5 14.5 212 4750
5 Adelie Torgers… 36.7 19.3 193 3450
6 Adelie Torgers… 42.5 20.7 197 4500
7 Adelie Dream 36.5 18 182 3150
8 Chinstrap Dream 46.8 16.5 189 3650
9 Chinstrap Dream 46.4 17.8 191 3700
10 Gentoo Biscoe 50.4 15.7 222 5750
# ℹ 2 more variables: sex <fct>, year <int>
14.9 Mutating and adding columns with mutate
The function mutate()
can add an new column or replace an existing one.
To make a new column called body_mass_kg
we can use
penguins |>
mutate(body_mass_kg = body_mass_g / 1000)
mutate: new variable 'body_mass_kg' (double) with 95 unique values and 1% NA
# A tibble: 344 × 9
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Torgersen 39.1 18.7 181 3750
2 Adelie Torgersen 39.5 17.4 186 3800
3 Adelie Torgersen 40.3 18 195 3250
# ℹ 341 more rows
# ℹ 3 more variables: sex <fct>, year <int>, body_mass_kg <dbl>
There are lots of functions that are useful to use with mutate. Any function that returns either a single value or as many values as are in the data can be used.
mutate()
is very useful when cleaning data.
- See text manipulation tutorial for cleaning text with the
stringr
package. - See date and time tutorial for cleaning dates and times with the
lubridate
package.
With the penguins
data,
- convert flipper length to cm
- add a column with the ratio of bill width to length
Hint
# hint 1
|>
penguins mutate(___ = ___)
14.10 Summarising data with summarise()
summarise()
lets us summarise data. We can use it if we want to calculate a summary statistic of the data. Remember to separate arguments with a comma.
penguins |> summarise(
flipper_len_mean = mean(flipper_length_mm, na.rm = TRUE),
flipper_len_sd = sd(flipper_length_mm, na.rm = TRUE)
)
summarise: now one row and 2 columns, ungrouped
# A tibble: 1 × 2
flipper_len_mean flipper_len_sd
<dbl> <dbl>
1 201. 14.1
Only the columns created in the summarise()
and any grouping columns (see below) will be kept.
With the penguins
data, find
- the maximum and minimum bill length
Hint
|>
penguins summarise(___ = ___(___))
14.10.1 Summarising multiple columns
Sometimes you want to summarise multiple columns at the same time. This can be done with the across()
helper function. across()
needs to be told which columns to process and what function or functions to use.
summarise: now one row and 2 columns, ungrouped
# A tibble: 1 × 2
bill_length_mm bill_depth_mm
<dbl> <dbl>
1 43.9 17.2
#using a list of functions
penguins |>
summarise(
across(.cols = starts_with("bill"),
.fns = list(sd = \(x)sd(x, na.rm = TRUE),
mean = \(x)mean(x, na.rm = TRUE))))
summarise: now one row and 4 columns, ungrouped
# A tibble: 1 × 4
bill_length_mm_sd bill_length_mm_mean bill_depth_mm_sd bill_depth_mm_mean
<dbl> <dbl> <dbl> <dbl>
1 5.46 43.9 1.97 17.2
You can also use across()
with mutate()
to mutate several columns at the same time.
14.11 Grouping data with group_by
group_by()
changes the way that many of the dplyr
functions work. Instead of working on the entire dataset, they now work on each group in the data
To find the mean flipper length for each species, we need to group_by()
species and then summarise()
.
penguins |>
group_by(species) |>
summarise(mean_flipper_length = mean(flipper_length_mm))
group_by: one grouping variable (species)
summarise: now 3 rows and 2 columns, ungrouped
# A tibble: 3 × 2
species mean_flipper_length
<fct> <dbl>
1 Adelie NA
2 Chinstrap 196.
3 Gentoo NA
Grouped data can be ungrouped with ungroup()
. This can help prevent surprises!
With the penguins
data, find
- the maximum and minimum bill depth of each species
Hint
|>
penguins group_by(___)
summarise(___ = ___(___))
14.12 Sorting with arrange()
To sort the data frame by one or more of the variables we can use arrange()
.
penguins |> arrange(bill_length_mm, bill_depth_mm)
# A tibble: 344 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Adelie Dream 32.1 15.5 188 3050
2 Adelie Dream 33.1 16.1 178 2900
3 Adelie Torgersen 33.5 19 190 3600
# ℹ 341 more rows
# ℹ 2 more variables: sex <fct>, year <int>
This will sort smallest first. To reverse the sort order, use desc()
# A tibble: 344 × 8
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
<fct> <fct> <dbl> <dbl> <int> <int>
1 Gentoo Biscoe 59.6 17 230 6050
2 Chinstrap Dream 58 17.8 181 3700
3 Gentoo Biscoe 55.9 17 228 5600
# ℹ 341 more rows
# ℹ 2 more variables: sex <fct>, year <int>
Sort the penguins
data by
- body mass
- island and body mass, with the largest birds first
Hint
# hint 1
|>
penguins arrange(___, ___)
# hint 2
?desc
14.13 Counting rows with count()
and n()
The function n
can count how many rows there are in the each group (or the entire data frame if it is not grouped). It can be used with either mutate()
or summarise()
.
penguins |>
group_by(species) |>
summarise(n = n())
group_by: one grouping variable (species)
summarise: now 3 rows and 2 columns, ungrouped
# A tibble: 3 × 2
species n
<fct> <int>
1 Adelie 152
2 Chinstrap 68
3 Gentoo 124
Or with count()
penguins |>
count(species)
count: now 3 rows and 2 columns, ungrouped
# A tibble: 3 × 2
species n
<fct> <int>
1 Adelie 152
2 Chinstrap 68
3 Gentoo 124
count()
is more concise, but group_by()
and summarise()
is useful when you need to calculate more variables (for example mean and standard deviation).
From the penguins
data, find
- how many penguins are there of each species
- how many penguins are there of each species in each island
Hint
|>
penguins count(___, ___)
# or
|>
penguins group_by(___) |>
summarise(___ = n())
14.14 Common problems
14.14.1 Non standard names
Ideally column names should follow the standard rules for naming objects in R - UPPER and lower case letters, numbers, “.” and “_” with the first character being a letter (or a dot if you want an invisible object). Sometimes when you import data, it has non-standard names with spaces or extra characters. If you need to refer to a column name that doesn’t follow the rules, you need to enclose it with back-ticks.
df <- tibble(`Region/Country` = "Norway", value = 42)
df
# A tibble: 1 × 2
`Region/Country` value
<chr> <dbl>
1 Norway 42
df |> rename(region_country = `Region/Country`)
rename: renamed one variable (region_country)
# A tibble: 1 × 2
region_country value
<chr> <dbl>
1 Norway 42
It is sometimes best to rename these columns to make them easier to refer to. janitor::clean_names()
is very efficient for making easy-to-use names.
Contributors
- Richard Telford