8  Transforming Data

8.1 head()and tail()

After reading the file, you can use the head() and tail() functions to examine the object:

Example:

head(dataset)
tail()

8.2 Changing Variable Types

A common task after reading in data is to consider variable types. For example, if a variable in a dataset is a categorical variable with a known set of possible values but you see it is represented as a character, then you will want to make it a factor. Or a numeric variable is a character variable because one of the observations is typed out as a character instead of a numeric (e.g. “five” instead of 5).

Example:

students |>
  mutate(meal_plan = factor(meal_plan),
  age = parse_number(if_else(age == "five", "5", age))

After this, you will that the type of variable denoted underneath the variable name has changed from character () to factor ().

8.3 Changing Columns Names

Note that you can assign your own column names after reading a dataframe using the names() function, or when you read it in, using the col.names argument, which should be assigned a character vector the same length as the number of columns.

If there is no header and no col.names argument, then R uses the names V1, V2, etc.

8.4 Columns No-Syntatic Names

Sometimes, you might also notice that the columns names are surrounded by backticks. That’s because they contain spaces, breaking R’s usual rules for variable names; they’re non-syntactic names.

*Example:*

``` R
students |> 
  rename(
    student_id = `Student ID`,
    full_name = `Full Name`
  )
```

An alternative approach is to use janitor::clean_names() to use some heuristics to turn them all into snake case at once1.

students |> janitor::clean_names()

8.5 Changing Numbers

In most cases, numbers will be in one of R’s numeric types: integer or double. In some cases, however, you’ll encounter them as strings, possibly because you’ve created them by pivoting from column headers or because something has gone wrong in your data import process.

readr provides two useful functions for parsing strings into numbers: parse_double() and parse_number().

Example

Example:

x <- c("1.2", "5.6", "1e3")
parse_double(x)

[1]    1.2    5.6 1000.0

x <- c("$1,234", "USD 3,513", "59%")
parse_number(x)

[1] 1234 3513   59

8.6 Removing Duplicates

You can remove duplicates based on the x variable using:

x <- c(1, 2, NA, 4, NA, 5) 
data[!duplicated(data$x), ]

8.7 Reshaping Datasets

Statistical analysis sometimes requires wide data and sometimes long data. In such cases, we need to be able to fluently and fluidly reshape the data to meet the requirements of statistical analysis. The function reshape() reshapes a dataframe between ‘wide’ format (with repeated measurements in separate columns of the same row) and ‘long’ format (with the repeated measurements in separate rows).

Data reshaping is just a rearrangement of the form of the data—it does not change the content of the dataset.

Example:

students <- data.frame(sid=c(1,1,2,2), exmterm=c(1,2,1,2), math=c(50,65,75,69), 
literature=c(40,45,55,59), language=c(70,80,75,78))

# Reshaping dataset using reshape function to wide format
wide_students <- reshape(students, direction="wide", idvar="sid", timevar="exmterm")

# Now again reshape to long format
long_students <- reshape (wide_students, direction="long", idvar="id")

8.7.1 The reshape package

• Melting data (molten data):

  • Though melting can be applied to different R objects, the most common use is to melt a data frame.
  • To perform melting operations using the melt function, we need to know what the identification variables and measured variables in the original input dataset are.
  • One important thing to note is that, whenever we use the melt function, all the measured variables should be of the same type, that is, the measured variables should be either numeric, factor, character, or date.
  • To deal with the implicit missing value, it is good to use na.rm = TRUE with the melt function to remove the structural missing value (i.e., it will fill empty cells in the data table with NA).

  Example:

  ## the format of the resulting table is id/variable/value
  melt(students, id=c("sid","exmterm"), measured=c("math", "literature", "language"))

• Casting molten data:

  • Once we have molten data, we can rearrange it in any layout using the cast function from the reshape package.
  • There are two main arguments required to cast molten data: data and formula.
  • The basic casting formula is col_var_1+col_var_2 ~ row_var_1+ row_var_2, which describes the variables to appear in columns and rows.

  Example:

  # to return to the original data structure  
  cast(molten_students, sid+exmterm ~ variable)

• For faster and large data rearrangement, use the reshape2 package and the functions dcast (data frames) and acast (arrays):

  Example:

  acast(molten_students, sid+exmterm~variable)

8.8 Missing Values

R represents missing observations through the data value NA. It is easiest to think of NA values as place holders for data that should have been there, but, for some reason, are not.

8.8.1 Detect NA

We can detect whether variables are missing value using:

• is.na() is used to test objects if they are NA.
• is.nan() is used to test for NaN.

Example:

x <- c(1, 2, NaN, NA, 4) 
is.na(x) 

[1] FALSE FALSE  TRUE  TRUE FALSE

is.nan(x)

[1] FALSE FALSE  TRUE FALSE FALSE

To check if there is any NA in a dataframe:

Example:

any(is.na(x))

[1] TRUE

8.8.2 Remove NAs

Example:

a <- c(11, NA, 13)
mean(a, na.rm = TRUE)

[1] 12

8.8.3 Extract NA from a vector

Example:

x <- c(1, 2, NA, 3) 
z <- is.na(x) 
x[!z]

[1] 1 2 3

8.8.4 NA vs NULL

Note that NA and NULL are not equivalent:

• NA is a place holder for something that exists but is missing.

• NULL stands for something that never existed at all.

8.8.5 Removing NA values

You can remove rows with NA values in any variables:

na.omit(data)

## Another example
x <- c(1, 2, NA, 4, NA, 5) 
bad <- is.na(x) 
x[!bad]

[1] 1 2 4 5

How can you take the subset with no missing values in any of those objects?:

Example:

good <- complete.cases(airquality) 
head(airquality[good,])