TDM 10100: Project 8 — Fall 2026

  • Data Grouping, Subsets, Looping through Data using Apply (sapply/tapply/lapply) Functions, and Splitting Data

Motivation: R provides powerful methods for looping through data. These methods are easy to use, once you get used to the syntax and the methodology.

Context: R allows us to split data into groups, apply functions to each group, and to organize an analysis of data…​ often all in one line!

Scope: We will demonstrate the usage of looping through data in groups in R, starting with the tapply function as a key example.

Learning Objectives:

  • Practice organizing and analyzing data in R.

  • Understand how R groups data and applies functions.

Please use the template found here, and the important information about project submissions here, when you are ready to submit your work.

Datasets

  • /anvil/projects/tdm/data/election/itcont1980.txt

  • /anvil/projects/tdm/data/icecream/combined/products.csv

  • /anvil/projects/tdm/data/icecream/combined/reviews.csv

  • /anvil/projects/tdm/data/flights/subset/1990.csv

  • /anvil/projects/tdm/data/taxi/yellow/yellow_tripdata_2017-06.csv

  • /anvil/projects/tdm/data/bay_area_bike_share/kaggle/trip.csv

  • /anvil/projects/tdm/data/beer/reviews_sample.csv

  • /anvil/projects/tdm/data/grouping/vgchartz-2024.csv

  • /anvil/projects/tdm/data/grouping/googleplaystore.csv

In this project, please only use 2 cores in your Jupyter Lab session:

notebook.anvilcloud.rcac.purdue.edu/

(do not use 4 cores or 16 cores for this project)

Dr. Ward thinks about the tapply function this way:

  • The data that you want to work with goes first in the tapply

  • The way that the data are grouped goes second in the tapply

  • The function that you want to apply to the data goes third in the tapply

For instance, tapply(mymiles, mypeople, sum) will add up the miles (stored in mymiles) but will do it one person at a time (according to mypeople).

As a way to know if you have the ordering of things correct, you can usually ask yourself if you could apply the function to the data itself, without grouping it, for instance, in this example, sum(mymiles) would be the total miles, but it would not make sense to write sum(mypeople) because you do not add up people.

There are numerous examples in these pages. It should help to take a look at the examples, before you try the questions.

You can read an overview about the tapply function:

the-examples-book.com/projects/fall2026/10100/tapplyoverview

as well as an election example:

the-examples-book.com/projects/fall2026/10100/electionexample

an ice cream example:

the-examples-book.com/projects/fall2026/10100/icecreamexample

an airport example:

the-examples-book.com/projects/fall2026/10100/airportexample

and some taxi examples:

the-examples-book.com/projects/fall2026/10100/taxiexamples

and some beer review examples:

the-examples-book.com/projects/fall2026/10100/beerreviews

and the video game examples:

the-examples-book.com/projects/fall2026/10100/videogameexamples

Questions

Question 1

The ice cream example given above was about the products.csv file:

/anvil/projects/tdm/data/icecream/combined/products.csv

Now let us turn our attention instead to the reviews.csv file:

/anvil/projects/tdm/data/icecream/combined/reviews.csv

For each of the 4 brand types of ice cream, find the average number of stars for that brand in two ways:

First find the average number of stars for each brand by using 4 separate lines of R, as follows:

mean(myDF$stars[myDF$brand == 'bj'])
mean(myDF$stars[myDF$brand == 'breyers'])
mean(myDF$stars[myDF$brand == 'hd'])
mean(myDF$stars[myDF$brand == 'talenti'])

Then find the average number of stars for each brand by using just 1 tapply command. Your result from the tapply function should show the same 4 average number of stars but will only need one line of R.

Deliverables

  • First find the average number of stars for each brand by using 4 separate lines of R.

  • Then find the average number of stars for each brand by using just 1 tapply command.

Question 2

First consider the airport example given above.

Next, find the average departure delays for each carrier. Hint: The carriers are stored in UniqueCarrier and you will also need to remove the NA values, as discussed in the carrier example. (There used to be more carriers in the airline industry, but you will recognize some of them, for instance, 'UA' is United, and 'AA' is American, etc.)

Finally, find the total number of miles flown on each carrier. Hint: The miles per flight are given in the Distance column.

Deliverables

  • Find the average departure delays for each carrier.

  • Find the total number of miles flown on each carrier.

Question 3

Re-consider the election example above: If you sum the transactions amounts (as in the video) and sort the results, you see that Texas, California, and New York are the top three states, in terms of the amount of money donated altogether, and then something strange happens for the 4th ranked state, namely, R gives a number for the name of the state. The reason is that there is an empty string for many donations. We can see that as follows:

names(sort(tapply(myDF$TRANSACTION_AMT, myDF$STATE, sum)))

Notice that the 4th ranked state is listed as '' (as an empty string).

It is up to the data provider how to deal with empty values. We might choose to change those empty values to (say) a value such as 'unknown state' or to NA. Here we will change the value to unknown state:

myDF$STATE[myDF$STATE == ''] <- 'unknown state'

After making this change, show the top 6 states, according to the total amount of money donated, including the 4th most popular state listed now as 'unknown state'. (So 5 states will be listed by name, and one entry among the top 6 will be listed as 'unknown state'.)

Now make a similar change for the city:

myDF$CITY[myDF$CITY == ''] <- 'unknown city'

Now use tapply to find the 6 most popular cities, but do not just use the city, i.e., do not do this:

tapply(myDF$TRANSACTION_AMT, myDF$CITY, sum)

instead, we want you to paste the city and state together, as the second entry in the tapply:

tapply(myDF$TRANSACTION_AMT, paste(myDF$CITY, myDF$STATE), sum)

Finally, show the top 6 donors (from the NAME column) according to the amount that they donated in the 1980 election.

Deliverables

  • Show the top 6 states, according to the total amount of money donated, including the 4th most popular state listed now as 'unknown state'.

  • Show the top 6 city-and-state pairs, according to the total amount of money donated, including the number 1 most popular city-and-state pair, listed now as 'unknown city unknown state'.

  • Show the top 6 donors (from the NAME column) according to the amount that they donated in the 1980 election

Question 4

In the Kaggle data about bicycle rides:

/anvil/projects/tdm/data/bay_area_bike_share/kaggle/trip.csv

there are two possible values of subscription_type. Find the sum of the duration of all rides, for each of these two values of subscription_type.

You can use the starting and ending locations of the rides, connected with the word 'to', like this:

paste(myDF$start_station_name, myDF$end_station_name, sep=" to ")

(do not run this paste all by itself, or otherwise Jupyter Lab will try to print all of those locations on your screen!)

You can use this paste as the second value in your tapply , to group the rides according to their starting-and-ending location pairs.

Now find the 6 starting-and-ending location pairs that have the greatest sum of the durations of rides on those routes.

As we can see, some of the rides have the same starting-and-ending locations (these could be called loops), and some do not. Let’s split the data into two types:

selfloopsDF <- subset(myDF, myDF$start_station_name == myDF$end_station_name)
notloopsDF <- subset(myDF, myDF$start_station_name != myDF$end_station_name)

For the rides that are self-loops, find the 6 starting-and-ending locations pairs (they will be the same, since they are self-loops) that have the greatest sum of the durations of rides.

For the rides that are not self-loops, find the 6 starting-and-ending locations pairs that have the greatest sum of the durations of rides.

Deliverables

  • Find the sum of the duration of all rides, for each of these two values of subscription_type.

  • Find the 6 starting-and-ending location pairs that have the greatest sum of the durations of rides on those routes.

  • For the rides that are self-loops, find the 6 starting-and-ending locations pairs (they will be the same, since they are self-loops) that have the greatest sum of the durations of rides.

  • For the rides that are not self-loops, find the 6 starting-and-ending locations pairs that have the greatest sum of the durations of rides.

Question 5

Consider the video game data:

/anvil/projects/tdm/data/grouping/vgchartz-2024.csv

In all three of the subquestions within Question 5, because some of the total sales values are NA, make sure to use the na.rm=T option in the tapply.

Find the top 6 publishers accordings to the total sales.

Now find the sum of the total sales in each year (grouping by the year(myDF$release_date)).

Finally, find the sum of the total sales in each month-and-year pair (grouping y list(year(myDF$release_date), month(myDF$release_date))).

Deliverables

  • Find the top 6 publishers accordings to the total sales.

  • Find the sum of the total sales in each year.

  • Find the sum of the total sales in each month-and-year pair.

Submitting your Work

Once you have completed the questions, save your Jupyter notebook. You can then download the notebook and submit it to Gradescope.

Items to submit

  • firstname_lastname_project8.ipynb

You must double check your .ipynb after submitting it in gradescope. A very common mistake is to assume that your .ipynb file has been rendered properly and contains your code, markdown, and code output even though it may not. Please take the time to double check your work. See here for instructions on how to double check this.

You will not receive full credit if your .ipynb file does not contain all of the information you expect it to, or if it does not render properly in Gradescope. Please ask a TA if you need help with this.