In mkearney/rtweet.download: Automate Large Downloads with 'rtweet'

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
library(rtweet.download)
options(width = 90)

rtweet.download

Automating the collection of Twitter data (via {rtweet})

Installation

You can install the released version of {rtweet.download} from CRAN with:

install.packages("rtweet.download")

And the development version from GitHub with:

# install.packages("remotes")
remotes::install_github("mkearney/rtweet.download")

Friends IDs

Twitter's "friends/ids" API endpoint is rate limited to 15 requests (or the friend IDs of 15* accounts) per 15 minutes. So while a single call using rtweet::get_friends() can retrieve the friend IDs of up to 15 users, a single call using rtweet.download::get_friends_download() can retrieve the friend IDs of hundreds or even thousands of users!

| | | |----------------------------------------------|--------------------------| | API Feature | Value | |   Endpoint | "friends/ids" | |   Rate limit (per 15 min.) | 15 | |   Friends per request | 5000* | | R Package | Function | |   {rtweet} | get_friends() | |   {rtweet.download} | get_friends_download() |

The example below uses get_friends_download() to automate the collection of friend (accounts followed by) IDs of users on \@Teradata's list of data science influencers.

## get members on data science influencers influence
data_sci_influencers <- rtweet::lists_members(
  owner_user = "Teradata", slug = "data-science-influencers"
)

## download friend IDs for each user
fds <- get_friends_download(data_sci_influencers$screen_name)

## preview data
head(fds)

Users data

Twitter's "users/lookup" API endpoint is rate limited to 900 requests (or 90,000 users) per 15 minutes. So while a single call using rtweet::lookup_users() can retrieve data on up to 90,000 users, a single call using rtweet.download::lookup_users_download() can collect data on hundreds of thousands or even millions of users!

| | | |----------------------------------------------|---------------------------| | API Feature | Value | |   Endpoint | "users/lookup" | |   Rate limit (per 15 min.) | 900 | |   Users per request | 100 | | R Package | Function | |   {rtweet} | lookup_users() | |   {rtweet.download} | lookup_users_download() |

The example below uses lookup_users_download() to automate data collection for the previously collected accounts followed by data science influencers.

## download users data
fds_data <- lookup_users_download(fds$user_id)

## preview data
head(fds)

Notes

* The "friends/ids" endpoint returns the up to 5,000 friend IDs of a single user, so 15 requests can only return all the friend IDs of 15 users if all 15 of those users follow 5,000 or fewer accounts. To retrieve all the friend IDs for users following more than 5,000 accounts, multiple requests (friends_count / 5,000) are required.

mkearney/rtweet.download documentation built on Dec. 14, 2019, 7:27 p.m.