copy_to: Copy a (very small) local data frame to Impala

In implyr: R Interface for Apache Impala

Copy a (very small) local data frame to Impala

Description

copy_to inserts the contents of a local data frame into a new Impala table. copy_to is intended to be used only with very small data frames. It uses the SQL INSERT ... VALUES() technique, which is not suitable for loading large amounts of data. By default, this function will throw an error if you attempt to copy a data frame with more than 1000 row/column positions. You can increase this limit at your own risk by setting the option implyr.copy_to_size_limit to a higher number.

This package does not provide tools for loading larger amounts of local data into Impala tables. This is because Impala can query data stored in several different filesystems and storage systems (HDFS, Apache Kudu, Apache HBase, Amazon S3, Microsoft ADLS, and Dell EMC Isilon) and Impala does not include built-in capability for loading local data into these systems.

Usage

## S3 method for class 'src_impala'
copy_to(
  dest,
  df,
  name = deparse(substitute(df)),
  overwrite = FALSE,
  types = NULL,
  temporary = TRUE,
  unique_indexes = NULL,
  indexes = NULL,
  analyze = FALSE,
  external = FALSE,
  force = FALSE,
  field_terminator = NULL,
  line_terminator = NULL,
  file_format = NULL,
  ...
)

Arguments

dest	an object with class with class src_impala
df	a (very small) local data frame
name	name for the new Impala table
overwrite	whether to overwrite existing table data (currently ignored)
types	a character vector giving variable types to use for the columns
temporary	must be set to FALSE
unique_indexes	not used
indexes	not used
analyze	whether to run COMPUTE STATS after adding data to the new table
external	whether the new table will be externally managed
force	whether to silently continue if the table already exists
field_terminator	the deliminter to use between fields in text file data. Defaults to the ASCII control-A (hex 01) character
line_terminator	the line terminator. Defaults to "\n"
file_format	the storage format to use. Options are "TEXTFILE" (default) and "PARQUET"
...	other arguments passed on to methods

Value

An object with class tbl_impala, tbl_sql, tbl_lazy, tbl

Note

Impala does not support temporary tables. When using copy_to() to insert local data into an Impala table, you must set temporary = FALSE.

Examples

library(nycflights13)
dim(airlines) # airlines data frame is very small
# [1] 16  2

## Not run: 
copy_to(impala, airlines, temporary = FALSE)
## End(Not run)

implyr documentation built on May 29, 2024, 8 a.m.