knitr::opts_chunk$set( collapse = TRUE, comment = "#>" )
The goal is to generate a data set which contains no real units, therefore safe for public release and retains the structure of the data.
In other words, one can say that synthetic data contains all the characteristics of original data minus the sensitive content.
Synthetic data is generally made to validate mathematical models. This data is used to compare the behaviour of the real data against the one generated by the model.
The principle is to observe real-world statistic distributions from the original data and reproduce fake data by drawing simple numbers.
Consider a data set with $p$ variables. In a nutshell, synthesis follows these steps:
Fitting statistical models to the original data and generating completely new records for public release. Joint distribution $f(x_1,x_2,x_3,…,x_p)$ is approximated by a set of conditional distributions $f(x_2|x_1)$.
The data this is based on is the NEWS Score devised by the Royal College of Physicians.
Synthetic data can be generated from new data, utilising the above methodology, on the real observed data:
library(readr) library(dplyr) df <- suppressWarnings(read_csv("https://raw.githubusercontent.com/StatsGary/SyntheticNEWSData/main/observed_news_data.csv") %>% dplyr::select(everything(), -X1)) glimpse(df)
This reads in the observed NEWS data from the GitHub repository. Now, we will utilise the
synthpop package to create a synthetically generated dataset.
As stated, now we will use the real observed data and generate a synthetic set, utilising the equations and process mapped out in the preceding sections:
library(synthpop) syn_df <- syn(df,seed=4321) #### synthetic data synthetic_news_data <- syn_df$syn glimpse(synthetic_news_data)
library(ggplot2) #Create temperature tibbles to compare observed vs synthetically generated labels obs <- tibble(label="observed_data", value = df$temp) synth <- tibble(label="synthetic_data",value = synthetic_news_data$temp) #Merge the frames together to get a comparison merged <- obs %>% bind_rows(synth) #Create the plot plot <- merged %>% ggplot(aes(value, fill = label)) + geom_histogram(alpha = 0.9, position = 'identity') + theme_minimal() + scale_fill_manual(values=c("#BCBDC1", "#2061AC")) + labs(title="Observed vs Synthetically NEWS values", subtitle="Based on NEWS Temperature score", x="NEWS Temperature Score", y="Score frequency") + theme(legend.position = "none") print(plot)
This dataset is available from the NHSRDatasets package and similar comparisons can be made with the above. These examples can be used for data wrangling and data visualisation.
For mode information about the synthpop package.
In many ways, synthetic data reflects George Box’s observation that “all models are wrong, but some are useful” while providing a “useful approximation [of] those found in the real world,”
The connection between the clinical outcomes of a patient visits and costs rarely exist in practice, so being able to assess these trade-offs in synthetic data allow for measurement and enhancement of the value of care – cost divided by outcomes.
Synthetic data is likely not a 100% accurate depiction of real-world outcomes, like cost and clinical quality, but rather a useful approximation of these variables. Moreover, synthetic data is constantly improving, and methods like validation and calibration will continue to make these data sources more realistic.
Besides synthetic data used to protect the privacy and confidentiality of set of data, it can be used for testing fraud detection systems by creating realistic behaviour profiles for users and attackers. In machine learning, it can also be used to train and test models. The synthetic data can aid in creating a baseline for future testing or studies such as clinical trial studies.
Any scripts or data that you put into this service are public.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.