calculateMABias: calculateMABias
In reproducer: Reproduce Statistical Analyses and Meta-Analyses
calculateMABias R Documentation
calculateMABias
Description
The function simulates multiple five group families of either two-group or four-group experiments and estimates the power, individual estimate error, and the small sample bias obtained after synthesizing the analysis results obtained from the experiments in each family. The power is estimated as the percentage of families for which the overall mean of the five experiments was significantly different from zero. The experiment data may be one of four different type: Normal, Log-normal, Gamma or Laplace. The simulations can be repeated for different mean differences between the control mean and treatment mean depending on the parameter diff. The output is a table of values identifying the observed values of three effect sizes: Cliff's d, PHat and StdMD, estimate error and their related small sample bias and power for each set of simulated families. The synthesis method for all the effect sizes is based on calculating the overall mean and variance for experiments in each family and then using those values to calculate the overall effect size and its variance. This function supports the production of the values reported in data tables in the paper "Recommendations for Analyzing Small Sample Size Software Engineering Experiments" and its Supplementary Material.
Usage
calculateMABias(
mean = 0,
sd = 1,
N,
reps,
diff = c(0.2, 0.5, 0.8),
Experiments = 5,
Expected.StdMD = c(0.2, 0.5, 0.8),
Expected.PHat = c(0.556, 0.638, 0.714),
type = "n",
FourG = FALSE,
seed = 223,
StdAdj = 0,
Blockmean = 0,
StdExp = 0,
MAMethod = "PM",
alpha = 0.05
)
Arguments
mean
This is the mean value of the control and treatment group(s) used in the simulations of each experiment of each family for simulations of a specified sample size (default 0).
sd
This is the standard deviation value of the control group(s) and treatment group(s) used in the simulations of each experiment of each family for simulations of a specified sample size (default 1).
N
This specifies the sample size per group that will be used in each set of simulations.
reps
The number of families simulated for each sample size.
diff
This specifies the difference between the control and treatment that will be used in each set of simulations. It must always have three values representing small, medium and large values (default c(0.2, 0.5, 0.8)).
Experiments
The number of experiments in each family (default 5).
Expected.StdMD
This defines the expected value of the overall average StdMD for each mean difference (default c(0.2,0.5,0.8)).
Expected.PHat
This defines the expected population value of the overall average Phat for each mean difference (default c(0.556,0.638,0.714)).
type
This specifies the distribution of the data samples that will be simulated. Options ae "n" for Normal, "l", for Log-normal,'g" for Gamma, "lap" for LaPlace (default "n").
FourG
If FourG is FALSE (default) the individual experiments in each family will be two-group experiments, otherwise the individual experiments will be four-group families.
seed
A seed for the simulations (default 123).
StdAdj
Used to introduce variance heterogeneity for Laplace and Normal samples (default 0).
Blockmean
Used to set a fixed block effect for four-group experiments (default 0).
StdExp
Used to introduce heterogeneity among families of experiments (default 0).
MAMethod
Not used (default "PM").
alpha
The significance level for statistical tests (default 0.05).
Value
Design. Specifies the type of experiment 2G or 4G, the sample distribution (n,l,g,lap), and whether variance heterogeneity was added (het)
BEIncluded. Specifies whether or not a block effect was introduced. Always set to "No" for two-group experiments.
GrpSize. Specifies the size of each group in the individual experiments.
Diff. The size of the difference between the control and treatment converted to an ordinal scale (Small, Medium, Large)
NPBias The relative difference between the average of the observed values of either Cliff's d or centralised PHat and the population value
StdMDBias. The relative difference between the average of the observed values of StdMDBias and the theoretical value
NPMdMRE The median of the absolute relative difference between the observed values of either Cliff's d or centralised PHat and the theoretical value for each experiment.
StdMDMdMRE The median of the absolute relative difference between the observed values of StdMD and the population value for each experiment.
ObsPHat. The average of the average Phat value found for each family in the set of simulations.
ObsCliffd. The average of the average Cliffd value found for each family in the set of simulations.
ObsStdES. The average of StdMD calculated for each family in the set of simulations.
PHatPower. The percentage of the simulations, for a specific mean difference, for which the overall Phat estimate was significantly different from zero at the nominated alpha level using one-sided tests.
CliffdPower. The percentage of the simulations, for a specific mean difference, for which the overall Cliff's d estimate was significantly different from zero at the nominated alpha level using one-sided tests.
StdMDPower. The percentage of the simulations, for a specific mean difference, for which the overall StdMD estimate was significantly different from zero at the nominated alpha level using one-sided tests.
Author(s)
Barbara Kitchenham and Lech Madeyski
Examples
# as.data.frame(calculateMABias(mean=0,sd=1,N=10,diff=c(0.2,0.5,0.8), Experiments=5,reps=10,
# Expected.StdMD=c(0.2,0.5,0.8), Expected.PHat=c(0.556,0.638,0.714), type="n",FourG=FALSE,
# seed= 123, StdAdj = 0, Blockmean=0, StdExp=0))
# Design Blockmean GrpSize Diff NPBias StdMDBias NPMdMRE StdMDMdMRE ObsPHat ObsCliffd..
#1 2G_n No 10 Small 0.09285714 0.02606704 0.8928571 1.0741432 0.5612 0.1224..
#2 2G_n No 10 Medium 0.03768116 0.01740262 0.2391304 0.4171896 0.6432 0.2864..
#3 2G_n No 10 Large 0.03738318 0.01523651 0.2009346 0.2490287 0.7220 0.4440..
# PHatPower CliffdPower StdESPower
#1 0.2 0.2 0.3
#2 0.7 0.7 0.7
#3 1.0 1.0 1.0
as.data.frame(calculateMABias(mean=0,sd=1,N=10,diff=c(0.2,0.5,0.8), Experiments=5,reps=4,
Expected.StdMD=c(0.2,0.5,0.8), Expected.PHat=c(0.556,0.638,0.714), type="n",FourG=TRUE,
seed= 123,StdAdj = 0.5,Blockmean=0.5,StdExp=0))
#Results for reps=10
# Design Blockmean GrpSize Diff NPBias StdMDBias NPMdMRE StdMDMdMRE ObsPHat ObsClif..
#1 4G_n_het Yes 10 Small -0.1321429 -0.1372277 0.6696429 0.4698935 0.5486 0.0972..
#2 4G_n_het Yes 10 Medium -0.1869565 -0.1882479 0.2318841 0.1472392 0.6122 0.2244..
#3 4G_n_het Yes 10 Large -0.1864486 -0.2010029 0.1612150 0.1531253 0.6741 0.3482..
# PHatPower CliffdPower StdESPower
#1 0.4 0.4 0.4
#2 0.9 0.9 0.8
#3 1.0 1.0 1.0
reproducer documentation built on Oct. 18, 2023, 5:10 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
| calculateMABias | R Documentation |
calculateMABias
Description
The function simulates multiple five group families of either two-group or four-group experiments and estimates the power, individual estimate error, and the small sample bias obtained after synthesizing the analysis results obtained from the experiments in each family. The power is estimated as the percentage of families for which the overall mean of the five experiments was significantly different from zero. The experiment data may be one of four different type: Normal, Log-normal, Gamma or Laplace. The simulations can be repeated for different mean differences between the control mean and treatment mean depending on the parameter diff. The output is a table of values identifying the observed values of three effect sizes: Cliff's d, PHat and StdMD, estimate error and their related small sample bias and power for each set of simulated families. The synthesis method for all the effect sizes is based on calculating the overall mean and variance for experiments in each family and then using those values to calculate the overall effect size and its variance. This function supports the production of the values reported in data tables in the paper "Recommendations for Analyzing Small Sample Size Software Engineering Experiments" and its Supplementary Material.
Usage
calculateMABias(
mean = 0,
sd = 1,
N,
reps,
diff = c(0.2, 0.5, 0.8),
Experiments = 5,
Expected.StdMD = c(0.2, 0.5, 0.8),
Expected.PHat = c(0.556, 0.638, 0.714),
type = "n",
FourG = FALSE,
seed = 223,
StdAdj = 0,
Blockmean = 0,
StdExp = 0,
MAMethod = "PM",
alpha = 0.05
)
Arguments
mean |
This is the mean value of the control and treatment group(s) used in the simulations of each experiment of each family for simulations of a specified sample size (default 0). |
sd |
This is the standard deviation value of the control group(s) and treatment group(s) used in the simulations of each experiment of each family for simulations of a specified sample size (default 1). |
N |
This specifies the sample size per group that will be used in each set of simulations. |
reps |
The number of families simulated for each sample size. |
diff |
This specifies the difference between the control and treatment that will be used in each set of simulations. It must always have three values representing small, medium and large values (default c(0.2, 0.5, 0.8)). |
Experiments |
The number of experiments in each family (default 5). |
Expected.StdMD |
This defines the expected value of the overall average StdMD for each mean difference (default c(0.2,0.5,0.8)). |
Expected.PHat |
This defines the expected population value of the overall average Phat for each mean difference (default c(0.556,0.638,0.714)). |
type |
This specifies the distribution of the data samples that will be simulated. Options ae "n" for Normal, "l", for Log-normal,'g" for Gamma, "lap" for LaPlace (default "n"). |
FourG |
If FourG is FALSE (default) the individual experiments in each family will be two-group experiments, otherwise the individual experiments will be four-group families. |
seed |
A seed for the simulations (default 123). |
StdAdj |
Used to introduce variance heterogeneity for Laplace and Normal samples (default 0). |
Blockmean |
Used to set a fixed block effect for four-group experiments (default 0). |
StdExp |
Used to introduce heterogeneity among families of experiments (default 0). |
MAMethod |
Not used (default "PM"). |
alpha |
The significance level for statistical tests (default 0.05). |
Value
Design. Specifies the type of experiment 2G or 4G, the sample distribution (n,l,g,lap), and whether variance heterogeneity was added (het)
BEIncluded. Specifies whether or not a block effect was introduced. Always set to "No" for two-group experiments.
GrpSize. Specifies the size of each group in the individual experiments.
Diff. The size of the difference between the control and treatment converted to an ordinal scale (Small, Medium, Large)
NPBias The relative difference between the average of the observed values of either Cliff's d or centralised PHat and the population value
StdMDBias. The relative difference between the average of the observed values of StdMDBias and the theoretical value
NPMdMRE The median of the absolute relative difference between the observed values of either Cliff's d or centralised PHat and the theoretical value for each experiment.
StdMDMdMRE The median of the absolute relative difference between the observed values of StdMD and the population value for each experiment.
ObsPHat. The average of the average Phat value found for each family in the set of simulations.
ObsCliffd. The average of the average Cliffd value found for each family in the set of simulations.
ObsStdES. The average of StdMD calculated for each family in the set of simulations.
PHatPower. The percentage of the simulations, for a specific mean difference, for which the overall Phat estimate was significantly different from zero at the nominated alpha level using one-sided tests.
CliffdPower. The percentage of the simulations, for a specific mean difference, for which the overall Cliff's d estimate was significantly different from zero at the nominated alpha level using one-sided tests.
StdMDPower. The percentage of the simulations, for a specific mean difference, for which the overall StdMD estimate was significantly different from zero at the nominated alpha level using one-sided tests.
Author(s)
Barbara Kitchenham and Lech Madeyski
Examples
# as.data.frame(calculateMABias(mean=0,sd=1,N=10,diff=c(0.2,0.5,0.8), Experiments=5,reps=10,
# Expected.StdMD=c(0.2,0.5,0.8), Expected.PHat=c(0.556,0.638,0.714), type="n",FourG=FALSE,
# seed= 123, StdAdj = 0, Blockmean=0, StdExp=0))
# Design Blockmean GrpSize Diff NPBias StdMDBias NPMdMRE StdMDMdMRE ObsPHat ObsCliffd..
#1 2G_n No 10 Small 0.09285714 0.02606704 0.8928571 1.0741432 0.5612 0.1224..
#2 2G_n No 10 Medium 0.03768116 0.01740262 0.2391304 0.4171896 0.6432 0.2864..
#3 2G_n No 10 Large 0.03738318 0.01523651 0.2009346 0.2490287 0.7220 0.4440..
# PHatPower CliffdPower StdESPower
#1 0.2 0.2 0.3
#2 0.7 0.7 0.7
#3 1.0 1.0 1.0
as.data.frame(calculateMABias(mean=0,sd=1,N=10,diff=c(0.2,0.5,0.8), Experiments=5,reps=4,
Expected.StdMD=c(0.2,0.5,0.8), Expected.PHat=c(0.556,0.638,0.714), type="n",FourG=TRUE,
seed= 123,StdAdj = 0.5,Blockmean=0.5,StdExp=0))
#Results for reps=10
# Design Blockmean GrpSize Diff NPBias StdMDBias NPMdMRE StdMDMdMRE ObsPHat ObsClif..
#1 4G_n_het Yes 10 Small -0.1321429 -0.1372277 0.6696429 0.4698935 0.5486 0.0972..
#2 4G_n_het Yes 10 Medium -0.1869565 -0.1882479 0.2318841 0.1472392 0.6122 0.2244..
#3 4G_n_het Yes 10 Large -0.1864486 -0.2010029 0.1612150 0.1531253 0.6741 0.3482..
# PHatPower CliffdPower StdESPower
#1 0.4 0.4 0.4
#2 0.9 0.9 0.8
#3 1.0 1.0 1.0
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.