In jason-p-pickering/data-pack-importer: Utilities for importing the ICPI Data pack into DATIM

Tier II Data Validation Guide

This manual should guide through the Tier II DataPack / DisaggTool process for COP18.

Status: DRAFT

Set up your environment

• Follow the instructions here: data-pack-importer-vagrant
• Update your support files and download them to the support_files folder

Download the support files

They are Excel spreadsheets links to Sharepoint files (they must not be attached). Download those into the repository from above. The country team needs to indicate the distribution method that should be used, either it's 2017 (FY17 Results) or 2018 (FY18 Targets).

DisaggTool validation

Run a first validation. Adjust the following code and paste it into RStudio code (http://localhost:8787) and hit ENTER.

# ADJUST THIS --->
filename="DisaggTool_filename.xlsx"
distribution_year=2017

# DO NOT CHANGE
library(devtools)
install_github("jason-p-pickering/data-pack-importer", ref="prod")
library(datapackimporter)

support_files="/vagrant/support_files/"
check_support_files(support_files)
disagg_tools="/vagrant/disagg_tools/"
disagg_tool=paste0(disagg_tools, filename)

#Parse the PNSU data
d<-DataPackR(disagg_tool, distribution_method = distribution_year, support_files_path = support_files)

It creates the following files:

• Site Level Review File - SiteLevelReview_*.xlsx -> send back to the country team via Datim Support. There may be errors, include that log into your response so the country team can correct their Disagg Tools and re-submit it. Re-run the DisaggTool validation.
• PSNU data: *_import_*.csv -> depending of issues, continue to run Import File Validation or act as below.

TBD

Warnings / error handling:

• Hard stop: Tier II does not proceed with anything else until this is fixed by the country team.
• Continue: Tier II proceeds with validation #2 of the PSNU CSV (*_import_*.csv)

Possible issues and their consequences:

• Negative values: hard stop
• Schema not valid: hard stop
• DisAggTool corrupted: hard stop
• Missing PSNUs: continue

Site-Level Review Tool Validation

You receive back the Site Level Review Tool, reviewed by the country. This needs another validation. Adjust the filename and re-run.

# ADJUST THIS --->
filename="SiteLevelReview.xlsx"
distribution_year=2017

(re-use the same script from above)

Import File Validation

In above step a PSNU CSV file was generated. This step checks if it is a valid DATIM import file. See these instructions for doing so: https://github.com/jason-p-pickering/datim-validation/blob/master/vignettes/validating_data.Rmd

The gist of it:

• Clear the console in RStudio
• Create a ~/.secrets/datim.json file, or use the /vagrant/datim.json

{
  "dhis": {
    "baseurl": "https://dev-de.datim.org/",
    "username": "admin",
    "password": "district"
  }
}

Make it only readable by your user:

chmod 0600 ~/.secrets/datim.json

or

chmod 0600 /path/to/vagrant/datim.json

• Install datim-validation and load secrets:

# ADJUST THIS -->
filename="/vagrant/disagg_tools/path_to_psnu.csv"
secrets_file="/vagrant/datim.json"

# DON'T CHANGE
devtools::install_github('jason-p-pickering/datim-validation')
require(datimvalidation)
secrets<-secrets_file
loadSecrets(secrets)
d<-d2Parser(file=filename,
            type = "csv",
            dataElementIdScheme = "id",
            orgUnitIdScheme = "id",
            idScheme = "id",
            invalidData = TRUE)

then run these one by one:

getInvalidMechanisms(d)

For site level tools, invoke this command:

getInvalidDataElements(d,c("eyI0UOWJnDk","l796jk9SW7q"))

For PSNU level tools, invoke this command

getInvalidDataElements(d,c("msJImKkTJkV","pTuDWXzkAkJ"))

For both of these commands, you should not get any rows returned from the function.

Then run remaining code (one by one or together should work):

checkValueTypeCompliance(d)
vr_violations<-validateData(data = d,
                            return_violations_only = TRUE,
                            parallel = FALSE)

vr_rules<-getValidationRules()
cop_18_des<-getValidDataElements(datasets=c("l796jk9SW7q","eyI0UOWJnDk","msJImKkTJkV"))
match<-paste(unique(cop_18_des$dataelementuid),sep="",collapse="|")
vr_filter<-vr_rules[grepl(match,vr_rules$leftSide.expression) & grepl(match,vr_rules$rightSide.expression),"id"]
vr_violations<-vr_violations[ vr_violations$id %in% vr_filter,]
diff<-gsub(" <= ","/",vr_violations$formula)
vr_violations$diff<-sapply(diff,function(x) { round( ( eval(parse(text=x)) -1 ) * 100 , 2) })
write.csv(vr_violations,file="validation_violations.csv")

You can download the resulting file by checking the checkbox next to the appropriate file in the file panel, then by clicking More and selecting Export.

If the validation passes, reroute file to Tier III.

If it does not pass, respond with the logs to the country team.

jason-p-pickering/data-pack-importer documentation built on Oct. 27, 2019, 9:42 a.m.