NEWS.md
In pepfar-datim/datapackr: A Package that Packs and Unpacks all Data Packs and Target Setting Tools

datapackr 8.2.0

Updates to enable the use of a TSNU

Updated Analytics checks to be reflective of COP25
Updated DATIM and PAW export to work for COP25
Updated COC map to be reflective for COP25
Updated Schema to be reflective for COP25
Updated valid org units to be reflective for COP25
Updated Template to be reflective for COP25
Updated validation rules pulling from DATIM
Updated test as needed

datapackr 8.1.0

Updated how messages are handled via labeling by tool and their associated tests

Removed 2025 from unPackTool as there will be no year 2 data

Updated schema and the template to capture COP25 changes
Updated the Home tab to allow for COP25

datapackr 8.0.1

Addressed issues related to mechanisms and the Regionalization of countries in DATIM

datapackr 8.0.0

Ability to process COP25 datapacks full version
Addition of the comparison plus file

datapackr 7.6.2

Updated checkMechanisms to handle shifts in rationalization as it relates to 23 OPU's

datapackr 7.6.1

Updated dataset_levels to reflect org hierarchy change

datapackr 7.6.0

Sunset COP21 and COP22
Added original versions of the COP25 templates

reenabled droping invalid disaggs from the year 2 sheet
Updated Valid Org Units for 2024 to account for regionalization.
Removed COP Season parameter

Updated strip_wb_nas

datapackr 7.5.0

Add utility functions for interacting with PDAP Jobs

Updated dependencies
Updated lintr to be more restrictive and updated code to fit new restrictions
Removed TODO's from the entire code base

datapackr 7.4.4

Updated exported functions and back end param handling.
Updated documentation

datapackr 7.4.3

Updated checkanalytics and their associated tests. Relating to PMTCT & TB_STAT columns being based on COP Year.

Added tests related to writePSNUxIM
Updated getCurrentCOPYear to 2024, and started development on sunsetting this function.
Removed non-ASCII character from file unPackingChecks
Fix issue with package imports flagged by devtools::check

datapackr 7.4.1

Updated valid Org units with changes from 2024

datapackr 7.4.0

Created createPAWExport to prepare data to be exported to PAW
Modified createDATIMExport to prepare data to be exported only to DATIM
Modified packForDATIM to feed both of the aforementioned export functions

Added tests around both createPAWExport,createDATIMExport, and packForDATIM
Please note a few hot fixes related to visual formatting went into main between 7.3.3 and 7.4.0.

datapackr 7.3.3

Updated packForDATIM to include only certain pop indicators for COP24 in createDATIMExport.
Patched update_de_coc_co_map to allow for certain data elements to be included even though they were left off the data entry forms
Updated getCOPDataFromDATIM, as it relates to imports, to work with COP24
Updated getMemoIndicators to allow for COP24
Updated memoStructure to allow for COP24

datapackr 7.3.2

Updated packForDATIM to included pop indicators for COP24 in createDATIMExport.

datapackr 7.3.1

Updated how checkMechanisms handles dedupe mechs.

datapackr 7.3.0

Updated COP validation rules for 2024
Updated checkAnalytics to analyze 2024 psnu data
Updated writePSNUxIM to be able to write 2024 PSNUxIM tabs

Updated unit tests related to the New features above

datapackr 7.2.1

Updates to COP24 Template

Completed updates related to OPU Data Pack's conversion PSNUxIM tools
Updated country list for COP24 Target Setting Tool Generation script
Updated regular expression related to Year2 KP data

datapackr 7.2.0

Finalized model
Able to produce production 2024 Target Setting Tools

Updated unPackYear2Sheet regular expression to better capture data.

Update to Year 2 tab for OVC to include 18-20 age band
update default cell to A1 for all tabs
Updated packDataPack to hide rows in Year 2 and AGYW sheets
Updated documentation related to getValidOrgUnits
Updated documentation related to extractDuplicateRows

datapackr 7.1.1

Added support for either PSNUxIM or OPU Data Packs for COP23.

upgrade related to check params and Added exclusion of COP24 for regions

Reorganized COP24 files
Cached valid_OrgUnits as CSV
Updated createDatimExport to handle COP24
Saved DEsCOCs metadata as CSV accessible via API

datapackr 7.1.0

Updated logic related to the PSNUxIm.
Updated various configuration files for year 2024

Updated home tab title to reflect new fiscal year syntax.
Updated createKycahinInfo to reflect new fiscal year syntax
Moved getPSNUs into its own file
Updated metadata, reviewed formulas, and formatting.
Added tests related to unpackSNUxIM and dataset creation
Updated year 23 data pack template and schema
Various small updates to how configuration files are generated

datapackr 7.0.3

Updated datapackVSDatim to be more representative.
Updated COP22 schema update script
Updated cached map script for cogs for year 2024
Fixed getRDA for unit tests
Updated unit test related to create schema.

Added year 2024 to getHTSModality
Updated unPackSchema in how it handles multi uid patterns.
Updated update_de_coc_co_map for year 2024
Added an utility function related to DHIS2 UID's patterns
Updated various configuration files

datapackr 7.0.2

Opens generation & processing of COP24 Data Packs.

Corrects formatting of randomized Spectrum data saved in testing files from RData to rds.

Exports writeSpectrumData to allow functionalized writing of Spectrum data to Spectrum tab of Data Packs.
Updated some minor R package documentation.
compareData_DatapackVsDatim updated to no longer include a dedupe data frame. This is more in line with the current operations of datim, and will assist with imports.

datapackr 7.0.1

COP24 PSNUxIM Tool Template created

Updated dataset levels
Fixed issue related to ordering by age in the memo tables

COP24 Data Pack Template updated to incorporate formulas flagged for edit during COP23.
COP24 Data Pack Template updated to increment FY references in Year 2 tab.
Package dependencies updated

datapackr 7.0.0

Cop24 Data Pack Template created

Updated valid Org Units to include UG SNU 2 PSNU's

datapackr 6.3.3

Added season as a parameter in most unpacking and packing functions to better align with new workflows — PSNUxIM tools separate, and both PSNUxIM tool & Data Pack valid for both COP & OPU. This is in preparation for the deprecation of the concept of a separate "OPU Tool", as this is now functionally identical to the PSNUxIM tool used during main COP.

datapackr 6.3.2

Fixed issue with merging of message queues when unpacking multiple tools

Dedupe Handling
Tool to Datim comparison improvements
Handling of mech file refresh
Updated all dependencies to current versions

datapackr 6.3.1

Updated handling of zero values for non dedupe mechs.
Functionalized the process for mapping mechanism data from Data Pack indicators to DATIM dataElements & categoryOptionCombos.

datapackr 6.3.0

Updated handling of KP_ESTIMATES dataElements to tag with the current COP_YEAR period, rather than the previous. This is different from how other IMPATT estimates are handled, but was a direct request from OGAC.

datapackr 6.2.3

Adds function to update PSNUxIM values from the main tabs

Addresses some minor issues with duplication in the Data element/COC map

Fixed an issue with certain tests failing in certain testing infrastructures
Updated cached organisation unit map from DATIM

datapackr 6.2.2

Fixed an issue with countries in regions being flagged as having invalid prioritizations
Fixed an issue with analytics checks when only using a PSNUxIM tool

datapackr 6.2.1

Adds support for parsing COP23 PSNUxIM sheets
Adds wrapper function unPackToolSet to handle multiple tools
Adds functionality for handling Year 2 exports
Adds utility function extractRawColumnData which can retrieve raw data from a sheet
Created a centralized method for creating DATIM exports

Added unit tests for PSNU's for COP23 generation
Updated COP 23 validation rules to match DATIM
Updated analytics checks for COP23
Analytics checks comparisons are made with rounded numbers to two significant digits
Resolved issue with age band differences between tools and DATIM
Fixed critical formulas
Updated DE/COC map

datapackr 6.2.0

Removed support for COP21

Added initial support for parsing COP23 Datapacks
Added functionality to export COP23 data to export formats
Added function writeSpectrumData which can be used to populate the DataPack Spectrum tab during testing.
Added testing helper functions to deal with peculiarities of the CI testing environment.
Added various tests and testing files for COP23 tools.
Adds initial parsing methods for COP23 Year2 tabs

Upgraded CI testing environment to R 4.2.1
Fixed bug in unPackSchemarelated to detecting invalid column and value types.
Fixed create schema unit test.
Removed superfluous warning related to missing PSNUs.
Fixed issue in unPackingChecks related to the lack of the SNU1 column in COP23 tools.
Fixed a testing issue related to choosing the correct template to use for testing.
Updated several unit tests to favor COP23 over COP21.
Disabled two unit tests for COP21.
Altered test method from devtools::check to devtools::test, which skips CRAN package checks.

datapackr 6.1.1

Script for generating COP 23 test data

Add additional unit tests to support COP 23
Various formatting improvements as it relates to the exported file
Updated valid Org Units and Dataset Levels.

datapackr 6.1.0

Fixes issues with getMechanismView. The current implementation was too slow, and prone to timing out. The entire view will be downloaded (which is time consuming) once and then all filters happen on the full cached file.
Fixes a bug with COP22 data element/category option combo maps. 50+ age bands were being dropped with full COP22 datapacks.

Adds a check for defunct disaggregates in the PSNUxim tab.
Adds an object dataset_levels which provides the organisation unit hierarchy of each country by COP year.
Adds a test for blank_rows_datim_export which should inform if any rows of any export files to DATIM contain any blanks.
Fixes dedupe handling in regards to OPU's

Fixes an issue with add_dp_label for COP22 datapacks. The orgunit type was missing.
Removes deprecated DATIM API functions, which have been moved to datimutils.
Add various unit tests.

datapackr 6.0.0

Users are advised to usegetValidOrgunits(cop_year) to obtain a list of valid organisation units for a given COP year.

Support generation of COP23 target setting tools
Correctly supply a list of organisation units by COP year.

Numerous code fixes and test improvements.

datapackr 5.5.1

Added TX_PVLS(D) to memo

datapackr 5.5.0

Use datimvalidation's expression lexer to validate incoming indicator expression

Refactored how unPackSNUxIM function operates.
Retire internal API utility functions and replace with datimutils functions
Removed duplicative code in packDataPack that was already in packTool
Added tests for packDataPack

datapackr 5.4.0

Adds support for COP22 OPU Data Pack processing
Adds mergeDatapack function for use in merging multiple datapacks along with related unit test.

Fixes an issue with COP Memo indicator calculations
New unit tests for OPU generation
Fixes an issue with dedupes being dropped in the PSNUxIM tab

datapackr 5.3.2

Removes getFY22Prioritizations in favor of clearer, more stable, internal functions.
Replaces getPSNUs with with more versatile (and appropriately named) getDataPackOrgUnits. Many of the org units used in Data Packs are actually not PSNUs (e.g., DREAMS SNUs, or Military SNUs), making many of the columns created previously less than sensible. This function now also produces a tibble instead of a data frame. In some cases, column names in this output have also slightly changed.
Similarly, replaces add_dp_psnu with add_dp_label.
adorn_import_file can now correctly assign prioritizations to org units below PSNU (e.g., for DREAMS SNUs in Rwanda & Eswatini).

Adds the functions cache_is_fresh and fresh_cache_part for use in anticipated functions allowing the caching of metadata objects retrieved from the DATIM API, allowing faster processing for repeat requests.
Adds the function checkToolEmptySheets to allow checking and unpacking processes to detect sheets with either no rows of data, or no text in key header columns (e.g., PSNU, Age, Sex, KeyPop). In these cases, unpacking and checking functions cannot accurately assess data, so these sheets should be skipped. This function results in updates in unPackingChecks, unPackSheets, and unPackDataPackSheet.

datapackr 5.3.0

Removes the following functions from the package:
checkComments
checkDuplicateRows
checkExternalLinks
checkMissingMetadata
checkNumericValues
checkStructure
compareTemplateToSchema
defunctDisaggs
exportSubnatToDATIM
getDataValueSets
packForDATIM_MER
packForDATIM_OPU
packForDATIM_UndistributedMER
Refactors packDataPack and packOPUDataPack to work with new packTool wrapper function.
Common functionality is removed from individual functions and moved into packTool to remove duplication.
The packDataPack function now only takes in d and d2_session as arguments.
The packOPUDataPack function now takes in a d object, an undistributed_mer_data dataframe, and a d2_session object as arguments. When an undistributed_mer_data object is provided, it will use these target values for distributing rather than taking target values from DATIM.
Each function now checks if appropriate model filepath has been provided and then writes tabs into pre-generated file.

Add the following functions, along with related tests:
loadSheets
readSheet
checkFormulas
checkDisaggs (instead of defunctDisaggs)
checkInvalidPrioritizations
checkInvalidOrgUnits
checkNegativeValues
checkDecimalValues
checkMissingMetadata
checkNonNumeric (instead of checkNumericValues)
checkDupeRows (instead of checkDuplicateRows)
checkColumnStructure (instead of checkColStructure)
checkToolStructure (instead of checkStructure)
checkToolConnections (instead of checkExternalLinks)
checkToolComments (instead of checkComments)
checkSheets
extract_uid
extract_uid_all
Creates a new wrapper function packTool around packDataPack and packOPUDataPack
Adds checkOutputFolder function to check_params set that validates that an output folder has been provided, that the directory exists, and can be written to.

Refactors unPackDataPackSheet to reduce cyclomatic complexity and make it functional over multiple sheets at once.
Updates unPackDataPack to use new check functions (see above).
Updates unPackTool to use loadDataPack instead of createKeychainInfo.
Simplifies and generalizes unPackDataPackSheet.
Generalizes separateDataSets
Turns off checkMissingMetadata in unPackSNUxIM for now.
Updates createKeychainInfo to be more accurate in deducing metadata from submitted Data Packs.
Updates loadDataPack to be production ready.
Refactors packForDATIM to streamline all related functions into one and removes packForDATIM_MER, packForDATIM_OPU, and packForDATIM_UndistributedMER. Updates related functions accordingly.
Removes deprecated code for exportSubnatToDatim function.
Removes deprecated compareTemplateToSchema function after removing last instance of usage in packDataPack.
Creates data-raw/COP21OPU_PSNUxIM_generation_from_Data_Pack.R which allows the generation of a COP21 OPU tool that includes the target values from a COP21 Data Pack tool and the % distributions from DATIM.
Fixes tests for the following functions:
listWorkbookContents
packForDATIM
separateDataSets
unPackSheets
prepareMemoData
createAnalytics

datapackr 5.2.3

Deprecates pullDATIMCodeList and pullFullCodeList functions in favor of getCodeList
Retires api_sql_call function (DP-603)

Creates new function getCodeList which utilizes datimutils API utilities and replaces pullDATIMCodeList and pullFullCodeList (DP-599)
getCodeList uses cop_year as a parameter rather than the FY parameter previously used by pullDATIMCodeList and pullFullCodeList

Adds unit tests for get_Map_DataPack_DATIM_DEs_COCs and checkPSNUData (DP-272)
Refactors the following functions to utilize datimutils API utilities:
getHTSModality (DP-597)
map_Cs_to_COs (DP-605)
Refactors package to remove hard dependencies on stringi, glue, data.table, and readr packages (DP-670, DP-671, DP-687, DP-693)
Refactors package to standardize on cop_year over fiscal_year and FY for parameters (DP-691)
Refactors package to standardize usage of pick_schema across functions (DP-294)
Refactors packSNUxIM to reduce cyclomatic complexity (DP-662)

datapackr 5.2.2

Fixes checkPSNUData and prepareMemoData to stop crashes on Windows by avoiding paralellization if Windows OS detected
Adds check to messageQueue to avoid potential mismatches of message level and message text

Merges getMechanismViewFromDATIM into getMechanismView

Tightens checks of tool and cop_year against one another. Adds datapackrSupports, supportedCOPYears, & supportedTools.

Refactors the following functions to utilize datimutils instead of datapackr API functions:
getMechanismView
map_COCs_to_COs
Refactors getDataSetUids, writePSNUxIM, and packPSNUxIM to reduce cyclomatic complexity of both functions
Switches getDataSetUids to use cop_year instead of FY in parameters
Simplifies pullFullCodeList
Increments getCurrentCOPYear
Adjusts paste_oxford to accommodate length = 2
Updates checkInvalidIndicatorCodes to utilize pick_schema instead of hard coded schema
Refactors package to remove dependencies on R6 (DP-694), plyr (DP-672), and rlist (DP-684)
Clears a number of warnings and notes during build checks:
Adds missing documentation
Adds missing package declarations
Fixes incorrect argument full.name to full.names in list.files call in extractWorkbook function
Updates datapackr.R to add missing global variables
Adds test for the following functions:
canReadFile
checkDuplicateRows
checkInvalidIndicatorCodes
checkMechanisms
defuntDisaggs
fetchPrioritizationTable
getHTSModality
getOPUDataFromDATIM
Adds more tests for memo generation, including memoStructure and prepareMemoData
Fixes pullFullCodeList test for FY2022
Fixes broken test for getCurrentCOPYear incrementing
Disables play-spectrum test. Also increments cop_year within this test
Adds more files to .gitignore and .Rbuildignore
Refactors prepareMemoData and prepareMemoDataByPSNU to avoid parallelization on CI environments
Update Circle CI configuration to specify large resource class and add insights snapshot to README.md

datapackr 5.2.1

Fix to packOPUDataPack by reverting changes to parameter names in exportPackr calls that were accidentally made during release v5.2.0.

datapackr 5.2.0

Removes the following content as part of the deprecation of COP20 OPU handling:
R/packSNUxIM_2020.R
R/packSNUxIM_OPU.R
R/unPackOPU_PSNUxIM.R
data/cop20_data_pack_schema.rda
data/cop20_map_DataPack_DATIM_DEs_COCs.rda
data/cop20OPU_data_pack_schema.rda
data/data_pack_schema.rda
data/updated_indicator_codes.rda
data/valid_category_options.rda
data-raw/cop20_validation_rules.json
data-raw/COP20OPU_Data_Pack_generation_script.R
data-raw/COP20OPU_Data_Pack_processing_script.R
data-raw/COP20OPU_Data_Pack_validation_script.R
data-raw/cop21_datapack_indicator_code_updates.csv
data-raw/GetDataForGlobalFund.R
data-raw/TEST_COP20_OPU_Data_Pack_Template.xlsx
data-raw/update_cop20_datapack_template.R
data-raw/update_cop20OPU_datapack_schema.R
data-raw/update_cached_valid_COs.R
data-raw/valid_COCs_COs.rda
inst/extdata/COP20_Data_Pack_Template_Final.xlsx
inst/extdata/COP20_OPU_Data_Pack_Template.xlsx
Removes the following legacy content from COP19:
data/data_pack_schema.rda
data-raw/checkTX_CURR.R
data-raw/produceConfigFile.R
Removes COP20 processing in the following functions:
adorn_import_file
check_cop_year
check_tool
checkMechanisms
createAnalytics
createKeychainInfo
createTestDataset
compareData_OpuDatapackVsDatim
getCOPDataFromDATIM
getDataSetUids
getHTSModality
getMapDataPack_DATIM_DEs_COCs
getMemoIndicators
getOPUDataFromDATIM
headerRow
memoStructure
packDataPack
packForDATIM_OPU
pick_schema
pick_template_path
skip_tabs
unPackOPUDataPack
unPackSNUxIM
unPackSchema_datapack
writePSNUxIM
Removes the following functions that are no longer used anywhere in the package:
adornMechanisms
adornPSNUs
deriveTargets
getCountries
getIMPATTLevels
getNumeratorDenominator
getTechArea
getValidCategoryOptions
getValidCOs
packForPAW
pull_COPindicators
Renames .testInvalidIndicatorCodes to checkInvalidIndicatorCodes

Fixes a bug in updateExistingPrioritization that was preventing tools from being unpacked or memos from being generated in the apps

Changes default schema parameter from data_pack_schema to pick_schema() for the following functions:
packDataPackSheet
packSheets
prepareSheetData
Removes lazyeval and datapackcommons as required packages as neither are used in the package anywhere

datapackr 5.1.7

Removes PrEP_CT >= PrEP_NEW validation rule from checks. (DP-552)
Removes DIAGNOSED_SUBNAT (N, SUBNAT, HIVStatus) TARGET >= DIAGNOSED_SUBNAT (N, SUBNAT, Age/Sex/HIV) TARGET + DIAGNOSED_SUBNAT (N, SUBNAT, HIV/Sex) TARGET validation rule from checks. (DP-554)
Restores "affected_rows" column to the altered formula tab in the validation rules output file. (DP-572)
Fixes bug that prevented Technical Assistance (TA) only implementing mechanisms (IMs) from populating in the PSNUxIM tab. (DP-575)
Fixes bug in checkMechanisms.
Ensures that default/unallocated data is included in COP Approval Memo tables.
Reverts changes to function unPackSNUxIM in R/unPackSNUxIM.R back to 5.1.5.

Adds new utility function formatSetStrings to assist in the checkFormulas function. Formats a vector of numbers into a string of ranges.

Cleans up code and adds documentation to functions in the packPSNUxIM` tree. (DP-473)
Strengthens validation of PSNUs: (DP-509)
- Creates new function parsePSNUs
- Errors out invalid PSNU strings are detected
- Provides a warning only if NO PSNUs are detected
- Adds unit tests
Replaces deprecated code in swapColumns function located in R/utilities.R: (DP-547)
- Replaces mutate_ with base R fix to remove deprecation warning when writing PSNUXIM tab.
- Added tests for swapColumns in tests/testthat/test-utilities.R
Update FY23 MER DOD Targets COC UID to "o71WtN5JrUu" in the following locations: (DP-576 and DP-577)
- R/utilities.R
- tests/testthat/test-GetDatasetUids.R
- data-raw/update_cop22_de_coc_co_map.R
- data/cop22_map_DataPack_DATIM_DEs_COCs.rda
- data/cop22_map_adorn_import_file.rda
Adding whole package to linting checks. Lints data-raw and tests folders now in addition to R folder. (DP-578)
Aligns on datastreams parameter to consolidate parameter names across functions (DP-579)
- Changes datastream parameter for pullFullCodeList to datastreams
- Changes streams parameter for getCOPDataFromDATIM to datastreams
- Changes parameters in all relevant locations
- Moves definition of datastreams parameter to datapackr_params
- Removes duplicative parameter definitions from getCOPDataFromDATIM and inherits them from datapackr_params instead
Adds missing d2_session parameter to getCountries function in R/utilities.R (DP-580)
Aligns on output_folder as parameter for all functions that export files (DP-581)
- Changes output_path parameter in exportPackr to output_folder
- Replaces parameter names for exportPackr in all relevant locations
- Modifies output_folder definition in datapackr_params to remove outdated reference to defaulting to working directory.
- Removes duplicative parameter definitions in functions that were touched and replaces with inheritance from datapackr_params
Remove unused parameters (DP-582)
- Removes type argument from deriveTargets.
- Removes country_names argument from getOPUDataFromDATIM.
- Removes duplicative parameter definitions from getOPUDataFromDATIM and inherits definitions from datapackr_params.

datapackr 5.1.6

Fixes mismatch between text and metadata on certain validation messages that caused some warnings to be highlighted red as if they were errors and caused some errors not to be highlighted red. (DP-523)
Fixes missing PrEP_CT data on approval memo target tables. (DP-561)
Fixes missing AGYW_PREV data on approval memo target tables. (DP-562)
Fixes reference to memo template in generateMemoTemplate.

Adds a system environment variable MAX_CORES which will limit the number of cores to be used in parallel processing. This environment variable is detected by the new function getMaxCores.
Adds parameter append to writePSNUxIM. This parameter is TRUE by default and will append new rows to the existing PSNUxIM tab. If FALSE, a new PSNUxIM tab will be output only containing the new rows needed in the validated tool.
Adds parameter include_default to getMechanismView and adorn_import_file that will include the default mechanism to the dataframe that is returned.
Adds new function checkHasPSNUxIM to detect whether a COP Data Pack has a PSNUxIM tab.

Updates alterned formula warning from checkFormulas to clarify that users may edit formulas in Green columns without permission, but need to receive clearance from DUIT prior to editing formulas in Grey columns. (DP-241)
Makes checkColStructure more robust.
Adds back stringi dependency due to new use of stringi::stri_replace_all. This will be replaced with a stringr function in a later version to reduce dependencies.

datapackr 5.1.5

Adds COP Approval Memo target table generation to the package and enables COP22 Approvale Memo target table generation support (DP-503, DP-528, DP-534, DP-536)
Creates the following new functions to aid in the fetching, cleaning, and formatting of data used in the various target tables of COP approval memos:
- evaluateIndicators
- fetchPrioritizationTable
- generateComparisonTable
- getMemoIndicators
- memoStructure
- prepareExistingDataAnalytics
- prepareMemoData
- prepareMemoDataByAgency
- prepareMemoDataByPSNU
- prepareMemoDataByPartner
- prepareMemoDataByPrio
- prepareMemoMetadata
- updateExistingPrioritization
Creates the following new functions to aid in the creation and formatting of COP Approval Memo target tables:
- getMemoTemplate selects whether to supply a watermarked "Draft Memo" Word document or a blank document for final memo tables.
- renderAgencyTable, renderPartnerTable, and renderPrioTable each generate formatted tables for their respective sections of the memo.
- generateApprovalMemo compiles all target tables together in a single document.
- default_memo_font, default_memo_style_header, and default_memo_style_para contain information on standard memo formatting.
- zeros_to_dashes is a utility function used in formatting memos appropriately.
Adds cop_validation_rules as new dataset object within package
- Adds data-raw file to create and update cop_validation_rules object
Adds draft_memo_template.docx under inst/extdata folder for use in draft memo generation
Adds tests for new functions:
- Adds test-get-memo-indicators to test getMemoIndicators
- Adds test-indicators to test evaluateIndicators
- Adds test-memo-structure to test memoStructure
Adds additional non-memo related functionality from the datapackr-app into the package (DP-504)
Creates checkMechanisms and checkPSNUData to replace validateMechanisms and validatePSNUData from the datapackr-app, respectively. These functions are used in validating and analyzing Data Pack targets.
Adds sane_name, source_user, and operating_unit as new variables under d$info in keychain
- Creates getSaneName and getOUFromCountryUIDs as new functions
- Modifies createKeychainInfo to create new variables under d$info
- Modifies packDataPack to create new variables under d$info
- Adds d2_session argument to createKeychainInfo
- Updates relevant tests to incorporate new elements of keychain
Enables the ability to get COP22 data using getCOPDataFromDATIM (DP-536)

Adds parallel, gdtools, flextable, and officer as suggested packages

datapackr 5.1.4

Ignore duplicated Not PEPFAR columns in PSNUxIM
Fixes for handling 50+ age bands in PSNUxIM packing and unpacking

datapackr 5.1.3

Downgrades openxslx to version 4.2.3 to prevent error with PSNUxIM packing.
Changes the HTS modality group set to HTS Modality (USE ONLY for FY22 Results/FY23 Targets).
Change compareData_OpuDatapackVsDatim to NOT compare AGYW data which are not part of OPU data packs.

Changes arguments for unPackSchema_datapack and validateSchema to use standard argument names. Replaces filepath with template_path and replaces their usage in all relevant locations.

Reexports magrittr %<>% function.
Adds missing function interactive_warning.
Removes function getPass since this code is commented out and functionality has been replaced by datimutils.
Cleans up mismatched brackets in data.R file and allows for creation of documentation for all datasets.
Updates .Rbuildignore to include all appropriate files and folder.
Clears error with R dependency by downgrading required version to R 4.1.0.
Updates package dependencies:
- Adds dependency and remote for datimutils.
- Adds dependency for methods.
- Suggests waldo package.
- Removes scales and stringi as dependencies as they are not used.
- Moves assertthat from Imports to Suggests since it is only used in data-raw files.
Moves the following variables from data-raw files to environment variables. Users can add these to an .Rprofile to streamline their workflow:
- SECRETS_FOLDER
- OUTPUT_FOLDER
- MODEL_DATA_PATH
- SNUXIM_MODEL_DATA_PATH

datapackr 5.1.2

Corrects the % of EID by 12 months from 95% (same as by 2 months) to 100%.
Updates comments and conditional formatting in the Cascade Tab over HTS_INDEX yield columns to align with COP Guidance (<10% as warning, rather than 20%).

For COP22 Data Packs, enables appending data to the bottom of the PSNUxIM tab when data is detected in the main tabs of the Data Pack, but not reflected in the PSNUxIM tab.

datapackr 5.1.1

This maintenance release fixes a bug with unPackCountryUIDs that was introduced in v5.0.1 due to linting. This bug preventing the processing of COP and OPU tools in datapackr-app as well as broke the createAnalytics function for regional tools.

datapackr 5.1.0

The function packForDATIM_UndistributedMER now takes in arguments for MER data and COP year instead of an entire d object and returns a table of undistributed MER data rather than return a d object with data nested under d$data$UndistributedMER.

Updates COP22 Data Pack template and processing code.
- Removes "Summary" tab from COP22 tools and processing code.
- Includes "Not PEPFAR" column to PSNUxIM tab and adds support for processing this data. Drops dataf from this column before creating the analytics table and MER exports.
Adds item to d object for unallocated IMs nested under d$info$unallocatedIMs.

Additional handling for default Category Option Combos.
Adds dataset UIDs for COP22 to the getDatasetUids function.
Updates getMapDataPack_DATIM_DEs_COCs to include handling for COP22 map.
Adds test for getDatasetUids
Adds explicit comparison checks when updating schemas and Data Pack-DATIM mapping files using the waldo package.

datapackr 5.0.3

Initial launch of COP22 Data Pack processing!

Now requires R version 4.1.1 or higher.

Updated and improved documentation of datasets in datapackr.
Improves handling of default categoryOptionCombo.
Improves documentation of packDataPackSheet, packSheets, and prepareSheetData.

loginToDATIM is retired in favor of the same function in datimutils. All instances of this function being invoked have been replaced appropriately.
- The functions DHISLogin, GetCredentialsFromConsole, LoadConfigFile, and ValidateConfig were not exported and are now deprecated as well. They were previously only used by loginToDATIM.
isLoggedIn is retired as it was only used in getMechList and loginToDATIM.

datapackr 5.0.2

Resolves a bug with packOPUDataPack where createDataPack was not implemented correctly in version 5.0.1.
Patches a bug with getOPUDataFromDATIM where getCOPDataFromDATIM returns a dataframe where the default Category Option Combo UID is listed as default rather than the appropriate DATIM UID. This will be removed in favor of a more permanent solution in future updates.

Significantly improves handling of parameter checks and standardizes their validation and defaults. Documentation for these checks is also added.
Adds functionality for producing COP22 Beta Packs and test data.

Removes getDataPackSchema in favor of consolidated pick_schema.

getDataPackSchema has been deprecated in favor of pick_schema and has been replaced in the two locations where it was previously used.

Improves and updates tests related to parameter checks and schemas.
Introduces many new small utilities functions such as %missing% and %||%.
Improves automation of Data Pack Template/schema validation.

datapackr 5.0.1

loadDataPack is a new function that returns a Data Pack object conserving styles and formatting of the original Data Pack .xlsx file, as well as other metadata necessary for processing and analysing data in the Data Pack.
.testInvalidIndicatorCodes was previously an internal function that is now documented and exported by the package. This function tests for invalid indicator codes in a d Datapackr object.
datapack_cogs is a new dataset containing Datapack Category option groups (id and name) along with their individual category options (id and name) as a nested data frame.

createWorkbook has been renamed createDataPack to deconflict with the openxlsx function createWorkbook. This function now returns a d datapackr object rather than an openxlsx workbook object.
The d2_session argument has been removed from the following functions:
- check_params
- createDataPack (previously createWorkbook)
- unPackSchema_datapack

unPackSchema_datapack was modified in the following ways:
- Now uses the datapack_cogs data set rather than making a query to DATIM.
- Inherits parameters from datapackr_params.
writeHomeTab was modified in the following ways:
- The wb and datapack_name arguments default to NULL.
- Checks and assigns parameters using the check_params function.
- Lists country names on the Home tab in addition to Country UIDs.
Minor corrections were made to Excel functions written by packSNUxIM that had been erroneously changed during previous linting.
Internal changes were made to variable names and functions used inside the check_params function.
A new file has been added to data-raw to generate the datapack_cogs data set.
Documentation is now provided for the cop20OPU_data_pack_schema data set.

pepfar-datim/datapackr documentation built on Jan. 14, 2025, 7:11 a.m.

rdrr.io home R language documentation Run R code online

CRAN packages Bioconductor packages R-Forge packages GitHub packages

Note that we can't provide technical support on individual packages. You should contact the package authors for that.

pepfar-datim/datapackr A Package that Packs and Unpacks all Data Packs and Target Setting Tools

NEWS.md In pepfar-datim/datapackr: A Package that Packs and Unpacks all Data Packs and Target Setting Tools

datapackr 8.2.0

New features

Bug fixes

Minor improvements and fixes

datapackr 8.1.0

New features

Bug fixes

Minor improvements and fixes

datapackr 8.0.1

New features

Bug fixes

Minor improvements and fixes

datapackr 8.0.0

New features

Bug fixes

Minor improvements and fixes

datapackr 7.6.2

New features

Bug fixes

Minor improvements and fixes

datapackr 7.6.1

New features

Bug fixes

Minor improvements and fixes

datapackr 7.6.0

New features

Bug fixes

Minor improvements and fixes

datapackr 7.5.0

New features

Bug fixes

Minor improvements and fixes

datapackr 7.4.4

New features

Bug fixes

Minor improvements and fixes

datapackr 7.4.3

New features

Bug fixes

Minor improvements and fixes

datapackr 7.4.1

Bug fixes

datapackr 7.4.0

New features

Bug fixes

Minor improvements and fixes

datapackr 7.3.3

New features

Bug fixes

Minor improvements and fixes

datapackr 7.3.2

New features

Bug fixes

Minor improvements and fixes

datapackr 7.3.1

New features

Bug fixes

Minor improvements and fixes

datapackr 7.3.0

New features

Bug fixes

Minor improvements and fixes

datapackr 7.2.1

New features

Bug fixes

Minor improvements and fixes

datapackr 7.2.0

New features

Bug fixes

Minor improvements and fixes

datapackr 7.1.1

New features

Bug fixes

Minor improvements and fixes

datapackr 7.1.0

New features

Bug fixes

Minor improvements and fixes

pepfar-datim/datapackr
A Package that Packs and Unpacks all Data Packs and Target Setting Tools

NEWS.md
In pepfar-datim/datapackr: A Package that Packs and Unpacks all Data Packs and Target Setting Tools