README.md
In NERC-CEH/ukem: Package Title (one line, title case)

title: " ukem: UK Emissions Model - spatio-temporal modelling of emissions of greenhouse gases and air pollutants" author: "Peter Levy" edits: "Sam Tomlinson" date: "2022-01-04" output: html_document: keep_md: yes

This document describes the ukem code written in R for producing spatio-temporal predictions of UK emissions of GHGs and air pollutants for the period 1990 to near-present. The package uses the data structures provided by the raster package (https://CRAN.R-project.org/package=raster), but output can be written to generic netCDF files for use in other programs.

New target : tar_target(lookup_CRF, "data/lookups/CRF_to_SNAP.csv", format = "file")
New in targets: in target(dt_raw_uk) i have passed the target lookup_NRF as an argument so that updates in the csv not only trigger the lookup_NRF target but also trigger a re-running of the dt_raw_uk, which previously it was not. PETE TO CHECK
- should the same be done as above but for lookup_SIC and lookup_CRF ?
scale_stack_to_nationalTotal() changed to include adding SNAP4 to SNAP3 in maps and national totals. Snap 4 converted to 0 emissions. This is done pre-scaling.
Another thing to note with the above is that in getAlpha_byYear(), alpha_year is generated but is not used in scaling rasters. However it is plotted. This would obviously not line up once SN03 & SN04 were added.
various changes made, removing hard-coded years and pollutant etc.
Should CH4 (and others) have a blank 12th sector map if they dont get marine emissions, to make the same structure?
PETE: I think your CH4 solvents map might exist because there are ~120 CH4 point sources (small total) that had nothing to be added to, and it might be those points that you were using for ethane. Furthermore it's the only GHG with a missing map, but it is a bit more common in APs (SOx, NH3...) and metals, so i've changed the way the maps stacks are generated.

consider everything below this line to be out of date or superceded

Cursiouly, if you change something in the Eire GHG lookup table, targets wont pick it up - is this a case of just making a new target for it? e.g. tar_target(lookup_CRF, "data/lookups/CRF_to_SNAP.csv", format = "file")
Switching to UNFCC data for CO2/N2O/CH4 for Eire - much better agreement with MapEire data and categories, while awkward as hell to make a lookup for, are better than the text description EPA ones.
At the moment, for CO2, land conversion emissions/sinks not coded in SNAP/NFR lookup - Pete please advise.
ISSUE: point sources incorrectly allocated to SNAP sectors in getPointSourceData()
match() does not correctly match pollutant_id as two pm2.5 rows, causes errors.
SectorID to SNAP changes dependent on pollutant so match() was creating lots of incorrect lookups.
Therefore the function produces a coherent table but is incorrect
must match SectorID by pollutant.
Resolved: add pollutant IDs to all pollutants in SIC_to_SNAP.csv and use join syntax.
Diffuse maps read in are fine but points added were incorrect, now fine.
ISSUE: point sources not added correctly ('+' not working?)
data lost due to calculation not working, changed to 'calc', setting no points to blank map. Fixed.
Scaling of stack to national total will also scale points - just a flag.
why not do this post re-projection? to conserve totals AND only do it once. ENABLED
WORRYING: The 2019 CO2 maps I have on P:/ (and therefore onto Git) are different to ones I have just downloaded from NAEI (20/01/22). Total area CO2 down from 81Mt to 69Mt. Replacing on Git with newest. Only CO2.
ISSUE: CO2 maps: Points are not converted to CO2 (from C) as for diffuse and nat totals. Fixed.
ISSUE: CO2 maps: UK Nat total = ~410Mt, diffuse + point maps = ~382Mt
ISSUE: CO2 maps: Sector totals from UK Nat totals and from diff+point maps are quite different. Results in quite dramatic scaling factors. e.g. Nat totals for SN3 = 44Mt & SN4 = 12.6Mt ; maps + points for SN3 = 63Mt & SN4 = 0.5Mt (they look connected). That means some maps get scaled and give undue influence? SN5 bad, SN6 bad but small, SN8 bad, SN9 bad but small, SNAp10 bad. SNAPs 1,2 & 7 fine.
ISSUE: CO2 maps: Eire CO2 from UNFCC totals for SN3 = 4.6Mt & SN4 = 2.1Mt ; maps for SN3 = 7.1Mt & SN4 = 0Mt (they look connected). SNAP 1,2 are ok match. SNAP 5 empty in UNFCC, tiny on map. SNAP 6 tiny in UNFCC & map. SNAP 7, 8, 9 & 10 great. SNAP11 double in UNFCC (6.3Mt vs 3.6Mt). Good agreement on total though (ME = 44.3 Mt, UNFCC = 47.2 Mt).
ISSUE: CO2 maps: Therefore when scaling Eire to nat totals, you lose SN4 and half of SN3 but SN11 doubles. Do we need to make a decision about industrial CO2?
ISSUE: Marine: it appears to be included with both UK and IE stacks? surely just the once. Temp Fix: removed from Eire get_maps_static() so as only added once to uK. PETE CHECK.
ISSUE: n_sectors: In get_maps_static(), changed to make marine 12th sector as previously it acted as 12th layer/SNAP, but the functions worked on a basis of 11, causing errors that dont cause runtime errors in the functions. Fixed. (might need to add 12th blank layer for CH4?)

I have changed the hard-coded year=2018 to year=2019 in both readUKDiffuseData() & readIEDiffuseData(). Perhaps we should consider a change to this.
ERROR!!! there is no 'solvents' CH4 map from NAEI in 2019. Causing error for CH4!
have copied 2018 over (and renamed) FOR NOW. Where did Pete get 2018?
i think all functions need hard-coded 2018 stripping out?
Have linked the './data/lookups/NFR19_to_SNAP_vNov21.csv' file.
Have linked to 2019 NAEI maps but have extended the directory name in readUKDiffuseData()
Have linked to 2019 MapEire maps but have extended the directory name in readIEDiffuseData()
Consider removing all the preset year_ref, startYear and endYear values?
Consider rescaling the reprojected data back to it's total when in BNG ; get_maps_static ()
i think this rescaling is important, the mapped data is not the same as NAEI_ts_long.csv
ISSUE: CO2 maps have some negative values (check re-projection for this, can happen with points)
ISSUE: NAEI summaries are correct but mapped totals appear wrong. Check pre re-projected totals

functions run - UK & Eire maps appear to be 'flatter' than before. I think the primary issue was in 'get_maps_static' whereby the Irish layer was somehow overwriting the UK emissions with zero. Have adjusted. Need to consider rescaling reprojected data.
New NFR/SNAP lookup uploaded (./data/lookups/NFR19_to_SNAP_vNov21.csv) - it currently doesnt have CRF or IPCC2006 coding - is this a requirement?
MapEire 2019 maps uploaded (./data-raw/diffuse_ie/ME19/ folder) but not linked in function yet
NAEI 2019 maps uploaded (./data-raw/diffuse_uk/UK19/ folder) but not linked in function yet. Also, as ".tif" files.
Marine emissions component remains untouched by SJT.

I could add a slightly updated NFR19 to SNAP lookup, developed under DUKEMs?
I have MapEire 2019 surfaces, to update the 2016 version
upload the 2019 NAEI maps

Why set the default endYear to be 2018 in all the functions?
Have added new query IDs for NAEI emission downloads
Added NAEI 2019 point data to ./data-raw/point_sources/
Error in getEmissionTimeSeriesFromBEIS() - the appending of APs to GHGs causes error. The GHG data written first is from 1990 while the AP data appended to it is from 1970 - this misaligns the cols for AP below the GHG and causes function error. Changed to rbindlist() and fwrite()
23/11/21: NAEI_ts_long.csv matches my output totals for UK
What is dt_ts in run_ukem_targets.R (L149)? a combo of dt_ts_uk & dt_ts_ie? (perhaps ignore)
think about converting readUKDiffuseData() (and others?) to terra
Not adjusted yet: GNFR P_IntShipping should probably be mapped to SNAP8 as it is in the NFR19 to SNAP.
not really clear where scaling of maps takes place (scale_stack_to_nationalTotal ?) and point/diffuse integration takes place (add_points_to_diffuse ?)