automotive only portfolio can not run stress testing

https://dev.azure.com/2DegreesInvesting/2DegreesInvesting/_workitems/edit/3214

When a user's portfolio that resolves to having only Automotive relevant assets, stress testing fails because there will be only one scenario here (SDS)

r2dii.climate.stress.test/web_tool_stress_test.R

Lines 289 to 294 in 42a283a

    
           # check scenario availability across data inputs for equity 
        
           check_scenario_availability( 
        
             portfolio = pacta_equity_results, 
        
             scen_data = scenario_data, 
        
             scenarios = scenarios_filter 
        
           )

The allowable scenarios are defined in the model_parameters config file, e.g.

r2dii.climate.stress.test/model_parameters/model_parameters_PA2021NO.yml

Lines 11 to 14 in 42a283a

    
           scenarios: 
        
             scenario_to_follow_baseline: NPS 
        
             scenario_to_follow_ls: SDS 
        
             scenario_to_follow_ls_aligned: SDS

Which is read in here and set to scenarios_filter

r2dii.climate.stress.test/web_tool_stress_test.R

Lines 116 to 134 in 42a283a

    
           #### Model variables---------------------------------------- 
        
           #### OPEN: This should be moved into a StressTestModelParameters.yml 
        
           cfg_mod <- config::get(file = file.path(stress_test_path, "model_parameters", paste0("model_parameters_", project_code, ".yml"))) 
        
           # OPEN: wrap reading in of params in function and move to global_functions 
        
           end_year <- cfg_mod$end_year # Set to 2040 cause current scenario data goes until 2040. can be extended when WEO2020 turns out extended horizon 
        
           # Scenarios in the model_parameters.yml file must have the short names (SDS, NPS, etc) 
        
           scenario_to_follow_baseline <- cfg_mod$scenarios$scenario_to_follow_baseline # sets which scenario trajectory the baseline scenario follows 
        
           scenario_to_follow_ls <- cfg_mod$scenarios$scenario_to_follow_ls # sets which scenario trajectory LS scenario follows after shock period 
        
           scenario_to_follow_ls_aligned <- cfg_mod$scenarios$scenario_to_follow_ls_aligned 
        
           scenarios_filter <- unique( 
        
             c( 
        
               scenario_to_follow_baseline, 
        
               scenario_to_follow_ls, 
        
               scenario_to_follow_ls_aligned 
        
             ) 
        
           )

and then applied as a filter here

r2dii.climate.stress.test/web_tool_stress_test.R

Lines 263 to 277 in 42a283a

    
           pacta_equity_results <- pacta_equity_results_full %>% 
        
             tidyr::complete( 
        
               year = seq(start_year, start_year + time_horizon), 
        
               nesting(!!!syms(nesting_vars)) 
        
             ) %>% 
        
             dplyr::mutate(plan_tech_prod = dplyr::if_else(is.na(plan_tech_prod), 0, plan_tech_prod)) %>% 
        
             apply_filters( 
        
               investor = investor_name, 
        
               sectors = sectors, 
        
               technologies = technologies, 
        
               scenario_geography_filter = scenario_geography_filter, 
        
               scenarios = scenarios_filter, 
        
               allocation_method = allocation_method_equity, 
        
               start_analysis = start_year 
        
             ) %>%

Here's a test portfolio
TestPortfolio_Input.csv

Thanks for the detailed overview on the issue @cjyetman

As it affects the webtool I would suggest you take it over @jacobvjk

Some thoughts though:

I only allowed use of portfolios with only 1 sector for the user workflows in this PR, in order to not alter webtool behavior 05ebe6a#diff-b6b6854a02ae177f8860f49654ff250c1554d021ae434b6efdbe99d00bdc6055R23 this PR can likely give some orientation though
taking a step back, you said that in general including portfolios with only 1 sector does not make sense for stresstesting, how do you evaluate this in the webtool case?
If there was only SDS scenario included there would be no baseline, so no shocking can happen. This should not be a consequence of only including 1 sector, is this description of the behavior correct in this regard?

why would there be no NPS scenario for automotive @cjyetman ? In all previous versions of the webtool, whenever holdings from the Auto sector were shocked, there must have been NPS as well. So it seems odd this would not be the case when a portfolio only has Auto holdings.

If you run the test portfolio I provided through PACTA, the "Equity_results_portfolio.rda" will have NPS and SDS for Automotive sector (e.g. in the attached file). That's the relevant PACTA result file that gets read in here in the stress test web tool.
Equity_results_portfolio.rda.zip

okay thanks. for some weird reason my mac does not want to properly uncompress the zip file, but what you describe sound like the problem is indeed that we only have one sector and the code will expect holdings that allow using both ETP and WEO. We had a similar issue with the banks stress test recently. If it is okay for your timeline, we would take a detailed look tomorrow morning @cjyetman

found the issue. Automotive gives us scenario ET2017_NPSRTS. It used to be ETP2017_NPS. The filters are kicking that out, because we do not expect to have to do the string manipulation at that point. We can move the corresponding line which turns NPSRTS into NPS to make it pass the filter. But I would suggest checking why we changed what comes through from pacta. We did not make any changes to this section lately, so to me this looks like a data artefact. I am also reluctant to simply changing this because it worked with other projects, so I do not want to risk their integrity with this fix. @cjyetman do we know if any of the Norwegian portfolios show results for automotive?

"results for automotive" as in any automotive data driving any of the charts or tables in the interactive report?

sorry, that was imprecise. I am wondering if any of the stress test charts for Norwegian portfolios display a shock in the Automotive sector (section 4.2). My assumption would be that they all don't do that, but only the ones that have all their PACTA-related holdings in that one sector will fail?

I don't want to mess around with a live PA2021NO user's account, but... if I add one line to the above test portfolio with an isin for a Power company, the stress test code does NOT fail/error-out, the report is generated, and I get these charts in the stress testing section/s...

does that answer your question?

yes, thanks @cjyetman . It still leaves open the question on how best to tackle this though. My assumption is that the scenario data we use on the transition monitor now passes ETP2017_NPSRTS through PACTA. And I would assume that it used to be only ETP2017_NPS. We should validate that and think about what we want it to use there. I am reluctant to changing anything on our side until we have clarity on whether or not the data source used in PACTA changed.
fyi @MirijaHa

ok, thanks... I'll investigate and get back to you

in pacta-data...

sort(unique(readRDS("2019Q4/equity_ald_scenario.rda")$scenario))
#>  [1] "ETP2017_B2DS"  "ETP2017_NPS"   "ETP2017_SDS"   "GECO2019_1.5c" "GECO2019_2c_m" "GECO2019_ref"  "SBTI_SBTI"    
#>  [8] "WEO2019_CPS"   "WEO2019_NPS"   "WEO2019_SDS"   "WEO2020_NPS"   "WEO2020_SDS"  
sort(unique(readRDS("2020Q4/equity_ald_scenario.rda")$scenario))
#>  [1] "ETP2017_B2DS"         "ETP2017_NPSRTS"       "ETP2017_SDS"          "GECO2019_1.5c"        "GECO2019_2c_m"       
#>  [6] "GECO2019_ref"         "IPR2021_IPR FPS 2021" "SBTI_SBTI"            "WEO2019_CPS"          "WEO2019_SDS"         
#> [11] "WEO2019_SPS"          "WEO2020_SDS"          "WEO2020_SPS"

also on Dropbox...

dropbox_analysis_inputs_path <- "~/Dropbox (2° Investing)/PortCheck/00_Data/07_AnalysisInputs"

scenarios_analysisinput_2021 <- 
  readRDS(file.path(dropbox_analysis_inputs_path, 
                    "2020Q4_transitionmonitor/scenarios_analysisinput_2021.rds"))
sort(unique(paste0(scenarios_analysisinput_2021$scenario_source, "_", scenarios_analysisinput_2021$scenario)))
#> [1] "ETP2017_B2DS"   "ETP2017_NPSRTS" "ETP2017_SDS"    "GECO2019_1.5c" 
#> [5] "GECO2019_2c_m"  "GECO2019_ref"   "SBTI_SBTI"      "WEO2019_CPS"   
#> [9] "WEO2019_SDS"    "WEO2019_SPS"    "WEO2020_SDS"    "WEO2020_SPS"   


scenarios_analysisinput_2021_v0.2.3 <- 
  readRDS(file.path(dropbox_analysis_inputs_path, 
                    "2020Q4_transitionmonitor_v0.2.3/scenarios_analysisinput_2021.rds"))
sort(unique(paste0(scenarios_analysisinput_2021_v0.2.3$scenario_source, "_", scenarios_analysisinput_2021_v0.2.3$scenario)))
#> [1] "ETP2017_B2DS"   "ETP2017_NPSRTS" "ETP2017_SDS"    "GECO2019_1.5c" 
#> [5] "GECO2019_2c_m"  "GECO2019_ref"   "SBTI_SBTI"      "WEO2019_CPS"   
#> [9] "WEO2019_SDS"    "WEO2019_SPS"    "WEO2020_SDS"    "WEO2020_SPS"

okay, so the scenario name changed. That's what's causing it. I think it is safe to fix it on our side, because there is no chance to create duplicates, as only one of the two ET2017_NPS or ETP2017_NPSRTS will ever be in the same data set.
What do you think @MirijaHa ?

would it be possible to parameterize this, at least in web_tool_external_stress_test.R and/or web_tool_stress_test.R, so it's up to us to set this properly, rather than the stress testing team trying to anticipate what we're going to send you?

@cjyetman Can u please concretise what you mean by setting this properly?
If I understand it correctly the current problem is a mix of 2 things:

inconsistency on NPS/NPSTRS because there is a configuration to use NPS

r2dii.climate.stress.test/model_parameters/model_parameters_PA2021NO.yml

Lines 11 to 14 in 42a283a

    
           scenarios: 
        
             scenario_to_follow_baseline: NPS 
        
             scenario_to_follow_ls: SDS 
        
             scenario_to_follow_ls_aligned: SDS

, while in "2020Q4/equity_ald_scenario.rda" and "<>scenarios_analysisinput_2021<>" this scenario is named NPSRTS

some pretty confusing and currently incorrectly placed handling in stresstesting code to map NPSRTS to NPS

In general: We are planning to parameterise use of scenarios in the, much easier and better tested, code for the user facing workflow. I would prefer to implement it there first, so we can get our head around all methodological pitfalls there and then translate it to the webtool if still needed. Therefore in the current case I would prefer to go with the simple fix of putting the NPS NPSTRS mapping to the correct location.

I guess I mean... in the long term it probably makes most sense for PACTA to be able to tell the stress test code with some parameters in the PACTA_analysis repo which scenarios to use/expect, rather than hard-coding in a switch from one thing to another. So maybe moving something similar to r2dii.climate.stress.test/model_parameters/model_parameters_PA2021NO.yml into the PACTA_analysis repo that could then somehow be passed to or read by the stress test web tool code. Does that clarify it?

or maybe it makes most sense, right now anyway, to edit all of the model_parameters/model_parameters_XXXX.yml style files here that target projects that are meant to use 2020Q4 data to use

scenario_to_follow_baseline: NPSRTS

instead of

scenario_to_follow_baseline: NPS

though that's only really a partial fix because GENERAL users could potentially be using 2019Q4 or 2020Q4

Sure, in the long, or even medium run this is certainly the way to go where:

scenarios to use are configured on analysis level by PACTA
PACTA takes care of consistency between the requested scenarios and availability in the passed data.

For now I would still want to go with the hardcoded mapping so we have this out of the way and revisit it after implementing parameterisation for the user workflow, which is scheduled to be done in Q1/2022.

thanks!

Merged, can you please retrigger the failing calculation (or start the process that will result in the user doing that) and close the issue if succesful" 😄

yes! using the test portfolio I posted in the original message, I pass stress test with no error and I see Automotive data in the stress test chart. thanks!

	# check scenario availability across data inputs for equity
	check_scenario_availability(
	portfolio = pacta_equity_results,
	scen_data = scenario_data,
	scenarios = scenarios_filter
	)

	scenarios:
	scenario_to_follow_baseline: NPS
	scenario_to_follow_ls: SDS
	scenario_to_follow_ls_aligned: SDS

	#### Model variables----------------------------------------
	#### OPEN: This should be moved into a StressTestModelParameters.yml
	cfg_mod <- config::get(file = file.path(stress_test_path, "model_parameters", paste0("model_parameters_", project_code, ".yml")))

	# OPEN: wrap reading in of params in function and move to global_functions
	end_year <- cfg_mod$end_year # Set to 2040 cause current scenario data goes until 2040. can be extended when WEO2020 turns out extended horizon

	# Scenarios in the model_parameters.yml file must have the short names (SDS, NPS, etc)
	scenario_to_follow_baseline <- cfg_mod$scenarios$scenario_to_follow_baseline # sets which scenario trajectory the baseline scenario follows
	scenario_to_follow_ls <- cfg_mod$scenarios$scenario_to_follow_ls # sets which scenario trajectory LS scenario follows after shock period
	scenario_to_follow_ls_aligned <- cfg_mod$scenarios$scenario_to_follow_ls_aligned

	scenarios_filter <- unique(
	c(
	scenario_to_follow_baseline,
	scenario_to_follow_ls,
	scenario_to_follow_ls_aligned
	)
	)

	pacta_equity_results <- pacta_equity_results_full %>%
	tidyr::complete(
	year = seq(start_year, start_year + time_horizon),
	nesting(!!!syms(nesting_vars))
	) %>%
	dplyr::mutate(plan_tech_prod = dplyr::if_else(is.na(plan_tech_prod), 0, plan_tech_prod)) %>%
	apply_filters(
	investor = investor_name,
	sectors = sectors,
	technologies = technologies,
	scenario_geography_filter = scenario_geography_filter,
	scenarios = scenarios_filter,
	allocation_method = allocation_method_equity,
	start_analysis = start_year
	) %>%