merge industry subsector structuremappingIO files back into originals

Question

merge industry subsector structuremappingIO files back into originals

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q opened this issue a year ago · 23 comments

0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q commented a year ago

Running calcOutput("IO", subtype = "output", file = "f04_IO_output.cs4r") once with structuremappingIO_outputs.csv and once with structuremappingIO_outputs_Industry_subsectors.csv as the mapping file yields two results:
f04_IO_output.csv
f04_IO_output_Industry_subsectors.csv

comp <- full_join(
  read_csv(file = '/tmp/f04_IO_output.csv',
                      col_names = c('t', 'regi', 'enty.1', 'enty.2', 'te',
                                    'value_outputs'),
                      col_types = 'iccccd',
                      comment = '*') %>% 
  arrange(t, regi, enty.1, enty.2, te),
  
  read_csv(file = '/tmp/f04_IO_output_Industry_subsectors.csv',
                         col_names = c('t', 'regi', 'enty.1', 'enty.2', 'te',
                                       'value_subsectors'),
                         col_types = 'iccccd',
                         comment = '*') %>% 
  arrange(t, regi, enty.1, enty.2, te),
  
  c('t', 'regi', 'enty.1', 'enty.2', 'te')
)

Differences where outputs and subsectors have different values are limited to

comp %>% 
  filter(value_outputs != value_subsectors) %>% 
  mutate(diff = value_subsectors - value_outputs) %>% 
  distinct(enty.1, enty.2, te)

  enty.1   enty.2   te      
  <chr>    <chr>    <chr>   
1 pecoal   segafos  coalgas 
2 pecoal   sesofos  coaltr  
3 seliqfos fehoi_cs tdhoi_cs

Differences where outputs has values, but subsectors does not:

comp %>% 
  filter(is.na(value_subsectors)) %>% 
  distinct(enty.1, enty.2, te)

   enty.1   enty.2   te      
   <chr>    <chr>    <chr>   
 1 seel     feelcb   tdelb   
 2 seel     feeli    tdeli   
 3 segabio  fegai    tdbiogai
 4 segafos  fegai    tdfosgai
 5 segafos  fegai_cs tdgai_cs
 6 sehe     fehei    tdhei   
 7 seliqbio fehoi    tdbiohoi
 8 seliqfos fehoi    tdfoshoi
 9 sesobio  fesoi    tdbiosoi
10 sesofos  fesoi    tdfossoi

Differences where subsectors has values, but outputs does not:

comp %>% 
  filter(is.na(value_outputs)) %>% 
  distinct(enty.1, enty.2, te) %>% 
  print(n = Inf)

   enty.1   enty.2             te                
   <chr>    <chr>              <chr>             
 1 seel     feel_cement        tdel_cement       
 2 seel     feel_steel         tdel_steel        
 3 seel     feelwlth_chemicals tdelwlth_chemicals
 4 seel     feelwlth_otherInd  tdelwlth_otherInd 
 5 segabio  fega_cement        tdga_cement       
 6 segabio  fega_chemicals     tdga_chemicals    
 7 segabio  fega_otherInd      tdga_otherInd     
 8 segabio  fega_steel         tdga_steel        
 9 segafos  fega_cement        tdga_cement       
10 segafos  fega_chemicals     tdga_chemicals    
11 segafos  fega_otherInd      tdga_otherInd     
12 segafos  fega_steel         tdga_steel        
13 segafos  fegai_cs           tdfosgai_cs       
14 sehe     fehe_otherInd      tdhe_otherInd     
15 seliqbio feli_cement        tdli_cement       
16 seliqbio feli_chemicals     tdli_chemicals    
17 seliqbio feli_otherInd      tdli_otherInd     
18 seliqbio feli_steel         tdli_steel        
19 seliqfos feli_cement        tdli_cement       
20 seliqfos feli_chemicals     tdli_chemicals    
21 seliqfos feli_otherInd      tdli_otherInd     
22 seliqfos feli_steel         tdli_steel        
23 sesobio  feso_cement        tdso_cement       
24 sesobio  feso_chemicals     tdso_chemicals    
25 sesobio  feso_otherInd      tdso_otherInd     
26 sesobio  feso_steel         tdso_steel        
27 sesofos  feso_cement        tdso_cement       
28 sesofos  feso_chemicals     tdso_chemicals    
29 sesofos  feso_otherInd      tdso_otherInd     
30 sesofos  feso_steel         tdso_steel

Answer 1 · 2023-11-21T14:10:48.000Z

add summation of subsector FE use, should provide fe(so|he|ga|h2|he|el)i
check segafos . fegai_cs . tdfosgai_cs ./. segafos . fegai_cs . tdgai_cs (genuine inconsistency?)
what is seel . feelcb . tdelb?
map out coalgas, coaltr, and tdhoi_cs differences

Answer 2 · 2023-11-22T13:55:27.000Z

* [ ]  what is `seel . feelcb . tdelb`?

I guess I should know but to be honest, I never understood why we have stationary in the model and so I don't know what this is supposed to be. From REMIND code I understand that tdelb is feels -> feelb and not the full t&d from secondary level. Also feelcb would be just the non-heating part of buildings electricity which is incomplete. Sorry for this much cluelessness...

Answer 3 · 2023-11-23T08:07:18.000Z

It is sufficient to know that it is used somewhere in REMIND.
I guess that Antoine added it after I created structuremappingIO_outputs_Industry_subsectors.csv. If only changes to those files would have been tracked …
Question @robinhasse: is buildings/simple used and operational?

Answer 4 · 2023-11-23T10:14:59.000Z

is buildings/simple used and operational?

It is the only realisation that is used and operational.
Also, seel.feelcb.tdelb likely comes from @Renato-Rodrigues as there was no differentiation of electricity back when Antoine worked on this.

Answer 5 · 2023-11-23T10:55:57.000Z

It is the only realisation that is used and operational.

Ouch.

Answer 6 · 2023-11-23T10:57:34.000Z

@robinhasse "stationary" is the aggregate of buildings, industry, services, etc... everything that is not moving. That was a aggregate used many years ago, when we did not want to increase the number of ces-nests too much. As it is: such things keep on living in code after they were abandoned. Anyways...
regarding the feelcb it seems to me that this is what used to be feelb. I see the same entries in structuremappingIO_outputs.csv with feelcb that were used in structuremappingIO_outputs_old.csv but with feelb. That seeme to be a renaming or it is simply a bug.
On the entries in the table I agree with @robertpietzcker about his remakrs about deleting unused mappings.

Answer 7 · 2023-11-23T11:08:36.000Z

Thanks, Nico. It is not just a renaming. We used to only have feelb (all electricity in buildings). Now it is the sum of feelcb (non-heating), feelrhb (resistive heating) and feelhpb (heat pumps). So I actually think that seel.feelcb.tdelb is wrong. I though that tdels is seel --> feels and than we have the two links to buildings and industry tdeli: feels --> feeli and tdelb: feels --> feelb.

Answer 8 · 2023-11-23T11:31:41.000Z

@robinhasse did I understand correctly that this increased the number of primary inputs into the overall CES-tree? If so, I recommend to strongly evaluate whether this is really necessary. Every CES-nest and -branch makes the model more difficult to calibrate.

Answer 9 · 2023-11-23T11:58:05.000Z

@nicobauer, you are about two and a half years late with that admonition. And the calibration can deal with that just fine.

Answer 10 · 2023-11-23T15:11:01.000Z

Also feelcb would be just the non-heating part of buildings electricity which is incomplete.

but is there actually a "heat pump electricity" input for 2005 in REMIND? because how would you get such a value if IEA doesn't tell you how electricity for residential splits into the different uses?

or is there some buildings-related adjustment on the IEA numbers that splits the elec going into buildings by a 10%/90% split into heat pump / other elec?

so it seems to me that (after checking it is correct) this part would need to be transferred into a new "structuremapping_IO_merged" - or how is your process proposal for this, Michaja? I would think creating a third "merged" mapping will best allow us to determine if it really produces all the right output when the two old instances are replaced by the new merged one.

Answer 11 · 2023-11-23T15:43:09.000Z

My plan is to recreate the output of the outputs mapping & code with the outputs_Industry_subsectors mapping & code verbatim, except for the instances where they should be extended or are wrong.

Answer 12 · 2023-11-27T13:58:42.000Z

but is there actually a "heat pump electricity" input for 2005 in REMIND? because how would you get such a value if IEA doesn't tell you how electricity for residential splits into the different uses?

could you briefly think about this and confirm/correct, @robinhasse ?

Answer 13 · 2023-11-27T14:30:24.000Z

but is there actually a "heat pump electricity" input for 2005 in REMIND? because how would you get such a value if IEA doesn't tell you how electricity for residential splits into the different uses?

could you briefly think about this and confirm/correct, @robinhasse ?

Irrelevant to this issue.

No, there is only feelcb in f04_IO_output.cs4r:

$ tar -Oxf /p/projects/rd3mod/inputdata/output/rev6.605_62eff8f7_remind.tgz ./f04_IO_output.cs4r | awk -F ',' '$1 !~ /^\*/ && !a[$4] ++ { print $4 }'
seliqfos
fehoi
fesoi
fesob
sesofos
seel
sehe
fedie
fehob
seliqbio
fegai
fegab
segabio
fegat
fepet
fehoi_cs
segafos
sesobio
feeli
feel
feelt
fehei
feheb
fegai_cs
feelcb
feelb

Answer 14 · 2023-11-27T14:51:38.000Z

Irrelevant to this issue.

Hm, I had thought the original question if/whether seel . feelcb . tdelb is something that should be there or something that should in fact be different/be called differently was still relevant, because I did not get a clear idea from Robin's reactions if this is indeed needed or not.

in your list I see feelcb and feelb, and I currently think they should NOT both be outputs from f04_IO_output.cs4r, but only one of the two. (based on the assumption that the IEA data cannot differentiate between the two)

Answer 15 · 2023-11-27T15:18:47.000Z

Hm, I had thought the original question if/whether seel . feelcb . tdelb is something that should be there or something that should in fact be different/be called differently was still relevant, because I did not get a clear idea from Robin's reactions if this is indeed needed or not.

If that is the case, it applies to the output mapping as well. I will just recreate that mapping (where industry is not concerned). So this is not relevant here.

Answer 16 · 2023-11-27T15:36:20.000Z

Ok, I understand.

Is there any further support I can offer for anything in this issue, or is it mainly "finding time" ?

Answer 17 · 2023-11-27T15:38:35.000Z

Nothing right now.

Answer 18 · 2024-02-29T15:41:25.000Z

pushing this up again - is there any progress? if not - what are the chances this advances over the next week, and how can they be increased?

Answer 19 · 2024-03-01T08:03:02.000Z

This is not going to be finished next week, and the chances of changing that are zero (short of Joshua 10:13).

Answer 20 · 2024-03-01T08:43:00.000Z

thanks for the info. (and I wanted to write "weeks", not "week", but given your reply on the other issue I guess the answer remains the same

Answer 21 · 2024-03-01T08:51:34.000Z

But to forward this issue in general:

check segafos . fegai_cs . tdfosgai_cs ./. segafos . fegai_cs . tdgai_cs (genuine inconsistency?)

@robertpietzcker are fehoi_cs and fegai_cs still used?
They do appear in f04_IO_output.cs4r (generated by calcIO()), but in the code they only show up in ./core/sets.gms as part of all_enty [↑], but in no mappings. And that appears to have been the case for the last four years [↑].
So, is there some magical mechanism by which these two have an bearing on the PE-to-SE calibration, or can we just cut them?

Answer 22 · 2024-09-13T11:55:32.000Z

But to forward this issue in general:

check segafos . fegai_cs . tdfosgai_cs ./. segafos . fegai_cs . tdgai_cs (genuine inconsistency?)

@robertpietzcker are fehoi_cs and fegai_cs still used? They do appear in f04_IO_output.cs4r (generated by calcIO()), but in the code they only show up in ./core/sets.gms as part of all_enty [↑], but in no mappings. And that appears to have been the case for the last four years [↑]. So, is there some magical mechanism by which these two have an bearing on the PE-to-SE calibration, or can we just cut them?

*poke* @robertpietzcker