NNPDF/nnpdf

Separate `eko.tar` and fastkernel tables in the server

scarlehoff opened this issue · 3 comments

scarlehoff commented

Given that the tables for our current NNLO theory are about ~700 MB and the eko for the postfit evolution instead ~1 GB, I think it would be very convenient to separate both.

I think this would bring only advantages:

  1. ekos are not needed for validphys, so no need to have them in your computer to make plots
  2. If you need to copy stuff to your nodes when doing a fit, you halve the size of what you are copying for the fit itself.
  3. We have many theories without an eko, where the eko is actually shared (e.g. theory 200, 400, 700, 708, 600, 40000000). Making it a separate entity means you can have just one 1GB object in the server instead of many.

I would not change anything of the architecture once they are downloaded since they get used in the same manner. I would just create another resource which then gets downloaded to the right theory id and evolven3fit will depend on it.

RoyStegeman commented

So would there be two download scripts as well? The advantages seem very minor to me (memory is very cheap and bandwidth when downloading is hardly an issue) so I'm not really sure this would be worth the effort and increased complexity of the theory management (in particular due to point 3)

scarlehoff commented

So would there be two download scripts as well?

No, it is the same resource loader.

memory is very cheap

Just wait until you get your mac.
Jokes aside, I'm getting my jobs throttled in the cluster due to this.

(and we had to ask CERN for extra space because we exhausted the disk space that was allocated to us)

bandwidth when downloading is hardly an issue

At home in Germany I usually need a few attempts because getting 2GB of data in one go is too much for Vodafone.

so I'm not really sure this would be worth the effort and increased complexity of the theory management

There are quite a few theories without an eko inside. And right now it might not be complex but it is somewhat broken (#2003). Also, I'd say doing it in this way would reduce complexity, not increase it.

scarlehoff commented

After talking a bit with @comane, this gives us yet another advantage.
If the eko is not included in the theory (and given that most of the weight of the theories is now the eko) we can try to use the normal theories for the tests. The fktables will be heavier that theories like 399 but it also allows us to test exactly the same thing we are running.