Transfer Requests for Run2016 data

Question

Transfer Requests for Run2016 data

mdunser opened this issue 8 years ago · 22 comments

Hi all,

I'm starting a new issue for data-transfers of Run2016. Many of the datasets, despite being subscribed originally to CERN are being deleted fairly rapidly.

After some interaction with CompOps it has become clear that there is no longer any guarantee that any dataset of any run-period in 2016 will remain at the CERN T2. Computing resources are hitting their absolute limit pretty much everywhere, so whatever workflow that used to work in 2015 is no longer guaranteed to work in 2016.

For users of heppy and heppy_batch that means that they should have a look into using one of two options:

useAAA which copies the root files locally to /tmp via xrd and runs on them from there. In my experience this works well, though it sometimes takes many resubmissions for it to finally run through. This requires setting the environment variable X509_USER_PROXY to an empty file that exists.
crab3 works well, with some adaptions to the run-config and the crab/ directory. If you plan on running loads of data now or in the future, this might be worthwhile checking out. In my experience this work well.

It is, of course, still possible to transfer datasets to our local buffer, but be advised that this may take up to a week or more from your request on this thread to the dataset finally being present at CERN, and in addition the buffer has a limited size, so there is no way we can store all data in it just for the sake of it.

Best,
-m

Answer 1 · 2016-07-26T10:12:35.000Z

On Tue, Jul 26, 2016 at 12:02 PM, mdunser notifications@github.com wrote:

Hi all,

I'm starting a new issue for data-transfers of Run2016. Many of the
datasets, despite being subscribed originally to CERN are being deleted
fairly rapidly.

After some interaction with CompOps it has become clear that there is no
longer any guarantee that any dataset of any run-period in 2016 will
remain at the CERN T2. Computing resources are hitting their absolute limit
pretty much everywhere, so whatever workflow that used to work in 2015 is
no longer guaranteed to work in 2016.

For users of heppy and heppy_batch that means that they should have a look
into using one of two options:

useAAA which copies the root files locally to /tmp via xrd and runs on
them from there. In my experience this works well, though it sometimes
takes many resubmissions for it to finally run through. This requires
setting the environment variable X509_USER_PROXY to an empty file that
exists.

crab3 works well, with some adaptions to the run-config and the crab/
directory. If you plan on running loads of data now or in the future, this
might be worthwhile checking out. In my experience this work well.

It is, of course, still possible to transfer datasets to our local buffer,
but be advised that this may take up to a week or more from your request on
this thread to the dataset finally being present at CERN, and in addition
the buffer has a limited size, so there is no way we can store all data in
it just for the sake of it.

We should discuss in a CMG group meeting - I believe the group should be
able to find the resources for hosting the data at CERN.
MC is less of an issue since individual job failures do not affect the
latency (one doesn't need 100.0% of complete jobs to be able to use a MC)

Giovanni

Best,
-m

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#666, or mute the thread
https://github.com/notifications/unsubscribe-auth/AEbbR2lmmafnsL0cT57J7gDIY_SnOB0xks5qZdspgaJpZM4JU-5I
.

Answer 2 · 2016-08-24T10:56:14.000Z

Hi Marc,

Can you transfer

/SingleMuon/Run2016G-PromptReco-v1/MINIAOD ?

Thanks,
Jan

Answer 3 · 2016-08-24T11:16:25.000Z

Voila: https://cmsweb.cern.ch/phedex/prod/Request::View?request=765363
-m

On 24 Aug 2016, at 12:56, Jan Steggemann <notifications@github.com mailto:notifications@github.com> wrote:

Hi Marc,

Can you transfer

/SingleMuon/Run2016G-PromptReco-v1/MINIAOD ?

Thanks,
Jan

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/666#issuecomment-242026014, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEdevChErlIW1wy5AeIxxaJKjzTq9Hnfks5qjCNPgaJpZM4JU-5I.

Answer 4 · 2016-10-07T16:39:13.000Z

Hi Marc,

could you please transfer

/MET/Run2016F-PromptReco-v1/MINIAOD
/DoubleMuon/Run2016G-PromptReco-v1/MINIAOD

Many thanks,
cristina

Answer 5 · 2016-10-07T16:41:37.000Z

sorry, I meant:
/MET/Run2016F-PromptReco-v1/MINIAOD
/DoubleMuon/Run2016F-PromptReco-v1/MINIAOD

Answer 6 · 2016-10-07T16:42:43.000Z

hi cristina,

RunG i had requested some time ago, so that one should be there.
I just requested all RunF datasets to be replicated at CERN, but that
transfer will take a while I suppose.

-m

On 07 Oct 2016, at 18:39, cbotta <notifications@github.com mailto:notifications@github.com> wrote:

Hi Marc,

could you please transfer

/MET/Run2016F-PromptReco-v1/MINIAOD
/DoubleMuon/Run2016G-PromptReco-v1/MINIAOD

Many thanks,
cristina

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/666#issuecomment-252300878, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEdevLiIGwqj1qPBZxmF50MmweKmyKoGks5qxnWygaJpZM4JU-5I.

Answer 7 · 2016-10-08T10:02:58.000Z

just for the record, here the request: https://cmsweb.cern.ch/phedex/prod/Request::View?request=802579
-m

On 07 Oct 2016, at 18:42, Marc Dunser <marc.dunser@cern.ch mailto:marc.dunser@cern.ch> wrote:

hi cristina,

RunG i had requested some time ago, so that one should be there.
I just requested all RunF datasets to be replicated at CERN, but that
transfer will take a while I suppose.

-m

On 07 Oct 2016, at 18:39, cbotta <notifications@github.com mailto:notifications@github.com> wrote:

Hi Marc,

could you please transfer

/MET/Run2016F-PromptReco-v1/MINIAOD
/DoubleMuon/Run2016G-PromptReco-v1/MINIAOD

Many thanks,
cristina

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/666#issuecomment-252300878, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEdevLiIGwqj1qPBZxmF50MmweKmyKoGks5qxnWygaJpZM4JU-5I.

Answer 8 · 2016-10-31T19:57:21.000Z

Hi Marc,

I need to look into these data somewhat urgently

/SingleMuon/Run2016H-PromptReco-v1/MINIAOD
/SingleMuon/Run2016H-PromptReco-v2/MINIAOD
/SingleMuon/Run2016H-PromptReco-v3/MINIAOD

Given your initial message, would a transfer request still make sense or I'd be better off using crab / AAA?

Thanks,
Riccardo

Answer 9 · 2016-10-31T20:18:30.000Z

Hi Riccardo,

so the first dataset (v1) is already at CERN so you should be able to access it
without crab or AAA.

The other two don’t exist.

Best,
-m

On 31 Oct 2016, at 20:57, Riccardo Manzoni <notifications@github.com mailto:notifications@github.com> wrote:

Hi Marc,

I need to look into these data somewhat urgently

/SingleMuon/Run2016G-PromptReco-v1/MINIAOD
/SingleMuon/Run2016G-PromptReco-v2/MINIAOD
/SingleMuon/Run2016G-PromptReco-v3/MINIAOD

Given your initial message, would a transfer request still make sense or I'd be better off using crab / AAA?

Thanks,
Riccardo

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/666#issuecomment-257402686, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEdevLTtWW7ZCT42zFhpELvwptJIUSipks5q5kgigaJpZM4JU-5I.

Answer 10 · 2016-10-31T20:19:44.000Z

Sorry Marc, I corrected the first message, I meant 2016H v1 to v3.
Thanks,
Riccardo

Answer 11 · 2016-10-31T20:27:50.000Z

Hi,

I made the request for all RunH (v1,v2,v3) samples just now:

https://cmsweb.cern.ch/phedex/prod/Request::View?request=817882

It will probably be approved tomorrow, and only then will the transfer start, so
you can expect the samples earliest tomorrow ~evening.

Depending on your interpretation of “urgently” you might want to consider
using AAA / crab.

Best,
-m

On 31 Oct 2016, at 21:19, Riccardo Manzoni <notifications@github.com mailto:notifications@github.com> wrote:

Sorry Marc, I corrected the first message, I meant 2016H v1 to v3.
Thanks,
Riccardo

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/666#issuecomment-257408766, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEdevJRogB-2ElcP7Bk8QN7f0BeLX7erks5q5k1jgaJpZM4JU-5I.

Answer 12 · 2016-11-11T10:48:17.000Z

Hello Marc,

could you please add to
https://cmsweb.cern.ch/phedex/prod/Request::View?request=822372
also
/DoubleMuon/Run2016_-23Sep2016-v_/MINIAOD
?

Many thanks
cheers,
Cristina

Answer 13 · 2016-11-11T10:57:47.000Z

Hi,

this was already part of this request:
https://cmsweb.cern.ch/phedex/prod/Request::View?request=820680
And the files should already be here...

Or am I missing something?

-m

On 11 Nov 2016, at 11:48, cbotta <notifications@github.com mailto:notifications@github.com> wrote:

Hello Marc,

could you please add to
https://cmsweb.cern.ch/phedex/prod/Request::View?request=822372
also
/DoubleMuon/Run2016-23Sep2016-v/MINIAOD
?

Many thanks
cheers,
Cristina

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/666#issuecomment-259931784, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEdevBx_HE895lc1ngMcYPIASbzHvPRuks5q9EfygaJpZM4JU-5I.

Answer 14 · 2016-11-11T11:02:46.000Z

Ciao Marc.

Would be great if you can add the

/SinglePhoton/Run2016B-23Sep2016-v3/MINIAOD
/SinglePhoton/Run2016C-23Sep2016-v1/MINIAOD
/SinglePhoton/Run2016D-23Sep2016-v1/MINIAOD
/SinglePhoton/Run2016E-23Sep2016-v1/MINIAOD
/SinglePhoton/Run2016F-23Sep2016-v1/MINIAOD
/SinglePhoton/Run2016G-23Sep2016-v1/MINIAOD

do not see in the https://cmsweb.cern.ch/phedex/prod/Request::View?request=820680

Thanks

Maria

Answer 15 · 2016-11-11T11:14:29.000Z

Hi Maria, all,

here’s the list of the datasets in the local space:

https://cmsweb.cern.ch/phedex/prod/Data::Subscriptions#state=create_since%3D1344868584%3Bgroup%3Dlocal%3Bnode%3D1561https://cmsweb.cern.ch/phedex/prod/Data::Subscriptions#state=create_since=1344868584;group=local;node=1561

The request which has the SinglePhotons in it:

https://cmsweb.cern.ch/phedex/prod/Request::View?request=820681

For all I can tell right now, all these datasets are here.

-m

On 11 Nov 2016, at 12:02, mariadalfonso <notifications@github.com mailto:notifications@github.com> wrote:

Ciao Marc.

Would be great if you can add the

/SinglePhoton/Run2016B-23Sep2016-v3/MINIAOD
/SinglePhoton/Run2016C-23Sep2016-v1/MINIAOD
/SinglePhoton/Run2016D-23Sep2016-v1/MINIAOD
/SinglePhoton/Run2016E-23Sep2016-v1/MINIAOD
/SinglePhoton/Run2016F-23Sep2016-v1/MINIAOD
/SinglePhoton/Run2016G-23Sep2016-v1/MINIAOD

do not see in the https://cmsweb.cern.ch/phedex/prod/Request::View?request=820680

Thanks

Maria

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com//issues/666#issuecomment-259934215, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEdevCkChh2zc1jHELRaaLZKnEmMIt5Oks5q9EtWgaJpZM4JU-5I.

Answer 16 · 2017-01-11T13:30:08.000Z

Hi all,

new year, new cleaning-up campaign. Hooray!

The situation is that the extended local space which is now 300 TB is overfull. We are a few TB above our quota, so there is no way around cleaning up a certain amount of the data we have subscribed to the local space.

So, unless there are some solid reasons for keeping any of the following datasets in the local space, I will request their deletion. This will free up roughly 50 TB, which should buy us some time before we (I) have to think about something new/smarter.

These are all PromptReco of 2016 data PDs.

Before you request keeping them, please have a good look and some solid reasoning. Keeping datasets just for funsies won't cut it anymore, unfortunately.

Also: datasets can clearly be retrieved back at a later point if that were necessary.

Best,
-m

/BTagCSV/Run2016*-PromptReco-v*/MINIAOD
/BTagMu/Run2016*-PromptReco-v*/MINIAOD
/Charmonium/Run2016*-PromptReco-v*/MINIAOD
/Commissioning/Run2016*-PromptReco-v*/MINIAOD
/DisplacedJet/Run2016*-PromptReco-v*/MINIAOD
/EmptyBX/Run2016*-PromptReco-v*/MINIAOD
/FSQJets/Run2016*-PromptReco-v*/MINIAOD
/HINCaloJets/Run2016*-PromptReco-v*/MINIAOD
/HINPFJets/Run2016*-PromptReco-v*/MINIAOD
/HINPhoton/Run2016*-PromptReco-v*/MINIAOD
/HLTPhysics/Run2016*-PromptReco-v*/MINIAOD
/HLTPhysicsBunchTrains/Run2016*-PromptReco-v*/MINIAOD
/HLTPhysicsIsolatedBunch/Run2016*-PromptReco-v*/MINIAOD
/HcalHPDNoise/Run2016*-PromptReco-v*/MINIAOD
/HcalNZS/Run2016*-PromptReco-v*/MINIAOD
/HighMultiplicityEOF/Run2016*-PromptReco-v*/MINIAOD
/L1MinimumBia*/Run2016*-PromptReco-v*/MINIAOD
/MinimumBias/Run2016*-PromptReco-v*/MINIAOD
/MuOnia/Run2016*-PromptReco-v*/MINIAOD
/NoBPTX/Run2016*-PromptReco-v*/MINIAOD
/ParkingScoutingMonitor/Run2016*-PromptReco-v*/MINIAOD
/ZeroBia*/Run2016*-PromptReco-v*/MINIAOD

Answer 17 · 2017-03-21T09:31:13.000Z

Hi Marc,

Can you transfer the following datasets to CERN:

/Tau/Run2016B-03Feb2017_ver2-v2/MINIAOD
/Tau/Run2016E-03Feb2017-v1/MINIAOD
/Tau/Run2016G-03Feb2017-v1/MINIAOD

Thanks!
Jan

Answer 18 · 2017-03-21T16:16:32.000Z

Hi Jan, request is made: https://cmsweb.cern.ch/phedex/prod/Request::View?request=967091 Best, -m On 21 Mar 2017, at 10:31, Jan Steggemann <notifications@github.com<mailto:notifications@github.com>> wrote: Hi Marc, Can you transfer the following datasets to CERN: /Tau/Run2016B-03Feb2017_ver2-v2/MINIAOD /Tau/Run2016E-03Feb2017-v1/MINIAOD /Tau/Run2016G-03Feb2017-v1/MINIAOD Thanks! Jan — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#666 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AEdevKmD46V1W99CvdHtXT_KDV-aVMatks5rn5jhgaJpZM4JU-5I>.

Answer 19 · 2017-09-21T14:21:01.000Z

@mdunser
Can you please transfer this dataset /DoubleMuon/Run2016E-03Feb2017-v1/MINIAOD
Maria

Answer 20 · 2017-09-25T08:19:22.000Z

sorry for being a bit late on this, request is here: https://cmsweb.cern.ch/phedex/prod/Request::View?request=1113675 -m On 21 Sep 2017, at 16:21, mariadalfonso <notifications@github.com<mailto:notifications@github.com>> wrote: @mdunser<https://github.com/mdunser> Can you please transfer this dataset /DoubleMuon/Run2016E-03Feb2017-v1/MINIAOD Maria — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#666 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AEdevJOEynsZ5x0nwt31dSQIHBq1mu-iks5sknDTgaJpZM4JU-5I>.

Answer 21 · 2017-10-13T09:16:35.000Z

@mdunser

can you please transfer , with high priority,
/DoubleEG/Run2016D-03Feb2017-v1/MINIAOD
/DoubleMuon/Run2016H-03Feb2017_ver3-v1/MINIAOD
/SinglePhoton/Run2016H-03Feb2017_ver3-v1/MINIAOD

Thanks

Answer 22 · 2017-10-16T16:51:21.000Z

here’s the request: https://cmsweb.cern.ch/phedex/prod/Request::View?request=1132385 -m On 13 Oct 2017, at 11:16, mariadalfonso <notifications@github.com<mailto:notifications@github.com>> wrote: @mdunser<https://github.com/mdunser> can you please transfer , with high priority, /DoubleEG/Run2016D-03Feb2017-v1/MINIAOD /DoubleMuon/Run2016H-03Feb2017_ver3-v1/MINIAOD /SinglePhoton/Run2016H-03Feb2017_ver3-v1/MINIAOD Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#666 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AEdevDSTDNlwRa0GvaeEpWkvpvYywu2iks5sryp1gaJpZM4JU-5I>.