reformat_GSOD() runs infinitely with 0 activity

Question

reformat_GSOD() runs infinitely with 0 activity

taraskaduk opened this issue 4 years ago · 8 comments

So, this is a strange issue which I'm not sure how to diagnose.
reformat_GSOD() does not behave as expected. It runs, but with no end, no output, no and CPU/GPU activity shown on the activity monitor by Mac. It's like nothing is happening at all.

I narrowed everything down to one single file, but nothing.

Extra info: this is the first attempt to run the function after a clean install of R, RStudio, and all packages on a new MBP with M1 chip. Not sure if this is relevant at all

─ Session info ──────────────
setting value
version R version 4.0.3 (2020-10-10)
os macOS Big Sur 10.16
system x86_64, darwin17.0
ui RStudio
language (EN)
collate en_US.UTF-8
ctype en_US.UTF-8
tz America/New_York
date 2021-01-11

GSODR * 2.1.2.9000 2021-01-11 [1] Github (63b1ea8)

library(GSODR)

list.files("data/gsod/2010/")
[1] "02431099999.csv"

data <- reformat_GSOD(dsn = "data/gsod/2010")

Answer 1 · 2021-01-12T00:52:38.000Z

I'm aware. I found this issue some time ago. Unfortunately, there's no easy solution I can find yet.

I managed to pinpoint the function, it's where I use future.apply::future_lapply() to reformat the CSV files.

Also, I've just moved, so when I found it I was packing. Now I'm still unpacking and trying to get settle so I don't have much time to devote to this right now.

The quickest fix is to just insert lapply() in for future_lapply() and run single-threaded.

Answer 2 · 2021-01-12T00:56:41.000Z

I'm intrigued. Is it only this one file that causes issues for you?

Answer 3 · 2021-01-12T00:58:11.000Z

Sorry for confusion. No, not one file. I just stripped everything down to one file so that I can't blame it on anything else. Even on simply one file, it doesn't work.

Answer 4 · 2021-01-12T00:59:14.000Z

OK, clear. I thought maybe other files were working so there was a clue in the files themselves.

Answer 5 · 2021-01-12T01:09:31.000Z

I found your issue over at future.apply at HenrikBengtsson/future.apply#74.
At least I know I'm not crazy now.

Answer 6 · 2021-01-21T03:54:31.000Z

I've reverted to using just the base lapply() for now to fix this. Parallel functionality has been disabled see: 81ad3fb, it may come back in the future.

Answer 7 · 2021-01-21T16:20:20.000Z

Thanks for the update. FWIW, I wonder if the issue is really on the future.apply side... All things held constant, a swap for lapply shows the code works...
Thank you for your work on this, and sorry I couldn't have been a better help

Answer 8 · 2021-01-22T02:55:02.000Z

I have no idea where the issue is, whether it's data.table or future.apply or something else. It happened around the time that both were updated late last year. I tried a few times to install older versions of the packages but couldn't seem to find the culprit. Once my life has settled a bit I'll try to revisit this.