Tmp file not found on Heroku
zerolab opened this issue · 8 comments
While attempting to run this on Heroku, we got:
FileNotFoundError /admin/redirect-importer/import/
error[Errno 2] No such file or directory: '/tmp/tmp4i9b_poi'
In the end, resorted to importing via the management command via a temporary Heroku CLI container
Hi @zerolab and thanks for the bug report! Where in the flow did you get this error and did you try with different file formats and different file sizes?
Reading through the docs about ephemeral filesystem dynamos use I couldn't find anything that could cause this issue, might need to spin up a dynamo myself and take a peak. To be continued. (And any help is of course much appreciated).
Awesome that the CLI worked :)
Did not get a chance to dive further into this as it was right before a go-live.
As far as I understand it, got to the import page, select CSV to import, submit the form > 500 (FileNotFound)
Opened the issue because I could not find anything in the Heroku docs that would indicate what the issue is. Will try to reproduce on a staging environment and (hopefully) come back with more info.
Thank you for the prompt responses!
Awesome, let me know how it goes! I will do a series of updates to this library in the upcoming weeks and see this critical bug as something we need to address :)
Hi again @zerolab, I have started to research the Heroku issue as well, but are not able to recreate.
This is what I used when verifying:
- A copy of the bakery demo (along with wagtail-redirect-importer) (https://github.com/marteinn/wagtail-redirect-importer-heroku-test)
- I adjusted the deploy to Heroku to use my template
- Ran installation
- Logged in on https://wagtail-redirect-importer-test.herokuapp.com (just ask me if you want to login and poke around)
- Imported this example file: https://docs.google.com/spreadsheets/d/1jbuMgaQ6bNrmgCPOke__8qLKbpUDpvSA7ghMs9LEyNE/edit?usp=sharing
Any hint on what I can do to reproduce? Happy easter 🐰!
@zerolab Just wanted to follow up, did you get any further when researching this issue? Thanks!
Hey @marteinn,
Sorry for the delayed reply. Life got in the way..
I was unable to reproduce on a Heroku site with one web dyno.
We had the issue with on a production instance using 2 or 3 dynos, which leads me to believe the issue stems from the lack of dyno persistence. i.e. one request went to dyno1, the other to dyno 2, while the temporary file was on dyno one
Deployed the same to a test heroku app, which I then scaled to 2 x Pro dynos. and can confirm requests alternate between the two as, annoyingly, had to keep re-logging in.
Will see if I can retry on the same live system we experienced it first hand.
Hi @zerolab! Awesome research, I totally agree, it definitely sounds like a load balancing issue. I will look into recreating the issue myself on Heroku. Some of the inspiration in how I deal with tmp files comes from https://github.com/django-import-export/django-import-export, so will probably include this library in my testing.
I discovered a similar issue with django-import-export (more details django-import-export/django-import-export#130). I will investigate their solution of using a different storage engine for horizontally scaled apps.
For anyone coming to this issue, work is taking place here: wagtail/wagtail#5949