PHP Warning: fopen(): failed to open stream: "Quota exceeded"
juanluisbaptiste opened this issue · 8 comments
Hi Dave,
I'm having a very strange error with some wordpress sites I'm running using your wordpress image, but I think if there's an issue it lies on php-fpm, so if I'm wrong my apologies and I will move this to the wordpress image.
On some sites I'm getting the following error:
NOTICE: PHP message: PHP Warning: fopen(/www/wordpress/wp-content/languages/plugins/wordpress-seo-es_ES.mo): failed to open stream: "Quota exceeded" in /www/wordpress/wp-includes/pomo/streams.php on line 148
Because of it nginx sometimes fails with a 502 bad gateway error. It happens with many files, not just that one from the quoted error.
I cannot find anything on this, all I can find is about "Disk quota exceeded", but this is just "Quota exceeded". The error only happens with this image, I have other php sites running on the same nodes that use another php+apache (no php-fpm) image and the error is not present.
Could it be php-fpm parameters ? this is my config that is shared among containers:
FPM_MAX_CHILDREN=120
FPM_MAX_REQUESTS=1000
PHP_MEMORY_LIMIT=512M
Those are low traffic sites if that's important. This is the first time that I bump into this error. Any hints would be greatly appreciated.
HI there, wow that is a new one to me too. I seem to recall you using NFS for some of your work, could that be related? If it is the image that is doing it I would make a guess that it would be Nginx that would be the actual cause of it with its heavy caching. The closest thing I can think of right now would be to try NGINX_ENABLE_SITE_OPTIMIZATIONS=FALSE - I'd also wonder what the Wordpress Plugin is actually doing to get to that response, as I feel it could be OS level. Perhaps Open File Limits inside the container?
Yes, site content is stored on an aws EFS shared filesystem, not sure if this is as I can manually read and create files on it. I did a test and started a copy of one of the sites and nginx failed to start with the same error:
** [php-fpm] Starting php-fpm 7.0.33
nginx: [emerg] open() "/www/wordpress/.htaccess.nginx" failed (122: Quota exceeded) in /etc/nginx/conf.d/default.conf:28
** [nginx] Starting nginx 1.17.6
nginx: [emerg] open() "/www/wordpress/.htaccess.nginx" failed (122: Quota exceeded) in /etc/nginx/conf.d/default.conf:28
[04-May-2020 20:01:42] ERROR: failed to open error_log (/www/logs/php-fpm/php-fpm.log): Quota exceeded (122)
[04-May-2020 20:01:42] ERROR: failed to post process the configuration
[04-May-2020 20:01:42] ERROR: FPM initialization failed
** [php-fpm] Starting php-fpm
[04-May-2020 20:01:42] ERROR: failed to open error_log (/www/logs/php-fpm/php-fpm.log): Quota exceeded (122)
[04-May-2020 20:01:42] ERROR: failed to post process the configuration
[04-May-2020 20:01:42] ERROR: FPM initialization failed
So it seems the error comes from the OS ?
Yes it looks like it. Hard to say if its a data transfer error, or a size quota - Have you checked the size of your volumes by chance? While not entirely verbose, Quota could been alot of things, however I tend to default around transfer speeds personally.
I just noticed that the error is only with one of the nodes on the swarm cluster, because all of the sites with the error have one thing in common: one of the two containers they run are on the same node.
Volumes are bind volumes that mount a directory on the EFS filesystem, on EFS there is no size quota as it is elastic and will grow infinitely, and if there was any kind of quota issue with EFS then all other nodes would be having the same issue I think. What transfer speeds you mean ? how would you check them ?
Anyway I opened a case with AWS because I'm completely lost with this error, if you look it up on google the only link you will find is my own question on stackoverflow from last night hehe.
My thoughts on transfer quota is that perhaps we were creating too many temporary files (PHP Sessions perhaps/Temporary Cache files?) and it was saying that it couldn't handle it. Also on Transfer Quota, meaning that they would only allow a certain amount of files (whether it be actual files, or sizes inside the files) before throwing that error. I don't have much experience with AWS other than Lightsail, but what I do know is that they are very lenient on restrictions and are more than happy to give you what they need as it means more money in their pocket :)
I'll do some digging around, but am truly stumped.
Any news on this one? I've asked around and people seem aligned with my thoughts that it is a filesystem issue. My curiousity is piqued.
Not yet, but it seems to be EFS related, in the aws case it was escalated to the EFS team because its not clear which quota limit is being exceeded. I'll update when I know more...
At the end we were not able to find the exact cause of this but it seems it was related to an AWS EFS limit being surpassed, bu I was not able to find the cause.
Closing this as it was not related to this image.