Adding projects to new instance results in 413 and 422 errors
albancrommer opened this issue · 10 comments
Describe the bug
Nginx / rails are behaving strangely. Can't add new projects.
I know this is supposed to be an nginx bug (413) but the mix with 422 is confusing.
Maybe getting more logs from rails would help ?
To Reproduce
Steps to reproduce the behavior:
- Run fab-manager with docker-compose
- Add a new project
- Add a title and a description. cf. screenshot 1
- It fails with
error 422 "Unprocessable Entity"cf. screenshot 2 - Add more informations, like author, material, tags, ...
- Same 422 error.
- Add a valid SVG CAO file.
- It fails with
error 413 "Request Entity Too Large"cf. screenshot 3. Note: you MUST open the browser console to observe.
Expected behavior
It should succeed, or at least print errors in a meaningful way.
Screenshots
Screenshot 1
Screenshot 2
Screenshot 3
Server (please complete the following information):
- OS: Debian 11
- Kernel: Linux fabmanager 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux
- Fab-manager version : release-v5.7.0
Browser (please complete the following information):
- Name: Chromium Linux
- Version: Version 107.0.5304.87
Additional context
This is a standard, sh script deploy / docker-compose based install
I pulled the latest version, it didn't fix it. Before that I had the 2023-01-09T14:26:00.781041181Z / 28ee9e18f2dc not on the hub anymore.
Nginx client_max_body_size
I have checked the nginx configuration for the parameter related to 413 and even injected it in the nginx.conf.
This is the conf running nginx -Tlive in the nginx docker :
# configuration file /etc/nginx/nginx.conf:
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
client_max_body_size 0;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
}
# configuration file /etc/nginx/mime.types:
types {
text/html html htm shtml;
text/css css;
text/xml xml;
image/gif gif;
image/jpeg jpeg jpg;
application/javascript js;
application/atom+xml atom;
application/rss+xml rss;
text/mathml mml;
text/plain txt;
text/vnd.sun.j2me.app-descriptor jad;
text/vnd.wap.wml wml;
text/x-component htc;
image/avif avif;
image/png png;
image/svg+xml svg svgz;
image/tiff tif tiff;
image/vnd.wap.wbmp wbmp;
image/webp webp;
image/x-icon ico;
image/x-jng jng;
image/x-ms-bmp bmp;
font/woff woff;
font/woff2 woff2;
application/java-archive jar war ear;
application/json json;
application/mac-binhex40 hqx;
application/msword doc;
application/pdf pdf;
application/postscript ps eps ai;
application/rtf rtf;
application/vnd.apple.mpegurl m3u8;
application/vnd.google-earth.kml+xml kml;
application/vnd.google-earth.kmz kmz;
application/vnd.ms-excel xls;
application/vnd.ms-fontobject eot;
application/vnd.ms-powerpoint ppt;
application/vnd.oasis.opendocument.graphics odg;
application/vnd.oasis.opendocument.presentation odp;
application/vnd.oasis.opendocument.spreadsheet ods;
application/vnd.oasis.opendocument.text odt;
application/vnd.openxmlformats-officedocument.presentationml.presentation
pptx;
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
xlsx;
application/vnd.openxmlformats-officedocument.wordprocessingml.document
docx;
application/vnd.wap.wmlc wmlc;
application/wasm wasm;
application/x-7z-compressed 7z;
application/x-cocoa cco;
application/x-java-archive-diff jardiff;
application/x-java-jnlp-file jnlp;
application/x-makeself run;
application/x-perl pl pm;
application/x-pilot prc pdb;
application/x-rar-compressed rar;
application/x-redhat-package-manager rpm;
application/x-sea sea;
application/x-shockwave-flash swf;
application/x-stuffit sit;
application/x-tcl tcl tk;
application/x-x509-ca-cert der pem crt;
application/x-xpinstall xpi;
application/xhtml+xml xhtml;
application/xspf+xml xspf;
application/zip zip;
application/octet-stream bin exe dll;
application/octet-stream deb;
application/octet-stream dmg;
application/octet-stream iso img;
application/octet-stream msi msp msm;
audio/midi mid midi kar;
audio/mpeg mp3;
audio/ogg ogg;
audio/x-m4a m4a;
audio/x-realaudio ra;
video/3gpp 3gpp 3gp;
video/mp2t ts;
video/mp4 mp4;
video/mpeg mpeg mpg;
video/quicktime mov;
video/webm webm;
video/x-flv flv;
video/x-m4v m4v;
video/x-mng mng;
video/x-ms-asf asx asf;
video/x-ms-wmv wmv;
video/x-msvideo avi;
}
# configuration file /etc/nginx/conf.d/fabmanager.conf:
upstream puma {
server fabmanager:3000;
}
server {
listen 80;
server_name fabmanager.local;
root /usr/src/app/public;
location ^~ /packs/ {
gzip_static on;
expires max;
add_header Cache-Control public;
}
try_files $uri/index.html $uri @puma;
location @puma {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;
proxy_pass http://puma;
}
client_max_body_size 4G;
keepalive_timeout 10;
error_page 500 502 504 /500.html;
error_page 503 @503;
# Return a 503 error if the maintenance page exists.
if (-f /usr/src/app/public/maintenance.html) {
return 503;
}
location @503 {
# Serve static assets if found.
if (-f $request_filename) {
break;
}
# Set root to the shared directory.
root /usr/src/app/public/;
rewrite ^(.*)$ /maintenance.html break;
}
location /.well-known/acme-challenge {
root /etc/letsencrypt/webrootauth;
default_type "text/plain";
}
# no spam bot
if ($http_referer ~* (guardlink.org|free-share-buttons|social-buttons|buy-cheap-online.info|social-buttons.com|free-share-buttons.com|darodar.com|blackhatworth.com|hulfingtonpost.com|priceg.com|semalt.com|imaspammer.com|iedit.ilovevitaly.com|7makemoneyonline.com|iedit.ilovevitaly.com|7makemoneyonline.com|gamersyde.com|iloveitaly.com|econom.co|semalt.com|forum.topic44637676.darodar.com|darodar.com|iskalko.ru|ilovevitaly.ru|ilovevitaly.com|ilovevitaly.co|o-o-8-o-o.ru|o-o-6-o-o.ru|buttons-for-website.com|semalt.semalt.com|cenoval.ru|priceg.com|darodar.com|cenokos.ru|seoexperimenty.ru|gobongo.info|vodkoved.ru|adcash.com|websocial.me|cityadspix.com|luxup.ru|ykecwqlixx.ru|superiends.org|slftsdybbg.ru|edakgfvwql.ru|socialseet.ru|screentoolkit.com|econom.co|semalt.com|savetubevideo.com|shopping.ilovevitaly.com|iedit.ilovevitaly.com|forum.topic52548358.darodar.com|forum.topic53813291.darodar.com|share-buttons.com|event-tracking.com|success-seo.com|free-floating-buttons.com|get-free-social-traffic.com|chinese-amezon.com|get-free-traffic-now.com|free-social-buttons.com|videos-for-your-business.com)) { return 403; }
}
I guess the problem here is that the project image was missing during your first tries, so it result in 422 unprocessable entity. In your last try you added an image, so it was ok for the requirement but it exceed the maximum allowed size, so it resulted in 413 request entity too large. Can you check the provided image size doesn't exceed the maximum set in the MAX_IMAGE_SIZE variable (see /apps/fabmanager/config/env file)?
Anyway, I agree that the resulting error message doesn't help. We already have plans to improve that in the near future, as part of the react migration.
Content of the config env file :
root@fabmanager:/apps/fabmanager# grep MAX config/env
MAX_IMPORT_SIZE='5242880'
MAX_IMAGE_SIZE='10485760'
MAX_CAO_SIZE='20971520'
MAX_SUPPORTING_DOCUMENT_FILE_SIZE='5242880'
New try with the project image
The image weights 600Ko, is a standard JPEG. Still got the 422 unprocessable entity.
Hints / Questions
422 : Is is possible to get more logs from rails ? Like switch to "DEBUG" the logger level ?
413 : My guts tell me this is strictly an nginx problem, because all searches on that code result with nginx and client_max_body_size.
I saw that your nginx configuration file contains client_max_body_size 0; in the http {} section. Maybe it can be responsible for this? Otherwise, we set the limit to 4 Gb in the nginx fabmanager's configuration file so I hope the problem was not there... Which nginx version are you running, the default one?
You can adjust the log level by setting LOG_LEVEL=debug in config/env and restarting the app (docker-compose down && docker-compose up -d). Moreover, you may get more details in the browser's network inspector, by clicking on the request which terminates on a 422 error.
- The
client_max_body_size 0is defined in the official nginx documentation as the way to disable it
Setting size to 0 disables checking of client request body size.
- The log level was already on DEBUG, but I think I should see more informations. This is weird.
There was not so much more information in the 422 error requests in network inspector.
I just made some tests ... I cannot reproduce the issue when connected as a member or as an admin, when saving the project as draft or publishing it, with an image attached or with no image.
Very strange that 422 errors do not show any details in the resulting JSON because that's what the code does if saving the project fails. You're probably right about nginx handling the request and causing the issue but as long as I can't reproduce it I have no idea of what kind of configuration is causing the trouble... You can try bypassing nginx by adding the following directive in the docker-compose.yml file:
services:
fabmanager:
ports:
- "3000:3000"Then restart the service and try to access your instance using myhostname.com:3000 and see if the error still occurs. Note that you may have to set ALLOW_INSECURE_HTTP=true in config/env to be able to log in.
Sorry, issue #438 blocking my path for now, will try next.
Alright, some progress.
413 Errors: Solved
I forgot I had another nginx reverse proxy at the gateway of our infrastructure. Adding the client_max_body_size directive to that nginx got rid of the problem.
422 Errors
The log/production.log has some information about the error.
I, [2023-03-01T12:58:38.022390 #7] INFO -- : Started POST "/api/projects/" for 172.22.0.6 at 2023-03-01 12:58:38 +0000
I, [2023-03-01T12:58:38.023864 #7] INFO -- : Processing by API::ProjectsController#create as JSON
I, [2023-03-01T12:58:38.024030 #7] INFO -- : Parameters: {"project"=>{"name"=>"Story box", "project_image_attributes"=>{"attachment"=>#<ActionDispatch::Http::UploadedFile:0x00007f3ffd3d01a0 @tempfile=#<Tempfile:/tmp/RackMultipart20230301-7-blocfv.png>, @original_filename="1*YI3QuyzhfVCzKQeGLcY4Mw.png", @content_type="image/png", @headers="Content-Disposition: form-data; name=\"project[project_image_attributes][attachment]\"; filename=\"1*YI3QuyzhfVCzKQeGLcY4Mw.png\"\r\nContent-Type: image/png\r\n">}, "description"=>"<p>This is a projet</p>", "status_id"=>"", "component_ids"=>[""], "machine_ids"=>[""], "user_ids"=>[""], "licence_id"=>"", "theme_ids"=>[""], "tags"=>"", "state"=>"published"}, "authenticity_token"=>"7pE75FpLaIgjgVeBPiILMF4ShaSXIq/BSTUFhm1hqJMIFtFUY5Wlnh5duUtv4G2+tu8mNVQKShWKAc7lJ2j3OQ=="}
W, [2023-03-01T12:58:38.024433 #7] WARN -- : HTTP Origin header (https://<DOMAIN.NAME>) didn't match request.base_url (http://DOMAIN.NAME>)
I, [2023-03-01T12:58:38.024892 #7] INFO -- : Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
F, [2023-03-01T12:58:38.026506 #7] FATAL -- :
F, [2023-03-01T12:58:38.026572 #7] FATAL -- : ActionController::InvalidAuthenticityToken (ActionController::InvalidAuthenticityToken):
F, [2023-03-01T12:58:38.026631 #7] FATAL -- :
The error seems to come from a HTTP/HTTPS error in the form handler "forgery protection" filter.
Other forms work on the app, like machine creation. So far only the projects category does the 422.
My proxy conf goes like this :
- on GATEWAY server : Nginx/Letsencrypt Proxy. 443/HTTPS =>
- on FABMANAGER VM : Nginx Docker-compose. 80/HTTP =>
- on FABMANAGER VM : Rails Docker-compose. 3000/HTTP
The docker-compose env config file has the following settings:
DEFAULT_HOST=fabmanager.crommer.net
DEFAULT_PROTOCOL=https
Hi @sylvainbx sorry got delayed do you have any idea about what would be the reason / a way to test or fix it ?
Hi @albancrommer, I'm sorry I don't see anything I can do from FM's point of view as long as this issue is not reproductible on any of my setups. I guess this is related to your environment. Maybe your other nginx is lacking a directive to set the HTTP Origin header? For example, in our nginx config, we set proxy_set_header Host $http_host;, maybe you have to do something similar on your other nginx...
Hey @sylvainbx, thanks a lot for your feedback!
I managed to solve the problem by adding nginx headers, as you recommanded :)
I let you close the issue.
Solution
Adding any of these configs to the nginx vhost worked.
- Add an
Origin
proxy_set_header Origin $scheme$host;- Add headers according to this Rails Framework issue comment
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Forwarded-Port 443;
proxy_set_header X-Forwarded-Host $host;The Host header was already there on the first proxy, tried to add X-Forwarded-For, X-Forwarded-Proto, X-Forwarded-Server but didn't work. Tried Origin, it worked, and then the 3 issues comment fields, and they worked too.
The conf was added on the SSL/TLS termination proxy, but it should work on others.