API errors when sending utf-8 data (emoji) POST inspection/store

[crunchie84]

Browser: Chrome (Chromebook) - Version 90.0.4430.218 (Official Build) (64-bit)

Reproduction scenario:

• Add new inspection to a beehive. I only filled in the note field with "testing support of emojis 🤞 "
• Save inspection

Expected behaviour:

• inspection should be saved

Actual behaviour:

• CORS error for POST request after the OPTIONS was successfull:

Access to XMLHttpRequest at 'https://api.beep.nl/api/inspections/store' from origin 'https://app.beep.nl' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Possible cause:

https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSMissingAllowOrigin

[crunchie84]

Somehow the required headers which were added to the OPTIONS request were not available on the POST request:

Access-Control-Allow-Headers: AUTHORIZATION,CONTENT-TYPE
Access-Control-Allow-Methods: POST
Access-Control-Allow-Origin: https://app.beep.nl

OPTIONS response:

vs POST response:

[crunchie84]

When removing the emoji the response is 201 created and the CORS headers are OK

[crunchie84]

request body which breaks:

{"impression":-1,"attention":-1,"reminder":"","reminder_date":"","notes":"Some note with an emoji: 🤞 works?","date":"2021-06-06 22:47","items":{},"valid":true,"unfilled_required_item_names":[],"checklist_id":4260,"hive_id":"31553"}

[crunchie84]

https://dev.mysql.com/doc/refman/5.7/en/charset.html#:~:text=The%20default%20MySQL%20server%20character,column%2C%20and%20string%20literal%20levels.

MySQL includes character set support that enables you to store data using a variety of character sets and perform comparisons according to a variety of collations. The default MySQL server character set and collation are latin1 and latin1_swedish_ci, but you can specify character sets at the server, database, table, column, and string literal levels.

• Looking at the docker-compose.yaml no diverging from this default has been made (https://github.com/beepnl/BEEP/blob/master/docker-compose.yaml)
• Searching in this codebase for utf returns no database-related results

My working hypothesis is that the Inspection::create (https://github.com/beepnl/BEEP/blob/master/app/Inspection.php#L119) throws an error from the database because it is not able to store an emoji (which is utf-8 encoded) in a text field with character encoding latin1. This error then causes the middlewares to fail and not wrap the response with the correct CORS headers; effectively hiding the actual error.

[crunchie84]

using cURL to ignore the CORS we indeed get a 500 error:

BEARERTOKEN=__LOGIN_YOURSELF_AND_GET_THIS
curl 'https://api.beep.nl/api/inspections/store' \
  -H 'Connection: keep-alive' \
  -H 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"' \
  -H 'DNT: 1' \
  -H 'Accept-Language: nl' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H "Authorization: Bearer $BEARERTOKEN" \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/plain, */*' \
  -H 'User-Agent: Mozilla/5.0 (X11; CrOS x86_64 13816.82.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.218 Safari/537.36' \
  -H 'Origin: https://app.beep.nl' \
  -H 'Sec-Fetch-Site: same-site' \
  -H 'Sec-Fetch-Mode: cors' \
  -H 'Sec-Fetch-Dest: empty' \
  -H 'Referer: https://app.beep.nl/' \
  --data-raw '{"impression":-1,"attention":-1,"reminder":"","reminder_date":"","notes":"derp 🤞","date":"2021-06-06 23:03","items":{},"valid":true,"unfilled_required_item_names":[],"checklist_id":4260,"hive_id":"31553"}' \
  --compressed

response:

{
    "message": "Server Error"
}

[pvgennip]

Dear @crunchie84 thanks for the explicit explanation, we will look into this. The inspections table of our live database has UTF-8 Unicode encoded fields for the notes:

[pvgennip]

Fixed by converting all Utf8 columns to Utf8mb4.
@crunchie84 😊

[crunchie84]

yay! Can finally note that my 👑 🐝 rocks💪 ! Thanks!