Charset issue
jakobeissler opened this issue · 34 comments
Hello, I have a Symfony 3 project with iso charset and getting strange charater problems
Is ther any chance to change to iso?
@jakobeissler Can you give me more informations such as your Symfony verson, your PHP version and your bundle version. Additionally, what is you page encoding? UTF-8? And What is the internal encoding used by PHP in you configuration (php.ini)?
Sure:
- Symfony version: 3.1.8
- PHP version: 5.6
- Bundle version: 4.0
- Page encoding ISO-8859-1
php.ini encoding by default is UTF-8
Hum Isn't it the issue? PHP generated UTF-8 chars whereas you specify it as ISO-8859-1 in your page. Why are you doing such strategy?
Because our Oracle database is encoded with ISO-8859-1. Shouldn't ckeditor assume HTMLs charset?
I have never played with CKEditor charset... (UTF-8 everywhere...). IMO, it should rely on your page encoding but I'm not a CKEditor expert.. Maybe there is an option in order to specify the encoding to the CKEditor. I don't know need more research...
This issue (https://dev.ckeditor.com/ticket/9755) makes me feel we can maybe do something... According to this issue, what we can try is to put a charset utf-8 on the each script tags. Update here in your vendor by adding charset="utf-8" on all script tags and give a try. Then, come back here and give me some feedbacks :) If it works, instead of hardcoding the charset, we will see how we can make it more developer friendly...
I tried but it didn't work.
I have also played with this config options... no result:
'entities' => true,
'entities_latin' => true,
'basicEntities' => true,
I need to set up a fork of the Symfony SE & give it a try...
OK, just one thing, i've the following function in AppKernel.php because i need to force the charset for my entire site to iso-8859-1:
public function getCharset()
{
return 'ISO-8859-1';
}
If i remove that function or change it to UTF-8, ckeditor works as expected.
Ok thanks for this information. Anything else I should know before give it a try? Do we agree you put <meta charset="ISO-8859-1"> in the head of your page? As well as PHP uses UTF-8 internally?
Maybe stupid but do you try to also move PHP to ISO-8859-1?
Yes that is correct. Thank you.
@jakobeissler So... after almost one hour of debug, i think I got it... The issue seems coming from how CKEditor loads its translation files.
I notice that all broken texts comes from a translation file loaded by CKEditor itself and since the page uses ISO-8895-1 as charset, I think the browser directly interprets this file with this encoding whereas the file is encoded using UTF-8...
I have updated my nginx conf by forcing the UTF-8 encoding on this specific files (using the charset option on the entire CKEditor directory) and then everything is working fine.
The idea is to have Content-Type: content-type; charset=UTF-8 in the response if you use a different server.
This is not elegant but since CKEditor does not provide an option in order to configure the charset and we can't control how it loads its translation files, I don't see an other solution...
That is interesting. Just a quick question, where are the language files loaded from? I tried to edit manually the file vendor/egeloen/ckeditor-bundle/Resources/public/lang/ but no change, it seems they are loaded from a different location.
For a different approach i could try to override the language file with correct values but i can't figure out from where they are loaded.
@jakobeissler I can't find where it loads it since the javascript is minnified... but IMO, you have copied your file in the web directory with bin/console assets:install and so, updating the file in the vendor is not enought, you need to re-install assets in the web directory.
Futhermore editing file in the vendor is definitively not the way to go since when you will deploy your app, the vendor is downloaded from the source... If you wan to use your own version of CKEditor, the bundle supports it: http://symfony.com/doc/current/bundles/IvoryCKEditorBundle/usage/version.html
OK, i found out that the language file that is loaded is: web/bundles/ivoryckeditor/lang/es.js
If i open this file i see the text like this:
"checkbox":"Casilla de Verificación"
That doesn't look quite right does it?
Hum, this is weird when I open it using https://github.com/egeloen/IvoryCKEditorBundle/blob/master/Resources/public/lang/es.js it looks good.
I deleted the assets folder and installed again:
php bin/console assets:install web
Didn't work.
When you say, I open the file and see "checkbox":"Casilla de Verificación", where do you open it? in your browser? in your IDE?
You are right, it is the IDE (Netbeans), opening the file in browser displays the language file ok.
OK, so your IDE detects the file as ISO-8895-1... You have do something manually to get this behavior? My IDE (PHPStorm) detects the file as UTF-8 automatically.
An other alternative is to use dos2unix with the -iso option in order to convert all CKEDitor files using ISO-8895-1.
Something like find ckeditor-path -type f -exec dos2unix -iso {} \;
I set the charset in projects properties. When i deploy the symfony project to production server (linux) it doesn't work neither so has nothing to do with the IDE. I set up another ckeditor without bundle and it is working ok. I give up.
Never give up :) Maybe it is related on the way I update CKEditor, where do you download CKEditor? In the bundle, I rely on this custom script: https://github.com/egeloen/IvoryCKEditorBundle/blob/master/Resources/bin/ckeditor-sync.sh
It came with a template I bought (metronic). Ckeditor version is 4.4.5
Can you run my script and remove the last line (dos2unix) and re-install assets in the web directory?
Didn't work
NO!!! wait, it DID work... I just had to clear the browsers cache.
So, what does work? Removing the dos2unix line? or it works by default? or something else?
I executed the .sh script without the last line, installed assets and it is working on my localhost and on production server. It has definitely to do with:
find $ckeditorPath -type f -exec dos2unix {} \;
Ok, so there is something to fix. I can't remember why I needed to convert Windowd EOL to Unix ones... Need to search through the git history... Hopefully, you don't give up :)
Ok, many many thanks for the help. Please let me know when you tag a new version with the fix so I can install with composer.
This is a great and very neccesary bundle.
Here we are, your issue will be fixed by the referenced PR :)
@jakobeissler I just release 5.0.1
Thank you, it's working!!!