UTF-8 Output problem

Your page is not output as UTF-8. If you look at the characters you quoted, you will see that they are UTF-8 sequences rendered as something else. For example, the second character is E3 81 AF, which should be rendered as は, but instead is rendered as individual bytes. In FireFox, go to Tools > Page Info to see the actual encoding.

Do you have a sample I can look at? You can PM me the link if you'd rather not post it.

This definitely seems like a database/backend script problem. I had a similar problem with MySQL before I upgraded the ODBC driver I was using. What appears to be happening is that your default table character set is most likely ISO-8859-3 (Latin-3). Check MySQl manual how to verify/change the character set on a particular column. When these characters are returned to your server-side script, they are incorrectly converted into UTF-8 by simply taking their byte values. That is, the first two characters appear to be おは, which is E3 81 8A E3 81 AF in UTF-8. However, looking at your post, the sequence contains Š, which is 8A in Latin-3. Converting Š to 8A yields correct UTF-8.

Andre

What I meant is that your database driver may interpret encoding incorrectly, even if it's set correctly in the database. However, since you are saying that other parts of the site using the same database driver work fine, it's something else. FCKeditor shouldn't be the problem here because it's the browser that reads characters and FCKeditor gets the chance to see the characters after they are converted from UTF-8 to UCS-2, which is what the JavaScript using.

Disable FCKeditor on this page and just try to render HTML directly into a textarea. This will show you that it's a server-side problem.

You need to make sure data is not coming in mangled. Use a debugging proxy, like Charles (http://www.charlesproxy.com/) to see what exactly the browser receives. If it's well-formed UTF-8, then it's something in the browser. When working with the proxy, never mind the characters it displays - some of them fail to pick up fonts correctly. Look at the bytes instead.

Or try fiddler for IE. It's free. I've been able to tackle down some bugs with it. It offers a great HEX view for the request and response.

FCKeditor, being a JS app, can't really cause this. It could only be the character set of one of the documents that host the editor. Capturing HTTP traffic with a debugging proxy will tell you if the characters are coming in properly-encoded (you should see UTF-8 characters in the capture). The next thing you can do is to install DOM Inspector in FireFox and point to the #document objects on the way from the editor (click on the finder tool at the top-left corner to find) to your hosting page. In the right-hand pane of the DOM Inspector, you will see a bunch of properties for each document. Find characterSet and make sure it's UTF-8.

I have the same problem.

All application is utf-8 encoded.

Saving is OK, since I can see the stored data being returned through a DB query.

When going back to fckeditor, text is replaced by a lot of rubish. I tried IE7 FF3/Chrome: same thing.

Have you found a fix for this?

UPDATE: I found how to fix this, change fckconfig.js
FCKConfig.ProcessNumericEntities = true ;

Do you have a link to a test page?

claudounet wrote:UPDATE: I found how to fix this, change fckconfig.js
FCKConfig.ProcessNumericEntities = true ;

Bad idea. Instead of actual Unicode characters like these:

一 丁 丂 七 丄 丅 丆

you will be storing this in the database:

一 丁 丂 七 丄 丅 丆

Try searching for words or patterns in a database like this, not to mention 8 bytes per character that would take 3 if stored in UTF-8.

I have problem with installing FCK editor to my Russian website http://chans.in.ua
And everything is ok with my english website http://www.moneymakersportal.com
What's wrong with russian language?

unifinbroker wrote:I have problem with installing FCK editor to my Russian website http://chans.in.ua
And everything is ok with my english website http://www.moneymakersportal.com
What's wrong with russian language?

You are right A.M. regarding extra storage and search. The suggestion above works as a work around. We are investigating several areas including ODBC connector and IIS.

Just make sure you understand all character conversions. That is, you receive posted text as UTF-8 (on the wire), but ASP converts it for you to UCS-2, which is what VB/JS scripts use. When you send it to the database, make sure to use a Unicode-compatible DB driver. For example, the old MySQL ODBC driver (v3.51 I think, but may be forgetting the actual version) would mangle UCS-2 text, so make sure to use most recent one. When database receives it, it will convert whatever the driver sent to your specified storage, usually UTF-8, as it takes less storage and MySQL used to have limitations around UCS-2. On the way back from the database, the sequence is similar. Just make sure that you specify proper codepage in ASP (should be 65001) and content type in the response.

Andre