BUG: UTF-8 pages not displaying correctly (suggested solution included)

[flyingdutchie]

There are various thread dealing with improper displaying of international characters (just search for UTF-8 on the support forums).

1. Smugmug does correctly generate pages with this meta tag:
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />
2. However, the HTTP response-header "Content-type" is incorrect! When i do a 'graburl' to my site, i get this http-response:
HTTP/1.1 200 OK
Date: Sat, 08 Oct 2005 16:07:01 GMT
Server: Apache
X-Powered-By: smugmug/1.2.0
Set-Cookie: SMSESS=026e577c53333abbcfcfd9247f880068; path=/; domain=.smugmug.com
Cache-Control: private, max-age=1, must-revalidate
Pragma:
X-Extra: 0.26807594299316
ETag: sm-a54c5ea41ed6f8c1233cdecdc2405552-sm
Content-Length: 29046
Connection: close
Content-Type: text/html; charset=ISO-8859-1

The incorrect Content-Type header causes the browsers to select ISO-8859-1 (Western European (ISO)) instead. Can this be fixed?

The last line (in red) is incorrect. It should read:
Content-Type: text/html; charset=UTF-8


Related threads:
http://www.dgrin.com/showthread.php?p=175958#post175958
http://www.dgrin.com/showthread.php?t=18290

Addition: I just saw that Luke Church did suggest the same solution:
http://www.dgrin.com/showpost.php?p=162986&postcount=12

[Scala]

The people at smugmug must be well aware of all this, but here we go again:

In addition to the suggested fix above the content that was once saved as ISO-8859-1 (and is still saved as such if you type in text using the customize gallery screen, for example) will need to be converted to UTF-8. If not, my guess is that a lot of currently working international characters will end up displaying incorrectly.

Here's an example. Go to http://scala.smugmug.com and select UTF-8 as the content encoding. There will be characters in my bio text displaying incorrectly or not at all.

[flyingdutchie]

The people at smugmug must be well aware of all this, but here we go again:

In addition to the suggested fix above the content that was once saved as ISO-8859-1 (and is still saved as such if you type in text using the customize gallery screen, for example) will need to be converted to UTF-8. If not, my guess is that a lot of currently working international characters will end up displaying incorrectly.

Here's an example. Go to http://scala.smugmug.com (http://scala.smugmug.com/) and select UTF-8 as the content encoding. There will be characters in my bio text displaying incorrectly or not at all.
I see,
but Smugmug should choose one or the other, since the meta tag in the pages looks like this:
<meta http-equiv="content-type" content="text/html; charset=UTF-8" />

So, the http response-header tells the browser to use one encoding, the <meta> tag to use another... it is still a bug.
-- Anton.

[Scala]

So, the http response-header tells the browser to use one encoding, the <meta> tag to use another... it is still a bug.
-- Anton.
Indeed. And the fix is long overdue.

[{JT}]

So this should be fixed, would love feedback from someone. I don't think previous entries are fixed yet - some might be though.

Indeed. And the fix is long overdue.

[Scala]

So this should be fixed, would love feedback from someone. I don't think previous entries are fixed yet - some might be though.
I tried entering some Finnish and Vietnamese text in comments and gallery description and it seems to work. Thanks!

There's still the issue of the META tag implying UTF-8 but the browser selecting ISO-8859-1 but I guess that's how you want it. Either way there's something advanced going on because I thought ISO-8859-1 did not include all those strange Vietnamese characters :)

Previous entries are still wrong.

And the new "Updating..." notification does not go away if you delete the last (only) gallery comment. :):

The end user experience is a lot better now internationally, thanks again!

[rainforest1155]

So this should be fixed, would love feedback from someone. I don't think previous entries are fixed yet - some might be though.I successfully added a comment containing these characters:
á Á à À â Â é É è È ê Ê í Í ì Ì î Î ó Ó ò Ò ô Ô ú Ú ù Ù û Û
ä Ä ö Ö ü Ü ß

Just the €uro isn't working, yet.

The comments didn't change and are still wrong (here (http://rainforest1155.smugmug.com/gallery/706437/4/34687505/Medium#comment) - ö = ö and here (http://rainforest1155.smugmug.com/gallery/850584) ü = ü)

Sebastian

[Scala]

Just the €uro isn't working, yet.
I tried the €uro in a comment and it worked for me in IE6 and Firefox 1.0.7. But I agree that in theory it should not work since ISO-8859-1 doesn't have the euro character :)

[{JT}]

As longs as it works for now, I am happy :)

I tried the €uro in a comment and it worked for me in IE6 and Firefox 1.0.7. But I agree that in theory it should not work since ISO-8859-1 doesn't have the euro character :)

[rainforest1155]

As longs as it works for now, I am happy :)Yeah, works now for me, but before it clearly didn't. Did some finetuning, didn't we? Nice! :thumb

Sebastian

[flyingdutchie]

As longs as it works for now, I am happy :)
{JT}, as a fellow software engineer, I'm just wondering...
Why is it so tricky to send over UTF-8 instead of ISO-8859-1? Has it to do with the existing text-data in your databases?

The http-response headers still say it is ISO-8859-1 and the <META> tags declare it to be UTF-8.. still a contradiction here.

-- Anton

[{JT}]

Has it to do with the existing text-data in your databases?

Bingo.

[flyingdutchie]

Bingo.Aha... migration is the issue here...! :)

But still, what about the contradicting http-response header and <META> tag? Shouldn't the <META> tag specify the ISO-8859-1 charset as well?

[{JT}]

Maybe :) As I stated earlier - right now everything is working, so I am hesitant to change anyting :)

Aha... migration is the issue here...! :)

But still, what about the contradicting http-response header and <META> tag? Shouldn't the <META> tag specify the ISO-8859-1 charset as well?

[DmitryS]

Maybe :) As I stated earlier - right now everything is working, so I am hesitant to change anyting :) Russian characters works now too.
However:
1. When I change description of the gallery using "Edit" button, I can start new lines. If I go to the gallery desription, new line symbols can't be used.
2. The limit for number of non-english characters in the description is very small, about 25 symbols. For enlish - it's much larger.
Not a big deal, though

[rainforest1155]

Maybe :) As I stated earlier - right now everything is working, so I am hesitant to change anyting :)I did some more testing and found out that it depends where the €uro symbol stands for it to work or not. :scratch
This doesn't work:€ á
€ á

but that does:
€ á
€ á

EDIT: In fact I boiled it down to this - if there's at least one space before the €uro, it won't work. It only works when the €uro is the first character. If there are more €uros in the comment it depends on a €uro being the first character of the comment or not for all others!

Works:
€
Test123 €
Another testline that is fine €

but if you add a space before the first €uro resulting in it not being the first character of the comment it's broken:
€
Test123 €
Another testline that is fine €

Any ideas on this?
Sebastian

[{JT}]

This happens to the euro only though, correct?

I might an idea as to why, but have to check first ...

I did some more testing and found out that it depends where the €uro symbol stands for it to work or not. :scratch
This doesn't work:€ á
€ á

but that does:
€ á
€ á

EDIT: In fact I boiled it down to this - if there's at least one space before the €uro, it won't work. It only works when the €uro is the first character. If there are more €uros in the comment it depends on a €uro being the first character of the comment or not for all others!

Works:
€
Test123 €
Another testline that is fine €

but if you add a space before the first €uro resulting in it not being the first character of the comment it's broken:
€
Test123 €
Another testline that is fine €

Any ideas on this?
Sebastian