I’ve got a CSV file with mixed up encoded strings:
1956;Mathé Altéry;Le Temps Perdu
1963;Alain Barrière;Elle était si jolie
2024;Gérard truc;Mon éden
2024;GÃ¥te;Ulveham
2023;Gergő Rácz;Zé Szabó
2210;Anna Mjöll;SjúbÃdú
2200;Ovidijus Vyšniauskas;Lopšinė mylimai
Some accents on french names are either UTF8 encoded (é ; è ) or are encoded in plain text (‘é’ ASCII ?). Also some characters (‘ő’) don’t exist with UTF8 but with Windows-1252.
What I’ve done is converting everyting from UTF8 to Windows-1252:
mb_convert_encoding($text, 'Windows-1252','UTF-8');
which works like a charm, .. except for french accents that are all detected as UTF8.
Here is sample code:
$csv = array("1956;Mathé Altéry;Le Temps Perdu",
"1963;Alain Barrière;Elle était si jolie",
"2024;Gérard truc;Mon éden",
"2024;GÃ¥te;Ulveham",
"2023;Gergő Rácz;Zé Szabó",
"2210;Anna Mjöll;SjúbÃdú",
"2200;Ovidijus Vyšniauskas;Lopšinė mylimai");
foreach ($csv as $row) {
echo "row=$rown";
$row_values = explode(';', $row);
foreach ($row_values as $value) {
echo sprintf("value=%-30s encoding=%-10s W1252=%-20sn",
$value,
mb_detect_encoding($value),
mb_convert_encoding($value, 'Windows-1252' ,'UTF-8'));
}
echo "n";
}
# php ./encodings.php
row=1956;Mathé Altéry;Le Temps Perdu
value=1956 encoding=ASCII W1252=1956
value=Mathé Altéry encoding=UTF-8 W1252=Mathé Altéry
value=Le Temps Perdu encoding=ASCII W1252=Le Temps Perdu
row=1963;Alain Barrière;Elle était si jolie
value=1963 encoding=ASCII W1252=1963
value=Alain Barrière encoding=UTF-8 W1252=Alain Barrière
value=Elle était si jolie encoding=UTF-8 W1252=Elle était si jolie
row=2024;Gérard truc;Mon éden
value=2024 encoding=ASCII W1252=2024
value=Gérard truc encoding=UTF-8 W1252=G▒rard truc
value=Mon éden encoding=UTF-8 W1252=Mon ▒den
row=2024;GÃ¥te;Ulveham
value=2024 encoding=ASCII W1252=2024
value=GÃ¥te encoding=UTF-8 W1252=Gåte
value=Ulveham encoding=ASCII W1252=Ulveham
row=2023;Gergő Rácz;Zé Szabó
value=2023 encoding=ASCII W1252=2023
value=GergÅ‘ Rácz encoding=UTF-8 W1252=Gergő Rácz
value=Zé Szabó encoding=UTF-8 W1252=Zé Szabó
row=2210;Anna Mjöll;SjúbÃdú
value=2210 encoding=ASCII W1252=2210
value=Anna Mjöll encoding=UTF-8 W1252=Anna Mjöll
value=SjúbÃdú encoding=UTF-8 W1252=Sjúbídú
row=2200;Ovidijus Vyšniauskas;Lopšinė mylimai
value=2200 encoding=ASCII W1252=2200
value=Ovidijus VyÅ¡niauskas encoding=UTF-8 W1252=Ovidijus Vyšniauskas
value=LopÅ¡inÄ— mylimai encoding=UTF-8 W1252=Lopšinė mylimai
You can see that values having french accents are detected as UTF-8 and converting them to W1252 adds weird characters meaning a wrong conversion.
Now, I can’t figure out how to detect which strings need to be encoded, which don’t need to. mb_detect_encoding doesn’t seem to return reliable results.
Any idea ?
Many thanks all !!