[Commits] [wesnoth/wesnoth] b165f6: Improve UTF-8 handling
GitHub
noreply at github.com
Sun Mar 23 17:47:55 UTC 2014
Branch: refs/heads/1.12
Home: https://github.com/wesnoth/wesnoth
Commit: b165f6e25e8044d10c0e354b7301adcc9dedd4fa
https://github.com/wesnoth/wesnoth/commit/b165f6e25e8044d10c0e354b7301adcc9dedd4fa
Author: aquileia <sk.aquileia at gmail.com>
Date: 2014-03-23 (Sun, 23 Mar 2014)
Changed paths:
M src/dialogs.cpp
M src/gui/dialogs/addon_list.cpp
M src/gui/widgets/password_box.cpp
M src/gui/widgets/text.cpp
M src/gui/widgets/text_box.cpp
M src/serialization/string_utils.cpp
M src/serialization/string_utils.hpp
M src/tests/test_serialization.cpp
M src/text.cpp
Log Message:
-----------
Improve UTF-8 handling
For manipulation of UTF-8 encoded strings (insert, delete, truncate, and
size), usually conversion to wstring and back was necessary. This has
two disadvantages:
* the size of wstring is implementation defined
* the
double conversion is slow
The purpose of this patch is to add functions
to handle these tasks by a layer handling the different indexing between
codepoints and characters.
This commit applies these functions in
several places to replace the old implementations.
The byte_size_from_utf8_first() implementation will be replaced in a
separate commit as it relies on the count_leading_ones function by 8680.
Signed-off-by: aquileia <sk.aquileia at gmail.com>
Commit: 1b71bf0b781787e0e5f175c6ee3a08aa035803f3
https://github.com/wesnoth/wesnoth/commit/1b71bf0b781787e0e5f175c6ee3a08aa035803f3
Author: aquileia <sk.aquileia at gmail.com>
Date: 2014-03-23 (Sun, 23 Mar 2014)
Changed paths:
M src/gui/widgets/text_box.cpp
Log Message:
-----------
Fix a wrongly passed parameter in text_box.cpp
Commit: 61fabaf294c92b64bb0e3e1f2d27552c2b9f0a65
https://github.com/wesnoth/wesnoth/commit/61fabaf294c92b64bb0e3e1f2d27552c2b9f0a65
Author: aquileia <sk.aquileia at gmail.com>
Date: 2014-03-23 (Sun, 23 Mar 2014)
Changed paths:
M src/tests/test_serialization.cpp
Log Message:
-----------
fix signature and test for UTF-8 functions
On the advise of 8680, the unicode string was replaced by the hex
transcription of the corresponding codepoints (which he kindly
provided).
This might help to pass the overzealous encoding check of clang.
This commit also fixes a discrepancy between the signature of u8index in
declaration and definition.
Conflicts:
src/serialization/string_utils.cpp
src/tests/test_serialization.cpp
Commit: 610dc87c314aae957cebf7d60d9bb231b2150609
https://github.com/wesnoth/wesnoth/commit/610dc87c314aae957cebf7d60d9bb231b2150609
Author: Alexander van Gessel <ai0867 at gmail.com>
Date: 2014-03-23 (Sun, 23 Mar 2014)
Changed paths:
M src/addon/manager_ui.cpp
M src/construct_dialog.cpp
M src/construct_dialog.hpp
M src/desktop_util.cpp
M src/help.cpp
M src/hotkey/hotkey_item.cpp
M src/marked-up_text.cpp
M src/marked-up_text.hpp
M src/race.cpp
M src/race.hpp
M src/serialization/string_utils.cpp
M src/serialization/string_utils.hpp
M src/server/player_network.cpp
M src/text.cpp
M src/text.hpp
M src/widgets/textbox.cpp
M src/widgets/textbox.hpp
M src/windows_tray_notification.cpp
M src/windows_tray_notification.hpp
Log Message:
-----------
Get rid of utils::wide_string
utils::wide_string was a std::vector<wchar_t>
On most unix systems, this is 32 bits wide, and therefore a UCS-4 encoding.
On windows however, wchar_t is 16 bits wide, and the naive approach results
in UCS-2, which can only represent the Basic Multilingual Plane.
Most functions that used wide_string have been moved over to ucs4_string.
The Win32 API has been moved over to a specially created utf16_string instead.
Specialty tolower #ifdefs have been removed, as the towlower function has
been available since OS X 10.3, OpenBSD 3.7 and probably ancient versions of
all other BSDs too. The only issue is that on windows, the function cannot
be applied to characters outside the BMP.
Conflicts:
src/dialogs.cpp
src/gui/dialogs/addon_list.cpp
src/gui/widgets/password_box.cpp
src/gui/widgets/text.cpp
src/gui/widgets/text_box.cpp
src/serialization/string_utils.cpp
src/serialization/string_utils.hpp
src/text.cpp
src/widgets/textbox.cpp
Commit: 72c16d00670d7bcb71f46ee2907e95d79257634c
https://github.com/wesnoth/wesnoth/commit/72c16d00670d7bcb71f46ee2907e95d79257634c
Author: Alexander van Gessel <ai0867 at gmail.com>
Date: 2014-03-23 (Sun, 23 Mar 2014)
Changed paths:
M src/windows_tray_notification.cpp
Log Message:
-----------
Prevent some possible buffer overflows
Commit: f265444ecb4f15ee0757f717db3c047782657610
https://github.com/wesnoth/wesnoth/commit/f265444ecb4f15ee0757f717db3c047782657610
Author: Alexander van Gessel <ai0867 at gmail.com>
Date: 2014-03-23 (Sun, 23 Mar 2014)
Changed paths:
M src/windows_tray_notification.hpp
Log Message:
-----------
Match declaration with definition
Commit: 20d4381a2512cc00833aadc12b3af38b62c8747d
https://github.com/wesnoth/wesnoth/commit/20d4381a2512cc00833aadc12b3af38b62c8747d
Author: Alexander van Gessel <ai0867 at gmail.com>
Date: 2014-03-23 (Sun, 23 Mar 2014)
Changed paths:
M src/windows_tray_notification.cpp
Log Message:
-----------
Use existing named constants
Commit: 433edbfea36f8fd0948c25b1eb09804203e45bf2
https://github.com/wesnoth/wesnoth/commit/433edbfea36f8fd0948c25b1eb09804203e45bf2
Author: Alexander van Gessel <ai0867 at gmail.com>
Date: 2014-03-23 (Sun, 23 Mar 2014)
Changed paths:
M src/windows_tray_notification.cpp
Log Message:
-----------
Fix an identifier. Spotted by aquileia
Commit: 22eb44c8af2063b4d155384f253be6879d4fa7ae
https://github.com/wesnoth/wesnoth/commit/22eb44c8af2063b4d155384f253be6879d4fa7ae
Author: Alexander van Gessel <ai0867 at gmail.com>
Date: 2014-03-23 (Sun, 23 Mar 2014)
Changed paths:
M changelog
Log Message:
-----------
Changelog entry for unicode handling improvements
Commit: 235e519b6a038b4613f7aff357592ef6a6b47aa8
https://github.com/wesnoth/wesnoth/commit/235e519b6a038b4613f7aff357592ef6a6b47aa8
Author: Alexander van Gessel <ai0867 at gmail.com>
Date: 2014-03-23 (Sun, 23 Mar 2014)
Changed paths:
M doc/manual/manual.cs.html
M doc/manual/manual.de.html
M doc/manual/manual.en.html
M doc/manual/manual.en_GB.html
M doc/manual/manual.es.html
M doc/manual/manual.et.html
M doc/manual/manual.fi.html
M doc/manual/manual.fr.html
M doc/manual/manual.gl.html
M doc/manual/manual.hu.html
M doc/manual/manual.id.html
M doc/manual/manual.it.html
M doc/manual/manual.ja.html
M doc/manual/manual.pl.html
M doc/manual/manual.pt.html
M doc/manual/manual.pt_BR.html
M doc/manual/manual.ru.html
M doc/manual/manual.sk.html
M doc/manual/manual.sr.html
M doc/manual/manual.sr at ijekavian.html
M doc/manual/manual.sr at ijekavianlatin.html
M doc/manual/manual.sr at latin.html
M doc/manual/manual.uk.html
M doc/manual/manual.vi.html
M doc/manual/manual.zh_CN.html
M doc/manual/manual.zh_TW.html
M po/wesnoth-ai/wesnoth-ai.pot
M po/wesnoth-anl/wesnoth-anl.pot
M po/wesnoth-aoi/wesnoth-aoi.pot
M po/wesnoth-did/wesnoth-did.pot
M po/wesnoth-dm/de.po
M po/wesnoth-dm/wesnoth-dm.pot
M po/wesnoth-dw/wesnoth-dw.pot
M po/wesnoth-editor/wesnoth-editor.pot
M po/wesnoth-ei/wesnoth-ei.pot
M po/wesnoth-help/wesnoth-help.pot
M po/wesnoth-httt/de.po
M po/wesnoth-httt/wesnoth-httt.pot
M po/wesnoth-l/wesnoth-l.pot
M po/wesnoth-lib/af.po
M po/wesnoth-lib/ang.po
M po/wesnoth-lib/ang at latin.po
M po/wesnoth-lib/ar.po
M po/wesnoth-lib/bg.po
M po/wesnoth-lib/ca.po
M po/wesnoth-lib/ca_ES at valencia.po
M po/wesnoth-lib/cs.po
M po/wesnoth-lib/da.po
M po/wesnoth-lib/de.po
M po/wesnoth-lib/el.po
M po/wesnoth-lib/en at shaw.po
M po/wesnoth-lib/en_GB.po
M po/wesnoth-lib/eo.po
M po/wesnoth-lib/es.po
M po/wesnoth-lib/et.po
M po/wesnoth-lib/eu.po
M po/wesnoth-lib/fi.po
M po/wesnoth-lib/fr.po
M po/wesnoth-lib/fur_IT.po
M po/wesnoth-lib/ga.po
M po/wesnoth-lib/gd.po
M po/wesnoth-lib/gl.po
M po/wesnoth-lib/he.po
M po/wesnoth-lib/hr.po
M po/wesnoth-lib/hu.po
M po/wesnoth-lib/id.po
M po/wesnoth-lib/is.po
M po/wesnoth-lib/it.po
M po/wesnoth-lib/ja.po
M po/wesnoth-lib/ko.po
M po/wesnoth-lib/la.po
M po/wesnoth-lib/lt.po
M po/wesnoth-lib/lv.po
M po/wesnoth-lib/mk.po
M po/wesnoth-lib/mr.po
M po/wesnoth-lib/nb_NO.po
M po/wesnoth-lib/nl.po
M po/wesnoth-lib/pl.po
M po/wesnoth-lib/pt.po
M po/wesnoth-lib/pt_BR.po
M po/wesnoth-lib/racv.po
M po/wesnoth-lib/ro.po
M po/wesnoth-lib/ru.po
M po/wesnoth-lib/sk.po
M po/wesnoth-lib/sl.po
M po/wesnoth-lib/sr.po
M po/wesnoth-lib/sr at ijekavian.po
M po/wesnoth-lib/sr at ijekavianlatin.po
M po/wesnoth-lib/sr at latin.po
M po/wesnoth-lib/sv.po
M po/wesnoth-lib/tl.po
M po/wesnoth-lib/tr.po
M po/wesnoth-lib/uk.po
M po/wesnoth-lib/vi.po
M po/wesnoth-lib/wesnoth-lib.pot
M po/wesnoth-lib/zh_CN.po
M po/wesnoth-lib/zh_TW.po
M po/wesnoth-low/wesnoth-low.pot
M po/wesnoth-manpages/af.po
M po/wesnoth-manpages/ang.po
M po/wesnoth-manpages/ang at latin.po
M po/wesnoth-manpages/ar.po
M po/wesnoth-manpages/bg.po
M po/wesnoth-manpages/ca.po
M po/wesnoth-manpages/ca_ES at valencia.po
M po/wesnoth-manpages/cs.po
M po/wesnoth-manpages/da.po
M po/wesnoth-manpages/de.po
M po/wesnoth-manpages/el.po
M po/wesnoth-manpages/en at shaw.po
M po/wesnoth-manpages/en_GB.po
M po/wesnoth-manpages/eo.po
M po/wesnoth-manpages/es.po
M po/wesnoth-manpages/et.po
M po/wesnoth-manpages/eu.po
M po/wesnoth-manpages/fi.po
M po/wesnoth-manpages/fr.po
M po/wesnoth-manpages/fur_IT.po
M po/wesnoth-manpages/ga.po
M po/wesnoth-manpages/gd.po
M po/wesnoth-manpages/gl.po
M po/wesnoth-manpages/he.po
M po/wesnoth-manpages/hr.po
M po/wesnoth-manpages/hu.po
M po/wesnoth-manpages/id.po
M po/wesnoth-manpages/it.po
M po/wesnoth-manpages/ja.po
M po/wesnoth-manpages/ko.po
M po/wesnoth-manpages/la.po
M po/wesnoth-manpages/lt.po
M po/wesnoth-manpages/lv.po
M po/wesnoth-manpages/mk.po
M po/wesnoth-manpages/mr.po
M po/wesnoth-manpages/nb_NO.po
M po/wesnoth-manpages/nl.po
M po/wesnoth-manpages/pl.po
M po/wesnoth-manpages/pt.po
M po/wesnoth-manpages/pt_BR.po
M po/wesnoth-manpages/racv.po
M po/wesnoth-manpages/ro.po
M po/wesnoth-manpages/ru.po
M po/wesnoth-manpages/sk.po
M po/wesnoth-manpages/sl.po
M po/wesnoth-manpages/sr.po
M po/wesnoth-manpages/sr at ijekavian.po
M po/wesnoth-manpages/sr at ijekavianlatin.po
M po/wesnoth-manpages/sr at latin.po
M po/wesnoth-manpages/sv.po
M po/wesnoth-manpages/tl.po
M po/wesnoth-manpages/tr.po
M po/wesnoth-manpages/uk.po
M po/wesnoth-manpages/vi.po
M po/wesnoth-manpages/wesnoth-manpages.pot
M po/wesnoth-manpages/zh_CN.po
M po/wesnoth-manpages/zh_TW.po
M po/wesnoth-manual/af.po
M po/wesnoth-manual/ang.po
M po/wesnoth-manual/ang at latin.po
M po/wesnoth-manual/ar.po
M po/wesnoth-manual/bg.po
M po/wesnoth-manual/ca.po
M po/wesnoth-manual/ca_ES at valencia.po
M po/wesnoth-manual/cs.po
M po/wesnoth-manual/da.po
M po/wesnoth-manual/de.po
M po/wesnoth-manual/el.po
M po/wesnoth-manual/en at shaw.po
M po/wesnoth-manual/en_GB.po
M po/wesnoth-manual/eo.po
M po/wesnoth-manual/es.po
M po/wesnoth-manual/et.po
M po/wesnoth-manual/eu.po
M po/wesnoth-manual/fi.po
M po/wesnoth-manual/fr.po
M po/wesnoth-manual/fur_IT.po
M po/wesnoth-manual/ga.po
M po/wesnoth-manual/gd.po
M po/wesnoth-manual/gl.po
M po/wesnoth-manual/he.po
M po/wesnoth-manual/hr.po
M po/wesnoth-manual/hu.po
M po/wesnoth-manual/id.po
M po/wesnoth-manual/is.po
M po/wesnoth-manual/it.po
M po/wesnoth-manual/ja.po
M po/wesnoth-manual/ko.po
M po/wesnoth-manual/la.po
M po/wesnoth-manual/lt.po
M po/wesnoth-manual/lv.po
M po/wesnoth-manual/mk.po
M po/wesnoth-manual/mr.po
M po/wesnoth-manual/nb_NO.po
M po/wesnoth-manual/nl.po
M po/wesnoth-manual/pl.po
M po/wesnoth-manual/pt.po
M po/wesnoth-manual/pt_BR.po
M po/wesnoth-manual/racv.po
M po/wesnoth-manual/ro.po
M po/wesnoth-manual/ru.po
M po/wesnoth-manual/sk.po
M po/wesnoth-manual/sl.po
M po/wesnoth-manual/sr.po
M po/wesnoth-manual/sr at ijekavian.po
M po/wesnoth-manual/sr at ijekavianlatin.po
M po/wesnoth-manual/sr at latin.po
M po/wesnoth-manual/sv.po
M po/wesnoth-manual/tl.po
M po/wesnoth-manual/tr.po
M po/wesnoth-manual/uk.po
M po/wesnoth-manual/vi.po
M po/wesnoth-manual/wesnoth-manual.pot
M po/wesnoth-manual/zh_CN.po
M po/wesnoth-manual/zh_TW.po
M po/wesnoth-multiplayer/wesnoth-multiplayer.pot
M po/wesnoth-nr/wesnoth-nr.pot
M po/wesnoth-sof/wesnoth-sof.pot
M po/wesnoth-sotbe/wesnoth-sotbe.pot
M po/wesnoth-tb/wesnoth-tb.pot
M po/wesnoth-test/POTFILES.in
M po/wesnoth-test/wesnoth-test.pot
M po/wesnoth-thot/af.po
M po/wesnoth-thot/ang.po
M po/wesnoth-thot/ang at latin.po
M po/wesnoth-thot/ar.po
M po/wesnoth-thot/bg.po
M po/wesnoth-thot/ca.po
M po/wesnoth-thot/ca_ES at valencia.po
M po/wesnoth-thot/cs.po
M po/wesnoth-thot/da.po
M po/wesnoth-thot/de.po
M po/wesnoth-thot/el.po
M po/wesnoth-thot/en at shaw.po
M po/wesnoth-thot/en_GB.po
M po/wesnoth-thot/eo.po
M po/wesnoth-thot/es.po
M po/wesnoth-thot/et.po
M po/wesnoth-thot/eu.po
M po/wesnoth-thot/fi.po
M po/wesnoth-thot/fr.po
M po/wesnoth-thot/fur_IT.po
M po/wesnoth-thot/ga.po
M po/wesnoth-thot/gd.po
M po/wesnoth-thot/gl.po
M po/wesnoth-thot/he.po
M po/wesnoth-thot/hr.po
M po/wesnoth-thot/hu.po
M po/wesnoth-thot/id.po
M po/wesnoth-thot/is.po
M po/wesnoth-thot/it.po
M po/wesnoth-thot/ja.po
M po/wesnoth-thot/ko.po
M po/wesnoth-thot/la.po
M po/wesnoth-thot/lt.po
M po/wesnoth-thot/lv.po
M po/wesnoth-thot/mk.po
M po/wesnoth-thot/mr.po
M po/wesnoth-thot/nb_NO.po
M po/wesnoth-thot/nl.po
M po/wesnoth-thot/pl.po
M po/wesnoth-thot/pt.po
M po/wesnoth-thot/pt_BR.po
M po/wesnoth-thot/racv.po
M po/wesnoth-thot/ro.po
M po/wesnoth-thot/ru.po
M po/wesnoth-thot/sk.po
M po/wesnoth-thot/sl.po
M po/wesnoth-thot/sr.po
M po/wesnoth-thot/sr at ijekavian.po
M po/wesnoth-thot/sr at ijekavianlatin.po
M po/wesnoth-thot/sr at latin.po
M po/wesnoth-thot/sv.po
M po/wesnoth-thot/tl.po
M po/wesnoth-thot/tr.po
M po/wesnoth-thot/uk.po
M po/wesnoth-thot/vi.po
M po/wesnoth-thot/wesnoth-thot.pot
M po/wesnoth-thot/zh_CN.po
M po/wesnoth-thot/zh_TW.po
M po/wesnoth-trow/wesnoth-trow.pot
M po/wesnoth-tsg/wesnoth-tsg.pot
M po/wesnoth-tutorial/wesnoth-tutorial.pot
M po/wesnoth-units/wesnoth-units.pot
M po/wesnoth-utbs/wesnoth-utbs.pot
M po/wesnoth/POTFILES.in
M po/wesnoth/af.po
M po/wesnoth/ang.po
M po/wesnoth/ang at latin.po
M po/wesnoth/ar.po
M po/wesnoth/bg.po
M po/wesnoth/ca.po
M po/wesnoth/ca_ES at valencia.po
M po/wesnoth/cs.po
M po/wesnoth/da.po
M po/wesnoth/de.po
M po/wesnoth/el.po
M po/wesnoth/en at shaw.po
M po/wesnoth/en_GB.po
M po/wesnoth/eo.po
M po/wesnoth/es.po
M po/wesnoth/et.po
M po/wesnoth/eu.po
M po/wesnoth/fi.po
M po/wesnoth/fr.po
M po/wesnoth/fur_IT.po
M po/wesnoth/ga.po
M po/wesnoth/gd.po
M po/wesnoth/gl.po
M po/wesnoth/he.po
M po/wesnoth/hr.po
M po/wesnoth/hu.po
M po/wesnoth/id.po
M po/wesnoth/is.po
M po/wesnoth/it.po
M po/wesnoth/ja.po
M po/wesnoth/ko.po
M po/wesnoth/la.po
M po/wesnoth/lt.po
M po/wesnoth/lv.po
M po/wesnoth/mk.po
M po/wesnoth/mr.po
M po/wesnoth/nb_NO.po
M po/wesnoth/nl.po
M po/wesnoth/pl.po
M po/wesnoth/pt.po
M po/wesnoth/pt_BR.po
M po/wesnoth/racv.po
M po/wesnoth/ro.po
M po/wesnoth/ru.po
M po/wesnoth/sk.po
M po/wesnoth/sl.po
M po/wesnoth/sr.po
M po/wesnoth/sr at ijekavian.po
M po/wesnoth/sr at ijekavianlatin.po
M po/wesnoth/sr at latin.po
M po/wesnoth/sv.po
M po/wesnoth/tl.po
M po/wesnoth/tr.po
M po/wesnoth/uk.po
M po/wesnoth/vi.po
M po/wesnoth/wesnoth.pot
M po/wesnoth/zh_CN.po
M po/wesnoth/zh_TW.po
Log Message:
-----------
Merge branch '1.12' of github.com:wesnoth/wesnoth into 1.12
Compare: https://github.com/wesnoth/wesnoth/compare/9414bbc700b2...235e519b6a03
More information about the Commits
mailing list