Feature request: Edits
tavianator opened this issue · 13 comments
tavianator commented
I'd like to use CaseMap and record the positions of the modifications.
ovalhub commented
I added support for Edits and its iterators in rev 174 just now.
Please, try it out and let me know if it works for you.
The Edits iterators return a tuple
(hasChange, oldLength, newLength,
sourceIndex, destinationIndex, replacementIndex)
These values are also available via props on the iterator objects.
Thanks !
Andi..
tavianator commented
Wow that was fast, thanks! Trying it now. I needed this patch to get it to build:
diff --git a/numberformat.cpp b/numberformat.cpp
index 8e9ee8e..f3ed0a5 100644
--- a/numberformat.cpp
+++ b/numberformat.cpp
@@ -788,7 +788,7 @@ static int t_decimalformatsymbols_init(t_decimalformatsymbols *self,
{
Locale *locale;
DecimalFormatSymbols *dfs;
-#if U_ICU_VERSION_HEX >= VERSION_HEX(63, 0, 0)
+#if U_ICU_VERSION_HEX >= VERSION_HEX(60, 0, 0)
NumberingSystem *system;
#endif
tavianator commented
The named properties don't seem to work, but the tuples do:
>>> import icu
>>> e = icu.Edits()
>>> icu.CaseMap.fold(0, "abcßDeF", e)
'abcssdef'
>>> i = e.getFineIterator()
>>> next(i)
(False, 3, 3, 0, 0, 0)
>>> i.hasChange
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'icu.EditsIterator' object has no attribute 'hasChange'
tavianator commented
From looking here it seems to be (..., replacementIndex, destinationIndex), not the other way around.
tavianator commented
One final note: it might be nice to support some overloads that take in UnicodeString as a destination, since the offsets in Edits will be UnicodeString offsets (i.e. UTF-16) not Python string offsets.
ovalhub commented
On Wed, 17 Oct 2018, Tavian Barnes wrote:
Wow that was fast, thanks! Trying it now. I needed this patch to get it to build:
```
diff --git a/numberformat.cpp b/numberformat.cpp
index 8e9ee8e..f3ed0a5 100644
--- a/numberformat.cpp
+++ b/numberformat.cpp
@@ -788,7 +788,7 @@ static int t_decimalformatsymbols_init(t_decimalformatsymbols *self,
{
Locale *locale;
DecimalFormatSymbols *dfs;
-#if U_ICU_VERSION_HEX >= VERSION_HEX(63, 0, 0)
+#if U_ICU_VERSION_HEX >= VERSION_HEX(60, 0, 0)
Yes, thanks, copy/paste error from last night.
Andi..
… NumberingSystem *system;
#endif
```
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#90 (comment)
ovalhub commented
On Wed, 17 Oct 2018, Tavian Barnes wrote:
The named properties don't seem to work, but the tuples do:
```
>>> import icu
>>> e = icu.Edits()
>>> icu.CaseMap.fold(0, "abcßDeF", e)
'abcssdef'
>>> i = e.getFineIterator()
>>> next(i)
(False, 3, 3, 0, 0, 0)
>>> i.hasChange
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'icu.EditsIterator' object has no attribute 'hasChange'
```
Indeed, the type needed to setup with that before INSTALL.
Andi..
…
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#90 (comment)
ovalhub commented
On Wed, 17 Oct 2018, Tavian Barnes wrote:
From looking [here](41713ed#diff-36a3d85296fc1fd40a81f27c4301f2efR1397) it seems to be (..., replacementIndex, destinationIndex), not the other way around.
The intention was source, destination, replacement but the docs has it the
way you found. Fixed to be the way I intended it.
Andi..
…
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#90 (comment)
ovalhub commented
On Wed, 17 Oct 2018, Tavian Barnes wrote:
One final note: it might be nice to support some overloads that take in `UnicodeString` as a destination, since the offsets in `Edits` will be `UnicodeString` offsets (i.e. UTF-16) not Python string offsets.
What API are you referring to ?
I see no string, char * or UnicodeString inputs here:
http://icu-project.org/apiref/icu4c/classicu_1_1Edits.html
Or are you referring to CaseMap ?
Andi..
…
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#90 (comment)
ovalhub commented
On Wed, 17 Oct 2018, Tavian Barnes wrote:
One final note: it might be nice to support some overloads that take in `UnicodeString` as a destination, since the offsets in `Edits` will be `UnicodeString` offsets (i.e. UTF-16) not Python string offsets.
>> from icu import *
>> CaseMap.toLower(UnicodeString('ABCD'))
u'abcd
Seems to work but maybe you're thinking of another API ?
Andi..
…
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#90 (comment)
ovalhub commented
All the fixes for bugs you reported are available in HEAD rev 177.
Thank you for the testing and bug reports !
Andi..
…On Wed, 17 Oct 2018, Tavian Barnes wrote:
One final note: it might be nice to support some overloads that take in `UnicodeString` as a destination, since the offsets in `Edits` will be `UnicodeString` offsets (i.e. UTF-16) not Python string offsets.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#90 (comment)
tavianator commented
Seems to work but maybe you're thinking of another API ?
Ah, I was talking about things like this:
>>> string = UnicodeString()
>>> name = locale.getDisplayName(string)
>>> name
<UnicodeString: Portuguese (Brazil)>
>>> name is string
True
I didn't realise there's an actual ICU API that looks like that, I thought that was just a pyicu convention for getting the output as an icu.UnicodeString instead of str.
Anyway, everything works great, thanks!
ovalhub commented
On Thu, 18 Oct 2018, Tavian Barnes wrote:
> Seems to work but maybe you're thinking of another API ?
Ah, I was talking about things like this:
```
>>> string = UnicodeString()
>>> name = locale.getDisplayName(string)
>>> name
<UnicodeString: Portuguese (Brazil)>
>>> name is string
True
```
I didn't realise there's an actual ICU API that looks like that, I thought that was just a pyicu convention for getting the output as an `icu.UnicodeString` instead of `str`.
Anyway, everything works great, thanks!
I'm still not sure what you mean but the goal is to have a pythonic API that
feels natural to use with ICU. Passing string to receive data is not very
pythonic. Thus, in Python 2:
>> locale.getItaly().getDisplayName()
u'Italian (Italy)'
is the pythonic way, but
>> locale.getItaly().getDisplayName(UnicodeString())
<UnicodeString: u'Italian (Italy)'>
is not.
You'll find that the latter form is only very rarely supported in PyICU.
Locale is one of the oldest classes wrapped by PyICU, a project I started in
2004, and the wrapping of ICU APIs (as well as the patterns in the ICU C++
APIs themselves) have evolved over the years.
Andi..
…
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#90 (comment)