GNU bug report logs - #53145
"cut" can't segment Chinese characters correctly?

Previous Next

To reply to this bug, email your comments to 53145 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: zendas <zendas <at> protonmail.com>
To: "bug-coreutils <at> gnu.org" <bug-coreutils <at> gnu.org>
Subject: "cut" can't segment Chinese characters correctly?
Date: Sun, 09 Jan 2022 18:53:29 +0000

[Message part 1 (text/plain, inline)]

Hello, I need to get Chinese characters from the string. I googled a lot of documents, it seems that the -c parameter of cut should be able to meet my needs, but I even directly execute the instructions on the web page, and the result is different from the demonstration. I have searched dozens of pages but the results are not the same as the demo, maybe this is a bug?

For example:
https://blog.csdn.net/xuzhangze/article/details/80930714
[20180705173450701.png]
the result of my attempt:
[螢幕快照 2022-01-10 02:49:46.png]

[Message part 2 (text/html, inline)]

[螢幕快照 2022-01-10 024946.png (image/png, inline)]

[20180705173450701.png (image/png, inline)]

Message #8 received at 53145 <at> debbugs.gnu.org (full text, mbox):

From: Bob Proulx <bob <at> proulx.com>
To: 53145 <at> debbugs.gnu.org, zendas <zendas <at> protonmail.com>
Subject: Re: bug#53145: "cut" can't segment Chinese characters correctly?
Date: Sun, 9 Jan 2022 12:40:20 -0700

zendas wrote:
> Hello, I need to get Chinese characters from the string. I googled a
> lot of documents, it seems that the -c parameter of cut should be
> able to meet my needs, but I even directly execute the instructions
> on the web page, and the result is different from the
> demonstration. I have searched dozens of pages but the results are
> not the same as the demo, maybe this is a bug?

Unfortunately the example was attached as images instead of as plain
text.  Please in the future copy and paste the example as text rather
than as an image.  As an image it is impossible to reproduce by trying
to copy and paste the image.  As an image it is impossible to search
for the strings.

The images were also lost somehow from the various steps in the
mailing list pipelines with this message.  First it was classified as
spam by the anti-spam robot (SpamAssassin-Bogofilter-CRM114).  I
caught it in review and re-sent the message.  That may have been the
problem specifically with images.

> For example:
> https://blog.csdn.net/xuzhangze/article/details/80930714
> [20180705173450701.png]
> the result of my attempt:
> [螢幕快照 2022-01-10 02:49:46.png]

One of the two images:

    https://debbugs.gnu.org/cgi/bugreport.cgi?msg=5;bug=53145;att=3;filename=20180705173450701.png

Second problem is that the first image shows as being corrupted.  I
can view the original however.  To my eye they are similar enough that
the one above is sufficient and I do not need to re-send the corrupted
image.

As to the problem you have reported it is due to lack of
internationalization support for characters.  -c is the same as -b at
this moment.

    https://www.gnu.org/software/coreutils/manual/html_node/cut-invocation.html#cut-invocation

    ‘-c CHARACTER-LIST’
    ‘--characters=CHARACTER-LIST’
         Select for printing only the characters in positions listed in
         CHARACTER-LIST.  The same as ‘-b’ for now, but internationalization
         will change that.  Tabs and backspaces are treated like any other
         character; they take up 1 character.  If an output delimiter is
         specified, (see the description of ‘--output-delimiter’), then
         output that string between ranges of selected bytes.

For multi-byte UTF-8 characters the -c option will operate the same as
the -b option as of the current version and is not suitable for
dealing with multi-byte characters.

    $ echo '螢幕快照'
    螢幕快照
    $ echo '螢幕快照' | cut -c 1
    ?
    $ echo '螢幕快照' | cut -c 1-3
    螢
    $ echo '螢幕快照' | cut -b 1-3
    螢

If the characters are known to be 3 bytes multi-characters then I
might suggest using -b to workaround the problem assuming 3 byte
characters.  Eventually when -c is coded to handle multi-byte
characters the handling as bytes will change.  Using -b would avoid
that change.

Some operating systems have patched that specific version of utilities
locally to add multi-byte character handling.  But the patches have
not been found acceptable for inclusion.  That is why there are
differences between different operating systems.

Bob

Message #11 received at 53145 <at> debbugs.gnu.org (full text, mbox):

From: zendas <zendas <at> protonmail.com>
To: 53145 <at> debbugs.gnu.org, zendas <zendas <at> protonmail.com>
Subject: Re: bug#53145: "cut" can't segment Chinese characters correctly?
Date: Sun, 09 Jan 2022 19:51:33 +0000

Create a new test2.txt, the content is
星期一
星期二
星期三
星期四
星期五
星期六
星期日
=============================
zendas <at> Backup-Server:/tmp$ cat test2.txt
星期一
星期二
星期三
星期四
星期五
星期六
星期日
zendas <at> Backup-Server:/tmp$
=============================
zendas <at> Backup-Server:/tmp$ cut -c 1 test2.txt
�
�
�
�
�
�
�
zendas <at> Backup-Server:/tmp$ cut -c 2 test2.txt
�
�
�
�
�
�
�
zendas <at> Backup-Server:/tmp$ cut -c 1-3 test2.txt
星
星
星
星
星
星
星
zendas <at> Backup-Server:/tmp$
=============================
Reference source:
https://blog.csdn.net/m0_38110132/article/details/79883827

my environment is:
zendas <at> Backup-Server:~$ cat /etc/debian_version
11.1
zendas <at> Backup-Server:~$ cut --version
cut (GNU coreutils) 8.32
Copyright (C) 2020 Free Software Foundation, Inc.
授權條款 GPLv3+：GNU 通用公共授權條款第 3 版或更新版本 <https://gnu.org/licenses/gpl.html>。
本軟體是自由軟體：您可以自由修改和重新發布它。
在法律範圍內沒有其他保證。

由 David M. Ihnat、David MacKenzie 和 Jim Meyering 編寫。

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

在 2022年1月10日 星期一 上午 3:40，Bob Proulx <bob <at> proulx.com> 寫道：

> zendas wrote:
>
> > Hello, I need to get Chinese characters from the string. I googled a
> >
> > lot of documents, it seems that the -c parameter of cut should be
> >
> > able to meet my needs, but I even directly execute the instructions
> >
> > on the web page, and the result is different from the
> >
> > demonstration. I have searched dozens of pages but the results are
> >
> > not the same as the demo, maybe this is a bug?
>
> Unfortunately the example was attached as images instead of as plain
>
> text. Please in the future copy and paste the example as text rather
>
> than as an image. As an image it is impossible to reproduce by trying
>
> to copy and paste the image. As an image it is impossible to search
>
> for the strings.
>
> The images were also lost somehow from the various steps in the
>
> mailing list pipelines with this message. First it was classified as
>
> spam by the anti-spam robot (SpamAssassin-Bogofilter-CRM114). I
>
> caught it in review and re-sent the message. That may have been the
>
> problem specifically with images.
>
> > For example:
> >
> > https://blog.csdn.net/xuzhangze/article/details/80930714
> >
> > [20180705173450701.png]
> >
> > the result of my attempt:
> >
> > [螢幕快照 2022-01-10 02:49:46.png]
>
> One of the two images:
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?msg=5;bug=53145;att=3;filename=20180705173450701.png
>
> Second problem is that the first image shows as being corrupted. I
>
> can view the original however. To my eye they are similar enough that
>
> the one above is sufficient and I do not need to re-send the corrupted
>
> image.
>
> As to the problem you have reported it is due to lack of
>
> internationalization support for characters. -c is the same as -b at
>
> this moment.
>
> https://www.gnu.org/software/coreutils/manual/html_node/cut-invocation.html#cut-invocation
>
> ‘-c CHARACTER-LIST’
>
> ‘--characters=CHARACTER-LIST’
>
> Select for printing only the characters in positions listed in
>
> CHARACTER-LIST. The same as ‘-b’ for now, but internationalization
>
> will change that. Tabs and backspaces are treated like any other
>
> character; they take up 1 character. If an output delimiter is
>
> specified, (see the description of ‘--output-delimiter’), then
>
> output that string between ranges of selected bytes.
>
> For multi-byte UTF-8 characters the -c option will operate the same as
>
> the -b option as of the current version and is not suitable for
>
> dealing with multi-byte characters.
>
> $ echo '螢幕快照'
>
> 螢幕快照
>
> $ echo '螢幕快照' | cut -c 1
>
> ?
>
> $ echo '螢幕快照' | cut -c 1-3
>
> 螢
>
> $ echo '螢幕快照' | cut -b 1-3
>
> 螢
>
> If the characters are known to be 3 bytes multi-characters then I
>
> might suggest using -b to workaround the problem assuming 3 byte
>
> characters. Eventually when -c is coded to handle multi-byte
>
> characters the handling as bytes will change. Using -b would avoid
>
> that change.
>
> Some operating systems have patched that specific version of utilities
>
> locally to add multi-byte character handling. But the patches have
>
> not been found acceptable for inclusion. That is why there are
>
> differences between different operating systems.
>
> Bob

Message #14 received at 53145 <at> debbugs.gnu.org (full text, mbox):

From: zendas <zendas <at> protonmail.com>
To: 53145 <at> debbugs.gnu.org, zendas <zendas <at> protonmail.com>
Subject: 回覆: Re: bug#53145: "cut" can't segment Chinese characters correctly?
Date: Mon, 10 Jan 2022 05:14:41 +0000

zendas <at> Backup-Server:/tmp$ echo "你好啊" | cut -c 1-3 | od -b
0000000 344 275 240 012
0000004
zendas <at> Backup-Server:/tmp$ echo "你好啊" | cut -c 1 | od -b
0000000 344 012
0000002
zendas <at> Backup-Server:/tmp$ echo "你好啊" | cut -b 1 | od -b
0000000 344 012
0000002
zendas <at> Backup-Server:/tmp$ echo "你好啊" | cut -nb 1 | od -b
0000000 344 012
0000002
zendas <at> Backup-Server:/tmp$ echo "你好啊" | cut -c 1-3
ä½ 
zendas <at> Backup-Server:/tmp$ echo "你好啊" | cut -c 1
�
zendas <at> Backup-Server:/tmp$
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

在 2022年1月10日 星期一 上午 3:51，zendas <zendas <at> protonmail.com> 寫道：

> Reference source:
>
> https://blog.csdn.net/m0_38110132/article/details/79883827
>
> my environment is:
>
> zendas <at> Backup-Server:~$ cat /etc/debian_version
>
> 11.1
>
> zendas <at> Backup-Server:~$ cut --version
>
> cut (GNU coreutils) 8.32
>
> Copyright (C) 2020 Free Software Foundation, Inc.
>
> 授權條款 GPLv3+：GNU 通用公共授權條款第 3 版或更新版本 https://gnu.org/licenses/gpl.html。
>
> 本軟體是自由軟體：您可以自由修改和重新發布它。
>
> 在法律範圍內沒有其他保證。
>
> 由 David M. Ihnat、David MacKenzie 和 Jim Meyering 編寫。
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>
> 在 2022年1月10日 星期一 上午 3:40，Bob Proulx bob <at> proulx.com 寫道：
>
> > zendas wrote:
> >
> > > Hello, I need to get Chinese characters from the string. I googled a
> > >
> > > lot of documents, it seems that the -c parameter of cut should be
> > >
> > > able to meet my needs, but I even directly execute the instructions
> > >
> > > on the web page, and the result is different from the
> > >
> > > demonstration. I have searched dozens of pages but the results are
> > >
> > > not the same as the demo, maybe this is a bug?
> >
> > Unfortunately the example was attached as images instead of as plain
> >
> > text. Please in the future copy and paste the example as text rather
> >
> > than as an image. As an image it is impossible to reproduce by trying
> >
> > to copy and paste the image. As an image it is impossible to search
> >
> > for the strings.
> >
> > The images were also lost somehow from the various steps in the
> >
> > mailing list pipelines with this message. First it was classified as
> >
> > spam by the anti-spam robot (SpamAssassin-Bogofilter-CRM114). I
> >
> > caught it in review and re-sent the message. That may have been the
> >
> > problem specifically with images.
> >
> > > For example:
> > >
> > > https://blog.csdn.net/xuzhangze/article/details/80930714
> > >
> > > [20180705173450701.png]
> > >
> > > the result of my attempt:
> > >
> > > [螢幕快照 2022-01-10 02:49:46.png]
> >
> > One of the two images:
> >
> > https://debbugs.gnu.org/cgi/bugreport.cgi?msg=5;bug=53145;att=3;filename=20180705173450701.png
> >
> > Second problem is that the first image shows as being corrupted. I
> >
> > can view the original however. To my eye they are similar enough that
> >
> > the one above is sufficient and I do not need to re-send the corrupted
> >
> > image.
> >
> > As to the problem you have reported it is due to lack of
> >
> > internationalization support for characters. -c is the same as -b at
> >
> > this moment.
> >
> > https://www.gnu.org/software/coreutils/manual/html_node/cut-invocation.html#cut-invocation
> >
> > ‘-c CHARACTER-LIST’
> >
> > ‘--characters=CHARACTER-LIST’
> >
> > Select for printing only the characters in positions listed in
> >
> > CHARACTER-LIST. The same as ‘-b’ for now, but internationalization
> >
> > will change that. Tabs and backspaces are treated like any other
> >
> > character; they take up 1 character. If an output delimiter is
> >
> > specified, (see the description of ‘--output-delimiter’), then
> >
> > output that string between ranges of selected bytes.
> >
> > For multi-byte UTF-8 characters the -c option will operate the same as
> >
> > the -b option as of the current version and is not suitable for
> >
> > dealing with multi-byte characters.
> >
> > $ echo '螢幕快照'
> >
> > 螢幕快照
> >
> > $ echo '螢幕快照' | cut -c 1
> >
> > ?
> >
> > $ echo '螢幕快照' | cut -c 1-3
> >
> > 螢
> >
> > $ echo '螢幕快照' | cut -b 1-3
> >
> > 螢
> >
> > If the characters are known to be 3 bytes multi-characters then I
> >
> > might suggest using -b to workaround the problem assuming 3 byte
> >
> > characters. Eventually when -c is coded to handle multi-byte
> >
> > characters the handling as bytes will change. Using -b would avoid
> >
> > that change.
> >
> > Some operating systems have patched that specific version of utilities
> >
> > locally to add multi-byte character handling. But the patches have
> >
> > not been found acceptable for inclusion. That is why there are
> >
> > differences between different operating systems.
> >
> > Bob

Message #17 received at 53145 <at> debbugs.gnu.org (full text, mbox):

From: zendas <zendas <at> protonmail.com>
To: 53145 <at> debbugs.gnu.org
Subject: 回覆: bug#53145: Acknowledgement ("cut" can't segment Chinese characters correctly?)
Date: Wed, 12 Jan 2022 11:19:01 +0000

I have considered dealing with this problem directly with three bytes instead, but I have two doubts, I can correctly use wc -m to recognize the bytes in the same environment (but cut can't?), and my script goal is to recognize Chinese, will The probability of execution is higher on platforms that support Chinese environment. In addition, the fixed three-byte approach cannot handle the mixed content of full shape and half shape. I need a lot of judgment and conversion, which will greatly increase the possibility of errors.

在 2022年1月10日 星期一 上午 3:13， <help-debbugs <at> gnu.org> 寫道：

> 53145: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=53145
>
> GNU Bug Tracking System
>
> Contact help-debbugs <at> gnu.org with problems

Message #20 received at 53145 <at> debbugs.gnu.org (full text, mbox):

From: Bernhard Voelker <mail <at> bernhard-voelker.de>
To: zendas <zendas <at> protonmail.com>, 53145 <at> debbugs.gnu.org
Subject: Re: bug#53145: 回覆: bug#53145: Acknowledgement ("cut" can't segment Chinese characters correctly?)
Date: Wed, 12 Jan 2022 13:25:17 +0100

On 1/12/22 12:19, zendas via GNU coreutils Bug Reports wrote:
> I have considered dealing with this problem directly with three bytes instead, but I have two doubts, I can correctly use wc -m to recognize the bytes in the same environment (but cut can't?), and my script goal is to recognize Chinese, will The probability of execution is higher on platforms that support Chinese environment. In addition, the fixed three-byte approach cannot handle the mixed content of full shape and half shape. I need a lot of judgment and conversion, which will greatly increase the possibility of errors.

As Bob wrote, some downstream distributions have multi-byte support in cut(1) for many years,
e.g. RHEL/Fedora and SUSE/openSUSE.

E.g. here on my openSUSE system:

  $ echo "你好啊" | LC_ALL=zh_CN.UTF-8 cut -c 1
  ä½ 

Have a nice day,
Berny

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.