From unknown Sat Aug 16 16:01:12 2025 X-Loop: help-debbugs@gnu.org Subject: bug#26319: sort -u deletes characters it shouldn't Resent-From: sort-u@re-gister.com Original-Sender: "Debbugs-submit" Resent-CC: bug-coreutils@gnu.org Resent-Date: Fri, 31 Mar 2017 04:35:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 26319 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: 26319@debbugs.gnu.org X-Debbugs-Original-To: bug-coreutils@gnu.org Reply-To: sort-u@re-gister.com Received: via spool by submit@debbugs.gnu.org id=B.149093486625780 (code B ref -1); Fri, 31 Mar 2017 04:35:02 +0000 Received: (at submit) by debbugs.gnu.org; 31 Mar 2017 04:34:26 +0000 Received: from localhost ([127.0.0.1]:53451 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ctoGX-0006hj-Vl for submit@debbugs.gnu.org; Fri, 31 Mar 2017 00:34:26 -0400 Received: from eggs.gnu.org ([208.118.235.92]:50324) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cthbv-0007q2-Ek for submit@debbugs.gnu.org; Thu, 30 Mar 2017 17:28:03 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cthbp-0001HF-26 for submit@debbugs.gnu.org; Thu, 30 Mar 2017 17:27:57 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=BAYES_40,HTML_MESSAGE autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:52183) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cthbo-0001HB-V7 for submit@debbugs.gnu.org; Thu, 30 Mar 2017 17:27:56 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:41294) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cthbn-0001k1-Ss for bug-coreutils@gnu.org; Thu, 30 Mar 2017 17:27:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cthbl-0001E9-AG for bug-coreutils@gnu.org; Thu, 30 Mar 2017 17:27:55 -0400 Received: from astra4649.startdedicated.de ([62.138.6.202]:51669) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cthbl-0001DG-3n for bug-coreutils@gnu.org; Thu, 30 Mar 2017 17:27:53 -0400 Received: by astra4649.startdedicated.de (Postfix, from userid 33) id 2F6083273F2; Thu, 30 Mar 2017 23:27:53 +0200 (CEST) X-PHP-Originating-Script: 0:Sendmail.php X-Originating-IP: 185.61.148.238 From: sort-u@re-gister.com Date: Thu, 30 Mar 2017 23:27:53 +0200 Content-Type: multipart/alternative; boundary="=_c786def06f4065524ab40309feabe585" MIME-Version: 1.0 Message-Id: <20170330212753.2F6083273F2@astra4649.startdedicated.de> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -5.0 (-----) X-Mailman-Approved-At: Fri, 31 Mar 2017 00:34:24 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -5.0 (-----) --=_c786def06f4065524ab40309feabe585 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable sort (GNU coreutils) 8.26local of the system: en_US.UTF-8contents of /et= c/locale.conf:LANG=3Den_US.UTF-8LC_TIME=3Dfr_FR.UTF-8LC_NUMERIC=3Dfr_FR.= UTF-8=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D$ echo $'=E4=9C=8C= \n=E9=BE=B9' | sort -uExpected output:=E9=BE=B9=E4=9C=8CActual output:= =E4=9C=8C --=_c786def06f4065524ab40309feabe585 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable sort (GNU coreutils) 8.26
local of the system: en_US.UTF-8

con= tents of /etc/locale.conf:
LANG=3Den_US.UTF-8
LC_TIME=3Dfr_FR.UTF-= 8
LC_NUMERIC=3Dfr_FR.UTF-8
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D
$ echo $'=E4=9C=8C\n=E9=BE=B9' | sort -u

Expected ou= tput:
=E9=BE=B9
=E4=9C=8C
Actual output:
=E4=9C=8C
--=_c786def06f4065524ab40309feabe585-- From unknown Sat Aug 16 16:01:12 2025 X-Loop: help-debbugs@gnu.org Subject: bug#26319: sort -u deletes characters it shouldn't Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady Original-Sender: "Debbugs-submit" Resent-CC: bug-coreutils@gnu.org Resent-Date: Fri, 31 Mar 2017 05:16:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 26319 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: sort-u@re-gister.com, 26319@debbugs.gnu.org Received: via spool by 26319-submit@debbugs.gnu.org id=B26319.149093731230097 (code B ref 26319); Fri, 31 Mar 2017 05:16:02 +0000 Received: (at 26319) by debbugs.gnu.org; 31 Mar 2017 05:15:12 +0000 Received: from localhost ([127.0.0.1]:53491 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ctou0-0007pN-M7 for submit@debbugs.gnu.org; Fri, 31 Mar 2017 01:15:12 -0400 Received: from midir.magicbluesmoke.com ([82.195.144.46]:38480 helo=mail.magicbluesmoke.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ctoty-0007pE-Am for 26319@debbugs.gnu.org; Fri, 31 Mar 2017 01:15:10 -0400 Received: from localhost.localdomain (mobile-166-137-178-014.mycingular.net [166.137.178.14]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.magicbluesmoke.com (Postfix) with ESMTPSA id 56E6C98B0; Fri, 31 Mar 2017 06:15:08 +0100 (IST) References: <20170330212753.2F6083273F2@astra4649.startdedicated.de> From: =?UTF-8?Q?P=C3=A1draig?= Brady Message-ID: <9e2554fc-c197-f438-538a-9804e1aa1b4c@draigBrady.com> Date: Thu, 30 Mar 2017 22:15:04 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20170330212753.2F6083273F2@astra4649.startdedicated.de> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) tag 26319 notabug close 26319 stop On 30/03/17 14:27, sort-u@re-gister.com wrote: > sort (GNU coreutils) 8.26 > $ echo $'䜌\n龹' | sort -u > Expected output:龹䜌 > Actual output:䜌 That's because the strcoll() in that locale is comparing them as equal. This needs to be addressed in the locale data for glibc etc. thanks, Pádraig From debbugs-submit-bounces@debbugs.gnu.org Sun Oct 28 22:59:32 2018 Received: (at control) by debbugs.gnu.org; 29 Oct 2018 02:59:32 +0000 Received: from localhost ([127.0.0.1]:49659 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gGxm8-0002oE-EP for submit@debbugs.gnu.org; Sun, 28 Oct 2018 22:59:32 -0400 Received: from mail-io1-f54.google.com ([209.85.166.54]:36001) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gGxm6-0002o2-EC for control@debbugs.gnu.org; Sun, 28 Oct 2018 22:59:30 -0400 Received: by mail-io1-f54.google.com with SMTP id o19-v6so4050442iod.3 for ; Sun, 28 Oct 2018 19:59:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=to:from:message-id:date:user-agent:mime-version:content-language :content-transfer-encoding; bh=1on9/KtzG+mSi3XociNLKVodJyPVYisVxk1I4SKDWSI=; b=Bbjd2Koof7DE7p7114bJbgfnwHM/sKSC0ll2Rpy9b0UmrrYUR0ViNMkguWKWRX55Et 117Q7iJaZHzewlJjD8N6d1+HRU14IS8H1SnEI5qTFXqTGkY5IM8M75EtSTwOZzgrc9ZH VdB83hPvskTTFflwAi50yOJ0GfWM9YhFhrSmoVXsltF9dKhApTsOeUOD2CNggfgQobdH Sb0+9EwMllzexffqxoTkhpCVVUSN7MSUj5W1GracHaqPwEI5Qn+xBmzSgGBCfmZA0Ux9 kS/CvFNVk2cjrjoBbITM25daO8/sONu/glaXROcPaGVbZ68Hfj0xdqhX4O2ZVS/pPkLw Dujw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:message-id:date:user-agent:mime-version :content-language:content-transfer-encoding; bh=1on9/KtzG+mSi3XociNLKVodJyPVYisVxk1I4SKDWSI=; b=EcOaVVHR3FBjzsx0eEsJWnuL1BIbATOU6D36/gkz6U3xHewZAZlDi2DjOFV3xSTkiU AV4ioSr4rG/GJXVt5SmXqSunYAaAvaNpgF2D1T1rMVS4kHjTeIXoJVOC5lIIKiQ5fzxF 1ws3GS7mq8Evc4UuWM5FZ50VmpB2ydu6mAfbj7yq44xT7KtkOmK6b4r56N4zQpV2i012 uN944oppqFjnDYIYu5JEK0hHt5Mf3x0QW8u+pwEsxaySNopD1K6Fy8+p3JZ7htTvIlY9 a3me0gwLGhWS5s4QhdPsTFURPf6S1UekBC6Vs6WgUsIN+a4uOK02kvOrfOrX6ZHCl7Of KEVA== X-Gm-Message-State: AGRZ1gKMzf96A8K1w4G8Ma8qwA5S9dN5i8XmMdWgKJEBTKiKivPYpCPw FBbw6yZA1Rekw9IVEmVBGnPMWJM++Ow= X-Google-Smtp-Source: AJdET5fExpLLNZNoVwHzj0jvKLd2Me8n8+lqeZlPNpkrPIYMfNR5P8KgwaGnNdGFQSEDpSTNZbZgEQ== X-Received: by 2002:a5e:cb07:: with SMTP id p7-v6mr2530455iom.178.1540781964410; Sun, 28 Oct 2018 19:59:24 -0700 (PDT) Received: from tomato.housegordon.com (moose.housegordon.com. [184.68.105.38]) by smtp.googlemail.com with ESMTPSA id u68-v6sm4437834itd.1.2018.10.28.19.59.21 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 28 Oct 2018 19:59:23 -0700 (PDT) To: control@debbugs.gnu.org From: Assaf Gordon Message-ID: Date: Sun, 28 Oct 2018 20:59:20 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: tags 26319 notabug close 26319 [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [209.85.166.54 listed in list.dnswl.org] 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (assafgordon[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record 1.8 MISSING_SUBJECT Missing Subject: header 0.2 NO_SUBJECT Extra score for no subject X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) tags 26319 notabug close 26319