Subject: bug#51698: closed (Re: [PATCH 1/1] tests: make surrogate-pair pass under Cygwin)

From: [email-masked] (GNU bug Tracking System)
Date: Wed, 24 Nov 2021 02:36:02 +0000

Your bug report

#27555: surrogate-pair test fails under Cygwin

which was filed against the grep package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
27555: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=27555
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: [PATCH 1/1] tests: make surrogate-pair pass under Cygwin
From: Paul Eggert <[email-masked]>
To: Duncan Roe <[email-masked]>
Cc: [email-masked]
Date: Tue, 23 Nov 2021 18:34:51 -0800
List-Id: <debbugs-submit.debbugs.gnu.org>

I installed the attached fancier patch instead; please give it a try.

I'm boldly marking the bug as fixed; we can unmark it later if I'm wrong.

From 1ef60a1566e6c2df610bf29bd2c2994d58dd1e89 Mon Sep 17 00:00:00 2001
From: Paul Eggert <[email-masked]>
Date: Tue, 23 Nov 2021 18:28:10 -0800
Subject: [PATCH] tests: skip surrogate-search test on Cygwin

Cygwin does not support surrogate-pair search strings, so
skip the test there (Bug#27555).
* tests/Makefile.am (TESTS): Add surrogate-search.
* tests/surrogate-pair: Remove surrogate-search test,
which is now done by surrogate-search.
* tests/surrogate-search: New test, which is skipped on Cygwin.
---
 tests/Makefile.am      |  1 +
 tests/surrogate-pair   |  6 ------
 tests/surrogate-search | 42 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 43 insertions(+), 6 deletions(-)
 create mode 100755 tests/surrogate-search

diff --git a/tests/Makefile.am b/tests/Makefile.am
index c84cdc0..92f194f 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -161,6 +161,7 @@ TESTS =						\
   stack-overflow				\
   status					\
   surrogate-pair				\
+  surrogate-search				\
   symlink					\
   triple-backref				\
   turkish-I					\
diff --git a/tests/surrogate-pair b/tests/surrogate-pair
index a91fa36..c1cbaba 100755
--- a/tests/surrogate-pair
+++ b/tests/surrogate-pair
@@ -51,10 +51,4 @@ case $io_pair in
   *) fail_ "unexpected output: $io_pair"; fail=1;;
 esac
 
-# Also test whether a surrogate-pair in the search string works.
-for opt in '' -i -E -F -iE -iF; do
-  grep --file=in $opt in > out 2>&1 || fail=1
-  compare out in || fail=1
-done
-
 Exit $fail
diff --git a/tests/surrogate-search b/tests/surrogate-search
new file mode 100755
index 0000000..2679c3c
--- /dev/null
+++ b/tests/surrogate-search
@@ -0,0 +1,42 @@
+#!/bin/sh
+# Check characters outside the Unicode BMP in the search string.
+
+# Copyright 2021 Free Software Foundation, Inc.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+. "${srcdir=.}/init.sh"; path_prepend_ ../src
+
+require_en_utf8_locale_
+require_compiled_in_MB_support
+
+case $(uname -s) in
+  CYGWIN*)
+    skip_ 'surrogate-pair search string on Cygwin, where it cannot work';;
+esac
+
+fail=0
+
+s_pair=$(printf '\360\220\220\205')
+printf '%s\n' "$s_pair" > in || framework_failure_
+
+LC_ALL=en_US.UTF-8
+export LC_ALL
+
+for opt in '' -i -E -F -iE -iF; do
+  grep --file=in $opt in > out 2>&1 || fail=1
+  compare out in || fail=1
+done
+
+Exit $fail
-- 
2.33.1

Subject: surrogate-pair test fails under Cygwin
From: Duncan Roe <[email-masked]>
To: [email-masked]
Cc: Duncan Roe <[email-masked]>
Date: Tue, 9 Nov 2021 13:48:02 +1100
List-Id: <debbugs-submit.debbugs.gnu.org>

The 3rd surrogate-pair test fails under Cygwin:
> # Also test whether a surrogate-pair in the search string works.
Fails at grep-3.7 or latest commit.

Reproduces easily enough from the command line:
> printf '%s\n' "$(printf '\360\220\220\205')" >in
> LANG=en_US.utf8
> locale
> src/grep --file=in in

Reports a match under Linux but not under Cygwin. Tested Cygwin64 on Windows 7
Home and Windows 10.

Comparing gdb sessions between the platforms, I noticed:
> linux:  sbclen = '\001' <repeats 128 times>, '\377' <repeats 66 times>, '\376' <repeats 60 times>, "\377\377"
> cygwin: sbclen = '\001' <repeats 128 times>, '\377' <repeats 64 times>, '\376' <repeats 53 times>, '\377' <repeats 11 times>
in `dfa` (i.e. dfa.localeinfo.sbclen).

Also this:
> linux:  enlistnew (cpp=0x, new=0x "\360\220\220\205") at dfa.c:3928
> cygwin: enlistnew (cpp=0x, new=0x "\360\355\260\205") at dfa.c:3928

Locale data is different for the same locale on the 2 systems. I investigated
this further by breakpointing the code as it starts to compute sbclen[250] which
is \376 ubder Linux but \377 under Cygwin. I captured the gdb sessions using
`script` and have attached them in the hope they are some help.

If your system rejects the tar.gz attachment I'll send them plaintext in
separate emails. They compare best in a side-by-side diff highlighting changed
characters. I find `tkdiff` good for this: from View choose "Show inline
comparison (recursive)".

Uninteresting changes between the sessions are removed:
 Automatic
 - strip hex numbers (addresses usually) to plain 0x
 - remove escape sequences (colouring &c.)
 - probably other stuff
 Specifics
 - force matching locale names
 - insert blank lines at linux:72 to line up return stmt
 - split linux:100 to more easily see later args

Cheers ... Duncan.