GNU bug report logs - #29802
"uniq -c" doesn't like counting lines with nulls

Previous Next

Full log

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "PD" <bug-bash.gnu.org <at> pkts.ca>
To: bug-coreutils <at> gnu.org
Subject: "uniq -c" doesn't like counting lines with nulls
Date: Thu, 21 Dec 2017 00:40:34 -0800

Uniq *sometimes* fails to combine lines containing a null character:

# uniq --version
uniq (GNU coreutils) 8.4

##### Count duplicate text lines:
# printf "\n\x00\n\x00\n" | cat -e | uniq -c
      1 $
      2 ^@$

##### Count duplicate binary lines:
# printf "\x00\n\x00\n\n" | uniq -c | cat -e
      2 ^@$
      1 $

##### Whoops, fail to count duplicate binary lines:
# printf "\n\x00\n\x00\n" | uniq -c | cat -e
      1 $
      1 ^@$
      1 ^@$

This was the smallest test case; the original file had hundreds of lines
with nulls (\x00) and Ctrl-A (\x01) characters, and it was quite a
surprise when the output of 'sort testfile | uniq -c' had many pages of '1
^@$' followed by '496 ^A$': it was counting the Ctrl-A lines correctly,
but failing on the null-character lines.

For automated testing with 'delta' or 'git bisect', this works:
---
#!/bin/bash
a=$(sort $1 | cat -e | uniq -c | md5sum -)
b=$(sort $1 | uniq -c | cat -e | md5sum -)
if [[ "$a" != "$b" ]]; then
  echo "PASS (bug present)"; exit 0
else
  echo "FAIL (bug absent)"; exit 1
fi
----

I regret not having the time to test this with coreutils 8.28, but I
couldn't see anything in the git log to suggest this has been fixed:
http://git.savannah.gnu.org/gitweb/?p=coreutils.git;a=history;f=src/uniq.c;h=d1dac93c010d7333ced4b54fccbd965cbd5729c2;hb=HEAD

Cheers,
PD

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.