Subject: bug#26263: closed (Re: bug#26263: sha256sum does not correctly parse file with PGP message header (coreutils 8.26))

From: [email-masked] (GNU bug Tracking System)
Date: Mon, 27 Mar 2017 00:15:01 +0000

Your bug report

#26263: sha256sum does not correctly parse file with PGP message header (coreutils 8.26)

which was filed against the coreutils package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
26263: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=26263
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#26263: sha256sum does not correctly parse file with PGP message header (coreutils 8.26)
From: Pádraig Brady <[email-masked]>
To: Toby Kelsey <[email-masked]>, [email-masked]
Date: Sun, 26 Mar 2017 17:13:56 -0700
List-Id: <debbugs-submit.debbugs.gnu.org>

On 26/03/17 03:09, Toby Kelsey wrote:
> I notice sha256sum does not correctly parse a file with a PGP header. Here is a script log:

> toby@toby-pc:/tmp/test$ sha256sum -c hashes.txt
> 
> sha256sum: ' monero-win-x64-v0.10.3.0.zip': No such file or directory
>   monero-win-x64-v0.10.3.0.zip: FAILED open or read

> There is a spurious extra space at the start of filenames in the larger file.
> If I just extract the hash lines it works:

> The hash file is from <https://getmonero.org/downloads/hashes.txt>

The attached should fix this issue.

thanks,
Pádraig

>From 1a1029ea037d70d13eef05d29d17980fac729e2a Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= <[email-masked]>
Date: Sun, 26 Mar 2017 17:04:36 -0700
Subject: [PATCH] md5sum,b2sum,sha*sum: don't erroneously trigger BSD reversed
 mode

* src/md5sum.c (split_3): Verify hex digits internally before
triggering the global bsd_reversed mode flag.
(bsd_split_3): Likewise.
* tests/misc/md5sum-bsd.sh: Add a test case.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/26263
---
 NEWS                     |  4 ++++
 src/md5sum.c             | 39 +++++++++++++++++++++------------------
 tests/misc/md5sum-bsd.sh |  7 +++++++
 3 files changed, 32 insertions(+), 18 deletions(-)

diff --git a/NEWS b/NEWS
index e8d6d34..4604318 100644
--- a/NEWS
+++ b/NEWS
@@ -7,6 +7,10 @@ GNU coreutils NEWS                                    -*- outline -*-
   split no longer exits when invocations of a --filter return EPIPE.
   [bug introduced in coreutils-8.26]
 
+  md5sum --check no longer incorrectly enables BSD reversed format mode when
+  ignoring some non checksum lines.  This also affects sha*sum and b2sum.
+  [bug introduced in coreutils-8.14]
+
 
 * Noteworthy changes in release 8.27 (2017-03-08) [stable]
 
diff --git a/src/md5sum.c b/src/md5sum.c
index e58a68e..91cdfb2 100644
--- a/src/md5sum.c
+++ b/src/md5sum.c
@@ -346,6 +346,21 @@ filename_unescape (char *s, size_t s_len)
   return s;
 }
 
+/* Return true if S is a NUL-terminated string of DIGEST_HEX_BYTES hex digits.
+   Otherwise, return false.  */
+static bool _GL_ATTRIBUTE_PURE
+hex_digits (unsigned char const *s)
+{
+  unsigned int i;
+  for (i = 0; i < digest_hex_bytes; i++)
+    {
+      if (!isxdigit (*s))
+        return false;
+      ++s;
+    }
+  return *s == '\0';
+}
+
 /* Split the checksum string S (of length S_LEN) from a BSD 'md5' or
    'sha1' command into two parts: a hexadecimal digest, and the file
    name.  S is modified.  Return true if successful.  */
@@ -386,7 +401,8 @@ bsd_split_3 (char *s, size_t s_len, unsigned char **hex_digest,
     i++;
 
   *hex_digest = (unsigned char *) &s[i];
-  return true;
+
+  return hex_digits (*hex_digest);
 }
 
 /* Split the string S (of length S_LEN) into three parts:
@@ -492,6 +508,9 @@ split_3 (char *s, size_t s_len,
 
   s[i++] = '\0';
 
+  if (! hex_digits (*hex_digest))
+    return false;
+
   /* If "bsd reversed" format detected.  */
   if ((s_len - i == 1) || (s[i] != ' ' && s[i] != '*'))
     {
@@ -521,21 +540,6 @@ split_3 (char *s, size_t s_len,
   return true;
 }
 
-/* Return true if S is a NUL-terminated string of DIGEST_HEX_BYTES hex digits.
-   Otherwise, return false.  */
-static bool _GL_ATTRIBUTE_PURE
-hex_digits (unsigned char const *s)
-{
-  unsigned int i;
-  for (i = 0; i < digest_hex_bytes; i++)
-    {
-      if (!isxdigit (*s))
-        return false;
-      ++s;
-    }
-  return *s == '\0';
-}
-
 /* If ESCAPE is true, then translate each NEWLINE byte to the string, "\\n",
    and each backslash to "\\\\".  */
 static void
@@ -702,8 +706,7 @@ digest_check (const char *checkfile_name)
         line[--line_length] = '\0';
 
       if (! (split_3 (line, line_length, &hex_digest, &binary, &filename)
-             && ! (is_stdin && STREQ (filename, "-"))
-             && hex_digits (hex_digest)))
+             && ! (is_stdin && STREQ (filename, "-"))))
         {
           ++n_misformatted_lines;
 
diff --git a/tests/misc/md5sum-bsd.sh b/tests/misc/md5sum-bsd.sh
index e2ad8db..dfd2cef 100755
--- a/tests/misc/md5sum-bsd.sh
+++ b/tests/misc/md5sum-bsd.sh
@@ -36,6 +36,13 @@ sed 's/  / /' check.md5sum > check.md5
 md5sum --strict -c check.md5sum || fail=1
 md5sum --strict -c check.md5 || fail=1
 
+# Ensure we don't trigger BSD reversed format with GPG headers etc.
+echo '____not_all_hex_so_no_match_____ blah' > check2.md5sum
+cat check.md5sum >> check2.md5sum
+md5sum -c check2.md5sum 2>check2.err || fail=1
+echo 'md5sum: WARNING: 1 line is improperly formatted' >check2.exp
+compare check2.exp check2.err || fail=1
+
 # If we skip the first entry in the BSD format checksums
 # then it'll be detected as standard format and error.
 # This unlikely caveat was thought better than mandating
-- 
2.9.3

Subject: sha256sum does not correctly parse file with PGP message header (coreutils 8.26)
From: Toby Kelsey <[email-masked]>
To: [email-masked]
Date: Sun, 26 Mar 2017 11:09:10 +0100
List-Id: <debbugs-submit.debbugs.gnu.org>

I notice sha256sum does not correctly parse a file with a PGP header. Here is a script log:

toby@toby-pc:/tmp/test$ ls -l
total 57068
-rw-r--r-- 1 toby toby     2046 Mar 26 10:30 hashes.txt
-rw-r--r-- 1 toby toby 58432932 Mar 26 10:30 monero-linux-x64-v0.10.3.0.tar.bz2

toby@toby-pc:/tmp/test$ grep linux-x64 hashes.txt > single.txt

toby@toby-pc:/tmp/test$ cat single.txt
f164112bea75dfd3eaaa905d21129820a996bbe6e5d65cb194609c672a71c3fe  monero-linux-x64-v0.10.3.0.tar.bz2

toby@toby-pc:/tmp/test$ sha256sum -c single.txt
monero-linux-x64-v0.10.3.0.tar.bz2: OK

toby@toby-pc:/tmp/test$ sha256sum -c hashes.txt

sha256sum: ' monero-win-x64-v0.10.3.0.zip': No such file or directory
  monero-win-x64-v0.10.3.0.zip: FAILED open or read
sha256sum: ' monero-win-x86-v0.10.3.0.zip': No such file or directory
  monero-win-x86-v0.10.3.0.zip: FAILED open or read
sha256sum: ' monero-mac-x64-v0.10.3.0.tar.bz2': No such file or directory
  monero-mac-x64-v0.10.3.0.tar.bz2: FAILED open or read
sha256sum: ' monero-linux-x64-v0.10.3.0.tar.bz2': No such file or directory
  monero-linux-x64-v0.10.3.0.tar.bz2: FAILED open or read
sha256sum: ' monero-linux-x86-v0.10.3.0.tar.bz2': No such file or directory
  monero-linux-x86-v0.10.3.0.tar.bz2: FAILED open or read
sha256sum: ' monero-linux-armv7-v0.10.3.0.tar.bz2': No such file or directory
  monero-linux-armv7-v0.10.3.0.tar.bz2: FAILED open or read
sha256sum: ' monero-linux-armv8-v0.10.3.0.tar.bz2': No such file or directory
  monero-linux-armv8-v0.10.3.0.tar.bz2: FAILED open or read
sha256sum: ' monero-freebsd-x64-v0.10.3.0.tar.bz2': No such file or directory
  monero-freebsd-x64-v0.10.3.0.tar.bz2: FAILED open or read
sha256sum: ' monero-dragonflybsd-x64-v0.10.3.0.tar.bz2': No such file or directory
  monero-dragonflybsd-x64-v0.10.3.0.tar.bz2: FAILED open or read
sha256sum: ' monero.gui.win.x64.beta.zip': No such file or directory
  monero.gui.win.x64.beta.zip: FAILED open or read
sha256sum: ' monero.gui.mac.x64.beta.tar.bz2': No such file or directory
  monero.gui.mac.x64.beta.tar.bz2: FAILED open or read
sha256sum: ' monero.gui.linux.x64.beta.tar.bz2': No such file or directory
  monero.gui.linux.x64.beta.tar.bz2: FAILED open or read
sha256sum: ' monero.gui.linux.x86.beta.tar.bz2': No such file or directory
  monero.gui.linux.x86.beta.tar.bz2: FAILED open or read
sha256sum: WARNING: 22 lines are improperly formatted
sha256sum: WARNING: 13 listed files could not be read

There is a spurious extra space at the start of filenames in the larger file.
If I just extract the hash lines it works:

toby@toby-pc:/tmp/test$ egrep zip\|bz2 hashes.txt > hashes1.txt

toby@toby-pc:/tmp/test$ sha256sum -c hashes1.txt

sha256sum: monero-win-x64-v0.10.3.0.zip: No such file or directory
monero-win-x64-v0.10.3.0.zip: FAILED open or read
sha256sum: monero-win-x86-v0.10.3.0.zip: No such file or directory
monero-win-x86-v0.10.3.0.zip: FAILED open or read
sha256sum: monero-mac-x64-v0.10.3.0.tar.bz2: No such file or directory
monero-mac-x64-v0.10.3.0.tar.bz2: FAILED open or read
monero-linux-x64-v0.10.3.0.tar.bz2: OK
sha256sum: monero-linux-x86-v0.10.3.0.tar.bz2: No such file or directory
monero-linux-x86-v0.10.3.0.tar.bz2: FAILED open or read
sha256sum: monero-linux-armv7-v0.10.3.0.tar.bz2: No such file or directory
monero-linux-armv7-v0.10.3.0.tar.bz2: FAILED open or read
sha256sum: monero-linux-armv8-v0.10.3.0.tar.bz2: No such file or directory
monero-linux-armv8-v0.10.3.0.tar.bz2: FAILED open or read
sha256sum: monero-freebsd-x64-v0.10.3.0.tar.bz2: No such file or directory
monero-freebsd-x64-v0.10.3.0.tar.bz2: FAILED open or read
sha256sum: monero-dragonflybsd-x64-v0.10.3.0.tar.bz2: No such file or directory
monero-dragonflybsd-x64-v0.10.3.0.tar.bz2: FAILED open or read
sha256sum: monero.gui.win.x64.beta.zip: No such file or directory
monero.gui.win.x64.beta.zip: FAILED open or read
sha256sum: monero.gui.mac.x64.beta.tar.bz2: No such file or directory
monero.gui.mac.x64.beta.tar.bz2: FAILED open or read
sha256sum: monero.gui.linux.x64.beta.tar.bz2: No such file or directory
monero.gui.linux.x64.beta.tar.bz2: FAILED open or read
sha256sum: monero.gui.linux.x86.beta.tar.bz2: No such file or directory
monero.gui.linux.x86.beta.tar.bz2: FAILED open or read
sha256sum: WARNING: 12 listed files could not be read

The hash file is from <https://getmonero.org/downloads/hashes.txt>

Regards,
Toby