GNU bug report logs - #7155
[md5sum] does not accept

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 7155 in the body.
You can then email your comments to 7155 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Rimas Kudelis <rq <at> rq.lt>
To: bug-coreutils <at> gnu.org
Subject: [md5sum] does not accept
Date: Sun, 03 Oct 2010 22:24:31 +0300

Hi,

I have a little problem with md5sum.

A FreeBSD box generates an md5 sum of a file, which I'm later trying to
check on a Linux box. The problem is that what FreeBSD's md5 outputs is
slightly different from what Linux's md5sum expects, which makes md5sum
complain. The difference is really trivial: md5 outputs one space
between the sum and the file name, and md5sum outputs/expects two:

rq <at> garage:~/Pootle/tmp$ md5sum -c  catkeys.zip
md5sum: catkeys.zip: no properly formatted MD5 checksum lines found

rq <at> garage:~/Pootle/tmp$ md5sum -c  catkeys.zip.md5
md5sum: catkeys.zip.md5: no properly formatted MD5 checksum lines found

rq <at> garage:~/Pootle/tmp$ md5sum catkeys.zip
f653761af3137c8a631284b390bae102  catkeys.zip

rq <at> garage:~/Pootle/tmp$ cat catkeys.zip.md5
f653761af3137c8a631284b390bae102 catkeys.zip

I don't know which format is the "right" one, but could md5sum perhaps
be fixed to support md5's checksum files too?

Thanks!
Rimas Kudelis

Message #8 received at 7155 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Rimas Kudelis <rq <at> rq.lt>
Cc: 7155 <at> debbugs.gnu.org
Subject: Re: bug#7155: [md5sum] does not accept
Date: Sun, 03 Oct 2010 23:27:24 +0100

On 03/10/10 20:24, Rimas Kudelis wrote:
> Hi,
> 
> I have a little problem with md5sum.
> 
> A FreeBSD box generates an md5 sum of a file, which I'm later trying to
> check on a Linux box. The problem is that what FreeBSD's md5 outputs is
> slightly different from what Linux's md5sum expects, which makes md5sum
> complain. The difference is really trivial: md5 outputs one space
> between the sum and the file name, and md5sum outputs/expects two:

md5 seems to output a different format here.

$ head -n1 /etc/motd
FreeBSD 8.0-RELEASE-p3 (GENERIC) #0: Wed May 26 05:45:12 UTC 2010
$ md5sum --version | head -n1
md5sum (GNU coreutils) 8.3
$ md5 file | tee t.md5
MD5 (file) = b85d6fb9ef4260dcf1ce0a1b0bff80d3
$ md5sum -c t.md5
file: OK

Could you verify what md5 utility you're using exactly.

cheers,
Pádraig.

Message #11 received at 7155 <at> debbugs.gnu.org (full text, mbox):

From: Rimas Kudelis <rq <at> rq.lt>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: 7155 <at> debbugs.gnu.org
Subject: Re: bug#7155: [md5sum] does not accept
Date: Sat, 16 Oct 2010 22:37:01 +0300

On Sun, 03 Oct 2010 23:27:24 +0100, Pádraig Brady <P <at> draigBrady.com>
wrote:
> On 03/10/10 20:24, Rimas Kudelis wrote:
>> Hi,
>> 
>> I have a little problem with md5sum.
>> 
>> A FreeBSD box generates an md5 sum of a file, which I'm later trying to
>> check on a Linux box. The problem is that what FreeBSD's md5 outputs is
>> slightly different from what Linux's md5sum expects, which makes md5sum
>> complain. The difference is really trivial: md5 outputs one space
>> between the sum and the file name, and md5sum outputs/expects two:
> 
> md5 seems to output a different format here.
> 
> $ head -n1 /etc/motd
> FreeBSD 8.0-RELEASE-p3 (GENERIC) #0: Wed May 26 05:45:12 UTC 2010
> $ md5sum --version | head -n1
> md5sum (GNU coreutils) 8.3
> $ md5 file | tee t.md5
> MD5 (file) = b85d6fb9ef4260dcf1ce0a1b0bff80d3
> $ md5sum -c t.md5
> file: OK
> 
> Could you verify what md5 utility you're using exactly.


Sorry for taking so long to answer, but I wasn't the person producing the
checksum, so I had to ask too. The command used to produce the checksum is:
$ md5 -r <filename>

FreeBSD release version is the same as yours. I've just tested the same
command with FreeBSD 6.2, and it only outputs one space too.

Rimas

Message #14 received at 7155 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Rimas Kudelis <rq <at> rq.lt>
Cc: 7155 <at> debbugs.gnu.org
Subject: Re: bug#7155: [md5sum] does not accept
Date: Sat, 16 Oct 2010 23:37:26 +0100

On 16/10/10 20:37, Rimas Kudelis wrote:
> 
> On Sun, 03 Oct 2010 23:27:24 +0100, Pádraig Brady <P <at> draigBrady.com>
> wrote:
>> On 03/10/10 20:24, Rimas Kudelis wrote:
>>> Hi,
>>>
>>> I have a little problem with md5sum.
>>>
>>> A FreeBSD box generates an md5 sum of a file, which I'm later trying to
>>> check on a Linux box. The problem is that what FreeBSD's md5 outputs is
>>> slightly different from what Linux's md5sum expects, which makes md5sum
>>> complain. The difference is really trivial: md5 outputs one space
>>> between the sum and the file name, and md5sum outputs/expects two:
>>
>> md5 seems to output a different format here.
>>
>> $ head -n1 /etc/motd
>> FreeBSD 8.0-RELEASE-p3 (GENERIC) #0: Wed May 26 05:45:12 UTC 2010
>> $ md5sum --version | head -n1
>> md5sum (GNU coreutils) 8.3
>> $ md5 file | tee t.md5
>> MD5 (file) = b85d6fb9ef4260dcf1ce0a1b0bff80d3
>> $ md5sum -c t.md5
>> file: OK
>>
>> Could you verify what md5 utility you're using exactly.
> 
> 
> Sorry for taking so long to answer, but I wasn't the person producing the
> checksum, so I had to ask too. The command used to produce the checksum is:
> $ md5 -r <filename>
> 
> FreeBSD release version is the same as yours. I've just tested the same
> command with FreeBSD 6.2, and it only outputs one space too.

Ah right, md5 -r produces the alternate format.
I suppose we could support a single space,
by trying to open("*abc") after trying to open ("abc").
There is still an ambiguity if both files are present,
though that is unlikely. I'll have a look.

cheers,
Pádraig.

Message #17 received at 7155 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Rimas Kudelis <rq <at> rq.lt>
Cc: 7155 <at> debbugs.gnu.org
Subject: Re: bug#7155: [md5sum] does not accept
Date: Wed, 14 Sep 2011 13:35:13 +0100

severity 7155 wishlist
tags 7155 + notabug

Comments below.

On 10/16/2010 11:37 PM, Pádraig Brady wrote:
> On 16/10/10 20:37, Rimas Kudelis wrote:
>>
>> On Sun, 03 Oct 2010 23:27:24 +0100, Pádraig Brady <P <at> draigBrady.com>
>> wrote:
>>> On 03/10/10 20:24, Rimas Kudelis wrote:
>>>> Hi,
>>>>
>>>> I have a little problem with md5sum.
>>>>
>>>> A FreeBSD box generates an md5 sum of a file, which I'm later trying to
>>>> check on a Linux box. The problem is that what FreeBSD's md5 outputs is
>>>> slightly different from what Linux's md5sum expects, which makes md5sum
>>>> complain. The difference is really trivial: md5 outputs one space
>>>> between the sum and the file name, and md5sum outputs/expects two:
>>>
>>> md5 seems to output a different format here.
>>>
>>> $ head -n1 /etc/motd
>>> FreeBSD 8.0-RELEASE-p3 (GENERIC) #0: Wed May 26 05:45:12 UTC 2010
>>> $ md5sum --version | head -n1
>>> md5sum (GNU coreutils) 8.3
>>> $ md5 file | tee t.md5
>>> MD5 (file) = b85d6fb9ef4260dcf1ce0a1b0bff80d3
>>> $ md5sum -c t.md5
>>> file: OK
>>>
>>> Could you verify what md5 utility you're using exactly.
>>
>>
>> Sorry for taking so long to answer, but I wasn't the person producing the
>> checksum, so I had to ask too. The command used to produce the checksum is:
>> $ md5 -r <filename>
>>
>> FreeBSD release version is the same as yours. I've just tested the same
>> command with FreeBSD 6.2, and it only outputs one space too.
> 
> Ah right, md5 -r produces the alternate format.
> I suppose we could support a single space,
> by trying to open("*abc") after trying to open ("abc").
> There is still an ambiguity if both files are present,
> though that is unlikely. I'll have a look.

Thinking more about this, there is a bit of a
security issue with mixing both formats.
Consider the case currently with a checksum in BSD format
of ' important_file' (with leading space).

b85d6fb9ef4260dcf1ce0a1b0bff80d3  important_file

Now an attacker does:

mv ' important_file' important_file
cp trojan ' important_file'

And since coreutils will check 'important_file' first,
then we'll not report any errors.

If coreutils supported both formats then this would be compounded.
I suppose we could mitigate that somewhat by detecting and
supporting only a single format per run, as done in the following diff.

Note we don't handle the case where the first or only
entry in a BSD format checksum file has a file name with a
leading ' ' or '*'.  We could support this and avoid detection
with an option to specify BSD format checksums, but
I don't think that's warranted. Note we could detect
in this situation too by retrying the open with the
leading char included, but that would introduce a security
issue. Consider the following standard format checksum file:

b85d6fb9ef4260dcf1ce0a1b0bff80d3  firewall_rules

Attackers could then do this undetected:

mv firewall_rules ' firewall_rules'

cheers,
Pádraig.

diff --git a/src/md5sum.c b/src/md5sum.c
index ff9538a..9de34a5 100644
--- a/src/md5sum.c
+++ b/src/md5sum.c
@@ -99,7 +99,7 @@
    not include any newline character at the end of a line.  */
 #define MIN_DIGEST_LINE_LENGTH \
   (DIGEST_HEX_BYTES /* length of hexadecimal message digest */ \
-   + 2 /* blank and binary indicator */ \
+   + 1 /* blank */ \
    + 1 /* minimum filename length */ )

 /* True if any of the files read were the standard input. */
@@ -126,6 +126,9 @@ static bool quiet = false;
    improperly formatted. */
 static bool strict = false;

+/* Whether a BSD reversed format checksum is detected.  */
+static int bsd_reversed = -1;
+
 /* For long options that have no equivalent short option, use a
    non-character as a pseudo short option, starting with CHAR_MAX + 1.  */
 enum
@@ -307,9 +310,24 @@ split_3 (char *s, size_t s_len,

   s[i++] = '\0';

-  if (s[i] != ' ' && s[i] != '*')
-    return false;
-  *binary = (s[i++] == '*');
+  /* If "bsd reversed" format detected.  */
+  if ((s_len - i == 1) || (s[i] != ' ' && s[i] != '*'))
+    {
+      /* Don't allow mixing bsd and standard formats,
+         to minimize security issues with attackers
+         renaming files with leading spaces.
+         This assumes that with bsd format checksums
+         that the first file name does not have
+         a leading ' ' or '*'.  */
+      if (bsd_reversed == 0)
+        return false;
+      bsd_reversed = 1;
+    }
+  else if (bsd_reversed != 1)
+    {
+      bsd_reversed = 0;
+      *binary = (s[i++] == '*');
+    }

   /* All characters between the type indicator and end of line are
      significant -- that includes leading and trailing white space.  */

Message #20 received at 7155 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: 7155 <at> debbugs.gnu.org, Rimas Kudelis <rq <at> rq.lt>
Subject: Re: bug#7155: [md5sum] does not accept
Date: Wed, 14 Sep 2011 16:12:58 +0200

Pádraig Brady wrote:
> severity 7155 wishlist
> tags 7155 + notabug
>
> Comments below.
>
> On 10/16/2010 11:37 PM, Pádraig Brady wrote:
>> On 16/10/10 20:37, Rimas Kudelis wrote:
>>>
>>> On Sun, 03 Oct 2010 23:27:24 +0100, Pádraig Brady <P <at> draigBrady.com>
>>> wrote:
>>>> On 03/10/10 20:24, Rimas Kudelis wrote:
>>>>> Hi,
>>>>>
>>>>> I have a little problem with md5sum.
>>>>>
>>>>> A FreeBSD box generates an md5 sum of a file, which I'm later trying to
>>>>> check on a Linux box. The problem is that what FreeBSD's md5 outputs is
>>>>> slightly different from what Linux's md5sum expects, which makes md5sum
>>>>> complain. The difference is really trivial: md5 outputs one space
>>>>> between the sum and the file name, and md5sum outputs/expects two:
>>>>
>>>> md5 seems to output a different format here.
>>>>
>>>> $ head -n1 /etc/motd
>>>> FreeBSD 8.0-RELEASE-p3 (GENERIC) #0: Wed May 26 05:45:12 UTC 2010
>>>> $ md5sum --version | head -n1
>>>> md5sum (GNU coreutils) 8.3
>>>> $ md5 file | tee t.md5
>>>> MD5 (file) = b85d6fb9ef4260dcf1ce0a1b0bff80d3
>>>> $ md5sum -c t.md5
>>>> file: OK
>>>>
>>>> Could you verify what md5 utility you're using exactly.
>>>
>>>
>>> Sorry for taking so long to answer, but I wasn't the person producing the
>>> checksum, so I had to ask too. The command used to produce the checksum is:
>>> $ md5 -r <filename>
>>>
>>> FreeBSD release version is the same as yours. I've just tested the same
>>> command with FreeBSD 6.2, and it only outputs one space too.
>>
>> Ah right, md5 -r produces the alternate format.
>> I suppose we could support a single space,
>> by trying to open("*abc") after trying to open ("abc").
>> There is still an ambiguity if both files are present,
>> though that is unlikely. I'll have a look.
>
> Thinking more about this, there is a bit of a
> security issue with mixing both formats.
> Consider the case currently with a checksum in BSD format
> of ' important_file' (with leading space).
>
> b85d6fb9ef4260dcf1ce0a1b0bff80d3  important_file
>
> Now an attacker does:
>
> mv ' important_file' important_file
> cp trojan ' important_file'
>
> And since coreutils will check 'important_file' first,
> then we'll not report any errors.
>
> If coreutils supported both formats then this would be compounded.
> I suppose we could mitigate that somewhat by detecting and
> supporting only a single format per run, as done in the following diff.
>
> Note we don't handle the case where the first or only
> entry in a BSD format checksum file has a file name with a
> leading ' ' or '*'.  We could support this and avoid detection
> with an option to specify BSD format checksums, but
> I don't think that's warranted. Note we could detect
> in this situation too by retrying the open with the
> leading char included, but that would introduce a security
> issue. Consider the following standard format checksum file:
>
> b85d6fb9ef4260dcf1ce0a1b0bff80d3  firewall_rules
>
> Attackers could then do this undetected:
>
> mv firewall_rules ' firewall_rules'

Good point.
I don't see a way to make GNU md5sum handle this automatically
and safely.  However, that's not a big deal: it's easy to
convert from one format to the other using sed or perl.

I'd be inclined to mark this "wontfix" (because we cannot)
or simply to close it.

Message #23 received at 7155 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Jim Meyering <jim <at> meyering.net>
Cc: 7155 <at> debbugs.gnu.org, Rimas Kudelis <rq <at> rq.lt>
Subject: Re: bug#7155: [md5sum] does not accept
Date: Wed, 14 Sep 2011 16:13:03 +0100

On 09/14/2011 03:12 PM, Jim Meyering wrote:
> Pádraig Brady wrote:
>> severity 7155 wishlist
>> tags 7155 + notabug
>>
>> Comments below.
>>
>> On 10/16/2010 11:37 PM, Pádraig Brady wrote:
>>> On 16/10/10 20:37, Rimas Kudelis wrote:
>>>>
>>>> On Sun, 03 Oct 2010 23:27:24 +0100, Pádraig Brady <P <at> draigBrady.com>
>>>> wrote:
>>>>> On 03/10/10 20:24, Rimas Kudelis wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I have a little problem with md5sum.
>>>>>>
>>>>>> A FreeBSD box generates an md5 sum of a file, which I'm later trying to
>>>>>> check on a Linux box. The problem is that what FreeBSD's md5 outputs is
>>>>>> slightly different from what Linux's md5sum expects, which makes md5sum
>>>>>> complain. The difference is really trivial: md5 outputs one space
>>>>>> between the sum and the file name, and md5sum outputs/expects two:
>>>>>
>>>>> md5 seems to output a different format here.
>>>>>
>>>>> $ head -n1 /etc/motd
>>>>> FreeBSD 8.0-RELEASE-p3 (GENERIC) #0: Wed May 26 05:45:12 UTC 2010
>>>>> $ md5sum --version | head -n1
>>>>> md5sum (GNU coreutils) 8.3
>>>>> $ md5 file | tee t.md5
>>>>> MD5 (file) = b85d6fb9ef4260dcf1ce0a1b0bff80d3
>>>>> $ md5sum -c t.md5
>>>>> file: OK
>>>>>
>>>>> Could you verify what md5 utility you're using exactly.
>>>>
>>>>
>>>> Sorry for taking so long to answer, but I wasn't the person producing the
>>>> checksum, so I had to ask too. The command used to produce the checksum is:
>>>> $ md5 -r <filename>
>>>>
>>>> FreeBSD release version is the same as yours. I've just tested the same
>>>> command with FreeBSD 6.2, and it only outputs one space too.
>>>
>>> Ah right, md5 -r produces the alternate format.
>>> I suppose we could support a single space,
>>> by trying to open("*abc") after trying to open ("abc").
>>> There is still an ambiguity if both files are present,
>>> though that is unlikely. I'll have a look.
>>
>> Thinking more about this, there is a bit of a
>> security issue with mixing both formats.
>> Consider the case currently with a checksum in BSD format
>> of ' important_file' (with leading space).
>>
>> b85d6fb9ef4260dcf1ce0a1b0bff80d3  important_file
>>
>> Now an attacker does:
>>
>> mv ' important_file' important_file
>> cp trojan ' important_file'
>>
>> And since coreutils will check 'important_file' first,
>> then we'll not report any errors.
>>
>> If coreutils supported both formats then this would be compounded.
>> I suppose we could mitigate that somewhat by detecting and
>> supporting only a single format per run, as done in the following diff.
>>
>> Note we don't handle the case where the first or only
>> entry in a BSD format checksum file has a file name with a
>> leading ' ' or '*'.  We could support this and avoid detection
>> with an option to specify BSD format checksums, but
>> I don't think that's warranted. Note we could detect
>> in this situation too by retrying the open with the
>> leading char included, but that would introduce a security
>> issue. Consider the following standard format checksum file:
>>
>> b85d6fb9ef4260dcf1ce0a1b0bff80d3  firewall_rules
>>
>> Attackers could then do this undetected:
>>
>> mv firewall_rules ' firewall_rules'
> 
> Good point.
> I don't see a way to make GNU md5sum handle this automatically
> and safely.  However, that's not a big deal: it's easy to
> convert from one format to the other using sed or perl.
> 
> I'd be inclined to mark this "wontfix" (because we cannot)
> or simply to close it.

Yes, it's debatable.

To summarize, the only caveat with my patch I think is that
it will give a false error for BSD format checksums
where the first entry has a file name starting with ' ' or '*'.
That should be exceedingly rare though, and is a lot better
than a false OK.
Also in this case even without the patch, we're susceptible
to the 'trojan' case above.

The workaround is easy as you suggest:

  sed 's/ /  /' files.md5 | md5sum -c

However that is not easily discoverable.
I'm 50:50, so I'll think a bit more.
Hmm I might just document in info that
the checksum utilities are compatible with
the BSD ones when processed like:

  md5 -r files... | sed 's/ /  /' > files.md5

cheers,
Pádraig.

Message #26 received at 7155 <at> debbugs.gnu.org (full text, mbox):

From: Rimas Kudelis <rq <at> rq.lt>
To: Jim Meyering <jim <at> meyering.net>, 7155 <at> debbugs.gnu.org
Subject: Re: bug#7155: [md5sum] does not accept
Date: Thu, 15 Sep 2011 01:43:18 +0300

2011.09.14 17:12, Jim Meyering rašė:
> Good point. I don't see a way to make GNU md5sum handle this 
> automatically and safely. However, that's not a big deal: it's easy to 
> convert from one format to the other using sed or perl. I'd be 
> inclined to mark this "wontfix" (because we cannot) or simply to close 
> it. 


I think you could also implement this as a commandline switch, right?

Rimas

Message #29 received at 7155 <at> debbugs.gnu.org (full text, mbox):

From: Jim Meyering <jim <at> meyering.net>
To: Pádraig Brady <P <at> draigBrady.com>
Cc: 7155 <at> debbugs.gnu.org, Rimas Kudelis <rq <at> rq.lt>
Subject: Re: bug#7155: [md5sum] does not accept
Date: Fri, 16 Sep 2011 09:29:07 +0200

Pádraig Brady wrote:
> To summarize, the only caveat with my patch I think is that
> it will give a false error for BSD format checksums
> where the first entry has a file name starting with ' ' or '*'.
> That should be exceedingly rare though, and is a lot better
> than a false OK.

Oh.  I misunderstood.
We already accept one BSD variant:

  MD5 (f) = d41d8cd98f00b204e9800998ecf8427e

so I guess it's hard to reject another.

> Also in this case even without the patch, we're susceptible
> to the 'trojan' case above.
>
> The workaround is easy as you suggest:
>
>   sed 's/ /  /' files.md5 | md5sum -c
>
> However that is not easily discoverable.
> I'm 50:50, so I'll think a bit more.
> Hmm I might just document in info that
> the checksum utilities are compatible with
> the BSD ones when processed like:
>
>   md5 -r files... | sed 's/ /  /' > files.md5

Whichever you prefer.

Message #34 received at 7155-done <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Rimas Kudelis <rq <at> rq.lt>
Cc: 7155-done <at> debbugs.gnu.org
Subject: Re: bug#7155: [md5sum] does not accept
Date: Fri, 16 Sep 2011 13:54:17 +0100

[Message part 1 (text/plain, inline)]

On 09/14/2011 11:43 PM, Rimas Kudelis wrote:
> 2011.09.14 17:12, Jim Meyering rašė:
>> Good point. I don't see a way to make GNU md5sum handle this automatically and safely. However, that's not a big deal: it's easy to convert from one format to the other using sed or perl. I'd be inclined to mark this "wontfix" (because we cannot) or simply to close it. 
> 
> 
> I think you could also implement this as a commandline switch, right?

Right. But that complicates the interface unnecessarily I think.

I'm going to apply the attached to give automatic support,
with the aforementioned caveat that BSD -r format checksums
where the _first_ file starts with a ' ' or '*' are not supported.
In that very unlikely case, one can always preprocess with sed 's/ /  /'

cheers,
Pádraig.

[md5sum-bsd.diff (text/plain, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.