From unknown Wed Jun 25 02:10:38 2025 X-Loop: help-debbugs@gnu.org Subject: bug#10355: Add an option to {md5,sha*} to ignore directories Resent-From: "Gilles Espinasse" Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Fri, 23 Dec 2011 13:47:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 10355 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: 10355@debbugs.gnu.org X-Debbugs-Original-To: Received: via spool by submit@debbugs.gnu.org id=B.13246479694797 (code B ref -1); Fri, 23 Dec 2011 13:47:02 +0000 Received: (at submit) by debbugs.gnu.org; 23 Dec 2011 13:46:09 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Re5Ro-0001FK-T7 for submit@debbugs.gnu.org; Fri, 23 Dec 2011 08:46:09 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Re5Rm-0001FD-G7 for submit@debbugs.gnu.org; Fri, 23 Dec 2011 08:46:07 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Re5PZ-0001Om-Oy for submit@debbugs.gnu.org; Fri, 23 Dec 2011 08:43:50 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.2 Received: from lists.gnu.org ([140.186.70.17]:48298) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Re5PZ-0001Oi-NV for submit@debbugs.gnu.org; Fri, 23 Dec 2011 08:43:49 -0500 Received: from eggs.gnu.org ([140.186.70.92]:53081) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Re5PY-0004gk-T3 for bug-coreutils@gnu.org; Fri, 23 Dec 2011 08:43:49 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Re5PS-0001No-Q6 for bug-coreutils@gnu.org; Fri, 23 Dec 2011 08:43:48 -0500 Received: from smtp3-g21.free.fr ([212.27.42.3]:56350) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Re5PS-0001Mx-3u for bug-coreutils@gnu.org; Fri, 23 Dec 2011 08:43:42 -0500 Received: from pii350 (unknown [82.236.101.3]) by smtp3-g21.free.fr (Postfix) with SMTP id 71E5AA6294 for ; Fri, 23 Dec 2011 14:43:33 +0100 (CET) Message-ID: <082801ccc179$17d154a0$f9b5a8c0@pii350> From: "Gilles Espinasse" Date: Fri, 23 Dec 2011 14:45:10 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.2001 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.2001 X-Antivirus: avast! (VPS 111223-0, 23/12/2011), Outbound message X-Antivirus-Status: Clean X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 140.186.70.17 X-Spam-Score: -4.7 (----) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.7 (----) I was using a way to check md5sum on a lot of file using for myfile in `cat ${ALLFILES}`; do if [ -f /${myfile} ]; then md5sum /$myfile >> $ALLFILES}.md5; fi; done But this is slow, comparing with xargs md5sum way. time (for myfile in `cat ${ALLFILES}`; do if [ -f /${myfile} ]; then md5sum /$myfile >> ${ALLFILES}.md5; fi; done) real 0m26.907s user 0m40.019s sys 0m10.253s This is faster using xargs md5sum. time (sed -e '/.\/$/d' -e 's|^.|/&|g' ${ALLFILES} | xargs md5sum >${ALLFILES}.md5) md5sum: /etc/ipsec.d/cacerts: Is a directory md5sum: /etc/ipsec.d/certs: Is a directory md5sum: /etc/ipsec.d/crls: Is a directory md5sum: /etc/ppp/chap-secrets: No such file or directory md5sum: /etc/ppp/pap-secrets: No such file or directory md5sum: /etc/squid/squid.conf: No such file or directory real 0m1.176s user 0m0.780s sys 0m0.400s That run mostly 30 times faster. In the above example, I already skipped most of the directories in the list, removing lines that end with / but not all directories in my list match on that condition. So the fast solution emit errors and end with status 123. I know I could hide error messages and status error but that start to be ugly. sed -e'/.\/$/d' -e 's|^.|/&|g' ${ALLFILES} | xargs md5sum > ${ALLFILES}.md5 2>/dev/null || test $? -eq 123 Would it not be great to support an option in {md5,sha*} to ignore directory error? I may even be able to produce a patch if there is a real interest. Gilles From unknown Wed Jun 25 02:10:38 2025 X-Loop: help-debbugs@gnu.org Subject: bug#10355: Add an option to {md5,sha*} to ignore directories Resent-From: Erik Auerswald Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Fri, 23 Dec 2011 15:06:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 10355 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: Gilles Espinasse Cc: 10355@debbugs.gnu.org Received: via spool by 10355-submit@debbugs.gnu.org id=B10355.132465270411855 (code B ref 10355); Fri, 23 Dec 2011 15:06:01 +0000 Received: (at 10355) by debbugs.gnu.org; 23 Dec 2011 15:05:04 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Re6gB-000359-V8 for submit@debbugs.gnu.org; Fri, 23 Dec 2011 10:05:04 -0500 Received: from mailgw1.uni-kl.de ([131.246.120.220]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Re6g9-00034k-KI for 10355@debbugs.gnu.org; Fri, 23 Dec 2011 10:05:02 -0500 Received: from sushi.unix-ag.uni-kl.de (sushi.unix-ag.uni-kl.de [IPv6:2001:638:208:ef34:0:ff:fe00:65]) by mailgw1.uni-kl.de (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id pBNF2ivr004428 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Fri, 23 Dec 2011 16:02:44 +0100 Received: from [127.0.0.1] (localhost [127.0.0.1]) by sushi.unix-ag.uni-kl.de (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id pBNF2hfx031349; Fri, 23 Dec 2011 16:02:43 +0100 Message-ID: <4EF49813.6030900@unix-ag.uni-kl.de> Date: Fri, 23 Dec 2011 16:02:43 +0100 From: Erik Auerswald User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.24) Gecko/20111114 Icedove/3.1.16 MIME-Version: 1.0 References: <082801ccc179$17d154a0$f9b5a8c0@pii350> In-Reply-To: <082801ccc179$17d154a0$f9b5a8c0@pii350> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -6.6 (------) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.6 (------) Hi Gilles, On 12/23/2011 02:45 PM, Gilles Espinasse wrote: > I was using a way to check md5sum on a lot of file using > for myfile in `cat ${ALLFILES}`; do if [ -f /${myfile} ]; then md5sum > /$myfile>> $ALLFILES}.md5; fi; done > > But this is slow, comparing with xargs md5sum way. > time (for myfile in `cat ${ALLFILES}`; do if [ -f /${myfile} ]; then md5sum > /$myfile>> ${ALLFILES}.md5; fi; done) > > real 0m26.907s > user 0m40.019s > sys 0m10.253s > > This is faster using xargs md5sum. > time (sed -e '/.\/$/d' -e 's|^.|/&|g' ${ALLFILES} | xargs md5sum >> ${ALLFILES}.md5) > md5sum: /etc/ipsec.d/cacerts: Is a directory > md5sum: /etc/ipsec.d/certs: Is a directory > md5sum: /etc/ipsec.d/crls: Is a directory > md5sum: /etc/ppp/chap-secrets: No such file or directory > md5sum: /etc/ppp/pap-secrets: No such file or directory > md5sum: /etc/squid/squid.conf: No such file or directory > > real 0m1.176s > user 0m0.780s > sys 0m0.400s > > That run mostly 30 times faster. > In the above example, I already skipped most of the directories in the list, > removing lines that end with / but not all directories in my list match on > that condition. How do you create the list of files to check? You could use "find $DIR -type f" to list regular files only. Erik From unknown Wed Jun 25 02:10:38 2025 X-Loop: help-debbugs@gnu.org Subject: bug#10355: Add an option to {md5,sha*} to ignore directories Resent-From: Bob Proulx Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Fri, 23 Dec 2011 17:20:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 10355 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: To: Gilles Espinasse , 10355@debbugs.gnu.org Received: via spool by 10355-submit@debbugs.gnu.org id=B10355.132466078623509 (code B ref 10355); Fri, 23 Dec 2011 17:20:02 +0000 Received: (at 10355) by debbugs.gnu.org; 23 Dec 2011 17:19:46 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Re8mX-000675-11 for submit@debbugs.gnu.org; Fri, 23 Dec 2011 12:19:45 -0500 Received: from joseki.proulx.com ([216.17.153.58]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Re8mT-00066s-RJ; Fri, 23 Dec 2011 12:19:42 -0500 Received: from hysteria.proulx.com (hysteria.proulx.com [192.168.230.119]) by joseki.proulx.com (Postfix) with ESMTP id 55564211D2; Fri, 23 Dec 2011 10:17:23 -0700 (MST) Received: by hysteria.proulx.com (Postfix, from userid 1000) id 049D72DCD7; Fri, 23 Dec 2011 10:17:22 -0700 (MST) Date: Fri, 23 Dec 2011 10:17:22 -0700 From: Bob Proulx Message-ID: <20111223171722.GA3518@hysteria.proulx.com> References: <082801ccc179$17d154a0$f9b5a8c0@pii350> <4EF49813.6030900@unix-ag.uni-kl.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4EF49813.6030900@unix-ag.uni-kl.de> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Score: -2.5 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.5 (--) severity 10355 wishlist tags 10355 + notabug wontfix moreinfo thanks Erik Auerswald wrote: > Gilles Espinasse wrote: > >I was using a way to check md5sum on a lot of file using > > for myfile in `cat ${ALLFILES}`; do if [ -f /${myfile} ]; then md5sum > >/$myfile>> $ALLFILES}.md5; fi; done >... > You could use "find $DIR -type f" to list regular files only. Yes. Exactly. The capability you ask for is already present. Please try this: find . -type f -exec md5sum {} + Replace '.' above with a directory if you wish it to find files in a different directory. Bob From unknown Wed Jun 25 02:10:38 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.427 (Entity 5.427) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: "Gilles Espinasse" Subject: bug#10355: closed (Re: bug#10355: Add an option to {md5,sha*} to ignore directories) Message-ID: References: <4EF4BED5.7030607@draigBrady.com> <082801ccc179$17d154a0$f9b5a8c0@pii350> X-Gnu-PR-Message: they-closed 10355 X-Gnu-PR-Package: coreutils X-Gnu-PR-Keywords: notabug moreinfo wontfix Reply-To: 10355@debbugs.gnu.org Date: Fri, 23 Dec 2011 17:51:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1324662662-29127-1" This is a multi-part message in MIME format... ------------=_1324662662-29127-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #10355: Add an option to {md5,sha*} to ignore directories which was filed against the coreutils package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 10355@debbugs.gnu.org. --=20 10355: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D10355 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1324662662-29127-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 10355-done) by debbugs.gnu.org; 23 Dec 2011 17:50:29 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Re9GH-0007Z4-0m for submit@debbugs.gnu.org; Fri, 23 Dec 2011 12:50:29 -0500 Received: from mail1.vodafone.ie ([213.233.128.43]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Re9GE-0007Yw-AY for 10355-done@debbugs.gnu.org; Fri, 23 Dec 2011 12:50:27 -0500 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ApMBAJW99E5tTiPl/2dsb2JhbAAMMQapX4VPAQEBBDIBRhALDQEKCRYPCQMCAQIBRQYNAQcBARe/JYN9hGSDLgSacYxF Received: from unknown (HELO [192.168.1.79]) ([109.78.35.229]) by mail1.vodafone.ie with ESMTP; 23 Dec 2011 17:48:08 +0000 Message-ID: <4EF4BED5.7030607@draigBrady.com> Date: Fri, 23 Dec 2011 17:48:05 +0000 From: =?ISO-8859-1?Q?P=E1draig_Brady?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:6.0) Gecko/20110816 Thunderbird/6.0 MIME-Version: 1.0 To: Gilles Espinasse Subject: Re: bug#10355: Add an option to {md5,sha*} to ignore directories References: <082801ccc179$17d154a0$f9b5a8c0@pii350> In-Reply-To: <082801ccc179$17d154a0$f9b5a8c0@pii350> X-Enigmail-Version: 1.3.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Spam-Score: -2.5 (--) X-Debbugs-Envelope-To: 10355-done Cc: 10355-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.5 (--) On 12/23/2011 01:45 PM, Gilles Espinasse wrote: > I was using a way to check md5sum on a lot of file using > for myfile in `cat ${ALLFILES}`; do if [ -f /${myfile} ]; then md5sum > /$myfile >> $ALLFILES}.md5; fi; done > > But this is slow, comparing with xargs md5sum way. > time (for myfile in `cat ${ALLFILES}`; do if [ -f /${myfile} ]; then md5sum > /$myfile >> ${ALLFILES}.md5; fi; done) > > real 0m26.907s > user 0m40.019s > sys 0m10.253s > > This is faster using xargs md5sum. > time (sed -e '/.\/$/d' -e 's|^.|/&|g' ${ALLFILES} | xargs md5sum >> ${ALLFILES}.md5) > md5sum: /etc/ipsec.d/cacerts: Is a directory > md5sum: /etc/ipsec.d/certs: Is a directory > md5sum: /etc/ipsec.d/crls: Is a directory > md5sum: /etc/ppp/chap-secrets: No such file or directory > md5sum: /etc/ppp/pap-secrets: No such file or directory > md5sum: /etc/squid/squid.conf: No such file or directory > > real 0m1.176s > user 0m0.780s > sys 0m0.400s > > That run mostly 30 times faster. > In the above example, I already skipped most of the directories in the list, > removing lines that end with / but not all directories in my list match on > that condition. > > So the fast solution emit errors and end with status 123. > I know I could hide error messages and status error but that start to be > ugly. > sed -e'/.\/$/d' -e 's|^.|/&|g' ${ALLFILES} | xargs md5sum > ${ALLFILES}.md5 > 2>/dev/null || test $? -eq 123 > > Would it not be great to support an option in {md5,sha*} to ignore directory > error? > I may even be able to produce a patch if there is a real interest. > > Gilles I don't think this is worthwhile TBH, as it is too unusual. One can easily exclude dirs from the source. Either trivially with find, or filtering like: LANG=C xargs -d'\n' -r stat -L -c "%F:%n" < ${ALLFILES} | # decorate sed '/^directory:/d; s/^[^:]*://' | # filter and undecorate xargs -d'\n' md5sum # process cheers, Pádraig. ------------=_1324662662-29127-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 23 Dec 2011 13:46:09 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Re5Ro-0001FK-T7 for submit@debbugs.gnu.org; Fri, 23 Dec 2011 08:46:09 -0500 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Re5Rm-0001FD-G7 for submit@debbugs.gnu.org; Fri, 23 Dec 2011 08:46:07 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Re5PZ-0001Om-Oy for submit@debbugs.gnu.org; Fri, 23 Dec 2011 08:43:50 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.2 Received: from lists.gnu.org ([140.186.70.17]:48298) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Re5PZ-0001Oi-NV for submit@debbugs.gnu.org; Fri, 23 Dec 2011 08:43:49 -0500 Received: from eggs.gnu.org ([140.186.70.92]:53081) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Re5PY-0004gk-T3 for bug-coreutils@gnu.org; Fri, 23 Dec 2011 08:43:49 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Re5PS-0001No-Q6 for bug-coreutils@gnu.org; Fri, 23 Dec 2011 08:43:48 -0500 Received: from smtp3-g21.free.fr ([212.27.42.3]:56350) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Re5PS-0001Mx-3u for bug-coreutils@gnu.org; Fri, 23 Dec 2011 08:43:42 -0500 Received: from pii350 (unknown [82.236.101.3]) by smtp3-g21.free.fr (Postfix) with SMTP id 71E5AA6294 for ; Fri, 23 Dec 2011 14:43:33 +0100 (CET) Message-ID: <082801ccc179$17d154a0$f9b5a8c0@pii350> From: "Gilles Espinasse" To: Subject: Add an option to {md5,sha*} to ignore directories Date: Fri, 23 Dec 2011 14:45:10 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.2001 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.2001 X-Antivirus: avast! (VPS 111223-0, 23/12/2011), Outbound message X-Antivirus-Status: Clean X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 140.186.70.17 X-Spam-Score: -4.7 (----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.7 (----) I was using a way to check md5sum on a lot of file using for myfile in `cat ${ALLFILES}`; do if [ -f /${myfile} ]; then md5sum /$myfile >> $ALLFILES}.md5; fi; done But this is slow, comparing with xargs md5sum way. time (for myfile in `cat ${ALLFILES}`; do if [ -f /${myfile} ]; then md5sum /$myfile >> ${ALLFILES}.md5; fi; done) real 0m26.907s user 0m40.019s sys 0m10.253s This is faster using xargs md5sum. time (sed -e '/.\/$/d' -e 's|^.|/&|g' ${ALLFILES} | xargs md5sum >${ALLFILES}.md5) md5sum: /etc/ipsec.d/cacerts: Is a directory md5sum: /etc/ipsec.d/certs: Is a directory md5sum: /etc/ipsec.d/crls: Is a directory md5sum: /etc/ppp/chap-secrets: No such file or directory md5sum: /etc/ppp/pap-secrets: No such file or directory md5sum: /etc/squid/squid.conf: No such file or directory real 0m1.176s user 0m0.780s sys 0m0.400s That run mostly 30 times faster. In the above example, I already skipped most of the directories in the list, removing lines that end with / but not all directories in my list match on that condition. So the fast solution emit errors and end with status 123. I know I could hide error messages and status error but that start to be ugly. sed -e'/.\/$/d' -e 's|^.|/&|g' ${ALLFILES} | xargs md5sum > ${ALLFILES}.md5 2>/dev/null || test $? -eq 123 Would it not be great to support an option in {md5,sha*} to ignore directory error? I may even be able to produce a patch if there is a real interest. Gilles ------------=_1324662662-29127-1-- From unknown Wed Jun 25 02:10:38 2025 X-Loop: help-debbugs@gnu.org Subject: bug#10355: Add an option to {md5,sha*} to ignore directories In-Reply-To: <082801ccc179$17d154a0$f9b5a8c0@pii350> Resent-From: "Gilles Espinasse" Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Fri, 23 Dec 2011 17:55:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 10355 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: notabug moreinfo wontfix To: "Bob Proulx" , <10355@debbugs.gnu.org> Received: via spool by 10355-submit@debbugs.gnu.org id=B10355.132466285729462 (code B ref 10355); Fri, 23 Dec 2011 17:55:02 +0000 Received: (at 10355) by debbugs.gnu.org; 23 Dec 2011 17:54:17 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Re9Jx-0007f9-1i for submit@debbugs.gnu.org; Fri, 23 Dec 2011 12:54:17 -0500 Received: from smtp3-g21.free.fr ([212.27.42.3]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Re9Ju-0007f0-Lr for 10355@debbugs.gnu.org; Fri, 23 Dec 2011 12:54:16 -0500 Received: from pii350 (unknown [82.236.101.3]) by smtp3-g21.free.fr (Postfix) with SMTP id C2B34A62B5; Fri, 23 Dec 2011 18:51:51 +0100 (CET) Message-ID: <089a01ccc19b$c771aff0$f9b5a8c0@pii350> From: "Gilles Espinasse" References: <082801ccc179$17d154a0$f9b5a8c0@pii350> <4EF49813.6030900@unix-ag.uni-kl.de> <20111223171722.GA3518@hysteria.proulx.com> Date: Fri, 23 Dec 2011 18:53:27 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.2001 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.2001 X-Antivirus: avast! (VPS 111223-0, 23/12/2011), Outbound message X-Antivirus-Status: Clean X-Spam-Score: -2.8 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.8 (--) ----- Original Message ----- From: "Bob Proulx" To: "Gilles Espinasse" ; <10355@debbugs.gnu.org> Sent: Friday, December 23, 2011 6:17 PM Subject: Re: bug#10355: Add an option to {md5,sha*} to ignore directories > severity 10355 wishlist > tags 10355 + notabug wontfix moreinfo > thanks > > Erik Auerswald wrote: > > Gilles Espinasse wrote: > > >I was using a way to check md5sum on a lot of file using > > > for myfile in `cat ${ALLFILES}`; do if [ -f /${myfile} ]; then md5sum > > >/$myfile>> $ALLFILES}.md5; fi; done > >... > > You could use "find $DIR -type f" to list regular files only. > Thank for the suggestion. ALLFILES is indirectly made using find $DIR, but I can't use -type f during that find. The primary usage of that list is to create a tar using --files-from and when a directory is empty, you need to include the directory name directly. > Yes. Exactly. The capability you ask for is already present. > > Please try this: > > find . -type f -exec md5sum {} + > > Replace '.' above with a directory if you wish it to find files in a > different directory. > > Bob This does not work too in my case. I didn't want to calculate md5 for each file found in my chroot. I care only for a shorter list of files to be include in a tar. The only change I find is derived from the slow version, but instead of running md5sum each time, adding true file names to a list that md5sum will only use at the end: rm /tmp/ALLFILES* time (for myfile in ${ALLFILES} | sed -e 's/^dev.*//' -e 's/^sys.*//'); do if [ -f /${myfile} ]; then echo /$myfile >>/tmp/ALLFILES; fi; done; xargs md5sum < /tmp/ALLFILES >/tmp/ALLFILES.md5) real 0m1.967s user 0m1.368s sys 0m0.604s This is approximatly 100% slower than the fast version but does not need hiding errors (from directory message and program status). That's fast enought for my need and divide by 5 the time from the first slow version. Gilles From unknown Wed Jun 25 02:10:38 2025 X-Loop: help-debbugs@gnu.org Subject: bug#10355: Add an option to {md5,sha*} to ignore directories References: <082801ccc179$17d154a0$f9b5a8c0@pii350> Resent-From: "Alan Curry" Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Fri, 23 Dec 2011 23:02:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 10355 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: notabug moreinfo wontfix To: bob@proulx.com (Bob Proulx) Cc: 10355@debbugs.gnu.org Received: via spool by 10355-submit@debbugs.gnu.org id=B10355.132468127223619 (code B ref 10355); Fri, 23 Dec 2011 23:02:02 +0000 Received: (at 10355) by debbugs.gnu.org; 23 Dec 2011 23:01:12 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ReE6x-00068u-VT for submit@debbugs.gnu.org; Fri, 23 Dec 2011 18:01:12 -0500 Received: from c-68-60-252-82.hsd1.in.comcast.net ([68.60.252.82] helo=kosh.dhis.org) by debbugs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1ReE6w-00068n-48 for 10355@debbugs.gnu.org; Fri, 23 Dec 2011 18:01:10 -0500 Received: (qmail 28564 invoked by uid 1000); 23 Dec 2011 22:58:51 -0000 Message-ID: <20111223225851.28563.qmail@kosh.dhis.org> From: "Alan Curry" Date: Fri, 23 Dec 2011 17:58:51 -0500 (GMT+5) In-Reply-To: <20111223171722.GA3518@hysteria.proulx.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Spam-Score: -0.7 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -0.7 (/) Bob Proulx writes: > > severity 10355 wishlist > tags 10355 + notabug wontfix moreinfo > thanks > > Erik Auerswald wrote: > > Gilles Espinasse wrote: > > >I was using a way to check md5sum on a lot of file using > > > for myfile in `cat ${ALLFILES}`; do if [ -f /${myfile} ]; then md5sum > > >/$myfile>> $ALLFILES}.md5; fi; done > >... > > You could use "find $DIR -type f" to list regular files only. > > Yes. Exactly. The capability you ask for is already present. Do you suppose we can convince GNU grep's maintainer to follow this philosphy? $ mkdir d $ touch d/foo $ grep foo * $ It opens and reads, gets EISDIR, and intentionally skips printing it. Grr. But wait, there's a -d option with 3 alternatives for what to do with directories! ...and none of choices is "just print the EISDIR so I'll know if I accidentally grepped a directory". -- Alan Curry From unknown Wed Jun 25 02:10:38 2025 X-Loop: help-debbugs@gnu.org Subject: bug#10355: Add an option to {md5,sha*} to ignore directories Resent-From: Bob Proulx Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Sat, 24 Dec 2011 01:00:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 10355 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: notabug moreinfo wontfix To: 10355@debbugs.gnu.org Received: via spool by 10355-submit@debbugs.gnu.org id=B10355.13246883851227 (code B ref 10355); Sat, 24 Dec 2011 01:00:02 +0000 Received: (at 10355) by debbugs.gnu.org; 24 Dec 2011 00:59:45 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ReFxf-0000Jj-Ot for submit@debbugs.gnu.org; Fri, 23 Dec 2011 19:59:45 -0500 Received: from joseki.proulx.com ([216.17.153.58]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ReFxa-0000JX-7R for 10355@debbugs.gnu.org; Fri, 23 Dec 2011 19:59:42 -0500 Received: from hysteria.proulx.com (hysteria.proulx.com [192.168.230.119]) by joseki.proulx.com (Postfix) with ESMTP id D9FB5211D2 for <10355@debbugs.gnu.org>; Fri, 23 Dec 2011 17:57:18 -0700 (MST) Received: by hysteria.proulx.com (Postfix, from userid 1000) id 7EC892DCDB; Fri, 23 Dec 2011 17:57:18 -0700 (MST) Date: Fri, 23 Dec 2011 17:57:18 -0700 From: Bob Proulx Message-ID: <20111224005718.GA13924@hysteria.proulx.com> References: <082801ccc179$17d154a0$f9b5a8c0@pii350> <20111223225851.28563.qmail@kosh.dhis.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111223225851.28563.qmail@kosh.dhis.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Score: -2.5 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.5 (--) Alan Curry wrote: > Do you suppose we can convince GNU grep's maintainer to follow this > philosphy? Too late. GNU grep already has --recursive. I think adding --recursive to grep was a mistake. It then requires most of 'find' to be added to it too. (--include*, --exclude*) > $ mkdir d > $ touch d/foo > $ grep foo * > $ > > It opens and reads, gets EISDIR, and intentionally skips printing it. Grr. All silently. For most cases I think your example would have been a a case of programming error. It would be better to make those cases noisy. The above seems to be a bug since it violates the documented action of 'read' for directories. It appears to be skipping by default. Even when --directories=read is specified. > But wait, there's a -d option with 3 alternatives for what to do with > directories! ...and none of choices is "just print the EISDIR so I'll know > if I accidentally grepped a directory". And the problems just go on and on. Bob From unknown Wed Jun 25 02:10:38 2025 X-Loop: help-debbugs@gnu.org Subject: bug#10355: Add an option to {md5,sha*} to ignore directories Resent-From: Paul Eggert Original-Sender: debbugs-submit-bounces@debbugs.gnu.org Resent-CC: bug-coreutils@gnu.org Resent-Date: Sat, 24 Dec 2011 10:45:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 10355 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: notabug moreinfo wontfix To: Alan Curry Cc: 10355@debbugs.gnu.org, Bob Proulx Received: via spool by 10355-submit@debbugs.gnu.org id=B10355.132472349019880 (code B ref 10355); Sat, 24 Dec 2011 10:45:02 +0000 Received: (at 10355) by debbugs.gnu.org; 24 Dec 2011 10:44:50 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ReP5t-0005Aa-F1 for submit@debbugs.gnu.org; Sat, 24 Dec 2011 05:44:50 -0500 Received: from smtp.cs.ucla.edu ([131.179.128.62]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1ReP5d-0005AF-Ms for 10355@debbugs.gnu.org; Sat, 24 Dec 2011 05:44:48 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp.cs.ucla.edu (Postfix) with ESMTP id 984CFA60004; Sat, 24 Dec 2011 02:42:12 -0800 (PST) X-Virus-Scanned: amavisd-new at smtp.cs.ucla.edu Received: from smtp.cs.ucla.edu ([127.0.0.1]) by localhost (smtp.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZE4LnFmXBunU; Sat, 24 Dec 2011 02:42:12 -0800 (PST) Received: from [192.168.1.10] (pool-71-189-109-235.lsanca.fios.verizon.net [71.189.109.235]) by smtp.cs.ucla.edu (Postfix) with ESMTPSA id 2C4D339E8006; Sat, 24 Dec 2011 02:42:12 -0800 (PST) Message-ID: <4EF5AC85.4020409@cs.ucla.edu> Date: Sat, 24 Dec 2011 02:42:13 -0800 From: Paul Eggert Organization: UCLA Computer Science Department User-Agent: Mozilla/5.0 (X11; Linux i686; rv:8.0) Gecko/20111124 Thunderbird/8.0 MIME-Version: 1.0 References: <082801ccc179$17d154a0$f9b5a8c0@pii350> <20111223225851.28563.qmail@kosh.dhis.org> In-Reply-To: <20111223225851.28563.qmail@kosh.dhis.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Spam-Score: -2.9 (--) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.9 (--) On 12/23/11 14:58, Alan Curry wrote: > Do you suppose we can convince GNU grep's maintainer to follow this > philosphy? We definitely should. I have filed a bug report (with patch) at .