GNU bug report logs - #4982
rgrep can't deal with .gz files

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 4982 in the body.
You can then email your comments to 4982 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: jidanni <at> jidanni.org
To: bug-gnu-emacs <at> gnu.org
Subject: rgrep can't deal with .gz files
Date: Fri, 20 Nov 2009 09:42:45 +0800

rgrep should be enhanced so if it encounters .gz files it knows what to do.

Message #10 received at 4982 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Juri Linkov <juri <at> jurta.org>
To: jidanni <at> jidanni.org
Cc: 4982 <at> debbugs.gnu.org
Subject: Re: bug#4982: rgrep can't deal with .gz files
Date: Fri, 20 Nov 2009 11:31:39 +0200

> rgrep should be enhanced so if it encounters .gz files
> it knows what to do.

This would be tricky to implement.  When searching in .gz files
we have to rely on zgrep (that has its own problems - it doesn't
highlight grep matches).

I suggest creating a new command (say, `zrgrep') as a wrapper for `zgrep'
and with UI like `rgrep'.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Message #15 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Thierry Volpiatto <thierry.volpiatto <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#4982: rgrep can't deal with .gz files
Date: Fri, 20 Nov 2009 11:07:08 +0100

Juri Linkov <juri <at> jurta.org> writes:

>> rgrep should be enhanced so if it encounters .gz files
>> it knows what to do.
>
> This would be tricky to implement.  When searching in .gz files
> we have to rely on zgrep (that has its own problems - it doesn't
> highlight grep matches).

traverselisp.el handle compressed archives using avfs.
http://www.emacswiki.org/cgi-bin/emacs/traverselisp.el
You can launch traverse-deep-rfind (similar to rgrep) on any archive.(tar)gz.

You could use also avfs to handle .gz files with rgrep.
http://sourceforge.net/projects/avf/
 
> I suggest creating a new command (say, `zrgrep') as a wrapper for `zgrep'
> and with UI like `rgrep'.

-- 
A + Thierry Volpiatto
Location: Saint-Cyr-Sur-Mer - France

Message #20 received at submit <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Kevin Rodgers <kevin.d.rodgers <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#4982: rgrep can't deal with .gz files
Date: Fri, 20 Nov 2009 03:43:40 -0700

Juri Linkov wrote:
> I suggest creating a new command (say, `zrgrep') as a wrapper for `zgrep'
> and with UI like `rgrep'.

Clearly, it should be called `rzgrep' :-)

-- 
Kevin Rodgers
Denver, Colorado, USA

Message #25 received at 4982 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: jidanni <at> jidanni.org
To: juri <at> jurta.org
Cc: 4982 <at> debbugs.gnu.org
Subject: Re: bug#4982: rgrep can't deal with .gz files
Date: Sat, 21 Nov 2009 05:17:02 +0800

>>>>> "JL" == Juri Linkov <juri <at> jurta.org> writes:
>> rgrep should be enhanced so if it encounters .gz files
>> it knows what to do.

JL> This would be tricky to implement.  When searching in .gz files
JL> we have to rely on zgrep

Ah, of course... OK, now in .emacs I'll just use
(setq grep-program "zgrep")
as a workaround, as zgrep also works fine on non compressed files.

JL>(that has its own problems - it doesn't highlight grep matches).

Seems to work for me.

JL> I suggest creating a new command (say, `zrgrep') as a wrapper for `zgrep'
JL> and with UI like `rgrep'.

Message #30 received at 4982 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Juri Linkov <juri <at> jurta.org>
To: jidanni <at> jidanni.org
Cc: 4982 <at> debbugs.gnu.org
Subject: Re: bug#4982: rgrep can't deal with .gz files
Date: Sun, 22 Nov 2009 12:28:31 +0200

> JL>(that has its own problems - it doesn't highlight grep matches).
>
> Seems to work for me.

Hmm, strange.  In http://thread.gmane.org/gmane.emacs.bugs/16956
you requested us to remove highlighting when there are additional
filters on the grep output.  So we changed `--color=always' into
`--color=auto' that broke the highlighting in zgrep output because
zgrep uses additional filters.

But now you say that highlighting in zgrep output works for you.
I wonder how this is possible?

-- 
Juri Linkov
http://www.jurta.org/emacs/

Message #35 received at 4982 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Juri Linkov <juri <at> jurta.org>
To: Kevin Rodgers <kevin.d.rodgers <at> gmail.com>
Cc: 4982 <at> debbugs.gnu.org
Subject: Re: bug#4982: rgrep can't deal with .gz files
Date: Sun, 22 Nov 2009 12:33:42 +0200

> > I suggest creating a new command (say, `zrgrep') as a wrapper for `zgrep'
> > and with UI like `rgrep'.
>
> Clearly, it should be called `rzgrep' :-)

See in the zgrep script:

  grep='${GREP-grep}'
  version='z$grep (gzip) 1.3.12'

So according to the naming convention of zgrep, it prepends the "z" prefix
to any grep command name :-)

-- 
Juri Linkov
http://www.jurta.org/emacs/

Message #40 received at 4982 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: jidanni <at> jidanni.org
To: juri <at> jurta.org
Cc: 4982 <at> debbugs.gnu.org
Subject: Re: bug#4982: rgrep can't deal with .gz files
Date: Mon, 23 Nov 2009 09:53:33 +0800

>>>>> "JL" == Juri Linkov <juri <at> jurta.org> writes:
JL> But now you say that highlighting in zgrep output works for you.
JL> I wonder how this is possible?
All I know is after I changed grep-command to zgrep, when I run
(rgrep "JL" "* .*" "~" nil)
it uses
find . \( -path \*/SCCS ... xargs -0 -e zgrep  -nH -e JL
and the front of this line is blue:
./KIDPIX/n_PICT4993.JPG:Binary file (standard input) matches
and this is green:
./.ssh/known_hosts:14:|1|HHSxNOckqq
Note for me
$ env|grep -ci grep
0
and I use the latest debian emacs-snapshot.

Message #45 received at 4982 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Stefan Monnier <monnier <at> iro.umontreal.ca>
To: Juri Linkov <juri <at> jurta.org>
Cc: 4982 <at> debbugs.gnu.org, jidanni <at> jidanni.org
Subject: Re: bug#4982: rgrep can't deal with .gz files
Date: Sun, 22 Nov 2009 23:26:49 -0500

> Hmm, strange.  In http://thread.gmane.org/gmane.emacs.bugs/16956
> you requested us to remove highlighting when there are additional
> filters on the grep output.  So we changed `--color=always' into
> `--color=auto' that broke the highlighting in zgrep output because
> zgrep uses additional filters.

Maybe zgrep is careful to distinguish filters on the input from filters
on the output.  AFAICT zgrep only uses a filter on the input, so it
should work.


        Stefan

Message #50 received at 4982 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Juri Linkov <juri <at> jurta.org>
To: jidanni <at> jidanni.org
Cc: 4982 <at> debbugs.gnu.org
Subject: Re: bug#4982: rgrep can't deal with .gz files
Date: Mon, 23 Nov 2009 11:58:32 +0200

> JL> But now you say that highlighting in zgrep output works for you.
> JL> I wonder how this is possible?
>
> All I know is after I changed grep-command to zgrep, when I run
> (rgrep "JL" "* .*" "~" nil)
> it uses
> find . \( -path \*/SCCS ... xargs -0 -e zgrep  -nH -e JL
> and the front of this line is blue:
> ./KIDPIX/n_PICT4993.JPG:Binary file (standard input) matches
> and this is green:
> ./.ssh/known_hosts:14:|1|HHSxNOckqq

So you don't see yellow matches?

-- 
Juri Linkov
http://www.jurta.org/emacs/

Message #55 received at 4982 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Juri Linkov <juri <at> jurta.org>
To: Stefan Monnier <monnier <at> iro.umontreal.ca>
Cc: 4982 <at> debbugs.gnu.org, jidanni <at> jidanni.org
Subject: Re: bug#4982: rgrep can't deal with .gz files
Date: Mon, 23 Nov 2009 11:57:43 +0200

> > Hmm, strange.  In http://thread.gmane.org/gmane.emacs.bugs/16956
> > you requested us to remove highlighting when there are additional
> > filters on the grep output.  So we changed `--color=always' into
> > `--color=auto' that broke the highlighting in zgrep output because
> > zgrep uses additional filters.

> Maybe zgrep is careful to distinguish filters on the input from
> filters on the output.  AFAICT zgrep only uses a filter on the input,
> so it should work.

IIUC, zgrep puts a sed filter on the grep output to replace
"(standard input)" strings with real file names.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Message #60 received at 4982 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: jidanni <at> jidanni.org
To: juri <at> jurta.org
Cc: 4982 <at> debbugs.gnu.org
Subject: Re: bug#4982: rgrep can't deal with .gz files
Date: Tue, 24 Nov 2009 04:18:21 +0800

>>>>> "JL" == Juri Linkov <juri <at> jurta.org> writes:
JL> So you don't see yellow matches?
OK, I do if I mouseover them. By the way, some ugliness now that I set
grep-command to zgrep:

gzip: stdout: Broken pipe
./libchewing3-data_0.3.0-2_i386.deb:Binary file (standard input) matches

gzip: stdout: Broken pipe
./Nokia_3315_APAC_UG_En.pdf:Binary file (standard input) matches

Message #65 received at 4982 <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Juri Linkov <juri <at> jurta.org>
To: jidanni <at> jidanni.org
Cc: 4982 <at> debbugs.gnu.org
Subject: Re: bug#4982: rgrep can't deal with .gz files
Date: Tue, 24 Nov 2009 19:04:43 +0200

> JL> So you don't see yellow matches?
> OK, I do if I mouseover them. By the way, some ugliness now that I set
> grep-command to zgrep:
>
> gzip: stdout: Broken pipe
> ./libchewing3-data_0.3.0-2_i386.deb:Binary file (standard input) matches
>
> gzip: stdout: Broken pipe
> ./Nokia_3315_APAC_UG_En.pdf:Binary file (standard input) matches

Yes, zgrep incorrectly processes non-gzipped files.  To search
recursively in gzipped files, I created a new command `zrgrep'.
It works like `rgrep' but uses `zgrep' for `grep-program', sets the default
file name to `*.gz', and sets `grep-highlight-matches' to `always'."

Other changes in the patch below include:

1. grep-highlight-matches has two new options `always' and `auto'.
This allows `zrgrep' to override the default value with `always'.

2. Move Windows/DOS specific --colors settings handling from
`grep-process-setup' to `grep-compute-defaults'.

Index: lisp/progmodes/grep.el
===================================================================
RCS file: /sources/emacs/emacs/lisp/progmodes/grep.el,v
retrieving revision 1.111
diff -c -r1.111 grep.el
*** lisp/progmodes/grep.el	19 Nov 2009 17:37:25 -0000	1.111
--- lisp/progmodes/grep.el	24 Nov 2009 17:04:11 -0000
***************
*** 69,90 ****
    :group 'grep)
  
  (defcustom grep-highlight-matches 'auto-detect
!   "If t, use special markers to highlight grep matches.
  
  Some grep programs are able to surround matches with special
  markers in grep output.  Such markers can be used to highlight
  matches in grep mode.
  
! This option sets the environment variable GREP_COLOR to specify
  markers for highlighting and GREP_OPTIONS to add the --color
  option in front of any explicit grep options before starting
  the grep.
  
  In interactive usage, the actual value of this variable is set up
! by `grep-compute-defaults'; to change the default value, use
! Customize or call the function `grep-apply-setting'."
    :type '(choice (const :tag "Do not highlight matches with grep markers" nil)
  		 (const :tag "Highlight matches with grep markers" t)
  		 (other :tag "Not Set" auto-detect))
    :set 'grep-apply-setting
    :version "22.1"
--- 69,102 ----
    :group 'grep)
  
  (defcustom grep-highlight-matches 'auto-detect
!   "Use special markers to highlight grep matches.
  
  Some grep programs are able to surround matches with special
  markers in grep output.  Such markers can be used to highlight
  matches in grep mode.
  
! This option sets the environment variable GREP_COLORS to specify
  markers for highlighting and GREP_OPTIONS to add the --color
  option in front of any explicit grep options before starting
  the grep.
  
+ When this option is `auto', grep uses `--color=auto' to highlight
+ matches only when it outputs to a terminal (when `grep' is the last
+ command in the pipe), thus avoiding the use of any potentially-harmful
+ escape sequences when standard output goes to a file or pipe.
+ 
+ To make grep highlight matches even into a pipe, you need the option
+ `always' that forces grep to use `--color=always' to unconditionally
+ output escape sequences.
+ 
  In interactive usage, the actual value of this variable is set up
! by `grep-compute-defaults' when the default value is `auto-detect'.
! To change the default value, use Customize or call the function
! `grep-apply-setting'."
    :type '(choice (const :tag "Do not highlight matches with grep markers" nil)
  		 (const :tag "Highlight matches with grep markers" t)
+ 		 (const :tag "Use --color=always" always)
+ 		 (const :tag "Use --color=auto" auto)
  		 (other :tag "Not Set" auto-detect))
    :set 'grep-apply-setting
    :version "22.1"
***************
*** 442,460 ****
  (defun grep-process-setup ()
    "Setup compilation variables and buffer for `grep'.
  Set up `compilation-exit-message-function' and run `grep-setup-hook'."
!   (unless (or (not grep-highlight-matches) (eq grep-highlight-matches t))
      (grep-compute-defaults))
!   (when (eq grep-highlight-matches t)
      ;; `setenv' modifies `process-environment' let-bound in `compilation-start'
      ;; Any TERM except "dumb" allows GNU grep to use `--color=auto'
      (setenv "TERM" "emacs-grep")
-     ;; `--color=auto' emits escape sequences on a tty rather than on a pipe,
-     ;; thus allowing to use multiple grep filters on the command line
-     ;; and to output escape sequences only on the final grep output
      (setenv "GREP_OPTIONS"
  	    (concat (getenv "GREP_OPTIONS")
! 		    ;; Windows and DOS pipes fail `isatty' detection in Grep.
! 		    " --color=" (if (memq system-type '(windows-nt ms-dos))
  				    "always" "auto")))
      ;; GREP_COLOR is used in GNU grep 2.5.1, but deprecated in later versions
      (setenv "GREP_COLOR" "01;31")
--- 454,469 ----
  (defun grep-process-setup ()
    "Setup compilation variables and buffer for `grep'.
  Set up `compilation-exit-message-function' and run `grep-setup-hook'."
!   (when (eq grep-highlight-matches 'auto-detect)
      (grep-compute-defaults))
!   (unless (or (eq grep-highlight-matches 'auto-detect)
! 	      (null grep-highlight-matches))
      ;; `setenv' modifies `process-environment' let-bound in `compilation-start'
      ;; Any TERM except "dumb" allows GNU grep to use `--color=auto'
      (setenv "TERM" "emacs-grep")
      (setenv "GREP_OPTIONS"
  	    (concat (getenv "GREP_OPTIONS")
! 		    " --color=" (if (eq grep-highlight-matches 'always)
  				    "always" "auto")))
      ;; GREP_COLOR is used in GNU grep 2.5.1, but deprecated in later versions
      (setenv "GREP_COLOR" "01;31")
***************
*** 579,592 ****
  			(t
  			 (format "%s . <X> -type f <F> -print | %s %s"
  				 find-program xargs-program gcmd))))))))
!     (unless (or (not grep-highlight-matches) (eq grep-highlight-matches t))
        (setq grep-highlight-matches
  	    (with-temp-buffer
  	      (and (grep-probe grep-program '(nil t nil "--help"))
  		   (progn
  		     (goto-char (point-min))
  		     (search-forward "--color" nil t))
! 		   t))))
  
      ;; Save defaults for this host.
      (setq grep-host-defaults-alist
--- 588,603 ----
  			(t
  			 (format "%s . <X> -type f <F> -print | %s %s"
  				 find-program xargs-program gcmd))))))))
!     (when (eq grep-highlight-matches 'auto-detect)
        (setq grep-highlight-matches
  	    (with-temp-buffer
  	      (and (grep-probe grep-program '(nil t nil "--help"))
  		   (progn
  		     (goto-char (point-min))
  		     (search-forward "--color" nil t))
! 		   ;; Windows and DOS pipes fail `isatty' detection in Grep.
! 		   (if (memq system-type '(windows-nt ms-dos))
! 		       'always 'auto)))))
  
      ;; Save defaults for this host.
      (setq grep-host-defaults-alist
***************
*** 978,983 ****
--- 993,1030 ----
  	  (if (eq next-error-last-buffer (current-buffer))
  	      (setq default-directory dir)))))))
  
+ ;;;###autoload
+ (defun zrgrep (regexp &optional files dir confirm grep-find-template)
+   "Recursively grep for REGEXP in gzipped FILES in tree rooted at DIR.
+ Like `rgrep' but uses `zgrep' for `grep-program', sets the default
+ file name to `*.gz', and sets `grep-highlight-matches' to `always'."
+   (interactive
+    (let ((grep-program "zgrep")
+ 	 (grep-find-command nil)
+ 	 (grep-find-template nil)
+ 	 (grep-host-defaults-alist nil)
+ 	 (grep-files-aliases '(("*.gz" . "*.gz") ; for `grep-read-files'
+ 			       ("all" . "* .*"))))
+      ;; Recompute defaults using let-bound values above.
+      (grep-compute-defaults)
+      (cond
+       ((and grep-find-command (equal current-prefix-arg '(16)))
+        (list (read-from-minibuffer "Run: " grep-find-command
+ 				   nil nil 'grep-find-history)))
+       ((not grep-find-template)
+        (error "grep.el: No `grep-find-template' available"))
+       (t (let* ((regexp (grep-read-regexp))
+ 		(files (grep-read-files regexp))
+ 		(dir (read-directory-name "Base directory: "
+ 					  nil default-directory t))
+ 		(confirm (equal current-prefix-arg '(4))))
+ 	   (list regexp files dir confirm grep-find-template))))))
+   ;; Set `grep-highlight-matches' to `always'
+   ;; since `zgrep' puts filters in the grep output.
+   (let ((grep-highlight-matches 'always))
+     ;; `rgrep' uses the dynamically bound value `grep-find-template'
+     ;; from the argument `grep-find-template' whose value is computed
+     ;; in the `interactive' spec.
+     (rgrep regexp files dir confirm)))
  
;;;###autoload
(defalias 'rzgrep 'zrgrep)
  
  (provide 'grep)
  
-- 
Juri Linkov
http://www.jurta.org/emacs/

Message #70 received at 4982-done <at> emacsbugs.donarmstrong.com (full text, mbox):

From: Juri Linkov <juri <at> jurta.org>
To: 4982-done <at> debbugs.gnu.org
Cc: jidanni <at> jidanni.org
Subject: Re: bug#4982: rgrep can't deal with .gz files
Date: Wed, 25 Nov 2009 19:28:14 +0200

>> By the way, some ugliness now that I set grep-command to zgrep:
>>
>> gzip: stdout: Broken pipe
>> ./libchewing3-data_0.3.0-2_i386.deb:Binary file (standard input) matches
>>
>> gzip: stdout: Broken pipe
>> ./Nokia_3315_APAC_UG_En.pdf:Binary file (standard input) matches
>
> Yes, zgrep incorrectly processes non-gzipped files.  To search
> recursively in gzipped files, I created a new command `zrgrep'.
> It works like `rgrep' but uses `zgrep' for `grep-program', sets the default
> file name to `*.gz', and sets `grep-highlight-matches' to `always'."

I installed a new command `zrgrep' to search recursively in gzipped files.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.