Subject: 24.5; ansi-color-apply minor problem on 24.5 and broken on 25.

From: Thierry Volpiatto <[email-masked]>
Date: Sun, 30 Aug 2015 19:05:08 +0200

Hi,

I use ansi-color-apply since long time now to colorize output of diverse
helm grep variations and this on emacs-24.5.

Until now it worked fine with ack-grep and grep though I had to modify
(only for grep) GREP_COLORS env var to fix the same problem as
following.

But now I have implemented helm for AG, I have the same problem I had in
the past with grep, the end of the string is skipped by ansi:

;; Use original regexp.
(let (ansi-color-context)
  (ansi-color-apply "[1;32mhelm-external.el[0m[K:[1;33m62[0m[K:([30;43mdefun[0m[K helm-external-commands-list-1 (&optional sort)"))
#("helm-external.el:62:(defun" 0 16 (font-lock-face
[...]

As you can see the last part of the string is skipped.

;; Use modified regexp.
(let (ansi-color-context
      (ansi-color-regexp "\033\\[\\(K\\|[0-9;]*m\\)")) 
  (ansi-color-apply "[1;32mhelm-external.el[0m[K:[1;33m62[0m[K:([30;43mdefun[0m[K helm-external-commands-list-1 (&optional sort)"))
#("helm-external.el:62:(defun helm-external-commands-list-1 (&optional sort)" 0 16 (font-lock-face 
[...]

Here the string is completely displayed.
The result on helm is that ansi is working with all backend with this
modified regexp.

So this is on emacs-24.5.

Now if I switch to emacs-25 in either cases ansi is broken by this
commit:

--8<---------------cut here---------------start------------->8---
Author:     Daniel Colascione <[email-masked]>
AuthorDate: Mon Aug 3 20:06:04 2015 -0700
Commit:     Daniel Colascione <[email-masked]>
CommitDate: Mon Aug 3 20:06:04 2015 -0700

    Improve ansi-color filtering of unrecognized escape sequences
    
    * lisp/ansi-color.el (ansi-color-drop-regexp): Recognize mode-setting
    escape sequences.
    (ansi-color-filter-apply): Filter out unrecognized escape sequences
    (ansi-color-apply): Filter out unrecognized escape sequences
--8<---------------cut here---------------end--------------->8---

Here the backtrace:

--8<---------------cut here---------------start------------->8---
Debugger entered--Lisp error: (args-out-of-range #("[1;32m/home/thierry/labo/github/helm/helm-command.el[0m:[1;33m143[0m:    (beginning-of-[30;43mdefun[0m)" 7 53 (font-lock-face ((foreground-color . "green3") bold)) 65 68 (font-lock-face ((foreground-color . "yellow3") bold)) 99 104 (font-lock-face ((background-color . "yellow3") (foreground-color . "black")))) 114)
  string-match("" #("[1;32m/home/thierry/labo/github/helm/helm-command.el[0m:[1;33m143[0m:    (beginning-of-[30;43mdefun[0m)" 7 53 (font-lock-face ((foreground-color . "green3") bold)) 65 68 (font-lock-face ((foreground-color . "yellow3") bold)) 99 104 (font-lock-face ((background-color . "yellow3") (foreground-color . "black")))) 114)
  (if (string-match "" string start) (let ((pos (match-beginning 0))) (setq fragment (substring string pos)) (setq result (cons (substring string start pos) result))) (setq result (cons (substring string start) result)))
  (let (fragment) (if (string-match "" string start) (let ((pos (match-beginning 0))) (setq fragment (substring string pos)) (setq result (cons (substring string start pos) result))) (setq result (cons (substring string start) result))) (setq ansi-color-context (if (or codes fragment) (list codes fragment))))
  (let ((codes (car ansi-color-context)) (start 0) end escape-sequence result colorized-substring) (if (car (cdr ansi-color-context)) (setq string (concat (car (cdr ansi-color-context)) string) ansi-color-context nil)) (while (setq end (string-match ansi-color-regexp string start)) (setq escape-sequence (match-string 1 string)) (if codes (progn (put-text-property start end (quote font-lock-face) (ansi-color--find-face codes) string))) (setq colorized-substring (substring string start end) start (match-end 0)) (while (string-match ansi-color-drop-regexp colorized-substring) (setq colorized-substring (replace-match "" nil nil colorized-substring))) (setq result (cons colorized-substring result)) (setq codes (ansi-color-apply-sequence escape-sequence codes))) (if codes (progn (put-text-property start (length string) (quote font-lock-face) (ansi-color--find-face codes) string))) (while (string-match ansi-color-drop-regexp string) (setq string (replace-match "" nil nil string))) (let (fragment) (if (string-match "" string start) (let ((pos (match-beginning 0))) (setq fragment (substring string pos)) (setq result (cons (substring string start pos) result))) (setq result (cons (substring string start) result))) (setq ansi-color-context (if (or codes fragment) (list codes fragment)))) (apply (quote concat) (nreverse result)))
  ansi-color-apply(#("[1;32m/home/thierry/labo/github/helm/helm-command.el[0m[K:[1;33m143[0m[K:    (beginning-of-[30;43mdefun[0m[K)" 7 53 (font-lock-face ((foreground-color . "green3") bold)) 68 71 (font-lock-face ((foreground-color . "yellow3") bold)) 105 110 (font-lock-face ((background-color . "yellow3") (foreground-color . "black")))))
  (if ansi-p (ansi-color-apply candidate) candidate)
  (let* ((root (or dir (and helm-grep-default-directory-fn (funcall helm-grep-default-directory-fn)))) ansi-color-context (ansi-p (string-match-p ansi-color-regexp candidate)) (line (if ansi-p (ansi-color-apply candidate) candidate)) (split (helm-grep-split-line line)) (fname (if (and root split) (expand-file-name (car split) root) (car-safe split))) (lineno (nth 1 split)) (str (nth 2 split))) (if (and fname lineno str) (progn (cons (concat (propertize (file-name-nondirectory fname) (quote face) (quote helm-grep-file) (quote help-echo) fname) ":" (propertize lineno (quote face) (quote helm-grep-lineno)) ":" (if ansi-p str (helm-grep-highlight-match str))) line))))
  helm-grep--filter-candidate-1(#("[1;32m/home/thierry/labo/github/helm/helm-command.el[0m[K:[1;33m143[0m[K:    (beginning-of-[30;43mdefun[0m[K)" 7 53 (font-lock-face ((foreground-color . "green3") bold)) 68 71 (font-lock-face ((foreground-color . "yellow3") bold)) 105 110 (font-lock-face ((background-color . "yellow3") (foreground-color . "black")))))
  helm-grep-filter-one-by-one(#("[1;32m/home/thierry/labo/github/helm/helm-command.el[0m[K:[1;33m143[0m[K:    (beginning-of-[30;43mdefun[0m[K)" 7 53 (font-lock-face ((foreground-color . "green3") bold)) 68 71 (font-lock-face ((foreground-color . "yellow3") bold)) 105 110 (font-lock-face ((background-color . "yellow3") (foreground-color . "black")))))
  read-from-minibuffer("pattern: " nil (keymap (f13 lambda nil (interactive) (helm-select-nth-action 12)) (f12 lambda nil (interactive) (helm-select-nth-action 11)) (f11 lambda nil (interactive) (helm-select-nth-action 10)) (f10 lambda nil (interactive) (helm-select-nth-action 9)) (f9 lambda nil (interactive) (helm-select-nth-action 8)) (f8 lambda nil (interactive) (helm-select-nth-action 7)) (f7 lambda nil (interactive) (helm-select-nth-action 6)) (f6 lambda nil (interactive) (helm-select-nth-action 5)) (f5 lambda nil (interactive) (helm-select-nth-action 4)) (f4 lambda nil (interactive) (helm-select-nth-action 3)) (f3 lambda nil (interactive) (helm-select-nth-action 2)) (f2 lambda nil (interactive) (helm-select-nth-action 1)) (menu-bar keymap (help-menu keymap (describe keymap (describe-mode . helm-help)))) (help keymap (109 . helm-help)) (f1 lambda nil (interactive) (helm-select-nth-action 0)) (8 keymap (109 . helm-help) (104 . undefined) (8 . undefined) (4 . helm-enable-or-switch-to-debug)) (20 . helm-toggle-resplit-and-swap-windows) (C-tab . undefined) (67108897 . helm-toggle-suspend-update) (3 keymap (63 . helm-help) (62 . helm-toggle-truncate-line) (21 . helm-refresh) (6 . helm-follow-mode) (9 . helm-copy-to-buffer) (11 . helm-kill-selection-and-quit) (25 . helm-yank-selection) (4 . helm-delete-current-selection) (45 . helm-swap-windows)) (67108987 . helm-enlarge-window) (67108989 . helm-narrow-window) (19 . undefined) (18 . undefined) (23 . helm-yank-text-at-point) (24 keymap (2 . helm-resume-list-buffers-after-quit) (98 . helm-resume-previous-session-after-quit) (6 . helm-quit-and-find-file)) (11 . helm-delete-minibuffer-contents) (67108896 . helm-toggle-visible-mark) (0 . helm-toggle-visible-mark) (C-M-up . helm-scroll-other-window-down) (C-M-down . helm-scroll-other-window) (M-prior . helm-scroll-other-window-down) (M-next . helm-scroll-other-window) (12 . helm-recenter-top-bottom-other-window) (15 . helm-next-source) (10 . helm-execute-persistent-action) (26 . helm-execute-persistent-action) (9 . helm-select-action) (13 . helm-maybe-exit-minibuffer) (left . helm-previous-source) (right . helm-next-source) (7 . helm-keyboard-quit) (22 . helm-next-page) (27 keymap (110 . next-history-element) (112 . previous-history-element) (115 . undefined) (5 . helm-display-all-sources) (1 . helm-show-all-in-this-source-only) (117 . helm-unmark-all) (97 . helm-mark-all) (109 . helm-toggle-all-marks) (41 . helm-next-visible-mark) (40 . helm-prev-visible-mark) (91) (32 . helm-toggle-visible-mark) (33554454 . helm-scroll-other-window-down) (25 . helm-scroll-other-window-down) (22 . helm-scroll-other-window) (12 . helm-reposition-window-other-window) (62 . helm-end-of-buffer) (60 . helm-beginning-of-buffer) (118 . helm-previous-page)) (next . helm-next-page) (prior . helm-previous-page) (C-up . helm-follow-action-backward) (C-down . helm-follow-action-forward) (16 . helm-previous-line) ...) nil nil nil t)
  helm-read-pattern-maybe(nil nil nil nil nil nil nil)
  helm-grep-ag-1("/home/thierry/labo/github/helm/")
  helm-do-grep-ag()
  funcall-interactively(helm-do-grep-ag)
  call-interactively(helm-do-grep-ag nil nil)
  command-execute(helm-do-grep-ag)
--8<---------------cut here---------------end--------------->8---

In GNU Emacs 24.5.1 (x86_64-unknown-linux-gnu, X toolkit)
 of 2015-04-12 on dell-14z
Windowing system distributor `The X.Org Foundation', version 11.0.11501000
System Description:	Ubuntu 14.04.3 LTS

Configured using:
 `configure --with-x-toolkit=lucid --without-toolkit-scroll-bars
 --without-gconf --without-gsettings'

Important settings:
  value of $LANG: fr_FR.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
  diff-auto-refine-mode: t
  psession-mode: t
  golden-ratio-mode: t
  global-git-gutter-mode: t
  git-gutter-mode: t
  winner-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  auto-image-file-mode: t
  eldoc-in-minibuffer-mode: t
  show-paren-mode: t
  display-time-mode: t
  recentf-mode: t
  savehist-mode: t
  dired-async-mode: t
  minibuffer-depth-indicate-mode: t
  helm-descbinds-mode: t
  helm-push-mark-mode: t
  eldoc-mode: t
  helm-mode: t
  shell-dirtrack-mode: t
  helm-adaptive-mode: t
  async-bytecomp-package-mode: t
  tooltip-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Checking for load-path shadows...done
[mu4e] Started mu4e with 22100 messages in store
Mark set [2 times]
Auto-saving...done
Message modified; kill anyway? (y or n) y
(No changes need to be saved)
ad-handle-definition: `vc-revert' got redefined
Revert current hunk ? (y or n) y
Saving file /home/thierry/labo/github/helm/helm-grep.el...
Wrote /home/thierry/labo/github/helm/helm-grep.el

Load-path shadows:
/home/thierry/.emacs.d/elpa/emms-20150810.832/tq hides /usr/local/share/emacs/24.5/lisp/emacs-lisp/tq
~/elisp/auctex/lpath hides ~/elisp/emacs-wget/lpath
/home/thierry/.emacs.d/elpa/zenburn-theme-20150809.43/zenburn-theme hides ~/.emacs.d/themes/zenburn-theme

Features:
(diff-mode view helm-ls-git vc-git vc cl-indent shadow epa-mail
mule-util emacsbug helm-command markdown-mode cc-langs cc-mode cc-fonts
cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs
rst vc-hg checkdoc help-mode package-build lisp-mnt sh-script smie
executable vc-dispatcher vc-rcs jedi auto-complete popup jedi-core
python-environment epc ctable concurrent deferred naquadah-theme em-unix
em-script em-prompt em-ls em-hist em-pred em-glob em-dirs em-cmpl
em-basic em-banner em-alias align-let server psession golden-ratio
git-gutter cus-edit winner undo-tree diff slime-xref-browser
slime-banner slime-tramp slime-asdf slime-fancy slime-trace-dialog
slime-fontifying-fu slime-package-fu slime-references
slime-compiler-notes-tree slime-scratch slime-presentations bridge
slime-mdot-fu slime-enclosing-context slime-fuzzy slime-fancy-trace
slime-fancy-inspector slime-c-p-c slime-editing-commands slime-autodoc
slime-repl slime-parse slime etags arc-mode archive-mode image-file
xdvi-search preview-latex tex-site auto-loads pcomplete-extension
pcmpl-unix pcmpl-gnu em-term term disp-table ehelp python eldoc-eval
warnings whitespace paren time recentf tree-widget savehist dired-async
smtpmail-async mu4e-config org-mu4e helm-mu mu4e-contrib mu4e
mu4e-speedbar speedbar sb-image ezimage dframe mu4e-main mu4e-view
mu4e-headers mu4e-compose mu4e-draft mu4e-actions ido rfc2368 smtpmail
sendmail mu4e-mark mu4e-message html2text mu4e-proc mu4e-utils
mu4e-lists mu4e-vars hl-line mu4e-meta config-w3m w3m-search w3m
doc-view jka-compr image-mode timezone w3m-hist w3m-fb bookmark-w3m
w3m-ems w3m-ccl ccl w3m-favicon w3m-image w3m-proc w3m-util iterator
lacarte iedit-rect rect iedit iedit-lib smallurl mm-url gnus gnus-ems
nnheader wid-edit ledger-config ledger esh-var esh-io esh-cmd esh-opt
esh-ext esh-proc esh-groups eshell esh-module esh-mode esh-arg esh-util
tv-utils pcvs vc-cvs pcvs-parse pcvs-info pcvs-defs pcvs-util ewoc
mb-depth cl-info hyperspec esh-toggle flymake no-word htmlize cl
dired-extension emms-vlc-config emms-librefm-stream
emms-librefm-scrobbler emms-playlist-limit emms-volume
emms-volume-amixer emms-i18n emms-history emms-score emms-stream-info
emms-metaplaylist-mode emms-bookmarks emms-cue emms-mode-line-icon
emms-browser sort emms-playlist-sort emms-last-played emms-player-xine
emms-player-mpd tq emms-playing-time emms-lyrics emms-url
emms-tag-editor emms-mark emms-mode-line emms-cache emms-info-ogginfo
emms-info-mp3info emms-playlist-mode emms-player-vlc emms-player-mplayer
emms-info emms-streams later-do emms-source-playlist emms-source-file
locate emms-player-simple emms-setup emms emms-compat org-config-thierry
ob-sh org-crypt appt diary-lib diary-loaddefs org-annotation-helper
addressbook-bookmark message rfc822 mml mml-sec mm-decode mm-bodies
mm-encode mail-parse rfc2231 rfc2047 rfc2045 ietf-drums mailabbrev
mail-utils gmm-utils mailheader bookmark-firefox-handler
bookmark-extensions org-location-google-maps org-agenda google-maps
google-maps-static google-maps-geocode google-maps-base json org
org-macro org-footnote org-pcomplete org-list org-faces org-entities
noutline outline org-version ob-emacs-lisp ob ob-tangle ob-ref ob-lob
ob-table ob-exp org-src ob-keys ob-comint ob-core ob-eval org-compat
org-macs org-loaddefs find-func cal-menu calendar cal-loaddefs
init-helm-thierry helm-descbinds helm-extensions-autoloads helm-ring
helm-elisp helm-eval edebug eldoc helm-mode helm-files image-dired tramp
tramp-compat tramp-loaddefs trampver shell pcomplete format-spec dired-x
dired-aux ffap thingatpt helm-buffers helm-elscreen helm-tags
helm-bookmark helm-adaptive helm-info bookmark pp helm-locate helm-grep
wgrep-helm wgrep grep helm-regexp helm-plugin helm-external helm-net
browse-url xml url url-proxy url-privacy url-expand url-methods
url-history url-cookie url-domsuf url-util url-parse url-vars mailcap
helm-utils compile comint ansi-color ring helm-help helm-types helm
easy-mmode helm-source helm-multi-match helm-lib dired helm-config
helm-autoloads helm-easymenu cl-macs gv async-bytecomp async
helm-aliases epa-file epa derived epg auth-source gnus-util mm-util
mail-prsvr password-cache w3m-wget edmacro kmacro advice help-fns
net-utils eieio byte-opt bytecomp byte-compile cl-extra cconv eieio-core
slime-autoloads info easymenu package epg-config time-date avoid
cus-start cus-load cl-loaddefs cl-lib tooltip electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar dnd
fontset image regexp-opt fringe tabulated-list newcomment lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
dbusbind gfilenotify dynamic-setting font-render-setting x-toolkit x
multi-tty emacs)

Memory information:
((conses 16 561069 61155)
 (symbols 48 63339 2)
 (miscs 40 466 1009)
 (strings 32 161022 26063)
 (string-bytes 1 4716715)
 (vectors 16 63434)
 (vector-slots 8 1717700 24022)
 (floats 8 1657 607)
 (intervals 56 3072 24)
 (buffers 960 157)
 (heap 1024 51555 2258))
-- 
Thierry
Get my Gnupg key:
gpg --keyserver pgp.mit.edu --recv-keys 59F29997