From debbugs-submit-bounces@debbugs.gnu.org Fri Sep 09 22:18:29 2016 Received: (at submit) by debbugs.gnu.org; 10 Sep 2016 02:18:29 +0000 Received: from localhost ([127.0.0.1]:55036 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1biXsC-000289-Ue for submit@debbugs.gnu.org; Fri, 09 Sep 2016 22:18:29 -0400 Received: from eggs.gnu.org ([208.118.235.92]:49022) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1biX4h-0000vY-Gd for submit@debbugs.gnu.org; Fri, 09 Sep 2016 21:27:20 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1biX4Z-00044A-2J for submit@debbugs.gnu.org; Fri, 09 Sep 2016 21:27:14 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,FREEMAIL_FROM autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:55413) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1biX4Y-00043U-Un for submit@debbugs.gnu.org; Fri, 09 Sep 2016 21:27:10 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39992) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1biX4U-0000SQ-LB for bug-gnu-emacs@gnu.org; Fri, 09 Sep 2016 21:27:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1biX4Q-00041K-E6 for bug-gnu-emacs@gnu.org; Fri, 09 Sep 2016 21:27:05 -0400 Received: from aibo.runbox.com ([91.220.196.211]:46634) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1biX4P-00040n-Ur for bug-gnu-emacs@gnu.org; Fri, 09 Sep 2016 21:27:02 -0400 Received: from [10.9.9.210] (helo=mailfront10.runbox.com) by bars.runbox.com with esmtp (Exim 4.71) (envelope-from ) id 1biX4N-0006cA-80 for bug-gnu-emacs@gnu.org; Sat, 10 Sep 2016 03:26:59 +0200 Received: from c-24-22-244-161.hsd1.wa.comcast.net ([24.22.244.161] helo=[172.20.1.143]) by mailfront10.runbox.com with esmtpsa (uid:179284 ) (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) id 1biX46-00064z-Mk for bug-gnu-emacs@gnu.org; Sat, 10 Sep 2016 03:26:43 +0200 From: Gemini Lasswell Content-Type: multipart/mixed; boundary="Apple-Mail=_6ECBAF97-8E4F-4898-9530-07141E44E087" Subject: 25.1.50; testcover-start breaks should-error Message-Id: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> Date: Fri, 9 Sep 2016 18:26:40 -0700 To: bug-gnu-emacs@gnu.org Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -4.1 (----) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Fri, 09 Sep 2016 22:18:27 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -4.1 (----) --Apple-Mail=_6ECBAF97-8E4F-4898-9530-07141E44E087 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Enabling code coverage on a file containing an ert test that uses = should-error causes the test to fail. Steps to reproduce: 1. Start emacs (I used: open nextstep/Emacs.app =E2=80=94args -Q) 2. Create a buffer containing the following code and save it as bug.el: (require 'ert) (require 'testcover) (ert-deftest should-error-bug () (should-error (my-error))) (defun my-error () (/ 1 0)) 3. M-x eval-buffer RET 4. M-x ert RET t RET 5. The test passes. 6. M-x testcover-start RET bug.el RET 7. M-x ert RET t RET 8. The test fails. TAB TAB d produced the attached debugger output. --Apple-Mail=_6ECBAF97-8E4F-4898-9530-07141E44E087 Content-Disposition: attachment; filename=bug.txt Content-Type: text/plain; name="bug.txt" Content-Transfer-Encoding: quoted-printable Debugger entered--Lisp error: (arith-error) apply(debug (error (arith-error))) ert--run-test-debugger([cl-struct-ert--test-execution-info = [cl-struct-ert-test should-error-bug nil (lambda nil (testcover-enter = (quote test@should-error-bug) (function (lambda nil (testcover-after 3 = (let ... ...)))))) nil :passed nil] [cl-struct-ert-test-failed nil nil = (arith-error) ((t (lambda nil (testcover-1value 1 (/ 1 0)))) (t = testcover-enter my-error (lambda nil (testcover-1value 1 (/ 1 0)))) (t = my-error) (nil list 2 (my-error)) (nil let ((fn-7 (function = testcover-after)) (args-8 (list 2 (my-error)))) (let ((value-9 (quote = ert-form-evaluation-aborted-10))) (let (form-description-11) (let (... = ...) (condition-case -condition- ... ...) (if errorp12 nil ...))) = value-9)) (nil testcover-after 3 (let ((fn-7 (function testcover-after)) = (args-8 (list 2 ...))) (let ((value-9 ...)) (let (form-description-11) = (let ... ... ...)) value-9))) (t (lambda nil (testcover-after 3 (let = (... ...) (let ... ... value-9))))) (t testcover-enter = test@should-error-bug (lambda nil (testcover-after 3 (let (... ...) (let = ... ... value-9))))) (t (lambda nil (testcover-enter (quote = test@should-error-bug) (function (lambda nil ...))))) (t = ert--run-test-internal #0) (t ert-run-test [cl-struct-ert-test = should-error-bug nil (lambda nil (testcover-enter (quote = test@should-error-bug) (function ...))) nil :passed nil]) (t = ert-run-or-rerun-test [cl-struct-ert--stats t [[cl-struct-ert-test = should-error-bug nil (lambda nil ...) nil :passed nil]] #s(hash-table = size 1 test eql rehash-size 1.5 rehash-threshold 0.8 data = (should-error-bug 0)) [nil] [(22483 23504 1016 0)] [(22483 22641 595275 = 0)] 0 0 0 0 0 (22483 22641 578681 0) (22483 22641 595513 0) nil = [cl-struct-ert-test should-error-bug nil (lambda nil (testcover-enter = ... ...)) nil :passed nil] 1473469392.100925] [cl-struct-ert-test = should-error-bug nil (lambda nil (testcover-enter (quote = test@should-error-bug) (function ...))) nil :passed nil] #[385 = "\306=02\307\"\203)=00\211\211G\310U\203=14=00\211@\202=1B=00\311\312\313=03= GD\"\301\314=02\302\242\300#\240\210\315\301\242!\207\306=02\316\"\203\222= =00\211\211G\317U\203A=00\211=01A\262=02\242\202H=00\311\312\313=03GD\"=01= @\303\320=02\204T=00\321\202U=00\322\323=05!\324=06=06!\325=06=07!\211\326= U\203j=00\321\202n=00\327\330=02\"\262=01\331=06=08!\211\326U\203~=00\321\= 202\202=00\327\332=02\"\262=01&=06\210\333r\301\242q\210\f)=03\"\207\306=02= \334\"\203\372=00\211\211G\317U\203\252=00\211=01A\262=02\242\202\261=00\3= 11\312\313=03GD\"=01@r\301\242q\210\f\335=03=03\"\336=02=02\"\211\204\311=00= \337\340!\210\211\317H\211\326H=0D= >\204\333=00\311\341\342=03D\"\210\211\211\310=06=07I\266=03=0E-=02\343\31= 3\344\"I\210\345=03=06=06\"\210\346=03=02\"\266\203)\207\306=02\347\"\203\= 204=01\211\211G\350U\203=12=01\211=01A\262=02\242\202=19=01\311\312\313=03= GD\"=01\211A\262=03\242=02@r\301\242q\210\f\335=04=04\"\336=02=02\"\211\31= 7H\211\326H=0D= >\204A=01\311\341\342=03D\"\210\211\317H\262=01\203g=01\211\317H\211\326H=0D= >\204[=01\311\341\342=03D\"\210\211\211\317\351=06 =06 = \"I\266=03=0E-=02\343=06=06\351=06 =06 = \"\"I\210\345=03=06=07\"\210\346=03=02\"\266\203)\207\352\353=03\354#\205\= 215=01\313\207" [nil (#) (#4) message ert--results-ewoc = cl-struct-ert--ewoc-entry-tags eql run-started 1 signal = wrong-number-of-arguments nil ert--setup-results-buffer pop-to-buffer = run-ended 2 "%sRan %s tests, %s results were as expected%s%s" "" = "Aborted: " ert-stats-total ert-stats-completed-expected = ert-stats-completed-unexpected 0 format ", %s unexpected" = ert-stats-skipped ", %s skipped" ert--results-update-stats-display = test-started ert--stats-test-pos ewoc-nth cl--assertion-failed node = wrong-type-argument ert--ewoc-entry ert-char-for-test-result t = ert--results-update-stats-display-maybe ewoc-invalidate test-ended 3 = ert-test-result-expected-p error "cl-ecase failed: %s, %s" (run-started = run-ended test-started test-ended) ert--results-progress-bar-string] 16 = "\n\n(fn EVENT-TYPE &rest EVENT-ARGS)"]) (t = ert-results-rerun-test-at-point) (t = ert-results-rerun-test-at-point-debugging-errors) (t = funcall-interactively ert-results-rerun-test-at-point-debugging-errors) = (t call-interactively ert-results-rerun-test-at-point-debugging-errors = nil nil) (t command-execute = ert-results-rerun-test-at-point-debugging-errors)) nil] #[0 = "\300\301\302\"\207" [throw --cl-block-error-- nil] 3] debug t] (error = (arith-error))) #[128 "\301\300=02\"\207" [[cl-struct-ert--test-execution-info = [cl-struct-ert-test should-error-bug nil (lambda nil (testcover-enter = (quote test@should-error-bug) (function (lambda nil ...)))) nil :passed = nil] [cl-struct-ert-test-failed nil nil (arith-error) ((t (lambda nil = (testcover-1value 1 ...))) (t testcover-enter my-error (lambda nil = (testcover-1value 1 ...))) (t my-error) (nil list 2 (my-error)) (nil let = ((fn-7 ...) (args-8 ...)) (let (...) (let ... ...) value-9)) (nil = testcover-after 3 (let (... ...) (let ... ... value-9))) (t (lambda nil = (testcover-after 3 ...))) (t testcover-enter test@should-error-bug = (lambda nil (testcover-after 3 ...))) (t (lambda nil (testcover-enter = ... ...))) (t ert--run-test-internal #2) (t ert-run-test = [cl-struct-ert-test should-error-bug nil (lambda nil ...) nil :passed = nil]) (t ert-run-or-rerun-test [cl-struct-ert--stats t = [[cl-struct-ert-test should-error-bug nil ... nil :passed nil]] = #s(hash-table size 1 test eql rehash-size 1.5 rehash-threshold 0.8 data = (should-error-bug 0)) [nil] [...] [...] 0 0 0 0 0 (22483 22641 578681 0) = (22483 22641 595513 0) nil [cl-struct-ert-test should-error-bug nil ... = nil :passed nil] 1473469392.100925] [cl-struct-ert-test should-error-bug = nil (lambda nil ...) nil :passed nil] #[385 = "\306=02\307\"\203)=00\211\211G\310U\203=14=00\211@\202=1B=00\311\312\313=03= GD\"\301\314=02\302\242\300#\240\210\315\301\242!\207\306=02\316\"\203\222= =00\211\211G\317U\203A=00\211=01A\262=02\242\202H=00\311\312\313=03GD\"=01= @\303\320=02\204T=00\321\202U=00\322\323=05!\324=06=06!\325=06=07!\211\326= U\203j=00\321\202n=00\327\330=02\"\262=01\331=06=08!\211\326U\203~=00\321\= 202\202=00\327\332=02\"\262=01&=06\210\333r\301\242q\210\f)=03\"\207\306=02= \334\"\203\372=00\211\211G\317U\203\252=00\211=01A\262=02\242\202\261=00\3= 11\312\313=03GD\"=01@r\301\242q\210\f\335=03=03\"\336=02=02\"\211\204\311=00= \337\340!\210\211\317H\211\326H=0D= >\204\333=00\311\341\342=03D\"\210\211\211\310=06=07I\266=03=0E-=02\343\31= 3\344\"I\210\345=03=06=06\"\210\346=03=02\"\266\203)\207\306=02\347\"\203\= 204=01\211\211G\350U\203=12=01\211=01A\262=02\242\202=19=01\311\312\313=03= GD\"=01\211A\262=03\242=02@r\301\242q\210\f\335=04=04\"\336=02=02\"\211\31= 7H\211\326H=0D= >\204A=01\311\341\342=03D\"\210\211\317H\262=01\203g=01\211\317H\211\326H=0D= >\204[=01\311\341\342=03D\"\210\211\211\317\351=06 =06 = \"I\266=03=0E-=02\343=06=06\351=06 =06 = \"\"I\210\345=03=06=07\"\210\346=03=02\"\266\203)\207\352\353=03\354#\205\= 215=01\313\207" [nil ... ... message ert--results-ewoc = cl-struct-ert--ewoc-entry-tags eql run-started 1 signal = wrong-number-of-arguments nil ert--setup-results-buffer pop-to-buffer = run-ended 2 "%sRan %s tests, %s results were as expected%s%s" "" = "Aborted: " ert-stats-total ert-stats-completed-expected = ert-stats-completed-unexpected 0 format ", %s unexpected" = ert-stats-skipped ", %s skipped" ert--results-update-stats-display = test-started ert--stats-test-pos ewoc-nth cl--assertion-failed node = wrong-type-argument ert--ewoc-entry ert-char-for-test-result t = ert--results-update-stats-display-maybe ewoc-invalidate test-ended 3 = ert-test-result-expected-p error "cl-ecase failed: %s, %s" ... = ert--results-progress-bar-string] 16 "\n\n(fn EVENT-TYPE &rest = EVENT-ARGS)"]) (t ert-results-rerun-test-at-point) (t = ert-results-rerun-test-at-point-debugging-errors) (t = funcall-interactively ert-results-rerun-test-at-point-debugging-errors) = (t call-interactively ert-results-rerun-test-at-point-debugging-errors = nil nil) (t command-execute = ert-results-rerun-test-at-point-debugging-errors)) nil] #[0 = "\300\301\302\"\207" [throw --cl-block-error-- nil] 3] debug t] = ert--run-test-debugger] 4 "\n\n(fn &rest ARGS)"](error (arith-error)) /(1 0) (testcover-1value 1 (/ 1 0)) (lambda nil (testcover-1value 1 (/ 1 0)))() testcover-enter(my-error (lambda nil (testcover-1value 1 (/ 1 0)))) my-error() (list 2 (my-error)) (let ((fn-7 (function testcover-after)) (args-8 (list 2 (my-error)))) = (let ((value-9 (quote ert-form-evaluation-aborted-10))) (let = (form-description-11) (let ((errorp12 nil) (form-description-fn-13 = (function (lambda nil form-description-11)))) (condition-case = -condition- (unwind-protect (setq value-9 (apply fn-7 args-8)) (setq = form-description-11 (nconc ... ... ... ...)) = (ert--signal-should-execution form-description-11)) (error (setq = errorp12 t) (ert--should-error-handle-error form-description-fn-13 = -condition- (quote error) nil) (setq value-9 -condition-))) (if errorp12 = nil (ert-fail (append (funcall form-description-fn-13) (list = :fail-reason "did not signal an error")))))) value-9)) (testcover-after 3 (let ((fn-7 (function testcover-after)) (args-8 = (list 2 (my-error)))) (let ((value-9 (quote = ert-form-evaluation-aborted-10))) (let (form-description-11) (let = ((errorp12 nil) (form-description-fn-13 (function ...))) (condition-case = -condition- (unwind-protect (setq value-9 ...) (setq form-description-11 = ...) (ert--signal-should-execution form-description-11)) (error (setq = errorp12 t) (ert--should-error-handle-error form-description-fn-13 = -condition- ... nil) (setq value-9 -condition-))) (if errorp12 nil = (ert-fail (append ... ...))))) value-9))) (lambda nil (testcover-after 3 (let ((fn-7 (function testcover-after)) = (args-8 (list 2 (my-error)))) (let ((value-9 (quote = ert-form-evaluation-aborted-10))) (let (form-description-11) (let = ((errorp12 nil) (form-description-fn-13 ...)) (condition-case = -condition- (unwind-protect ... ... ...) (error ... ... ...)) (if = errorp12 nil (ert-fail ...)))) value-9))))() testcover-enter(test@should-error-bug (lambda nil (testcover-after 3 = (let ((fn-7 (function testcover-after)) (args-8 (list 2 (my-error)))) = (let ((value-9 (quote ert-form-evaluation-aborted-10))) (let = (form-description-11) (let ((errorp12 nil) (form-description-fn-13 ...)) = (condition-case -condition- (unwind-protect ... ... ...) (error ... ... = ...)) (if errorp12 nil (ert-fail ...)))) value-9))))) (lambda nil (testcover-enter (quote test@should-error-bug) (function = (lambda nil (testcover-after 3 (let ((fn-7 ...) (args-8 ...)) (let (...) = (let ... ...) value-9)))))))() ert--run-test-internal([cl-struct-ert--test-execution-info = [cl-struct-ert-test should-error-bug nil (lambda nil (testcover-enter = (quote test@should-error-bug) (function (lambda nil (testcover-after 3 = (let ... ...)))))) nil :passed nil] [cl-struct-ert-test-failed nil nil = (arith-error) ((t (lambda nil (testcover-1value 1 (/ 1 0)))) (t = testcover-enter my-error (lambda nil (testcover-1value 1 (/ 1 0)))) (t = my-error) (nil list 2 (my-error)) (nil let ((fn-7 (function = testcover-after)) (args-8 (list 2 (my-error)))) (let ((value-9 (quote = ert-form-evaluation-aborted-10))) (let (form-description-11) (let (... = ...) (condition-case -condition- ... ...) (if errorp12 nil ...))) = value-9)) (nil testcover-after 3 (let ((fn-7 (function testcover-after)) = (args-8 (list 2 ...))) (let ((value-9 ...)) (let (form-description-11) = (let ... ... ...)) value-9))) (t (lambda nil (testcover-after 3 (let = (... ...) (let ... ... value-9))))) (t testcover-enter = test@should-error-bug (lambda nil (testcover-after 3 (let (... ...) (let = ... ... value-9))))) (t (lambda nil (testcover-enter (quote = test@should-error-bug) (function (lambda nil ...))))) (t = ert--run-test-internal #0) (t ert-run-test [cl-struct-ert-test = should-error-bug nil (lambda nil (testcover-enter (quote = test@should-error-bug) (function ...))) nil :passed nil]) (t = ert-run-or-rerun-test [cl-struct-ert--stats t [[cl-struct-ert-test = should-error-bug nil (lambda nil ...) nil :passed nil]] #s(hash-table = size 1 test eql rehash-size 1.5 rehash-threshold 0.8 data = (should-error-bug 0)) [nil] [(22483 23504 1016 0)] [(22483 22641 595275 = 0)] 0 0 0 0 0 (22483 22641 578681 0) (22483 22641 595513 0) nil = [cl-struct-ert-test should-error-bug nil (lambda nil (testcover-enter = ... ...)) nil :passed nil] 1473469392.100925] [cl-struct-ert-test = should-error-bug nil (lambda nil (testcover-enter (quote = test@should-error-bug) (function ...))) nil :passed nil] #[385 = "\306=02\307\"\203)=00\211\211G\310U\203=14=00\211@\202=1B=00\311\312\313=03= GD\"\301\314=02\302\242\300#\240\210\315\301\242!\207\306=02\316\"\203\222= =00\211\211G\317U\203A=00\211=01A\262=02\242\202H=00\311\312\313=03GD\"=01= @\303\320=02\204T=00\321\202U=00\322\323=05!\324=06=06!\325=06=07!\211\326= U\203j=00\321\202n=00\327\330=02\"\262=01\331=06=08!\211\326U\203~=00\321\= 202\202=00\327\332=02\"\262=01&=06\210\333r\301\242q\210\f)=03\"\207\306=02= \334\"\203\372=00\211\211G\317U\203\252=00\211=01A\262=02\242\202\261=00\3= 11\312\313=03GD\"=01@r\301\242q\210\f\335=03=03\"\336=02=02\"\211\204\311=00= \337\340!\210\211\317H\211\326H=0D= >\204\333=00\311\341\342=03D\"\210\211\211\310=06=07I\266=03=0E-=02\343\31= 3\344\"I\210\345=03=06=06\"\210\346=03=02\"\266\203)\207\306=02\347\"\203\= 204=01\211\211G\350U\203=12=01\211=01A\262=02\242\202=19=01\311\312\313=03= GD\"=01\211A\262=03\242=02@r\301\242q\210\f\335=04=04\"\336=02=02\"\211\31= 7H\211\326H=0D= >\204A=01\311\341\342=03D\"\210\211\317H\262=01\203g=01\211\317H\211\326H=0D= >\204[=01\311\341\342=03D\"\210\211\211\317\351=06 =06 = \"I\266=03=0E-=02\343=06=06\351=06 =06 = \"\"I\210\345=03=06=07\"\210\346=03=02\"\266\203)\207\352\353=03\354#\205\= 215=01\313\207" [nil (#) (#4) message ert--results-ewoc = cl-struct-ert--ewoc-entry-tags eql run-started 1 signal = wrong-number-of-arguments nil ert--setup-results-buffer pop-to-buffer = run-ended 2 "%sRan %s tests, %s results were as expected%s%s" "" = "Aborted: " ert-stats-total ert-stats-completed-expected = ert-stats-completed-unexpected 0 format ", %s unexpected" = ert-stats-skipped ", %s skipped" ert--results-update-stats-display = test-started ert--stats-test-pos ewoc-nth cl--assertion-failed node = wrong-type-argument ert--ewoc-entry ert-char-for-test-result t = ert--results-update-stats-display-maybe ewoc-invalidate test-ended 3 = ert-test-result-expected-p error "cl-ecase failed: %s, %s" (run-started = run-ended test-started test-ended) ert--results-progress-bar-string] 16 = "\n\n(fn EVENT-TYPE &rest EVENT-ARGS)"]) (t = ert-results-rerun-test-at-point) (t = ert-results-rerun-test-at-point-debugging-errors) (t = funcall-interactively ert-results-rerun-test-at-point-debugging-errors) = (t call-interactively ert-results-rerun-test-at-point-debugging-errors = nil nil) (t command-execute = ert-results-rerun-test-at-point-debugging-errors)) nil] #[0 = "\300\301\302\"\207" [throw --cl-block-error-- nil] 3] debug t]) ert-run-test([cl-struct-ert-test should-error-bug nil (lambda nil = (testcover-enter (quote test@should-error-bug) (function (lambda nil = (testcover-after 3 (let (... ...) (let ... ... value-9))))))) nil = :passed nil]) ert-run-or-rerun-test([cl-struct-ert--stats t [[cl-struct-ert-test = should-error-bug nil (lambda nil (testcover-enter (quote = test@should-error-bug) (function (lambda nil (testcover-after 3 ...))))) = nil :passed nil]] #s(hash-table size 1 test eql rehash-size 1.5 = rehash-threshold 0.8 data (should-error-bug 0)) [nil] [(22483 23504 1016 = 0)] [(22483 22641 595275 0)] 0 0 0 0 0 (22483 22641 578681 0) (22483 = 22641 595513 0) nil [cl-struct-ert-test should-error-bug nil (lambda nil = (testcover-enter (quote test@should-error-bug) (function (lambda nil = (testcover-after 3 (let ... ...)))))) nil :passed nil] = 1473469392.100925] [cl-struct-ert-test should-error-bug nil (lambda nil = (testcover-enter (quote test@should-error-bug) (function (lambda nil = (testcover-after 3 (let (... ...) (let ... ... value-9))))))) nil = :passed nil] #[385 = "\306=02\307\"\203)=00\211\211G\310U\203=14=00\211@\202=1B=00\311\312\313=03= GD\"\301\314=02\302\242\300#\240\210\315\301\242!\207\306=02\316\"\203\222= =00\211\211G\317U\203A=00\211=01A\262=02\242\202H=00\311\312\313=03GD\"=01= @\303\320=02\204T=00\321\202U=00\322\323=05!\324=06=06!\325=06=07!\211\326= U\203j=00\321\202n=00\327\330=02\"\262=01\331=06=08!\211\326U\203~=00\321\= 202\202=00\327\332=02\"\262=01&=06\210\333r\301\242q\210\f)=03\"\207\306=02= \334\"\203\372=00\211\211G\317U\203\252=00\211=01A\262=02\242\202\261=00\3= 11\312\313=03GD\"=01@r\301\242q\210\f\335=03=03\"\336=02=02\"\211\204\311=00= \337\340!\210\211\317H\211\326H=0D= >\204\333=00\311\341\342=03D\"\210\211\211\310=06=07I\266=03=0E-=02\343\31= 3\344\"I\210\345=03=06=06\"\210\346=03=02\"\266\203)\207\306=02\347\"\203\= 204=01\211\211G\350U\203=12=01\211=01A\262=02\242\202=19=01\311\312\313=03= GD\"=01\211A\262=03\242=02@r\301\242q\210\f\335=04=04\"\336=02=02\"\211\31= 7H\211\326H=0D= >\204A=01\311\341\342=03D\"\210\211\317H\262=01\203g=01\211\317H\211\326H=0D= >\204[=01\311\341\342=03D\"\210\211\211\317\351=06 =06 = \"I\266=03=0E-=02\343=06=06\351=06 =06 = \"\"I\210\345=03=06=07\"\210\346=03=02\"\266\203)\207\352\353=03\354#\205\= 215=01\313\207" [nil (#) (#0) message ert--results-ewoc = cl-struct-ert--ewoc-entry-tags eql run-started 1 signal = wrong-number-of-arguments nil ert--setup-results-buffer pop-to-buffer = run-ended 2 "%sRan %s tests, %s results were as expected%s%s" "" = "Aborted: " ert-stats-total ert-stats-completed-expected = ert-stats-completed-unexpected 0 format ", %s unexpected" = ert-stats-skipped ", %s skipped" ert--results-update-stats-display = test-started ert--stats-test-pos ewoc-nth cl--assertion-failed node = wrong-type-argument ert--ewoc-entry ert-char-for-test-result t = ert--results-update-stats-display-maybe ewoc-invalidate test-ended 3 = ert-test-result-expected-p error "cl-ecase failed: %s, %s" (run-started = run-ended test-started test-ended) ert--results-progress-bar-string] 16 = "\n\n(fn EVENT-TYPE &rest EVENT-ARGS)"]) ert-results-rerun-test-at-point() ert-results-rerun-test-at-point-debugging-errors() = funcall-interactively(ert-results-rerun-test-at-point-debugging-errors) call-interactively(ert-results-rerun-test-at-point-debugging-errors = nil nil) command-execute(ert-results-rerun-test-at-point-debugging-errors) --Apple-Mail=_6ECBAF97-8E4F-4898-9530-07141E44E087 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 In GNU Emacs 25.1.50.8 (x86_64-apple-darwin15.6.0, NS appkit-1404.47 = Version 10.11.6 (Build 15G1004)) of 2016-09-09 built on rainbow.local Windowing system distributor 'Apple', version 10.3.1404 Recent messages: For information about GNU Emacs and the GNU system, type C-h C-a. You can run the command =E2=80=98eval-buffer=E2=80=99 with M-x ev-b RET Ran 1 tests, 1 results were as expected Edebug: test@should-error-bug Edebug: my-error You can run the command =E2=80=98testcover-start=E2=80=99 with M-x tes-s = RET Edebug: my-error Ran 1 tests, 0 results were as expected, 1 unexpected Configured using: 'configure --with-ns --without-gnutls' Configured features: JPEG IMAGEMAGICK NOTIFY ACL LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS Important settings: value of $LANG: en_US.UTF-8 locale-coding-system: utf-8-unix Major mode: ERT-Results Minor modes in effect: tooltip-mode: t global-eldoc-mode: t electric-indent-mode: t mouse-wheel-mode: t tool-bar-mode: t menu-bar-mode: t file-name-shadow-mode: t global-font-lock-mode: t font-lock-mode: t blink-cursor-mode: t auto-composition-mode: t auto-encryption-mode: t auto-compression-mode: t buffer-read-only: t line-number-mode: t transient-mark-mode: t Load-path shadows: None found. Features: (shadow sort mail-extr emacsbug message subr-x puny dired dired-loaddefs format-spec rfc822 mml mml-sec password-cache epa derived epg epg-config gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils cl-seq cl-macs testcover edebug ert pp find-func seq byte-opt gv bytecomp byte-compile cl-extra help-mode cconv ewoc easymenu debug cl-loaddefs pcase cl-lib time-date mule-util tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type mwheel term/ns-win ns-win ucs-normalize term/common-win tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow timer select scroll-bar mouse jit-lock font-lock syntax facemenu font-core term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european ethiopic indian cyrillic chinese charscript case-table epa-hook jka-cmpr-hook help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs button faces cus-face macroexp files text-properties overlay sha1 md5 base64 format env code-pages mule custom widget hashtable-print-readable backquote kqueue cocoa ns multi-tty make-network-process emacs) --Apple-Mail=_6ECBAF97-8E4F-4898-9530-07141E44E087-- From debbugs-submit-bounces@debbugs.gnu.org Tue Sep 20 00:28:13 2016 Received: (at 24402) by debbugs.gnu.org; 20 Sep 2016 04:28:13 +0000 Received: from localhost ([127.0.0.1]:57765 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bmCfF-0002dL-0Q for submit@debbugs.gnu.org; Tue, 20 Sep 2016 00:28:13 -0400 Received: from aibo.runbox.com ([91.220.196.211]:60644) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bmCfC-0002dC-Du for 24402@debbugs.gnu.org; Tue, 20 Sep 2016 00:28:11 -0400 Received: from [10.9.9.210] (helo=mailfront10.runbox.com) by bars.runbox.com with esmtp (Exim 4.71) (envelope-from ) id 1bmCfA-0005Qz-U2 for 24402@debbugs.gnu.org; Tue, 20 Sep 2016 06:28:09 +0200 Received: from cpe-108-184-181-48.socal.res.rr.com ([108.184.181.48] helo=[192.168.1.142]) by mailfront10.runbox.com with esmtpsa (uid:179284 ) (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.82) id 1bmCez-0007vF-Ln for 24402@debbugs.gnu.org; Tue, 20 Sep 2016 06:27:57 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: More Information From: Gemini Lasswell In-Reply-To: Date: Mon, 19 Sep 2016 21:27:54 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> To: 24402@debbugs.gnu.org X-Mailer: Apple Mail (2.3124) X-Spam-Score: -3.8 (---) X-Debbugs-Envelope-To: 24402 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.8 (---) I=E2=80=99ve done some investigating of why this is happening. = testcover-start transforms: (should-error (my-error)) into: (should-error (testcover-after 2 (my-error))) Then the macro expansion of should-error separates the form which it is passed into a function symbol and list of arguments, and applies the function to the arguments inside of a condition-case that traps errors. The problem is that the arguments are evaluated first, outside of the condition-case, so errors in their evaluation do not get caught. This problem is not specific to testcover. In this example, the first test passes and the second fails: (defun div-by (n) (/ 100 n)) (defmacro log-div-by (n) `(message "div-by: %d" (div-by ,n))) (ert-deftest test-div-by () (should-error (div-by 0))) (ert-deftest test-log-div-by () (should-error (log-div-by 0))) =20 From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 03 23:29:09 2017 Received: (at 24402) by debbugs.gnu.org; 4 Jul 2017 03:29:09 +0000 Received: from localhost ([127.0.0.1]:51040 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dSEWT-0001lb-AE for submit@debbugs.gnu.org; Mon, 03 Jul 2017 23:29:09 -0400 Received: from mail-it0-f54.google.com ([209.85.214.54]:34581) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dSEWR-0001lB-3D; Mon, 03 Jul 2017 23:29:07 -0400 Received: by mail-it0-f54.google.com with SMTP id v202so18085420itb.1; Mon, 03 Jul 2017 20:29:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=Y4Gtvv2y5eZXN9JOC6kLSiGZPLbcFpOo4VoIsSySo6Q=; b=motDJKiAqCPSlkBMB9Efw2sUMf5sGsnsk0+vRicflKK/oHlLHNBcBldEi0+lRZwBjz OmrgkRHUK5V8sLu34yKSGtO2sk2PtCBz9aZf2lzvneZ50RAreupibhYSgD8HJs4P3ZLA xQH/zSsm0V667vg0pT4vBWje3YkenuMQyQIFlrSxXcK+7idXDwZQ5+ZQl0nIh8E/FT5M 9ZjKZ2ap6YM0lElEb2FNznuSX7iWbDlcTogeNNArXRNOwXXmvNnJsIzU6af07YOVCr8H uNt0yTq9VSRCLM0PWwq4yaLFjlh8UAwIq8FFvtGEK/ftE84B2qO0yx3Ex1aJU/SR8gmD iT1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=Y4Gtvv2y5eZXN9JOC6kLSiGZPLbcFpOo4VoIsSySo6Q=; b=cWvEOqeMVeMObxIi7Otxi5th53J0R3t5Kp6dP9WlLkOaDddGkvReb65g0BXGiR7X2T o1CvLrgzKYIfA7l9OEowECVP7hlZN1CNBcY48oOxPEWUBmpqr9p01rV/FcqXVhPy6dd8 NVaWRqc/R1TBDBomJCiIVaVefRpPREN5RsYMA9nWR92Ime//uI0Nub68nFWrAUZB+1QO 7iM7A8+0FMQ0nL6mwIZgQ4Xr0LWlY3GkoIt9kUMHolQU1zkJ0+Laq8MrmhjQPnD2tqdS wbix9dSJHSC6Zvc4vqw8K5S8KP5CJRnwkmp5v7IFV5z4CA0u3GUvHHvKHXgHZDMpWXD1 tUNw== X-Gm-Message-State: AIVw111IwPGKfuxqoKC06Yh+odmgHV8NaBKUiELpB1Ksc2K922VqbX9H 59tHlOWDp9dC9efL X-Received: by 10.36.141.134 with SMTP id w128mr11087776itd.84.1499138940928; Mon, 03 Jul 2017 20:29:00 -0700 (PDT) Received: from lylat (S010664777d9cebe3.ss.shawcable.net. [70.64.85.59]) by smtp.gmail.com with ESMTPSA id s135sm11068828ita.2.2017.07.03.20.28.59 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Jul 2017 20:28:59 -0700 (PDT) From: Alex To: Gemini Lasswell Subject: should-error doesn't catch all errors (Was:bug#24402: More Information) References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> Date: Mon, 03 Jul 2017 21:28:55 -0600 In-Reply-To: <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> (Gemini Lasswell's message of "Mon, 19 Sep 2016 21:27:54 -0700") Message-ID: <87zickhoco.fsf_-_@lylat> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: -0.2 (/) X-Debbugs-Envelope-To: 24402 Cc: 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.2 (/) tags + 24402 confirmed quit Gemini Lasswell writes: > I=E2=80=99ve done some investigating of why this is happening. testcover-= start > transforms: > (should-error (my-error)) > into: > (should-error (testcover-after 2 (my-error))) > > Then the macro expansion of should-error separates the form which it > is passed into a function symbol and list of arguments, and applies > the function to the arguments inside of a condition-case that traps > errors. The problem is that the arguments are evaluated first, outside > of the condition-case, so errors in their evaluation do not get > caught. This problem is not specific to testcover. In this example, > the first test passes and the second fails: > > (defun div-by (n) > (/ 100 n)) > > (defmacro log-div-by (n) > `(message "div-by: %d" (div-by ,n))) > > (ert-deftest test-div-by () > (should-error (div-by 0))) > > (ert-deftest test-log-div-by () > (should-error (log-div-by 0))) >=20=20=20 I just ran into this as well. Consider these two forms: (should-error (cl-fourth "1234") :type 'wrong-type-argument) (should-error (car (cdr (cdr (cdr "1234")))) :type 'wrong-type-argument) Only the second raises an error, even though cl-fourth is equivalent to the car/cdr chain. From debbugs-submit-bounces@debbugs.gnu.org Mon Jul 03 23:36:36 2017 Received: (at control) by debbugs.gnu.org; 4 Jul 2017 03:36:36 +0000 Received: from localhost ([127.0.0.1]:51052 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dSEdg-0001xc-BI for submit@debbugs.gnu.org; Mon, 03 Jul 2017 23:36:36 -0400 Received: from mail-it0-f45.google.com ([209.85.214.45]:38843) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dSEde-0001xP-9W for control@debbugs.gnu.org; Mon, 03 Jul 2017 23:36:34 -0400 Received: by mail-it0-f45.google.com with SMTP id k192so65236620ith.1 for ; Mon, 03 Jul 2017 20:36:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:user-agent:mime-version; bh=k8cWq+JPRWT7JaIagYY5teJs3s8rJ0xchRJuJ1HAy5c=; b=fqrhg4vUcvrbsrnAIhWjJYFsmgqe5plcX3EDVL3HnufzABza/Sc0tcP1cocxYDrWL4 YtlS6MyaDscOzNdtsf2+18K07IcAVa9Gid7vSkvLGJiYJf4aXIhNgAlaKXVNNeHFdGv5 d3RF6iuWBcyuuSGxVMhTlEcj9M0uiAhDLsrQXmowbr30RJOL/m1SFuwd7QwL4CLb9dyQ X8JodoFP6WLseoBvvJoL4hxcU1sxKTJo9twHDwLNW27DhVblW9hQbAPNv5xvHHM0MVg5 x0T2rH+p/l4frSKiajZZrCxOpxvFxbs1Pes8BoowdYP+YH58VDXuy8kg8IJsir8t+3Xl CuAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:user-agent :mime-version; bh=k8cWq+JPRWT7JaIagYY5teJs3s8rJ0xchRJuJ1HAy5c=; b=n6s0ik2+K9Ey9KxWStbqcEsSK0HzEBer7vxqbhuJ4ReMYDJLwHk+LDTrolV5fcWKSv ZcVBCz6PUUCCLq2d6AmAE40lbrdmu8pSBYmtHOQ057jyB6xrqcWzYXyU3V2yR3Ohw1mq A3jgFLtpwg0/sa2WuIahRGDl8DW4c87suJNJNqEDLzSfkq5LUz2xTSqrae1lU+ApniY6 csIoVcFBJovsfGB9BIADX11yA0jQ0ofysUpTD94YM6g+rHuX/YzGdDzw6PZbVL1PFsyZ tbLfSmJzgwSZ/FOkzfxmSvUf0KnTSGNbSItatz6Tpem72e+Z8Fh5rSrD1Ty3BuN3sg1G 6p4A== X-Gm-Message-State: AIVw110ToRyJig8TS7kx95Eooet/1HPJQz1j5JUNR7A6nOZjkyhFy9Qb dzi9s5PqifsrVYCx X-Received: by 10.36.65.18 with SMTP id x18mr11747321ita.88.1499139388348; Mon, 03 Jul 2017 20:36:28 -0700 (PDT) Received: from lylat (S010664777d9cebe3.ss.shawcable.net. [70.64.85.59]) by smtp.gmail.com with ESMTPSA id 138sm1267854itw.10.2017.07.03.20.36.27 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Jul 2017 20:36:27 -0700 (PDT) From: Alex To: control@debbugs.gnu.org Subject: control message for bug 24402 Date: Mon, 03 Jul 2017 21:36:25 -0600 Message-ID: <87van8ho06.fsf@lylat> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.5 (/) tags 24402 + confirmed found 24402 26.0.50 quit From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 04 20:00:20 2017 Received: (at 24402) by debbugs.gnu.org; 5 Jul 2017 00:00:20 +0000 Received: from localhost ([127.0.0.1]:52699 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dSXjv-00011M-W5 for submit@debbugs.gnu.org; Tue, 04 Jul 2017 20:00:20 -0400 Received: from mail-io0-f169.google.com ([209.85.223.169]:35628) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dSXjt-000112-5z; Tue, 04 Jul 2017 20:00:17 -0400 Received: by mail-io0-f169.google.com with SMTP id h134so77799905iof.2; Tue, 04 Jul 2017 17:00:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=j4EYMi3hESfB1xNFMZT+MbHgATz6XmpM38TvNqAyAd0=; b=NGrLfKb5p/1Jpq5v0Lo0wU/aaLRAuTqZ/zF1nkl2mJ9UAFmmNEBukOFOl4mDEGZGfu 2q3to23K5d4nwY6/brReDaRzGcW0znDzZHz8c0jb2cej/cW76ojTzC8NbV04FypdFLwr cLxsoLqH+14yjrNDeiB0m7rQvi5gsW+kDpL8BkEVZUiir5rXJEutWUhhbmZ1m05QN7Eg ncAxRm9MO2hv+isHN+wacZAQZj7Dcf7M0TlDBcCmPiLMTiVXqTA6Nb1HXufzC6dsjqOx Eurq9GEP+w4ICqnNAnvsxWcWD3scRVSA1ApSzUiuNcf5YbZZeogx9h7JXDLsRXYAAv10 VHCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=j4EYMi3hESfB1xNFMZT+MbHgATz6XmpM38TvNqAyAd0=; b=n0vZFP2QCPb90SuF49NM9eRvhXP8dDPy5I3Sz5SN720zVdd2/Pnzr636aDx48PClyA 53omPygWOfD9lKkcxfBCMfqtatFKXKtMx/oQ0RMu1YviQ9N9UFzoncKrFoJ+4wsRQcYG 0JGe7yycrGRv0sBp96OCvKY4j4BPC39ICIF4UZyq/3ah9zb+mVduNCRwzX506nOpazt3 XofVlmfgAViBlgMd4wGHk/JsMXTpicaGoZnA6yeIDsjMAakbJEU2nz6Gp5ke6+dMk+/t OgcYgUbM9xWosPa02lGg4w9jSvTH9Uk9wvWDP0y78vZmo/vqUdmfnE9bKqo88PdJfYZR +Y8A== X-Gm-Message-State: AIVw113o0jw49et4TvJ26sAnIX3JfFAVFj54l0kjx0xXkh1p0lsl85Ah bZSjJxqcMrAj9fmV X-Received: by 10.107.165.13 with SMTP id o13mr10533583ioe.47.1499212811230; Tue, 04 Jul 2017 17:00:11 -0700 (PDT) Received: from lylat (S010664777d9cebe3.ss.shawcable.net. [70.64.85.59]) by smtp.gmail.com with ESMTPSA id j12sm10508158iod.12.2017.07.04.17.00.09 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 04 Jul 2017 17:00:09 -0700 (PDT) From: Alex To: Gemini Lasswell Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> Date: Tue, 04 Jul 2017 18:00:04 -0600 In-Reply-To: <87zickhoco.fsf_-_@lylat> (Alex's message of "Mon, 03 Jul 2017 21:28:55 -0600") Message-ID: <877eznda7v.fsf@lylat> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 24402 Cc: 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable tags 24402 patch quit Alex writes: > Gemini Lasswell writes: > >> I=E2=80=99ve done some investigating of why this is happening. testcover= -start >> transforms: >> (should-error (my-error)) >> into: >> (should-error (testcover-after 2 (my-error))) >> >> Then the macro expansion of should-error separates the form which it >> is passed into a function symbol and list of arguments, and applies >> the function to the arguments inside of a condition-case that traps >> errors. The problem is that the arguments are evaluated first, outside >> of the condition-case, so errors in their evaluation do not get >> caught. This problem is not specific to testcover. In this example, >> the first test passes and the second fails: >> >> (defun div-by (n) >> (/ 100 n)) >> >> (defmacro log-div-by (n) >> `(message "div-by: %d" (div-by ,n))) >> >> (ert-deftest test-div-by () >> (should-error (div-by 0))) >> >> (ert-deftest test-log-div-by () >> (should-error (log-div-by 0))) >>=20=20=20 > > I just ran into this as well. Consider these two forms: > > (should-error (cl-fourth "1234") :type 'wrong-type-argument) > > (should-error (car (cdr (cdr (cdr "1234")))) :type 'wrong-type-argument) > > Only the second raises an error, even though cl-fourth is equivalent to > the car/cdr chain. Here's a diff that appears to fix this. It's a bit crude, but I'd rather not mess around too much with ert's internals. --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=ert-args.diff Content-Description: first diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el index 2c49a634e3..f5f3e30c0d 100644 --- a/lisp/emacs-lisp/ert.el +++ b/lisp/emacs-lisp/ert.el @@ -303,8 +303,12 @@ ert--expand-should-1 (args (cl-gensym "args-")) (value (cl-gensym "value-")) (default-value (cl-gensym "ert-form-evaluation-aborted-"))) - `(let ((,fn (function ,fn-name)) - (,args (list ,@arg-forms))) + `(let* ((,fn (function ,fn-name)) + (,args (condition-case err + (list ,@arg-forms) + (error (progn (setq ,fn #'signal) + (list (car err) + (cdr err))))))) (let ((,value ',default-value)) ,(funcall inner-expander `(setq ,value (apply ,fn ,args)) --=-=-= Content-Type: text/plain There's a similar issue with macro-expansions inside of should/should-error/should-not that could/should be fixed by wrapping the macroexpand call at the top of ert--expand-should-1 in a similar condition-case. Here's another diff that does just that: --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=ert-args2.diff Content-Description: second diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el index eb2b2e3e11..66ae31fd51 100644 --- a/lisp/emacs-lisp/ert.el +++ b/lisp/emacs-lisp/ert.el @@ -276,13 +276,15 @@ ert--special-operator-p (defun ert--expand-should-1 (whole form inner-expander) "Helper function for the `should' macro and its variants." (let ((form - (macroexpand form (append (bound-and-true-p - byte-compile-macro-environment) - (cond - ((boundp 'macroexpand-all-environment) - macroexpand-all-environment) - ((boundp 'cl-macro-environment) - cl-macro-environment)))))) + (condition-case err + (macroexpand-all form (append (bound-and-true-p + byte-compile-macro-environment) + (cond + ((boundp 'macroexpand-all-environment) + macroexpand-all-environment) + ((boundp 'cl-macro-environment) + cl-macro-environment)))) + (error `(signal ',(car err) ',(cdr err)))))) (cond ((or (atom form) (ert--special-operator-p (car form))) (let ((value (cl-gensym "value-"))) @@ -303,8 +305,13 @@ ert--expand-should-1 (args (cl-gensym "args-")) (value (cl-gensym "value-")) (default-value (cl-gensym "ert-form-evaluation-aborted-"))) - `(let ((,fn (function ,fn-name)) - (,args (list ,@arg-forms))) + `(let* ((,fn (function ,fn-name)) + (,args (condition-case err + (list ,@arg-forms) + ;; (ert-test-failed (signal (car err) (cdr err))) + (error (progn (setq ,fn #'signal) + (list (car err) + (cdr err))))))) (let ((,value ',default-value)) ,(funcall inner-expander `(setq ,value (apply ,fn ,args)) --=-=-= Content-Type: text/plain I ran "make check" and found only one test that the above diff breaks: ert-test-test-result-expected-p. I can't seem to figure out why it doesn't work. The test fails because of these two: (let ((test (make-ert-test :body (lambda () (ert-fail "failed"))))) (should-not (ert-test-result-expected-p test (ert-run-test test)))) (let ((test (make-ert-test :body (lambda () (ert-fail "failed")) :expected-result-type ':failed))) (should (ert-test-result-expected-p test (ert-run-test test)))) I tried to re-throw the ert-test-failed signal and still the above two forms raise error an error. --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 05 09:43:24 2017 Received: (at 24402) by debbugs.gnu.org; 5 Jul 2017 13:43:24 +0000 Received: from localhost ([127.0.0.1]:53099 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dSkaS-000737-1x for submit@debbugs.gnu.org; Wed, 05 Jul 2017 09:43:24 -0400 Received: from mail-pf0-f179.google.com ([209.85.192.179]:36318) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dSkaP-00072q-7y for 24402@debbugs.gnu.org; Wed, 05 Jul 2017 09:43:21 -0400 Received: by mail-pf0-f179.google.com with SMTP id q86so129980688pfl.3 for <24402@debbugs.gnu.org>; Wed, 05 Jul 2017 06:43:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=7wCmx+8jdrcaMdP4PatqdkAWyDYf732r9UcRQfcHWQI=; b=t+68t7nDy+xJDNp76uMngha5NJd6lE7HvvxQaITXfMl917nF8RdR+TThWjaAkJMtJm LiUKIH+Wl3XaPj/eSQmSl0XM14W11lbI49Tn9L7lMuvPKJjGHoeZAj5C0J1mzl1T8O23 1MWZR56kfFj45tjm8HcGwatxWpOiVOF62YjZRyOYmiMIg7NkKi+bmh0/2oGir55LvOzE bQigUZU0/IQF3xhtvcQ4HIRDLTi9TTOTOHLkEfHcMBOJ09dV17ob7kyAGtsR3ccScuOw /hBfmo4pSOunLDIPOYtPrq9Yc+e5EgeXcBSEqcpkMIXmQYqrKG3HqUiQY6ZAnqEbgVAh BuxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=7wCmx+8jdrcaMdP4PatqdkAWyDYf732r9UcRQfcHWQI=; b=MM4vbJFkXHcIqHaHqxc389uBD8CDYQFMNs7fX0XVJAKC0zZM6F06t8PGGj76/CWq/C CmudMwcgCcXcqDYO9kEzEEQ6DwmkUWO2LEqgaBo2znpIBm5vk/uv3w5xXYgkfKVWGh9T 425XpE2L8FV/UGSLcGgOohB5jFisunknDoShpl8CGv91SgqKg3WPd+bOwAugDtZS2UmA BG4LteBuyVegmRv0gM2HbAmN3CSwLQIsMB+QT9UU8XH+KZs7PG904G2/tBKClpZlr2Zc 2ngbTUscHmURuDKTG0kUdqsuAMDRL3SgaZYZkVSkZv1ayiMk7nQ4FSnDO54e8KZLSrBz ir0A== X-Gm-Message-State: AIVw112xSIZGvpIHsHSsRRtzTiONmRaPzl4IFCP2/BqtE9GHsT+rKYTv ZgMUxWRGcS5XNFze X-Received: by 10.98.107.133 with SMTP id g127mr20315000pfc.202.1499262195063; Wed, 05 Jul 2017 06:43:15 -0700 (PDT) Received: from calancha-pc (222.139.137.133.dy.bbexcite.jp. [133.137.139.222]) by smtp.gmail.com with ESMTPSA id s87sm34126171pfa.86.2017.07.05.06.43.13 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 05 Jul 2017 06:43:14 -0700 (PDT) From: Tino Calancha To: Alex Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> Date: Wed, 05 Jul 2017 22:43:09 +0900 In-Reply-To: <877eznda7v.fsf@lylat> (Alex's message of "Tue, 04 Jul 2017 18:00:04 -0600") Message-ID: <874lur0zki.fsf@calancha-pc> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) Alex writes: > I ran "make check" and found only one test that the above diff breaks: > ert-test-test-result-expected-p. > > I can't seem to figure out why it doesn't work. The test fails because > of these two: > > (let ((test (make-ert-test :body (lambda () (ert-fail "failed"))))) > (should-not (ert-test-result-expected-p test (ert-run-test test)))) > (let ((test (make-ert-test :body (lambda () (ert-fail "failed")) > :expected-result-type ':failed))) > (should (ert-test-result-expected-p test (ert-run-test test)))) > > I tried to re-throw the ert-test-failed signal and still the above two > forms raise error an error. Hi! I just arrived from teletransportation from Bug#27559. Very fast! (and cheap!). Thank you for looking on this. I think you go in the right direction to fix this problem. * I have updated your patch and all the test suite pass (even your proposed tests in Bug#27559 without requiring "(eval '....)"). * Bear in mind that I am far to be an expert on `ert.el', and i am already in my second beer, so please double check that the patch have sense. --8<-----------------------------cut here---------------start------------->8--- commit a07f99f062f3da3418060ef30e1a00030fa0b947 Author: Tino Calancha Date: Wed Jul 5 22:11:46 2017 +0900 Tweak Alex's 2nd patch diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el index eb2b2e3e11..2d131cf99e 100644 --- a/lisp/emacs-lisp/ert.el +++ b/lisp/emacs-lisp/ert.el @@ -276,13 +276,15 @@ ert--special-operator-p (defun ert--expand-should-1 (whole form inner-expander) "Helper function for the `should' macro and its variants." (let ((form - (macroexpand form (append (bound-and-true-p - byte-compile-macro-environment) - (cond - ((boundp 'macroexpand-all-environment) - macroexpand-all-environment) - ((boundp 'cl-macro-environment) - cl-macro-environment)))))) + (condition-case err + (macroexpand-all form (append (bound-and-true-p + byte-compile-macro-environment) + (cond + ((boundp 'macroexpand-all-environment) + macroexpand-all-environment) + ((boundp 'cl-macro-environment) + cl-macro-environment)))) + (error `(signal ',(car err) ',(cdr err)))))) (cond ((or (atom form) (ert--special-operator-p (car form))) (let ((value (cl-gensym "value-"))) @@ -303,8 +305,14 @@ ert--expand-should-1 (args (cl-gensym "args-")) (value (cl-gensym "value-")) (default-value (cl-gensym "ert-form-evaluation-aborted-"))) - `(let ((,fn (function ,fn-name)) - (,args (list ,@arg-forms))) + `(let* ((,fn (function ,fn-name)) + (,args (condition-case err + (list ,@arg-forms) + (error (if (or (eq (car err) 'ert-test-failed) + (eq (car err) 'ert-test-skipped)) + (list ,@arg-forms) + (setq ,fn #'signal) + (list (car err) (cdr err))))))) (let ((,value ',default-value)) ,(funcall inner-expander `(setq ,value (apply ,fn ,args)) --8<-----------------------------cut here---------------end--------------->8--- In GNU Emacs 26.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.22.11) of 2017-07-05 Repository revision: 5d62247323f53f3ae9c7d9f51e951635887b2fb6 From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 05 15:57:03 2017 Received: (at 24402) by debbugs.gnu.org; 5 Jul 2017 19:57:03 +0000 Received: from localhost ([127.0.0.1]:53750 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dSqQ2-0004Dv-TV for submit@debbugs.gnu.org; Wed, 05 Jul 2017 15:57:03 -0400 Received: from mail-it0-f66.google.com ([209.85.214.66]:32846) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dSqQ0-0004DR-Qr for 24402@debbugs.gnu.org; Wed, 05 Jul 2017 15:57:01 -0400 Received: by mail-it0-f66.google.com with SMTP id 188so15442748itx.0 for <24402@debbugs.gnu.org>; Wed, 05 Jul 2017 12:57:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=PrmOamwlTRfx+NnW74jzgiv6sH/Tf6cmh96SQF15OQg=; b=H1a423CZyDIOByqLqqYu/hvl0HyfCjV3RgDFb4HeGZTBy6bKRTu3FJbOz3Tflr9dPI O+r5YsdqHgYahAeOEIP4gOwmFvPM3pQJ41q8Q6zWk0XN/TlYX+cpkikyB/62ULLB+RQU J6/6nBa4k4C1iCkHPEfZw3uYZpCH6622qF39P9KupyHWT6wzU2tMjUpQMfNCt8Xa12g3 oT7q6/2xjJSWP0WqKhNlUkP/mVhiKNEyY+/1Nz+V1scPZ/c1vetH3klRipIHtS2b7OEC po9grR22Sa9+GaEKl4rfabXCEkwAAWsvtGd/iNV5NPTaaG8PxivWrUCAy/RmfM5izydD ZA7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=PrmOamwlTRfx+NnW74jzgiv6sH/Tf6cmh96SQF15OQg=; b=M7edIMV/L2vuIq+9D3ktq4aK4l1x4zIUp3UfILJyeqr6KADADNfTfdrhCaIsfBF2DR sG33RJmZTBATn3RKiyV3d2Cy2jvf143RsapLmCQ9fkgt/HOcD3Qg2MppEKY2+F9f0ufO 2MshJk0Dq10GsamHUfyvNMtE09diAE95xRTjzYVYtsfoszEw5Ak/4BiUXphr/J0KptHt EwWhVL1shESsc0VD3m1Pz4IWb7pITSlyxwa4jLiWtoYbCongF3V8K0dHMQg2b7cXLObb gOZ7c712I568ZRZwUvXbbAaXWmU0Vk2JoWH5vve14t5hQvJP4zS1jpuujJEX1DkDbdPE X/aw== X-Gm-Message-State: AKS2vOzTVRFv/dIdWAzDCGnUIfjUpqodANZKYTLtvJMLnWYokKKZ7XiG l2Pj/vFhCgEzH84y X-Received: by 10.36.93.142 with SMTP id w136mr44541319ita.13.1499284614944; Wed, 05 Jul 2017 12:56:54 -0700 (PDT) Received: from lylat (S010664777d9cebe3.ss.shawcable.net. [70.64.85.59]) by smtp.gmail.com with ESMTPSA id c4sm6505666ioc.18.2017.07.05.12.56.53 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 05 Jul 2017 12:56:53 -0700 (PDT) From: Alex To: Tino Calancha Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> Date: Wed, 05 Jul 2017 13:56:48 -0600 In-Reply-To: <874lur0zki.fsf@calancha-pc> (Tino Calancha's message of "Wed, 05 Jul 2017 22:43:09 +0900") Message-ID: <87o9sywtbz.fsf@lylat> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.5 (/) Tino Calancha writes: > Hi! > I just arrived from teletransportation from Bug#27559. Very fast! (and > cheap!). > > Thank you for looking on this. I think you go in the right direction to > fix this problem. > > * I have updated your patch and all the test suite pass (even your > proposed tests in Bug#27559 without requiring "(eval '....)"). > > * Bear in mind that I am far to be an expert on `ert.el', and i am > already in my second beer, so please double check that > the patch have sense. > > > commit a07f99f062f3da3418060ef30e1a00030fa0b947 > Author: Tino Calancha > Date: Wed Jul 5 22:11:46 2017 +0900 > > Tweak Alex's 2nd patch > > diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el > index eb2b2e3e11..2d131cf99e 100644 > --- a/lisp/emacs-lisp/ert.el > +++ b/lisp/emacs-lisp/ert.el > @@ -276,13 +276,15 @@ ert--special-operator-p > (defun ert--expand-should-1 (whole form inner-expander) > "Helper function for the `should' macro and its variants." > (let ((form > - (macroexpand form (append (bound-and-true-p > - byte-compile-macro-environment) > - (cond > - ((boundp 'macroexpand-all-environment) > - macroexpand-all-environment) > - ((boundp 'cl-macro-environment) > - cl-macro-environment)))))) > + (condition-case err > + (macroexpand-all form (append (bound-and-true-p > + byte-compile-macro-environment) > + (cond > + ((boundp 'macroexpand-all-environment) > + macroexpand-all-environment) > + ((boundp 'cl-macro-environment) > + cl-macro-environment)))) > + (error `(signal ',(car err) ',(cdr err)))))) > (cond > ((or (atom form) (ert--special-operator-p (car form))) > (let ((value (cl-gensym "value-"))) > @@ -303,8 +305,14 @@ ert--expand-should-1 > (args (cl-gensym "args-")) > (value (cl-gensym "value-")) > (default-value (cl-gensym "ert-form-evaluation-aborted-"))) > - `(let ((,fn (function ,fn-name)) > - (,args (list ,@arg-forms))) > + `(let* ((,fn (function ,fn-name)) > + (,args (condition-case err > + (list ,@arg-forms) > + (error (if (or (eq (car err) 'ert-test-failed) > + (eq (car err) 'ert-test-skipped)) > + (list ,@arg-forms) > + (setq ,fn #'signal) > + (list (car err) (cdr err))))))) > (let ((,value ',default-value)) > ,(funcall inner-expander > `(setq ,value (apply ,fn ,args)) Thanks, that helps with my understanding of the issue. Sadly, I don't think your tweak will work in all cases, namely whenever (list ,@arg-forms) has side-effects. Luckily this doesn't happen in any current tests, but I think those types of tests should be supported. I believe the following is why my previous diff doesn't work. Consider: (let ((test (make-ert-test :body (lambda () (ert-fail "failed"))))) (ert-run-test test)) The above returns a struct/record and does not error. (let ((test (make-ert-test :body (lambda () (ert-fail "failed"))))) (condition-case err (ert-run-test test) (error "woops"))) Even though ert-run-test by itself does not error, the error handler is ran. I believe this is because `ert--run-test-internal' binds `debugger' around the evaluation of the test to somehow suppress this error. Using condition-case-unless-debug gets around this, but now anytime debug-on-error is non-nil the condition-case won't catch the error. I'm not sure how to solve this. From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 07 06:15:27 2017 Received: (at 24402) by debbugs.gnu.org; 7 Jul 2017 10:15:27 +0000 Received: from localhost ([127.0.0.1]:55758 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dTQII-0002CY-UI for submit@debbugs.gnu.org; Fri, 07 Jul 2017 06:15:27 -0400 Received: from mail-pf0-f180.google.com ([209.85.192.180]:35372) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dTQIH-0002CL-8K for 24402@debbugs.gnu.org; Fri, 07 Jul 2017 06:15:25 -0400 Received: by mail-pf0-f180.google.com with SMTP id c73so15118840pfk.2 for <24402@debbugs.gnu.org>; Fri, 07 Jul 2017 03:15:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=cdABWZdT5DGKTzl2Q7APgbA1cpJq52fO+Uo7chgUyFc=; b=b3foczHrfjJYuqR31XKQhx02tKivhn+U+R3TqA2mjlnAmc5wEYLAWMHwAdO7DYy9SC 4l+hMMpmJLmHPxGFP21L6yZQaRqx5M3eI6MoeqxtgDpvUwyYdHsmA45H3kcbGh0N9Z7P Jzq825pSmB2czP5BmTLHGPAXbXwe5AYOcE+2e2QH39qcNyWg0Hm7rtiky32a27/Vezgj QqhReNsgpQMj2/wVpO5pdbyhibva/a48J9CJ+YvLM3dLzSeZEJBZnwUwlT8NwPw03BJp 5wR+4xBiHSoeuBjCeuASHzQ/JPqu0MB60ZxOFNhyGyel/b+hHXaXJ4kooFHOaApA+g2v AFnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=cdABWZdT5DGKTzl2Q7APgbA1cpJq52fO+Uo7chgUyFc=; b=GQ0vevNLK7S55YrOldDXRp2lLzWyfAr1yAfFjCBKma6OG20/6Eoz7xfwSVME2b85GT esWRZNAFOhw6uC3zokibWMcLYsxvUt9h5FEg1D0IMzAFJn93ax+Thi0nuNsR9t8FuKP0 50FfxxVGXncPXkYCxHBT4ETIq1h+pqJllq634omiwWpnwHPa0AT+oBN60K4MPk/bKSFK k4XhqWydUemEDeFp4emBYAn+c3eVahni6/FFc+zwBhcqxR2oEvEF3z8qFtyGIWuNTxeQ MlL7WNCxTZ/xZcJINuvFkkynEHrBMgg4udRwJLi9MpmSwvFwm64j7y0nZDUPfsjppZQO GJ0w== X-Gm-Message-State: AIVw110Kgaj92f1xBfhTIBNXgQwMgnIHXjpIEJeNqCgEUrYzWxH3uWZi Zs79zJhgwAxpoL7T X-Received: by 10.84.232.74 with SMTP id f10mr2096474pln.154.1499422519053; Fri, 07 Jul 2017 03:15:19 -0700 (PDT) Received: from calancha-pc (222.139.137.133.dy.bbexcite.jp. [133.137.139.222]) by smtp.gmail.com with ESMTPSA id e13sm5539361pfh.96.2017.07.07.03.15.17 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 07 Jul 2017 03:15:18 -0700 (PDT) From: Tino Calancha To: Alex Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> Date: Fri, 07 Jul 2017 19:15:13 +0900 In-Reply-To: <87o9sywtbz.fsf@lylat> (Alex's message of "Wed, 05 Jul 2017 13:56:48 -0600") Message-ID: <87d19clfim.fsf@calancha-pc> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) Alex writes: > I don't think your tweak will work in all cases, namely whenever > (list ,@arg-forms) has side-effects. Luckily this doesn't happen in any > current tests, but I think those types of tests should be supported. Good point. > I believe the following is why my previous diff doesn't work. Consider: > > (let ((test (make-ert-test :body (lambda () (ert-fail "failed"))))) > (ert-run-test test)) > > The above returns a struct/record and does not error. > > (let ((test (make-ert-test :body (lambda () (ert-fail "failed"))))) > (condition-case err > (ert-run-test test) > (error "woops"))) > > Even though ert-run-test by itself does not error, the error handler is > ran. I believe this is because `ert--run-test-internal' binds `debugger' > around the evaluation of the test to somehow suppress this error. > > Using condition-case-unless-debug gets around this, but now anytime > debug-on-error is non-nil the condition-case won't catch the error. I'm > not sure how to solve this. I've being thinking about this, and i cannot find something better than your second patch. My suggestion is: 1. We apply your 2nd patch. 2. We ammend the failing tests wrapping '> I don't think your tweak will work in all cases, namely whenever > (list ,@arg-forms) has side-effects. Luckily this doesn't happen in any > current tests, but I think those types of tests should be supported. Good point. > I believe the following is why my previous diff doesn't work. Consider: > > (let ((test (make-ert-test :body (lambda () (ert-fail "failed"))))) > (ert-run-test test)) > > The above returns a struct/record and does not error. > > (let ((test (make-ert-test :body (lambda () (ert-fail "failed"))))) > (condition-case err > (ert-run-test test) > (error "woops"))) > > Even though ert-run-test by itself does not error, the error handler is > ran. I believe this is because `ert--run-test-internal' binds `debugger' > around the evaluation of the test to somehow suppress this error. > > Using condition-case-unless-debug gets around this, but now anytime > debug-on-error is non-nil the condition-case won't catch the error. I'm > not sure how to solve this. I've being thinking about this, and i cannot find something better than your second patch. My suggestion is: 1. We apply your 2nd patch. 2. We handle the failing tests wrapping '(ert-fail "failed")' into `ignore-errors' as in below patch. Then, IMO we are in a better situation than we are know: That fix this bug, and if i am nt missing something, at a low price: just tweaking a bit 2 innocents tests. What do you think? --8<-----------------------------cut here---------------start------------->8--- diff --git a/test/lisp/emacs-lisp/ert-tests.el b/test/lisp/emacs-lisp/ert-tests.el index 317838b250..f3e4db192a 100644 --- a/test/lisp/emacs-lisp/ert-tests.el +++ b/test/lisp/emacs-lisp/ert-tests.el @@ -413,12 +413,14 @@ ert-test--which-file (let ((test (make-ert-test :body (lambda ())))) (should (ert-test-result-expected-p test (ert-run-test test)))) ;; unexpected failure - (let ((test (make-ert-test :body (lambda () (ert-fail "failed"))))) - (should-not (ert-test-result-expected-p test (ert-run-test test)))) - ;; expected failure - (let ((test (make-ert-test :body (lambda () (ert-fail "failed")) - :expected-result-type ':failed))) + (let ((test (make-ert-test + :body (lambda () (ignore-errors (ert-fail "failed")))))) (should (ert-test-result-expected-p test (ert-run-test test)))) + ;; expected failure + (let ((test (make-ert-test + :body (lambda () (ignore-errors (ert-fail "failed"))) + :expected-result-type ':failed))) + (should-not (ert-test-result-expected-p test (ert-run-test test)))) ;; `not' expected type (let ((test (make-ert-test :body (lambda ()) :expected-result-type '(not :failed)))) --8<-----------------------------cut here---------------end--------------->8--- From debbugs-submit-bounces@debbugs.gnu.org Sun Jul 09 03:04:39 2017 Received: (at 24402) by debbugs.gnu.org; 9 Jul 2017 07:04:39 +0000 Received: from localhost ([127.0.0.1]:58321 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dU6Gl-0004oJ-Hg for submit@debbugs.gnu.org; Sun, 09 Jul 2017 03:04:39 -0400 Received: from mail-io0-f196.google.com ([209.85.223.196]:36506) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dU6Gj-0004o6-9N for 24402@debbugs.gnu.org; Sun, 09 Jul 2017 03:04:39 -0400 Received: by mail-io0-f196.google.com with SMTP id h134so2419028iof.3 for <24402@debbugs.gnu.org>; Sun, 09 Jul 2017 00:04:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=m/phVQmNRp5hbtgo2w80fVa/pBk3XXG65KB9HqdYSdU=; b=ZvIugZAJzduufRwE8XIqFXvce94nJYWBDNBc38QfDlOF7+dSXldAF271DIumdO1Day BV8a1F8JfPVglp4qkYQfHEXgtiDutHMJbtj9//jNBBzBxfUiXAGGwc6AO+rFMD9qAaNs fcDnp4N/ZAQMGX38fVpACYbTaQVvtnXS6/UzteuAYmEp6waAePU1zDbyD8YuTCJHnD+W 1tcUwsyD4oxd+1AYI5QPw6n2shtDk3LB9Ypwy9nAJTQBPSStnsMRx5bch7GRlOeTcELB JhjKALSo36frUqMZupysR9JG+Q0r0jrTvUwKpemhzMb7g26bRBG0SNOvEcyENUkra78Z GVVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=m/phVQmNRp5hbtgo2w80fVa/pBk3XXG65KB9HqdYSdU=; b=Ykm7HIxjlxLRYg//MEa8+UZRzDGkcPz/8rWE07aH048dmi+CY2Yb08uhNqpV4jC3TZ FQvJFMz7SHpZ7qh9mGnYSlRT4jPPH7bD/HZcpvdftjbwtDE6ZaSwIzNe2mAinuH8Bw08 ZozxvzZAqDonEeJMJ20AocR0aV/dFU3362OeigbqV1fBHlJqGugJvop7UZsIItRZspLa gYh3DcWm0WQjmJE36ulC5/D0p02//j8H/KyO46iQ3O1GKbO/srakBqjb2dKnfelBbN+4 ohwFd34HxS0uxX/rZ36p8xFE2Oc+ypLdqv+gd10aCg7Cgflil44PAr5t+Esa7og4jm+L a0Qg== X-Gm-Message-State: AIVw110eIE4pUN++Itndkx9eVP2TN44WMo2ZMApSdDVEMaUM7pphQoMD AZqO0IO+OvjYjQ== X-Received: by 10.107.138.17 with SMTP id m17mr10068652iod.155.1499583871551; Sun, 09 Jul 2017 00:04:31 -0700 (PDT) Received: from lylat (S010664777d9cebe3.ss.shawcable.net. [70.64.85.59]) by smtp.gmail.com with ESMTPSA id 194sm1617129itl.4.2017.07.09.00.04.29 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sun, 09 Jul 2017 00:04:30 -0700 (PDT) From: Alex To: Tino Calancha Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87d19clfim.fsf@calancha-pc> Date: Sun, 09 Jul 2017 01:04:24 -0600 In-Reply-To: <87d19clfim.fsf@calancha-pc> (Tino Calancha's message of "Fri, 07 Jul 2017 19:15:13 +0900") Message-ID: <87bmounlaf.fsf@lylat> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , eliz@gnu.org, npostavs@users.sourceforge.net, 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) Tino Calancha writes: > I've being thinking about this, and i cannot find something better than > your second patch. > > My suggestion is: > > 1. We apply your 2nd patch. > 2. We handle the failing tests wrapping '(ert-fail "failed")' into > `ignore-errors' as in below patch. > > Then, IMO we are in a better situation than we are know: > That fix this bug, and if i am nt missing something, at a low price: just > tweaking a bit 2 innocents tests. > > What do you think? > > > diff --git a/test/lisp/emacs-lisp/ert-tests.el b/test/lisp/emacs-lisp/ert-tests.el > index 317838b250..f3e4db192a 100644 > --- a/test/lisp/emacs-lisp/ert-tests.el > +++ b/test/lisp/emacs-lisp/ert-tests.el > @@ -413,12 +413,14 @@ ert-test--which-file > (let ((test (make-ert-test :body (lambda ())))) > (should (ert-test-result-expected-p test (ert-run-test test)))) > ;; unexpected failure > - (let ((test (make-ert-test :body (lambda () (ert-fail "failed"))))) > - (should-not (ert-test-result-expected-p test (ert-run-test test)))) > - ;; expected failure > - (let ((test (make-ert-test :body (lambda () (ert-fail "failed")) > - :expected-result-type ':failed))) > + (let ((test (make-ert-test > + :body (lambda () (ignore-errors (ert-fail "failed")))))) > (should (ert-test-result-expected-p test (ert-run-test test)))) > + ;; expected failure > + (let ((test (make-ert-test > + :body (lambda () (ignore-errors (ert-fail "failed"))) > + :expected-result-type ':failed))) > + (should-not (ert-test-result-expected-p test (ert-run-test test)))) > ;; `not' expected type > (let ((test (make-ert-test :body (lambda ()) > :expected-result-type '(not :failed)))) I agree that it's a low price, and that it fixes more than it breaks, but it does involve switching the logic in a simple basic test. Also note that ignore-errors expands into a condition-case, so instead of changing the tests, the patch could be tweaked to return nil if the test signals ert-test-{failed, skipped}. Or the two tests could use condition-case directly and make sure that ert-test-failed is/isn't signalled. I'd like to get some more opinions on this bug, in the hopes that there's a better solution. Eli/Noam (or anyone happening upon this), do you have any ideas on how to solve this? From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 11 08:17:19 2017 Received: (at 24402) by debbugs.gnu.org; 11 Jul 2017 12:17:19 +0000 Received: from localhost ([127.0.0.1]:33418 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dUu6R-0007ti-4D for submit@debbugs.gnu.org; Tue, 11 Jul 2017 08:17:19 -0400 Received: from mail-it0-f66.google.com ([209.85.214.66]:34323) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dUu6P-0007tV-Mv for 24402@debbugs.gnu.org; Tue, 11 Jul 2017 08:17:18 -0400 Received: by mail-it0-f66.google.com with SMTP id o202so2785196itc.1 for <24402@debbugs.gnu.org>; Tue, 11 Jul 2017 05:17:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=TdZfaBKbSfubgoxSc6NIPerDz7+8taZ/IYYHTOr30bA=; b=rIYntnXAL4s/+nLIg7tTK4P5lUWka7q3r81TyjoniMA+XCCKfAF2GmJoOuM9CB8V/z ZMlpmpYC2SMW0ph4MeyEqOpViSAEEts/LTPn11nyt4wBokHPTJLApmgNJhxj7Fh6QCt9 9UsCcU+WPE0f0TGsXkpIDYEtUuifYsqUvEp7hIzmLCaYBrxRaPnGzejVp8r1y483hHWU VZu4ZzR3C++0i/R+UppLAzMDDbdsB6lO5QVJy6h/WjtM6Bg1hTplVUW9xQ1Tl3o6gxOG X08xy5YYC5h7j+wP0s0ItmJ3ZoHg4T1kLLNI06AJoIF50cgJkixU/7dpCE0uat7qHL1R cNJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:references:date :in-reply-to:message-id:user-agent:mime-version; bh=TdZfaBKbSfubgoxSc6NIPerDz7+8taZ/IYYHTOr30bA=; b=JRx8vLpGCCbDFiuhOHT6XqL1HLG/vY3iLPEAg49OXPiSOq+bMc4XrJ4B8epXQ6tu+x eu3ZKQUK8cgVGO69CTCHi8YRay4/0T/KNZXMj2rdqHlfg2TDXn+FBoLmqW7ATZ4oJf9b rSu4Jq/43JFmPYZRCBTGfhPpzj/kT6tHFdEX2jmsGX1EdahkeceGwgvx25hCTV953HtP o/UWkovxWGrcuLQ07UljQMtggwj5yYEJY2UVGeaxlEfDFjk5Rlj6odD+ouJFcTpSYZLU UiNNcEGK1ulNcSWFd2sQd+0Fxd+jP1qSkf9QQB5UeUG+KcPivRERLhcN94UEU9AQ66pR oRsA== X-Gm-Message-State: AIVw112S4tEmmgA/Y69XcA0TIp6ZWOmrqcKFDdlG9JAOQIs5yGJQti+p 4hvQ/cX3SqxtYQ== X-Received: by 10.36.118.80 with SMTP id z77mr16771899itb.53.1499775432200; Tue, 11 Jul 2017 05:17:12 -0700 (PDT) Received: from zony ([45.2.7.65]) by smtp.googlemail.com with ESMTPSA id l27sm2675005ioi.6.2017.07.11.05.17.10 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 11 Jul 2017 05:17:11 -0700 (PDT) From: npostavs@users.sourceforge.net To: Alex Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> Date: Tue, 11 Jul 2017 08:18:43 -0400 In-Reply-To: <87o9sywtbz.fsf@lylat> (Alex's message of "Wed, 05 Jul 2017 13:56:48 -0600") Message-ID: <87fue3f9p8.fsf@users.sourceforge.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.7 (/) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , 24402@debbugs.gnu.org, Tino Calancha X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.7 (/) Alex writes: > I believe the following is why my previous diff doesn't work. Consider: > > (let ((test (make-ert-test :body (lambda () (ert-fail "failed"))))) > (ert-run-test test)) > > The above returns a struct/record and does not error. > > (let ((test (make-ert-test :body (lambda () (ert-fail "failed"))))) > (condition-case err > (ert-run-test test) > (error "woops"))) > > Even though ert-run-test by itself does not error, the error handler is > ran. I believe this is because `ert--run-test-internal' binds `debugger' > around the evaluation of the test to somehow suppress this error. Yes, ert binds `debugger' in order to get full backtrace information when there is an error. This means it won't see errors caught by condition-case. That's good when it ignores errors caught by test code using condition-case, but does give rise to problems. There is some relevant discussion in Bugs #11218 and #24617. Espcially the suggestion in #24617 of using `signal-hook-function' to record error info instead of using `debugger', I think doing this could simplify things a lot. It is definitely going to require messing around with ert's internals though... From debbugs-submit-bounces@debbugs.gnu.org Tue Jul 11 23:47:55 2017 Received: (at 24402) by debbugs.gnu.org; 12 Jul 2017 03:47:55 +0000 Received: from localhost ([127.0.0.1]:34567 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dV8d1-0007uy-KK for submit@debbugs.gnu.org; Tue, 11 Jul 2017 23:47:55 -0400 Received: from mail-io0-f195.google.com ([209.85.223.195]:35947) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dV8d0-0007un-GL for 24402@debbugs.gnu.org; Tue, 11 Jul 2017 23:47:55 -0400 Received: by mail-io0-f195.google.com with SMTP id h134so1036156iof.3 for <24402@debbugs.gnu.org>; Tue, 11 Jul 2017 20:47:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=YttQ6BmdnrvQqRAs8Y0ohl2RuG8jo0P9/GhrCSAvYnw=; b=XyCgS/V31uObG81HmNKeuNLyZlPbhkWhdUTZXvrZEDAgvghWXlk3xmpJvra8fj0kgi 6FFzb51tFBJSW9A11r4EsMVTidQzzVrK4qX9w/ck4AlacbNWWnkpwQoXIvSo6x1/vrUQ Tq/sa2uAjdmAy+n7c1AiCwyRV9QhPwCj2xGm4wuUFFuzdlvYuqQaUnodGlR0CmYCSKJA bunYH6jmZ+3q/Md7PvJvHAHvPoCLAjqzSti3tCWJaq+XFoVdK8FIroZgnQ87lQ+unREo NExPa3JY52U3EBz7XzybjfDZ5g8dUUIl7DmP7xTI8AlpK7VdCNWLaLpPr768uh910hRS GRPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=YttQ6BmdnrvQqRAs8Y0ohl2RuG8jo0P9/GhrCSAvYnw=; b=ml0Y4/+vYNGsVxKlLrT+bet7v/4CbcvAzpKoPRmL83fbPfjvc+5P2ZsJZw86QJZtoN vAJgyoMGqaduPr3hK23kBUgwlKN/K/zwF76SotH5/Ldx5e1aQQLNGw4JSm5q+j0Z/AO7 hePiur8V4oXVmsSrwFEWfiRjPlfWud2ODkzP0jc6N+/0dmh3N+deZWjEo/lw6JQtwdjm FBbZKqH3hMstCA0h+AQ/RfHcK6Zu4jPgA4JJpJyIkRQJ0pgG7isyEBBwETC5dvT6BxHx ofI45ILQh2itQGuoaqcOwCl5iDD9KAIiQVruQvxG/WUyaE3gTGq2lqWwJYiGK9CUvYRM 6X/A== X-Gm-Message-State: AIVw113T2yEB7WxFuag1tZ2CHEWfZmx5aBSZ/Ol+JD7aPSJ14JWrtHoD UsfYfp5KSom/EA== X-Received: by 10.107.178.214 with SMTP id b205mr3593963iof.68.1499831267797; Tue, 11 Jul 2017 20:47:47 -0700 (PDT) Received: from lylat (S010664777d9cebe3.ss.shawcable.net. [70.64.85.59]) by smtp.gmail.com with ESMTPSA id k16sm666594itb.1.2017.07.11.20.47.45 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 11 Jul 2017 20:47:46 -0700 (PDT) From: Alex To: npostavs@users.sourceforge.net Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> Date: Tue, 11 Jul 2017 21:47:44 -0600 In-Reply-To: <87fue3f9p8.fsf@users.sourceforge.net> (npostavs's message of "Tue, 11 Jul 2017 08:18:43 -0400") Message-ID: <87vamyl3j3.fsf@lylat> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , 24402@debbugs.gnu.org, Tino Calancha X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) --=-=-= Content-Type: text/plain npostavs@users.sourceforge.net writes: > Yes, ert binds `debugger' in order to get full backtrace information > when there is an error. This means it won't see errors caught by > condition-case. That's good when it ignores errors caught by test code > using condition-case, but does give rise to problems. There is some > relevant discussion in Bugs #11218 and #24617. > > Espcially the suggestion in #24617 of using `signal-hook-function' to > record error info instead of using `debugger', I think doing this could > simplify things a lot. It is definitely going to require messing around > with ert's internals though... Thanks for the info. I may have discovered a workaround, but I'm not sure if there's any negative side-effects. All the tests pass, though. What do you think of it? It's obviously not ideal, but I think it at least fixes the issues at hand. --=-=-= Content-Type: text/x-diff Content-Disposition: inline; filename=ert-3.diff Content-Description: v3 diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el index eb2b2e3e11..beb32c48ce 100644 --- a/lisp/emacs-lisp/ert.el +++ b/lisp/emacs-lisp/ert.el @@ -266,6 +266,14 @@ ert--signal-should-execution (when ert--should-execution-observer (funcall ert--should-execution-observer form-description))) +;; See Bug#24402 for why this exists +(defun ert--signal-hook (error-symbol data) + "Stupid hack to stop `condition-case' from catching ert signals. +It should only be stopped when ran from inside ert--run-test-internal." + (when (and (not (symbolp debugger)) ; only run on anonymous procedures + (memq error-symbol '(ert-test-failed ert-test-skipped))) + (funcall debugger 'error data))) + (defun ert--special-operator-p (thing) "Return non-nil if THING is a symbol naming a special operator." (and (symbolp thing) @@ -276,13 +284,15 @@ ert--special-operator-p (defun ert--expand-should-1 (whole form inner-expander) "Helper function for the `should' macro and its variants." (let ((form - (macroexpand form (append (bound-and-true-p - byte-compile-macro-environment) - (cond - ((boundp 'macroexpand-all-environment) - macroexpand-all-environment) - ((boundp 'cl-macro-environment) - cl-macro-environment)))))) + (condition-case err + (macroexpand-all form (append (bound-and-true-p + byte-compile-macro-environment) + (cond + ((boundp 'macroexpand-all-environment) + macroexpand-all-environment) + ((boundp 'cl-macro-environment) + cl-macro-environment)))) + (error `(signal ',(car err) ',(cdr err)))))) (cond ((or (atom form) (ert--special-operator-p (car form))) (let ((value (cl-gensym "value-"))) @@ -303,8 +313,13 @@ ert--expand-should-1 (args (cl-gensym "args-")) (value (cl-gensym "value-")) (default-value (cl-gensym "ert-form-evaluation-aborted-"))) - `(let ((,fn (function ,fn-name)) - (,args (list ,@arg-forms))) + `(let* ((,fn (function ,fn-name)) + (,args (condition-case err + (let ((signal-hook-function #'ert--signal-hook)) + (list ,@arg-forms)) + (error (progn (setq ,fn #'signal) + (list (car err) + (cdr err))))))) (let ((,value ',default-value)) ,(funcall inner-expander `(setq ,value (apply ,fn ,args)) diff --git a/test/lisp/emacs-lisp/ert-tests.el b/test/lisp/emacs-lisp/ert-tests.el index 317838b250..b6a7eb68da 100644 --- a/test/lisp/emacs-lisp/ert-tests.el +++ b/test/lisp/emacs-lisp/ert-tests.el @@ -294,6 +294,15 @@ ert-self-test-and-exit "the error signaled was a subtype of the expected type"))))) )) +(ert-deftest ert-test-should-error-argument () + "Errors due to evaluating arguments should not break tests." + (should-error (identity (/ 1 0)))) + +(ert-deftest ert-test-should-error-macroexpansion () + "Errors due to expanding macros should not break tests." + (cl-macrolet ((test () (error "Foo"))) + (should-error (test)))) + (ert-deftest ert-test-skip-unless () ;; Don't skip. (let ((test (make-ert-test :body (lambda () (skip-unless t))))) --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 12 08:29:30 2017 Received: (at 24402) by debbugs.gnu.org; 12 Jul 2017 12:29:30 +0000 Received: from localhost ([127.0.0.1]:34870 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVGlm-0000XS-Fz for submit@debbugs.gnu.org; Wed, 12 Jul 2017 08:29:30 -0400 Received: from mail-io0-f170.google.com ([209.85.223.170]:35945) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVGll-0000XH-J1 for 24402@debbugs.gnu.org; Wed, 12 Jul 2017 08:29:29 -0400 Received: by mail-io0-f170.google.com with SMTP id z62so11580954ioi.3 for <24402@debbugs.gnu.org>; Wed, 12 Jul 2017 05:29:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=9Rh5L6LMmIuigdLufWoRQTiP+ct7wxgVhBJdhGfvO7Q=; b=p7UJBJBu7Ep6yjGfk3X/f6jhr/VDQAV0wBv+bsP/EXCsSzLAmq5KZmMraVNlkm6sAy 6cfc9VDAo6Kl2fDzp2gPvoy26hImzcvH1HaGvPm8q0wR+xvjZyoqTL7k6kkT78Wjfqx2 iycsvSteXKoOtN7V53Ug07ltBJNlA9yBAoz8+VL3mFQYG1QvCuAdX6Ks5CpaXsieRmun qaDYdTIg//sR/G5w5XJ8IM/JL/1VnzWHK4/+rfLhUPBI0dTjWmFmIgAa9FZnaU5orjfN EXRvBLjHZ253KV5zIYF34tXLvELBZXGPXbGVkJZTnbAHysTlgWV96mVWcoP07ydzxQWw gviw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:references:date :in-reply-to:message-id:user-agent:mime-version; bh=9Rh5L6LMmIuigdLufWoRQTiP+ct7wxgVhBJdhGfvO7Q=; b=tZYhSWhvUEPuf5V77IyM+NpSNxhmtsiRjNL2kSCQWwtbiaBrAvYlJi3ZVzO9GbmMXp +kLgAuqkb/cTCaRuuQbCkRy17AqchMHy5/7Mo9I8RCdhFGKlTfPZ65JKOwxgSrQl36iu L8kxBiKJXZuH5DwO3QSCCM01vQcZbKJQbvoH0ATog80QZlf2RfQBO8/g8HZVkL9JojiK t2fMHfUafckK/yMvXTQ1YWz6iIP4F5nWQ/N8pPyfRHLfcosojv9vYcsv/Z9cVBY0xKfj Lr9GETAlheXNIo3WPt5mbigg7N3kvf0KqiJASNjHYaIntEQkblQ4ofBFxyjwGXUZQ5nT bltA== X-Gm-Message-State: AIVw113Sg3UyOtI7Qt7MK+NZG2if4bh5joNCiS0BCU6i4d/SyaIrtfSP HeDXVS9ZTsEjAA== X-Received: by 10.107.46.230 with SMTP id u99mr4894545iou.52.1499862563920; Wed, 12 Jul 2017 05:29:23 -0700 (PDT) Received: from zony ([45.2.7.65]) by smtp.googlemail.com with ESMTPSA id c4sm1451642ioc.18.2017.07.12.05.29.22 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 12 Jul 2017 05:29:23 -0700 (PDT) From: npostavs@users.sourceforge.net To: Alex Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> Date: Wed, 12 Jul 2017 08:30:56 -0400 In-Reply-To: <87vamyl3j3.fsf@lylat> (Alex's message of "Tue, 11 Jul 2017 21:47:44 -0600") Message-ID: <87tw2het1b.fsf@users.sourceforge.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.1 (--) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , Tino Calancha , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.1 (--) Alex writes: > npostavs@users.sourceforge.net writes: > >> Yes, ert binds `debugger' in order to get full backtrace information >> when there is an error. This means it won't see errors caught by >> condition-case. That's good when it ignores errors caught by test code >> using condition-case, but does give rise to problems. There is some >> relevant discussion in Bugs #11218 and #24617. >> >> Espcially the suggestion in #24617 of using `signal-hook-function' to >> record error info instead of using `debugger', I think doing this could >> simplify things a lot. It is definitely going to require messing around >> with ert's internals though... > > Thanks for the info. I may have discovered a workaround, but I'm not > sure if there's any negative side-effects. All the tests pass, though. > > What do you think of it? It's obviously not ideal, but I think it at > least fixes the issues at hand. Does it also work when loading the elc version of the test file? (try 'make check TEST_LOAD_EL=no') What about tests like this? (ert-deftest check-error-handling () (should (eq 42 (condition-case () (/ 1 0) (arith-error 42))))) From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 12 12:45:28 2017 Received: (at 24402) by debbugs.gnu.org; 12 Jul 2017 16:45:29 +0000 Received: from localhost ([127.0.0.1]:35661 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVKlU-0000Qs-Nk for submit@debbugs.gnu.org; Wed, 12 Jul 2017 12:45:28 -0400 Received: from mail-it0-f65.google.com ([209.85.214.65]:34703) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVKlT-0000Qg-GC for 24402@debbugs.gnu.org; Wed, 12 Jul 2017 12:45:27 -0400 Received: by mail-it0-f65.google.com with SMTP id o202so2721145itc.1 for <24402@debbugs.gnu.org>; Wed, 12 Jul 2017 09:45:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=bfBhk6eSEx1mqGzrDrKTjEnkvnS2W6n+VqEiTF5vWks=; b=FHeVcwZHew6ozSLkxhqi1ywZMQKEIWV5YXrXnQRu1w2KrqVlsUId0/2UQAxui+FCH/ i02sBWRwDAK72BFByHnZ5kO29YwPNcEoF3gQprVTgiaR5kc5j6m/G9IJ9qB/8dMLS9d0 pabb4aOQ2bfbQiVnE4/IhSWZ/2+V8xEBhnLEBGty2X7kluQhaHTMwdeEgg9IyPNx+95l yo5XYIyVnkePk6OsMmSXyCJbECq6Ur5CU4hTbYI+fPOyJt0AmakTxwirgHFbKowd8czT 8vzNqhEGHhmjEtizKinNZhYHQ3p4UGIl8YaoOBDPnrUrUj8ZlDfMqvDITBS+0R76w4vA +OLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=bfBhk6eSEx1mqGzrDrKTjEnkvnS2W6n+VqEiTF5vWks=; b=UL4Tg58nTECqCN6H4Y77kQKXMgvUQ30bZHuz5ydFngL2Hxz9PD1D2BhAiPEVV7QOMc g+v84w6lN//dlasB1d13UN3d1KzG3vLQuBEU4tuSJWxDwudeBKabgEgtCQ3P9/xLV8X2 07+CanKPJbD/GgX0UTt3vzqAJX5LEdy3zJ02axSBWDsW+VyopfGCNw1xhu99osV6Kg7J kiaggCUKVRA2Rm7bNq89r4d1kkWVJigWaT0AzqvEh7pY7ysCG1JvJllyg6KOWejAAJ9h C8d699bUauzO3HAyj3agmlUko+PlJUH/Vs3Ja0gLkFl2ibn6zF1wdC8dOyoc0GfdSKg/ yvLA== X-Gm-Message-State: AIVw112VRdn6ewdJdh29qdPdkSPJqUV39rbJn20yLNykMIo/qD/iiiNh v2MoDpjBhDYk4w== X-Received: by 10.107.51.66 with SMTP id z63mr6828542ioz.44.1499877921649; Wed, 12 Jul 2017 09:45:21 -0700 (PDT) Received: from lylat (S010664777d9cebe3.ss.shawcable.net. [70.64.85.59]) by smtp.gmail.com with ESMTPSA id m15sm1461130iti.9.2017.07.12.09.45.19 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 12 Jul 2017 09:45:20 -0700 (PDT) From: Alex To: npostavs@users.sourceforge.net Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> Date: Wed, 12 Jul 2017 10:45:14 -0600 In-Reply-To: <87tw2het1b.fsf@users.sourceforge.net> (npostavs's message of "Wed, 12 Jul 2017 08:30:56 -0400") Message-ID: <874luhbo4l.fsf@lylat> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , Tino Calancha , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.5 (/) npostavs@users.sourceforge.net writes: > Alex writes: > >> npostavs@users.sourceforge.net writes: >> >>> Yes, ert binds `debugger' in order to get full backtrace information >>> when there is an error. This means it won't see errors caught by >>> condition-case. That's good when it ignores errors caught by test code >>> using condition-case, but does give rise to problems. There is some >>> relevant discussion in Bugs #11218 and #24617. >>> >>> Espcially the suggestion in #24617 of using `signal-hook-function' to >>> record error info instead of using `debugger', I think doing this could >>> simplify things a lot. It is definitely going to require messing around >>> with ert's internals though... >> >> Thanks for the info. I may have discovered a workaround, but I'm not >> sure if there's any negative side-effects. All the tests pass, though. >> >> What do you think of it? It's obviously not ideal, but I think it at >> least fixes the issues at hand. > > Does it also work when loading the elc version of the test file? (try > 'make check TEST_LOAD_EL=no') Oh, it doesn't load the elc version by default? That's surprising; I think that should be documented in the test README. I get 3 test failures with TEST_LOAD_EL=no, but I don't believe they're because of me. On a mostly clean master (d014a5e15) those 3 also error. One of them is simple to fix (the (require 'subr-x) should not be inside eval-when-compile in dom-tests). The other failing tests are subr-test-backtrace-integration-test and cl-lib-defstruct-record. > What about tests like this? > > (ert-deftest check-error-handling () > (should > (eq 42 > (condition-case () > (/ 1 0) > (arith-error 42))))) It works for me, yes. As long as `debugger' is set to a symbol. I can make it a bit more robust by using an additional defvar in ert--run-test-internal. Are you asking because it doesn't work for you? From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 12 21:29:56 2017 Received: (at 24402) by debbugs.gnu.org; 13 Jul 2017 01:29:57 +0000 Received: from localhost ([127.0.0.1]:35992 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVSx2-0004Xk-Lb for submit@debbugs.gnu.org; Wed, 12 Jul 2017 21:29:56 -0400 Received: from mail-it0-f53.google.com ([209.85.214.53]:38143) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVSx0-0004XV-AX for 24402@debbugs.gnu.org; Wed, 12 Jul 2017 21:29:54 -0400 Received: by mail-it0-f53.google.com with SMTP id k192so29792733ith.1 for <24402@debbugs.gnu.org>; Wed, 12 Jul 2017 18:29:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=sxQ1SktwsJ3F1AMgCcUE/9uQBYYWhlLZkmOUHXXDz7Q=; b=aHnOtmG4gmCTUP/8UCNi/qxzlXg8DK133YU9P9q90txKKRFrB+aWTIZbTJiP6YKoTw Z3Qu+wqBFI86M1m8ji7oz5dpaD8SqPDPJ+keqXMzGJPeTM2hQ5YCpx6Oa348X/qn5D47 iTQlfVqXDshlta2n5mcubD0ABwF7s5/UKsNoVDej9GmoSVbBeJFiHDWnkq9o55MjGFyI +tRksGzheX0o1C+jOd6dKxRI0iY/SON5qy6LwBGzDLI2jYpDDjvhiJpRH1RJEI1QlvQW emswCfK+5sW43MPwY5B7XRP/UBio54mYxhwPOcaacWncjGxoxT4icpqBFlGNEuylRcXi D0Ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:references:date :in-reply-to:message-id:user-agent:mime-version; bh=sxQ1SktwsJ3F1AMgCcUE/9uQBYYWhlLZkmOUHXXDz7Q=; b=bXr8Ce2m6eujFSZeWjERPEYYGc5keTPwpa6qjZJnuyxDsn/WVWL1+zSPUetc6eZaO1 +Ao1XyWUAKUfcs+dwx6Bg9jjDgIHkT6YURhAFz5rKhwL5jSV1EhxsLHL+JTW5azPkea+ PzwSIA58Pf5OvggKN2+2EjIM6HO24/eaGwdExdN8THn/UwZarT/tWdZVmu80o4dxpprB yUnuUnWOoP7Mn39aJlrVd3JkAsuiz18fVWKgB3HfvSU7B506v6qNhMGvlqx8e2Dg+aDy 3Op8SSYF6IVFwqb5Toh0Bm3OHeZYjl4hw3fMDvJAagJQKlR7s6ILkBubLGRMUdy7TXmB pWug== X-Gm-Message-State: AIVw113ZzWh1x35tz0N4v+KqPcqh10KcKZkh0bCY5cSSBq8EgFrDBVQX 2MvKqRkOGnB0+w== X-Received: by 10.36.71.79 with SMTP id t76mr11775943itb.118.1499909388629; Wed, 12 Jul 2017 18:29:48 -0700 (PDT) Received: from zony ([45.2.7.65]) by smtp.googlemail.com with ESMTPSA id 66sm2400534ioj.47.2017.07.12.18.29.46 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 12 Jul 2017 18:29:46 -0700 (PDT) From: npostavs@users.sourceforge.net To: Alex Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> <874luhbo4l.fsf@lylat> Date: Wed, 12 Jul 2017 21:31:19 -0400 In-Reply-To: <874luhbo4l.fsf@lylat> (Alex's message of "Wed, 12 Jul 2017 10:45:14 -0600") Message-ID: <87lgntdswo.fsf@users.sourceforge.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , Tino Calancha , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.0 (/) Alex writes: > npostavs@users.sourceforge.net writes: > >> Does it also work when loading the elc version of the test file? (try >> 'make check TEST_LOAD_EL=no') > > Oh, it doesn't load the elc version by default? That's surprising; I > think that should be documented in the test README. > > I get 3 test failures with TEST_LOAD_EL=no, but I don't believe they're > because of me. On a mostly clean master (d014a5e15) those 3 also error. > One of them is simple to fix (the (require 'subr-x) should not be inside > eval-when-compile in dom-tests). Ah, the `should' blocks inlining during compilation. Is that necessary? Probably yes if we expect to catch errors during macroexpansion I guess. > The other failing tests are > subr-test-backtrace-integration-test and cl-lib-defstruct-record. Hmm, I'll see if I can fix these. >> What about tests like this? >> >> (ert-deftest check-error-handling () >> (should >> (eq 42 >> (condition-case () >> (/ 1 0) >> (arith-error 42))))) > > It works for me, yes. As long as `debugger' is set to a symbol. I can > make it a bit more robust by using an additional defvar in > ert--run-test-internal. > > Are you asking because it doesn't work for you? No, I'm just trying to explore the edges of this solution. Isn't `debugger' bound to a non-symbol while running the the tests? I'm confused as to why this solution works. From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 12 23:04:56 2017 Received: (at 24402) by debbugs.gnu.org; 13 Jul 2017 03:04:56 +0000 Received: from localhost ([127.0.0.1]:36048 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVUQx-00009Z-MF for submit@debbugs.gnu.org; Wed, 12 Jul 2017 23:04:55 -0400 Received: from mail-it0-f66.google.com ([209.85.214.66]:32960) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVUQt-00009K-KU for 24402@debbugs.gnu.org; Wed, 12 Jul 2017 23:04:52 -0400 Received: by mail-it0-f66.google.com with SMTP id 188so5164559itx.0 for <24402@debbugs.gnu.org>; Wed, 12 Jul 2017 20:04:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=ys3HxOf0wNXEFIG/xW823tdo15GFlua9/ISfGQMykEk=; b=jmy/D/CkkoRnFhB/53Be+z9GqJ7c7iKVhGlCPCQMn6VplemJABykY5C5/vEGUkIuaY EajE2p7d7ar86vaoqY2rWQ2k7lVDqzKgaWMdIqEXtbxnmOHhugKdKoX7Lbz86bCtfHHB vydAWiLvAdVZu2Or2OjJy3a+YzxakhVfGhARSb4poAZfNmg0QjUrftwq+RY1bfd2pQIJ L35EMk9vrmv48ttI5EOxeUMOTgDtoR1k0WXvHzLcvsmu/dgybLBFlxyHeLLVXhc22/fc C0Migo0Pp0/ZGRS2ZlXaI4P1ciAS29Pd70LYA0z6wPO3/LZmYb5Q9bm9PUC4vlDscejq kX3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=ys3HxOf0wNXEFIG/xW823tdo15GFlua9/ISfGQMykEk=; b=lnEiDJ1PvFikGREL08GjAsj45nwFGBRn8bk9yRFv+i7Ra2ZfHSq0DmYVSYeIrIoY6A vOyj9BXo0uNNOUANVIJD8RgON3r0vTVd80GP7eUzJD0AtXIk5YRp9oUPKtcZkJHOZ8bK D5VfI9EJkIXsCmJRXDmoXSAZAydgS8LOlfxu9o1fY9LzMQ3NffFOI3gd5vr6T796j+uR nwahjwtDuiQhqtd0ctob3bi+jdx+on5FHkhWuS6/1safMUzJyb+0t2YFbsX+7cEGyT5u +yw3fbMiA4KkZj7cXIHQRkLhAlmrkZRj5nsINReQFs7mO4pF+yO/0FZYc84mM2PBOM+0 LcQg== X-Gm-Message-State: AIVw112JGIOo3yTLbyCa+JxmgUuFxYfGRy/oPxr9qI0Xau4xOLc9XYRq q6AF27gh2T2MjQ== X-Received: by 10.36.123.203 with SMTP id q194mr12641574itc.19.1499915086090; Wed, 12 Jul 2017 20:04:46 -0700 (PDT) Received: from lylat (S010664777d9cebe3.ss.shawcable.net. [70.64.85.59]) by smtp.gmail.com with ESMTPSA id q62sm2406855iof.57.2017.07.12.20.04.43 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 12 Jul 2017 20:04:44 -0700 (PDT) From: Alex To: npostavs@users.sourceforge.net Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> <874luhbo4l.fsf@lylat> <87lgntdswo.fsf@users.sourceforge.net> Date: Wed, 12 Jul 2017 21:04:38 -0600 In-Reply-To: <87lgntdswo.fsf@users.sourceforge.net> (npostavs's message of "Wed, 12 Jul 2017 21:31:19 -0400") Message-ID: <87k23d9gvt.fsf@lylat> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , Tino Calancha , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.5 (/) --=-=-= Content-Type: text/plain npostavs@users.sourceforge.net writes: > Alex writes: > >> npostavs@users.sourceforge.net writes: >> >>> Does it also work when loading the elc version of the test file? (try >>> 'make check TEST_LOAD_EL=no') >> >> Oh, it doesn't load the elc version by default? That's surprising; I >> think that should be documented in the test README. >> >> I get 3 test failures with TEST_LOAD_EL=no, but I don't believe they're >> because of me. On a mostly clean master (d014a5e15) those 3 also error. >> One of them is simple to fix (the (require 'subr-x) should not be inside >> eval-when-compile in dom-tests). > > Ah, the `should' blocks inlining during compilation. Is that necessary? > Probably yes if we expect to catch errors during macroexpansion I guess. Can you get errors by expanding inlined functions? Macros are expanded at compile-time with the current patch. If there are any macroexpansion errors, then the form is altered to be (error ). Perhaps inline functions could work similarly. Here's a diff to my patch that uses byte-compile-inline-expand. This fixes the dom-tests case. Do you think it's worth it? --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=ert-inline.diff Content-Description: inline diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el index beb32c48ce..f4d2f725ac 100644 --- a/lisp/emacs-lisp/ert.el +++ b/lisp/emacs-lisp/ert.el @@ -281,17 +281,19 @@ ert--special-operator-p (and (subrp definition) (eql (cdr (subr-arity definition)) 'unevalled))))) +(require 'byte-opt) + (defun ert--expand-should-1 (whole form inner-expander) "Helper function for the `should' macro and its variants." (let ((form (condition-case err - (macroexpand-all form (append (bound-and-true-p - byte-compile-macro-environment) - (cond - ((boundp 'macroexpand-all-environment) - macroexpand-all-environment) - ((boundp 'cl-macro-environment) - cl-macro-environment)))) + (byte-compile-inline-expand (macroexpand-all form (append (bound-and-true-p + byte-compile-macro-environment) + (cond + ((boundp 'macroexpand-all-environment) + macroexpand-all-environment) + ((boundp 'cl-macro-environment) + cl-macro-environment))))) (error `(signal ',(car err) ',(cdr err)))))) (cond ((or (atom form) (ert--special-operator-p (car form))) @@ -308,7 +310,8 @@ ert--expand-should-1 (cl-assert (or (symbolp fn-name) (and (consp fn-name) (eql (car fn-name) 'lambda) - (listp (cdr fn-name))))) + (listp (cdr fn-name))) + (byte-code-function-p fn-name))) (let ((fn (cl-gensym "fn-")) (args (cl-gensym "args-")) (value (cl-gensym "value-")) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable >> The other failing tests are >> subr-test-backtrace-integration-test and cl-lib-defstruct-record. > > Hmm, I'll see if I can fix these. Thanks. I noticed when byte-compiling cl-lib-tests, I got this warning: Unused lexical variable =E2=80=98cl-struct-foo=E2=80=99. >>> What about tests like this? >>> >>> (ert-deftest check-error-handling () >>> (should >>> (eq 42 >>> (condition-case () >>> (/ 1 0) >>> (arith-error 42))))) >> >> It works for me, yes. As long as `debugger' is set to a symbol. I can >> make it a bit more robust by using an additional defvar in >> ert--run-test-internal. >> >> Are you asking because it doesn't work for you? > > No, I'm just trying to explore the edges of this solution. Isn't > `debugger' bound to a non-symbol while running the the tests? I'm > confused as to why this solution works. Yes, that's why there's the second test that checks for error-symbol to be ert-test-{failed, skipped}. Basically what's happening is that ert--signal-hook forces the debugger to trigger even inside a condition-case, but only with a non-symbol `debugger' (since ert--run-test-internal binds it to a closure), and one of the above two errors. The only time this approach fails is when you bind `debugger' to a non-symbol and also signal ert-test-{failed, skipped}. This is relatively rare compared to the problems at hand (macro and argument errors), so unless there are unforeseen issues I think it's an acceptable stop-gap solution. Hopefully Someone=E2=84=A2 can properly fix t= his eventually. --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 12 23:42:55 2017 Received: (at 24402) by debbugs.gnu.org; 13 Jul 2017 03:42:55 +0000 Received: from localhost ([127.0.0.1]:36064 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVV1j-00016U-6n for submit@debbugs.gnu.org; Wed, 12 Jul 2017 23:42:55 -0400 Received: from mail-it0-f46.google.com ([209.85.214.46]:36081) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVV1h-00016I-26 for 24402@debbugs.gnu.org; Wed, 12 Jul 2017 23:42:53 -0400 Received: by mail-it0-f46.google.com with SMTP id m68so27935648ith.1 for <24402@debbugs.gnu.org>; Wed, 12 Jul 2017 20:42:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=l+XiAXUlRw+hHJUs2bw3MLgyC1zTphz2R4pm9d7RqzE=; b=lkwgRv4Zmu8axDIdfg/LFB4Q62vObCvQzUrIKnrlDyxy1pt0VR9CpNAbEDffkOAO+i E/8Swg6ORVIffM6zVRradfte/pUeNg73kGZzdtRA4vXMIffGa5tNDL+1MOl8II5iM50P UoMU+n6SHZmtgjqvDpNcNb2dX6f033c6xjZtDUnsafaytqabJRsD6INi01ivGhObhTpa Arpy8zAeikaKmP1PmZGPyBVrBXjLv4jqWRji3IOUZUCVJQQ+cVtGQEPEHotIXkgMMPRe 52bqKjB3XUkRIpzVezcSJNG815xcCi+t/NatmcgN1Y4+nEd8K1QvnSIxd9Is2HLrnKoO rExg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:references:date :in-reply-to:message-id:user-agent:mime-version :content-transfer-encoding; bh=l+XiAXUlRw+hHJUs2bw3MLgyC1zTphz2R4pm9d7RqzE=; b=e/ZwGnyyYr05YA0x5B5fhQmahKnkvYbA0yKNwg12Z9+6R+4k4Li/T+xS808ts9vFP+ JKg0ArN+18wJLsx5R8/YJNNl2D6KcM+fEuY3p2xU3T06hEaAu+We3Tfh+ZFP2j4ITTAb nK1KgnTK3VXmBfj8HHn1AelB3WLYd1ciRydj0ygzzdb+eqqye1OdtI0hh5HaHWN5Xm5j LYdo95dS7/aYS6r+lRVbxvr2UeKRDYoe/amJ04CYvxEmopIaxYvKHBvzyHtFW49iym4E LYtx55a8UhUi13LgEE0i07V6lgBIqLsfCBFM+MIl3r9Cot3XstpY0xjVTeePXAxgM2yG DgQw== X-Gm-Message-State: AIVw110yuwJHAx9PZyqCZHoTrTDLV8fA70XkHJXZTFR+Lg4jRZ9U5h2b x7Ga1JEOcF605A== X-Received: by 10.107.173.15 with SMTP id w15mr1537795ioe.99.1499917367412; Wed, 12 Jul 2017 20:42:47 -0700 (PDT) Received: from zony ([45.2.7.65]) by smtp.googlemail.com with ESMTPSA id v65sm2415961iod.28.2017.07.12.20.42.45 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 12 Jul 2017 20:42:46 -0700 (PDT) From: npostavs@users.sourceforge.net To: Alex Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> <874luhbo4l.fsf@lylat> <87lgntdswo.fsf@users.sourceforge.net> <87k23d9gvt.fsf@lylat> Date: Wed, 12 Jul 2017 23:44:19 -0400 In-Reply-To: <87k23d9gvt.fsf@lylat> (Alex's message of "Wed, 12 Jul 2017 21:04:38 -0600") Message-ID: <87d195dmr0.fsf@users.sourceforge.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.7 (/) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , Tino Calancha , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.7 (/) Alex writes: >>> I get 3 test failures with TEST_LOAD_EL=3Dno, but I don't believe they'= re >>> because of me. On a mostly clean master (d014a5e15) those 3 also error. >>> One of them is simple to fix (the (require 'subr-x) should not be inside >>> eval-when-compile in dom-tests). >> >> Ah, the `should' blocks inlining during compilation. Is that necessary? >> Probably yes if we expect to catch errors during macroexpansion I guess. > > Can you get errors by expanding inlined functions? Yes I think so, probably not from defsubst (except for wrong number of arguments maybe?), but I believe define-inline basically lets you run an arbitrary macro to produce the inlined call. > Macros are expanded at compile-time with the current patch. If there are > any macroexpansion errors, then the form is altered to be (error > ). Perhaps inline functions could work similarly. > > Here's a diff to my patch that uses byte-compile-inline-expand. This > fixes the dom-tests case. Do you think it's worth it? It would be nice if we can make code inside tests behave the same as outside. But we should make it conditional on whether the code is being compiled, otherwise code inside tests would behave differently when being interpreted. Anyway, we can leave this for a separate bug. > Yes, that's why there's the second test that checks for error-symbol to > be ert-test-{failed, skipped}. Basically what's happening is that > ert--signal-hook forces the debugger to trigger even inside a > condition-case, but only with a non-symbol `debugger' (since > ert--run-test-internal binds it to a closure), and one of the above two > errors. > > The only time this approach fails is when you bind `debugger' to a > non-symbol and also signal ert-test-{failed, skipped}. Okay, it sounds like we would only hit problems when running an ert test from inside an ert test. That should be pretty rare outside of ert's test suite, and we already have workarounds for that case, right? > This is relatively rare compared to the problems at hand (macro and > argument errors), so unless there are unforeseen issues I think it's an > acceptable stop-gap solution. Hopefully Someone=E2=84=A2 can properly fix= this > eventually. Yes, I think this approach is acceptable. From debbugs-submit-bounces@debbugs.gnu.org Thu Jul 13 18:45:56 2017 Received: (at 24402) by debbugs.gnu.org; 13 Jul 2017 22:45:56 +0000 Received: from localhost ([127.0.0.1]:37508 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVmrr-0006xv-PT for submit@debbugs.gnu.org; Thu, 13 Jul 2017 18:45:56 -0400 Received: from mail-it0-f65.google.com ([209.85.214.65]:36480) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVmrq-0006xj-LI for 24402@debbugs.gnu.org; Thu, 13 Jul 2017 18:45:55 -0400 Received: by mail-it0-f65.google.com with SMTP id k3so9730853ita.3 for <24402@debbugs.gnu.org>; Thu, 13 Jul 2017 15:45:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=/qLlY7vc/PYLZa7Vyp+7IYAgg0FwXWrTWTPNDhJZ2Is=; b=tgLfsoTqPAFc8DAm/hHLrsdbBFr32VLguXOmNZBm6HYO0/f9fOefarCjeOcL65NJo4 xExZ+CafpOBkiSpfGYcmeCUeYUWY4orGieJv7E6AlRFbCRWVcFP1ap5uULZjyE3JlAJa IPhf2ZhKa5C95IaHtYwRa4GzPlLjqKTZZ4iSGm7K1DESL1/L5qDKfAF3BkXBw+pwIxUi NH98CoR91gmqYvNrwNg3jHYF/+0ZLwltmFiQyU66gLjXf78m6kIYKJ5L8AaeZqAEJd+j tNDQennW/vjoqcCO+TF18+Bl/kswGALBeMsY9x7c3hHoTzcO3e9KjUzEBuO1NNt9OY3i 9XPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=/qLlY7vc/PYLZa7Vyp+7IYAgg0FwXWrTWTPNDhJZ2Is=; b=scQa4TMpLMBjRSaBQEQzts+byL+h8Z+jRH2Ce6j/VX5cTJI6NSGjdTAngh9vZA1yJA ZhS8/TFr24CAGnc7zyYwiEJnVuGzst6GGOOEaROB5NDNsKQsTAgUzLA1FzV/INRTHDe4 6Ug9vxVXINn5dDUPvotb7Z+nN+axeylmMPio5emDeAGjg31MJ3EkYdg5y33tblapyUoY W5XrNQhAQPmYM/A1WlHyTzJcIrMEvB7sF7Ov6T3dbExj+I9B7UTVaPuq9nCkCk7i55zi 96vcI6VYticRlDadws3zRIeQiW0eC4Y7j0N3uYbYRXYNa78lQdJPwDMDBQoeSU04G9j9 g4gA== X-Gm-Message-State: AIVw113A/TqHKKBCUb5RIqQFnQR2tl+MXDgJ/ugkv9UgmUO3brB8Jnjn A6b06YtRxOCS5Q== X-Received: by 10.107.138.148 with SMTP id c20mr6166964ioj.135.1499985948732; Thu, 13 Jul 2017 15:45:48 -0700 (PDT) Received: from lylat (S010664777d9cebe3.ss.shawcable.net. [70.64.85.59]) by smtp.gmail.com with ESMTPSA id k16sm374724itb.1.2017.07.13.15.45.46 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2017 15:45:47 -0700 (PDT) From: Alex To: npostavs@users.sourceforge.net Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> <874luhbo4l.fsf@lylat> <87lgntdswo.fsf@users.sourceforge.net> <87k23d9gvt.fsf@lylat> <87d195dmr0.fsf@users.sourceforge.net> Date: Thu, 13 Jul 2017 16:45:39 -0600 In-Reply-To: <87d195dmr0.fsf@users.sourceforge.net> (npostavs's message of "Wed, 12 Jul 2017 23:44:19 -0400") Message-ID: <87lgns7y7g.fsf@lylat> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , Tino Calancha , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.5 (/) --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable npostavs@users.sourceforge.net writes: > Alex writes: > >> Macros are expanded at compile-time with the current patch. If there are >> any macroexpansion errors, then the form is altered to be (error >> ). Perhaps inline functions could work similarly. >> >> Here's a diff to my patch that uses byte-compile-inline-expand. This >> fixes the dom-tests case. Do you think it's worth it? > > It would be nice if we can make code inside tests behave the same as > outside. But we should make it conditional on whether the code is being > compiled, otherwise code inside tests would behave differently when > being interpreted. Anyway, we can leave this for a separate bug. I agree, but that sounds like it'll require a fair bit of refactoring and knowledge of ert internals. OOC, is there a robust way to check whether or not you're currently byte-compiling? >> Yes, that's why there's the second test that checks for error-symbol to >> be ert-test-{failed, skipped}. Basically what's happening is that >> ert--signal-hook forces the debugger to trigger even inside a >> condition-case, but only with a non-symbol `debugger' (since >> ert--run-test-internal binds it to a closure), and one of the above two >> errors. >> >> The only time this approach fails is when you bind `debugger' to a >> non-symbol and also signal ert-test-{failed, skipped}. > > Okay, it sounds like we would only hit problems when running an ert test > from inside an ert test. That should be pretty rare outside of ert's > test suite, and we already have workarounds for that case, right? I tried a nested case using two ert-deftests and it worked, so it looks okay here too. >> This is relatively rare compared to the problems at hand (macro and >> argument errors), so unless there are unforeseen issues I think it's an >> acceptable stop-gap solution. Hopefully Someone=E2=84=A2 can properly fi= x this >> eventually. > > Yes, I think this approach is acceptable. I was going to ask if you would merge in a few days, but it appears that what should have been a simple rebase to master caused unforeseen consequences. For instance, for some reason I now get a segmentation fault when executing 'make cl-lib-tests TEST_LOAD_EL=3Dno'. I even reset to the commit I was at before and it still segfaults. Can you reproduce this with the following patch on master? --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=0001-Catch-argument-and-expansion-errors-in-ert.patch Content-Description: ert >From fdbf66f73672d9b69fc37273223ae90fff951eb2 Mon Sep 17 00:00:00 2001 From: Alexander Gramiak Date: Thu, 13 Jul 2017 14:54:35 -0600 Subject: [PATCH] Catch argument and expansion errors in ert This kludge catches errors caused by evaluating arguments in ert's should, should-not, and should-error macros; it also catches macro and inline function expansion errors inside of the above macros (Bug#24402). * lisp/emacs-lisp/ert.el: (ert--should-signal-hook): New function. (ert--expand-should-1): Catch expansion errors. * test/lisp/emacs-lisp/ert-tests.el (ert-test-should-error-argument) (ert-test-should-error-macroexpansion): Tests for argument and expansion errors. --- lisp/emacs-lisp/ert.el | 47 ++++++++++++++++++++++++++++++--------- test/lisp/emacs-lisp/ert-tests.el | 9 ++++++++ 2 files changed, 46 insertions(+), 10 deletions(-) diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el index eb2b2e3e11..c6e3ee8503 100644 --- a/lisp/emacs-lisp/ert.el +++ b/lisp/emacs-lisp/ert.el @@ -65,6 +65,7 @@ (require 'find-func) (require 'help) (require 'pp) +(require 'byte-opt) ; for ert--expand-should-1 kludge ;;; UI customization options. @@ -266,6 +267,14 @@ ert--signal-should-execution (when ert--should-execution-observer (funcall ert--should-execution-observer form-description))) +;; See Bug#24402 for why this exists +(defun ert--should-signal-hook (error-symbol data) + "Stupid hack to stop `condition-case' from catching ert signals. +It should only be stopped when ran from inside ert--run-test-internal." + (when (and (not (symbolp debugger)) ; only run on anonymous debugger + (memq error-symbol '(ert-test-failed ert-test-skipped))) + (funcall debugger 'error data))) + (defun ert--special-operator-p (thing) "Return non-nil if THING is a symbol naming a special operator." (and (symbolp thing) @@ -276,13 +285,20 @@ ert--special-operator-p (defun ert--expand-should-1 (whole form inner-expander) "Helper function for the `should' macro and its variants." (let ((form - (macroexpand form (append (bound-and-true-p - byte-compile-macro-environment) - (cond - ((boundp 'macroexpand-all-environment) - macroexpand-all-environment) - ((boundp 'cl-macro-environment) - cl-macro-environment)))))) + ;; catch all macro and inline function expansion errors + ;; FIXME: Code inside of tests should be evaluated like it is + ;; outside of tests, with the sole exception of error + ;; handling + (condition-case err + (byte-compile-inline-expand + (macroexpand-all form (append (bound-and-true-p + byte-compile-macro-environment) + (cond + ((boundp 'macroexpand-all-environment) + macroexpand-all-environment) + ((boundp 'cl-macro-environment) + cl-macro-environment))))) + (error `(signal ',(car err) ',(cdr err)))))) (cond ((or (atom form) (ert--special-operator-p (car form))) (let ((value (cl-gensym "value-"))) @@ -298,13 +314,20 @@ ert--expand-should-1 (cl-assert (or (symbolp fn-name) (and (consp fn-name) (eql (car fn-name) 'lambda) - (listp (cdr fn-name))))) + (listp (cdr fn-name))) + ;; for inline functions + (byte-code-function-p fn-name))) (let ((fn (cl-gensym "fn-")) (args (cl-gensym "args-")) (value (cl-gensym "value-")) (default-value (cl-gensym "ert-form-evaluation-aborted-"))) - `(let ((,fn (function ,fn-name)) - (,args (list ,@arg-forms))) + `(let* ((,fn (function ,fn-name)) + (,args (condition-case err + (let ((signal-hook-function #'ert--should-signal-hook)) + (list ,@arg-forms)) + (error (progn (setq ,fn #'signal) + (list (car err) + (cdr err))))))) (let ((,value ',default-value)) ,(funcall inner-expander `(setq ,value (apply ,fn ,args)) @@ -766,6 +789,10 @@ ert--run-test-internal ;; too expensive, we can remove it. (with-temp-buffer (save-window-excursion + ;; FIXME: Use `signal-hook-function' instead of `debugger' to + ;; handle ert errors. Once that's done, remove + ;; `ert--should-signal-hook'. See Bug#24402 and Bug#11218 for + ;; details. (let ((debugger (lambda (&rest args) (ert--run-test-debugger test-execution-info args))) diff --git a/test/lisp/emacs-lisp/ert-tests.el b/test/lisp/emacs-lisp/ert-tests.el index 317838b250..b6a7eb68da 100644 --- a/test/lisp/emacs-lisp/ert-tests.el +++ b/test/lisp/emacs-lisp/ert-tests.el @@ -294,6 +294,15 @@ ert-self-test-and-exit "the error signaled was a subtype of the expected type"))))) )) +(ert-deftest ert-test-should-error-argument () + "Errors due to evaluating arguments should not break tests." + (should-error (identity (/ 1 0)))) + +(ert-deftest ert-test-should-error-macroexpansion () + "Errors due to expanding macros should not break tests." + (cl-macrolet ((test () (error "Foo"))) + (should-error (test)))) + (ert-deftest ert-test-skip-unless () ;; Don't skip. (let ((test (make-ert-test :body (lambda () (skip-unless t))))) -- 2.13.2 --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Thu Jul 13 19:48:03 2017 Received: (at 24402) by debbugs.gnu.org; 13 Jul 2017 23:48:03 +0000 Received: from localhost ([127.0.0.1]:37524 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVnpz-00006R-3A for submit@debbugs.gnu.org; Thu, 13 Jul 2017 19:48:03 -0400 Received: from mail-it0-f52.google.com ([209.85.214.52]:36874) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVnpx-00005v-Pj for 24402@debbugs.gnu.org; Thu, 13 Jul 2017 19:48:02 -0400 Received: by mail-it0-f52.google.com with SMTP id m84so8849601ita.0 for <24402@debbugs.gnu.org>; Thu, 13 Jul 2017 16:48:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=eYyocEUldQ5I8FbSc9ULQub03fdi85jiZ0kIPnqrXcs=; b=IS2/UjL+6tTq9aaPiKoxQIp48NryH7WNaF0YXSpJZrs6QWlgAOnDzPXmb9N7OiANNL DbT7rhU6AnZGbA9u6sra7+z33xurFAJV3z1hx6T+rwm5jfEsYlXt219XDp0YoWD5qYe9 yijnVpzWHHteed89e879Zzv20+C82d4g3IpmWPoz93exU+CGqSzIJ7Lc4sZgDAFEkcKX cY+PKpX96UNyveVYL6IdeBUCJKwRbA+DoiYKswZwGUMFFpFFg9AHz//vpFYUs6aSNBDQ 8yLpJvszTpD1D9rGYSkqyJeRZCoI7Xf8xK7DV0IpXU/hGXoHnffob7bRpNCwROMkRnq3 Lttg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:references:date :in-reply-to:message-id:user-agent:mime-version; bh=eYyocEUldQ5I8FbSc9ULQub03fdi85jiZ0kIPnqrXcs=; b=qR7fArGbc+YIF9CbCHjhfEldPtNTO86DppK98WuaWoJ3Rt8BTHqZR0PbhCyUaDeuoL Wz6ww0OgMg5yid2SyB6PcwdP1wEaR/KcIYb36Gd0gYSTfVJVHV3Q8DspyGVi1W0ivlEW Z3r+1+xKfnnedMDwMXD+NRmrX2FTHLngBxGxRKIZ2LprUaW2KTfh0S/rRGmQ2oEIvHt8 DNLvkoJBlLetXcKTKHNwNiq9VfozxiVRc5RAyAx1sBC46NND2HVrtpNOtA20rAb+vphh gXs0L7/G/q6VuQ/M+5QlbkCaxnOM/X5l5e4fm+71XHfzhmYqZ24TBa2CvkL5H8uTV8UP z6ZA== X-Gm-Message-State: AIVw112ZUmClL+aT2yGnswW4ElxB9GgQGggNXyVzcdk93sEmJIb2GmHp TPdR1/rdOJi2mg== X-Received: by 10.107.152.85 with SMTP id a82mr5947710ioe.105.1499989675083; Thu, 13 Jul 2017 16:47:55 -0700 (PDT) Received: from zony ([45.2.7.65]) by smtp.googlemail.com with ESMTPSA id s135sm1802298ita.2.2017.07.13.16.47.52 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2017 16:47:53 -0700 (PDT) From: npostavs@users.sourceforge.net To: Alex Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> <874luhbo4l.fsf@lylat> <87lgntdswo.fsf@users.sourceforge.net> <87k23d9gvt.fsf@lylat> <87d195dmr0.fsf@users.sourceforge.net> <87lgns7y7g.fsf@lylat> Date: Thu, 13 Jul 2017 19:49:26 -0400 In-Reply-To: <87lgns7y7g.fsf@lylat> (Alex's message of "Thu, 13 Jul 2017 16:45:39 -0600") Message-ID: <877ezbew3d.fsf@users.sourceforge.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.7 (/) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , Tino Calancha , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.7 (/) Alex writes: > npostavs@users.sourceforge.net writes: >> It would be nice if we can make code inside tests behave the same as >> outside. But we should make it conditional on whether the code is being >> compiled, otherwise code inside tests would behave differently when >> being interpreted. Anyway, we can leave this for a separate bug. > > I agree, but that sounds like it'll require a fair bit of refactoring > and knowledge of ert internals. I don't think so, just a conditional to decide whether or not to call the extra expansion. Do you think there is anything else? > OOC, is there a robust way to check whether or not you're currently > byte-compiling? AFAIK, the usual trick is (bound-and-true-p byte-compile-current-file). It's probably good enough for most things. > I was going to ask if you would merge in a few days, but it appears that > what should have been a simple rebase to master caused unforeseen > consequences. For instance, for some reason I now get a segmentation > fault when executing 'make cl-lib-tests TEST_LOAD_EL=no'. I even reset > to the commit I was at before and it still segfaults. Can you reproduce > this with the following patch on master? Nope, I just get the failures on cl-lib-defstruct-record we already mentioned. From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 14 00:42:27 2017 Received: (at 24402) by debbugs.gnu.org; 14 Jul 2017 04:42:27 +0000 Received: from localhost ([127.0.0.1]:37600 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVsQt-00071K-JS for submit@debbugs.gnu.org; Fri, 14 Jul 2017 00:42:27 -0400 Received: from mail-it0-f66.google.com ([209.85.214.66]:34528) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVsQs-000715-DJ for 24402@debbugs.gnu.org; Fri, 14 Jul 2017 00:42:26 -0400 Received: by mail-it0-f66.google.com with SMTP id o202so10785312itc.1 for <24402@debbugs.gnu.org>; Thu, 13 Jul 2017 21:42:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=tbKbpU4GH0bBQr9LVi8tI9dQrPTtACr4cTVUi5/pEtY=; b=k0t4v8KqWcmI0oHXJf1OIGivHbcWWuttO4YbV15N8tZRfkMNufmowv4Xut+S8d0k5D w3UQN44UU3jawQPwmkCpyLZA8KCo7DD213L2u3X8HsHGULcVAIJ7EPu6FQBZEbCvVlBG g7IGGV5s5DDICSc7MHuFPd8CTNJC7nBpjuUT7fVk2Bedg3ztWFCMSSuQR9shBX3rb07/ 76a0S9Ehou9psuNVir7+7WWC9EcSveZki4A8G0pxtSHrqqkaAqrEO3yQ0zdeg5c3Y96e TInVnw/s2sxbGZ7YaBfHU4BJW8gVd1JBftE86HN2QHEc3hvZ3KZDvsBhdgr3l/X0ifH4 /TJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=tbKbpU4GH0bBQr9LVi8tI9dQrPTtACr4cTVUi5/pEtY=; b=BAm9m3IbvCjpAEg1iG0aVxBm37tuzOON4hUOu4aU2W2ZwykV+k2ozslF1L8Njlnmqo /wR/nubGE7VbBzSXotA7ydIkfJB4oDKzJqNirBOEebIinJOpuo9Q60VDCuHCFjuKgvkl o0BdajYuDHImvxuzbbWUvEqr7RpkAB0Nl9mTzhiWAUKpPKl3uP876ieYNMzdLE7aRa1a CFKq7uDBdGId5j0EF/G5jOlEZP6mjwhhxCZDY4DUSL5ZuTjXCQGBPs5FpScYw6AcrCjk 7MugG2ut9RLAzJ26fTDUweRtw4vDITnVWhXkasKlMoWwdqFTpkM+QynWn+Xw99t5lWcM xpCQ== X-Gm-Message-State: AIVw1125qBUD8C9/rLKMlzB/1jI2eyBWf2PdBAttHJCo81IsNov3eTY0 qbDuQnkRC+9qdQ== X-Received: by 10.107.131.9 with SMTP id f9mr6648713iod.225.1500007340585; Thu, 13 Jul 2017 21:42:20 -0700 (PDT) Received: from lylat (S010664777d9cebe3.ss.shawcable.net. [70.64.85.59]) by smtp.gmail.com with ESMTPSA id a65sm4216734ioj.51.2017.07.13.21.42.18 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2017 21:42:19 -0700 (PDT) From: Alex To: npostavs@users.sourceforge.net Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> <874luhbo4l.fsf@lylat> <87lgntdswo.fsf@users.sourceforge.net> <87k23d9gvt.fsf@lylat> <87d195dmr0.fsf@users.sourceforge.net> <87lgns7y7g.fsf@lylat> <877ezbew3d.fsf@users.sourceforge.net> Date: Thu, 13 Jul 2017 22:42:10 -0600 In-Reply-To: <877ezbew3d.fsf@users.sourceforge.net> (npostavs's message of "Thu, 13 Jul 2017 19:49:26 -0400") Message-ID: <87d193ljdp.fsf@lylat> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , Tino Calancha , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.5 (/) npostavs@users.sourceforge.net writes: > Alex writes: > >> npostavs@users.sourceforge.net writes: >>> It would be nice if we can make code inside tests behave the same as >>> outside. But we should make it conditional on whether the code is being >>> compiled, otherwise code inside tests would behave differently when >>> being interpreted. Anyway, we can leave this for a separate bug. >> >> I agree, but that sounds like it'll require a fair bit of refactoring >> and knowledge of ert internals. > > I don't think so, just a conditional to decide whether or not to call > the extra expansion. Do you think there is anything else? I was mostly referring to not binding `debugger', but also evaluating the code "normally" (i.e., not doing expansions first in one condition-case, evaluating arguments in another, and then the whole form in a third one). >> OOC, is there a robust way to check whether or not you're currently >> byte-compiling? > > AFAIK, the usual trick is (bound-and-true-p byte-compile-current-file). > It's probably good enough for most things. I believe the below patch does that, though it has some issues. >> I was going to ask if you would merge in a few days, but it appears that >> what should have been a simple rebase to master caused unforeseen >> consequences. For instance, for some reason I now get a segmentation >> fault when executing 'make cl-lib-tests TEST_LOAD_EL=no'. I even reset >> to the commit I was at before and it still segfaults. Can you reproduce >> this with the following patch on master? > > Nope, I just get the failures on cl-lib-defstruct-record we already > mentioned. The segfault appears to have been because I didn't wipe out the elc files when testing different implementations. I spent a lot longer than I'd like to admit finding this out. Is there a reason why "make clean" in the test directory doesn't wipe out elc files? I don't understand why there's a separate bootstrap-clean that does this. Can this and TEST_LOAD_EL please be documented in the test README? Anyway, I got everything back in order. Sadly, there's a couple extra tests that now fail for me in the patch that *doesn't* expand inline functions, and these don't fail for me in a clean master. They are in eieio-tests (23 and 24). With the inline expansion, I also get some errors in ert-tests. All of the errors, with the exception of subr-tests error, seem to be from cl-defstruct and cl-typep (which is defined by define-inline). Do you have any ideas? There should be 5 unexpected errors without the inline expansion, and 6 errors with it. Note that all tests pass in both cases without "TEST_LOAD_EL=no". If it's easy to fix the eieio tests and not the other ones, then it might be better to leave the inline-function expansion out for now. From debbugs-submit-bounces@debbugs.gnu.org Fri Jul 14 00:45:26 2017 Received: (at 24402) by debbugs.gnu.org; 14 Jul 2017 04:45:26 +0000 Received: from localhost ([127.0.0.1]:37604 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVsTm-00076D-32 for submit@debbugs.gnu.org; Fri, 14 Jul 2017 00:45:26 -0400 Received: from mail-it0-f41.google.com ([209.85.214.41]:33838) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dVsTk-00075x-GC for 24402@debbugs.gnu.org; Fri, 14 Jul 2017 00:45:24 -0400 Received: by mail-it0-f41.google.com with SMTP id v202so6491364itb.1 for <24402@debbugs.gnu.org>; Thu, 13 Jul 2017 21:45:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=tbIIip2GRJxJZvwV+kXxwVnhbfJx2lB5DAXJ/VMpLMo=; b=PV4+Lz8EShgNJD5s21rsZwabzVFdklODiv9gfTr562b6L5e3vTX4Fi/JOHSw8bCFce fhUpNQqn8zFyzsWR7dVZlOsH5JQRPtM3vt+1iugApadJwoQAI/iI993Q0ueTq2iY8DdM 7zaCesnU45Ho5Z5x6cqgAHrg/OjUw2L2e1x7a06conel3Tjy7i3Bl/XRMEr6rxtURVsF ajxizHwD2XK8C+I8E1ESwSjlKg0+CGIWhG2M1FDMXmPkj6rHmPuWNVK9V/9K5OWNLxyP edW7pfv6JZSvBBSxMjwwAuOuMcfGMYwyz7xNVKUNTLoSJthuGayuoDEmxqtVeGSIK+ca f2uQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=tbIIip2GRJxJZvwV+kXxwVnhbfJx2lB5DAXJ/VMpLMo=; b=gz0ziBW4Se6WUg/cdxnIkPWKGO8Z40w9dNDwK9dxRt8BH034ZbnnF86vtguASoyDaU j/vH4hwMT1WlK7bYbg8hkOv2HgLBakhXx5iQh8gf/gRDDBIsSmf06/rvijqGhBlcLt8x 23+s2yKITn6VYZeMJFZt3eaMWTMJrJr1vctlCQJbcgzp1ANFC1GAv39muHy3fkRrbPJp 4h2BXyuYzsxa6l3wp8wQqX5mrisvdOHyvagsNHPTlZksX1v/g25SMOZa5I3SWSt/WxIv rENptEFVh+S7DyWr06ufSGDy0qVLB2n/LSXPpRE1lZcGdorzWFelfAocC88m7EBeiXC0 UZJg== X-Gm-Message-State: AIVw113elAAcrGA+0HOHbPDyWg3O0XMeAPDHgfniaFes936r/VFEQym6 zjhS8ApaYAz4L2WX X-Received: by 10.107.164.207 with SMTP id d76mr6436459ioj.158.1500007518746; Thu, 13 Jul 2017 21:45:18 -0700 (PDT) Received: from lylat (S010664777d9cebe3.ss.shawcable.net. [70.64.85.59]) by smtp.gmail.com with ESMTPSA id n19sm4118459ioe.3.2017.07.13.21.45.17 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 13 Jul 2017 21:45:17 -0700 (PDT) From: Alex To: npostavs@users.sourceforge.net Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> <874luhbo4l.fsf@lylat> <87lgntdswo.fsf@users.sourceforge.net> <87k23d9gvt.fsf@lylat> <87d195dmr0.fsf@users.sourceforge.net> <87lgns7y7g.fsf@lylat> <877ezbew3d.fsf@users.sourceforge.net> <87d193ljdp.fsf@lylat> Date: Thu, 13 Jul 2017 22:45:15 -0600 In-Reply-To: <87d193ljdp.fsf@lylat> (Alex's message of "Thu, 13 Jul 2017 22:42:10 -0600") Message-ID: <878tjrlj8k.fsf@lylat> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 24402 Cc: 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.5 (/) --=-=-= Content-Type: text/plain Alex writes: > I believe the below patch does that, though it has some issues. Here's the patch, sorry: --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=0001-Catch-argument-and-expansion-errors-in-ert.patch >From ba2703bce46e7caa6e86d7406a712bd92d06f84d Mon Sep 17 00:00:00 2001 From: Alexander Gramiak Date: Thu, 13 Jul 2017 14:54:35 -0600 Subject: [PATCH] Catch argument and expansion errors in ert This kludge catches errors caused by evaluating arguments in ert's should, should-not, and should-error macros; it also catches macro and inline function expansion errors inside of the above macros (Bug#24402). * lisp/emacs-lisp/ert.el: (ert--should-signal-hook): New function. (ert--expand-should-1): Catch expansion errors. * test/lisp/emacs-lisp/ert-tests.el (ert-test-should-error-argument) (ert-test-should-error-macroexpansion): Tests for argument and expansion errors. --- lisp/emacs-lisp/ert.el | 53 +++++++++++++++++++++++++++++++-------- test/lisp/emacs-lisp/ert-tests.el | 9 +++++++ 2 files changed, 52 insertions(+), 10 deletions(-) diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el index eb2b2e3e11..353cbb8d57 100644 --- a/lisp/emacs-lisp/ert.el +++ b/lisp/emacs-lisp/ert.el @@ -266,6 +266,14 @@ DATA is displayed to the user and should state the reason for skipping." (when ert--should-execution-observer (funcall ert--should-execution-observer form-description))) +;; See Bug#24402 for why this exists +(defun ert--should-signal-hook (error-symbol data) + "Stupid hack to stop `condition-case' from catching ert signals. +It should only be stopped when ran from inside ert--run-test-internal." + (when (and (not (symbolp debugger)) ; only run on anonymous debugger + (memq error-symbol '(ert-test-failed ert-test-skipped))) + (funcall debugger 'error data))) + (defun ert--special-operator-p (thing) "Return non-nil if THING is a symbol naming a special operator." (and (symbolp thing) @@ -276,13 +284,27 @@ DATA is displayed to the user and should state the reason for skipping." (defun ert--expand-should-1 (whole form inner-expander) "Helper function for the `should' macro and its variants." (let ((form - (macroexpand form (append (bound-and-true-p - byte-compile-macro-environment) - (cond - ((boundp 'macroexpand-all-environment) - macroexpand-all-environment) - ((boundp 'cl-macro-environment) - cl-macro-environment)))))) + ;; catch all macro and inline function expansion errors + ;; FIXME: Code inside of tests should be evaluated like it is + ;; outside of tests, with the sole exception of error + ;; handling + (condition-case err + (funcall + ;; expand only when byte-compiling + (lambda (form) + (if (and (bound-and-true-p byte-compile-current-file) + (consp form)) + (byte-compile-inline-expand form) + form)) + (macroexpand-all form + (append (bound-and-true-p + byte-compile-macro-environment) + (cond + ((boundp 'macroexpand-all-environment) + macroexpand-all-environment) + ((boundp 'cl-macro-environment) + cl-macro-environment))))) + (error `(signal ',(car err) ',(cdr err)))))) (cond ((or (atom form) (ert--special-operator-p (car form))) (let ((value (cl-gensym "value-"))) @@ -298,13 +320,20 @@ DATA is displayed to the user and should state the reason for skipping." (cl-assert (or (symbolp fn-name) (and (consp fn-name) (eql (car fn-name) 'lambda) - (listp (cdr fn-name))))) + (listp (cdr fn-name))) + ;; for inline functions + (byte-code-function-p fn-name))) (let ((fn (cl-gensym "fn-")) (args (cl-gensym "args-")) (value (cl-gensym "value-")) (default-value (cl-gensym "ert-form-evaluation-aborted-"))) - `(let ((,fn (function ,fn-name)) - (,args (list ,@arg-forms))) + `(let* ((,fn (function ,fn-name)) + (,args (condition-case err + (let ((signal-hook-function #'ert--should-signal-hook)) + (list ,@arg-forms)) + (error (progn (setq ,fn #'signal) + (list (car err) + (cdr err))))))) (let ((,value ',default-value)) ,(funcall inner-expander `(setq ,value (apply ,fn ,args)) @@ -766,6 +795,10 @@ This mainly sets up debugger-related bindings." ;; too expensive, we can remove it. (with-temp-buffer (save-window-excursion + ;; FIXME: Use `signal-hook-function' instead of `debugger' to + ;; handle ert errors. Once that's done, remove + ;; `ert--should-signal-hook'. See Bug#24402 and Bug#11218 for + ;; details. (let ((debugger (lambda (&rest args) (ert--run-test-debugger test-execution-info args))) diff --git a/test/lisp/emacs-lisp/ert-tests.el b/test/lisp/emacs-lisp/ert-tests.el index 317838b250..b6a7eb68da 100644 --- a/test/lisp/emacs-lisp/ert-tests.el +++ b/test/lisp/emacs-lisp/ert-tests.el @@ -294,6 +294,15 @@ failed or if there was a problem." "the error signaled was a subtype of the expected type"))))) )) +(ert-deftest ert-test-should-error-argument () + "Errors due to evaluating arguments should not break tests." + (should-error (identity (/ 1 0)))) + +(ert-deftest ert-test-should-error-macroexpansion () + "Errors due to expanding macros should not break tests." + (cl-macrolet ((test () (error "Foo"))) + (should-error (test)))) + (ert-deftest ert-test-skip-unless () ;; Don't skip. (let ((test (make-ert-test :body (lambda () (skip-unless t))))) -- 2.13.2 --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sat Jul 15 17:55:49 2017 Received: (at 24402) by debbugs.gnu.org; 15 Jul 2017 21:55:49 +0000 Received: from localhost ([127.0.0.1]:40755 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dWV2T-000123-CW for submit@debbugs.gnu.org; Sat, 15 Jul 2017 17:55:49 -0400 Received: from mail-io0-f171.google.com ([209.85.223.171]:35383) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dWV2R-00011K-GG for 24402@debbugs.gnu.org; Sat, 15 Jul 2017 17:55:47 -0400 Received: by mail-io0-f171.google.com with SMTP id h134so28565658iof.2 for <24402@debbugs.gnu.org>; Sat, 15 Jul 2017 14:55:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=T7JO+wW+TiNf+LAMQRJj0rwur5vjZamkT0bHwE82l3E=; b=UBc1Owa5sFG+peAv1ViNbBaR035fg9AmfiT2rB+Ira37TyhfGHL0YNlXgNPh8Fa7x4 +PWy3X0MUQK9luoCk62qKFcCUcrV5rYyGfSp6CZRUkeWGRrbyigs/nlyj16ncAO9zWE1 urI12e1aft9Pr7sJRk/hNCulPx9uyhAU7MSZzw3xmV4TUIh0X+sOhgyGILznCnEZAVIn jOXuthHN/ahAjZ1f14BdzwdzpgshImLhDcwR+t3LVp4oDTWegbCNs6Hw1WKVGUXkP6qj hkgshj3s06NVH80rQ/o+psOl3mw6W91GMjokX2H2nxMxjpvQGdCCvpNFTWSNNk3BhOeH f7Wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:references:date :in-reply-to:message-id:user-agent:mime-version; bh=T7JO+wW+TiNf+LAMQRJj0rwur5vjZamkT0bHwE82l3E=; b=DIVNg9V8rSFAymG8levDZHA149dvWMlycLLbL8kz9AnidawA8Hjt+R2Uo0CPHBuqwt d6TZ731a3v59tL/ICK+ePEDCFrfNJl0lUGWiK3+NV9N7pW2/g12ifUKYsq0KPhIKx9Z8 PjvV1wSciXJ64j45AAksvpY/X8QRVJIdVrexTrPqWA0DAV8qJywl+IFMAucuN0E0BMoK DmjjvtsxdqKutK5QaBAsJctMnvnYVNi9tyU0A0QA4z+EdO71B7I0lYqkRkKwXaNCfEwQ x8Un6iXwserTcC1aHJwbHbcblG95UU6g4ovS4/CvpHyT8kXsQKb05jFyxZFRv7xuBGej TWgA== X-Gm-Message-State: AIVw1130CaIwE2+toYdVqa/jKfqp/dONgb0QM3HLRJF+2uYnZo7lB1oy DfufOX8NWnPyQw== X-Received: by 10.107.181.16 with SMTP id e16mr4565868iof.70.1500155741834; Sat, 15 Jul 2017 14:55:41 -0700 (PDT) Received: from zony ([45.2.7.65]) by smtp.googlemail.com with ESMTPSA id f19sm3458902itf.27.2017.07.15.14.55.40 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 15 Jul 2017 14:55:41 -0700 (PDT) From: npostavs@users.sourceforge.net To: Alex Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> <874luhbo4l.fsf@lylat> <87lgntdswo.fsf@users.sourceforge.net> <87k23d9gvt.fsf@lylat> <87d195dmr0.fsf@users.sourceforge.net> <87lgns7y7g.fsf@lylat> <877ezbew3d.fsf@users.sourceforge.net> <87d193ljdp.fsf@lylat> Date: Sat, 15 Jul 2017 17:57:14 -0400 In-Reply-To: <87d193ljdp.fsf@lylat> (Alex's message of "Thu, 13 Jul 2017 22:42:10 -0600") Message-ID: <87shhxcqit.fsf@users.sourceforge.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.6 (--) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , Tino Calancha , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.6 (--) Alex writes: > The segfault appears to have been because I didn't wipe out the elc > files when testing different implementations. I suspect getting a segfault might indicate an actual bug somewhere. > I spent a lot longer than I'd like to admit finding this out. Is there a > reason why "make clean" in the test directory doesn't wipe out elc > files? I don't understand why there's a separate bootstrap-clean that > does this. Can this and TEST_LOAD_EL please be documented in the test > README? I think it was basically copied from the other Makefiles, where cleaning all elc files would mean a very long subsequent compilation. It might make sense to break the pattern for the test/ subdirectory though. > Anyway, I got everything back in order. Sadly, there's a couple extra > tests that now fail for me in the patch that *doesn't* expand inline > functions, and these don't fail for me in a clean master. They are in > eieio-tests (23 and 24). I'm seeing eieio-tests failing also in master. This seems to be an actual bug, in the definition of `cl-typep' I think. I've opened a new bug for this (Bug#27718). > With the inline expansion, I also get some errors in ert-tests. All of > the errors, with the exception of subr-tests error, seem to be from > cl-defstruct and cl-typep (which is defined by define-inline). > > Do you have any ideas? There should be 5 unexpected errors without the > inline expansion, and 6 errors with it. Note that all tests pass in both > cases without "TEST_LOAD_EL=no". > > If it's easy to fix the eieio tests and not the other ones, then it > might be better to leave the inline-function expansion out for now. I have a fix for the subr-tests failed, as for the others, I don't know enough about the compilation process to untangle it yet. I think we should just leave the inline-function expansion part out for now, at which point I believe your patch won't be making anything worse, so it should be okay to install. From debbugs-submit-bounces@debbugs.gnu.org Sat Jul 15 23:49:21 2017 Received: (at 24402) by debbugs.gnu.org; 16 Jul 2017 03:49:22 +0000 Received: from localhost ([127.0.0.1]:40915 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dWaYb-0005VS-Ej for submit@debbugs.gnu.org; Sat, 15 Jul 2017 23:49:21 -0400 Received: from mail-io0-f196.google.com ([209.85.223.196]:36827) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dWaYZ-0005VE-5J for 24402@debbugs.gnu.org; Sat, 15 Jul 2017 23:49:19 -0400 Received: by mail-io0-f196.google.com with SMTP id h134so5151307iof.3 for <24402@debbugs.gnu.org>; Sat, 15 Jul 2017 20:49:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=cvF7pPFKCThi01/TaAiaNF9TdCesBwbNQ6dS+DbI8Ow=; b=YP0ETkjGhaCuDTb36e0jnhvv1qrF2abeoA2H24uewvgIfMd/f4dpEJLVOzljlWNVEv qX/+ZFyNzsJ4pmt4Qw16/qIFDt5g1Qu2MOGC+3kZgNgVzyqfrOPnkTii6t+Ezqmnt+Wt p5h6A3HRnztL/M8WngdQc0DEp/7Z4b87JcVpOmdLOfdzGuXnQp1PhyA7qoT8LzRVSf40 fb3IF594O+0fq7b+yFm38bcUIpAC82f3It1q2nEyXrybX2Gu9nA/K4+FmgxW8f+70bEs c9Zqll9Wkc3IWS89Sit3ehiDLlspBZxHLVdoGiDmf7HgSOVEtUnad9t4ydk3wcAuHYuc f4nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=cvF7pPFKCThi01/TaAiaNF9TdCesBwbNQ6dS+DbI8Ow=; b=eD0dNmFuqZG9lPG/+HumoYrXwb4U6LAOwDCUnPYeaMm5PbNnnr0xu7/pyyt8qHZey+ JtrNJakIZeE4s1G5qvHolUSG2PRuihSBEeAGrZX1znjdBCWr7KMyc4XtKFJsJxoZ6AYz FIW9y8VJzmbaaBmLs4t4Z1Sla2MGlqvM1B1e1+XaCG68FN9tLoOg9BoPDfdDduDF5pqO FXNyokZd27R5RYkTtFIoiiWZ7ZGx7oAMxVufYUW/SfAZOzQCyELI/qN6EBMO85tB5Pnn IDQBewgbQoMrndtorjmhw3uuhMiOM0yGvt/CxreDBKekOjbREMKpOjap4NKK+jA+AW8b oGSQ== X-Gm-Message-State: AIVw1102rwUFcGDad9MfnDLSuLFGhWQF8TqMVltOhvIf8619JMnqmYZo WjefDV4D5t+l+A== X-Received: by 10.107.162.6 with SMTP id l6mr16210862ioe.84.1500176953455; Sat, 15 Jul 2017 20:49:13 -0700 (PDT) Received: from lylat (S010664777d9cebe3.ss.shawcable.net. [70.64.85.59]) by smtp.gmail.com with ESMTPSA id b17sm4854483itd.0.2017.07.15.20.49.10 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sat, 15 Jul 2017 20:49:11 -0700 (PDT) From: Alex To: npostavs@users.sourceforge.net Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> <874luhbo4l.fsf@lylat> <87lgntdswo.fsf@users.sourceforge.net> <87k23d9gvt.fsf@lylat> <87d195dmr0.fsf@users.sourceforge.net> <87lgns7y7g.fsf@lylat> <877ezbew3d.fsf@users.sourceforge.net> <87d193ljdp.fsf@lylat> <87shhxcqit.fsf@users.sourceforge.net> Date: Sat, 15 Jul 2017 21:49:06 -0600 In-Reply-To: <87shhxcqit.fsf@users.sourceforge.net> (npostavs's message of "Sat, 15 Jul 2017 17:57:14 -0400") Message-ID: <87inithwi5.fsf@lylat> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , Tino Calancha , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) --=-=-= Content-Type: text/plain npostavs@users.sourceforge.net writes: > Alex writes: > >> The segfault appears to have been because I didn't wipe out the elc >> files when testing different implementations. > > I suspect getting a segfault might indicate an actual bug somewhere. Probably, but I can't reproduce it anymore since I got rid of the offending .elc files. >> I spent a lot longer than I'd like to admit finding this out. Is there a >> reason why "make clean" in the test directory doesn't wipe out elc >> files? I don't understand why there's a separate bootstrap-clean that >> does this. Can this and TEST_LOAD_EL please be documented in the test >> README? > > I think it was basically copied from the other Makefiles, where cleaning > all elc files would mean a very long subsequent compilation. It might > make sense to break the pattern for the test/ subdirectory though. That makes sense to me, but if others disagree then the behaviour being documented is the next best thing. >> Anyway, I got everything back in order. Sadly, there's a couple extra >> tests that now fail for me in the patch that *doesn't* expand inline >> functions, and these don't fail for me in a clean master. They are in >> eieio-tests (23 and 24). > > I'm seeing eieio-tests failing also in master. This seems to be an > actual bug, in the definition of `cl-typep' I think. I've opened a new > bug for this (Bug#27718). Oddly enough, I can't reproduce this on master. I cloned a new copy, ran "./autogen.sh && ./configure && make -j4", then ran "make eieio-tests TEST_LOAD_EL=no" with no error. I cloned from 30444c695, then tried again with 7a0fb008. I also tried "make check-expensive TEST_LOAD_EL=no" and got only 2 errors (dom-tests and cl-lib-tests). Perhaps odder is that I can still reproduce your recipe in Bug#27718 in that same repository. >> With the inline expansion, I also get some errors in ert-tests. All of >> the errors, with the exception of subr-tests error, seem to be from >> cl-defstruct and cl-typep (which is defined by define-inline). >> >> Do you have any ideas? There should be 5 unexpected errors without the >> inline expansion, and 6 errors with it. Note that all tests pass in both >> cases without "TEST_LOAD_EL=no". >> >> If it's easy to fix the eieio tests and not the other ones, then it >> might be better to leave the inline-function expansion out for now. > > I have a fix for the subr-tests failed, as for the others, I don't know > enough about the compilation process to untangle it yet. I think we > should just leave the inline-function expansion part out for now, at > which point I believe your patch won't be making anything worse, so it > should be okay to install. Sounds good. It would be lucky if a fix to Bug#27718 also resolves the other inline function errors so that we could use the previous patch. Here's an updated patch with inline function excluded: --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=0001-Catch-argument-and-macroexpansion-errors-in-ert.patch >From 05f0734dd0b066058d263756a2e98cafc4ef8515 Mon Sep 17 00:00:00 2001 From: Alexander Gramiak Date: Thu, 13 Jul 2017 14:54:35 -0600 Subject: [PATCH] Catch argument and macroexpansion errors in ert This kludge catches errors caused by evaluating arguments in ert's should, should-not, and should-error macros; it also catches macroexpansion errors inside of the above macros (Bug#24402). * lisp/emacs-lisp/ert.el: (ert--should-signal-hook): New function. (ert--expand-should-1): Catch macroexpansion errors. * test/lisp/emacs-lisp/ert-tests.el (ert-test-should-error-argument) (ert-test-should-error-macroexpansion): Tests for argument and expansion errors. --- lisp/emacs-lisp/ert.el | 41 ++++++++++++++++++++++++++++++--------- test/lisp/emacs-lisp/ert-tests.el | 9 +++++++++ 2 files changed, 41 insertions(+), 9 deletions(-) diff --git a/lisp/emacs-lisp/ert.el b/lisp/emacs-lisp/ert.el index eb2b2e3e11..caa4a9f389 100644 --- a/lisp/emacs-lisp/ert.el +++ b/lisp/emacs-lisp/ert.el @@ -266,6 +266,14 @@ DATA is displayed to the user and should state the reason for skipping." (when ert--should-execution-observer (funcall ert--should-execution-observer form-description))) +;; See Bug#24402 for why this exists +(defun ert--should-signal-hook (error-symbol data) + "Stupid hack to stop `condition-case' from catching ert signals. +It should only be stopped when ran from inside ert--run-test-internal." + (when (and (not (symbolp debugger)) ; only run on anonymous debugger + (memq error-symbol '(ert-test-failed ert-test-skipped))) + (funcall debugger 'error data))) + (defun ert--special-operator-p (thing) "Return non-nil if THING is a symbol naming a special operator." (and (symbolp thing) @@ -273,16 +281,22 @@ DATA is displayed to the user and should state the reason for skipping." (and (subrp definition) (eql (cdr (subr-arity definition)) 'unevalled))))) +;; FIXME: Code inside of here should probably be evaluated like it is +;; outside of tests, with the sole exception of error handling (defun ert--expand-should-1 (whole form inner-expander) "Helper function for the `should' macro and its variants." (let ((form - (macroexpand form (append (bound-and-true-p - byte-compile-macro-environment) - (cond - ((boundp 'macroexpand-all-environment) - macroexpand-all-environment) - ((boundp 'cl-macro-environment) - cl-macro-environment)))))) + ;; catch macroexpansion errors + (condition-case err + (macroexpand-all form + (append (bound-and-true-p + byte-compile-macro-environment) + (cond + ((boundp 'macroexpand-all-environment) + macroexpand-all-environment) + ((boundp 'cl-macro-environment) + cl-macro-environment)))) + (error `(signal ',(car err) ',(cdr err)))))) (cond ((or (atom form) (ert--special-operator-p (car form))) (let ((value (cl-gensym "value-"))) @@ -303,8 +317,13 @@ DATA is displayed to the user and should state the reason for skipping." (args (cl-gensym "args-")) (value (cl-gensym "value-")) (default-value (cl-gensym "ert-form-evaluation-aborted-"))) - `(let ((,fn (function ,fn-name)) - (,args (list ,@arg-forms))) + `(let* ((,fn (function ,fn-name)) + (,args (condition-case err + (let ((signal-hook-function #'ert--should-signal-hook)) + (list ,@arg-forms)) + (error (progn (setq ,fn #'signal) + (list (car err) + (cdr err))))))) (let ((,value ',default-value)) ,(funcall inner-expander `(setq ,value (apply ,fn ,args)) @@ -766,6 +785,10 @@ This mainly sets up debugger-related bindings." ;; too expensive, we can remove it. (with-temp-buffer (save-window-excursion + ;; FIXME: Use `signal-hook-function' instead of `debugger' to + ;; handle ert errors. Once that's done, remove + ;; `ert--should-signal-hook'. See Bug#24402 and Bug#11218 for + ;; details. (let ((debugger (lambda (&rest args) (ert--run-test-debugger test-execution-info args))) diff --git a/test/lisp/emacs-lisp/ert-tests.el b/test/lisp/emacs-lisp/ert-tests.el index 317838b250..b6a7eb68da 100644 --- a/test/lisp/emacs-lisp/ert-tests.el +++ b/test/lisp/emacs-lisp/ert-tests.el @@ -294,6 +294,15 @@ failed or if there was a problem." "the error signaled was a subtype of the expected type"))))) )) +(ert-deftest ert-test-should-error-argument () + "Errors due to evaluating arguments should not break tests." + (should-error (identity (/ 1 0)))) + +(ert-deftest ert-test-should-error-macroexpansion () + "Errors due to expanding macros should not break tests." + (cl-macrolet ((test () (error "Foo"))) + (should-error (test)))) + (ert-deftest ert-test-skip-unless () ;; Don't skip. (let ((test (make-ert-test :body (lambda () (skip-unless t))))) -- 2.13.2 --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Sun Jul 16 20:44:36 2017 Received: (at 24402) by debbugs.gnu.org; 17 Jul 2017 00:44:36 +0000 Received: from localhost ([127.0.0.1]:42898 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dWu9M-0003cw-99 for submit@debbugs.gnu.org; Sun, 16 Jul 2017 20:44:36 -0400 Received: from mail-io0-f170.google.com ([209.85.223.170]:32909) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dWu9K-0003ck-Lv for 24402@debbugs.gnu.org; Sun, 16 Jul 2017 20:44:35 -0400 Received: by mail-io0-f170.google.com with SMTP id h64so35751163iod.0 for <24402@debbugs.gnu.org>; Sun, 16 Jul 2017 17:44:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=Wux290nuf+TGHiCq/c7IYJG1Tkw1QOUoI7X0k9vAsOk=; b=SCk6tI1xnnfvApK3EUqbSh9VAIqnQE1aQAch2hjDTjyAxLCMaIZKe3T6g6FW2uaCPB jYVHovg3L8yZjUYI58VoT4uflD/4Oi5fp+LdBd6cxTnI+1GYN13un8CPxSx8ndGFG5ov 1Ccw25MlK6DKYCwubkH5fP2gqM7JsBE2Fj37HaEiL4DLba1TTfNswjLndghtYMR9ysF3 da3AtvdZf3ruX1qbT0eX4B+5Y6oGq7On0j2eS9BxLqLVBHHa6spfZNvdmDEC2fQOkOdQ 337Q6j7zIwE0SMAlYy5ENqLtOkCP9TvTf3bhlJssx/+f4vGP0r5zaLLbKgKDdmJqo62B t3Pg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:references:date :in-reply-to:message-id:user-agent:mime-version; bh=Wux290nuf+TGHiCq/c7IYJG1Tkw1QOUoI7X0k9vAsOk=; b=dBAdkr36+mGqA89j22fgLNevxp0eKgVvbGj6VrLHs5PfWwi1iU96YkhgPQuyP/xfeg EK+pom8TEdj5HTUMaSdaq/zWI4ZWKBq9re21/wyi/Q8uvN9PQWyyWUyASVIhhltHCXLP 9n2Tss+ucfytL123G9/DSzETQffhB/OpeWXtySYkMJtTN2tpnTil9DzEUHo5HhabaSVc 1ePw0zhKGFHJ2WBr4cIrbc4+DiNezRXlscO6AXZKp/vvSlKT1vSag8BVFbT+24RVLQct 8sLjNKRmE+wCrODiQARf8Zq3lWNO3UbXBmDtqwXZsLq2S7+UluS/Nd/trnJxDN02QqP8 f9cg== X-Gm-Message-State: AIVw110TIPbmAOMM3jDnC28h8Nlpso07fNkx4MY0Oic3XNiJMQ9p3Kw6 GoMYCepbXLlH7g== X-Received: by 10.107.173.15 with SMTP id w15mr16375077ioe.99.1500252269085; Sun, 16 Jul 2017 17:44:29 -0700 (PDT) Received: from zony ([45.2.7.65]) by smtp.googlemail.com with ESMTPSA id 189sm4864167itl.11.2017.07.16.17.44.27 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Sun, 16 Jul 2017 17:44:28 -0700 (PDT) From: npostavs@users.sourceforge.net To: Alex Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> <874luhbo4l.fsf@lylat> <87lgntdswo.fsf@users.sourceforge.net> <87k23d9gvt.fsf@lylat> <87d195dmr0.fsf@users.sourceforge.net> <87lgns7y7g.fsf@lylat> <877ezbew3d.fsf@users.sourceforge.net> <87d193ljdp.fsf@lylat> <87shhxcqit.fsf@users.sourceforge.net> <87inithwi5.fsf@lylat> Date: Sun, 16 Jul 2017 20:46:02 -0400 In-Reply-To: <87inithwi5.fsf@lylat> (Alex's message of "Sat, 15 Jul 2017 21:49:06 -0600") Message-ID: <87vamrc2lx.fsf@users.sourceforge.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.1 (--) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , Tino Calancha , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.1 (--) Alex writes: > Oddly enough, I can't reproduce this on master. I cloned a new copy, ran > "./autogen.sh && ./configure && make -j4", then ran "make eieio-tests > TEST_LOAD_EL=no" with no error. I cloned from 30444c695, then tried > again with 7a0fb008. I also tried "make check-expensive TEST_LOAD_EL=no" > and got only 2 errors (dom-tests and cl-lib-tests). > > Perhaps odder is that I can still reproduce your recipe in Bug#27718 in > that same repository. Uh, actually, now I'm getting the same results as you (i.e., make eieio-tests TEST_LOAD_EL=no passes on master for me, though the #27718 test case does fail). Perhaps I accidentally ran an eieio-tests.elc file produced by a patched Emacs using a non-patched Emacs before? >> I have a fix for the subr-tests failed, as for the others, I don't know >> enough about the compilation process to untangle it yet. I think we >> should just leave the inline-function expansion part out for now, at >> which point I believe your patch won't be making anything worse, so it >> should be okay to install. > > Sounds good. It would be lucky if a fix to Bug#27718 also resolves the > other inline function errors so that we could use the previous patch. > Here's an updated patch with inline function excluded: Okay, so the patch does make a couple more tests fail, but I think it's actually an improvement since #27718 shows that the scenario was already failing outside of ert-deftest, i.e., in this case your patch is making code inside the test more like code outside the test. We can just mark those as expected failures when compiled. I will merge to master in a few days, assuming there are no objections. From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 19 17:23:39 2017 Received: (at 24402) by debbugs.gnu.org; 19 Jul 2017 21:23:39 +0000 Received: from localhost ([127.0.0.1]:48194 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dXwRX-0005Oi-5U for submit@debbugs.gnu.org; Wed, 19 Jul 2017 17:23:39 -0400 Received: from aibo.runbox.com ([91.220.196.211]:36346) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dXwRU-0005OY-M2 for 24402@debbugs.gnu.org; Wed, 19 Jul 2017 17:23:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=runbox.com; s=rbselector1; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date: References:Subject:Cc:To:From; bh=w6zrNdaqgPLU7P/hymInqQkuR5mS2KVewcR+LGMFxZI=; b=Uo9Qk2rG0HEziglda+2Ugq4BO/ uTYjA5saX1ERGJvTA1eMB9W8mmFy3l+ZrzKu6OovrUyhWctddJcQcTrcMkbr2bbrCn2dA7HOPiYlA f8IJHLBxTLSsalXg5tTDsJ+OO2MPouk/F6LINmymsXZ7yM6wT++Ot4Cu4v+iCiJvnvu+KNeKI6lze oO1VRp9eUlTG2iaJz+voXFEouzIJufBbLAirs/gq9Nm+ERlEzCh1o/A/QStKZGradmu6IUGFV1lTV P990i0Jm7gPSZ8ddBOoI6pX70JWB/WKW4WwSKi3BVC2DgmEM0JtJo61Pe/L6FzMTBXlWL4DvxnmYg ofuMMXIw==; Received: from [10.9.9.210] (helo=mailfront10.runbox.com) by mailtransmit03.runbox with esmtp (Exim 4.86_2) (envelope-from ) id 1dXwRR-00086G-FU; Wed, 19 Jul 2017 23:23:33 +0200 Received: from c-24-22-244-161.hsd1.wa.comcast.net ([24.22.244.161] helo=chinook) by mailfront10.runbox.com with esmtpsa (uid:179284 ) (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) id 1dXwRJ-0007vq-Bx; Wed, 19 Jul 2017 23:23:25 +0200 From: Gemini Lasswell To: Alex Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> <874luhbo4l.fsf@lylat> <87lgntdswo.fsf@users.sourceforge.net> <87k23d9gvt.fsf@lylat> <87d195dmr0.fsf@users.sourceforge.net> <87lgns7y7g.fsf@lylat> <877ezbew3d.fsf@users.sourceforge.net> <87d193ljdp.fsf@lylat> <87shhxcqit.fsf@users.sourceforge.net> <87inithwi5.fsf@lylat> Date: Wed, 19 Jul 2017 14:23:12 -0700 In-Reply-To: <87inithwi5.fsf@lylat> (Alex's message of "Sat, 15 Jul 2017 21:49:06 -0600") Message-ID: <87eftcglz3.fsf@runbox.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 24402 Cc: 24402@debbugs.gnu.org, Tino Calancha , npostavs@users.sourceforge.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) --=-=-= Content-Type: text/plain Hello, First, thank you for working on this bug! I have a branch of Emacs which attempts to run all the tests under Testcover, and I've tried it with Alex's latest patch. The patch has fixed the problems with Testcover and should-error except for one weird case. Here is a file that will let you reproduce the weird case on master with the patch applied: --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=cyc-test.el (defun cyc1 (a) (let ((ls (make-list 10 a))) (nconc ls ls) ls)) (ert-deftest test-cycle-assq () (let ((c1 (cyc1 '(1)))) (should-error (assq 3 c1) :type 'circular-list))) (ert-deftest test-cycle-assoc () (let ((c1 (cyc1 '(1)))) (should-error (assoc 2 c1) :type 'circular-list))) (defvar testcover-started nil) (unless testcover-started (setq testcover-started t) (testcover-start (or load-file-name (buffer-file-name)))) --=-=-= Content-Type: text/plain This is an excerpt of test/src/fns-tests.el with a few lines added at the end to invoke Testcover. It contains two nearly identical tests which should both pass, but one passes and one fails. If you then edit the file and comment out test-cycle-assoc (the one that passes) and run the test again, then the failing test will pass. I've reproduced it both in batch mode (without TEST_LOAD_EL=no) and interactively. --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 19 18:40:17 2017 Received: (at 24402) by debbugs.gnu.org; 19 Jul 2017 22:40:17 +0000 Received: from localhost ([127.0.0.1]:48279 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dXxdg-0007AA-PF for submit@debbugs.gnu.org; Wed, 19 Jul 2017 18:40:17 -0400 Received: from mail-it0-f65.google.com ([209.85.214.65]:36073) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dXxdf-00079w-3s for 24402@debbugs.gnu.org; Wed, 19 Jul 2017 18:40:15 -0400 Received: by mail-it0-f65.google.com with SMTP id r9so67088ita.3 for <24402@debbugs.gnu.org>; Wed, 19 Jul 2017 15:40:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=h0r1TvyKbkbYyMPvGOs0Cby3hSdw8Sn+JgO65wESnH4=; b=Z+8kQrdgVtEJzzC/hQvSmsfMB1yUQhvTID+4nn72F1ENY6seedbtsNS4IPyOCLVMll ArVrqxHJr5slnXoTRvXLlSxWFlKN9yxz8h9HKIwa2rn7XdyN2YdJG69ftDzhc2KwnRWe F1bI3wLP49cKBFN2K0DnoGAZyXKGYmHlAcno76e09ZBOvUMbUvRbwA4Fx+IAjcXt3vWq ewOYgxvYGPvFnenqhlZ3ZVdkINSww3qe1w4K6X19lafCEpDPA7jWzdPkggZP53/5Elo4 jLOZCpvv8ZXgEcIZpW/7CL8bLkIk3lrmq/46LzxJBsYfjtyxY//jyu1zs71I+w0UwWtk +PXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=h0r1TvyKbkbYyMPvGOs0Cby3hSdw8Sn+JgO65wESnH4=; b=CZt2vQ2hDsUG24ivBF5R+bvlMMSEgJK9AhFqsUWS+Tty6/e4ajlcIQhUSrs3ZDzn7K fSora+cvHvIRrf3U091fU2ON6Sng/5A+2PZcf7jfvOhVoBltDqhCxWaNqd5L96FnHExi nh3AKpCaZMU77VGRa/k3DLWKxCYOWbUUC6DOuEGGRJ+kmd36sbB+W8hT+CQw840kCMNL o35JGTbqMc5lnxbJ1cl4pRRh7ZhHUvrwCohiFYejvxKkwakg68/OT63oeYrtLfC9Xxrl RpS5MvBtAAAKEmAFfxvYFTMviqGK/lebVViolNIXvJ2Hh2N2XPW4RehpyvBr/DX6Y8HJ hDYQ== X-Gm-Message-State: AIVw112ctU0/AycnXPfNzql3D8CIQS/haTcK3jACpiQ384IcSi6UwqyG WK5wOVNuVh9DIQ== X-Received: by 10.36.239.196 with SMTP id i187mr1419582ith.161.1500504009433; Wed, 19 Jul 2017 15:40:09 -0700 (PDT) Received: from lylat (S010664777d9cebe3.ss.shawcable.net. [70.64.85.59]) by smtp.gmail.com with ESMTPSA id j76sm544422ioi.14.2017.07.19.15.40.07 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 19 Jul 2017 15:40:08 -0700 (PDT) From: Alex To: Gemini Lasswell Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> <874luhbo4l.fsf@lylat> <87lgntdswo.fsf@users.sourceforge.net> <87k23d9gvt.fsf@lylat> <87d195dmr0.fsf@users.sourceforge.net> <87lgns7y7g.fsf@lylat> <877ezbew3d.fsf@users.sourceforge.net> <87d193ljdp.fsf@lylat> <87shhxcqit.fsf@users.sourceforge.net> <87inithwi5.fsf@lylat> <87eftcglz3.fsf@runbox.com> Date: Wed, 19 Jul 2017 16:40:03 -0600 In-Reply-To: <87eftcglz3.fsf@runbox.com> (Gemini Lasswell's message of "Wed, 19 Jul 2017 14:23:12 -0700") Message-ID: <87shhsuk3g.fsf@lylat> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 24402 Cc: 24402@debbugs.gnu.org, Tino Calancha , npostavs@users.sourceforge.net X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.5 (/) Gemini Lasswell writes: > Hello, > > First, thank you for working on this bug! > > I have a branch of Emacs which attempts to run all the tests under > Testcover, and I've tried it with Alex's latest patch. The patch has > fixed the problems with Testcover and should-error except for one weird > case. Here is a file that will let you reproduce the weird case on > master with the patch applied: > > > > This is an excerpt of test/src/fns-tests.el with a few lines added at > the end to invoke Testcover. It contains two nearly identical tests > which should both pass, but one passes and one fails. If you then edit > the file and comment out test-cycle-assoc (the one that passes) and run > the test again, then the failing test will pass. I've reproduced it both > in batch mode (without TEST_LOAD_EL=3Dno) and interactively. Hmm, this isn't the behaviour that I see. I get the following errors both with and without my patch, and both interactively and in batch mode: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Edebug: cyc1 Edebug: test@test-cycle-assq Edebug: test@test-cycle-assoc Edebug: cyc1 Edebug: test@test-cycle-assq Edebug: test@test-cycle-assoc ... Edebug: cyc1 Edebug: test@test-cycle-assq Compiler-macro error for cl--block-wrapper: (error "Lisp nesting exceeds = =E2=80=98max-lisp-eval-depth=E2=80=99") Edebug: test@test-cycle-assoc Compiler-macro error for cl--block-wrapper: (error "Lisp nesting exceeds = =E2=80=98max-lisp-eval-depth=E2=80=99") Edebug: cyc1 Edebug: test@test-cycle-assq Compiler-macro error for cl--block-wrapper: (error "Lisp nesting exceeds = =E2=80=98max-lisp-eval-depth=E2=80=99") Eager macro-expansion failure: (error "Lisp nesting exceeds =E2=80=98max-li= sp-eval-depth=E2=80=99") Compiler-macro error for cl--block-wrapper: (error "Lisp nesting exceeds = =E2=80=98max-lisp-eval-depth=E2=80=99") Eager macro-expansion failure: (error "Lisp nesting exceeds =E2=80=98max-li= sp-eval-depth=E2=80=99") Compiler-macro error for cl--block-wrapper: (error "Lisp nesting exceeds = =E2=80=98max-lisp-eval-depth=E2=80=99") Entering debugger... cl--generic-make-next-function: Symbol=E2=80=99s function definition is voi= d: t =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D The ... above represents Edebug cycling, for some reason, between the 3 functions. Noam, can you reproduce this? From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 19 19:03:31 2017 Received: (at 24402) by debbugs.gnu.org; 19 Jul 2017 23:03:31 +0000 Received: from localhost ([127.0.0.1]:48303 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dXy0A-0007j7-PB for submit@debbugs.gnu.org; Wed, 19 Jul 2017 19:03:30 -0400 Received: from mail-io0-f178.google.com ([209.85.223.178]:34856) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dXy07-0007it-TS for 24402@debbugs.gnu.org; Wed, 19 Jul 2017 19:03:28 -0400 Received: by mail-io0-f178.google.com with SMTP id p185so2717730iof.2 for <24402@debbugs.gnu.org>; Wed, 19 Jul 2017 16:03:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=8zfMefqZkWZyqkdGUTDenDkmENInEgJsKhQDTwFCPNQ=; b=fEWo+Nsb2oN3TZIVlak0hrKQmsS69K2GFsxckZyr1QaIMoUlzxq498A6XoJU0mC7Gf abtTme8ZzgccEi8q75GCgAQsC0r3b/MvXn1Cf9xrdYXmOhIC7zh62h9pQ6r+5wR9QXb/ PLrDatNrNxwKbRwp0Hy2gy/3ye20/MIDB7WFVqG89yh8nwafAD5tq3aJ/Qg7AOWdiHK4 q7YJ2DV2mtfdXvqBtHxw1o60IL/g2SRq0MLvhI8xIaGQ8IXRGghKctbBeMVpOjckjmc1 6WeUSIa0KP8p1M3SC3QQmzi0RwPeztqH025f47dnTx3CiTHnDwcSvTu8GG6leVmASWwm Mk1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:references:date :in-reply-to:message-id:user-agent:mime-version; bh=8zfMefqZkWZyqkdGUTDenDkmENInEgJsKhQDTwFCPNQ=; b=uZ1pJc6iDLBslMPwZc6Timt+jEx4p+WnIzrk0oJadcsKEJglYD6Cmq3c1z/6aQatHJ UCh7VkVWprN/NrsiHP5YZGDEiXSLpxmoZfCLpW0v7qg0M6SlDGhCimy19MQGfwRebrRQ YSsDVvZMqK7MvIeuDtUGt965db1+tVg3pYxsB+Z6u7JJwBVs3FmKy1Vg5JwvPcdYXuGJ 4G5IcrD8hFqpAdAVu/DBV/r10SB2MgNzB+jrM7Jiomqj3j0hCFdI1PbxSEBGFgAHw/wU 7vSxonzEv3uezotM4Ntw+vju6bqUvKH06/iYNt8xDd37ffqK80RMXYiHVBDILxOnUEbx 8A2Q== X-Gm-Message-State: AIVw1106fP45GzkxQt/tl7iBjH1cc1xo5uhdzjkv+BueXFqTlGKJG0YG y4d84bXWYCPqOw== X-Received: by 10.107.146.139 with SMTP id u133mr1910591iod.58.1500505402305; Wed, 19 Jul 2017 16:03:22 -0700 (PDT) Received: from zony ([45.2.7.65]) by smtp.googlemail.com with ESMTPSA id 134sm551821ioz.75.2017.07.19.16.03.21 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 19 Jul 2017 16:03:21 -0700 (PDT) From: npostavs@users.sourceforge.net To: Gemini Lasswell Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> <874luhbo4l.fsf@lylat> <87lgntdswo.fsf@users.sourceforge.net> <87k23d9gvt.fsf@lylat> <87d195dmr0.fsf@users.sourceforge.net> <87lgns7y7g.fsf@lylat> <877ezbew3d.fsf@users.sourceforge.net> <87d193ljdp.fsf@lylat> <87shhxcqit.fsf@users.sourceforge.net> <87inithwi5.fsf@lylat> <87eftcglz3.fsf@runbox.com> Date: Wed, 19 Jul 2017 19:04:57 -0400 In-Reply-To: <87eftcglz3.fsf@runbox.com> (Gemini Lasswell's message of "Wed, 19 Jul 2017 14:23:12 -0700") Message-ID: <87y3rk9gfa.fsf@users.sourceforge.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2.50 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -2.1 (--) X-Debbugs-Envelope-To: 24402 Cc: Tino Calancha , Alex , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.1 (--) --=-=-= Content-Type: text/plain Gemini Lasswell writes: > This is an excerpt of test/src/fns-tests.el with a few lines added at > the end to invoke Testcover. It contains two nearly identical tests > which should both pass, but one passes and one fails. If I run them again in the same Emacs session then they both fail. The problem seems to be that testcover saves values produced in tests, and when it tries to compare the 2 circular lists produced by these tests, a `circular-list' error is thrown. In other words, this is actually a totally separate bug (although it's hidden until a fix for #24402 is applied). The following appears to fix it, though perhaps we should use a smarter equal function that would consider the circular lists to actually be equal instead of bailing out and returning nil on circularity. --=-=-= Content-Type: text/x-diff Content-Disposition: inline; filename=v1-0001-Don-t-error-on-circular-values-in-testcover.patch Content-Description: patch >From 7160e8e45ef9e074360e394f99120d11b7ed4b8a Mon Sep 17 00:00:00 2001 From: Noam Postavsky Date: Wed, 19 Jul 2017 18:48:50 -0400 Subject: [PATCH v1] Don't error on circular values in testcover * lisp/emacs-lisp/testcover.el (testcover-after, testcover-1value): Consider circular lists to be non-equal instead of signaling error. --- lisp/emacs-lisp/testcover.el | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lisp/emacs-lisp/testcover.el b/lisp/emacs-lisp/testcover.el index 433ad38a14..bc6004a709 100644 --- a/lisp/emacs-lisp/testcover.el +++ b/lisp/emacs-lisp/testcover.el @@ -463,7 +463,9 @@ (defun testcover-after (idx val) (cond ((eq (aref testcover-vector idx) 'unknown) (aset testcover-vector idx val)) - ((not (equal (aref testcover-vector idx) val)) + ((not (condition-case () + (equal (aref testcover-vector idx) val) + (circular-list nil))) (aset testcover-vector idx 'ok-coverage))) val) @@ -475,7 +477,9 @@ (defun testcover-1value (idx val) ((eq (aref testcover-vector idx) '1value) (aset testcover-vector idx (cons '1value val))) ((not (and (eq (car-safe (aref testcover-vector idx)) '1value) - (equal (cdr (aref testcover-vector idx)) val))) + (condition-case () + (equal (cdr (aref testcover-vector idx)) val) + (circular-list nil)))) (error "Value of form marked with `1value' does vary: %s" val))) val) -- 2.11.1 --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Alex writes: > Hmm, this isn't the behaviour that I see. I get the following errors > both with and without my patch, and both interactively and in batch > mode: > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > Edebug: cyc1 > Edebug: test@test-cycle-assq > Edebug: test@test-cycle-assoc > Edebug: cyc1 > Edebug: test@test-cycle-assq > Edebug: test@test-cycle-assoc > ... > Edebug: cyc1 > Edebug: test@test-cycle-assq > Compiler-macro error for cl--block-wrapper: (error "Lisp nesting exceeds = =E2=80=98max-lisp-eval-depth=E2=80=99") Hmm, that's strange, I don't get that error (or any "Edebug: cyc1" repetitions after the first) on master, or with your patch. --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Thu Jul 20 00:04:19 2017 Received: (at 24402) by debbugs.gnu.org; 20 Jul 2017 04:04:19 +0000 Received: from localhost ([127.0.0.1]:48405 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dY2hH-0001LS-6o for submit@debbugs.gnu.org; Thu, 20 Jul 2017 00:04:19 -0400 Received: from mail-it0-f68.google.com ([209.85.214.68]:34966) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dY2hF-0001LE-CW for 24402@debbugs.gnu.org; Thu, 20 Jul 2017 00:04:17 -0400 Received: by mail-it0-f68.google.com with SMTP id v127so1243565itd.2 for <24402@debbugs.gnu.org>; Wed, 19 Jul 2017 21:04:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-transfer-encoding; bh=SONQATy7q7fn/VW87Yxf0920D0FDuKw2Ws/c5fFTuGY=; b=uB8hp/Vll1oQISS3atIUKWpJTOodf6oCsHmNDBpJIMqmb0rWOeu1dg4rMzjvdkeFOl 30f9BZeJ9Qj6osQGN+IJmgElY0vUddzR2vHfukfEAaNMQd4+JEuIdc0ldN5jLdEJ03WN G3P4rxIN3Qj5AZhVEQvev5hAKhsTv4364I2Ig/UVDhOhSzWfSNwIhkWXA/YnxXchU2jR 6eCWovgB+R6LYPnnshbOJ9SjfGxaAolVPxYaOeoskOSNKy3efHAOtX3CUYr+6QK4Vp/s 1vjNZaleBj2Bc/QixMSJz4GZAeh5ax/N3crs6UwSz7PfL3QELfOiVrpa8vyUsctkrcUz cBOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version:content-transfer-encoding; bh=SONQATy7q7fn/VW87Yxf0920D0FDuKw2Ws/c5fFTuGY=; b=oy/BCHX3MeGXtoqKlCY3FI+GfVRxFokl3nC1RJsGo6sH4Jd0kac3A4L0t/0Q458YOU LOpUrt2B/DQLrUSFDI+3DO39QV8KEFSQklfQJRms4kGalua0p2hzrv1saYYQ8tDxywID 3tALQ9rFmMUVXQCKT6UIlBgL9Irpu+csqIsJRqskMNIZevmt4emFNBCnWRv1ubbLrH+a zwZWHvHOcDJcR/yZI05bh3Db+O0kUjrhu5yP26RUqq8pbtYF6m+WNEFwN82tShkCkC1t 88LGzGwfiz7w8jgnBPFLeFryk7DnPx929FjECforiyO//dCyaHX0DcNVc6PDFdj0lG5V icUQ== X-Gm-Message-State: AIVw113t+B6jh+zedLAvXo4h5seT+InbqKHe8GrOp24lUap4q3ta8NGU VC+E6yJ3Lfjs+g== X-Received: by 10.36.142.193 with SMTP id h184mr2116543ite.4.1500523451772; Wed, 19 Jul 2017 21:04:11 -0700 (PDT) Received: from lylat (S010664777d9cebe3.ss.shawcable.net. [70.64.85.59]) by smtp.gmail.com with ESMTPSA id a70sm821139ioe.88.2017.07.19.21.04.09 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Wed, 19 Jul 2017 21:04:10 -0700 (PDT) From: Alex To: npostavs@users.sourceforge.net Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> <874luhbo4l.fsf@lylat> <87lgntdswo.fsf@users.sourceforge.net> <87k23d9gvt.fsf@lylat> <87d195dmr0.fsf@users.sourceforge.net> <87lgns7y7g.fsf@lylat> <877ezbew3d.fsf@users.sourceforge.net> <87d193ljdp.fsf@lylat> <87shhxcqit.fsf@users.sourceforge.net> <87inithwi5.fsf@lylat> <87eftcglz3.fsf@runbox.com> <87y3rk9gfa.fsf@users.sourceforge.net> Date: Wed, 19 Jul 2017 22:04:03 -0600 In-Reply-To: <87y3rk9gfa.fsf@users.sourceforge.net> (npostavs's message of "Wed, 19 Jul 2017 19:04:57 -0400") Message-ID: <87fudrvjnw.fsf@lylat> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 24402 Cc: Gemini Lasswell , Tino Calancha , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.5 (/) npostavs@users.sourceforge.net writes: > Gemini Lasswell writes: > >> This is an excerpt of test/src/fns-tests.el with a few lines added at >> the end to invoke Testcover. It contains two nearly identical tests >> which should both pass, but one passes and one fails. > > If I run them again in the same Emacs session then they both fail. The > problem seems to be that testcover saves values produced in tests, and > when it tries to compare the 2 circular lists produced by these tests, a > `circular-list' error is thrown. In other words, this is actually a > totally separate bug (although it's hidden until a fix for #24402 is > applied). > > The following appears to fix it, though perhaps we should use a smarter > equal function that would consider the circular lists to actually be > equal instead of bailing out and returning nil on circularity. That would be nice. > Alex writes: >=20 > > Hmm, this isn't the behaviour that I see. I get the following errors > > both with and without my patch, and both interactively and in batch > > mode: > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > Edebug: cyc1 > > Edebug: test@test-cycle-assq > > Edebug: test@test-cycle-assoc > > Edebug: cyc1 > > Edebug: test@test-cycle-assq > > Edebug: test@test-cycle-assoc > > ... > > Edebug: cyc1 > > Edebug: test@test-cycle-assq > > Compiler-macro error for cl--block-wrapper: (error "Lisp nesting exceed= s =E2=80=98max-lisp-eval-depth=E2=80=99") >=20 > Hmm, that's strange, I don't get that error (or any "Edebug: cyc1" > repetitions after the first) on master, or with your patch. Whoops, I changed the defvar to a setq and forgot to change it back. Doing this results in the error I posted. (Which makes sense since it keeps reloading the file, resulting in the "Edebug:" loop. The compiler-macro errors and `t' being called as a function are a bit odd, though). I get the same behaviour as you and Gemini now. Sorry about the noise. From debbugs-submit-bounces@debbugs.gnu.org Thu Jul 20 15:23:52 2017 Received: (at 24402) by debbugs.gnu.org; 20 Jul 2017 19:23:52 +0000 Received: from localhost ([127.0.0.1]:50032 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dYH39-0005ix-N3 for submit@debbugs.gnu.org; Thu, 20 Jul 2017 15:23:52 -0400 Received: from aibo.runbox.com ([91.220.196.211]:41808) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dYH37-0005in-IR for 24402@debbugs.gnu.org; Thu, 20 Jul 2017 15:23:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=runbox.com; s=rbselector1; h=Content-Type:MIME-Version:Message-ID:In-Reply-To:Date: References:Subject:Cc:To:From; bh=ngknYz6HSFN6AxbRCgpqoQVxW7qvfUY9b52FahD9qN0=; b=b1dy/tCai8By7czUuJwIWZ+5cb hp00t9pHzKCOxcXtZLYUrbcIeQdvJ4kKp+dMd1lpFUQg16/7arNItw8XCChv+gCOUoQpRCu31yZq9 qWZ0dMj4nQicLHG/kG4lXaWoRVo8xGZkDSj69EtEC9GlNK5mtkizm+yd60A1bvPZIBKMoBBO7vMdT g0SwoMbYD7ASrKHG64EtJHlAlvbFxb2gpcT3gPIOYgifVm4arAn9cmcUjN7t26zV8DVek2QWKif7u U2TdAvF8ivoTBL5DATzFoYvzj9NOljjbZzljuDaL0AIlNU/44hjCx2BhMXR1sMkAL2h+89SBmKDar gaCbcRFg==; Received: from [10.9.9.211] (helo=mailfront11.runbox.com) by mailtransmit03.runbox with esmtp (Exim 4.86_2) (envelope-from ) id 1dYH34-0002Tt-VD; Thu, 20 Jul 2017 21:23:47 +0200 Received: from c-24-22-244-161.hsd1.wa.comcast.net ([24.22.244.161] helo=chinook) by mailfront11.runbox.com with esmtpsa (uid:179284 ) (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) id 1dYH2m-0006hO-9S; Thu, 20 Jul 2017 21:23:28 +0200 From: Gemini Lasswell To: npostavs@users.sourceforge.net Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> <874luhbo4l.fsf@lylat> <87lgntdswo.fsf@users.sourceforge.net> <87k23d9gvt.fsf@lylat> <87d195dmr0.fsf@users.sourceforge.net> <87lgns7y7g.fsf@lylat> <877ezbew3d.fsf@users.sourceforge.net> <87d193ljdp.fsf@lylat> <87shhxcqit.fsf@users.sourceforge.net> <87inithwi5.fsf@lylat> <87eftcglz3.fsf@runbox.com> <87y3rk9gfa.fsf@users.sourceforge.net> Date: Thu, 20 Jul 2017 12:23:25 -0700 In-Reply-To: <87y3rk9gfa.fsf@users.sourceforge.net> (npostavs@users.sourceforge.net's message of "Wed, 19 Jul 2017 19:04:57 -0400") Message-ID: <87bmoehpzm.fsf@runbox.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.1 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 24402 Cc: Tino Calancha , Alex , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.7 (/) --=-=-= Content-Type: text/plain npostavs@users.sourceforge.net writes: > The following appears to fix it, though perhaps we should use a smarter > equal function that would consider the circular lists to actually be > equal instead of bailing out and returning nil on circularity. Thanks for tracking this one down and making a patch, which looks good to me. Does that smarter equal function already exist? It doesn't seem worth the effort of writing one for this purpose only. Here is a test to add to your patch, which fails without it and passes with it: --=-=-= Content-Type: text/plain Content-Disposition: inline; filename=0001-Add-a-test-of-handling-of-circular-values-to-testcov.patch >From f401e3169ff3887b8215b6625d111d70e5340efe Mon Sep 17 00:00:00 2001 From: Gemini Lasswell Date: Thu, 20 Jul 2017 12:01:42 -0700 Subject: [PATCH] Add a test of handling of circular values to testcover-tests * test/lisp/emacs-lisp-testcover-resources/testcases.el (testcover-testcase-cyc1): New function. (testcover-tests-circular-lists-bug-24402): New test. --- test/lisp/emacs-lisp/testcover-resources/testcases.el | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/test/lisp/emacs-lisp/testcover-resources/testcases.el b/test/lisp/emacs-lisp/testcover-resources/testcases.el index 1eb791a993..c9a5a6daac 100644 --- a/test/lisp/emacs-lisp/testcover-resources/testcases.el +++ b/test/lisp/emacs-lisp/testcover-resources/testcases.el @@ -490,4 +490,14 @@ testcover-testcase-how-do-i-know-you (should (eq (testcover-testcase-how-do-i-know-you "Liz") 'unknown)) +;; ==== circular-lists-bug-24402 ==== +"Testcover captures and ignores circular list errors." +;; ==== +(defun testcover-testcase-cyc1 (a) + (let ((ls (make-list 10 a%%%))) + (nconc ls ls) + ls)) +(testcover-testcase-cyc1 1) +(testcover-testcase-cyc1 1) + ;; testcases.el ends here. -- 2.12.2 --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Mon Aug 07 21:13:48 2017 Received: (at 24402) by debbugs.gnu.org; 8 Aug 2017 01:13:48 +0000 Received: from localhost ([127.0.0.1]:50907 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1det5g-0007lY-3N for submit@debbugs.gnu.org; Mon, 07 Aug 2017 21:13:48 -0400 Received: from mail-it0-f68.google.com ([209.85.214.68]:38171) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1det5c-0007lA-E9; Mon, 07 Aug 2017 21:13:44 -0400 Received: by mail-it0-f68.google.com with SMTP id f16so1511199itb.5; Mon, 07 Aug 2017 18:13:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=XGBs8tMgz/VjskFa0fp4I6jm5swU9vC+wuF30b+TGQo=; b=rc8nWGmCz5ikSPivan4+E3Pf6hEBhBtLRPoXS1kYfN4CjK7uhrsrO7Kgetdu2AsnjQ Ew2l3BsDkd20cI1CtTe77wQ1JYC9GNb21PRnbH7Vnx5Cs5c81ZBWRLpBkJ8dKiPc3WaP DPBqADWHPftVe6fUIueyw5vfAXyugnhjRlA9AsoSLptLkEqWh+EaabrxeR09QyO+7kBb vow5r3KAHZj1V63uO8HQCjWtM4TTFg0yhbroDWgW1bDlh73xZalmpTPXvX2GGH9EZt6x Dq3WKGFeN2cOdgk6ph0rt68k08pWcDIs8fsIz/AszFOvytocyQn5Ffc/XpF89wbmT3KH 8Qow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:references:date :in-reply-to:message-id:user-agent:mime-version; bh=XGBs8tMgz/VjskFa0fp4I6jm5swU9vC+wuF30b+TGQo=; b=ZEHm2vcJLPQVmZsUDWSqPcMNzL/1SE84tx18kR2GVvjlHBinEkMtjXmKooshmMOP3r uKPVN/utBOYHxk55nqAdEeRTLaOIwRQ/KrR/ucZvCF3j5pwSeMo83Lz+VbgAR2v8DDc/ VgtqxMv+/XWWxe+4w4fvOSNlsn3xVJ+6JZpNMvncwqqGc2iWzF9dccX4CqzDpzfSokpK M3fjOR+GiZb98XE1LHv6q/O29YCKUs8aFgFJ42oqPCl5AT4mdiJDFLHbXseyJuQYDevA 7rhJkPRvhUyNSspC/Hc80NRD3HN2J+I3CBBNWArY7O+TRJWk+tQv+FnlUlYoFma25o/M 1ahQ== X-Gm-Message-State: AIVw113qv7OIGV5w83eIa9kTJ/MlQ/ZQDHOTSeLqb1JVrWx8haJf2yBh wz5sAYeGcRPyfJNX X-Received: by 10.36.121.207 with SMTP id z198mr2806191itc.83.1502154818673; Mon, 07 Aug 2017 18:13:38 -0700 (PDT) Received: from zony ([45.2.119.49]) by smtp.googlemail.com with ESMTPSA id y28sm3709973ita.2.2017.08.07.18.13.36 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 07 Aug 2017 18:13:37 -0700 (PDT) From: npostavs@users.sourceforge.net To: Gemini Lasswell Subject: Re: bug#24402: should-error doesn't catch all errors References: <3654D8E9-D3CB-402B-922F-B132C1871E9F@runbox.com> <596E65D2-E780-43A1-A75B-603B61B6F9F4@runbox.com> <87zickhoco.fsf_-_@lylat> <877eznda7v.fsf@lylat> <874lur0zki.fsf@calancha-pc> <87o9sywtbz.fsf@lylat> <87fue3f9p8.fsf@users.sourceforge.net> <87vamyl3j3.fsf@lylat> <87tw2het1b.fsf@users.sourceforge.net> <874luhbo4l.fsf@lylat> <87lgntdswo.fsf@users.sourceforge.net> <87k23d9gvt.fsf@lylat> <87d195dmr0.fsf@users.sourceforge.net> <87lgns7y7g.fsf@lylat> <877ezbew3d.fsf@users.sourceforge.net> <87d193ljdp.fsf@lylat> <87shhxcqit.fsf@users.sourceforge.net> <87inithwi5.fsf@lylat> <87eftcglz3.fsf@runbox.com> <87y3rk9gfa.fsf@users.sourceforge.net> <87bmoehpzm.fsf@runbox.com> Date: Mon, 07 Aug 2017 21:15:14 -0400 In-Reply-To: <87bmoehpzm.fsf@runbox.com> (Gemini Lasswell's message of "Thu, 20 Jul 2017 12:23:25 -0700") Message-ID: <87efsm507x.fsf@users.sourceforge.net> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.2.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: 0.7 (/) X-Debbugs-Envelope-To: 24402 Cc: Tino Calancha , Alex , 24402@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.7 (/) tags 24402 fixed close 24402 26.1 quit Sorry for the delay, I was moving apartment, and then I got a bit distracted by my 'grep --null' mess. Gemini Lasswell writes: > npostavs@users.sourceforge.net writes: > >> The following appears to fix it, though perhaps we should use a smarter >> equal function that would consider the circular lists to actually be >> equal instead of bailing out and returning nil on circularity. > > Thanks for tracking this one down and making a patch, which looks good > to me. Does that smarter equal function already exist? It doesn't seem > worth the effort of writing one for this purpose only. Yeah, I left it as is with a TODO. We can fix it later if it turns out to cause an actual problem. > Here is a test to add to your patch, which fails without it and passes > with it: Thanks, I pushed it to master along with the others. [1: 054c198c12]: 2017-08-07 18:43:54 -0400 Catch argument and macroexpansion errors in ert http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=054c198c120c1f01a8ff753892d52710b740acc6 [2: 95a04fd26c]: 2017-08-07 18:43:55 -0400 ; Avoid test failures when running from compiled test files http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=95a04fd26c91e6c6c9191a629d26886f136e30fc [3: 0508045ed7]: 2017-08-07 18:54:44 -0400 Don't error on circular values in testcover http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=0508045ed7159bce5b5ea3b5fb72cf78b8b4ee8e [4: 00f7e31110]: 2017-08-07 18:54:48 -0400 Add a test of handling of circular values to testcover-tests http://git.savannah.gnu.org/cgit/emacs.git/commit/?id=00f7e31110a27e568529192d7441d9631b9096bc From unknown Fri Aug 15 16:56:31 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Tue, 05 Sep 2017 11:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator From debbugs-submit-bounces@debbugs.gnu.org Thu Mar 08 09:39:33 2018 Received: (at control) by debbugs.gnu.org; 8 Mar 2018 14:39:33 +0000 Received: from localhost ([127.0.0.1]:50382 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1etwhg-0007CL-On for submit@debbugs.gnu.org; Thu, 08 Mar 2018 09:39:33 -0500 Received: from cloud103.planethippo.com ([78.129.138.110]:56946) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1etwhe-0007C4-VB for control@debbugs.gnu.org; Thu, 08 Mar 2018 09:39:31 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=russet.org.uk; s=default; h=Content-Type:MIME-Version:Message-ID:Date: Subject:To:From:Sender:Reply-To:Cc:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=YwS4rojbZrasdudcSaWYoYboO1R+FR3EUkngZp6Fhzs=; b=igTpvnBayVYuUWid6T1XBfEuyU 3xDPWll5T9qocmGtpQ1+MaG1gfxJiHX9ZYI9D6dfkkj68/M45DQGkZqRjGYv7sGEaig3G62E9cJUb iw1cfRtrGeMIoY5m1IDGNBCEK/NN/dSy8qLb0KF2V9KihWxE8/guROWC/YA/RE1lxwyppUNtQVjIE tXghpTqMPzKdBYyj61k3p2RZ4p/5QBneGkPkqFWmAtiRLii+aQc0BGzI26N4N1rSWmHODZDHCGsgr 4jqn2xP7NmI2o4qtxx55J5VQ57m2Thapz/UsaR6rsmfZ2IDSlBNMkQq3sN6VFxhdXvfYmXBZKJyu4 nAH0Z8PQ==; Received: from cpc142652-benw12-2-0-cust953.16-2.cable.virginm.net ([82.21.43.186]:38264 helo=russet.org.uk) by cloud103.planethippo.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.89_1) (envelope-from ) id 1etwhY-001hVl-6J for control@debbugs.gnu.org; Thu, 08 Mar 2018 14:39:24 +0000 From: phillip.lord@russet.org.uk (Phillip Lord) To: control@debbugs.gnu.org Subject: Date: Thu, 08 Mar 2018 14:39:22 +0000 Message-ID: <87h8pqpr6t.fsf@russet.org.uk> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.91 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cloud103.planethippo.com X-AntiAbuse: Original Domain - debbugs.gnu.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - russet.org.uk X-Get-Message-Sender-Via: cloud103.planethippo.com: authenticated_id: phillip.lord@russet.org.uk X-Authenticated-Sender: cloud103.planethippo.com: phillip.lord@russet.org.uk X-Source: X-Source-Args: X-Source-Dir: X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: unarchive 24402 [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [78.129.138.110 listed in list.dnswl.org] 2.0 BLANK_SUBJECT Subject is present but empty 0.0 T_DKIM_INVALID DKIM-Signature header exists but is not valid X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: unarchive 24402 [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [78.129.138.110 listed in list.dnswl.org] 0.0 TVD_SPACE_RATIO No description available. 2.0 BLANK_SUBJECT Subject is present but empty 0.0 T_DKIM_INVALID DKIM-Signature header exists but is not valid unarchive 24402 From debbugs-submit-bounces@debbugs.gnu.org Thu Mar 08 09:49:40 2018 Received: (at 24402) by debbugs.gnu.org; 8 Mar 2018 14:49:40 +0000 Received: from localhost ([127.0.0.1]:50392 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1etwrU-0000y6-Gy for submit@debbugs.gnu.org; Thu, 08 Mar 2018 09:49:40 -0500 Received: from cloud103.planethippo.com ([78.129.138.110]:58308) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1etwrS-0000xl-82; Thu, 08 Mar 2018 09:49:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=russet.org.uk; s=default; h=Content-Type:MIME-Version:References:Message-ID :Date:In-Reply-To:Subject:Cc:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=HNJMmzv7pxFy8vwGpH4XAVl0kaBu58ylJb8L5nINc38=; b=lXZ4dQFahrBt4azlziG0mf5y3 Arcxu0srurDPUpXV+/KzttW/paof9F0O5vkz0TZVLPA8tM1s8M+PR8/kMAF9YBr9vH3SMHAWboYk1 KICR8FFSngRA/OBPPU4nXCJD0Y+iaxQvDp3qcFHqJhpBdT3HNqwMJzKcgG75X4EogPvGjVd96luP5 wyiZKG1M93AFxnFq14ZFJZU6fi5MRI1l73O58rNAo3OThIZbpSJ3seRHIuq4Py7rsgaJUWy/vrIwv dv/zv2DHGowCi/LEIrFT5sWSOZuqyRkoAWce3SdwVTlFH/+EASlUD6FG7wMCiY6dZFKg6WtGZ+Uht 3gHdkpWUw==; Received: from cpc142652-benw12-2-0-cust953.16-2.cable.virginm.net ([82.21.43.186]:38534 helo=russet.org.uk) by cloud103.planethippo.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.89_1) (envelope-from ) id 1etwrL-001pyC-55; Thu, 08 Mar 2018 14:49:31 +0000 From: phillip.lord@russet.org.uk (Phillip Lord) To: 24402@debbugs.gnu.org Subject: Re: bug#30745: 26.0.91; ert should macros nest strangely In-Reply-To: <87tvtr34nt.fsf@russet.org.uk> (Phillip Lord's message of "Wed, 07 Mar 2018 22:24:38 +0000") Date: Thu, 08 Mar 2018 14:49:27 +0000 Message-ID: <877eqmpqq0.fsf@russet.org.uk> References: <874llr4lk1.fsf@russet.org.uk> <87tvtr34nt.fsf@russet.org.uk> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.0.91 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cloud103.planethippo.com X-AntiAbuse: Original Domain - debbugs.gnu.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - russet.org.uk X-Get-Message-Sender-Via: cloud103.planethippo.com: authenticated_id: phillip.lord@russet.org.uk X-Authenticated-Sender: cloud103.planethippo.com: phillip.lord@russet.org.uk X-Source: X-Source-Args: X-Source-Dir: X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 24402 Cc: 30745@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 0.0 (/) Just noting that the fix for this bug has caused a regression. From unknown Fri Aug 15 16:56:31 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Fri, 06 Apr 2018 11:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator