From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 12 04:35:52 2020 Received: (at submit) by debbugs.gnu.org; 12 Jun 2020 08:35:52 +0000 Received: from localhost ([127.0.0.1]:38887 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jjfAF-0004LL-QC for submit@debbugs.gnu.org; Fri, 12 Jun 2020 04:35:52 -0400 Received: from lists.gnu.org ([209.51.188.17]:52910) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jjfAE-0004LE-Ny for submit@debbugs.gnu.org; Fri, 12 Jun 2020 04:35:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45826) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjfAE-00047j-Jy for bug-dejagnu@gnu.org; Fri, 12 Jun 2020 04:35:50 -0400 Received: from mx2.suse.de ([195.135.220.15]:45488) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jjfAC-0003rl-Pj for bug-dejagnu@gnu.org; Fri, 12 Jun 2020 04:35:50 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 52CC4AC79 for ; Fri, 12 Jun 2020 08:35:51 +0000 (UTC) To: bug-dejagnu@gnu.org From: Tom de Vries Subject: Dejagnu's unknown proc aborts testsuite run when triggered in test-case Autocrypt: addr=tdevries@suse.de; keydata= xsBNBF0ltCcBCADDhsUnMMdEXiHFfqJdXeRvgqSEUxLCy/pHek88ALuFnPTICTwkf4g7uSR7 HvOFUoUyu8oP5mNb4VZHy3Xy8KRZGaQuaOHNhZAT1xaVo6kxjswUi3vYgGJhFMiLuIHdApoc u5f7UbV+egYVxmkvVLSqsVD4pUgHeSoAcIlm3blZ1sDKviJCwaHxDQkVmSsGXImaAU+ViJ5l CwkvyiiIifWD2SoOuFexZyZ7RUddLosgsO0npVUYbl6dEMq2a5ijGF6/rBs1m3nAoIgpXk6P TCKlSWVW6OCneTaKM5C387972qREtiArTakRQIpvDJuiR2soGfdeJ6igGA1FZjU+IsM5ABEB AAHNH1RvbSBkZSBWcmllcyA8dGRldnJpZXNAc3VzZS5kZT7CwKsEEwEIAD4WIQSsnSe5hKbL MK1mGmjuhV2rbOJEoAUCXSW0JwIbAwUJA8JnAAULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAh CRDuhV2rbOJEoBYhBKydJ7mEpsswrWYaaO6FXats4kSgc48H/Ra2lq5p3dHsrlQLqM7N68Fo eRDf3PMevXyMlrCYDGLVncQwMw3O/AkousktXKQ42DPJh65zoXB22yUt8m0g12xkLax98KFJ 5NyUloa6HflLl+wQL/uZjIdNUQaHQLw3HKwRMVi4l0/Jh/TygYG1Dtm8I4o708JS4y8GQxoQ UL0z1OM9hyM3gI2WVTTyprsBHy2EjMOu/2Xpod95pF8f90zBLajy6qXEnxlcsqreMaqmkzKn 3KTZpWRxNAS/IH3FbGQ+3RpWkNGSJpwfEMVCeyK5a1n7yt1podd1ajY5mA1jcaUmGppqx827 8TqyteNe1B/pbiUt2L/WhnTgW1NC1QDOwE0EXSW0JwEIAM99H34Bu4MKM7HDJVt864MXbx7B 1M93wVlpJ7Uq+XDFD0A0hIal028j+h6jA6bhzWto4RUfDl/9mn1StngNVFovvwtfzbamp6+W pKHZm9X5YvlIwCx131kTxCNDcF+/adRW4n8CU3pZWYmNVqhMUiPLxElA6QhXTtVBh1RkjCZQ Kmbd1szvcOfaD8s+tJABJzNZsmO2hVuFwkDrRN8Jgrh92a+yHQPd9+RybW2l7sJv26nkUH5Z 5s84P6894ebgimcprJdAkjJTgprl1nhgvptU5M9Uv85Pferoh2groQEAtRPlCGrZ2/2qVNe9 XJfSYbiyedvApWcJs5DOByTaKkcAEQEAAcLAkwQYAQgAJhYhBKydJ7mEpsswrWYaaO6FXats 4kSgBQJdJbQnAhsMBQkDwmcAACEJEO6FXats4kSgFiEErJ0nuYSmyzCtZhpo7oVdq2ziRKD3 twf7BAQBZ8TqR812zKAD7biOnWIJ0McV72PFBxmLIHp24UVe0ZogtYMxSWKLg3csh0yLVwc7 H3vldzJ9AoK3Qxp0Q6K/rDOeUy3HMqewQGcqrsRRh0NXDIQk5CgSrZslPe47qIbe3O7ik/MC q31FNIAQJPmKXX25B115MMzkSKlv4udfx7KdyxHrTSkwWZArLQiEZj5KG4cCKhIoMygPTA3U yGaIvI/BGOtHZ7bEBVUCFDFfOWJ26IOCoPnSVUvKPEOH9dv+sNy7jyBsP5QxeTqwxC/1ZtNS DUCSFQjqA6bEGwM22dP8OUY6SC94x1G81A9/xbtm9LQxKm0EiDH8KBMLfQ== Message-ID: Date: Fri, 12 Jun 2020 10:35:43 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Received-SPF: pass client-ip=195.135.220.15; envelope-from=tdevries@suse.de; helo=mx2.suse.de X-detected-operating-system: by eggs.gnu.org: First seen = 2020/06/11 23:44:17 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x (no timestamps) [generic] X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) Hi, the gdb testsuite uses the dejagnu framework. If I add a call to foobar at the end of a test-case in that testsuite, and I do a run of the testsuite, then the test-case aborts, and subsequently the testsuite run aborts, that is, no further tests are run. While it is as expected that the test-case aborts, it's not desirable that the testsuite run aborts. We've installed a hack in ${tool}_init/${tool}_finish to workaround this in the gdb testsuite ( https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=26783bce15adc0316f9167a216519cebcf1ccac3 ). It would preferable though if this problem was addressed in dejagnu, such that we can eventually remove the hack. A possible solution to this problem could be to make the exiting on error (which is done in dejagnu's version of proc unknown in framework.exp) optional. Thanks, - Tom From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 16 00:05:46 2020 Received: (at 41824) by debbugs.gnu.org; 16 Jun 2020 04:05:46 +0000 Received: from localhost ([127.0.0.1]:47911 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jl2r3-0001M2-Rc for submit@debbugs.gnu.org; Tue, 16 Jun 2020 00:05:46 -0400 Received: from mail-oi1-f180.google.com ([209.85.167.180]:34431) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jl2r1-0001Lo-E0 for 41824@debbugs.gnu.org; Tue, 16 Jun 2020 00:05:44 -0400 Received: by mail-oi1-f180.google.com with SMTP id b8so18111073oic.1 for <41824@debbugs.gnu.org>; Mon, 15 Jun 2020 21:05:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=YgAWT3lNIINJ9dR5W9I9J9PvZ8iLWCCDz6nRf4wfTSI=; b=mMoniFR+ggE8lgneSs1EdgxPLGYNYGrjbBB37QbT1HvApCBR6EVFcx2f6IZex/gI1q UcHtGIc+R7gNrYFWrPequdrup7zGp2ePrTxnfl4bGo8NQzHzUCA+jJidGSQbGbmuwL3d YQDWW1cK0nmzPKU6Z2bQeFZK9FGSD79ijnLRLje2YIWKrxeMmKALn4Lamv5l0zxEjVVX NeeVAKGxDpZNH1yzfQqaKscHHnTsy2vCLL2OW0D986kacHlsE9Fp8tysHCV0vBg7rGQw n0fSHi7zbX5Ximvo4on2Nm+8BlpPoATu0RA/87bINs9m3gsRKS7sX7+qlQRFrXjD7aCq i7Dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:reply-to:user-agent :mime-version:to:cc:subject:references:in-reply-to :content-transfer-encoding; bh=YgAWT3lNIINJ9dR5W9I9J9PvZ8iLWCCDz6nRf4wfTSI=; b=MIq7JrUKbONeC3O7LEoevT+BDAROWtg42/p0M1npOAp7k4sznjIbRhoO71i5VJyf+M Kxrlctp+vywem67CArJtIfiPMMK2mohTxvakMJoGIQ8QK91Q8n5qARY8m/Q5iV70Cwbq I8HqjSXZS6kiMODYb1+/Em6Q4z8mtd/9OtBS5me/8Q7JkN4o6Bfi2rNO4l5pziSXtfSU ABy/i4Hs93lCkgxdavVH4294Ow2cZc1ByIGxERSRkfUenPKMMwS/DuymrG4K/+ACFALX FBVDuTV+jRsYDW4ooz3vzu1NHkpbQwYlxD5N9ZTBVRnNMoJS2embd2Z+joHKbnql/zI1 OOqA== X-Gm-Message-State: AOAM532PUmESho1CukkmVTgRVmGoF/s0TNMbaTho7QsI+hQZNCVNqNs2 4y8ssmIFNGt+pp/FHfQc6Rs= X-Google-Smtp-Source: ABdhPJyPHp2BN6RMniZfYcr4MVPyN/hkAIPMG5XOYW8FDjGYyHy4aAF3EAri2Qv1BNlJ6j0uVacVsA== X-Received: by 2002:aca:1308:: with SMTP id e8mr1856181oii.119.1592280337495; Mon, 15 Jun 2020 21:05:37 -0700 (PDT) Received: from [192.168.2.42] (adsl-70-133-146-38.dsl.ablntx.sbcglobal.net. [70.133.146.38]) by smtp.gmail.com with ESMTPSA id p18sm15325oto.11.2020.06.15.21.05.35 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 15 Jun 2020 21:05:36 -0700 (PDT) Message-ID: <5EE8450D.9080609@gmail.com> Date: Mon, 15 Jun 2020 23:05:33 -0500 From: Jacob Bachmeyer User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Tom de Vries Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case References: In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 41824 Cc: 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jcb62281@gmail.com Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) Tom de Vries wrote: > Hi, > > the gdb testsuite uses the dejagnu framework. > Apologies for the delay in responding; I took a few days to think about this issue because it requires a careful balance, as silently continuing after aborting a testcase could cause errors to go unnoticed in large testsuites. > If I add a call to foobar at the end of a test-case in that testsuite, > and I do a run of the testsuite, then the test-case aborts, and > subsequently the testsuite run aborts, that is, no further tests are run. > > While it is as expected that the test-case aborts, it's not desirable > that the testsuite run aborts. > The main problem I have with this is that calling an unknown procedure is presumably a very serious error. Aborting the entire run is a good way to ensure that the problem is noticed, but perhaps with a testsuite as large (and long-running) as the GDB testsuite has become, continuing may also be reasonable to salvage at least some results. Where in the GDB testsuite is this a problem? When is it appropriate to continue after such a serious error? How to ensure that the error is not buried in the logs amidst the normal output from other tests and is actually noticed? > We've installed a hack in ${tool}_init/${tool}_finish to workaround this > in the gdb testsuite ( > https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=26783bce15adc0316f9167a216519cebcf1ccac3 > ). > > It would preferable though if this problem was addressed in dejagnu, > such that we can eventually remove the hack. > That hack (if not reverted entirely) needs to be made conditional on the actual existence of ::tcl_unknown as soon as possible -- the existence of ::tcl_unknown (kept to allow Tcl autoloading to work) is very much an internal implementation detail, and it *will* be moved out of the global namespace and eventually *will* cease to exist entirely in the interpreters that run testcases. Really, tcl_unknown is not supposed to be there and relying on it introduces a latent bug into the GDB testsuite. I am sorry, and I will seek to work with you to fix these problems with a minimum of breakage, but I have to put my foot down on this: I *cannot* treat every internal symbol as long-term stable API. That hack will *not* be supported long-term. Any GDB releases that include it *will* break with some as-yet-undetermined future DejaGnu release. I have to draw a line somewhere or I may as well declare DejaGnu unmaintained and frozen at 1.6.2 forever. I have no inclination to do the latter. > A possible solution to this problem could be to make the exiting on > error (which is done in dejagnu's version of proc unknown in > framework.exp) optional. > This is obviously a good solution, but leads to the obvious question of how (command-line option? (Make normally aborts on error, but has a --keep-going option.) configuration variable?) that should be determined, or if exit-on-unknown-procedure should ever be a default. Could simply reporting an UNRESOLVED test (indicating that the testcase script aborted due to calling an undefined command) be a better option? It would show up in the summary report at the end. At first, I was planning to schedule this for the 1.7 series, but I checked and runtest.exp:runtest already catches Tcl errors, so fixing this does not require a significant change to the normal control flow. Would simply reporting an "UNRESOLVED: testcase $file aborted on unknown command '$what'" result, then propagating the Tcl error in the ::unknown proc be a workable solution? -- Jacob From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 16 11:54:03 2020 Received: (at 41824) by debbugs.gnu.org; 16 Jun 2020 15:54:03 +0000 Received: from localhost ([127.0.0.1]:49462 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jlDuU-0004N3-V9 for submit@debbugs.gnu.org; Tue, 16 Jun 2020 11:54:03 -0400 Received: from mx2.suse.de ([195.135.220.15]:44934) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jlDuS-0004MZ-KD for 41824@debbugs.gnu.org; Tue, 16 Jun 2020 11:54:01 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id DD204ADF7; Tue, 16 Jun 2020 15:53:58 +0000 (UTC) Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case To: jcb62281@gmail.com References: <5EE8450D.9080609@gmail.com> From: Tom de Vries Autocrypt: addr=tdevries@suse.de; keydata= xsBNBF0ltCcBCADDhsUnMMdEXiHFfqJdXeRvgqSEUxLCy/pHek88ALuFnPTICTwkf4g7uSR7 HvOFUoUyu8oP5mNb4VZHy3Xy8KRZGaQuaOHNhZAT1xaVo6kxjswUi3vYgGJhFMiLuIHdApoc u5f7UbV+egYVxmkvVLSqsVD4pUgHeSoAcIlm3blZ1sDKviJCwaHxDQkVmSsGXImaAU+ViJ5l CwkvyiiIifWD2SoOuFexZyZ7RUddLosgsO0npVUYbl6dEMq2a5ijGF6/rBs1m3nAoIgpXk6P TCKlSWVW6OCneTaKM5C387972qREtiArTakRQIpvDJuiR2soGfdeJ6igGA1FZjU+IsM5ABEB AAHNH1RvbSBkZSBWcmllcyA8dGRldnJpZXNAc3VzZS5kZT7CwKsEEwEIAD4WIQSsnSe5hKbL MK1mGmjuhV2rbOJEoAUCXSW0JwIbAwUJA8JnAAULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAh CRDuhV2rbOJEoBYhBKydJ7mEpsswrWYaaO6FXats4kSgc48H/Ra2lq5p3dHsrlQLqM7N68Fo eRDf3PMevXyMlrCYDGLVncQwMw3O/AkousktXKQ42DPJh65zoXB22yUt8m0g12xkLax98KFJ 5NyUloa6HflLl+wQL/uZjIdNUQaHQLw3HKwRMVi4l0/Jh/TygYG1Dtm8I4o708JS4y8GQxoQ UL0z1OM9hyM3gI2WVTTyprsBHy2EjMOu/2Xpod95pF8f90zBLajy6qXEnxlcsqreMaqmkzKn 3KTZpWRxNAS/IH3FbGQ+3RpWkNGSJpwfEMVCeyK5a1n7yt1podd1ajY5mA1jcaUmGppqx827 8TqyteNe1B/pbiUt2L/WhnTgW1NC1QDOwE0EXSW0JwEIAM99H34Bu4MKM7HDJVt864MXbx7B 1M93wVlpJ7Uq+XDFD0A0hIal028j+h6jA6bhzWto4RUfDl/9mn1StngNVFovvwtfzbamp6+W pKHZm9X5YvlIwCx131kTxCNDcF+/adRW4n8CU3pZWYmNVqhMUiPLxElA6QhXTtVBh1RkjCZQ Kmbd1szvcOfaD8s+tJABJzNZsmO2hVuFwkDrRN8Jgrh92a+yHQPd9+RybW2l7sJv26nkUH5Z 5s84P6894ebgimcprJdAkjJTgprl1nhgvptU5M9Uv85Pferoh2groQEAtRPlCGrZ2/2qVNe9 XJfSYbiyedvApWcJs5DOByTaKkcAEQEAAcLAkwQYAQgAJhYhBKydJ7mEpsswrWYaaO6FXats 4kSgBQJdJbQnAhsMBQkDwmcAACEJEO6FXats4kSgFiEErJ0nuYSmyzCtZhpo7oVdq2ziRKD3 twf7BAQBZ8TqR812zKAD7biOnWIJ0McV72PFBxmLIHp24UVe0ZogtYMxSWKLg3csh0yLVwc7 H3vldzJ9AoK3Qxp0Q6K/rDOeUy3HMqewQGcqrsRRh0NXDIQk5CgSrZslPe47qIbe3O7ik/MC q31FNIAQJPmKXX25B115MMzkSKlv4udfx7KdyxHrTSkwWZArLQiEZj5KG4cCKhIoMygPTA3U yGaIvI/BGOtHZ7bEBVUCFDFfOWJ26IOCoPnSVUvKPEOH9dv+sNy7jyBsP5QxeTqwxC/1ZtNS DUCSFQjqA6bEGwM22dP8OUY6SC94x1G81A9/xbtm9LQxKm0EiDH8KBMLfQ== Message-ID: Date: Tue, 16 Jun 2020 17:53:54 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <5EE8450D.9080609@gmail.com> Content-Type: multipart/mixed; boundary="------------F860BC22FFD7297290218B0F" Content-Language: en-US X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 41824 Cc: 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) This is a multi-part message in MIME format. --------------F860BC22FFD7297290218B0F Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit On 6/16/20 6:05 AM, Jacob Bachmeyer wrote: > Tom de Vries wrote: >> Hi, >> >> the gdb testsuite uses the dejagnu framework. >>   > > Apologies for the delay in responding; I took a few days to think about > this issue because it requires a careful balance, as silently continuing > after aborting a testcase could cause errors to go unnoticed in large > testsuites. > >> If I add a call to foobar at the end of a test-case in that testsuite, >> and I do a run of the testsuite, then the test-case aborts, and >> subsequently the testsuite run aborts, that is, no further tests are run. >> >> While it is as expected that the test-case aborts, it's not desirable >> that the testsuite run aborts. >>   > > The main problem I have with this is that calling an unknown procedure > is presumably a very serious error.  Aborting the entire run is a good > way to ensure that the problem is noticed, but perhaps with a testsuite > as large (and long-running) as the GDB testsuite has become, continuing > may also be reasonable to salvage at least some results. > > Where in the GDB testsuite is this a problem?  When is it appropriate to > continue after such a serious error? It's not a problem in a specific test-case, but a generic problem. If somebody makes a typo in a test-case and checks it in, there's no need to abort the test run. > How to ensure that the error is > not buried in the logs amidst the normal output from other tests and is > actually noticed? > I monitor ERROR and WARNING in my test runs, so I didn't think of this, but indeed, those are not tracked in the summary, so that could be a problem, agreed. >> We've installed a hack in ${tool}_init/${tool}_finish to workaround this >> in the gdb testsuite ( >> https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=26783bce15adc0316f9167a216519cebcf1ccac3 >> >> ). >> >> It would preferable though if this problem was addressed in dejagnu, >> such that we can eventually remove the hack. >>   > > That hack (if not reverted entirely) needs to be made conditional on the > actual existence of ::tcl_unknown as soon as possible -- the existence > of ::tcl_unknown (kept to allow Tcl autoloading to work) is very much an > internal implementation detail, and it *will* be moved out of the global > namespace and eventually *will* cease to exist entirely in the > interpreters that run testcases.  Really, tcl_unknown is not supposed to > be there and relying on it introduces a latent bug into the GDB testsuite. > > I am sorry, and I will seek to work with you to fix these problems with > a minimum of breakage, but I have to put my foot down on this:  I > *cannot* treat every internal symbol as long-term stable API.  That hack > will *not* be supported long-term.  Any GDB releases that include it > *will* break with some as-yet-undetermined future DejaGnu release.  I > have to draw a line somewhere or I may as well declare DejaGnu > unmaintained and frozen at 1.6.2 forever.  I have no inclination to do > the latter. > Ack, makes sense. >> A possible solution to this problem could be to make the exiting on >> error (which is done in dejagnu's version of proc unknown in >> framework.exp) optional. >>   > > This is obviously a good solution, but leads to the obvious question of > how (command-line option?  (Make normally aborts on error, but has a > --keep-going option.)  configuration variable?) that should be > determined, or if exit-on-unknown-procedure should ever be a default.  > Could simply reporting an UNRESOLVED test (indicating that the testcase > script aborted due to calling an undefined command) be a better option?  > It would show up in the summary report at the end. > > At first, I was planning to schedule this for the 1.7 series, but I > checked and runtest.exp:runtest already catches Tcl errors, so fixing > this does not require a significant change to the normal control flow.  > Would simply reporting an "UNRESOLVED:  testcase $file aborted on > unknown command '$what'" result, then propagating the Tcl error in the > ::unknown proc be a workable solution? I think so, yes. I've tried that out in attached gdb patch, which also solves the reliance on ::tcl_unknown. Then when running into the unknown command in the test-case, we have in gdb.sum: ... Running /home/vries/gdb_versions/devel/src/gdb/testsuite/gdb.ada/O2_float_param.exp ... PASS: gdb.ada/O2_float_param.exp: compilation foo.adb PASS: gdb.ada/O2_float_param.exp: frame UNRESOLVED: gdb.ada/O2_float_param.exp: testcase aborted on unknown command 'foo' ERROR: tcl error sourcing /home/vries/gdb_versions/devel/src/gdb/testsuite/gdb.ada/O2_float_param.exp. ERROR: invalid command name "foo" while executing "::gdb_unknown foo" ("uplevel" body line 1) invoked from within "uplevel 1 ::gdb_unknown $args" (procedure "::unknown" line 4) invoked from within "foo" (file "/home/vries/gdb_versions/devel/src/gdb/testsuite/gdb.ada/O2_float_param.exp" line 33) invoked from within "source /home/vries/gdb_versions/devel/src/gdb/testsuite/gdb.ada/O2_float_param.exp" ("uplevel" body line 1) invoked from within "uplevel #0 source /home/vries/gdb_versions/devel/src/gdb/testsuite/gdb.ada/O2_float_param.exp" invoked from within "catch "uplevel #0 source $test_file_name"" === gdb Summary === # of expected passes 2 # of unresolved testcases 1 ... I get similar results if I disable the hack in the gdb testsuite, and use attached runtest.exp demonstrator patch. Note btw that in both cases I didn't put $file in the unresolved message, because that's already present in pf_prefix, but that might not always be true. Thanks, - Tom --------------F860BC22FFD7297290218B0F Content-Type: text/x-patch; charset=UTF-8; name="0001-fix.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="0001-fix.patch" fix --- gdb/testsuite/lib/gdb.exp | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/gdb/testsuite/lib/gdb.exp b/gdb/testsuite/lib/gdb.exp index f502eb157d..5cb9e7907c 100644 --- a/gdb/testsuite/lib/gdb.exp +++ b/gdb/testsuite/lib/gdb.exp @@ -5177,6 +5177,13 @@ proc gdb_cleanup_globals {} { } } +# Create gdb_unknown, a copy tcl's ::unknown. +set temp [interp create] +set old_args [interp eval $temp "info args ::unknown"] +set old_body [interp eval $temp "info body ::unknown"] +interp delete $temp +eval proc gdb_unknown {$old_args} {$old_body} + proc gdb_init { test_file_name } { # Reset the timeout value to the default. This way, any testcase # that changes the timeout value without resetting it cannot affect @@ -5285,12 +5292,13 @@ proc gdb_init { test_file_name } { # Dejagnu overrides proc unknown. The dejagnu version may trigger in a # test-case but abort the entire test run. To fix this, we install a - # local version here, which reverts dejagnu's override, and restore - # dejagnu's version in gdb_finish. + # local version here, which instead calls unresolved, and then propagates the + # error to tcl's unknown. In gdb_finish, we restore dejagnu's version. rename ::unknown ::dejagnu_unknown proc unknown { args } { - # Dejagnu saves the original version in ::tcl_unknown, use it. - return [uplevel 1 ::tcl_unknown $args] + # Restore ::unknown. + unresolved "testcase aborted on unknown command '$args'" + return [uplevel 1 ::gdb_unknown $args] } return $res --------------F860BC22FFD7297290218B0F Content-Type: text/x-patch; charset=UTF-8; name="runtest-exp.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="runtest-exp.patch" --- runtest.exp.orig 2020-06-16 17:36:15.038208916 +0200 +++ runtest.exp 2020-06-16 17:38:19.189251033 +0200 @@ -1431,6 +1431,11 @@ return $found } +proc test_case_unknown { args } { + unresolved "testcase aborted on unknown command '$args'" + return [uplevel 1 ::tcl_unknown $args] +} + # # Source the testcase in TEST_FILE_NAME. # @@ -1457,6 +1462,9 @@ } } + rename ::unknown dejagnu_unknown + rename ::test_case_unknown ::unknown + if { [catch "uplevel #0 source $test_file_name"] == 1 } { # If we have a Tcl error, propagate the exit status so # that 'make' (if it invokes runtest) notices the error. @@ -1476,6 +1484,9 @@ } } + rename ::unknown ::test_case_unknown + rename dejagnu_unknown ::unknown + if {[info exists tool]} { if { [info procs "${tool}_finish"] != "" } { ${tool}_finish --------------F860BC22FFD7297290218B0F-- From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 16 22:26:06 2020 Received: (at control) by debbugs.gnu.org; 17 Jun 2020 02:26:07 +0000 Received: from localhost ([127.0.0.1]:49991 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jlNmA-0001ew-Od for submit@debbugs.gnu.org; Tue, 16 Jun 2020 22:26:06 -0400 Received: from mail-ot1-f47.google.com ([209.85.210.47]:36862) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jlNm6-0001eO-PP for control@debbugs.gnu.org; Tue, 16 Jun 2020 22:26:05 -0400 Received: by mail-ot1-f47.google.com with SMTP id 97so426025otg.3 for ; Tue, 16 Jun 2020 19:26:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:reply-to:user-agent:mime-version:to:subject :content-transfer-encoding; bh=z5zCvEbVOcZbNBi1o05No8/O3c2Y4s3lWA2PmR5Ts30=; b=pF2EDzJSrjY6y3Io4VtWLu3byLuH8Pde57f+LoW6HlUtQlEQnz29q11U0TKbGhjLkP 87z6SK1CXZ7EJ7COHSyHLbtpwFY8RL79vKrqHioBXqSPjIzQB2WQzCfyZG6mzAlywMc3 WF0H3x+Fy1ysx0vv8020Qwfw6zVT+DX2jcT1bRK6LKSqPSKlkW+dogvLh3ERG15WsPd5 fnLSnKbDYZ2ISX162Xz752yeajjq0WQPWK4SJjVb1/rLbj2dgZ/NP3eipjkY+DEv0Rba 26ij6aWJnKDYOEq9DgauiRHGfigiv9e6CFDuxNezSxiut9RmrU5PmeHq4AOLmgbjkeGe D2TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:reply-to:user-agent :mime-version:to:subject:content-transfer-encoding; bh=z5zCvEbVOcZbNBi1o05No8/O3c2Y4s3lWA2PmR5Ts30=; b=hOnUb+4IwM9c6fL6IeSUqUd5ondGeywbGKWhRKVfifmqmi6OoAQGYnhg1jNkH/7MjE Dt3KWlZz9+/kr+p6HvrLpcvyKtixLYNXXNYvdu8HshzIU9TiKt3JeYJG3XePd231g5u0 bW18BCjWeKSuAZUlfPKRlFzgSxGChDyZc3msXvVMrIcu+uWHN7pWXCVrD/B7xUiHVBw4 UZPywFj6d2OkD3qXAjCGoW4UNQ1mbxo2+rDy+yjbyVdy3hOi1ff4Y16UqFdq6gmbK2JH LlZgAuMGqHHJ+KI/5gGD4BpaTvnofjgt38zea3cDfup6nq6qGeDpVaklgATLTvD8XeXZ arFQ== X-Gm-Message-State: AOAM533lIP8eS1zYKFt7B4zbsYJz/en8+Ku/kafTFxCKUWUw12dMumLB VJDiz1S7DabYfh12wQdX3PDbExkj X-Google-Smtp-Source: ABdhPJxc/DGwhP4gZsdeClOtG67HPOuwMRSAH9pOjc/AzpFYUOp5MyZ2yo3g3BFx7J9AbiM5ahMwUg== X-Received: by 2002:a05:6830:18c8:: with SMTP id v8mr5132310ote.119.1592360756836; Tue, 16 Jun 2020 19:25:56 -0700 (PDT) Received: from [192.168.2.42] (adsl-70-133-145-204.dsl.ablntx.sbcglobal.net. [70.133.145.204]) by smtp.gmail.com with ESMTPSA id s20sm4497470otd.62.2020.06.16.19.25.55 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 16 Jun 2020 19:25:56 -0700 (PDT) Message-ID: <5EE97F32.4010802@gmail.com> Date: Tue, 16 Jun 2020 21:25:54 -0500 From: Jacob Bachmeyer User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: GNU bug tracker automated control server Subject: details on old bugs; handle #41823, #41824 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jcb62281@gmail.com Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) package dejagnu send-detail 36293 send-detail 35069 send-detail 41295 send-detail 41296 send-detail 41561 send-detail 41585 send-detail 41587 send-detail 41647 owner 41823 ! owner 41824 ! thanks From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 16 23:26:36 2020 Received: (at 41824) by debbugs.gnu.org; 17 Jun 2020 03:26:36 +0000 Received: from localhost ([127.0.0.1]:50100 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jlOii-0005Jp-1t for submit@debbugs.gnu.org; Tue, 16 Jun 2020 23:26:36 -0400 Received: from mail-ot1-f42.google.com ([209.85.210.42]:45648) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jlOif-0005Jb-6A for 41824@debbugs.gnu.org; Tue, 16 Jun 2020 23:26:34 -0400 Received: by mail-ot1-f42.google.com with SMTP id m2so473667otr.12 for <41824@debbugs.gnu.org>; Tue, 16 Jun 2020 20:26:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=sHn4m+otCtMGunzea1FVQiuZeqp6SArL53f9ZrUhfSI=; b=nNN6AmcYVhmUJfR2qTdHd3zMgx+WiLTr0t2JOuRZU1z1W0mdXLeEDDK+9t4FCWf4/0 i6A74+tCdAeodbifKahAgPwJaCHkNHUYV56lRs1ycQoC+nexCEHYgf3FUokINnNLLAPA lnK+PJH/HAmXLd9gWPEN5ICASd/YE9Zljjv3DsPJ5emQ1tGyTPIzskbHv26DhKfprEKV ceUXgQSHczTBfpRdIeaMmPIkVYKo9yP1AP0MZ706Dytmzx7lPYlnfjwSD6FjQvIFZ0FK 65+nig4LjtA2lMDBcupehkBL4JMUfezf08j/GvvM2NkmaW4QnyDZmWuRXwP/OVANpMU6 Nf2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:reply-to:user-agent :mime-version:to:cc:subject:references:in-reply-to :content-transfer-encoding; bh=sHn4m+otCtMGunzea1FVQiuZeqp6SArL53f9ZrUhfSI=; b=BALiDoLhK/IHe4ghDxrhKnPOt1yYnHOi3GkAF+1QjNYyQlH2pGuh13ai/dv6PVwJqK VcscpS48y2Nqqoh+mWR64Zdj1siq7L5t7ba4rEm0KTsXa0SFPCzhUawP+qq0qasVePH7 9tgkERVUko/noc7xbSA+8g2r9Ea2AvTEz9MHifgibcznr2r4HJfal9G0HaJ4Fy9FkkmQ GY/F1pmdAOVyfqPsJwBi7ytZfG449Y3yiRQTAC5Up67UTOke/kN5DxbpZk5NSEe746fG W3fkDK9n2LAwRP//cuwLi4KqxbxdL6xmNXeA55/HJ1LS5hskbvFYrsy7qooucg0QMLE1 nUvg== X-Gm-Message-State: AOAM530812Cc3jDZOE7qAEmmKWqrN0p2lSR3yzvEi+a8tuuVEix9gZww /Xor74wOGzwTNG2eV/7WK4c= X-Google-Smtp-Source: ABdhPJxu7jLV17C1cUlBJLWwWBv64SeSSJVbosTx0CeZWVqAEmaOofUIFmgia2mtXgfLL8DHhnaE7Q== X-Received: by 2002:a9d:909:: with SMTP id 9mr5243742otp.325.1592364387496; Tue, 16 Jun 2020 20:26:27 -0700 (PDT) Received: from [192.168.2.42] (adsl-70-133-145-204.dsl.ablntx.sbcglobal.net. [70.133.145.204]) by smtp.gmail.com with ESMTPSA id l9sm4560008oov.46.2020.06.16.20.26.25 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 16 Jun 2020 20:26:26 -0700 (PDT) Message-ID: <5EE98D60.2090005@gmail.com> Date: Tue, 16 Jun 2020 22:26:24 -0500 From: Jacob Bachmeyer User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Tom de Vries Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case References: <5EE8450D.9080609@gmail.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 41824 Cc: 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jcb62281@gmail.com Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) Tom de Vries wrote: > On 6/16/20 6:05 AM, Jacob Bachmeyer wrote: > >> Tom de Vries wrote: >> >>> If I add a call to foobar at the end of a test-case in that testsuite, >>> and I do a run of the testsuite, then the test-case aborts, and >>> subsequently the testsuite run aborts, that is, no further tests are run. >>> >>> While it is as expected that the test-case aborts, it's not desirable >>> that the testsuite run aborts. >>> >>> >> The main problem I have with this is that calling an unknown procedure >> is presumably a very serious error. Aborting the entire run is a good >> way to ensure that the problem is noticed, but perhaps with a testsuite >> as large (and long-running) as the GDB testsuite has become, continuing >> may also be reasonable to salvage at least some results. >> >> Where in the GDB testsuite is this a problem? When is it appropriate to >> continue after such a serious error? >> > > It's not a problem in a specific test-case, but a generic problem. If > somebody makes a typo in a test-case and checks it in, there's no need > to abort the test run. > So the root cause problem is people committing changes to the GDB testsuite without actually testing their own changes first, at all? >> How to ensure that the error is >> not buried in the logs amidst the normal output from other tests and is >> actually noticed? >> > > I monitor ERROR and WARNING in my test runs, so I didn't think of this, > but indeed, those are not tracked in the summary, so that could be a > problem, agreed. > Blowing up the entire test run is a good way to ensure that problems (like typos in testcases) are noticed, but admittedly, that does not help when people fail to even run the tests before checking in their typos and then other developers' test runs blow up. Murphy's Law says the offenders then carry on to check in more typos in other code without testing their own work there either... >>> A possible solution to this problem could be to make the exiting on >>> error (which is done in dejagnu's version of proc unknown in >>> framework.exp) optional. >>> >>> >> This is obviously a good solution, but leads to the obvious question of >> how (command-line option? (Make normally aborts on error, but has a >> --keep-going option.) configuration variable?) that should be >> determined, or if exit-on-unknown-procedure should ever be a default. >> Could simply reporting an UNRESOLVED test (indicating that the testcase >> script aborted due to calling an undefined command) be a better option? >> It would show up in the summary report at the end. >> >> At first, I was planning to schedule this for the 1.7 series, but I >> checked and runtest.exp:runtest already catches Tcl errors, so fixing >> this does not require a significant change to the normal control flow. >> Would simply reporting an "UNRESOLVED: testcase $file aborted on >> unknown command '$what'" result, then propagating the Tcl error in the >> ::unknown proc be a workable solution? >> > > I think so, yes. I've tried that out in attached gdb patch, which also > solves the reliance on ::tcl_unknown. > That patch is subtly wrong in other ways. First, it assumes that ::unknown is a procedure and not a command; there is no guarantee of this in the Tcl manual. Second, it will fail if run inside a safe interpreter where the default "unknown" does not exist. Third, it breaks Tcl's default autoloading because it adds an UNRESOLVED result regardless of the result of Tcl's autoload search. The best solution is to just revert the hacks and wait for this to be solved upstream, where it is expected to land for the 1.6.3 release. > [...] > Note btw that in both cases I didn't put $file in the unresolved > message, because that's already present in pf_prefix, but that might not > always be true. > The use of pf_prefix appears to be a feature of the GDB testsuite, so the upstream implementation of this feature will not be able to rely on it. I am hesitant to alter default behavior for 1.6.3, so this will probably be added as a "--keep-going" option to runtest. I should be able to land this for 1.6.3; it is simple enough to do and we are already getting some other new command-line options in 1.6.3 anyway that were needed to fix DejaGnu's own testsuite. As of yet, there is no (supported) way planned to set the --keep-going option from an init file, only on the actual command-line. This may change in the future; if it does, there will be some general API or just a special parse of ~/.dejagnurc looking for command-line options. These are planned for 1.7 at the earliest. -- Jacob From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 17 10:13:26 2020 Received: (at 41824) by debbugs.gnu.org; 17 Jun 2020 14:13:27 +0000 Received: from localhost ([127.0.0.1]:51944 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jlYog-0005E6-Ee for submit@debbugs.gnu.org; Wed, 17 Jun 2020 10:13:26 -0400 Received: from mx2.suse.de ([195.135.220.15]:56848) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jlYoe-0005Dt-02 for 41824@debbugs.gnu.org; Wed, 17 Jun 2020 10:13:25 -0400 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id 85703AD78; Wed, 17 Jun 2020 14:13:22 +0000 (UTC) Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case To: jcb62281@gmail.com References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> From: Tom de Vries Autocrypt: addr=tdevries@suse.de; keydata= xsBNBF0ltCcBCADDhsUnMMdEXiHFfqJdXeRvgqSEUxLCy/pHek88ALuFnPTICTwkf4g7uSR7 HvOFUoUyu8oP5mNb4VZHy3Xy8KRZGaQuaOHNhZAT1xaVo6kxjswUi3vYgGJhFMiLuIHdApoc u5f7UbV+egYVxmkvVLSqsVD4pUgHeSoAcIlm3blZ1sDKviJCwaHxDQkVmSsGXImaAU+ViJ5l CwkvyiiIifWD2SoOuFexZyZ7RUddLosgsO0npVUYbl6dEMq2a5ijGF6/rBs1m3nAoIgpXk6P TCKlSWVW6OCneTaKM5C387972qREtiArTakRQIpvDJuiR2soGfdeJ6igGA1FZjU+IsM5ABEB AAHNH1RvbSBkZSBWcmllcyA8dGRldnJpZXNAc3VzZS5kZT7CwKsEEwEIAD4WIQSsnSe5hKbL MK1mGmjuhV2rbOJEoAUCXSW0JwIbAwUJA8JnAAULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAh CRDuhV2rbOJEoBYhBKydJ7mEpsswrWYaaO6FXats4kSgc48H/Ra2lq5p3dHsrlQLqM7N68Fo eRDf3PMevXyMlrCYDGLVncQwMw3O/AkousktXKQ42DPJh65zoXB22yUt8m0g12xkLax98KFJ 5NyUloa6HflLl+wQL/uZjIdNUQaHQLw3HKwRMVi4l0/Jh/TygYG1Dtm8I4o708JS4y8GQxoQ UL0z1OM9hyM3gI2WVTTyprsBHy2EjMOu/2Xpod95pF8f90zBLajy6qXEnxlcsqreMaqmkzKn 3KTZpWRxNAS/IH3FbGQ+3RpWkNGSJpwfEMVCeyK5a1n7yt1podd1ajY5mA1jcaUmGppqx827 8TqyteNe1B/pbiUt2L/WhnTgW1NC1QDOwE0EXSW0JwEIAM99H34Bu4MKM7HDJVt864MXbx7B 1M93wVlpJ7Uq+XDFD0A0hIal028j+h6jA6bhzWto4RUfDl/9mn1StngNVFovvwtfzbamp6+W pKHZm9X5YvlIwCx131kTxCNDcF+/adRW4n8CU3pZWYmNVqhMUiPLxElA6QhXTtVBh1RkjCZQ Kmbd1szvcOfaD8s+tJABJzNZsmO2hVuFwkDrRN8Jgrh92a+yHQPd9+RybW2l7sJv26nkUH5Z 5s84P6894ebgimcprJdAkjJTgprl1nhgvptU5M9Uv85Pferoh2groQEAtRPlCGrZ2/2qVNe9 XJfSYbiyedvApWcJs5DOByTaKkcAEQEAAcLAkwQYAQgAJhYhBKydJ7mEpsswrWYaaO6FXats 4kSgBQJdJbQnAhsMBQkDwmcAACEJEO6FXats4kSgFiEErJ0nuYSmyzCtZhpo7oVdq2ziRKD3 twf7BAQBZ8TqR812zKAD7biOnWIJ0McV72PFBxmLIHp24UVe0ZogtYMxSWKLg3csh0yLVwc7 H3vldzJ9AoK3Qxp0Q6K/rDOeUy3HMqewQGcqrsRRh0NXDIQk5CgSrZslPe47qIbe3O7ik/MC q31FNIAQJPmKXX25B115MMzkSKlv4udfx7KdyxHrTSkwWZArLQiEZj5KG4cCKhIoMygPTA3U yGaIvI/BGOtHZ7bEBVUCFDFfOWJ26IOCoPnSVUvKPEOH9dv+sNy7jyBsP5QxeTqwxC/1ZtNS DUCSFQjqA6bEGwM22dP8OUY6SC94x1G81A9/xbtm9LQxKm0EiDH8KBMLfQ== Message-ID: Date: Wed, 17 Jun 2020 16:13:17 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <5EE98D60.2090005@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 41824 Cc: 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.3 (---) On 6/17/20 5:26 AM, Jacob Bachmeyer wrote: > Tom de Vries wrote: >> On 6/16/20 6:05 AM, Jacob Bachmeyer wrote: >>   >>> Tom de Vries wrote: >>>     >>>> If I add a call to foobar at the end of a test-case in that testsuite, >>>> and I do a run of the testsuite, then the test-case aborts, and >>>> subsequently the testsuite run aborts, that is, no further tests are >>>> run. >>>> >>>> While it is as expected that the test-case aborts, it's not desirable >>>> that the testsuite run aborts. >>>>         >>> The main problem I have with this is that calling an unknown procedure >>> is presumably a very serious error.  Aborting the entire run is a good >>> way to ensure that the problem is noticed, but perhaps with a testsuite >>> as large (and long-running) as the GDB testsuite has become, continuing >>> may also be reasonable to salvage at least some results. >>> >>> Where in the GDB testsuite is this a problem?  When is it appropriate to >>> continue after such a serious error? >>>     >> >> It's not a problem in a specific test-case, but a generic problem. If >> somebody makes a typo in a test-case and checks it in, there's no need >> to abort the test run. >>   > > So the root cause problem is people committing changes to the GDB > testsuite without actually testing their own changes first, at all? > >>> How to ensure that the error is >>> not buried in the logs amidst the normal output from other tests and is >>> actually noticed? >>>     >> >> I monitor ERROR and WARNING in my test runs, so I didn't think of this, >> but indeed, those are not tracked in the summary, so that could be a >> problem, agreed. >>   > > Blowing up the entire test run is a good way to ensure that problems > (like typos in testcases) are noticed, but admittedly, that does not > help when people fail to even run the tests before checking in their > typos and then other developers' test runs blow up.  Murphy's Law says > the offenders then carry on to check in more typos in other code without > testing their own work there either... > >>>> A possible solution to this problem could be to make the exiting on >>>> error (which is done in dejagnu's version of proc unknown in >>>> framework.exp) optional. >>>>         >>> This is obviously a good solution, but leads to the obvious question of >>> how (command-line option?  (Make normally aborts on error, but has a >>> --keep-going option.)  configuration variable?) that should be >>> determined, or if exit-on-unknown-procedure should ever be a default. >>> Could simply reporting an UNRESOLVED test (indicating that the testcase >>> script aborted due to calling an undefined command) be a better >>> option? It would show up in the summary report at the end. >>> >>> At first, I was planning to schedule this for the 1.7 series, but I >>> checked and runtest.exp:runtest already catches Tcl errors, so fixing >>> this does not require a significant change to the normal control >>> flow. Would simply reporting an "UNRESOLVED:  testcase $file aborted on >>> unknown command '$what'" result, then propagating the Tcl error in the >>> ::unknown proc be a workable solution? >>>     >> >> I think so, yes.  I've tried that out in attached gdb patch, which also >> solves the reliance on ::tcl_unknown. >>   > > That patch is subtly wrong in other ways.  First, it assumes that > ::unknown is a procedure and not a command; there is no guarantee of > this in the Tcl manual.  Second, it will fail if run inside a safe > interpreter where the default "unknown" does not exist.  Third, it > breaks Tcl's default autoloading because it adds an UNRESOLVED result > regardless of the result of Tcl's autoload search. > Your last remark here got me thinking about dejagnu's unknown proc, and I've filed two bugs: - bug#41914: Propagate return value of auto-loaded command - bug#41918: Propagate error value of auto-loaded command > The best solution is to just revert the hacks and wait for this to be > solved upstream, where it is expected to land for the 1.6.3 release. > >> [...] >> Note btw that in both cases I didn't put $file in the unresolved >> message, because that's already present in pf_prefix, but that might not >> always be true. >>   > The use of pf_prefix appears to be a feature of the GDB testsuite, so > the upstream implementation of this feature will not be able to rely on it. > > > I am hesitant to alter default behavior for 1.6.3, so this will probably > be added as a "--keep-going" option to runtest.>  I should be able to > land this for 1.6.3; it is simple enough to do and we are already > getting some other new command-line options in 1.6.3 anyway that were > needed to fix DejaGnu's own testsuite.  As of yet, there is no > (supported) way planned to set the --keep-going option from an init > file, only on the actual command-line.  This may change in the future; > if it does, there will be some general API or just a special parse of > ~/.dejagnurc looking for command-line options.  These are planned for > 1.7 at the earliest. I guess it would be convenient if we could switch this on somehow from lib/${tool}.exp. Thanks, - Tom From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 17 19:19:03 2020 Received: (at 41824) by debbugs.gnu.org; 17 Jun 2020 23:19:03 +0000 Received: from localhost ([127.0.0.1]:52403 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jlhKg-0001bg-Sb for submit@debbugs.gnu.org; Wed, 17 Jun 2020 19:19:03 -0400 Received: from mail-oi1-f193.google.com ([209.85.167.193]:44057) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jlhKf-0001bC-8r for 41824@debbugs.gnu.org; Wed, 17 Jun 2020 19:19:02 -0400 Received: by mail-oi1-f193.google.com with SMTP id x202so3342823oix.11 for <41824@debbugs.gnu.org>; Wed, 17 Jun 2020 16:19:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject :references:in-reply-to; bh=voWcocsPQ5nelXxQwb/sE8Z+a2QTdIYCf2fPFs/eAvM=; b=MiZG2lKx7h2J144cOwBOc4RdEgE1tOF5eFStKNEoCWQaqKnHZDoVkiU3ed0ANUx+7E 3gIG2pfZNcYANGm6HI7DZ/YQtArbnRjUGHJT4g78FWTlB625MjgCHq3K1mr4GgPaLR8m WQWmp8ZlxrXbpe9dX1m2imPsCU9TFiYV4kZckJIF9RQetSgZ1LcZlVXy3wftrvEZP86O Ndy5yd2W+KYe3Qiham+Lqp+eC/V7Pfj/puIswkvzVfk6OYS0NIYs6M4S0PhhRPluKh1G cLpR3UWxhDxDtcr2PGKN7fyl35KWWkHEhNKMCwgjl+69NFW29Md3gcsyEabCzsiMSu9P EIOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:reply-to:user-agent :mime-version:to:cc:subject:references:in-reply-to; bh=voWcocsPQ5nelXxQwb/sE8Z+a2QTdIYCf2fPFs/eAvM=; b=KhwqiGkxTJHOaBYsqZ3CAn8YuK0tVRfJcFIIUx+Qjzl4hx9GwdM8CgArbkOjW/2O5k Vkr4qVcEtLABJTidIlRBUzM4xr6vV62myy2hpq5EDKfzol4iF0ejOHR/fwHiF5PjM2lI YjFagnCJuVnnYAWlx5vY4vxnSKNNSUz67jnpyGapmP/Otl8i5yEo2DH1o7pCzapZZtpK 2q83WHtLHeC66kPev6i45N+evW6iwxRz84e8XhIlKyVADCgal50+/Y71uVA/PCiY51Ix 6QHoa0hLzY/1E+9n2jQB0hH8W2bZqvl3W/u8Rc2T6tqcucGnRof21ooRL49KvlS4FOP6 E0Jg== X-Gm-Message-State: AOAM532yHMPm+iXGklxspQDUFkCdBIdkahUMmzpsADTcQBoCfIMIPKVi +9x4Y7JIEj1BR+6xuS8cZPU= X-Google-Smtp-Source: ABdhPJz8/9eqgu1Y8c2ymmTojLZ6ziOnrP5XJHDYIkDpimQWkUMwNrj6RXo8zB8w5ixcxz9TxOQvNA== X-Received: by 2002:aca:849:: with SMTP id 70mr832652oii.153.1592435935563; Wed, 17 Jun 2020 16:18:55 -0700 (PDT) Received: from [192.168.2.42] (adsl-70-133-145-204.dsl.ablntx.sbcglobal.net. [70.133.145.204]) by smtp.gmail.com with ESMTPSA id g5sm330425otb.20.2020.06.17.16.18.53 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 17 Jun 2020 16:18:54 -0700 (PDT) Message-ID: <5EEAA4DC.8070909@gmail.com> Date: Wed, 17 Jun 2020 18:18:52 -0500 From: Jacob Bachmeyer User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Tom de Vries Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> In-Reply-To: Content-Type: multipart/mixed; boundary="------------080008000608070707030003" X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 41824 Cc: 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jcb62281@gmail.com Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) This is a multi-part message in MIME format. --------------080008000608070707030003 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Tom de Vries wrote: > On 6/17/20 5:26 AM, Jacob Bachmeyer wrote: > >> Tom de Vries wrote: >> >>> On 6/16/20 6:05 AM, Jacob Bachmeyer wrote: >>> >>> >>>> Tom de Vries wrote: >>>> >>>>> A possible solution to this problem could be to make the exiting on >>>>> error (which is done in dejagnu's version of proc unknown in >>>>> framework.exp) optional. >>>>> >>>>> >>>> This is obviously a good solution, but leads to the obvious question of >>>> how (command-line option? (Make normally aborts on error, but has a >>>> --keep-going option.) configuration variable?) that should be >>>> determined, or if exit-on-unknown-procedure should ever be a default. >>>> Could simply reporting an UNRESOLVED test (indicating that the testcase >>>> script aborted due to calling an undefined command) be a better >>>> option? It would show up in the summary report at the end. >>>> >>>> At first, I was planning to schedule this for the 1.7 series, but I >>>> checked and runtest.exp:runtest already catches Tcl errors, so fixing >>>> this does not require a significant change to the normal control >>>> flow. Would simply reporting an "UNRESOLVED: testcase $file aborted on >>>> unknown command '$what'" result, then propagating the Tcl error in the >>>> ::unknown proc be a workable solution? >>>> >>>> >>> I think so, yes. I've tried that out in attached gdb patch, which also >>> solves the reliance on ::tcl_unknown. >>> >>> >> That patch is subtly wrong in other ways. First, it assumes that >> ::unknown is a procedure and not a command; there is no guarantee of >> this in the Tcl manual. Second, it will fail if run inside a safe >> interpreter where the default "unknown" does not exist. Third, it >> breaks Tcl's default autoloading because it adds an UNRESOLVED result >> regardless of the result of Tcl's autoload search. >> > > Your last remark here got me thinking about dejagnu's unknown proc, and > I've filed two bugs: > - bug#41914: Propagate return value of auto-loaded command > - bug#41918: Propagate error value of auto-loaded command > I believe both of these were fixed while working on this issue in the attached patch. Apologies for the delay; I wanted tests for the new feature. >> The best solution is to just revert the hacks and wait for this to be >> solved upstream, where it is expected to land for the 1.6.3 release. >> >> >>> [...] >>> Note btw that in both cases I didn't put $file in the unresolved >>> message, because that's already present in pf_prefix, but that might not >>> always be true. >>> >>> >> The use of pf_prefix appears to be a feature of the GDB testsuite, so >> the upstream implementation of this feature will not be able to rely on it. >> >> >> I am hesitant to alter default behavior for 1.6.3, so this will probably >> be added as a "--keep-going" option to runtest.> I should be able to >> land this for 1.6.3; it is simple enough to do and we are already >> getting some other new command-line options in 1.6.3 anyway that were >> needed to fix DejaGnu's own testsuite. As of yet, there is no >> (supported) way planned to set the --keep-going option from an init >> file, only on the actual command-line. This may change in the future; >> if it does, there will be some general API or just a special parse of >> ~/.dejagnurc looking for command-line options. These are planned for >> 1.7 at the earliest. >> > > I guess it would be convenient if we could switch this on somehow from > lib/${tool}.exp. The problem here is that this detects an invalidity in the testsuite. Allowing testsuites to select --keep_going is asking (Murphy's Law again) for people to turn it on and then ignore the errors from broken test scripts "because it works" -- never mind that it is obviously broken, but an extra UNRESOLVED or three can still be easily ignored. The entire test run stopping cold unless a special command-line option is given that is specifically documented to suppress abort-on-error is much harder to explain away or even try to justify as acceptable. -- Jacob --------------080008000608070707030003 Content-Type: text/plain; name="0001-Allow-testing-to-continue-after-an-undefined-command.patch" Content-Transfer-Encoding: base64 Content-Disposition: inline; filename*0="0001-Allow-testing-to-continue-after-an-undefined-command.pa"; filename*1="tch" RnJvbSBjNWIyMWYxZjFjZmFhYmYxNDMxMDEwYzMxNGFhZGNjMGI3YjcwOGYwIE1vbiBTZXAg MTcgMDA6MDA6MDAgMjAwMQpGcm9tOiBKYWNvYiBCYWNobWV5ZXIgPGpjYjYyMjgxK2RldkBn bWFpbC5jb20+CkRhdGU6IFdlZCwgMTcgSnVuIDIwMjAgMTg6MDg6NTcgLTA1MDAKU3ViamVj dDogW1BBVENIXSBBbGxvdyB0ZXN0aW5nIHRvIGNvbnRpbnVlIGFmdGVyIGFuIHVuZGVmaW5l ZCBjb21tYW5kIGlzIGNhbGxlZAoKLS0tCiBDaGFuZ2VMb2cgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICB8ICAgMjYgKysrKysrKwogTWFrZWZpbGUuYW0gICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAyICstCiBNYWtlZmls ZS5pbiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB8ICAgIDIgKy0K IE5FV1MgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwg ICAgMiArCiBkb2MvZGVqYWdudS50ZXhpICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICB8ICAgIDUgKwogZG9jL3J1bnRlc3QuMSAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgfCAgICAzICsKIGxpYi9mcmFtZXdvcmsuZXhwICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgIHwgICAyNCArKysrKy0KIHJ1bnRlc3QuZXhwICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHwgICAxMiArKysKIHRlc3RzdWl0ZS9y dW50ZXN0Lm1haW4vYWJvcnQuZXhwICAgICAgICAgICAgICAgICAgIHwgICA3OCArKysrKysr KysrKysrKysrKysrKwogLi4uL2Fib3J0L3Rlc3RzdWl0ZS9hYm9ydC50ZXN0L2Fib3J0LXVu ZGVmLmV4cCAgICAgfCAgIDI1ICsrKysrKwogLi4uL2Fib3J0L3Rlc3RzdWl0ZS9hYm9ydC50 ZXN0L3NpbXBsZS5leHAgICAgICAgICAgfCAgIDIxICsrKysrCiAxMSBmaWxlcyBjaGFuZ2Vk LCAxOTUgaW5zZXJ0aW9ucygrKSwgNSBkZWxldGlvbnMoLSkKIGNyZWF0ZSBtb2RlIDEwMDY0 NCB0ZXN0c3VpdGUvcnVudGVzdC5tYWluL2Fib3J0LmV4cAogY3JlYXRlIG1vZGUgMTAwNjQ0 IHRlc3RzdWl0ZS9ydW50ZXN0Lm1haW4vYWJvcnQvdGVzdHN1aXRlL2Fib3J0LnRlc3QvYWJv cnQtdW5kZWYuZXhwCiBjcmVhdGUgbW9kZSAxMDA2NDQgdGVzdHN1aXRlL3J1bnRlc3QubWFp bi9hYm9ydC90ZXN0c3VpdGUvYWJvcnQudGVzdC9zaW1wbGUuZXhwCgpkaWZmIC0tZ2l0IGEv Q2hhbmdlTG9nIGIvQ2hhbmdlTG9nCmluZGV4IDY5YTgwZWYuLjIzNTExMDEgMTAwNjQ0Ci0t LSBhL0NoYW5nZUxvZworKysgYi9DaGFuZ2VMb2cKQEAgLTEsMyArMSwyOSBAQAorMjAyMC0w Ni0xNyAgSmFjb2IgQmFjaG1leWVyICA8amNiNjIyODErZGV2QGdtYWlsLmNvbT4KKworCVBS IDQxODI0CisKKwkqIE5FV1M6IEFkZCBpdGVtIGZvciAtLWtlZXBfZ29pbmcgb3B0aW9uLgor CisJKiBNYWtlZmlsZS5hbSAoQ0xFQU5GSUxFUyk6IEFkZCBhYm9ydC1pbml0LmV4cCB0byBs aXN0LgorCisJKiBkb2MvZGVqYWdudS50ZXhpIChJbnZva2luZyBydW50ZXN0KTogRG9jdW1l bnQgbmV3IC0ta2VlcF9nb2luZworCWNvbW1hbmQgbGluZSBvcHRpb24uCisJKiBkb2MvcnVu dGVzdC4xOiBMaWtld2lzZS4KKworCSogbGliL2ZyYW1ld29yay5leHAgKHVua25vd24pOiBS ZXBvcnQgYW4gVU5SRVNPTFZFRCByZXN1bHQgaWYgYW4KKwl1bmtub3duIGNvbW1hbmQgaXMg aW52b2tlZC4gIEF2b2lkIGV4aXRpbmcgYW5kIHByb3BhZ2F0ZSB0aGUgZXJyb3IKKwlmcm9t IFRjbCdzICJ1bmtub3duIiBwcm9jZWR1cmUgaWYgLS1rZWVwX2dvaW5nIHdhcworCXNwZWNp ZmllZC4gQnJhY2UgcHJvY2VkdXJlIGFyZ3VtZW50IGxpc3QuCisJKiBydW50ZXN0LmV4cCAo ZGVqYWdudTo6b3B0KTogTmV3IG5hbWVzcGFjZS4KKwlBZGQgb3B0aW9uIC0ta2VlcF9nb2lu ZyB0byBjb250aW51ZSBydW5uaW5nIHRlc3RzIGFmdGVyIGEgdGVzdAorCXNjcmlwdCBhYm9y dHMgZHVlIHRvIGNhbGxpbmcgYW4gdW5kZWZpbmVkIGNvbW1hbmQuCisKKwkqIHRlc3RzdWl0 ZS9ydW50ZXN0Lm1haW4vYWJvcnQuZXhwOiBOZXcgZmlsZS4KKwkqIHRlc3RzdWl0ZS9ydW50 ZXN0Lm1haW4vYWJvcnQvdGVzdHN1aXRlL2Fib3J0LnRlc3QvYWJvcnQtdW5kZWYuZXhwOgor CU5ldyBmaWxlLgorCSogdGVzdHN1aXRlL3J1bnRlc3QubWFpbi9hYm9ydC90ZXN0c3VpdGUv YWJvcnQudGVzdC9zaW1wbGUuZXhwOgorCU5ldyBmaWxlLgorCiAyMDIwLTA2LTAyICBKYWNv YiBCYWNobWV5ZXIgIDxqY2I2MjI4MStkZXZAZ21haWwuY29tPgogCiAJUFIgNDE2NDcKZGlm ZiAtLWdpdCBhL01ha2VmaWxlLmFtIGIvTWFrZWZpbGUuYW0KaW5kZXggNmM0OGMyMC4uOWEy YmE4MSAxMDA2NDQKLS0tIGEvTWFrZWZpbGUuYW0KKysrIGIvTWFrZWZpbGUuYW0KQEAgLTI2 LDcgKzI2LDcgQEAgRVhUUkFfRElTVCA9IENoYW5nZUxvZy0xOTkyIE1BSU5UQUlORVJTIGRl amFnbnUgcnVudGVzdCBcCiAJJChjb21tYW5kc19EQVRBKSAkKFRFU1RTVUlURV9GSUxFUykg JChURVhJTkZPX1RFWClcCiAJJChDT05UUklCKQogCi1DTEVBTkZJTEVTID0gb3B0aW9ucy1p bml0LmV4cCBzdGF0cy1pbml0LmV4cAorQ0xFQU5GSUxFUyA9IGFib3J0LWluaXQuZXhwIG9w dGlvbnMtaW5pdC5leHAgc3RhdHMtaW5pdC5leHAKIAogY2xlYW4tbG9jYWw6IGNsZWFuLWxv Y2FsLWNoZWNrCiAuUEhPTlk6IGNsZWFuLWxvY2FsLWNoZWNrCmRpZmYgLS1naXQgYS9NYWtl ZmlsZS5pbiBiL01ha2VmaWxlLmluCmluZGV4IDFjZDIxMWMuLmJjOWQxZTUgMTAwNjQ0Ci0t LSBhL01ha2VmaWxlLmluCisrKyBiL01ha2VmaWxlLmluCkBAIC0zODEsNyArMzgxLDcgQEAg RVhUUkFfRElTVCA9IENoYW5nZUxvZy0xOTkyIE1BSU5UQUlORVJTIGRlamFnbnUgcnVudGVz dCBcCiAJJChjb21tYW5kc19EQVRBKSAkKFRFU1RTVUlURV9GSUxFUykgJChURVhJTkZPX1RF WClcCiAJJChDT05UUklCKQogCi1DTEVBTkZJTEVTID0gb3B0aW9ucy1pbml0LmV4cCBzdGF0 cy1pbml0LmV4cAorQ0xFQU5GSUxFUyA9IGFib3J0LWluaXQuZXhwIG9wdGlvbnMtaW5pdC5l eHAgc3RhdHMtaW5pdC5leHAKIGJpbl9TQ1JJUFRTID0gZGVqYWdudSBydW50ZXN0CiBpbmNs dWRlX0hFQURFUlMgPSBkZWphZ251LmgKIHBrZ2RhdGFfREFUQSA9IFwKZGlmZiAtLWdpdCBh L05FV1MgYi9ORVdTCmluZGV4IDIwOWU5YzQuLjQzNTQ0MjIgMTAwNjQ0Ci0tLSBhL05FV1MK KysrIGIvTkVXUwpAQCAtNyw2ICs3LDggQEAgQ2hhbmdlcyBzaW5jZSAxLjYuMjoKICAgIHNo b3VsZCB1c2UgdGhpcyBwcm9jLiBUaGUgJ2lzX3JlbW90ZScgcHJvYyBpcyBkZXByZWNhdGVk LgogMi4gcnVudGVzdCBub3cgYWNjZXB0cyAtLWxvY2FsX2luaXQgYW5kIC0tZ2xvYmFsX2lu aXQgb3B0aW9ucyB0byBvdmVycmlkZQogICAgdGhlIGRlZmF1bHQgb2YgcmVhZGluZyAic2l0 ZS5leHAiLiAgU2VlIHRoZSBtYW51YWwgZm9yIGRldGFpbHMuCitYLiBydW50ZXN0IG5vdyBh Y2NlcHRzIGEgLS1rZWVwX2dvaW5nIG9wdGlvbiB0byBjb250aW51ZSB3aXRoIG90aGVyIHRl c3QKKyAgIHNjcmlwdHMgYWZ0ZXIgYSB0ZXN0IHNjcmlwdCBpbnZva2VzIGFuIHVuZGVmaW5l ZCBjb21tYW5kLgogMy4gQSB1dGlsaXR5IHByb2NlZHVyZSByZWxhdGl2ZV9maWxlbmFtZSBo YXMgYmVlbiBhZGRlZC4gIFRoaXMgcHJvY2VkdXJlCiAgICBjb21wdXRlcyBhIHJlbGF0aXZl IGZpbGUgbmFtZSB0byBhIGdpdmVuIGRlc3RpbmF0aW9uIGZyb20gYSBnaXZlbiBiYXNlLgog NC4gVGhlIHV0aWxpdHkgcHJvY2VkdXJlICdncmVwJyBub3cgYWNjZXB0cyBhICctbicgb3B0 aW9uIHRoYXQKZGlmZiAtLWdpdCBhL2RvYy9kZWphZ251LnRleGkgYi9kb2MvZGVqYWdudS50 ZXhpCmluZGV4IGJiMzg2ZTguLmNkOGM0ZTkgMTAwNjQ0Ci0tLSBhL2RvYy9kZWphZ251LnRl eGkKKysrIGIvZG9jL2RlamFnbnUudGV4aQpAQCAtNjQ0LDYgKzY0NCwxMSBAQCBUaGUgaG9z dCBib2FyZCB0byB1c2UuCiBAaXRlbSBAY29kZXstLWlnbm9yZSBbdGVzdHMocyldIH0KIFRo ZSBuYW1lKHMpIG9mIHNwZWNpZmljIHRlc3RzIHRvIGlnbm9yZS4KIAorQGl0ZW0gQGNvZGV7 LS1rZWVwX2dvaW5nfQorQ29udGludWUgdGVzdGluZyB3aGVuIHRlc3Qgc2NyaXB0cyBlbmNv dW50ZXIgcmVjb3ZlcmFibGUgZmF0YWwgZXJyb3JzLgorU3VjaCBlcnJvcnMgYWx3YXlzIGNh dXNlIHRoZSBvZmZlbmRpbmcgdGVzdCBzY3JpcHQgdG8gYWJvcnQKK2ltbWVkaWF0ZWx5LCBi dXQgRGVqYUdudSBtYXkgYmUgYWJsZSB0byBjb250aW51ZSB3aXRoIHRoZSBuZXh0IHNjcmlw dC4KKwogQGl0ZW0gQGNvZGV7LS1sb2NhbF9pbml0IFtuYW1lXX0KIFVzZSBAZW1waHtuYW1l fSBhcyB0aGUgdGVzdHN1aXRlIGxvY2FsIGluaXQgZmlsZSBpbnN0ZWFkIG9mCiBAZmlsZXtz aXRlLmV4cH0gaW4gdGhlIGN1cnJlbnQgZGlyZWN0b3J5IGFuZCBpbiBAZW1waHtvYmpkaXJ9 LiAgVGhlCmRpZmYgLS1naXQgYS9kb2MvcnVudGVzdC4xIGIvZG9jL3J1bnRlc3QuMQppbmRl eCBkMDQzZWU3Li5lODkxM2IxIDEwMDY0NAotLS0gYS9kb2MvcnVudGVzdC4xCisrKyBiL2Rv Yy9ydW50ZXN0LjEKQEAgLTQ1LDYgKzQ1LDkgQEAgVGhlIGhvc3QgYm9hcmQgZGVmaW5pdGlv biB0byB1c2UuCiAuQkkgLS1pZ25vcmUgXCB0ZXN0MS5leHBcIHRlc3QyLmV4cFwgLi4uCiBE byBub3QgcnVuIHRoZSBzcGVjaWZpZWQgdGVzdHMuCiAuVFAKKy5CIC0ta2VlcF9nb2luZwor RG8gbm90IGFib3J0IHRlc3QgcnVuIGlmIGEgdGVzdCBzY3JpcHQgZW5jb3VudGVycyBhIGZh dGFsIGVycm9yLgorLlRQCiAuQkkgLS1sb2NhbF9pbml0IFwgTkFNRQogVGhlIE5BTUUgdG8g dXNlIGZvciB0aGUgdGVzdHN1aXRlIGxvY2FsIGluaXQgZmlsZSBpbiBib3RoIHRoZSBjdXJy ZW50CiBkaXJlY3RvcnkgYW5kIG9iamRpci4KZGlmZiAtLWdpdCBhL2xpYi9mcmFtZXdvcmsu ZXhwIGIvbGliL2ZyYW1ld29yay5leHAKaW5kZXggZTZjZTE5Ny4uNTMzMzNhZCAxMDA2NDQK LS0tIGEvbGliL2ZyYW1ld29yay5leHAKKysrIGIvbGliL2ZyYW1ld29yay5leHAKQEAgLTI1 NywyMSArMjU3LDM5IEBAIHByb2MgaXNuYXRpdmUgeyB9IHsKICMgVGhpcyBhbGxvd3MgVGNs IHBhY2thZ2UgYXV0b2xvYWRpbmcgdG8gd29yayBpbiB0aGUgbW9kZXJuIGFnZS4KIAogcmVu YW1lIDo6dW5rbm93biA6OnRjbF91bmtub3duCi1wcm9jIHVua25vd24gYXJncyB7Ci0gICAg aWYge1tjYXRjaCB7dXBsZXZlbCAxIDo6dGNsX3Vua25vd24gJGFyZ3N9IG1zZ119IHsKK3By b2MgdW5rbm93biB7IGFyZ3MgfSB7CisgICAgc2V0IGNvZGUgW2NhdGNoIHt1cGxldmVsIDEg Ojp0Y2xfdW5rbm93biAkYXJnc30gbXNnXQorICAgIGlmIHsgJGNvZGUgIT0gMCB9IHsKIAln bG9iYWwgZXJyb3JDb2RlCiAJZ2xvYmFsIGVycm9ySW5mbwogCWdsb2JhbCBleGl0X3N0YXR1 cwogCisJc2V0IHJldF9jbWQgW2xpc3QgcmV0dXJuIC1jb2RlICRjb2RlXQorCiAJY2xvbmVf b3V0cHV0ICJFUlJPUjogKERlamFHbnUpIHByb2MgXCIkYXJnc1wiIGRvZXMgbm90IGV4aXN0 LiIKIAlpZiB7W2luZm8gZXhpc3RzIGVycm9yQ29kZV19IHsKKwkgICAgbGFwcGVuZCByZXRf Y21kIC1lcnJvcmNvZGUgJGVycm9yQ29kZQogCSAgICBzZW5kX2Vycm9yICJUaGUgZXJyb3Ig Y29kZSBpcyAkZXJyb3JDb2RlXG4iCiAJfQogCWlmIHtbaW5mbyBleGlzdHMgZXJyb3JJbmZv XX0geworCSAgICAjIG9taXR0aW5nIGVycm9ySW5mbyBmcm9tIHRoZSBwcm9wYWdhdGVkIGVy cm9yIG1ha2VzIHRoaXMgY29kZQorCSAgICAjIGludmlzaWJsZSB3aXRoIHRoZSBiYWNrdHJh Y2UgcG9pbnRpbmcgZGlyZWN0bHkgdG8gdGhlIHByb2JsZW0KIAkgICAgc2VuZF9lcnJvciAi VGhlIGluZm8gb24gdGhlIGVycm9yIGlzOlxuJGVycm9ySW5mb1xuIgogCX0KIAlzZXQgZXhp dF9zdGF0dXMgMgotCWxvZ19hbmRfZXhpdAorCisJc2V0IHVucmVzb2x2ZWRfbXNnICJ0ZXN0 Y2FzZSAnW3VwbGV2ZWwgaW5mbyBzY3JpcHRdJyBhYm9ydGVkIgorCWFwcGVuZCB1bnJlc29s dmVkX21zZyAiIGF0IGNhbGwgdG8gdW5rbm93biBjb21tYW5kICckYXJncyciCisJdW5yZXNv bHZlZCAkdW5yZXNvbHZlZF9tc2cKKworCWxhcHBlbmQgcmV0X2NtZCAkbXNnCisJaWYgeyAk OjpkZWphZ251OjpvcHQ6OmtlZXBfZ29pbmcgfSB7CisJICAgIGV2YWwgJHJldF9jbWQKKwl9 IGVsc2UgeworCSAgICBsb2dfYW5kX2V4aXQKKwl9CisgICAgfSBlbHNlIHsKKwlyZXR1cm4g JG1zZwogICAgIH0KIH0KIApkaWZmIC0tZ2l0IGEvcnVudGVzdC5leHAgYi9ydW50ZXN0LmV4 cAppbmRleCBiYTQ5ZTczLi4wMjhhZDViIDEwMDY0NAotLS0gYS9ydW50ZXN0LmV4cAorKysg Yi9ydW50ZXN0LmV4cApAQCAtOTcsNiArOTcsMTMgQEAgc2V0IGdsb2JhbF9pbml0X2ZpbGUJ c2l0ZS5leHAJOyMgZ2xvYmFsIGluaXQgZmlsZSBuYW1lCiBzZXQgdGVzdHN1aXRlZGlyCSJ0 ZXN0c3VpdGUiCTsjIHRvcC1sZXZlbCB0ZXN0c3VpdGUgc291cmNlIGRpcmVjdG9yeQogc2V0 IHRlc3RidWlsZGRpcgkidGVzdHN1aXRlIgk7IyB0b3AtbGV2ZWwgdGVzdHN1aXRlIG9iamVj dCBkaXJlY3RvcnkKIAorIworIyBUaGVzZSBhcmUgdXNlZCBmb3IgaW50ZXJuYWwgY29tbWFu ZC1saW5lIGZsYWdzLgorIworbmFtZXNwYWNlIGV2YWwgOjpkZWphZ251OjpvcHQgeworICAg IHZhcmlhYmxlIGtlZXBfZ29pbmcJMAk7IyBjb250aW51ZSBhZnRlciBhIGZhdGFsIGVycm9y IGluIHRlc3RjYXNlPworfQorCiAjIFZhcmlvdXMgY2NhY2hlIHZlcnNpb25zIHByb3ZpZGUg aW5jb3JyZWN0IGRlYnVnIGluZm8gc3VjaCBhcyBpZ25vcmluZwogIyBkaWZmZXJlbnQgY3Vy cmVudCBkaXJlY3RvcnksIGJyZWFraW5nIEdEQiB0ZXN0c3VpdGUuCiBzZXQgZW52KENDQUNI RV9ESVNBQkxFKSAxCkBAIC0zNzIsNiArMzc5LDcgQEAgcHJvYyB1c2FnZSB7IH0gewogICAg IHNlbmRfdXNlciAiXHQtLWhvc3QgXFt0cmlwbGV0XF1cdFRoZSBjYW5vbmljYWwgdHJpcGxl dCBvZiB0aGUgaG9zdCBtYWNoaW5lXG4iCiAgICAgc2VuZF91c2VyICJcdC0taG9zdF9ib2Fy ZCBcW25hbWVcXVx0VGhlIGhvc3QgYm9hcmQgdG8gdXNlXG4iCiAgICAgc2VuZF91c2VyICJc dC0taWdub3JlIFxbbmFtZShzKVxdXHRUaGUgbmFtZXMgb2Ygc3BlY2lmaWMgdGVzdHMgdG8g aWdub3JlXG4iCisgICAgc2VuZF91c2VyICJcdC0ta2VlcF9nb2luZ1x0XHRDb250aW51ZSB0 ZXN0aW5nIGV2ZW4gaWYgYSBzY3JpcHQgYWJvcnRzXG4iCiAgICAgc2VuZF91c2VyICJcdC0t bG9jYWxfaW5pdCBcW25hbWVcXVx0VGhlIGZpbGUgdG8gbG9hZCBmb3IgbG9jYWwgY29uZmln dXJhdGlvblxuIgogICAgIHNlbmRfdXNlciAiXHQtLWxvZ19kaWFsb2dcdFx0XEVtaXQgRXhw ZWN0IG91dHB1dCBvbiBzdGRvdXRcbiIKICAgICBzZW5kX3VzZXIgIlx0LS1tYWlsIFxbbmFt ZShzKVxdXHRXaG9tIHRvIG1haWwgdGhlIHJlc3VsdHMgdG9cbiIKQEAgLTEyMDEsNiArMTIw OSwxMCBAQCBmb3IgeyBzZXQgaSAwIH0geyAkaSA8ICRhcmdjIH0geyBpbmNyIGkgfSB7CiAJ ICAgIGNvbnRpbnVlCiAJfQogCisJIi0tayoiIHsJCQkjICgtLWtlZXBfZ29pbmcpIHJlZHVj ZSBmYXRhbCBlcnJvcnMKKwkgICAgc2V0IDo6ZGVqYWdudTo6b3B0OjprZWVwX2dvaW5nIDEK Kwl9CisKIAkiLS1tKiIgewkJCSMgKC0tbWFpbCkgbWFpbCB0aGUgb3V0cHV0CiAJICAgIHNl dCBtYWlsaW5nX2xpc3QgJG9wdGFyZwogCSAgICBzZXQgbWFpbF9sb2dzIDEKZGlmZiAtLWdp dCBhL3Rlc3RzdWl0ZS9ydW50ZXN0Lm1haW4vYWJvcnQuZXhwIGIvdGVzdHN1aXRlL3J1bnRl c3QubWFpbi9hYm9ydC5leHAKbmV3IGZpbGUgbW9kZSAxMDA2NDQKaW5kZXggMDAwMDAwMC4u YzVmNzAxNAotLS0gL2Rldi9udWxsCisrKyBiL3Rlc3RzdWl0ZS9ydW50ZXN0Lm1haW4vYWJv cnQuZXhwCkBAIC0wLDAgKzEsNzggQEAKKyMgQ29weXJpZ2h0IChDKSAxOTk1LTIwMTYsIDIw MTgsIDIwMjAgRnJlZSBTb2Z0d2FyZSBGb3VuZGF0aW9uLCBJbmMuCisjCisjIFRoaXMgZmls ZSBpcyBwYXJ0IG9mIERlamFHbnUuCisjCisjIERlamFHbnUgaXMgZnJlZSBzb2Z0d2FyZTsg eW91IGNhbiByZWRpc3RyaWJ1dGUgaXQgYW5kL29yIG1vZGlmeSBpdAorIyB1bmRlciB0aGUg dGVybXMgb2YgdGhlIEdOVSBHZW5lcmFsIFB1YmxpYyBMaWNlbnNlIGFzIHB1Ymxpc2hlZCBi eQorIyB0aGUgRnJlZSBTb2Z0d2FyZSBGb3VuZGF0aW9uOyBlaXRoZXIgdmVyc2lvbiAzIG9m IHRoZSBMaWNlbnNlLCBvcgorIyAoYXQgeW91ciBvcHRpb24pIGFueSBsYXRlciB2ZXJzaW9u LgorIworIyBEZWphR251IGlzIGRpc3RyaWJ1dGVkIGluIHRoZSBob3BlIHRoYXQgaXQgd2ls bCBiZSB1c2VmdWwsIGJ1dAorIyBXSVRIT1VUIEFOWSBXQVJSQU5UWTsgd2l0aG91dCBldmVu IHRoZSBpbXBsaWVkIHdhcnJhbnR5IG9mCisjIE1FUkNIQU5UQUJJTElUWSBvciBGSVRORVNT IEZPUiBBIFBBUlRJQ1VMQVIgUFVSUE9TRS4gIFNlZSB0aGUgR05VCisjIEdlbmVyYWwgUHVi bGljIExpY2Vuc2UgZm9yIG1vcmUgZGV0YWlscy4KKyMKKyMgWW91IHNob3VsZCBoYXZlIHJl Y2VpdmVkIGEgY29weSBvZiB0aGUgR05VIEdlbmVyYWwgUHVibGljIExpY2Vuc2UKKyMgYWxv bmcgd2l0aCBEZWphR251OyBpZiBub3QsIHdyaXRlIHRvIHRoZSBGcmVlIFNvZnR3YXJlIEZv dW5kYXRpb24sCisjIEluYy4sIDUxIEZyYW5rbGluIFN0cmVldCAtIEZpZnRoIEZsb29yLCBC b3N0b24sIE1BIDAyMTEwLTEzMDEsIFVTQS4KKworIyBUaGlzIGZpbGUgdGVzdHMgaGFuZGxp bmcgb2YgZmF0YWwgZXJyb3JzIGluIHRlc3RjYXNlcy4KKyMgVGhlIHdheSB3ZSBkbyB0aGlz IGlzIHRvIHJlY3Vyc2l2ZWx5IGludm9rZSBvdXJzZWx2ZXMgb24gYSBzbWFsbCB0ZXN0c3Vp dGUKKyMgYW5kIGFuYWx5emUgdGhlIHJlc3VsdHMuCisKK2xvYWRfbGliIHV0aWwtZGVmcy5l eHAKKworaWYgeyFbaW5mbyBleGlzdHMgdG1wZGlyXX0geworICAgIHNldCB0bXBkaXIgW3Rl c3RzdWl0ZSBmaWxlIC1vYmplY3QgLXRvcCB0bXBkaXJdCit9CisKK3NldCBmZCBbb3BlbiBh Ym9ydC1pbml0LmV4cCB3XQorcHV0cyAkZmQgInNldCBzcmNkaXIgW3Rlc3RzdWl0ZSBmaWxl IC1zb3VyY2UgLXRlc3QgYWJvcnRdIgorcHV0cyAkZmQgInNldCBvYmpkaXIgW3Rlc3RzdWl0 ZSBmaWxlIC1vYmplY3QgLXRlc3QgYWJvcnRdIgorcHV0cyAkZmQgInNldCB0bXBkaXIgJHRt cGRpciIKK2Nsb3NlICRmZAorCitpZiB7IVtmaWxlIGlzZGlyZWN0b3J5ICR0bXBkaXJdfSB7 CisgICAgY2F0Y2ggImZpbGUgbWtkaXIgJHRtcGRpciIKK30KKworaWYgeyFbZmlsZSBpc2Rp cmVjdG9yeSBbdGVzdHN1aXRlIGZpbGUgLW9iamVjdCAtdGVzdCBhYm9ydF1dfSB7CisgICAg Y2F0Y2gge2ZpbGUgbWtkaXIgW3Rlc3RzdWl0ZSBmaWxlIC1vYmplY3QgLXRlc3QgYWJvcnRd fQorfQorCitzZXQgdGVzdHMgeworICAgIHsgInJ1biBvbmx5IHNpbXBsZSB0ZXN0IgorCSJz aW1wbGUuZXhwIgorCSJQQVNTOiBzaW1wbGUgdGVzdC4qXAorCSpleHBlY3RlZCBwYXNzZXNc WyBcdFxdKzFcbiIgfQorICAgIHsgImFib3J0IG9uIHVuZGVmaW5lZCBjb21tYW5kIgorCSJh Ym9ydC11bmRlZi5leHAiCisJIlBBU1M6IHJ1bm5pbmcgYWJvcnQtdW5kZWYuZXhwLipcCisJ KlVOUkVTT0xWRUQ6IC4qIGFib3J0ZWQgYXQgY2FsbCB0byB1bmtub3duIGNvbW1hbmQuKlwK KwkqZXhwZWN0ZWQgcGFzc2VzXFsgXHRcXSsxXG4uKnVucmVzb2x2ZWQgdGVzdGNhc2VzXFsg XHRcXSsxXG4iIH0KKyAgICB7ICJzdG9wIGF0IGFib3J0IHdpdGhvdXQgLS1rZWVwX2dvaW5n IgorCSJhYm9ydC11bmRlZi5leHAgc2ltcGxlLmV4cCIKKwkiUEFTUzogcnVubmluZyBhYm9y dC11bmRlZi5leHAuKlwKKwkqVU5SRVNPTFZFRDogLiogYWJvcnRlZCBhdCBjYWxsIHRvIHVu a25vd24gY29tbWFuZC4qXAorCSpleHBlY3RlZCBwYXNzZXNcWyBcdFxdKzFcbi4qdW5yZXNv bHZlZCB0ZXN0Y2FzZXNcWyBcdFxdKzFcbiIgfQorICAgIHsgImNvbnRpbnVlIGFmdGVyIGFi b3J0IHdpdGggLS1rZWVwX2dvaW5nIgorCSItLWtlZXBfZ29pbmcgYWJvcnQtdW5kZWYuZXhw IHNpbXBsZS5leHAiCisJIlBBU1M6IHJ1bm5pbmcgYWJvcnQtdW5kZWYuZXhwLipcCisJKlVO UkVTT0xWRUQ6IC4qIGFib3J0ZWQgYXQgY2FsbCB0byB1bmtub3duIGNvbW1hbmQuKlwKKwkq UEFTUzogc2ltcGxlIHRlc3QuKlwKKwkqZXhwZWN0ZWQgcGFzc2VzXFsgXHRcXSsyXG4uKnVu cmVzb2x2ZWQgdGVzdGNhc2VzXFsgXHRcXSsxXG4iIH0KK30KKworZm9yZWFjaCB0ICR0ZXN0 cyB7CisgICAgaWYgW3V0aWxfdGVzdCAkUlVOVEVTVCBcCisJICAgICItLWxvY2FsX2luaXQg YWJvcnQtaW5pdC5leHBcCisJCS0tb3V0ZGlyICR0bXBkaXIgLWEgW2xpbmRleCAkdCAxXSIg XAorCSAgICAiIiBcCisJICAgIFtsaW5kZXggJHQgMl1dIHsKKwlmYWlsIFtsaW5kZXggJHQg MF0KKyAgICB9IGVsc2UgeworCXBhc3MgW2xpbmRleCAkdCAwXQorICAgIH0KK30KKworZmls ZSBkZWxldGUgLWZvcmNlICR0bXBkaXIKZGlmZiAtLWdpdCBhL3Rlc3RzdWl0ZS9ydW50ZXN0 Lm1haW4vYWJvcnQvdGVzdHN1aXRlL2Fib3J0LnRlc3QvYWJvcnQtdW5kZWYuZXhwIGIvdGVz dHN1aXRlL3J1bnRlc3QubWFpbi9hYm9ydC90ZXN0c3VpdGUvYWJvcnQudGVzdC9hYm9ydC11 bmRlZi5leHAKbmV3IGZpbGUgbW9kZSAxMDA2NDQKaW5kZXggMDAwMDAwMC4uZTVmNDgwMwot LS0gL2Rldi9udWxsCisrKyBiL3Rlc3RzdWl0ZS9ydW50ZXN0Lm1haW4vYWJvcnQvdGVzdHN1 aXRlL2Fib3J0LnRlc3QvYWJvcnQtdW5kZWYuZXhwCkBAIC0wLDAgKzEsMjUgQEAKKyMgQ29w eXJpZ2h0IChDKSAyMDIwIEZyZWUgU29mdHdhcmUgRm91bmRhdGlvbiwgSW5jLgorIworIyBU aGlzIGZpbGUgaXMgcGFydCBvZiBEZWphR251LgorIworIyBEZWphR251IGlzIGZyZWUgc29m dHdhcmU7IHlvdSBjYW4gcmVkaXN0cmlidXRlIGl0IGFuZC9vciBtb2RpZnkgaXQKKyMgdW5k ZXIgdGhlIHRlcm1zIG9mIHRoZSBHTlUgR2VuZXJhbCBQdWJsaWMgTGljZW5zZSBhcyBwdWJs aXNoZWQgYnkKKyMgdGhlIEZyZWUgU29mdHdhcmUgRm91bmRhdGlvbjsgZWl0aGVyIHZlcnNp b24gMyBvZiB0aGUgTGljZW5zZSwgb3IKKyMgKGF0IHlvdXIgb3B0aW9uKSBhbnkgbGF0ZXIg dmVyc2lvbi4KKyMKKyMgRGVqYUdudSBpcyBkaXN0cmlidXRlZCBpbiB0aGUgaG9wZSB0aGF0 IGl0IHdpbGwgYmUgdXNlZnVsLCBidXQKKyMgV0lUSE9VVCBBTlkgV0FSUkFOVFk7IHdpdGhv dXQgZXZlbiB0aGUgaW1wbGllZCB3YXJyYW50eSBvZgorIyBNRVJDSEFOVEFCSUxJVFkgb3Ig RklUTkVTUyBGT1IgQSBQQVJUSUNVTEFSIFBVUlBPU0UuICBTZWUgdGhlIEdOVQorIyBHZW5l cmFsIFB1YmxpYyBMaWNlbnNlIGZvciBtb3JlIGRldGFpbHMuCisjCisjIFlvdSBzaG91bGQg aGF2ZSByZWNlaXZlZCBhIGNvcHkgb2YgdGhlIEdOVSBHZW5lcmFsIFB1YmxpYyBMaWNlbnNl CisjIGFsb25nIHdpdGggRGVqYUdudTsgaWYgbm90LCB3cml0ZSB0byB0aGUgRnJlZSBTb2Z0 d2FyZSBGb3VuZGF0aW9uLAorIyBJbmMuLCA1MSBGcmFua2xpbiBTdHJlZXQgLSBGaWZ0aCBG bG9vciwgQm9zdG9uLCBNQSAwMjExMC0xMzAxLCBVU0EuCisKKyMgSW52b2tlIGFuIHVuZGVm aW5lZCBjb21tYW5kLCBjYXVzaW5nIGEgZmF0YWwgZXJyb3IuCisKK3Bhc3MgInJ1bm5pbmcg YWJvcnQtdW5kZWYuZXhwIgorCitib2d1c19jb21tYW5kIDEgMiAzIDQKKworZmFpbCAic2Ny aXB0IGRpZCBub3QgYWJvcnQiCmRpZmYgLS1naXQgYS90ZXN0c3VpdGUvcnVudGVzdC5tYWlu L2Fib3J0L3Rlc3RzdWl0ZS9hYm9ydC50ZXN0L3NpbXBsZS5leHAgYi90ZXN0c3VpdGUvcnVu dGVzdC5tYWluL2Fib3J0L3Rlc3RzdWl0ZS9hYm9ydC50ZXN0L3NpbXBsZS5leHAKbmV3IGZp bGUgbW9kZSAxMDA2NDQKaW5kZXggMDAwMDAwMC4uOTNhMDNlNwotLS0gL2Rldi9udWxsCisr KyBiL3Rlc3RzdWl0ZS9ydW50ZXN0Lm1haW4vYWJvcnQvdGVzdHN1aXRlL2Fib3J0LnRlc3Qv c2ltcGxlLmV4cApAQCAtMCwwICsxLDIxIEBACisjIENvcHlyaWdodCAoQykgMjAyMCBGcmVl IFNvZnR3YXJlIEZvdW5kYXRpb24sIEluYy4KKyMKKyMgVGhpcyBmaWxlIGlzIHBhcnQgb2Yg RGVqYUdudS4KKyMKKyMgRGVqYUdudSBpcyBmcmVlIHNvZnR3YXJlOyB5b3UgY2FuIHJlZGlz dHJpYnV0ZSBpdCBhbmQvb3IgbW9kaWZ5IGl0CisjIHVuZGVyIHRoZSB0ZXJtcyBvZiB0aGUg R05VIEdlbmVyYWwgUHVibGljIExpY2Vuc2UgYXMgcHVibGlzaGVkIGJ5CisjIHRoZSBGcmVl IFNvZnR3YXJlIEZvdW5kYXRpb247IGVpdGhlciB2ZXJzaW9uIDMgb2YgdGhlIExpY2Vuc2Us IG9yCisjIChhdCB5b3VyIG9wdGlvbikgYW55IGxhdGVyIHZlcnNpb24uCisjCisjIERlamFH bnUgaXMgZGlzdHJpYnV0ZWQgaW4gdGhlIGhvcGUgdGhhdCBpdCB3aWxsIGJlIHVzZWZ1bCwg YnV0CisjIFdJVEhPVVQgQU5ZIFdBUlJBTlRZOyB3aXRob3V0IGV2ZW4gdGhlIGltcGxpZWQg d2FycmFudHkgb2YKKyMgTUVSQ0hBTlRBQklMSVRZIG9yIEZJVE5FU1MgRk9SIEEgUEFSVElD VUxBUiBQVVJQT1NFLiAgU2VlIHRoZSBHTlUKKyMgR2VuZXJhbCBQdWJsaWMgTGljZW5zZSBm b3IgbW9yZSBkZXRhaWxzLgorIworIyBZb3Ugc2hvdWxkIGhhdmUgcmVjZWl2ZWQgYSBjb3B5 IG9mIHRoZSBHTlUgR2VuZXJhbCBQdWJsaWMgTGljZW5zZQorIyBhbG9uZyB3aXRoIERlamFH bnU7IGlmIG5vdCwgd3JpdGUgdG8gdGhlIEZyZWUgU29mdHdhcmUgRm91bmRhdGlvbiwKKyMg SW5jLiwgNTEgRnJhbmtsaW4gU3RyZWV0IC0gRmlmdGggRmxvb3IsIEJvc3RvbiwgTUEgMDIx MTAtMTMwMSwgVVNBLgorCisjIFJldHVybiBhIHBhc3NpbmcgcmVzdWx0CisKK3Bhc3MgInNp bXBsZSB0ZXN0IgotLSAKMS43LjQuMQoK --------------080008000608070707030003-- From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 23 14:45:15 2020 Received: (at 41824) by debbugs.gnu.org; 23 Jun 2020 18:45:15 +0000 Received: from localhost ([127.0.0.1]:37111 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jnnv1-0005Mw-Bh for submit@debbugs.gnu.org; Tue, 23 Jun 2020 14:45:15 -0400 Received: from us-smtp-2.mimecast.com ([207.211.31.81]:37080 helo=us-smtp-delivery-1.mimecast.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jnnuy-0005Mm-PM for 41824@debbugs.gnu.org; Tue, 23 Jun 2020 14:45:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1592937912; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OaL1tguCNWJWnzKDTB4J3dF5wIh9AqZddOQCIfEx3A0=; b=dcNaYNMRTu5VmCgtD10Zlsrt6qjadQWDfQYZCLvgI1EJlxp8uVUatg4Nv4jX8nQJZMR3pS FAl0R0Giw4GiNHj9bN61bzj6rr+W2p4969yXn5YcnQpFNs/NLfWrTDa+H67daBWfDgwlOt 890NOBWtO6KZDJIRRJ8U8vwUNyxIrHk= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-121-2Xy2DXM5MVmdHeh4vZaX6w-1; Tue, 23 Jun 2020 14:45:08 -0400 X-MC-Unique: 2Xy2DXM5MVmdHeh4vZaX6w-1 Received: by mail-wr1-f72.google.com with SMTP id f5so16544576wrv.22 for <41824@debbugs.gnu.org>; Tue, 23 Jun 2020 11:45:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=OaL1tguCNWJWnzKDTB4J3dF5wIh9AqZddOQCIfEx3A0=; b=nSMmTKvIkJ1oPJEYSA3bTzNHROyIaQlF1SS1h+WB71SKIUP30qXYY++iYbCPxms776 jQg25Lxc/8sJrWFIGQ9BgFDMtCYiaTfKLSwRadjk48ioYggCemjn2E/gKucPUbyMhIhY toeaWXnxqqhgvkrypYu5AQuhlIPjK5vJ6yhkDo31Qcfp0gxpiOPmpEA+bagLlyhG96qG f15d5k86VFnUrD8teoWqJdC41dSIW/wKCRihBoSemNQZXhjUrldB6SRNF85IgfXmEMyV FuC13hye3Gn6kqrtNAR3V4B5M/PWWugn4YRgZoJmgvoW7Z6F3UwlGVBAJPoBVWraDSEu GHgw== X-Gm-Message-State: AOAM532YHVi+YNSXCj5tmABw2TlgPpIH8upmwiA1lVh1soJY8l/EgLny jNv/fM3ml/sxo+J/Orodf3rrdpZfCn4C0Y4MdND6cfrQL3mkMViH8Ygr1NArijWpFN1uMMFOpYM 59ikvaAAi/ojc7fU= X-Received: by 2002:adf:81c7:: with SMTP id 65mr25622818wra.47.1592937906764; Tue, 23 Jun 2020 11:45:06 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwhVVgftzm9XGPzMPjXG5IRs0qBBOYuJycqTc2Bw8Vhesf11O6kJwh0RqA3IYppS6dqEpEyGQ== X-Received: by 2002:adf:81c7:: with SMTP id 65mr25622800wra.47.1592937906563; Tue, 23 Jun 2020 11:45:06 -0700 (PDT) Received: from ?IPv6:2001:8a0:f922:c400:56ee:75ff:fe8d:232b? ([2001:8a0:f922:c400:56ee:75ff:fe8d:232b]) by smtp.gmail.com with ESMTPSA id v66sm5048674wme.13.2020.06.23.11.45.05 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 23 Jun 2020 11:45:05 -0700 (PDT) Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case To: jcb62281@gmail.com, Tom de Vries References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> From: Pedro Alves Message-ID: <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> Date: Tue, 23 Jun 2020 19:45:04 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <5EE98D60.2090005@gmail.com> Content-Language: en-US Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=palves@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 41824 Cc: 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 6/17/20 4:26 AM, Jacob Bachmeyer wrote: >> >> I monitor ERROR and WARNING in my test runs, so I didn't think of this, >> but indeed, those are not tracked in the summary, so that could be a >> problem, agreed. >>   > > Blowing up the entire test run is a good way to ensure that problems (like typos in testcases) are noticed, but admittedly, that does not help when people fail to even run the tests before checking in their typos and then other developers' test runs blow up.  Murphy's Law says the offenders then carry on to check in more typos in other code without testing their own work there either... There's conditional execution in the gdb testsuite, where tests may run different code paths depending on the target. Typos can happen in code paths that the developer isn't able to exercise. You can imagine someone missing something when doing some large refactoring in the testsuite. I'd think that tracking ERROR and WARNING in the summary would sort out this "test run blew up but no one noticed" issue. Thanks, Pedro Alves From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 23 19:37:01 2020 Received: (at 41824) by debbugs.gnu.org; 23 Jun 2020 23:37:01 +0000 Received: from localhost ([127.0.0.1]:37411 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jnsTN-0001ZE-Av for submit@debbugs.gnu.org; Tue, 23 Jun 2020 19:37:01 -0400 Received: from mail-ot1-f46.google.com ([209.85.210.46]:43925) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jnsTK-0001Yv-P2 for 41824@debbugs.gnu.org; Tue, 23 Jun 2020 19:36:59 -0400 Received: by mail-ot1-f46.google.com with SMTP id u23so166193otq.10 for <41824@debbugs.gnu.org>; Tue, 23 Jun 2020 16:36:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=A4MRonOiIxsyQWL5KqD9+XvhOFJrji+wVcG8ZgE1/Yo=; b=jdyQmhiAbXhjJ5hzURZSxIqfhq8yJkZp3B+0/8t/OL5hM3iOod/Go0k21AfeuF6Y65 5/xVPGJokBYfNwfb5kTIcgGBtKs4O3Lo0oxRpWpG+zXeFBWtTResGnK4B9SuJd/100ke 7n9AveZorCB3KicMS8aqvCmovnHh7gc6ay914BUxdDP+6RBljg2A8Kb/lNELerBoxMLZ +uDGkkJcVWwZM+UhStw03pnVGjQE7y+WpYD2cNPjtGqnvIQB2VOoU7agjM9OnCEWL/fk Sw/O0ShZH5RNSfX8EjQND4uHhNBa+XJMPrsX0xAPK/5gq5JlIxM3kxnzUzagDFlxmhNY QcZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:reply-to:user-agent :mime-version:to:cc:subject:references:in-reply-to :content-transfer-encoding; bh=A4MRonOiIxsyQWL5KqD9+XvhOFJrji+wVcG8ZgE1/Yo=; b=TeVNdUOI+GxmmFQ9tbsFGs60DH+RxIRHgDlJkv7ErQKWAIIDQ7fVMsnLfTDgBZvU32 g/GHBJGxW6bvNfwdVdWgoamfeAU9Bpb9sEw2BaLq6vnQLrYn3gQf4ezNL5XAUG4ubavi NCM6cyc/5a2wVxt/p+q7oLAiKBpcZEXyG40wFzHUujKHNDxv7Mw16kzWsIwGUmL8Cs9c 5M2Dnfm7jiWi6ZEykmGihYOaB2YIhfRz7l5Owj+28DJIeQSTnLSAkYSgf9cGTMYqZAGO 5UOGdTBPxilbFsAgcEvxqMr2jV5juCysXnljGc7L5MwQadrTgtnNOlp4xAHJj5JsEq4h v56g== X-Gm-Message-State: AOAM533G+ficr5slQSwYZ2YztAFOxtaMrze8bYq/dkLlofXUvqE+QJl3 gOQDCUMeUgftErw5AIcmqq0= X-Google-Smtp-Source: ABdhPJx9znMuhqkHxe/kaWmBw3I3JXREhWF2TinfNMZxK+MteANSpHy2anL/8vi6uKQWVTb76FjOfw== X-Received: by 2002:a9d:4cd:: with SMTP id 71mr21945553otm.188.1592955412988; Tue, 23 Jun 2020 16:36:52 -0700 (PDT) Received: from [192.168.2.42] (adsl-70-133-144-251.dsl.ablntx.sbcglobal.net. [70.133.144.251]) by smtp.gmail.com with ESMTPSA id 2sm4342896oiq.30.2020.06.23.16.36.51 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 23 Jun 2020 16:36:52 -0700 (PDT) Message-ID: <5EF29212.1050601@gmail.com> Date: Tue, 23 Jun 2020 18:36:50 -0500 From: Jacob Bachmeyer User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Pedro Alves Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> In-Reply-To: <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jcb62281@gmail.com Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) Pedro Alves wrote: > On 6/17/20 4:26 AM, Jacob Bachmeyer wrote: > >>> I monitor ERROR and WARNING in my test runs, so I didn't think of this, >>> but indeed, those are not tracked in the summary, so that could be a >>> problem, agreed. >>> >>> >> Blowing up the entire test run is a good way to ensure that problems (like typos in testcases) are noticed, but admittedly, that does not help when people fail to even run the tests before checking in their typos and then other developers' test runs blow up. Murphy's Law says the offenders then carry on to check in more typos in other code without testing their own work there either... >> > > There's conditional execution in the gdb testsuite, where tests may > run different code paths depending on the target. Typos can happen in > code paths that the developer isn't able to exercise. You can > imagine someone missing something when doing some large refactoring > in the testsuite. > This is a good point and is far more reasonable than checking in code without testing it at all. That sounds like an architectural issue with the GDB testsuite, but could also be a legitimate problem if the per-target handling really is different for some targets, especially if simulators for those targets are unavailable or insufficient for the tests. > I'd think that tracking ERROR and WARNING in the summary would sort > out this "test run blew up but no one noticed" issue. The problem is that the DejaGnu internals often generate ERROR and WARNING "directly" without using the procedures that update counters. The solution I am using in the current PR41824 patch (which has been rolled into the PR41918 patch) generates an extra UNRESOLVED result when a Tcl error is caught in runtest, which both appears in the summary and clearly indicates that the test run did not produce valid results. I am currently considering also adding a '--no-keep-going'/'--no_keep_going' option and making '--keep-going' the default for the rest of the 1.6 series. Aborting on uncaught Tcl errors will be the default in 1.7, but I am trying to decide whether changing it in 1.6 counts more as fixing a bug (DejaGnu quietly sweeping Tcl errors under the proverbial rug) or introducing an environment change (needing '--keep-going' to reach the end of some testsuites that "worked" before). This is further complicated by the fact that any testsuite needing that option to reach the end means that, technically, that testsuite is invalid -- it *did* throw a Tcl error during testing. -- Jacob From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 24 05:40:53 2020 Received: (at 41824) by debbugs.gnu.org; 24 Jun 2020 09:40:53 +0000 Received: from localhost ([127.0.0.1]:37813 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jo1tk-0001be-SO for submit@debbugs.gnu.org; Wed, 24 Jun 2020 05:40:53 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:30738 helo=us-smtp-1.mimecast.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jo1ti-0001bV-PJ for 41824@debbugs.gnu.org; Wed, 24 Jun 2020 05:40:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1592991650; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=91Z5dZVO2dpxbYkZvTysUcUeMSU5vKVzBLCLvhCv9rY=; b=K3Y1Esvz737uyxPL4IA/YLgbwU2HZOwPnYhR8FeeXnVCfKNyhcJ51py3RFiayqFC6k4hos /HVGnKmwD9Ehmpy+C8ijt4jt5Ev5dyz8xiBIP3fWwCt9q04f19KO8woyimMNda244wz2oZ +2aDbW8K7fT9mmuHFNoRt2f1wNR9IyI= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-504-3FzsgLNhOIO6OxMZQkOqFg-1; Wed, 24 Jun 2020 05:40:48 -0400 X-MC-Unique: 3FzsgLNhOIO6OxMZQkOqFg-1 Received: by mail-wr1-f72.google.com with SMTP id i5so1062231wra.9 for <41824@debbugs.gnu.org>; Wed, 24 Jun 2020 02:40:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=91Z5dZVO2dpxbYkZvTysUcUeMSU5vKVzBLCLvhCv9rY=; b=KmP5hzxIODmwwXO+F07dtSgp6tOhUPRyfO3s+t/yNfvT4kkSJwvTZoZDGlc7rKp6DA +0eP3Vzw5C5UTcQP7Tovk20POsgsw2i79VzTGBnz7/3aOYpyItLWu7ec8svIGOnggmIm aSHeKw61p6DXOKPHp+vEyeqDZ02V5XsXhn+C+xLKVgvXHrGvHWnudgH/cnnNAkhTvFyz PBayTaC0sG+6C7/JUt0spoFd+6zeqpeH9NK1qRoUy7dVpGF4UC2FnH31RI8xjx40tukO Vevir1CX88j0XbVWrIfEWFRAECUwCW8LXgXMgMSPFrZto3sQYdxkXzRpTJ+CkogB7hbP xpLQ== X-Gm-Message-State: AOAM532W8+ewzeW7rFd7SEcVPS9Lf9SNzJIKCFV/2diGK3oEALGt/s9j C8hGilQ4CwUKo7YW7C/mRt0mKqQo1ll4Tp44XT/sg1uB3A5mv6X6w59L8VnL6A7sgrkmozQVuzy bppaJV3z8DlRfGM4= X-Received: by 2002:a5d:6a46:: with SMTP id t6mr3041387wrw.374.1592991647053; Wed, 24 Jun 2020 02:40:47 -0700 (PDT) X-Google-Smtp-Source: ABdhPJynpKcHADQMscsvGHk81E8/gKdEZoitOK7uROs5wmU8412PP52+J2IE6cTQ3dwZ28DVw+v/eg== X-Received: by 2002:a5d:6a46:: with SMTP id t6mr3041360wrw.374.1592991646707; Wed, 24 Jun 2020 02:40:46 -0700 (PDT) Received: from ?IPv6:2001:8a0:f922:c400:56ee:75ff:fe8d:232b? ([2001:8a0:f922:c400:56ee:75ff:fe8d:232b]) by smtp.gmail.com with ESMTPSA id c25sm7168389wml.46.2020.06.24.02.40.45 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 24 Jun 2020 02:40:46 -0700 (PDT) Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case To: jcb62281@gmail.com References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> From: Pedro Alves Message-ID: <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> Date: Wed, 24 Jun 2020 10:40:45 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <5EF29212.1050601@gmail.com> Content-Language: en-US X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 6/24/20 12:36 AM, Jacob Bachmeyer wrote: > Pedro Alves wrote: >> On 6/17/20 4:26 AM, Jacob Bachmeyer wrote: >>   >>>> I monitor ERROR and WARNING in my test runs, so I didn't think of this, >>>> but indeed, those are not tracked in the summary, so that could be a >>>> problem, agreed. >>>>         >>> Blowing up the entire test run is a good way to ensure that problems (like typos in testcases) are noticed, but admittedly, that does not help when people fail to even run the tests before checking in their typos and then other developers' test runs blow up.  Murphy's Law says the offenders then carry on to check in more typos in other code without testing their own work there either... >>>     >> >> There's conditional execution in the gdb testsuite, where tests may >> run different code paths depending on the target.  Typos can happen in >> code paths that the developer isn't able to exercise.  You can >> imagine someone missing something when doing some large refactoring >> in the testsuite. >>   > > This is a good point and is far more reasonable than checking in code without testing it at all.  That sounds like an architectural issue with the GDB testsuite, but could also be a legitimate problem if the per-target handling really is different for some targets, especially if simulators for those targets are unavailable or insufficient for the tests. It's not an architectural issue -- simply different targets support a different set of features, support the same features but differently, run different OSs or even no OS at all, are built with different compilers, support a different set of languages, etc., etc. > >> I'd think that tracking ERROR and WARNING in the summary would sort >> out this "test run blew up but no one noticed" issue. > > The problem is that the DejaGnu internals often generate ERROR and WARNING "directly" without using the procedures that update counters.  I understand that that's what DejaGnu does currently, but couldn't it be changed? Would it not be desirable? The solution I am using in the current PR41824 patch (which has been rolled into the PR41918 patch) generates an extra UNRESOLVED result when a Tcl error is caught in runtest, which both appears in the summary and clearly indicates that the test run did not produce valid results. > > I am currently considering also adding a '--no-keep-going'/'--no_keep_going' option and making '--keep-going' the default for the rest of the 1.6 series.  Aborting on uncaught Tcl errors will be the default in 1.7, but I am trying to decide whether changing it in 1.6 counts more as fixing a bug (DejaGnu quietly sweeping Tcl errors under the proverbial rug) or introducing an environment change (needing '--keep-going' to reach the end of some testsuites that "worked" before).  This is further complicated by the fact that any testsuite needing that option to reach the end means that, technically, that testsuite is invalid -- it *did* throw a Tcl error during testing. It seems your designing this under the assumption that if a testsuite run "blows up", it's not a big deal to just fix it and rerun the testsuite. What that misses is that in some environments, particularly with slow embedded system boards, running the full testsuite can take _days_. This is all compounded by the fact that the testsuite must be run in a large set of different modes and boards to cover everything for a single target. Even on a modern GNU/Linux system, running all combinations of modes for full coverage can take hours. Having the testrun stop midway because of some specific testcase blowing up can waste a lot of time. The fact that the testsuite can take a while to fully run results in automated testing systems, like GDB's buildbot, struggling to keep up with testing all the commits that land in git. On some buildbot setups, it's just not possible to test all commits. So if some testcase blows up the whole test run, the next time you run the testsuite, it'll likely be that a large number of unrelated commits went into the sources. Losing a large chunk of the testsuite results because of some small typo in one testcase is thus undesirable, because noticing the typo in the first place can take a while, and also you may be stuck with a "last good run" for many of the testcases that gets older and older until the typo is finally fixed. Then there's the issue about testsuite order. DejaGnu runs testcases in alphabetical filename order, in sequence. If the testrun blows up in the first testcase run, say, a.exp, the whole testrun stops before b.exp, etc. are run. If the testcase that blows up is the last one, say named zzz.exp, most of the testsuite runs. This is not "fair". If I rename a.exp to zzzzz.exp, then suddenly that testcase gains more "importance" in the sense that it can no longer blow up the whole test run. A testcase's name should not be significant like that. On the same vein, consider parallel mode. This is not native to DejaGnu, but both GDB and GCC have parallel modes, where their makefiles set up a make check such that they spawn a number of runtest processes running in parallel, each exercising a subset the whole testsuite. Imagine a testsuite composed of exactly 32 testcases. And further imagine that you run it under "make check -j32". That is, you'll end up with 32 runtest instances, each running exactly one testcase. Now, if one of the testcases blows up, all the other 31 will still run. Again, this shows that testcase name, and testrun order should not be important. There is no real reason that a sequential run in that scenario ends up with different results (testcase a.exp blew up, so no result for the other tests) compared to a parallel run (all tests ran successfully but test a.exp). Instead, I advocate that the testresults summary should indicate that a problem happened, so that "blow up event" isn't lost. But do let the whole testsuite run. Thanks, Pedro Alves From debbugs-submit-bounces@debbugs.gnu.org Wed Jun 24 22:44:08 2020 Received: (at 41824) by debbugs.gnu.org; 25 Jun 2020 02:44:08 +0000 Received: from localhost ([127.0.0.1]:39550 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1joHrz-0007ns-NG for submit@debbugs.gnu.org; Wed, 24 Jun 2020 22:44:08 -0400 Received: from mail-oi1-f194.google.com ([209.85.167.194]:41724) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1joHrw-0007nG-KG for 41824@debbugs.gnu.org; Wed, 24 Jun 2020 22:44:06 -0400 Received: by mail-oi1-f194.google.com with SMTP id a21so3661712oic.8 for <41824@debbugs.gnu.org>; Wed, 24 Jun 2020 19:44:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=EsgF92L38VfpQnYEhCXjdEKOcX7qtctPuW9ds0mejoI=; b=WYJ3uKOChjxvRbRg9BNOH8KopO2F4yB8m+9elq6bKYyjyc98dHaeSBUGPSLcoWnsO+ St2TI2QcVVXRmoL8eltlzpLlZUj02E+DAJfoO2IBU7ZSu69YWwUMexgNGvT/uzXIxhEr htvc1Rd/+pJ1y31G+1NxUpi0eLWQ1vEPft0AlOPZPtf+5/THIsBJxgZGZXyzv7Ey2Lyw YnIjnGGuCW/cSqAqcyquG69y9DxQmdOoo4nBObHeWPmWh/7pdkAm1IZJIQiKEugJrOEW 87qxF5eGYCSNxdC5o1MreBigo1HA5BFxRILhAI18OwgD2Wc0x8um5HcE3ePGVsRYRtoU i2GQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:reply-to:user-agent :mime-version:to:cc:subject:references:in-reply-to :content-transfer-encoding; bh=EsgF92L38VfpQnYEhCXjdEKOcX7qtctPuW9ds0mejoI=; b=I1cIRBiV1bvFl30qdwjoRsgHiVHDSYQ0KA/zLhOGjenqD2snXIM2xPQxpr7aNnRE1a qykeANNx4tjSoBObhp5sFX6nXU8/JD5A+2VyCX1xlUjhLqD8145nIdCZb35AozoMSwPn yZLAVFrNKCcC9CXSQdk4nDbU7KZZydzcPamwiUfbg5CwvmS27LfYdzyP/wC+RKlV7oKp G9KeauIymz+wwlfw3TkXTiT2mewksyj0qUb50aWX9rGQIhH+W6J7JO6nTVsQ7n9FXJHP yJjPr7nb5SFPE4Oy0zmARAwQpfCHLNeO8wl7zNQJEcf05foEciVSsopDrb6brf85Zw0C +ADA== X-Gm-Message-State: AOAM531YcjsfrzR68+jFEeCn2hsjkhgYfN40YN8i5TYH5f/32409UoT8 iIMut4i0lboStyHMez0MWic= X-Google-Smtp-Source: ABdhPJwgpQt4HL5mS6bAAREf9NaQcTRNQus7b6QSI+oZWE/DSqtlVSRfAM7nx6hFzVyI5CHSnGEO+A== X-Received: by 2002:aca:d454:: with SMTP id l81mr621857oig.152.1593053038759; Wed, 24 Jun 2020 19:43:58 -0700 (PDT) Received: from [192.168.2.42] (adsl-70-133-144-251.dsl.ablntx.sbcglobal.net. [70.133.144.251]) by smtp.gmail.com with ESMTPSA id z7sm5218940oto.2.2020.06.24.19.43.57 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 24 Jun 2020 19:43:57 -0700 (PDT) Message-ID: <5EF40F6C.8010004@gmail.com> Date: Wed, 24 Jun 2020 21:43:56 -0500 From: Jacob Bachmeyer User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Pedro Alves Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> In-Reply-To: <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jcb62281@gmail.com Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) Pedro Alves wrote: > On 6/24/20 12:36 AM, Jacob Bachmeyer wrote: > >> Pedro Alves wrote: >> >>> On 6/17/20 4:26 AM, Jacob Bachmeyer wrote: >>> >>> I'd think that tracking ERROR and WARNING in the summary would sort >>> out this "test run blew up but no one noticed" issue. >>> >> The problem is that the DejaGnu internals often generate ERROR and WARNING "directly" without using the procedures that update counters. >> > > I understand that that's what DejaGnu does currently, but couldn't it be changed? > Would it not be desirable? > It could be changed, but those are much larger changes to the code and overall behavior than simply adding an UNRESOLVED result to report the failure, and if I understand correctly, the presence of UNRESOLVED results marks the test results as invalid or partially-valid at best, which is correct if a test script failed with a Tcl error. Also, the current warning and error counters are not global, but are reset after each test result. Exceeding thresholds for warnings or errors causes the next test result to be changed to UNRESOLVED before the counters are reset. >> I am currently considering also adding a '--no-keep-going'/'--no_keep_going' option and making '--keep-going' the default for the rest of the 1.6 series. Aborting on uncaught Tcl errors will be the default in 1.7, but I am trying to decide whether changing it in 1.6 counts more as fixing a bug (DejaGnu quietly sweeping Tcl errors under the proverbial rug) or introducing an environment change (needing '--keep-going' to reach the end of some testsuites that "worked" before). This is further complicated by the fact that any testsuite needing that option to reach the end means that, technically, that testsuite is invalid -- it *did* throw a Tcl error during testing. >> > It seems your designing this under the assumption that if a testsuite > run "blows up", it's not a big deal to just fix it and rerun the testsuite. > For most packages, this is correct. For GDB, perhaps it is a bigger problem. On the other hand, "blowing up" at least indicates clearly that something is wrong and prevents "whistling past the graveyard" where tests are not actually running, but everyone thinks the situation is fine. > What that misses is that in some environments, particularly with slow embedded > system boards, running the full testsuite can take _days_. This is all > compounded by the fact that the testsuite must be run in a large set of > different modes and boards to cover everything for a single target. Even on > a modern GNU/Linux system, running all combinations of modes for full coverage > can take hours. > > Having the testrun stop midway because of some specific testcase > blowing up can waste a lot of time. > > The fact that the testsuite can take a while to fully run results in automated > testing systems, like GDB's buildbot, struggling to keep up with testing > all the commits that land in git. On some buildbot setups, it's just not > possible to test all commits. So if some testcase blows up the whole > test run, the next time you run the testsuite, it'll likely be that a > large number of unrelated commits went into the sources. Losing a large > chunk of the testsuite results because of some small typo in one > testcase is thus undesirable, because noticing the typo in the first > place can take a while, and also you may be stuck with a "last good run" > for many of the testcases that gets older and older until > the typo is finally fixed. > I presume buildbot can be easily configured to pass a '--keep_going' option, perhaps as 'RUNTESTFLAGS=--keep_going' if needed? Still, needing a new option to get old behavior is changing the environment interface, so I will make --keep_going the default, at least for the rest of 1.6. That default is likely to change for 1.7. For automated systems like buildbot, passing '--keep_going' should be perfectly reasonable, which is why the option exists. (While the documented options will be changed to use hyphens to align with the rest of the GNU system in 1.7, the current option names will continue to be accepted, since the aliases are obvious.) This has been pushed on the temporary PR41918 branch because the fix for PR41824 has been rolled into the fix for PR41918, as the bugs are closely related. > Then there's the issue about testsuite order. DejaGnu runs testcases > in alphabetical filename order, in sequence. If the testrun blows up in > the first testcase run, say, a.exp, the whole testrun stops before b.exp, > etc. are run. If the testcase that blows up is the last one, say named > zzz.exp, most of the testsuite runs. This is not "fair". If I rename > a.exp to zzzzz.exp, then suddenly that testcase gains more "importance" in > the sense that it can no longer blow up the whole test run. A > testcase's name should not be significant like that. > > On the same vein, consider parallel mode. This is not native to DejaGnu, > but both GDB and GCC have parallel modes, where their makefiles set up a > make check such that they spawn a number of runtest processes running in > parallel, each exercising a subset the whole testsuite. Imagine a testsuite > composed of exactly 32 testcases. And further imagine that you run it > under "make check -j32". That is, you'll end up with 32 runtest instances, > each running exactly one testcase. Now, if one of the testcases blows up, all > the other 31 will still run. Again, this shows that testcase name, and > testrun order should not be important. There is no real reason that a > sequential run in that scenario ends up with different results (testcase a.exp > blew up, so no result for the other tests) compared to a parallel > run (all tests ran successfully but test a.exp). Instead, I advocate > that the testresults summary should indicate that a problem happened, > so that "blow up event" isn't lost. But do let the whole testsuite run. > There are plans to eventually implement a native parallel testing mode. You are correct that the order in which testcases are run should be insignificant, and for valid testsuites that do not crash, it is insignificant -- all tests are run. The problems start when a testcase *crashes* and aborts with a Tcl error. The entire reason for stopping the testsuite immediately when a Tcl error occurs is that it is undeniable that something is *very* *wrong* when the testsuite stops early. Software testing is hard enough when you know your testsuite is good, but it is far worse when you have bugs in the testsuite masking bugs in the actual program because a test script is crashing and part of the program that is supposed to be tested is not being exercised. -- Jacob From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 25 12:01:22 2020 Received: (at 41824) by debbugs.gnu.org; 25 Jun 2020 16:01:22 +0000 Received: from localhost ([127.0.0.1]:40950 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1joUJV-0002cz-DN for submit@debbugs.gnu.org; Thu, 25 Jun 2020 12:01:21 -0400 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:58006 helo=us-smtp-delivery-1.mimecast.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1joUJT-0002cp-BX for 41824@debbugs.gnu.org; Thu, 25 Jun 2020 12:01:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1593100878; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gti9YYPCaf3FBzYuZEQlRFZD0dc7iPgWRWfYqrHLLyw=; b=FMVxbCBkdj71wICZ4x+6yz7p+TJP1FyZtneHCIzUCPdleYZLoMen9w+PeUHpC2UkByGxlm VWjz0nu6nhK454bObznZs3QRznDYH2/USVuP/o2pHEhQtas+Sj6fmyLlZYg1FSdQhjVkPz YGvLlq2280BmhLjZ5kngIS1pLi5T+II= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-201-KaxDp1oPN3-EMtNNt0ysXg-1; Thu, 25 Jun 2020 12:01:17 -0400 X-MC-Unique: KaxDp1oPN3-EMtNNt0ysXg-1 Received: by mail-wr1-f71.google.com with SMTP id d6so7082030wrn.1 for <41824@debbugs.gnu.org>; Thu, 25 Jun 2020 09:01:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=gti9YYPCaf3FBzYuZEQlRFZD0dc7iPgWRWfYqrHLLyw=; b=dVJ451Cekkuakr8swr6UFs35uO8YmC50KLNPmuOiPHfqv4Z5oMBoz+Ww4Q0HJ6B0Ze DiY5wqFSnpenJfXJxvdOBgX18/axH4i4k0Mn+S80Oh/stkeMUa/02zF8oy2ps6XwN5VM FRLSeu0zSO4T0cms4beTMKJPJbcw84WI1yKY2TetIgTMSNO2FbKIgU+gUylaJZOYCRPl mD9QiXAoP9qlO2vXWTnm62ZojMbHBNWTuIKJ92ygdpRBRSrdQ+KCa6gvZWXoYnMhXUxT w1Tu8SB0D73pN/fSw4eKYW1w6UNUfMAbzg/miOBbE/jOJapLgKJEJH8agk50JRd8lXZe 0zRg== X-Gm-Message-State: AOAM532Nt0GRqYWS5+i71MnPVQDvo+XChpaH2clC4sVxXJGn6yet2Xjy NxJwK/mQvqwW9Jm7vDHJ13Dv5cEC6j/qcBt2m5prX0ctiE9a8yhY0d7QioS+lN3OOLWvE4CMYVb nmFcbbWKAChcyzYg= X-Received: by 2002:adf:a507:: with SMTP id i7mr40451632wrb.0.1593100872727; Thu, 25 Jun 2020 09:01:12 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxypeD4JU76nk335mOSQjCHob3U1u3Hnq7JcphmikLxXni20zOkncM8cXdC9lFm9Tz/32cf3Q== X-Received: by 2002:adf:a507:: with SMTP id i7mr40451562wrb.0.1593100872279; Thu, 25 Jun 2020 09:01:12 -0700 (PDT) Received: from ?IPv6:2001:8a0:f922:c400:56ee:75ff:fe8d:232b? ([2001:8a0:f922:c400:56ee:75ff:fe8d:232b]) by smtp.gmail.com with ESMTPSA id x13sm33831058wre.83.2020.06.25.09.01.11 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 25 Jun 2020 09:01:11 -0700 (PDT) Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case To: jcb62281@gmail.com References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> From: Pedro Alves Message-ID: Date: Thu, 25 Jun 2020 17:01:10 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <5EF40F6C.8010004@gmail.com> Content-Language: en-US X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 6/25/20 3:43 AM, Jacob Bachmeyer wrote: > Pedro Alves wrote: >> On 6/24/20 12:36 AM, Jacob Bachmeyer wrote: >>   >>> Pedro Alves wrote: >>>     >>>> On 6/17/20 4:26 AM, Jacob Bachmeyer wrote: >>>>       I'd think that tracking ERROR and WARNING in the summary would sort >>>> out this "test run blew up but no one noticed" issue. >>>>       >>> The problem is that the DejaGnu internals often generate ERROR and WARNING "directly" without using the procedures that update counters.      >> >> I understand that that's what DejaGnu does currently, but couldn't it be changed? >> Would it not be desirable? >>   > > It could be changed, but those are much larger changes to the code and overall behavior than simply adding an UNRESOLVED result to report the failure, and if I understand correctly, the presence of UNRESOLVED results marks the test results as invalid or partially-valid at best, which is correct if a test script failed with a Tcl error. > > Also, the current warning and error counters are not global, but are reset after each test result.  Exceeding thresholds for warnings or errors causes the next test result to be changed to UNRESOLVED before the counters are reset. > >>> I am currently considering also adding a '--no-keep-going'/'--no_keep_going' option and making '--keep-going' the default for the rest of the 1.6 series.  Aborting on uncaught Tcl errors will be the default in 1.7, but I am trying to decide whether changing it in 1.6 counts more as fixing a bug (DejaGnu quietly sweeping Tcl errors under the proverbial rug) or introducing an environment change (needing '--keep-going' to reach the end of some testsuites that "worked" before).  This is further complicated by the fact that any testsuite needing that option to reach the end means that, technically, that testsuite is invalid -- it *did* throw a Tcl error during testing. >>>     >> It seems your designing this under the assumption that if a testsuite >> run "blows up", it's not a big deal to just fix it and rerun the testsuite. >>   > > For most packages, this is correct.  For GDB, perhaps it is a bigger problem. > > On the other hand, "blowing up" at least indicates clearly that something is wrong and prevents "whistling past the graveyard" where tests are not actually running, but everyone thinks the situation is fine. > >> What that misses is that in some environments, particularly with slow embedded >> system boards, running the full testsuite can take _days_.   This is all >> compounded by the fact that the testsuite must be run in a large set of >> different modes and boards to cover everything for a single target.  Even on >> a modern GNU/Linux system, running all combinations of modes for full coverage >> can take hours. >> >> Having the testrun stop midway because of some specific testcase >> blowing up can waste a lot of time. >> >> The fact that the testsuite can take a while to fully run results in automated >> testing systems, like GDB's buildbot, struggling to keep up with testing >> all the commits that land in git.  On some buildbot setups, it's just not >> possible to test all commits.  So if some testcase blows up the whole >> test run, the next time you run the testsuite, it'll likely be that a >> large number of unrelated commits went into the sources.  Losing a large >> chunk of the testsuite results because of some small typo in one >> testcase is thus undesirable, because noticing the typo in the first >> place can take a while, and also you may be stuck with a "last good run" >> for many of the testcases that gets older and older until >> the typo is finally fixed. >>   > > I presume buildbot can be easily configured to pass a '--keep_going' option, perhaps as 'RUNTESTFLAGS=--keep_going' if needed? > What's more likely is we'll tweak GDB's Makefile so that make check enables that option automatically. If GCC does the same, which I would encourage them to, since they also run the testsuite against systems that can take hours/days to complete, then I'll get to wonder who the default is for. :-) We'll need to somehow detect that the option is supported, though, so it's of course doable, just will require a configure check or some little bit of shell to probe the option or something of the sort. > Still, needing a new option to get old behavior is changing the environment interface, so I will make --keep_going the default, at least for the rest of 1.6.  That default is likely to change for 1.7.  For automated systems like buildbot, passing '--keep_going' should be perfectly reasonable, which is why the option exists.  (While the documented options will be changed to use hyphens to align with the rest of the GNU system in 1.7, the current option names will continue to be accepted, since the aliases are obvious.) > Currently with DejaGnu only the case of calling an unknown function stops the test run AFAIK. Any other tcl error like calling a proc with the wrong number of arguments, or treating a non-array variable as an array, etc. is caught by the catch in the runtest proc in runtest.exp, and the testrun continues. That to me clearly indicates that the original intention was to catch errors and continue. That also gives the tool's $(tool)_finish proc a chance to tear down correctly. It is just that the "unknown" case wasn't thought of. So I argue that not making unknown proc calls abort the run is a bug fix, and making DejaGnu abort the run for other errors by default is a behavior change that no real testsuite built around DejaGnu is asking for. > This has been pushed on the temporary PR41918 branch because the fix for PR41824 has been rolled into the fix for PR41918, as the bugs are closely related. > >> Then there's the issue about testsuite order.  DejaGnu runs testcases >> in alphabetical filename order, in sequence.  If the testrun blows up in >> the first testcase run, say, a.exp, the whole testrun stops before b.exp, >> etc. are run.  If the testcase that blows up is the last one, say named >> zzz.exp, most of the testsuite runs.  This is not "fair".  If I rename >> a.exp to zzzzz.exp, then suddenly that testcase gains more "importance" in >> the sense that it can no longer blow up the whole test run.  A >> testcase's name should not be significant like that. >> >> On the same vein, consider parallel mode.  This is not native to DejaGnu, >> but both GDB and GCC have parallel modes, where their makefiles set up a >> make check such that they spawn a number of runtest processes running in >> parallel, each exercising a subset the whole testsuite.  Imagine a testsuite >> composed of exactly 32 testcases.  And further imagine that you run it >> under "make check -j32".  That is, you'll end up with 32 runtest instances, >> each running exactly one testcase.  Now, if one of the testcases blows up, all >> the other 31 will still run.  Again, this shows that testcase name, and >> testrun order should not be important.  There is no real reason that a >> sequential run in that scenario ends up with different results (testcase a.exp >> blew up, so no result for the other tests) compared to a parallel >> run (all tests ran successfully but test a.exp).  Instead, I advocate >> that the testresults summary should indicate that a problem happened, >> so that "blow up event" isn't lost.  But do let the whole testsuite run. >>   > > There are plans to eventually implement a native parallel testing mode.  You are correct that the order in which testcases are run should be insignificant, and for valid testsuites that do not crash, it is insignificant -- all tests are run. > The problems start when a testcase *crashes* and aborts with a Tcl error. It's hopefully obvious that I'm aware of the latter. > > The entire reason for stopping the testsuite immediately when a Tcl error occurs is that it is undeniable that something is *very* *wrong* when the testsuite stops early.  Software testing is hard enough when you know your testsuite is good, but it is far worse when you have bugs in the testsuite masking bugs in the actual program because a test script is crashing and part of the program that is supposed to be tested is not being exercised. I think it's safe to say that we all understand the general principle, but IMO that applies more to continuing the same testcase (if somehow that were possible, like "unknown" suppressing the error), rather than continuing with a different testcase. Continuing the testrun when one testcase errors out does not mask any bug in the program that is supposed to be tested. Why would it? The testcase aborts, it doesn't continue. The remaining testcases start afresh. Anyway, given the option, it won't be bad, just a little annoying. Thanks, Pedro Alves From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 25 18:40:11 2020 Received: (at 41824) by debbugs.gnu.org; 25 Jun 2020 22:40:11 +0000 Received: from localhost ([127.0.0.1]:41465 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1joaXT-0006oh-4v for submit@debbugs.gnu.org; Thu, 25 Jun 2020 18:40:11 -0400 Received: from mail-oi1-f182.google.com ([209.85.167.182]:44823) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1joaXO-0006o0-Sb for 41824@debbugs.gnu.org; Thu, 25 Jun 2020 18:40:10 -0400 Received: by mail-oi1-f182.google.com with SMTP id k6so2553818oij.11 for <41824@debbugs.gnu.org>; Thu, 25 Jun 2020 15:40:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=BtHOyfijebLV7q6Tr6ShLXwppsFu4ADz1GQALb3CWA8=; b=Q0udBQMqgOy/D0A9/vEeB1GauINb7rJXrm4vQJG8ZUpYl6UC5/q7zC2eYV/vbTQqyb b5yvaJaxivrmSvB+WgQWQkZQ4da6/Ug2+3lKXVPTiLzlc+6Pu3m1P8glBxqAS4YkHz9r UoaRrTzsm6hm82MA58wWFJYqeMQmx1IPlKslDKRr4hYgt2iKPCTazzS7BhB9vQXQQhO8 GZ6qE2h03/GDAKMvOJh42/mZrxwJKZD7UNwmYC+0X5GlVjWCE1TIDRhjjNM3+4wGX/Vy gnXysIac3gUumOrOt1AGEHab+Vs19vf37yuWKOyVGPNwdTRNdnWBmudtQs5ZT0zF5c+6 /C1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:reply-to:user-agent :mime-version:to:cc:subject:references:in-reply-to :content-transfer-encoding; bh=BtHOyfijebLV7q6Tr6ShLXwppsFu4ADz1GQALb3CWA8=; b=RdpDDpCL7jMt7RgvZNK96ui8sbYJrz7/mAwbwBE56g25zyRoeospbOsmdgi8GOKW6Y uIKo3YIa6OUhC8xKfQkHpgFYsNgOVYEQqx2O7HkGAgu6fsDy46+uxu4RJUObd5OfDyfX 2XNjIdu4sRcnnZUNvt/3IrLFZcaza7LjDwY3X612UmFSDgUVYqF4AIlfbKH4lNleSIMG 4SFn807e/LxZ+6hp0gkFX6svDzD3sI4Nxvn0snVyMiesGXW74+DyHn6IzTlofAx4korx gWR1iz8rMFjAjSzBZgouHFbco5g+Yvn7BkG1DfZ5ItcI0lugBzKKMaUeknSHSSj6t+37 mptg== X-Gm-Message-State: AOAM532rDOGEwuGonR7FqZSe7JXYYelzV5x7t7cFhcLS1hvNmMk+7nwn BNQ4MQtMKau40mQ1J5OmMgw= X-Google-Smtp-Source: ABdhPJw+LkumYSzP822bNwlLppmm05E4fqxofBHTOv2wwY8Q9A5D0d50E2vd9UDwEAx/WiPJNhRsGw== X-Received: by 2002:aca:358b:: with SMTP id c133mr239072oia.174.1593124800794; Thu, 25 Jun 2020 15:40:00 -0700 (PDT) Received: from [192.168.2.42] (adsl-70-133-144-251.dsl.ablntx.sbcglobal.net. [70.133.144.251]) by smtp.gmail.com with ESMTPSA id j89sm5827866otc.72.2020.06.25.15.39.59 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 25 Jun 2020 15:39:59 -0700 (PDT) Message-ID: <5EF527BE.9000806@gmail.com> Date: Thu, 25 Jun 2020 17:39:58 -0500 From: Jacob Bachmeyer User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Pedro Alves Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jcb62281@gmail.com Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) Pedro Alves wrote: > On 6/25/20 3:43 AM, Jacob Bachmeyer wrote: > >> Pedro Alves wrote: >> >>> What that misses is that in some environments, particularly with slow embedded >>> system boards, running the full testsuite can take _days_. This is all >>> compounded by the fact that the testsuite must be run in a large set of >>> different modes and boards to cover everything for a single target. Even on >>> a modern GNU/Linux system, running all combinations of modes for full coverage >>> can take hours. >>> >>> Having the testrun stop midway because of some specific testcase >>> blowing up can waste a lot of time. >>> >>> The fact that the testsuite can take a while to fully run results in automated >>> testing systems, like GDB's buildbot, struggling to keep up with testing >>> all the commits that land in git. On some buildbot setups, it's just not >>> possible to test all commits. So if some testcase blows up the whole >>> test run, the next time you run the testsuite, it'll likely be that a >>> large number of unrelated commits went into the sources. Losing a large >>> chunk of the testsuite results because of some small typo in one >>> testcase is thus undesirable, because noticing the typo in the first >>> place can take a while, and also you may be stuck with a "last good run" >>> for many of the testcases that gets older and older until >>> the typo is finally fixed. >>> >>> >> I presume buildbot can be easily configured to pass a '--keep_going' option, perhaps as 'RUNTESTFLAGS=--keep_going' if needed? >> > > What's more likely is we'll tweak GDB's Makefile so that make check > enables that option automatically. If GCC does the same, which I would > encourage them to, since they also run the testsuite against systems > that can take hours/days to complete, then I'll get to wonder who the > default is for. :-) > The default is for developers working on testsuites -- and for testing release candidates and releases because if your testcases crash in a release, you have problems. :-) Are you suggesting withdrawing the --keep-going/--no-keep-going options and always generating an UNRESOLVED result, resuming with the next test script, and possibly storing a list of aborted test scripts somewhere to repeat at the very end of a test run with a big warning about test cases that crashed? >> Still, needing a new option to get old behavior is changing the environment interface, so I will make --keep_going the default, at least for the rest of 1.6. That default is likely to change for 1.7. For automated systems like buildbot, passing '--keep_going' should be perfectly reasonable, which is why the option exists. (While the documented options will be changed to use hyphens to align with the rest of the GNU system in 1.7, the current option names will continue to be accepted, since the aliases are obvious.) >> > > Currently with DejaGnu only the case of calling an unknown function stops > the test run AFAIK. Any other tcl error like calling a proc with the wrong > number of arguments, or treating a non-array variable as an array, etc. is > caught by the catch in the runtest proc in runtest.exp, and the testrun > continues. That to me clearly indicates that the original intention was > to catch errors and continue. > > That also gives the tool's $(tool)_finish proc a chance to tear down > correctly. > > It is just that the "unknown" case wasn't thought of. So I argue that not > making unknown proc calls abort the run is a bug fix, and making DejaGnu > abort the run for other errors by default is a behavior change that no real > testsuite built around DejaGnu is asking for. > As it was explained to me, it is the other way around -- the catch in runtest was overlooked when ::unknown was added to catch bugs by aborting if an undefined procedure is called. Is there a ${tool}_finish or do you mean ${tool}_exit? Either way, you make a good point about running cleanup procedures when aborting a test run. >> The entire reason for stopping the testsuite immediately when a Tcl error occurs is that it is undeniable that something is *very* *wrong* when the testsuite stops early. Software testing is hard enough when you know your testsuite is good, but it is far worse when you have bugs in the testsuite masking bugs in the actual program because a test script is crashing and part of the program that is supposed to be tested is not being exercised. >> > > I think it's safe to say that we all understand the general principle, > but IMO that applies more to continuing the same testcase (if somehow that > were possible, like "unknown" suppressing the error), rather than continuing > with a different testcase. > > Continuing the testrun when one testcase errors out does not mask any > bug in the program that is supposed to be tested. Why would it? The > testcase aborts, it doesn't continue. The remaining testcases start > afresh. > Consider a hypothetical case where GDB has exec.exp (which tests starting a debugged process) and attach.exp (which tests attaching to a process). Unknown to the GDB developers, the "attach" command does not work and causes GDB to dump core (oops!) but attach.exp fails with a Tcl error before reaching the point where "attach" is issued to the inferior GDB. If DejaGnu stops immediately, it is obvious that the testsuite is broken, the bug in the testsuite is fixed, and the bug in GDB is found. If DejaGnu sweeps the Tcl error under the proverbial rug (as previous versions did), the bug in the testsuite effectively hides the bug in GDB , because the developers think the "attach" command is being exercised, but those tests are being (almost) silently skipped. Now a bug that obvious will be caught very quickly by users (likely the GDB developers themselves) but obscure regressions are for more insidious, especially in large testsuites such as those for GCC or GDB. If a regression test is being skipped with an easily-missed error, that regression can slip in unnoticed. Would simply turning aborted test scripts into UNRESOLVED results be enough to get them fixed quickly? -- Jacob From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 25 19:55:34 2020 Received: (at 41824) by debbugs.gnu.org; 25 Jun 2020 23:55:34 +0000 Received: from localhost ([127.0.0.1]:41503 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jobiQ-0000Bd-8g for submit@debbugs.gnu.org; Thu, 25 Jun 2020 19:55:34 -0400 Received: from gnashdev.org ([65.183.81.79]:55342 helo=welcomehome.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jobiC-0000BD-QN for 41824@debbugs.gnu.org; Thu, 25 Jun 2020 19:55:32 -0400 Received: from eyes.moongulch.net (dsl.moongulch.net [216.160.182.223]) (authenticated bits=0) by welcomehome.org (8.15.2/8.15.2) with ESMTPSA id 05PNrvCY006106 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Thu, 25 Jun 2020 17:54:17 -0600 Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case To: Pedro Alves , jcb62281@gmail.com References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> From: Rob Savoye Autocrypt: addr=rob@senecass.com; prefer-encrypt=mutual; keydata= mQGiBEezIb8RBACsVR1XKKx2+bmShsPssv98frN9CRwnpRwx+aZhCL2FbpxIn00FkpCg8lHg ftFKxNytbyIdXvqKKVQFd75wIsszjAs07M5dTRfWkunkQ8FR6RKRJuzNkGG6YR7k7YDE3f1k 5K/k54Y1aTgP2rhsn10bX537VV3gIRpknNyfNCNL8wCglCTxTOuqiVGTG/gOTo96f94Q0/kD /2EDQFYYUx17rxlyYJIvF7cfYvkw7nvBMZn0u/kDKirwPs6Sx2o30TxvvgjA55HTN6UNVwiL Iglr3DyBgi6LGD7RAxMG15FnK4xx28wmu8EXYDJuNqbsy+PoPs1pMdlsVERIJfzDmG65/pAD LFRymDL2DgnU9v/pH0HrLwY34A5kA/9Ywdn7G3FFR/oR/WGEavzM+mvYq2GRSDLEobaGRdPp tAEUpyMGsCVUG8TIWRA9B6CHD0YTh2kLKIH7QK6/3fcnROCrCd3/dIYwkaQV0Icrl17eUpLX 1xBmbR7ji3inCd58uceQBIksKdrcqeFB5yB6RBWPpgDELg020x7x6UhKSLkCDQRHsyG/EAgA uU0hNVK0CfKhFVzUBCcSXCn+Ck7sh7oFYdMKJyg6t3r56sWtgzlSuq1k1jwaDlaOSwF9DO0z KHSdJ2KvmoheBGxbUfSguUbgGC8XvERuaMHaHaCIal2D6MdisbL5w4QGtKmTSGjlqZj1tCS7 s6LBHrQAkVkBSBSv12sl4wz8tbL1aublznu/5LXusg2wmae+ynRbubnggAdDGIJv+AQsbYL+ yukEZkFT+AvJzCENFEaBxRj6B7obBx9eSKcRgv0ON0CghiM2iYN6iuUzhcaBl+v9+4XR9tDm vR2s/CBP0A14yNRXQ+B9HG/kzmSLSHttY9jiySO91YbRsrWZbn2wxwAECwgAn5vmpCw2T4/H gu14o9qnY4Lr0PPp9uCw9T7oaZDdjibZSvcgc9WabUuSP6BOlcpE84d92gGJn0mTvuTsSf6Z yWNYj0Rdt71MZbstU/lUfXmOBS3bwhas0LnH4zmoOZJ2/IvZT8hbfrld3eQCeT347rkBt2vM g4RZEXutvpQUDtT6HO+v202/hHgwlLZppy/QgTaUAZcYOoIw72/OR6Mb2y7gROJyYEAw/DJD Jf0KXIn5kpsrSDiEJHTW7YBDx1opnp+FwhRBUb2xGr7ldZcZxrOmjEHMAHzAi6vcG4gdlhZ4 /UwTkycgftroQ+nNhyhwY9UXdD02ab6eZzjIE9Pc/YhJBBgRAgAJBQJHsyG/AhsMAAoJEFC5 oRCgttP+kZcAnjB3S9tuyK9PqDKzEcoovKs/LowRAJ9griQbLMDG7R9HvCLOfvg9iRGH1A== Organization: Seneca Software & Solar, Inc. Message-ID: <32b06c6c-560b-3637-0c0d-84248357e591@senecass.com> Date: Thu, 25 Jun 2020 17:53:57 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.senecass.com X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 6/25/20 10:01 AM, Pedro Alves wrote: >> It could be changed, but those are much larger changes to the code and overall behavior than simply adding an UNRESOLVED result to report the failure, and if I understand correctly, the presence of UNRESOLVED results marks the test results as invalid or partially-valid at best, which is correct if a test script failed with a Tcl error. Adding an additional call makes it harder to diff the .sum files. Not a huge big deal, but still... UNRESOLVED says a human has to look at this test case output and decide, the actual error should be in the .log file. UNTESTED means something went wrong with the test, and the test is invalid, which is probably what you'd want as the test state for a Tcl error. > Currently with DejaGnu only the case of calling an unknown function stops > the test run AFAIK. Any other tcl error like calling a proc with the wrong > number of arguments, or treating a non-array variable as an array, etc. is > caught by the catch in the runtest proc in runtest.exp, and the testrun > continues. That to me clearly indicates that the original intention was > to catch errors and continue. The original intent was to catch some errors, nd ignore others. :-) Catch is used for errors that can be ignored or worked around. Unknown was basically to catch programming errors, since as noted, there are many paths though the code as you change options or environments. They'd get lost in the output unless it aborted. > It is just that the "unknown" case wasn't thought of. So I argue that not > making unknown proc calls abort the run is a bug fix, and making DejaGnu > abort the run for other errors by default is a behavior change that no real > testsuite built around DejaGnu is asking for. I think the default should be to abort on a unknown Tcl error, and have an option to ignore them and continue. As a person that has run many long test runs, I agree I don't want some errors to abort the test run. But to be honest, a dependable test run should mean no bugs in the test framework. A buggy test framework means you can trust the test results. Aborting was for use during debugging of test framework code itself, so supposedly you should should never have an Tcl error during a test run. There are times when even during debugging you'd want it to continue, and then collect all the bugs later to fix in batch mode. - rob - From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 25 21:53:58 2020 Received: (at 41824) by debbugs.gnu.org; 26 Jun 2020 01:53:58 +0000 Received: from localhost ([127.0.0.1]:41613 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jodYz-00037c-VO for submit@debbugs.gnu.org; Thu, 25 Jun 2020 21:53:58 -0400 Received: from mail-oi1-f194.google.com ([209.85.167.194]:43007) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jodYx-00037P-Fb for 41824@debbugs.gnu.org; Thu, 25 Jun 2020 21:53:56 -0400 Received: by mail-oi1-f194.google.com with SMTP id s21so6747708oic.9 for <41824@debbugs.gnu.org>; Thu, 25 Jun 2020 18:53:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=qJEaFYLTES4Mv8c5z0y6UJ3mtnrxhhCMrvwFki3qvFI=; b=d9tjpPGc22fUMEf0N5MTO3DOkMPdeeYZFaslbyjORcBVBST6DYb52joZqlzD6kFkBG pkT1yu36VMqvcqKlVXQOYAoW/jxOkaM/1v4gnEKBCcAXI9ELFtyeL5zgimxtEf3+DU6P cR3OLaUT2AjjVcfs033FlAvvb05vm/SLjXOqp32VD2lgHCU3gMCwidiW+8wIFjLOzf3r dIkAbjc6mprIggU0TKy1/26mKukxFxy4Z5VIzbuE/x2K77WoZmjKuNtL8x9kEgyHwudZ QGUrrBZiL5kSX+TVZcWEcp+8JEhxQYOpoJA4gGL+MErymBhiNk52URGqtBAyyACh9T1y MljQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:reply-to:user-agent :mime-version:to:cc:subject:references:in-reply-to :content-transfer-encoding; bh=qJEaFYLTES4Mv8c5z0y6UJ3mtnrxhhCMrvwFki3qvFI=; b=EMwTM8rE9mPap91B5kjDV86M0TZAsmsp9y/uTdKD7GBMRBhbK+16UqurXA42MnpYIE 1SY8ERYAm8oiSmnqc1/ioyavEWlX++6yRGaT9lL2Zcs+nM1ebw+ufMKGsMTMTODk+dOZ o7n2TZqVNvKDa85Yq64G4exkcnSfins0/ywO7SZFZ2txEvJOrR3KAGySFgCWebcra0Co JsFO4jzx+Wd1MawZtF+ivHX2AGoePxAnANBnr7lBcZXBFplm9Wzlc5iSxN+gzVfwI04S zx78aUQ2MwIIpsOSM7UvGgoDr4PBLFjnFhIkNWIlWw0B0FPrZRcStz2kJoUGIDVqnf46 GSNQ== X-Gm-Message-State: AOAM530BOyT7LlLLWadeCqovpeP3HwzKuXjlqOXaN061upTD3d+JwpkS D1IuTMSjpFiex7eB44ndSJc= X-Google-Smtp-Source: ABdhPJzDmjUXsR7WaLCjJYFfb7dXPxbRywK73OAdnikFxY7aDJVySbTtoBlkzMyXgBdTSZMmT5s/Xw== X-Received: by 2002:aca:5d46:: with SMTP id r67mr697156oib.143.1593136429770; Thu, 25 Jun 2020 18:53:49 -0700 (PDT) Received: from [192.168.2.42] (adsl-70-133-144-251.dsl.ablntx.sbcglobal.net. [70.133.144.251]) by smtp.gmail.com with ESMTPSA id 7sm5884408oti.25.2020.06.25.18.53.48 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 25 Jun 2020 18:53:48 -0700 (PDT) Message-ID: <5EF5552A.6040001@gmail.com> Date: Thu, 25 Jun 2020 20:53:46 -0500 From: Jacob Bachmeyer User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Rob Savoye Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <32b06c6c-560b-3637-0c0d-84248357e591@senecass.com> In-Reply-To: <32b06c6c-560b-3637-0c0d-84248357e591@senecass.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 41824 Cc: Pedro Alves , Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jcb62281@gmail.com Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) Rob Savoye wrote: > On 6/25/20 10:01 AM, Pedro Alves wrote: > > Adding an additional call makes it harder to diff the .sum files. Not > a huge big deal, but still... UNRESOLVED says a human has to look at > this test case output and decide, the actual error should be in the .log > file. UNTESTED means something went wrong with the test, and the test is > invalid, which is probably what you'd want as the test state for a Tcl > error. > The original rationale for using UNRESOLVED here was that DejaGnu converts a test result to UNRESOLVED if too many errors or warnings were produced. The DejaGnu manual indicates (section "A POSIX Compliant Test Framework") that UNRESOLVED is correct for a test where execution was interrupted or was set up incorrectly. UNTESTED is specifically listed as a placeholder for an as-yet-unwritten testcase. A "typical" GDB testsuite run has almost a hundred UNTESTED results but (without this patch) zero UNRESOLVED results. To me, this seems that UNRESOLVED is correct here, or that the manual has an error. >> Currently with DejaGnu only the case of calling an unknown function stops >> the test run AFAIK. Any other tcl error like calling a proc with the wrong >> number of arguments, or treating a non-array variable as an array, etc. is >> caught by the catch in the runtest proc in runtest.exp, and the testrun >> continues. That to me clearly indicates that the original intention was >> to catch errors and continue. >> > > The original intent was to catch some errors, nd ignore others. :-) > Catch is used for errors that can be ignored or worked around. Unknown > was basically to catch programming errors, since as noted, there are > many paths though the code as you change options or environments. They'd > get lost in the output unless it aborted. > So continuing with the next test script could be reasonable if the error (or at least an indication that the error occurred) is "replayed" at the end of the test run? >> It is just that the "unknown" case wasn't thought of. So I argue that not >> making unknown proc calls abort the run is a bug fix, and making DejaGnu >> abort the run for other errors by default is a behavior change that no real >> testsuite built around DejaGnu is asking for. >> > > I think the default should be to abort on a unknown Tcl error, and > have an option to ignore them and continue. As a person that has run > many long test runs, I agree I don't want some errors to abort the test > run. But to be honest, a dependable test run should mean no bugs in the > test framework. A buggy test framework means you can trust the test results. > > Aborting was for use during debugging of test framework code itself, > so supposedly you should should never have an Tcl error during a test > run. There are times when even during debugging you'd want it to > continue, and then collect all the bugs later to fix in batch mode. > This succinctly states the original motivation for having the --keep-going option, but having "make check" always pass --keep-going defeats the purpose. -- Jacob From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 26 05:53:50 2020 Received: (at 41824) by debbugs.gnu.org; 26 Jun 2020 09:53:50 +0000 Received: from localhost ([127.0.0.1]:41848 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jol3N-0000NV-NI for submit@debbugs.gnu.org; Fri, 26 Jun 2020 05:53:50 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:37266 helo=us-smtp-1.mimecast.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jol3I-0000NF-Fg for 41824@debbugs.gnu.org; Fri, 26 Jun 2020 05:53:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1593165223; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tR+61ZK3GBPQjeHncs1jqCoO65SIHBZZibYJJzBVANc=; b=hHA1y/rD0Ufw9wiZtcE+jhQUTP2omVqjA/bfmVY7yB3EYIaw+STY2/v9XreG2FzFAgBcIQ pYvFe++SpGgKpEzSSCUoSj2HoY67xjJ5MPpYx9cn3RW7G2UNpMPk/knoiDKub2vR1WQFkY cf2IY82WIzt7uC3upJ9xmdnLjoDv3Z0= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-160-1LEG3uBFN5KgoEfb1LQoPA-1; Fri, 26 Jun 2020 05:53:39 -0400 X-MC-Unique: 1LEG3uBFN5KgoEfb1LQoPA-1 Received: by mail-wm1-f69.google.com with SMTP id h25so8974080wmb.0 for <41824@debbugs.gnu.org>; Fri, 26 Jun 2020 02:53:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=tR+61ZK3GBPQjeHncs1jqCoO65SIHBZZibYJJzBVANc=; b=ObVcmE4B5O+NBuw5oF+DPxCVLaHnbRoPpjXE0VnEAcpPUuVVRzNcPE01vK9jgOU62P ELEa2nb2x5PiIwDKI1JYgiFGuae2uK296/yivXjRWwx4hf13rgYZ+JeAbTjHKS6jqBrI q1umJ2IByIlWaFUeXiJjSYElsBmiLtCnMMp34nNw8iVLKPhoFA8JlkhpW6pCrNZyMBEs hEl801aZoVoUKtNfFtbpTScTR3Z4LlYTl2fq71hhZHulXtESoau7TMrpRInAuUkjBSCh 5c4sn0SFgx1kBY1kQ6J/2wI2a2SOLd+XiugVUzrtuRu44XysXkKMu9SvhNYFCTMli52o ebOg== X-Gm-Message-State: AOAM532nzmkDsSDUgUOynu2ZXpVOyoGDCEsGVnikoBleRC77Qn5YIemu gaqHAqLYqJQmh0WsKyTPjn9Z+Bytbt4d+XZUxDEBiz5YGYPKXjzEaXDdK6Obu4J3xJ8Okn9J1kY WHf5Im8LnF/3cbEI= X-Received: by 2002:a1c:f301:: with SMTP id q1mr2532725wmq.110.1593165218409; Fri, 26 Jun 2020 02:53:38 -0700 (PDT) X-Google-Smtp-Source: ABdhPJymExPCHCC47U3gKz88lUjgoyzEy9XAefn+rajf+ZTluSDbLG1T8vZ6px2PiHOi/9HlOGtXiw== X-Received: by 2002:a1c:f301:: with SMTP id q1mr2532693wmq.110.1593165217983; Fri, 26 Jun 2020 02:53:37 -0700 (PDT) Received: from ?IPv6:2001:8a0:f922:c400:56ee:75ff:fe8d:232b? ([2001:8a0:f922:c400:56ee:75ff:fe8d:232b]) by smtp.gmail.com with ESMTPSA id 133sm18507776wme.5.2020.06.26.02.53.36 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 26 Jun 2020 02:53:37 -0700 (PDT) Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case To: jcb62281@gmail.com References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <5EF527BE.9000806@gmail.com> From: Pedro Alves Message-ID: Date: Fri, 26 Jun 2020 10:53:34 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <5EF527BE.9000806@gmail.com> Content-Language: en-US X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 6/25/20 11:39 PM, Jacob Bachmeyer wrote: > Pedro Alves wrote: >> On 6/25/20 3:43 AM, Jacob Bachmeyer wrote: >>   >>> Pedro Alves wrote: >>>     >>>> What that misses is that in some environments, particularly with slow embedded >>>> system boards, running the full testsuite can take _days_.   This is all >>>> compounded by the fact that the testsuite must be run in a large set of >>>> different modes and boards to cover everything for a single target.  Even on >>>> a modern GNU/Linux system, running all combinations of modes for full coverage >>>> can take hours. >>>> >>>> Having the testrun stop midway because of some specific testcase >>>> blowing up can waste a lot of time. >>>> >>>> The fact that the testsuite can take a while to fully run results in automated >>>> testing systems, like GDB's buildbot, struggling to keep up with testing >>>> all the commits that land in git.  On some buildbot setups, it's just not >>>> possible to test all commits.  So if some testcase blows up the whole >>>> test run, the next time you run the testsuite, it'll likely be that a >>>> large number of unrelated commits went into the sources.  Losing a large >>>> chunk of the testsuite results because of some small typo in one >>>> testcase is thus undesirable, because noticing the typo in the first >>>> place can take a while, and also you may be stuck with a "last good run" >>>> for many of the testcases that gets older and older until >>>> the typo is finally fixed. >>>>         >>> I presume buildbot can be easily configured to pass a '--keep_going' option, perhaps as 'RUNTESTFLAGS=--keep_going' if needed? >>>     >> >> What's more likely is we'll tweak GDB's Makefile so that make check >> enables that option automatically.  If GCC does the same, which I would >> encourage them to, since they also run the testsuite against systems >> that can take hours/days to complete, then I'll get to wonder who the >> default is for.  :-) >>   > > The default is for developers working on testsuites -- and for testing release candidates and releases because if your testcases crash in a release, you have problems.  :-) > > Are you suggesting withdrawing the --keep-going/--no-keep-going options Yes, but if you would like to add it, it's OK with me, though then you can say that I'm more discussing the default. But really I'm more worried about making sure that the people maintaining the GDB testsuite (me included), and the DejaGnu people understand each other's needs better. > and always generating an UNRESOLVED result, > resuming with the next test script, and possibly storing a list of aborted test scripts somewhere Yes (with --keep-going if you insist on having the option), but note that I suggested storing the list of ERRORs and WARNINGs. > to repeat at the very end of a test run with a big warning about test cases that crashed? I did not suggest to repeat anything. > >>> Still, needing a new option to get old behavior is changing the environment interface, so I will make --keep_going the default, at least for the rest of 1.6.  That default is likely to change for 1.7.  For automated systems like buildbot, passing '--keep_going' should be perfectly reasonable, which is why the option exists.  (While the documented options will be changed to use hyphens to align with the rest of the GNU system in 1.7, the current option names will continue to be accepted, since the aliases are obvious.) >>>     >> >> Currently with DejaGnu only the case of calling an unknown function stops >> the test run AFAIK.  Any other tcl error like calling a proc with the wrong >> number of arguments, or treating a non-array variable as an array, etc. is >> caught by the catch in the runtest proc in runtest.exp, and the testrun >> continues.  That to me clearly indicates that the original intention was >> to catch errors and continue. >> >> That also gives the tool's $(tool)_finish proc a chance to tear down >> correctly. >> >> It is just that the "unknown" case wasn't thought of.  So I argue that not >> making unknown proc calls abort the run is a bug fix, and making DejaGnu >> abort the run for other errors by default is a behavior change that no real >> testsuite built around DejaGnu is asking for. >>   > > As it was explained to me, it is the other way around -- the catch in runtest was overlooked when ::unknown was added to catch bugs by aborting if an undefined procedure is called. > > Is there a ${tool}_finish or do you mean ${tool}_exit?  Either way, you make a good point about running cleanup procedures when aborting a test run. Look at proc runtest in runtest.exp, it calls ${tool}_init, sources the testcase under catch (using ERROR:, so here you could run a tally of such caught errors), and then calls ${tool}_finish. > >>> The entire reason for stopping the testsuite immediately when a Tcl error occurs is that it is undeniable that something is *very* *wrong* when the testsuite stops early.  Software testing is hard enough when you know your testsuite is good, but it is far worse when you have bugs in the testsuite masking bugs in the actual program because a test script is crashing and part of the program that is supposed to be tested is not being exercised. >>>     >> >> I think it's safe to say that we all understand the general principle, >> but IMO that applies more to continuing the same testcase (if somehow that >> were possible, like "unknown" suppressing the error), rather than continuing >> with a different testcase. >> >> Continuing the testrun when one testcase errors out does not mask any >> bug in the program that is supposed to be tested.  Why would it?  The >> testcase aborts, it doesn't continue.  The remaining testcases start >> afresh. >>   > > Consider a hypothetical case where GDB has exec.exp (which tests starting a debugged process) and attach.exp (which tests attaching to a process).  Unknown to the GDB developers, the "attach" command does not work and causes GDB to dump core (oops!) but attach.exp fails with a Tcl error before reaching the point where "attach" is issued to the inferior GDB.  If DejaGnu stops immediately, it is obvious that the testsuite is broken, the bug in the testsuite is fixed, and the bug in GDB is found.  If DejaGnu sweeps the Tcl error under the proverbial rug (as previous versions did), the bug in the testsuite effectively hides the bug in GDB , because the developers think the "attach" command is being exercised, but those tests are being (almost) silently skipped. First, that scenario doesn't normally cause a Tcl error. If GDB trips an internal error, the GDB testsuite catches it with a FAIL. If GDB just dies (whether it dumps core or not) the GDB testsuite catches that with UNRESOLVED. Also, even if those weren't already considered, the developer won't think the testcase is just passing correctly, because being a good GDB developer, he knows he needs to diff the new gdb.sum file against a previous run's, and sees that some PASSes are missing, and some ERROR: tcl error showed up in their stead. Nobody should rely on just looking at the summary. Everyone _must_ diff between a previous run and the current run. Now some people may use test result diffing scripts that forget to look at ERROR: lines in .sum. Those would be buggy, but then again, if those ERRORs were accounted for in the DejaGnu summary, they would be harder to miss. And again, people run the GDB testsuite in parallel mode, so your assumption that the testsuite would stop immediately anyway doesn't hold. One of the parallel runtest invocations bombs out, while the others keep running, so by the time you get back the prompt, other testcases already executed, and you don't see the ERROR in the current terminal anymore. You'll need to look at the testsuite .sum .log files anyway. Only, you will find out that one testcase bombed out, 50000 tests ran, but 1000 others didn't run because of that one testcase. And now you're missing those 1000 results, which maybe included those which you were actually interested in, because you were doing a builbot try run across a number of systems. Now, when running the testsuite against slow embedded boards, then you won't normally be using the parallel mode, the board just can't support multiple parallel connections. In that case, you set the testsuite running, and go on your merry way for a couple hours or 20. When you come back, maybe the next day, you notice that the testsuite actually stopped running 30 mins in, because someone changed the testsuite upstream just before you rebased in way that caused a tcl error for you in some testcase. Oops. You just lost a day. Now you wish you had remembered to type --keep-going, but it's not the default, and nobody ever writes Tcl bugs, so you didn't think of it before. That's why I think --keep-going should be the default. When you're developing a new testcase, you'll run that testcase in isolation a number of times. It's very hard to miss any silly error then -- there's no need to optimize for this use case by default, IMO. > > Now a bug that obvious will be caught very quickly by users (likely the GDB developers themselves) but obscure regressions are for more insidious, especially in large testsuites such as those for GCC or GDB.  If a regression test is being skipped with an easily-missed error, that regression can slip in unnoticed. > > Would simply turning aborted test scripts into UNRESOLVED results be enough to get them fixed quickly? Yes. (Though I still think a separate count for ERROR (and WARNING) would be better.) Thanks, Pedro Alves From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 26 09:41:30 2020 Received: (at 41824) by debbugs.gnu.org; 26 Jun 2020 13:41:30 +0000 Received: from localhost ([127.0.0.1]:42042 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1joobh-00020R-Bx for submit@debbugs.gnu.org; Fri, 26 Jun 2020 09:41:30 -0400 Received: from mail-wm1-f49.google.com ([209.85.128.49]:39715) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jolHQ-0000pI-8f for 41824@debbugs.gnu.org; Fri, 26 Jun 2020 06:08:23 -0400 Received: by mail-wm1-f49.google.com with SMTP id t194so8816166wmt.4 for <41824@debbugs.gnu.org>; Fri, 26 Jun 2020 03:08:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=8dSKWkRixYd3szU1Ae0jNC702uezeMD9iupaUaqOwdk=; b=S6bKI0oH2Gv+BRDB/qEkLQk+fG/N9+SWV/ZcRjE4Pef/TgY2cUiuqEhPqasZCpJt8q 9h19AWhEkPeXwkgJHoIyqzojU4fNKHBpCtoUeIwa4ep9YrCln2IAljKR3m8+FcP+4oJl qPn8ec38M46Tcek/hhKqHhR8drJsDfV730E69mbUq4/Lemj6gHRthIEYJR+dXHhy8exr xxRU96Xf4WqsW8p7PuPWtHyQpDo5HIHZoKGoJ1hTqluSzeh2CVND5pl2Gq9hv5pTfcNy zeDrhUSpUxTGtK5WK+kvx9AHN9T6RvKl74x6ucmShV9EiidS0CMLJbmgNNRpp1Sn4KPm 1ayw== X-Gm-Message-State: AOAM5311hIRjTOAKllARYUEmlHmZKCi02a1mzAN2IR1alpmZFrHNeCdE XwKOjvNMMKNycILRYP4vpbZivzz1wnXI9g== X-Google-Smtp-Source: ABdhPJzXoyrsOvCZKVoOMEBIV+D0XAUQk+CATFi7caVT1/kjOxOqOweovR5qz2YHDVMjuIkKZA3YMQ== X-Received: by 2002:a1c:3c82:: with SMTP id j124mr2723940wma.155.1593166093352; Fri, 26 Jun 2020 03:08:13 -0700 (PDT) Received: from ?IPv6:2001:8a0:f922:c400:56ee:75ff:fe8d:232b? ([2001:8a0:f922:c400:56ee:75ff:fe8d:232b]) by smtp.gmail.com with ESMTPSA id l190sm16313099wml.12.2020.06.26.03.08.11 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 26 Jun 2020 03:08:12 -0700 (PDT) Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case To: Pedro Alves , jcb62281@gmail.com References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <5EF527BE.9000806@gmail.com> From: Pedro Alves Message-ID: Date: Fri, 26 Jun 2020 11:08:11 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 41824 X-Mailman-Approved-At: Fri, 26 Jun 2020 09:41:24 -0400 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 6/26/20 10:53 AM, Pedro Alves wrote: > >> to repeat at the very end of a test run with a big warning about test cases that crashed? > I did not suggest to repeat anything. > I misunderstood you here -- I somehow thought that that by "repeat" you meant re-running the testcase. Still, I would not suggest to repeat the error. Let me clarify -- with this: ~~~~ diff --git c/gdb/testsuite/gdb.base/break.exp w/gdb/testsuite/gdb.base/break.exp index 8c7ce42d805..056252c0677 100644 --- c/gdb/testsuite/gdb.base/break.exp +++ w/gdb/testsuite/gdb.base/break.exp @@ -15,6 +15,9 @@ # This file was written by Rob Savoye. (rob@cygnus.com) +set a 1 +set a(1) 1 + ~~~~ You get, if you run two testcases, and the other one manages to run first: $ make check TESTS="*/break.exp */schedlock.exp" make[1]: Entering directory '/home/pedro/brno/pedro/gdb/mygit/build/gdb/testsuite' Nothing to be done for all... make check-single make[2]: Entering directory '/home/pedro/brno/pedro/gdb/mygit/build/gdb/testsuite' rootme=`pwd`; export rootme; srcdir=/home/pedro/gdb/mygit/src/gdb/testsuite ; export srcdir ; EXPECT=`if [ "${READ1}" != "" ] ; then echo ${rootme}/expect-read1; elif [ -f ${rootme}/../../expect/expect ] ; then echo ${rootme}/../../expect/expect ; else echo expect ; fi` ; export EXPECT ; EXEEXT= ; export EXEEXT ; LD_LIBRARY_PATH=$rootme/../../expect:$rootme/../../libstdc++:$rootme/../../tk/unix:$rootme/../../tcl/unix:$rootme/../../bfd:$rootme/../../opcodes:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; if [ -f ${rootme}/../../expect/expect ] ; then TCL_LIBRARY=${srcdir}/../../tcl/library ; export TCL_LIBRARY ; fi ; runtest --status gdb.base/break.exp gdb.threads/schedlock.exp NOTE: Dejagnu's default_target_compile is missing support for Go, using local override NOTE: Dejagnu's default_target_compile is missing support for Rust, using local override Test run by pedro on Fri Jun 26 11:03:38 2020 Native configuration is x86_64-pc-linux-gnu === gdb tests === Schedule of variations: unix Running target unix Using /usr/share/dejagnu/baseboards/unix.exp as board description file for target. Using /usr/share/dejagnu/config/unix.exp as generic interface file for target. Using /home/pedro/gdb/mygit/src/gdb/testsuite/config/unix.exp as tool-and-target-specific interface file. Running /home/pedro/gdb/mygit/src/gdb/testsuite/gdb.threads/schedlock.exp ... Running /home/pedro/gdb/mygit/src/gdb/testsuite/gdb.base/break.exp ... ERROR: tcl error sourcing /home/pedro/gdb/mygit/src/gdb/testsuite/gdb.base/break.exp. ERROR: can't set "a(1)": variable isn't array while executing "set a(1) 1" (file "/home/pedro/gdb/mygit/src/gdb/testsuite/gdb.base/break.exp" line 19) invoked from within "source /home/pedro/gdb/mygit/src/gdb/testsuite/gdb.base/break.exp" ("uplevel" body line 1) invoked from within "uplevel #0 source /home/pedro/gdb/mygit/src/gdb/testsuite/gdb.base/break.exp" invoked from within "catch "uplevel #0 source $test_file_name"" === gdb Summary === # of expected passes 208 /home/pedro/brno/pedro/gdb/mygit/build/gdb/gdb version 10.0.50.20200625-git -nw -nx -data-directory /home/pedro/brno/pedro/gdb/mygit/build/gdb/testsuite/../data-directory make[2]: *** [Makefile:206: check-single] Error 2 make[2]: Leaving directory '/home/pedro/brno/pedro/gdb/mygit/build/gdb/testsuite' make[1]: *** [Makefile:159: check] Error 2 make[1]: Leaving directory '/home/pedro/brno/pedro/gdb/mygit/build/gdb/testsuite' make: *** [Makefile:1628: check] Error 2 ~~~~ Now, you already have an indication that something went wrong because make existed with error. But people may miss that. The issue for me is that the "gdb Summary" tally did not mention the aborted testcase. I would like to see something like this instead: # of expected passes 208 # of nasty OMG FIX! tcl errors 1 :-) Your UNRESOLVED is of course better than the status quo. Thanks, Pedro Alves From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 26 11:31:58 2020 Received: (at 41824) by debbugs.gnu.org; 26 Jun 2020 15:31:58 +0000 Received: from localhost ([127.0.0.1]:42941 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1joqKc-0004rm-0I for submit@debbugs.gnu.org; Fri, 26 Jun 2020 11:31:58 -0400 Received: from gnashdev.org ([65.183.81.79]:59018 helo=welcomehome.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1joqKO-0004rP-IE for 41824@debbugs.gnu.org; Fri, 26 Jun 2020 11:31:57 -0400 Received: from eyes.moongulch.net (dsl.moongulch.net [216.160.182.223]) (authenticated bits=0) by welcomehome.org (8.15.2/8.15.2) with ESMTPSA id 05QFUMon027044 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Fri, 26 Jun 2020 09:30:42 -0600 Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case To: Pedro Alves , jcb62281@gmail.com References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <5EF527BE.9000806@gmail.com> From: Rob Savoye Autocrypt: addr=rob@senecass.com; prefer-encrypt=mutual; keydata= mQGiBEezIb8RBACsVR1XKKx2+bmShsPssv98frN9CRwnpRwx+aZhCL2FbpxIn00FkpCg8lHg ftFKxNytbyIdXvqKKVQFd75wIsszjAs07M5dTRfWkunkQ8FR6RKRJuzNkGG6YR7k7YDE3f1k 5K/k54Y1aTgP2rhsn10bX537VV3gIRpknNyfNCNL8wCglCTxTOuqiVGTG/gOTo96f94Q0/kD /2EDQFYYUx17rxlyYJIvF7cfYvkw7nvBMZn0u/kDKirwPs6Sx2o30TxvvgjA55HTN6UNVwiL Iglr3DyBgi6LGD7RAxMG15FnK4xx28wmu8EXYDJuNqbsy+PoPs1pMdlsVERIJfzDmG65/pAD LFRymDL2DgnU9v/pH0HrLwY34A5kA/9Ywdn7G3FFR/oR/WGEavzM+mvYq2GRSDLEobaGRdPp tAEUpyMGsCVUG8TIWRA9B6CHD0YTh2kLKIH7QK6/3fcnROCrCd3/dIYwkaQV0Icrl17eUpLX 1xBmbR7ji3inCd58uceQBIksKdrcqeFB5yB6RBWPpgDELg020x7x6UhKSLkCDQRHsyG/EAgA uU0hNVK0CfKhFVzUBCcSXCn+Ck7sh7oFYdMKJyg6t3r56sWtgzlSuq1k1jwaDlaOSwF9DO0z KHSdJ2KvmoheBGxbUfSguUbgGC8XvERuaMHaHaCIal2D6MdisbL5w4QGtKmTSGjlqZj1tCS7 s6LBHrQAkVkBSBSv12sl4wz8tbL1aublznu/5LXusg2wmae+ynRbubnggAdDGIJv+AQsbYL+ yukEZkFT+AvJzCENFEaBxRj6B7obBx9eSKcRgv0ON0CghiM2iYN6iuUzhcaBl+v9+4XR9tDm vR2s/CBP0A14yNRXQ+B9HG/kzmSLSHttY9jiySO91YbRsrWZbn2wxwAECwgAn5vmpCw2T4/H gu14o9qnY4Lr0PPp9uCw9T7oaZDdjibZSvcgc9WabUuSP6BOlcpE84d92gGJn0mTvuTsSf6Z yWNYj0Rdt71MZbstU/lUfXmOBS3bwhas0LnH4zmoOZJ2/IvZT8hbfrld3eQCeT347rkBt2vM g4RZEXutvpQUDtT6HO+v202/hHgwlLZppy/QgTaUAZcYOoIw72/OR6Mb2y7gROJyYEAw/DJD Jf0KXIn5kpsrSDiEJHTW7YBDx1opnp+FwhRBUb2xGr7ldZcZxrOmjEHMAHzAi6vcG4gdlhZ4 /UwTkycgftroQ+nNhyhwY9UXdD02ab6eZzjIE9Pc/YhJBBgRAgAJBQJHsyG/AhsMAAoJEFC5 oRCgttP+kZcAnjB3S9tuyK9PqDKzEcoovKs/LowRAJ9griQbLMDG7R9HvCLOfvg9iRGH1A== Organization: Seneca Software & Solar, Inc. Message-ID: <6f1ae2a8-7719-0a7d-7a7a-3c6dcdf25a33@senecass.com> Date: Fri, 26 Jun 2020 09:30:22 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.senecass.com X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 6/26/20 3:53 AM, Pedro Alves wrote: > you can say that I'm more discussing the default. But really I'm more > worried about making sure that the people maintaining the GDB > testsuite (me included), and the DejaGnu people understand each > other's needs better. As the author of the GDB testsuite, I understand quite well how it needs to work, or at least used to. Testing GDB was the #1 requirement that lead to creating DejaGnu. Both have evolved considerably over the years, so working together on improving the current implementation is a good idea. What I get out of this discussion, which is a good one, is how DejaGnu should handle errors. Unknown is one case, but there others. Whether something is a warning, error, UNRESOLVED, UNTESTED, UNSUPPORTED, etc... is often up to the developer. Maybe what needs to be improved is better tracking of issues in a test run. > at ERROR: lines in .sum. Those would be buggy, but then again, if > those ERRORs were accounted for in the DejaGnu summary, they would be > harder to miss. Right, one reason I started on database support with better analysis tools. Diffing text files has limited functionality. > That's why I think --keep-going should be the default. When > you're developing a new testcase, you'll run that testcase in > isolation a number of times. It's very hard to miss any > silly error then -- there's no need to optimize for this use > case by default, IMO. I personally don't consider development or debugging done till I have done a full testrun native and cross to catch the errors running a test in isolation misses. Yes, I know this can take days, but I prefer that to assuming a validation testrun is not going to have problems. I'll note that there is a $HOME/.dejagnurc, where you can set the default to whatever you want. :-) - rob- From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 26 18:37:35 2020 Received: (at 41824) by debbugs.gnu.org; 26 Jun 2020 22:37:35 +0000 Received: from localhost ([127.0.0.1]:43286 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jowyV-0000Uj-3Q for submit@debbugs.gnu.org; Fri, 26 Jun 2020 18:37:35 -0400 Received: from mail-ot1-f65.google.com ([209.85.210.65]:38849) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jowyS-0000UO-BQ for 41824@debbugs.gnu.org; Fri, 26 Jun 2020 18:37:33 -0400 Received: by mail-ot1-f65.google.com with SMTP id 64so10074799oti.5 for <41824@debbugs.gnu.org>; Fri, 26 Jun 2020 15:37:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=KTjxkaeArnHnN4tzYD7ismn3Z5oeWyIE9gPF4KA9DPU=; b=OBb7pLV+GybCBXlNeCF6wXSeYQaC8M7XT/ZHyLocVZF0Wz4+ee1f+/aM/1Ei5rEllP cgGQIpf8gbTM6wHvSJ9CdNAphtyNgrDgiRC9L0xPGENHR7/sP2x0YkU7GuKwcE526WYQ kQcR0LQwsvoAXoAYzflLXu4I/nrQ2ovfmO7qo4muQ+4QyMq1z0LxoRspDVIhV1dJDk8V osy8Zexvv602vItwVe6ikSEAhMnOi7aDpzvHSaypNtoXeZmx1PE6ETamsNnlUBNSllMX fM7qajUlzyanwTiITCazb81yUag7lAuyojiEcac0PtRJwC32Iwa8mvIX63cXCxbkIiW8 nrhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:reply-to:user-agent :mime-version:to:cc:subject:references:in-reply-to :content-transfer-encoding; bh=KTjxkaeArnHnN4tzYD7ismn3Z5oeWyIE9gPF4KA9DPU=; b=ddt5BLpNw3K8jStH+2OLI1eWvTYGTCc5B2yKcrWCKK6N/h1GPb9E6crKQem5vzKGBY l6TytDNb509wTHQ7ZVpuz0mQsCbJQL8h+7pzc8qceM6cnmzMGndej5hQRRqfgs7iNkvV ETPcajXQySETxcHd2BFZLDOWcuOihZdjZOxT5Gpyh+4LBBUW21EabIgvLMpJghcGW+B7 ZpEDElvBiaUOu1lCI1aVylw3b8z7H1TAHxkyKFGINo1AwljietvABBsaGcFNQ+ETiy/W zQmJrFDzxX4rzp+PN9IjOl3b5ApErMenpQqzMR7xdwtP9VwYm50ms+7nOr3woM1CPDH3 985w== X-Gm-Message-State: AOAM5338BQSZ934UJBVj6FT9RBJs5BIxJS8akjoqUQW2tTzzyA4dFG3N QtCwlAEjtF3ve1lFn5XqYCA= X-Google-Smtp-Source: ABdhPJzKUd869QpM4MqJWf3/S5LHy0vxNkmmYvybCtjoSSp7d8ifKz3Ec0vnVseAUSSxZihWzEi8Bw== X-Received: by 2002:a05:6830:1d4e:: with SMTP id p14mr4418968oth.185.1593211046406; Fri, 26 Jun 2020 15:37:26 -0700 (PDT) Received: from [192.168.2.42] (adsl-70-133-144-251.dsl.ablntx.sbcglobal.net. [70.133.144.251]) by smtp.gmail.com with ESMTPSA id j2sm6487042oiw.24.2020.06.26.15.37.24 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Jun 2020 15:37:25 -0700 (PDT) Message-ID: <5EF678A3.9040607@gmail.com> Date: Fri, 26 Jun 2020 17:37:23 -0500 From: Jacob Bachmeyer User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Pedro Alves Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <5EF527BE.9000806@gmail.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jcb62281@gmail.com Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) Pedro Alves wrote: > On 6/25/20 11:39 PM, Jacob Bachmeyer wrote: > >> Pedro Alves wrote: >> >>> On 6/25/20 3:43 AM, Jacob Bachmeyer wrote: >>> >>> >>>> Pedro Alves wrote: >>>> >>>> >>>>> [...] >>>>> >>>> I presume buildbot can be easily configured to pass a '--keep_going' option, perhaps as 'RUNTESTFLAGS=--keep_going' if needed? >>>> >>>> >>> What's more likely is we'll tweak GDB's Makefile so that make check >>> enables that option automatically. If GCC does the same, which I would >>> encourage them to, since they also run the testsuite against systems >>> that can take hours/days to complete, then I'll get to wonder who the >>> default is for. :-) >>> >>> >> The default is for developers working on testsuites -- and for testing release candidates and releases because if your testcases crash in a release, you have problems. :-) >> >> Are you suggesting withdrawing the --keep-going/--no-keep-going options >> > > Yes, but if you would like to add it, it's OK with me, though then > you can say that I'm more discussing the default. But really I'm more > worried about making sure that the people maintaining the GDB > testsuite (me included), and the DejaGnu people understand each > other's needs better. > My position here stems from the need for reliable testing, and part of that is that crashes in the testsuite scripts need to be made very obvious, where DejaGnu had previously swept them under the proverbial rug. Sure, there would be a message in the log, but that is easily missed in a testsuite as large as GDB's and nowhere was a need to grep the log for those messages documented. >> and always generating an UNRESOLVED result, >> resuming with the next test script, and possibly storing a list of aborted test scripts somewhere >> > > Yes (with --keep-going if you insist on having the option), but note that > I suggested storing the list of ERRORs and WARNINGs. > Arguably, a crashed test script is a distinct category from the ERRORs and WARNINGs that test scripts can report when something does not go as expected. The former is a problem with the testsuite *itself*, while the latter indicate problems detected *by* the testsuite. >>>> Still, needing a new option to get old behavior is changing the environment interface, so I will make --keep_going the default, at least for the rest of 1.6. That default is likely to change for 1.7. For automated systems like buildbot, passing '--keep_going' should be perfectly reasonable, which is why the option exists. (While the documented options will be changed to use hyphens to align with the rest of the GNU system in 1.7, the current option names will continue to be accepted, since the aliases are obvious.) >>>> >>>> >>> Currently with DejaGnu only the case of calling an unknown function stops >>> the test run AFAIK. Any other tcl error like calling a proc with the wrong >>> number of arguments, or treating a non-array variable as an array, etc. is >>> caught by the catch in the runtest proc in runtest.exp, and the testrun >>> continues. That to me clearly indicates that the original intention was >>> to catch errors and continue. >>> >>> That also gives the tool's $(tool)_finish proc a chance to tear down >>> correctly. >>> >>> It is just that the "unknown" case wasn't thought of. So I argue that not >>> making unknown proc calls abort the run is a bug fix, and making DejaGnu >>> abort the run for other errors by default is a behavior change that no real >>> testsuite built around DejaGnu is asking for. >>> >>> >> As it was explained to me, it is the other way around -- the catch in runtest was overlooked when ::unknown was added to catch bugs by aborting if an undefined procedure is called. >> >> Is there a ${tool}_finish or do you mean ${tool}_exit? Either way, you make a good point about running cleanup procedures when aborting a test run. >> > > Look at proc runtest in runtest.exp, it calls ${tool}_init, > sources the testcase under catch (using ERROR:, so here you could > run a tally of such caught errors), and then calls ${tool}_finish. > Well, sure enough, there are ${tool}_init and ${tool}_finish procedures called "around" each script if they exist... and no documentation... another item added to my local TODO list for improving the manual. >>>> The entire reason for stopping the testsuite immediately when a Tcl error occurs is that it is undeniable that something is *very* *wrong* when the testsuite stops early. Software testing is hard enough when you know your testsuite is good, but it is far worse when you have bugs in the testsuite masking bugs in the actual program because a test script is crashing and part of the program that is supposed to be tested is not being exercised. >>>> >>>> >>> I think it's safe to say that we all understand the general principle, >>> but IMO that applies more to continuing the same testcase (if somehow that >>> were possible, like "unknown" suppressing the error), rather than continuing >>> with a different testcase. >>> >>> Continuing the testrun when one testcase errors out does not mask any >>> bug in the program that is supposed to be tested. Why would it? The >>> testcase aborts, it doesn't continue. The remaining testcases start >>> afresh. >>> >>> >> Consider a hypothetical case where GDB has exec.exp (which tests starting a debugged process) and attach.exp (which tests attaching to a process). Unknown to the GDB developers, the "attach" command does not work and causes GDB to dump core (oops!) but attach.exp fails with a Tcl error before reaching the point where "attach" is issued to the inferior GDB. If DejaGnu stops immediately, it is obvious that the testsuite is broken, the bug in the testsuite is fixed, and the bug in GDB is found. If DejaGnu sweeps the Tcl error under the proverbial rug (as previous versions did), the bug in the testsuite effectively hides the bug in GDB , because the developers think the "attach" command is being exercised, but those tests are being (almost) silently skipped. >> > > First, that scenario doesn't normally cause a Tcl error. If GDB trips > an internal error, the GDB testsuite catches it with a FAIL. If GDB just dies > (whether it dumps core or not) the GDB testsuite catches that > with UNRESOLVED. > In the hypothetical scenario, attach.exp crashes with a Tcl error *before* issuing the "attach" command to the GDB-under-test. Previous releases of DejaGnu mention a Tcl error in the log and carry on. With this patch, DejaGnu catches that attach.exp crashed and itself inserts an UNRESOLVED result. I also take this as more support for using the UNRESOLVED status here: the test script crashing is little different from GDB exiting unexpectedly. > Also, even if those weren't already considered, the developer won't think the > testcase is just passing correctly, because being a good GDB developer, > he knows he needs to diff the new gdb.sum file against a previous run's, and > sees that some PASSes are missing, and some ERROR: tcl error showed up > in their stead. Nobody should rely on just looking at the summary. > Everyone _must_ diff between a previous run and the current run. Now some > people may use test result diffing scripts that forget to look > at ERROR: lines in .sum. Those would be buggy, but then again, if > those ERRORs were accounted for in the DejaGnu summary, they would be > harder to miss. > If this is the case, then what is the problem with aborting the test run upon error? "Good GDB developers" will always diff the .sum files, notice that the expected new PASS results are not there, and fix their code before checking it in, so they will never have a problem with DejaGnu aborting upon encountering a Tcl error. :-) > And again, people run the GDB testsuite in parallel mode, so your > assumption that the testsuite would stop immediately anyway doesn't > hold. One of the parallel runtest invocations bombs > out, while the others keep running, so by the time you get > back the prompt, other testcases already executed, and you don't > see the ERROR in the current terminal anymore. This is a fundamental problem with slicing up the testsuite external to DejaGnu and running multiple instances of runtest -- output that is supposed to be "last at the end of the run" can be hidden away by other ongoing subset runs. > You'll need to > look at the testsuite .sum .log files anyway. Only, you will > find out that one testcase bombed out, 50000 tests ran, but 1000 others > didn't run because of that one testcase. And now you're missing > those 1000 results, which maybe included those which you were > actually interested in, because you were doing a builbot try run > across a number of systems. > This also applies on a smaller scale for which we have no workaround: gdb.foo/bar.exp bombs out and the remaining /N/ tests in that file did not run. The only real solution is for the test scripts to avoid causing Tcl errors in the first place; anything else is papering over the problem with duct tape. > Now, when running the testsuite against slow embedded boards, then > you won't normally be using the parallel mode, the board just can't > support multiple parallel connections. In that case, you set the > testsuite running, and go on your merry way for a couple hours or 20. > When you come back, maybe the next day, you notice that the testsuite > actually stopped running 30 mins in, because someone changed > the testsuite upstream just before you rebased in way that caused a > tcl error for you in some testcase. Oops. You just lost a day. This is more of an annoyance, and yes, I have come back the next day to find that a long-running command failed shortly after being started, so I know this irritation. > Now > you wish you had remembered to type --keep-going, but it's not > the default, and nobody ever writes Tcl bugs, so you didn't think > of it before. > Note that the snapshot of GDB I have been using for testing this patch does have such a Tcl bug in its testsuite. I did not make any particular effort to choose this snapshot, it was just the current state of GDB master when I started. I have not even looked at *which* commit it is. Thinking "nobody ever writes Tcl bugs" is what causes these problems in the first place. I would expect that, had DejaGnu always aborted immediately upon catching Tcl errors from testsuites, that issue would have been found and fixed quickly. > That's why I think --keep-going should be the default. When > you're developing a new testcase, you'll run that testcase in > isolation a number of times. It's very hard to miss any > silly error then -- there's no need to optimize for this use > case by default, IMO. > > >> Now a bug that obvious will be caught very quickly by users (likely the GDB developers themselves) but obscure regressions are for more insidious, especially in large testsuites such as those for GCC or GDB. If a regression test is being skipped with an easily-missed error, that regression can slip in unnoticed. >> >> Would simply turning aborted test scripts into UNRESOLVED results be enough to get them fixed quickly? >> > > Yes. (Though I still think a separate count for ERROR (and WARNING) > would be better.) I think we need the UNRESOLVED results for at least a credible best-effort at claiming POSIX compliance, but additionally tracking how many UNRESOLVED results are due to script crashes would not be difficult. -- Jacob From debbugs-submit-bounces@debbugs.gnu.org Fri Jun 26 18:58:21 2020 Received: (at 41824) by debbugs.gnu.org; 26 Jun 2020 22:58:21 +0000 Received: from localhost ([127.0.0.1]:43312 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1joxIb-000172-JC for submit@debbugs.gnu.org; Fri, 26 Jun 2020 18:58:21 -0400 Received: from mail-ot1-f49.google.com ([209.85.210.49]:40930) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1joxIX-00016i-Jr for 41824@debbugs.gnu.org; Fri, 26 Jun 2020 18:58:20 -0400 Received: by mail-ot1-f49.google.com with SMTP id q21so2569825otc.7 for <41824@debbugs.gnu.org>; Fri, 26 Jun 2020 15:58:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=2NZCqjl5hDv3QMvLyrgz5nTSiK/Ph5EjB8M7VeBjf3U=; b=rLhZbEgyJGT2ro9bEXtFnn6bmB6kBKDDieIoYZM5YbK/DwHWbUNC36HKeSnRf1CWdE aWSGKZGlPTycMSl4eZPnz3sBodjS4+Mcy4C/7pmbsS1emtrYV/CvGjZVxA3MfAr+tHAM mjs4HXGk1a4cr84xHJAt16+rFU3K2v754sT0kbjWn9aj3VKmrW9egPiRvPeY0ZonczQt +Cga+fzuc7vZPaaPAcE9sEUo5/1EeTtUG9zmsTpOe46qlA6rnOUAQzlzepMFL9755pJm Km6zpacIBRd5R2ZcZma7HUVvOVviQLtA4JIlW235n1iSSGQnApSOJGWPsub5qf/b8ipf DETA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:reply-to:user-agent :mime-version:to:cc:subject:references:in-reply-to :content-transfer-encoding; bh=2NZCqjl5hDv3QMvLyrgz5nTSiK/Ph5EjB8M7VeBjf3U=; b=aGuYu6eayw2t3DyoY3fyeVCYIIXqJVkPv36HBlU6b4kFm/ve8YNkH9Yq9/XOQHAJwP +9zghESi+W0k79YlDb1tMeJA8Rj3in+df6MAPG8qS6KoH/q73F9yO8uk80oJMxwO82Kg rxldRHs9w/OzliRmlUwUki+Av85Ae5reJxdALNs9nQ9ZHkt/Irtt7ZufrfjIYxYObZnS 2NNC5jRB/OTj+jFrwWlz8gh7ypiO1dqWtozN+A1xS8YUHoWmdR44c5oU5ZN1nlY23EQP UC87GqFuRci/ZWFVDObMHwlNzMA1syuxGAoDQh3oYBChM55V/ynQcaju/Cai5aLxt9rK 3Ehw== X-Gm-Message-State: AOAM530L0ig/iD1oUV+/kPb1EP7EFk9ff6tdSEO3XBtaFBhQHqRspriJ n/82AhacAdnBlPOWtkdkAMI= X-Google-Smtp-Source: ABdhPJwcQL5jpOG4gnLeSontEy66o5bVoOXnCFouuqsmKeobkwgqL5LIjHjoeWFi2mtE5o/yNW/IZw== X-Received: by 2002:a4a:964d:: with SMTP id r13mr4405720ooi.57.1593212291665; Fri, 26 Jun 2020 15:58:11 -0700 (PDT) Received: from [192.168.2.42] (adsl-70-133-144-251.dsl.ablntx.sbcglobal.net. [70.133.144.251]) by smtp.gmail.com with ESMTPSA id q15sm304679ota.71.2020.06.26.15.58.09 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 26 Jun 2020 15:58:10 -0700 (PDT) Message-ID: <5EF67D81.30604@gmail.com> Date: Fri, 26 Jun 2020 17:58:09 -0500 From: Jacob Bachmeyer User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Pedro Alves Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <5EF527BE.9000806@gmail.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 41824 Cc: Pedro Alves , Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jcb62281@gmail.com Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) Pedro Alves wrote: > On 6/26/20 10:53 AM, Pedro Alves wrote: > >>> to repeat at the very end of a test run with a big warning about test cases that crashed? >>> >> I did not suggest to repeat anything. >> > > I misunderstood you here -- I somehow thought that that by "repeat" you meant > re-running the testcase. > No, repeating output of at least the name of the file, and possibly the entire error dump. > Still, I would not suggest to repeat the error. Let me clarify -- with this: > > [...] > > Now, you already have an indication that something went wrong > because make existed with error. But people may miss that. > They will easily miss that -- DejaGnu exits with a failure status if any tests produced "surprising" results: FAIL, XPASS, or UNRESOLVED. The GDB testsuite typically has multiple FAIL results, so exiting with an error code is routine. > The issue for me is that the "gdb Summary" tally did not > mention the aborted testcase. I would like to see something > like this instead: > > # of expected passes 208 > # of nasty OMG FIX! tcl errors 1 > > :-) > > Your UNRESOLVED is of course better than the status quo. > UNRESOLVED is a step in the right direction, but I agree that more is needed. We are somewhat limited by POSIX, which does not define a separate result type for "test case crashed" and seems to roll that into UNRESOLVED, although POSIX appears to require a testsuite to have a strictly-defined set of tests and that a run must produce results for exactly all of them, which requires information DejaGnu does not normally have about the running testsuite. So, while UNRESOLVED is needed, we could also maintain a separate count of UNRESOLVED-due-to-crash, or simply store away the error message/errorCode/errorInfo tuples and emit them a second time at the end of the run, also easy in Tcl 8. I am leaning towards this latter solution precisely because it would be loud and obnoxious, while still maintaining a professional tone. (Listing a count of "nasty OMG FIX! >:-( tcl errors", while certainly expressing the proper intent, probably should not find its way into a DejaGnu release, nor should an ASCII-art C'thulu reminding the user that broken testsuites drive programmers to madness, complete with randomly-chosen "your remaining sanity" score... :-) ) Repeating error dumps at the end of the run also fits well into an XML log model, where the collected errors could go into their own structure at the end of the file, without complicating the tag model used for the main test results. Simple importers would still get the inserted UNRESOLVED result, while more thorough or specialized tools could analyze the errors. -- Jacob From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 27 07:40:23 2020 Received: (at 41824) by debbugs.gnu.org; 27 Jun 2020 11:40:23 +0000 Received: from localhost ([127.0.0.1]:43594 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jp9C3-00089x-1A for submit@debbugs.gnu.org; Sat, 27 Jun 2020 07:40:23 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:40829 helo=us-smtp-1.mimecast.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jp9By-00089k-ID for 41824@debbugs.gnu.org; Sat, 27 Jun 2020 07:40:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1593258018; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pdWYXh3/VYqZ8oTXVlFoLqVEK6YYuDq3rHtvi6mzGHY=; b=KVuVaIFOBaQG+Vdc+5QY2ejBMDBH+tbodi8QD4Oh8ygCn5yxwvq1xZhTRXIAS4mbQO3TmA tC+9gH1W9lYX12fipVwDcpNHVmcmKfvylTHgzKca3Czr9Tu7UqpDG6KDmxqL67/TxBoCCA waOdQEdPgULlZrOneUPBCK5/1So88Ak= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-331-qtvduHtMPA-To-XoJeK7cw-1; Sat, 27 Jun 2020 07:40:14 -0400 X-MC-Unique: qtvduHtMPA-To-XoJeK7cw-1 Received: by mail-wm1-f70.google.com with SMTP id o138so12841357wme.4 for <41824@debbugs.gnu.org>; Sat, 27 Jun 2020 04:40:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=pdWYXh3/VYqZ8oTXVlFoLqVEK6YYuDq3rHtvi6mzGHY=; b=CZXIBbPFV3dGrE4HAVpWVXHau5VOoCAr2kW+0DjCQV2Er2lWNCmsWHzSZTfdNBjmYi 9g1e4KIbGxnyG16SkCPzcxHOifVxtqwqskZcdL2j53qUviIo6GLdt95qHx57BP/w74AE QXa6YfS/SHU6Ho24iCaEBV6KvKiFCRD0Mkmc9GLAoXg95xb4HeyergIBgY6rHLFNHGV8 ZoyFfikkkxxHqkQTC9ap2LjADwHM34mHFRIrg7G0kBKupNVt2joYIgfue4TmxoCDj2g5 lCPHdFDIFn1NRrKung+jCCl1anL3LftwODlJ1D6uR12ivnri1b23TiLICowwTEF4yod/ 2moQ== X-Gm-Message-State: AOAM532nUAfzhHQU+XV4YybAg2p3v/ieOQvF4hbb7FcuAc/5HXbvVotN 4H8eB6io8NWpqP89+SlTF9MQVc7KnvydV65d0lF+P3XkKGl7YqH2Uaw1NyGf92jwp9pDrTgixUz Bpn3DKzskiuIiW1o= X-Received: by 2002:a1c:9cd0:: with SMTP id f199mr7697649wme.94.1593258013055; Sat, 27 Jun 2020 04:40:13 -0700 (PDT) X-Google-Smtp-Source: ABdhPJw+oH7HfG+3rTtzQUnX2MS465aB0gsElfJDujgOzqTJLBb91HPcjV5k2y2XiPjE5UBm2fSh/A== X-Received: by 2002:a1c:9cd0:: with SMTP id f199mr7697631wme.94.1593258012786; Sat, 27 Jun 2020 04:40:12 -0700 (PDT) Received: from ?IPv6:2001:8a0:f922:c400:d9b9:b3d0:d289:9385? ([2001:8a0:f922:c400:d9b9:b3d0:d289:9385]) by smtp.gmail.com with ESMTPSA id s8sm34555068wru.38.2020.06.27.04.40.11 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 27 Jun 2020 04:40:12 -0700 (PDT) Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case To: jcb62281@gmail.com, Pedro Alves References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <5EF527BE.9000806@gmail.com> <5EF67D81.30604@gmail.com> From: Pedro Alves Message-ID: <16ece36c-7173-0afe-8e0e-7efc5d1e3eab@redhat.com> Date: Sat, 27 Jun 2020 12:40:10 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <5EF67D81.30604@gmail.com> Content-Language: en-US X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 6/26/20 11:58 PM, Jacob Bachmeyer wrote: >> The issue for me is that the "gdb Summary" tally did not >> mention the aborted testcase.  I would like to see something >> like this instead: >> >>  # of expected passes            208 >>  # of nasty OMG FIX! tcl errors    1 >> >> :-) >> >> Your UNRESOLVED is of course better than the status quo. >>   > > UNRESOLVED is a step in the right direction, but I agree that more is needed.  We are somewhat limited by POSIX, which does not define a separate result type for "test case crashed" and seems to roll that into UNRESOLVED, although POSIX appears to require a testsuite to have a strictly-defined set of tests and that a run must produce results for exactly all of them, which requires information DejaGnu does not normally have about the running testsuite.  So, while UNRESOLVED is needed, we could also maintain a separate count of UNRESOLVED-due-to-crash, or simply store away the error message/errorCode/errorInfo tuples and emit them a second time at the end of the run, also easy in Tcl 8. I'm not sure how much is POSIX compliance important these days. I would hope that that would not limit DejaGnu's development and limit its potential to be more usable. Even if POSIX compliance is important, I would think that that could be put behind a --posixly_correct option, and/or POSIXLY_CORRECT environment variable, like other GNU tools do: https://www.gnu.org/prep/standards/html_node/Non_002dGNU-Standards.html Or conversely, we could have a --gnu mode, though I would say that a GNU tool should behave in GNU mode by default. I really don't buy that adding different result types is an issue, as hinted earlier. We've actually done it in GDB, for a loooong while. We have distinct XFAIL vs KFAIL output messages show up in the .sum file (though those aren't distinguished in the summary file, doing so would be convenient, hence I see that as a GDB testsuite bug). And just very recently, we've added PATH and DUPLICATE output messages, and those do show up in the summary: # of expected passes 89593 # of unexpected failures 547 # of expected failures 146 # of known failures 104 # of untested testcases 20 # of unresolved testcases 5 # of unsupported tests 97 # of paths in test names 1 <<<<< # of duplicate test names 370 <<<<< I would think that even these two would be useful to have in DejaGnu proper for all testsuites. v1/rationale - https://sourceware.org/pipermail/gdb-patches/2020-April/167847.html v3/final version - https://sourceware.org/pipermail/gdb-patches/2020-April/168105.html Thanks, Pedro Alves From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 27 07:40:33 2020 Received: (at 41824) by debbugs.gnu.org; 27 Jun 2020 11:40:33 +0000 Received: from localhost ([127.0.0.1]:43597 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jp9CC-0008AJ-CS for submit@debbugs.gnu.org; Sat, 27 Jun 2020 07:40:33 -0400 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:60230 helo=us-smtp-1.mimecast.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jp9CA-0008AB-8o for 41824@debbugs.gnu.org; Sat, 27 Jun 2020 07:40:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1593258030; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RaOiwctT0fxo18xGJkSF8fLM+6aQq3F/UBgI8yqpC9k=; b=QBBhi7d6/evR3nUrJ5A16CKyW8+VbD+sOXNi1TEoMrMe9IpTG8O2YrtAzb/OupV0TbBic5 M0B3o4rj1a90D5tb6OxnT+mKVa6YNI21Su5O9Q0bjc19NBpF0R4LFi2cLvP21PEagMZZZ/ /9p7pgJksjYaUfLMrBXi6eu4Bl7kqAA= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-41-bBrAPR3GPNavycbKCzunhw-1; Sat, 27 Jun 2020 07:40:26 -0400 X-MC-Unique: bBrAPR3GPNavycbKCzunhw-1 Received: by mail-wm1-f71.google.com with SMTP id b13so11867591wme.9 for <41824@debbugs.gnu.org>; Sat, 27 Jun 2020 04:40:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=RaOiwctT0fxo18xGJkSF8fLM+6aQq3F/UBgI8yqpC9k=; b=MfEyi0dsqVbswnvUD+LwaaxoyG89Em96/VRyvD8TtamTDPp150Qei8ujaFGMokolby EYd9dIbA7Tj/PB3lnZPDYFm+f8I/LTtvtWQRMtvai25JHpen22W0hpgyE5FxChq88jba 3qkfSYqy3+UgOJENLOeT0JRXk6HvXFlT4IWP54E0XBv3YAyHXYBM2agVwPT6JIfQs9cY 5coK2fIoIZ3sMKx+Wn1XBbOLEkhUYSzXQmjGhFqwGqaWNcuZ6rM9SocrdZaVWxCJ9eO9 qJ72yeCoWX8xZWpyMM4ZUG8cNBrJGgVnqRKqEjtipNdVWVUfhfatbJ9kFIyfEaraXJrp uI5g== X-Gm-Message-State: AOAM53077paQqTRiAPRAMbkWpzeV07riUZ2ftAL/DKn71f43BN58/Svz Xe/QCeMFKKGWD1vDlpKzf9EBgfeum1JI/Pcz1oIWD8B8pDca3M0E5QgA6wJIAooSqfi/OAdSdWe JZJvNu4kwkim83rc= X-Received: by 2002:a1c:2485:: with SMTP id k127mr7649676wmk.138.1593258024853; Sat, 27 Jun 2020 04:40:24 -0700 (PDT) X-Google-Smtp-Source: ABdhPJybFw6KmXDk4GV8MHjKzpn6JSXHULTEEnx5MM7+rFWhVcCxxfA4PpafSGaC5OCE+niNUXim8g== X-Received: by 2002:a1c:2485:: with SMTP id k127mr7649643wmk.138.1593258024337; Sat, 27 Jun 2020 04:40:24 -0700 (PDT) Received: from ?IPv6:2001:8a0:f922:c400:d9b9:b3d0:d289:9385? ([2001:8a0:f922:c400:d9b9:b3d0:d289:9385]) by smtp.gmail.com with ESMTPSA id c65sm20715955wme.8.2020.06.27.04.40.23 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 27 Jun 2020 04:40:23 -0700 (PDT) Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case To: jcb62281@gmail.com References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <5EF527BE.9000806@gmail.com> <5EF678A3.9040607@gmail.com> From: Pedro Alves Message-ID: Date: Sat, 27 Jun 2020 12:40:20 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <5EF678A3.9040607@gmail.com> Content-Language: en-US Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=palves@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 6/26/20 11:37 PM, Jacob Bachmeyer wrote: > Pedro Alves wrote: >> On 6/25/20 11:39 PM, Jacob Bachmeyer wrote: >>   >>> Pedro Alves wrote: >>>     >>>> On 6/25/20 3:43 AM, Jacob Bachmeyer wrote: >>>>   >>>>       >>>>> Pedro Alves wrote: >>>>>            >>>>>> [...] >>>>>>           >>>>> I presume buildbot can be easily configured to pass a '--keep_going' option, perhaps as 'RUNTESTFLAGS=--keep_going' if needed? >>>>>             >>>> What's more likely is we'll tweak GDB's Makefile so that make check >>>> enables that option automatically.  If GCC does the same, which I would >>>> encourage them to, since they also run the testsuite against systems >>>> that can take hours/days to complete, then I'll get to wonder who the >>>> default is for.  :-) >>>>         >>> The default is for developers working on testsuites -- and for testing release candidates and releases because if your testcases crash in a release, you have problems.  :-) >>> >>> Are you suggesting withdrawing the --keep-going/--no-keep-going options >>>     >> >> Yes, but if you would like to add it, it's OK with me, though then >> you can say that I'm more discussing the default.  But really I'm more >> worried about making sure that the people maintaining the GDB >> testsuite (me included), and the DejaGnu people understand each >> other's needs better. >>   > > My position here stems from the need for reliable testing, and part of that is that crashes in the testsuite scripts need to be made very obvious, where DejaGnu had previously swept them under the proverbial rug.  Sure, there would be a message in the log, but that is easily missed in a testsuite as large as GDB's and nowhere was a need to grep the log for those messages documented. Sure, but we don't need to drastically change DejaGnu's behavior to address that. > >>>>> Still, needing a new option to get old behavior is changing the environment interface, so I will make --keep_going the default, at least for the rest of 1.6.  That default is likely to change for 1.7.  For automated systems like buildbot, passing '--keep_going' should be perfectly reasonable, which is why the option exists.  (While the documented options will be changed to use hyphens to align with the rest of the GNU system in 1.7, the current option names will continue to be accepted, since the aliases are obvious.) >>>>>             >>>> Currently with DejaGnu only the case of calling an unknown function stops >>>> the test run AFAIK.  Any other tcl error like calling a proc with the wrong >>>> number of arguments, or treating a non-array variable as an array, etc. is >>>> caught by the catch in the runtest proc in runtest.exp, and the testrun >>>> continues.  That to me clearly indicates that the original intention was >>>> to catch errors and continue. >>>> >>>> That also gives the tool's $(tool)_finish proc a chance to tear down >>>> correctly. >>>> >>>> It is just that the "unknown" case wasn't thought of.  So I argue that not >>>> making unknown proc calls abort the run is a bug fix, and making DejaGnu >>>> abort the run for other errors by default is a behavior change that no real >>>> testsuite built around DejaGnu is asking for. >>>>         >>> As it was explained to me, it is the other way around -- the catch in runtest was overlooked when ::unknown was added to catch bugs by aborting if an undefined procedure is called. >>> >>> Is there a ${tool}_finish or do you mean ${tool}_exit?  Either way, you make a good point about running cleanup procedures when aborting a test run. >>>     >> >> Look at proc runtest in runtest.exp, it calls ${tool}_init, sources the testcase under catch (using ERROR:, so here you could >> run a tally of such caught errors), and then calls ${tool}_finish. >>   > > Well, sure enough, there are ${tool}_init and ${tool}_finish procedures called "around" each script if they exist... and no documentation... another item added to my local TODO list for improving the manual. > >>>>> The entire reason for stopping the testsuite immediately when a Tcl error occurs is that it is undeniable that something is *very* *wrong* when the testsuite stops early.  Software testing is hard enough when you know your testsuite is good, but it is far worse when you have bugs in the testsuite masking bugs in the actual program because a test script is crashing and part of the program that is supposed to be tested is not being exercised. It's actually deniable that there's something *very* *wrong*. For one, all tests run under the same runtest process, and are sourced in the global namespace. So it is easy for a global variable to leak from one testcase to another. That's usually harmless, except when you have set foo 1 in testcase a.exp, and then b.exp does: set foo(1) 1 Whoops, TCL doesn't like that, so it aborts the testcase. Could this be solved by putting each testcase in its own address space? Yes. Do people actually do that? No -- it's a lot of make work and the problem is rare. Is that reuse global as array really revealing such a MAJOR issue with the GDB testcase that it warrants stopping the whole run? No... I would argue that instead it is the DejaGnu framework that should make it so that each testcase is run under as much isolation as possible. So I call that a design issue in DejaGnu itself. Whether that could be fixed by something like sourcing each testcase under its own Tcl slave interpreter, I'm not sure. Another example of a Tcl bug that sometimes we trip on. Say we have a testcase that needs to extract some address from the inferior in its setup phase, something like this: set test "get address" gdb_test_multiple "x /i \$pc" $test { -re "($hex).*$gdb_prompt $" { set addr $expect_out(1,string) pass $test } } if {$addr == 0} { # The above failed, so no use continuing. return } Now, gdb_test_multiple is a wrapper around expect, that internally catches GDB crashes, internal GDB errors, timeouts and other things, so the bug above is often missed. The bug is of course that when e.g., a timeout occurs, $addr is left uninitialized, so you get a Tcl error referencing a nonexisting variable. But the thing is, we already issued a FAIL, and we want to bail out of the testcase anyway. Crashing the whole testrun for a bug that typically happens in already-failing code is too strict. And these are the code paths that are not exercised in a normal run, so are easy to miss, and are just impossible to have full coverage for. >>>>>             >>>> I think it's safe to say that we all understand the general principle, >>>> but IMO that applies more to continuing the same testcase (if somehow that >>>> were possible, like "unknown" suppressing the error), rather than continuing >>>> with a different testcase. >>>> >>>> Continuing the testrun when one testcase errors out does not mask any >>>> bug in the program that is supposed to be tested.  Why would it?  The >>>> testcase aborts, it doesn't continue.  The remaining testcases start >>>> afresh. >>>>         >>> Consider a hypothetical case where GDB has exec.exp (which tests starting a debugged process) and attach.exp (which tests attaching to a process).  Unknown to the GDB developers, the "attach" command does not work and causes GDB to dump core (oops!) but attach.exp fails with a Tcl error before reaching the point where "attach" is issued to the inferior GDB.  If DejaGnu stops immediately, it is obvious that the testsuite is broken, the bug in the testsuite is fixed, and the bug in GDB is found.  If DejaGnu sweeps the Tcl error under the proverbial rug (as previous versions did), the bug in the testsuite effectively hides the bug in GDB , because the developers think the "attach" command is being exercised, but those tests are being (almost) silently skipped. >>>     >> >> First, that scenario doesn't normally cause a Tcl error.  If GDB trips >> an internal error, the GDB testsuite catches it with a FAIL.  If GDB just dies >> (whether it dumps core or not) the GDB testsuite catches that >> with UNRESOLVED. >>   > > In the hypothetical scenario, attach.exp crashes with a Tcl error *before* issuing the "attach" command to the GDB-under-test.  Previous releases of DejaGnu mention a Tcl error in the log and carry on.  With this patch, DejaGnu catches that attach.exp crashed and itself inserts an UNRESOLVED result. > > I also take this as more support for using the UNRESOLVED status here:  the test script crashing is little different from GDB exiting unexpectedly. Well, I'm actually not sure why doesn't GDB FAIL in the crash case instead of UNRESOLVED. To me a GDB internal error or the GDB process crashing doesn't look very different. Both require pretty much the same level of attention from a developer. But that decision predates me. Who knows, maybe GDB internal error support was only added after the testsuite was originally written, and thus the crash handling predates the internal error handling. > >> Also, even if those weren't already considered, the developer won't think the >> testcase is just passing correctly, because being a good GDB developer, >> he knows he needs to diff the new gdb.sum file against a previous run's, and >> sees that some PASSes are missing, and some ERROR: tcl error showed up >> in their stead.  Nobody should rely on just looking at the summary. >> Everyone _must_ diff between a previous run and the current run.  Now some >> people may use test result diffing scripts that forget to look >> at ERROR: lines in .sum.  Those would be buggy, but then again, if >> those ERRORs were accounted for in the DejaGnu summary, they would be >> harder to miss. >>   > > If this is the case, then what is the problem with aborting the test run upon error?  "Good GDB developers" will always diff the .sum files, notice that the expected new PASS results are not there, and fix their code before checking it in, so they will never have a problem with DejaGnu aborting upon encountering a Tcl error.  :-) As I've explained, the problem is that one testcase prevented a large number of others from running, potentially substantially all of the testsuite. So you've now wasted a lot of time and maybe prevented other people from using the target system meanwhile. All because of needless interference between testcases. > >> And again, people run the GDB testsuite in parallel mode, so your >> assumption that the testsuite would stop immediately anyway doesn't >> hold.  One of the parallel runtest invocations bombs >> out, while the others keep running, so by the time you get >> back the prompt, other testcases already executed, and you don't >> see the ERROR in the current terminal anymore. > > This is a fundamental problem with slicing up the testsuite external to DejaGnu and running multiple instances of runtest -- output that is supposed to be "last at the end of the run" can be hidden away by other ongoing subset runs. That's like arguing against parallelization, and saying that everyone should run the testsuite in serial mode, and be OK with it taking 2 hours instead of 30 mins to complete a run... The parallel mode aggregates the summaries from the different runtest invocations as a single summary at the end, https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=contrib/dg-extract-results.py;h=30aa68771d491451f72d6dbd18f6d5ac339451b5;hb=HEAD so I don't see what problem here is unsolvable. Again, the issue is that Tcl ERRORs aren't tallied up on the DejaGnu summary, so the aggregated summary doesn't contain those either. > >>   You'll need to >> look at the testsuite .sum .log files anyway.  Only, you will >> find out that one testcase bombed out, 50000 tests ran, but 1000 others >> didn't run because of that one testcase.  And now you're missing >> those 1000 results, which maybe included those which you were >> actually interested in, because you were doing a builbot try run >> across a number of systems. >>   > > This also applies on a smaller scale for which we have no workaround:  gdb.foo/bar.exp bombs out and the remaining /N/ tests in that file did not run.  The only real solution is for the test scripts to avoid causing Tcl errors in the first place; anything else is papering over the problem with duct tape. That's not the same thing. The remaining /N/ tests in the file depend on the previous /M/ tests. E.g., gdb_test "step" gdb_test "print foo" If the "step" fails, the "print foo" test is meaningless. However, there should be absolutely no dependency between foo.exp and bar.exp. It should be possible to run those two different testcases in any order. Thus, it should be irrelevant to foo.exp whether bar.exp bombs out of not. Consequently, there's no real reason to not let foo.exp run regardless of what happens to bar.exp. > >> Now, when running the testsuite against slow embedded boards, then >> you won't normally be using the parallel mode, the board just can't >> support multiple parallel connections.  In that case, you set the >> testsuite running, and go on your merry way for a couple hours or 20. >> When you come back, maybe the next day, you notice that the testsuite >> actually stopped running 30 mins in, because someone changed >> the testsuite upstream just before you rebased in way that caused a >> tcl error for you in some testcase.  Oops.  You just lost a day. > > This is more of an annoyance, and yes, I have come back the next day to find that a long-running command failed shortly after being started, so I know this irritation. There you go. > >>   Now >> you wish you had remembered to type --keep-going, but it's not >> the default, and nobody ever writes Tcl bugs, so you didn't think >> of it before. >>   > > Note that the snapshot of GDB I have been using for testing this patch does have such a Tcl bug in its testsuite.  I did not make any particular effort to choose this snapshot, it was just the current state of GDB master when I started.  I have not even looked at *which* commit it is. Can you tell me what the error looks like? > Thinking "nobody ever writes Tcl bugs" is what causes these problems in the first place. I don't know what you mean by this. What I meant is that people likely won't remember to use --keep-going when invoking runtest interactively. Why force people to go through the pain of learning the option? > > I would expect that, had DejaGnu always aborted immediately upon catching Tcl errors from testsuites, that issue would have been found and fixed quickly. As I've explained, that "aborted immediately" is just impossible by design with parallel mode. You could approximate it perhaps, by making make abort all the parallel runtests when one of the runtests bombs out. It won't be immediately. And I'd of course think that adding such feature wouldn't help anyone in reality. And with your "quickly", you're assuming that a Tcl errors typically happen in the normal code paths. If it happens on some timeout path as I explained, then it may be that it's not that easy to trigger the error, unless you happen to run the testsuite on a "lucky" system with a "wrong" combination of environment (compiler, glibc, kernel, etc.). Thanks, Pedro Alves From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 27 07:44:31 2020 Received: (at 41824) by debbugs.gnu.org; 27 Jun 2020 11:44:31 +0000 Received: from localhost ([127.0.0.1]:43604 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jp9G3-0008GA-G7 for submit@debbugs.gnu.org; Sat, 27 Jun 2020 07:44:31 -0400 Received: from us-smtp-1.mimecast.com ([205.139.110.61]:31545 helo=us-smtp-delivery-1.mimecast.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jp9G0-0008G1-Ny for 41824@debbugs.gnu.org; Sat, 27 Jun 2020 07:44:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1593258268; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8oXLhl64GwLeLVcJjDHEyRNXgd2jL9elYOfYv1Bp7Do=; b=QQunu/2pPzNX5e6cEF+bCNnK385U66jZub8uu8ei5byHkCUsQCTTNowwPrEt9Pxzyfohd7 dWf7wDjnLVYBiSrEGqN5bjdvd4+Oyh72I0/7HkzVWG7K41oElgAFRV5UAx0ikJNFedCHEd WYsmOw6ojfYizRdmUWVACcIhEVODImc= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-306-t4U0-jKhP4SY-c4r3JAQ2Q-1; Sat, 27 Jun 2020 07:44:26 -0400 X-MC-Unique: t4U0-jKhP4SY-c4r3JAQ2Q-1 Received: by mail-wm1-f70.google.com with SMTP id g138so5233366wme.7 for <41824@debbugs.gnu.org>; Sat, 27 Jun 2020 04:44:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:cc:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=8oXLhl64GwLeLVcJjDHEyRNXgd2jL9elYOfYv1Bp7Do=; b=nc1C9j9y6FVNarsnwsd5XUmFK82LV860JTjlBvAgXbT78HfE4u5KJkc+PD7HMubN4E aiE+vxjCnt50MsSqUfscnzPWsX3DkwKiagVblzUTAFW2JCGYJgNr2nGlyHLhkESO5iFi QBj3XumynGrk6DlWL5fqXYbL7CSU7LgM4FOQCWezzRpo35fWnL2LPHjQFAXuPTC9EEhf cNO1rG4RLbNDYsRHhcGSITtW/lgFXZy84s+O4zbQ3NMFfP/aNgSm7O6wnyEZiQ6WWZ/M meAb9oOGtPensngq4D0Pj599w8gMksLv/VdA1z5rS7AN1jyIX9rm75remQcNoSieEml9 aRdg== X-Gm-Message-State: AOAM531vq8i9Hh0JlumQphaR6/BPOTR4ZseSFvNn4odrY2qPPyy4jDAO scVbNVA5lFLUSzvU4H0w5Z+DgG4WwCEhpKFb6eArMZ74MlN/x6G+CnrYB6HIIRQwWkXXFIe1D9h 24hCsEP9agow5xXA= X-Received: by 2002:a1c:3286:: with SMTP id y128mr7538073wmy.29.1593258264981; Sat, 27 Jun 2020 04:44:24 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxc45s6VIlXWBgBDsGKwAlr/kCwG71e0QdZIsSIkAmdHfyJGPpV+y90ZQ7IlBeEGDZAv36P9A== X-Received: by 2002:a1c:3286:: with SMTP id y128mr7538065wmy.29.1593258264787; Sat, 27 Jun 2020 04:44:24 -0700 (PDT) Received: from ?IPv6:2001:8a0:f922:c400:d9b9:b3d0:d289:9385? ([2001:8a0:f922:c400:d9b9:b3d0:d289:9385]) by smtp.gmail.com with ESMTPSA id t16sm20254874wru.9.2020.06.27.04.44.23 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 27 Jun 2020 04:44:24 -0700 (PDT) Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case From: Pedro Alves To: jcb62281@gmail.com References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <5EF527BE.9000806@gmail.com> <5EF678A3.9040607@gmail.com> Message-ID: Date: Sat, 27 Jun 2020 12:44:22 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 6/27/20 12:40 PM, Pedro Alves wrote: > > Could this be solved by putting each testcase in its own address space? I meant to say namespace instead. Thanks, Pedro Alves From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 27 08:51:28 2020 Received: (at 41824) by debbugs.gnu.org; 27 Jun 2020 12:51:28 +0000 Received: from localhost ([127.0.0.1]:43668 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpAIp-0001Ot-Rb for submit@debbugs.gnu.org; Sat, 27 Jun 2020 08:51:28 -0400 Received: from gnashdev.org ([65.183.81.79]:35562 helo=welcomehome.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpAIb-0001OU-VF for 41824@debbugs.gnu.org; Sat, 27 Jun 2020 08:51:26 -0400 Received: from eyes.moongulch.net (mobile-166-137-107-145.mycingular.net [166.137.107.145] (may be forged)) (authenticated bits=0) by welcomehome.org (8.15.2/8.15.2) with ESMTPSA id 05RCnovJ023938 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Sat, 27 Jun 2020 06:50:11 -0600 Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case To: jcb62281@gmail.com References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <32b06c6c-560b-3637-0c0d-84248357e591@senecass.com> <5EF5552A.6040001@gmail.com> From: Rob Savoye Autocrypt: addr=rob@senecass.com; prefer-encrypt=mutual; keydata= mQGiBEezIb8RBACsVR1XKKx2+bmShsPssv98frN9CRwnpRwx+aZhCL2FbpxIn00FkpCg8lHg ftFKxNytbyIdXvqKKVQFd75wIsszjAs07M5dTRfWkunkQ8FR6RKRJuzNkGG6YR7k7YDE3f1k 5K/k54Y1aTgP2rhsn10bX537VV3gIRpknNyfNCNL8wCglCTxTOuqiVGTG/gOTo96f94Q0/kD /2EDQFYYUx17rxlyYJIvF7cfYvkw7nvBMZn0u/kDKirwPs6Sx2o30TxvvgjA55HTN6UNVwiL Iglr3DyBgi6LGD7RAxMG15FnK4xx28wmu8EXYDJuNqbsy+PoPs1pMdlsVERIJfzDmG65/pAD LFRymDL2DgnU9v/pH0HrLwY34A5kA/9Ywdn7G3FFR/oR/WGEavzM+mvYq2GRSDLEobaGRdPp tAEUpyMGsCVUG8TIWRA9B6CHD0YTh2kLKIH7QK6/3fcnROCrCd3/dIYwkaQV0Icrl17eUpLX 1xBmbR7ji3inCd58uceQBIksKdrcqeFB5yB6RBWPpgDELg020x7x6UhKSLkCDQRHsyG/EAgA uU0hNVK0CfKhFVzUBCcSXCn+Ck7sh7oFYdMKJyg6t3r56sWtgzlSuq1k1jwaDlaOSwF9DO0z KHSdJ2KvmoheBGxbUfSguUbgGC8XvERuaMHaHaCIal2D6MdisbL5w4QGtKmTSGjlqZj1tCS7 s6LBHrQAkVkBSBSv12sl4wz8tbL1aublznu/5LXusg2wmae+ynRbubnggAdDGIJv+AQsbYL+ yukEZkFT+AvJzCENFEaBxRj6B7obBx9eSKcRgv0ON0CghiM2iYN6iuUzhcaBl+v9+4XR9tDm vR2s/CBP0A14yNRXQ+B9HG/kzmSLSHttY9jiySO91YbRsrWZbn2wxwAECwgAn5vmpCw2T4/H gu14o9qnY4Lr0PPp9uCw9T7oaZDdjibZSvcgc9WabUuSP6BOlcpE84d92gGJn0mTvuTsSf6Z yWNYj0Rdt71MZbstU/lUfXmOBS3bwhas0LnH4zmoOZJ2/IvZT8hbfrld3eQCeT347rkBt2vM g4RZEXutvpQUDtT6HO+v202/hHgwlLZppy/QgTaUAZcYOoIw72/OR6Mb2y7gROJyYEAw/DJD Jf0KXIn5kpsrSDiEJHTW7YBDx1opnp+FwhRBUb2xGr7ldZcZxrOmjEHMAHzAi6vcG4gdlhZ4 /UwTkycgftroQ+nNhyhwY9UXdD02ab6eZzjIE9Pc/YhJBBgRAgAJBQJHsyG/AhsMAAoJEFC5 oRCgttP+kZcAnjB3S9tuyK9PqDKzEcoovKs/LowRAJ9griQbLMDG7R9HvCLOfvg9iRGH1A== Organization: Seneca Software & Solar, Inc. Message-ID: <1d504891-c277-af13-ab49-d6d6df30221a@senecass.com> Date: Sat, 27 Jun 2020 06:49:30 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <5EF5552A.6040001@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on mail.senecass.com X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 41824 Cc: Pedro Alves , Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 6/25/20 7:53 PM, Jacob Bachmeyer wrote: > The original rationale for using UNRESOLVED here was that DejaGnu > converts a test result to UNRESOLVED if too many errors or warnings were > produced.  The DejaGnu manual indicates (section "A POSIX Compliant Test > Framework") that UNRESOLVED is correct for a test where execution was > interrupted or was set up incorrectly.  UNTESTED is specifically listed > as a placeholder for an as-yet-unwritten testcase.  A "typical" GDB > testsuite run has almost a hundred UNTESTED results but (without this > patch) zero UNRESOLVED results. > > To me, this seems that UNRESOLVED is correct here, or that the manual > has an error. So having written both, there is always the chance I've interpreted things differently. If the test is interrupted, or has errors/warnings, like timing problems for example, then it's UNRESOLVED. But a bug in Tcl code enough to trigger unknown is UNTESTED, as the test never really ran. I'd go with which definition the toolchain teams prefer, as this effects validation testruns. I could go either way. I don't know if POSIX 1003.3 is still considered a standard, that was a long time ago. The testsuites were primarily focused on XOpen and POSIX conformance tests, not toolchains. I was on the standards committee, so used them as a model cause they seemed a good standard. But the details of the interpretation can now be whatever we want it to be. We're probably the only ones left running TET compliant testsuites anyway. :-) Anyway, whether a test run aborts on an error or not is not defined, so it's up to us to decide. - rob - From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 27 09:15:35 2020 Received: (at 41824) by debbugs.gnu.org; 27 Jun 2020 13:15:35 +0000 Received: from localhost ([127.0.0.1]:43703 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpAgA-00023z-RB for submit@debbugs.gnu.org; Sat, 27 Jun 2020 09:15:35 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:54667 helo=us-smtp-1.mimecast.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpAg6-00023o-QP for 41824@debbugs.gnu.org; Sat, 27 Jun 2020 09:15:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1593263730; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WyFL0Nn9+6RPrdkaZQ+4OQ6odGPwD44oYNbNEj9Ne2Y=; b=RG0fvsUlNI9M5ckeHQ4QD4ykDlIvFD8oprJXH49Tl2wfB30C/DM9LzjZ+kEzqWGgdNn5VL yztZj23+ylfOS03iZLM/73QL6xNPkcQe8JRP2hhx8iTkNz7WC2fQyHl5/y+KoND5VFSjt7 Lh51DG9t95bPni4oIbQ9A0iQLjoDeqU= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-178-8LqWs4JEM0GIoaRtx4CvBg-1; Sat, 27 Jun 2020 09:15:28 -0400 X-MC-Unique: 8LqWs4JEM0GIoaRtx4CvBg-1 Received: by mail-wr1-f69.google.com with SMTP id a18so13126281wrm.14 for <41824@debbugs.gnu.org>; Sat, 27 Jun 2020 06:15:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:cc:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=WyFL0Nn9+6RPrdkaZQ+4OQ6odGPwD44oYNbNEj9Ne2Y=; b=iLOPCjIrtN0EJ7v/lf2R7anAUVmHi7V4/EaUHrpNVxprpnOOGZ8I3e6xrQl7fj/nOC UFNMIstCSwhoIxQblX9M9wbNEw6s7b+33Ymth7XHEFy4NcDKOw9DO9XoQXSEVfAzE8EG NY5ZZWxK29KZeqVRtqf5rszI/oNPTXTWHcQiEXwviKhHaxjwBNtt7asMNnFqEy81apRa dMDkoad/qdLWzflkG3eQRPpMjKBqo8XsIPbZtz1Adm/dj9ogPcTHRD897J77YeWiAeBC es3b7yxhi4doDV60uk5T/NI0nEWw4iaIVQ+Dv6hlpjbkFKAfd4k2kAy0IuTEVEselzsk UmHg== X-Gm-Message-State: AOAM530KK4XoUseR/JDCe8Kicim+gb9KxskaaxKBQWG1UFoQ4GODqJY7 RjnvWioTomFUj/2Dn0y3scNVoigo6GD2n8WP4IGU/QAu71GqSnI8m3tvQd8Rb64j4WLZKWOBqFM sXWTgG+/fCbTnUgg= X-Received: by 2002:adf:ef4d:: with SMTP id c13mr8324722wrp.315.1593263727152; Sat, 27 Jun 2020 06:15:27 -0700 (PDT) X-Google-Smtp-Source: ABdhPJww8oId0jidt/zhUvoJu6CVLXyfmDQNjE+1MhazN0JxY6m7u5MCm96K8GLIyWjLV+fztNhx9A== X-Received: by 2002:adf:ef4d:: with SMTP id c13mr8324709wrp.315.1593263726976; Sat, 27 Jun 2020 06:15:26 -0700 (PDT) Received: from ?IPv6:2001:8a0:f922:c400:d9b9:b3d0:d289:9385? ([2001:8a0:f922:c400:d9b9:b3d0:d289:9385]) by smtp.gmail.com with ESMTPSA id c70sm20751876wme.32.2020.06.27.06.15.25 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 27 Jun 2020 06:15:26 -0700 (PDT) Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case From: Pedro Alves To: jcb62281@gmail.com, Pedro Alves References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <5EF527BE.9000806@gmail.com> <5EF67D81.30604@gmail.com> <16ece36c-7173-0afe-8e0e-7efc5d1e3eab@redhat.com> Message-ID: <2a6211cf-c810-b39e-b188-3b5ac3f97cdc@redhat.com> Date: Sat, 27 Jun 2020 14:15:20 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <16ece36c-7173-0afe-8e0e-7efc5d1e3eab@redhat.com> Content-Language: en-US X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 6/27/20 12:40 PM, Pedro Alves wrote: > We > have distinct XFAIL vs KFAIL output messages show up in the .sum file (though > those aren't distinguished in the summary file, doing so would be convenient, > hence I see that as a GDB testsuite bug). > Sorry, somehow I got confused and thought KFAIL was GDB specific because AFAIK GCC doesn't use it. DejaGnu does show the KFAILs separated in the summary. I don't why I got confused, I've actually read the kfail code in DejaGnu before. Oh well. The point still stands with DUPLICATE and PATH output results. And KFAIL isn't POSIX either, I think? Thanks, Pedro Alves From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 27 13:21:47 2020 Received: (at 41824) by debbugs.gnu.org; 27 Jun 2020 17:21:47 +0000 Received: from localhost ([127.0.0.1]:44528 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpEWR-0008Q6-3L for submit@debbugs.gnu.org; Sat, 27 Jun 2020 13:21:47 -0400 Received: from mail-ot1-f42.google.com ([209.85.210.42]:44102) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpEWM-0008Pr-Vs for 41824@debbugs.gnu.org; Sat, 27 Jun 2020 13:21:45 -0400 Received: by mail-ot1-f42.google.com with SMTP id 5so9610987oty.11 for <41824@debbugs.gnu.org>; Sat, 27 Jun 2020 10:21:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=St4WVrCKfgcu8eOHDzipLG+0M8GgE6pOjeEMdEBX15A=; b=qXdvfWj4VE28RgNh5Zl85vwRbFCA8SDmsnYY+iAWiIyyvYSJiO1pm8RiwtPvYOwSwd 6Rzp+yopuO9aWqXsi2Qh78wo4KAPp1DooWTcOgK7UgFrNvG1Y7VMTftVk9SUgkYZVrXu sPCXO7zSM4oA4sQ2qszDOOQ/kJ6Ff0v9FuuISwyY60T2stdXd8g4FpgfaoCtwKEOgX0l PmF1ppAGtOcQutGCEgnF8NGCF9uUjUNwDX7E3+AA5YVhTyy0HpqxxduGRZpYxdp8Si/T tbzTb0T1eQI2rhuhq32h2lU5cvQ9SQ6daF/sKvXgZdAd7vpUuHQi/83k+uDIBI7omwUR vPqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:reply-to:user-agent :mime-version:to:cc:subject:references:in-reply-to :content-transfer-encoding; bh=St4WVrCKfgcu8eOHDzipLG+0M8GgE6pOjeEMdEBX15A=; b=aaH8U36kfswm//ywsGOZgIM8roIsAcvqR1BV9KloIoSLXc87o4aehDcw1im94ta2PZ 9UT4hM/tMFGivOXVhudodTUjMAQKsl/dcQiKfuMU6IWxTpuvfCi9OtN5VwxpdZry1ttS sIPgLHEyJMJi/FVZfOa7qNm3p8wlOAq3EZBRlbiFKvfOV3M9ET+oxfgwmFD3XuYiXS/z tlLiLPywkBe2Vq05fq/fSCpuaa1nXorTXe5PKfMfFmS9N8HSH7MCgvQ8ihi8feGFtOH1 xLLDnRJDR6lRpOJpoN1Hdfuk9rOu4WcvwB/aVl+wpxFv4HkGRxAZZ2BaDyhQw32P5nuD juhw== X-Gm-Message-State: AOAM531yx4csakl1oc0/djq9K/nXAWQwQsRy/zJ29zoRN0ivYtdzxjlH JBHeV9EGr44B/f7sjzU/glk= X-Google-Smtp-Source: ABdhPJwQ9VSDtFLEX3ES1cvk0mu/9lmXZtS3IQJ65fkNX0es1QJxwYh8OpqnRiOpzoEu/Ajrlq403w== X-Received: by 2002:a9d:6294:: with SMTP id x20mr7320441otk.18.1593278497220; Sat, 27 Jun 2020 10:21:37 -0700 (PDT) Received: from [192.168.2.42] (adsl-70-133-144-251.dsl.ablntx.sbcglobal.net. [70.133.144.251]) by smtp.gmail.com with ESMTPSA id o2sm6957474ota.14.2020.06.27.10.21.35 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 27 Jun 2020 10:21:36 -0700 (PDT) Message-ID: <5EF7801E.3040909@gmail.com> Date: Sat, 27 Jun 2020 12:21:34 -0500 From: Jacob Bachmeyer User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Pedro Alves Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <5EF527BE.9000806@gmail.com> <5EF678A3.9040607@gmail.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jcb62281@gmail.com Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) Pedro Alves wrote: > On 6/26/20 11:37 PM, Jacob Bachmeyer wrote: > >> Pedro Alves wrote: >> >>> On 6/25/20 11:39 PM, Jacob Bachmeyer wrote: >>> >>> >>>> Pedro Alves wrote: >>>> >>>> > > [...] > > Could this be solved by putting each testcase in its own address space? > Yes. Do people actually do that? No -- it's a lot of make work and the > problem is rare. Is that reuse global as array really revealing such a MAJOR > issue with the GDB testcase that it warrants stopping the whole run? No... > > I would argue that instead it is the DejaGnu framework that should make > it so that each testcase is run under as much isolation as possible. > So I call that a design issue in DejaGnu itself. Whether that could > be fixed by something like sourcing each testcase under its own > Tcl slave interpreter, I'm not sure. > This is the current plan -- to eventually run test scripts in independent slave interpreters, which enforces the isolation necessary at the Tcl level for future native parallel testing and also prevents test scripts from accidentally clobbering globals that are important to DejaGnu, some of which have very generic names. > Another example of a Tcl bug that sometimes we trip on. Say we have a > testcase that needs to extract some address from the inferior > in its setup phase, something like this: > > set test "get address" > gdb_test_multiple "x /i \$pc" $test { > -re "($hex).*$gdb_prompt $" { > set addr $expect_out(1,string) > pass $test > } > } > > if {$addr == 0} { > # The above failed, so no use continuing. > return > } > > Now, gdb_test_multiple is a wrapper around expect, that internally catches > GDB crashes, internal GDB errors, timeouts and other things, so the bug above > is often missed. The bug is of course that when e.g., a timeout occurs, $addr > is left uninitialized, so you get a Tcl error referencing a nonexisting variable. > > But the thing is, we already issued a FAIL, and we want to bail out of > the testcase anyway. Crashing the whole testrun for a bug that typically > happens in already-failing code is too strict. And these are the > code paths that are not exercised in a normal run, so are > easy to miss, and are just impossible to have full coverage for. > Fine; I will loosen that, but DejaGnu has no way to know that it is bailing out of a failing testcase and that there are no more tests from that script, so an UNRESOLVED result complaining about the error will still be inserted. >>>>> I think it's safe to say that we all understand the general principle, >>>>> but IMO that applies more to continuing the same testcase (if somehow that >>>>> were possible, like "unknown" suppressing the error), rather than continuing >>>>> with a different testcase. >>>>> >>>>> Continuing the testrun when one testcase errors out does not mask any >>>>> bug in the program that is supposed to be tested. Why would it? The >>>>> testcase aborts, it doesn't continue. The remaining testcases start >>>>> afresh. >>>>> >>>>> >>>> Consider a hypothetical case where GDB has exec.exp (which tests starting a debugged process) and attach.exp (which tests attaching to a process). Unknown to the GDB developers, the "attach" command does not work and causes GDB to dump core (oops!) but attach.exp fails with a Tcl error before reaching the point where "attach" is issued to the inferior GDB. If DejaGnu stops immediately, it is obvious that the testsuite is broken, the bug in the testsuite is fixed, and the bug in GDB is found. If DejaGnu sweeps the Tcl error under the proverbial rug (as previous versions did), the bug in the testsuite effectively hides the bug in GDB , because the developers think the "attach" command is being exercised, but those tests are being (almost) silently skipped. >>>> >>>> >>> First, that scenario doesn't normally cause a Tcl error. If GDB trips >>> an internal error, the GDB testsuite catches it with a FAIL. If GDB just dies >>> (whether it dumps core or not) the GDB testsuite catches that >>> with UNRESOLVED. >>> >>> >> In the hypothetical scenario, attach.exp crashes with a Tcl error *before* issuing the "attach" command to the GDB-under-test. Previous releases of DejaGnu mention a Tcl error in the log and carry on. With this patch, DejaGnu catches that attach.exp crashed and itself inserts an UNRESOLVED result. >> >> I also take this as more support for using the UNRESOLVED status here: the test script crashing is little different from GDB exiting unexpectedly. >> > > Well, I'm actually not sure why doesn't GDB FAIL in the crash case instead of > UNRESOLVED. To me a GDB internal error or the GDB process crashing doesn't > look very different. Both require pretty much the same level of attention > from a developer. But that decision predates me. Who knows, maybe GDB > internal error support was only added after the testsuite was originally > written, and thus the crash handling predates the internal error handling. > Considering that testing GDB was one of the motivations for writing DejaGnu and how old DejaGnu is, I suspect that the testsuite had to handle GDB crashes before GDB had support for reporting internal errors. In some sense, UNRESOLVED is more serious than FAIL; the GDB testsuite has about a hundred FAIL results but typically no UNRESOLVED results. >>> And again, people run the GDB testsuite in parallel mode, so your >>> assumption that the testsuite would stop immediately anyway doesn't >>> hold. One of the parallel runtest invocations bombs >>> out, while the others keep running, so by the time you get >>> back the prompt, other testcases already executed, and you don't >>> see the ERROR in the current terminal anymore. >>> >> This is a fundamental problem with slicing up the testsuite external to DejaGnu and running multiple instances of runtest -- output that is supposed to be "last at the end of the run" can be hidden away by other ongoing subset runs. >> > > That's like arguing against parallelization, and saying that everyone > should run the testsuite in serial mode, and be OK with it taking 2 hours > instead of 30 mins to complete a run... > It is a problem with parallelization, just like aborting a test run is a problem when running parallel tests. > The parallel mode aggregates the summaries from the different runtest > invocations as a single summary at the end, > > https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=contrib/dg-extract-results.py;h=30aa68771d491451f72d6dbd18f6d5ac339451b5;hb=HEAD > > so I don't see what problem here is unsolvable. Again, the issue is that Tcl > ERRORs aren't tallied up on the DejaGnu summary, so the aggregated summary > doesn't contain those either. > The ERRORs will be printed a second time, so all the script needs to do is collect text that appears after the summary and starts with "ERROR:". There seems to also be some confusion between Tcl errors and the ERROR messages produce by DejaGnu's perror procedure (or its internal equivalents). Would it be better to report Tcl errors with "CRASH" instead of "ERROR"? >>> You'll need to >>> look at the testsuite .sum .log files anyway. Only, you will >>> find out that one testcase bombed out, 50000 tests ran, but 1000 others >>> didn't run because of that one testcase. And now you're missing >>> those 1000 results, which maybe included those which you were >>> actually interested in, because you were doing a builbot try run >>> across a number of systems. >>> >>> >> This also applies on a smaller scale for which we have no workaround: gdb.foo/bar.exp bombs out and the remaining /N/ tests in that file did not run. The only real solution is for the test scripts to avoid causing Tcl errors in the first place; anything else is papering over the problem with duct tape. >> > > That's not the same thing. The remaining /N/ tests in the file depend > on the previous /M/ tests. E.g., > > gdb_test "step" > gdb_test "print foo" > > If the "step" fails, the "print foo" test is meaningless. > > However, there should be absolutely no dependency between foo.exp and > bar.exp. It should be possible to run those two different testcases > in any order. Thus, it should be irrelevant to foo.exp whether bar.exp > bombs out of not. Consequently, there's no real reason to not let > foo.exp run regardless of what happens to bar.exp. > This will need to be documented... another item for the local TODO list. >>> Now >>> you wish you had remembered to type --keep-going, but it's not >>> the default, and nobody ever writes Tcl bugs, so you didn't think >>> of it before. >>> >>> >> Note that the snapshot of GDB I have been using for testing this patch does have such a Tcl bug in its testsuite. I did not make any particular effort to choose this snapshot, it was just the current state of GDB master when I started. I have not even looked at *which* commit it is. >> > > Can you tell me what the error looks like? > With commit 7b958a48e1322880f23cdb0a1c35643dd27d3ddb and GDB built without Python support: ERROR: tcl error sourcing /home/jacob/arena/gdb-check/build-queue-test/gdb/testsuite/../../../src/gdb/testsuite/gdb.python/py-format-string.exp. ERROR: invoked "continue" outside of a loop -- Jacob From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 27 13:29:54 2020 Received: (at 41824) by debbugs.gnu.org; 27 Jun 2020 17:29:54 +0000 Received: from localhost ([127.0.0.1]:44532 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpEeI-00009f-8E for submit@debbugs.gnu.org; Sat, 27 Jun 2020 13:29:54 -0400 Received: from mail-ot1-f54.google.com ([209.85.210.54]:45908) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpEeH-00009S-4w for 41824@debbugs.gnu.org; Sat, 27 Jun 2020 13:29:53 -0400 Received: by mail-ot1-f54.google.com with SMTP id m2so11603253otr.12 for <41824@debbugs.gnu.org>; Sat, 27 Jun 2020 10:29:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=lZaVwkABzUQn2nAT3EtzebmC2B89LfUOkb20s15C0TE=; b=izI38jtFtwrHODJEgG0MwV0Ukzs0NEewGrPVd/sQfpphfsiX3W2O7K+CSX3erM3ARs UqGM/oVubUHoolwrhI6m8ShzXvd7A83xdTaCvojCj8/6y5aWckTDtAlpCeIavq734uRs CznNoERvNT9S0wTJnFRUacykcIEqlWMKSM7zNears1H+36BoW+ZJHisssIDf+tTcDwfK GNcCGR8h4IWtITJywPLYW0ZoI/Q6buiScqjFQSM0eEH6GVWwuYkHr8rgKZZBXVrr7lvE Dw9vUR+LxbgB26CQpU/PUDcbtJRRRLEUS/vwWpLiQBMN1iLV/Hn2uchAULWC13bMQ1yx DcpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:reply-to:user-agent :mime-version:to:cc:subject:references:in-reply-to :content-transfer-encoding; bh=lZaVwkABzUQn2nAT3EtzebmC2B89LfUOkb20s15C0TE=; b=CBqdaqs5CpARPTRd8ua2VVstE6dGo8NnOPF+FdtlJ+Jr+vODKSvHv/TrcbmDNgpKsI bXpHYFrUtSZb3p+uBcgU31Kb5CIVkKK+PASRlH7UepvYrv8FTXhCnsocZkNnAqR/A1FA WLOGu2hUX6NNeThZ2LLTz51J2yi5lQ3ChKVvbFBKZdjA32hRbVqi0w3eGoA5+LmO8DsV kcbo375PwdygCxyOC3N4Uq59gaOSxpxcbiVFEKJsj6PVmgqniKJzQLha1kht4w1Go//c 8vcos5FFYdlqDoHJy2LkDPhXzUljShePje/md/5VE//Vbx8A2OmrpE+USkiTUdRq5/es etmA== X-Gm-Message-State: AOAM532JWz1zXimbw7BEuiZhkSol+GD2QqyYQBUkYQZv+aJot8eNeprm +j4HEPP3/CEKQOHx+NvWABU= X-Google-Smtp-Source: ABdhPJxoVIBUAidsbq2NPJHnSaoaxl2GJiBeGJ+yWtqG4PFsYNWBpCaZA8wVHTytwE8EqSSmA8Nltw== X-Received: by 2002:a9d:3b8b:: with SMTP id k11mr6975359otc.240.1593278987573; Sat, 27 Jun 2020 10:29:47 -0700 (PDT) Received: from [192.168.2.42] (adsl-70-133-144-251.dsl.ablntx.sbcglobal.net. [70.133.144.251]) by smtp.gmail.com with ESMTPSA id w75sm1756541oie.58.2020.06.27.10.29.46 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 27 Jun 2020 10:29:46 -0700 (PDT) Message-ID: <5EF78209.8000509@gmail.com> Date: Sat, 27 Jun 2020 12:29:45 -0500 From: Jacob Bachmeyer User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Rob Savoye Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <32b06c6c-560b-3637-0c0d-84248357e591@senecass.com> <5EF5552A.6040001@gmail.com> <1d504891-c277-af13-ab49-d6d6df30221a@senecass.com> In-Reply-To: <1d504891-c277-af13-ab49-d6d6df30221a@senecass.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 41824 Cc: Pedro Alves , Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jcb62281@gmail.com Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) Rob Savoye wrote: > On 6/25/20 7:53 PM, Jacob Bachmeyer wrote: > >> The original rationale for using UNRESOLVED here was that DejaGnu >> converts a test result to UNRESOLVED if too many errors or warnings were >> produced. The DejaGnu manual indicates (section "A POSIX Compliant Test >> Framework") that UNRESOLVED is correct for a test where execution was >> interrupted or was set up incorrectly. UNTESTED is specifically listed >> as a placeholder for an as-yet-unwritten testcase. A "typical" GDB >> testsuite run has almost a hundred UNTESTED results but (without this >> patch) zero UNRESOLVED results. >> >> To me, this seems that UNRESOLVED is correct here, or that the manual >> has an error. >> > > So having written both, there is always the chance I've interpreted > things differently. If the test is interrupted, or has errors/warnings, > like timing problems for example, then it's UNRESOLVED. But a bug in Tcl > code enough to trigger unknown is UNTESTED, as the test never really > ran. I'd go with which definition the toolchain teams prefer, as this > effects validation testruns. I could go either way. > I see UNTESTED as a placeholder for a yet-to-be-written test; DejaGnu does not produce an error exit status after reporting UNTESTED. Once Tcl has begun to execute "source $test_file_name", I see the test as running from the framework's perspective, so an error exit should be UNRESOLVED: we do not know the result that would have been produced had the Tcl error not occurred. > I don't know if POSIX 1003.3 is still considered a standard, that was > a long time ago. The testsuites were primarily focused on XOpen and > POSIX conformance tests, not toolchains. I was > on the standards committee, so used them as a model cause they seemed a > good standard. But the details of the interpretation can now be whatever > we want it to be. We're probably the only ones left running TET > compliant testsuites anyway. :-) > POSIX 1003.3 appears to have been withdrawn. > Anyway, whether a test run aborts on an error or not is not defined, > so it's up to us to decide. Apparently, GDB has baked an assumption that the test run will not abort, ever, into their testsuite, and we had never documented circumstances that cause DejaGnu to abort, so I cannot really answer that with "your assumption is wrong". -- Jacob From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 27 13:42:14 2020 Received: (at 41824) by debbugs.gnu.org; 27 Jun 2020 17:42:14 +0000 Received: from localhost ([127.0.0.1]:44561 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpEqE-0000Ta-HF for submit@debbugs.gnu.org; Sat, 27 Jun 2020 13:42:14 -0400 Received: from mail-ot1-f41.google.com ([209.85.210.41]:41399) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpEqD-0000TC-Dh for 41824@debbugs.gnu.org; Sat, 27 Jun 2020 13:42:13 -0400 Received: by mail-ot1-f41.google.com with SMTP id k15so11615898otp.8 for <41824@debbugs.gnu.org>; Sat, 27 Jun 2020 10:42:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=6GbXzGCyuS5ZdWjBnoeAyXZAj5cMNWHo9BPSiNqV0NA=; b=kHFEBuSI8SOu+VPk07j8gUTWXFKWPd8OKUyvrsndwNSS1rcDMj73ezgTGBnfLWhCLF D+bNWFSSa+QK8UXLBt2wY6SccjR//ji4fxRGaSfsOKUNBTHpDGyKNFjM7x3lSHxVvt+/ 0Im44VfD4Nli/jmlaGmaULeXo3GpVsFYsdqU1nTtV7k6P9m82u2eNGVRf6SZdg2ju9dz tU2y+airhicsWs9rJfLMeSJcAg1ZPXNqNhO5iA9+UzNnA4kmRH/GNVDJTaPJ3LiEER/j 06JJxq1aXp3DHKbCLzl5VU5ERZ08hFFkYXvrV9cadQpLLO3P6Mb2lrA/RQKhcTbEZcdm tsvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:reply-to:user-agent :mime-version:to:cc:subject:references:in-reply-to :content-transfer-encoding; bh=6GbXzGCyuS5ZdWjBnoeAyXZAj5cMNWHo9BPSiNqV0NA=; b=a0h8MpUBv6FINwHdvFIyqBKE7Cz2leObJnz7q+h5OQqewiEZF8hqGvJKWJQHk3mR/T ysjBg98vWrWc/w5XzxFQbFZ4NVkDl7Grz4nFoms3pUBsNn29FioL9mfpi1K0DCAOU3Ni zYxnHa1DnRn8WvXUBZlgHvw0J5+XEi0n/ZIdm7ezIP2auDDuNP28Ns5HmpTk34zFb+TB 3kx714XZsvifwvCQcmJuXadbCK/+rsxDqlvFri+fPrrDORt4yIZb40RJ7XCDDPiApwwM gVpEsYIXM6p47LOQQy1vsg2K8XY4Yw6KgVcJOvJYG5mugm0VZkQKOLXftFTXeMNOM32+ 73gQ== X-Gm-Message-State: AOAM532/DWNvy2ZEmzO0kYfu5TLYPuj5ch0tqa32nDQK74Wp76h0UIED uuX8r0ODSKQfhv2Fw9TQhSY= X-Google-Smtp-Source: ABdhPJyxdqwMfP9qdpBb7acw8QZl7f40115eRTduIuDy3LPRqrrwQ6pJ0GrvO72IoVgB5Bm1oTqSuw== X-Received: by 2002:a9d:3bb6:: with SMTP id k51mr6993128otc.168.1593279727756; Sat, 27 Jun 2020 10:42:07 -0700 (PDT) Received: from [192.168.2.42] (adsl-70-133-144-251.dsl.ablntx.sbcglobal.net. [70.133.144.251]) by smtp.gmail.com with ESMTPSA id q128sm6983122oig.25.2020.06.27.10.42.06 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 27 Jun 2020 10:42:07 -0700 (PDT) Message-ID: <5EF784ED.4030405@gmail.com> Date: Sat, 27 Jun 2020 12:42:05 -0500 From: Jacob Bachmeyer User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Pedro Alves Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <5EF527BE.9000806@gmail.com> <5EF67D81.30604@gmail.com> <16ece36c-7173-0afe-8e0e-7efc5d1e3eab@redhat.com> In-Reply-To: <16ece36c-7173-0afe-8e0e-7efc5d1e3eab@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 41824 Cc: 41824@debbugs.gnu.org, Tom de Vries , Pedro Alves X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jcb62281@gmail.com Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) Pedro Alves wrote: > On 6/26/20 11:58 PM, Jacob Bachmeyer wrote: > > >>> The issue for me is that the "gdb Summary" tally did not >>> mention the aborted testcase. I would like to see something >>> like this instead: >>> >>> # of expected passes 208 >>> # of nasty OMG FIX! tcl errors 1 >>> >>> :-) >>> >>> Your UNRESOLVED is of course better than the status quo. >>> >>> >> UNRESOLVED is a step in the right direction, but I agree that more is needed. We are somewhat limited by POSIX, which does not define a separate result type for "test case crashed" and seems to roll that into UNRESOLVED, although POSIX appears to require a testsuite to have a strictly-defined set of tests and that a run must produce results for exactly all of them, which requires information DejaGnu does not normally have about the running testsuite. So, while UNRESOLVED is needed, we could also maintain a separate count of UNRESOLVED-due-to-crash, or simply store away the error message/errorCode/errorInfo tuples and emit them a second time at the end of the run, also easy in Tcl 8. >> > > I'm not sure how much is POSIX compliance important these days. I would hope that that > would not limit DejaGnu's development and limit its potential to be more usable. We already have extensions; POSIX only allowed PASS/FAIL/UNTESTED/UNRESOLVED/UNSUPPORTED. [KX]{PASS,FAIL} are GNU extensions. > Even > if POSIX compliance is important, I would think that that could be put behind > a --posixly_correct option, and/or POSIXLY_CORRECT environment variable, like other > GNU tools do: > > https://www.gnu.org/prep/standards/html_node/Non_002dGNU-Standards.html > > Or conversely, we could have a --gnu mode, though I would say that a GNU tool > should behave in GNU mode by default. > POSIX compliance in DejaGnu depended on the testsuite being run. Any results generated by the framework itself need to be in the POSIX set, though testsuites can use extensions, as GDB has done. Actually adding more test result codes is dubious, as that will interact badly with XML output and efficient database processing, both of which can reasonably enumerate the nine result codes. ({,K,X}{PASS,FAIL}, UN{TEST,RESOLV,SUPPORT}ED) > I really don't buy that adding different result types is an issue, as > hinted earlier. We've actually done it in GDB, for a loooong while. We > have distinct XFAIL vs KFAIL output messages show up in the .sum file (though > those aren't distinguished in the summary file, doing so would be convenient, > hence I see that as a GDB testsuite bug). > > And just very recently, we've added PATH and DUPLICATE output messages, > and those do show up in the summary: > > # of expected passes 89593 > # of unexpected failures 547 > # of expected failures 146 > # of known failures 104 > # of untested testcases 20 > # of unresolved testcases 5 > # of unsupported tests 97 > # of paths in test names 1 <<<<< > # of duplicate test names 370 <<<<< > > I would think that even these two would be useful to have in > DejaGnu proper for all testsuites. > > v1/rationale - https://sourceware.org/pipermail/gdb-patches/2020-April/167847.html > v3/final version - https://sourceware.org/pipermail/gdb-patches/2020-April/168105.html > I will look at the exact details later, but this sounds like it is probably a monkeypatch that will stop working in some future version when the procedures you are overriding disappear into an internal namespace, so we need to be thinking about what would be needed to provide extensible summary line sets, just as we need some way to extend language support in default_target_compile without overriding the entire procedure. -- Jacob From debbugs-submit-bounces@debbugs.gnu.org Sat Jun 27 13:53:28 2020 Received: (at 41824) by debbugs.gnu.org; 27 Jun 2020 17:53:28 +0000 Received: from localhost ([127.0.0.1]:44571 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpF16-0000k9-4L for submit@debbugs.gnu.org; Sat, 27 Jun 2020 13:53:28 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:29644 helo=us-smtp-1.mimecast.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jpF13-0000k1-HV for 41824@debbugs.gnu.org; Sat, 27 Jun 2020 13:53:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1593280404; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lbGnNIO3kIZnj6YvBtN3cr+oITgY9aAZ3WaKhdeyBfA=; b=ANGzxu4jSKRAIi4y1zSliTq4NRROiRpjIYGt0x4BW3t4KzamrHtxWqQS0r/uuS0lkeueOg S7wWj/He1aTqNIgoOCkGMkZvdYx0Q6jB1JxhpEij1ZW9JMeiXLC23gXGeP8FoH940g/Tyg Ctl5otpVcsqLAubqkMbCfm+IrovZaLE= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-134-QBXu4a-6O_uD2LrWdu_wtw-1; Sat, 27 Jun 2020 13:53:23 -0400 X-MC-Unique: QBXu4a-6O_uD2LrWdu_wtw-1 Received: by mail-wr1-f70.google.com with SMTP id w4so11656890wrm.5 for <41824@debbugs.gnu.org>; Sat, 27 Jun 2020 10:53:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=lbGnNIO3kIZnj6YvBtN3cr+oITgY9aAZ3WaKhdeyBfA=; b=e3c7AllI4ppLGbxzygYBVvlSfkegVnZm2mB3WZ4jw0QYxmuXsmB8beZuuNeD/gyVSe ZhKaTlu9kW0yRCi3YkSvzRFtUh3m98LP3/n94Y3endoF0iK+jwv8imh78lqzGz4IMjWS jfGi9JItfpnZ7nrgnua94AyDHiIo/yiSe9jye+ASOKh2A75RPTCxbbHW0z87oAPeMd5B X5oY5OtwG/63pMHJkVTTH6T+EmglUFSsvLHJZ0ODJOZ0zmDwBxjLZfgfO7MsesrnP9E2 wBWJlga3NerbF3/6AudvrSnLURrhAvXwV0C3SUQCvjKWKU26g+jH1BqHtVd+XFIoea33 ifAQ== X-Gm-Message-State: AOAM530NSUm6p4wbDBr6/qr0KPcfkuSR343qCC7nAqH2mEzmkbHaAQCq y44tspdk/Rci7t9tAlvbHMF8i/7HsaQlJClNvCwkd7Mih201YvW5P4SNg0c2QHG1glQAvG0TYkU M0RvIeiBacZDpiek= X-Received: by 2002:a5d:46c7:: with SMTP id g7mr9371650wrs.365.1593280401587; Sat, 27 Jun 2020 10:53:21 -0700 (PDT) X-Google-Smtp-Source: ABdhPJysbTyGU9RLP1GJgjtZENcn159c5S+/ZjN1ZvAOgu0y0n/Ph8ri8YRznvlHXoQtU06vDvON3g== X-Received: by 2002:a5d:46c7:: with SMTP id g7mr9371638wrs.365.1593280401368; Sat, 27 Jun 2020 10:53:21 -0700 (PDT) Received: from ?IPv6:2001:8a0:f922:c400:d9b9:b3d0:d289:9385? ([2001:8a0:f922:c400:d9b9:b3d0:d289:9385]) by smtp.gmail.com with ESMTPSA id s8sm35747658wru.38.2020.06.27.10.53.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 27 Jun 2020 10:53:20 -0700 (PDT) Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case To: jcb62281@gmail.com References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <5EF527BE.9000806@gmail.com> <5EF678A3.9040607@gmail.com> <5EF7801E.3040909@gmail.com> From: Pedro Alves Message-ID: <1e51f125-3fde-6d3c-17fa-a0ae38303444@redhat.com> Date: Sat, 27 Jun 2020 18:53:19 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <5EF7801E.3040909@gmail.com> Content-Language: en-US X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 6/27/20 6:21 PM, Jacob Bachmeyer wrote: > Pedro Alves wrote: >> On 6/26/20 11:37 PM, Jacob Bachmeyer wrote: >>> Note that the snapshot of GDB I have been using for testing this patch does have such a Tcl bug in its testsuite.  I did not make any particular effort to choose this snapshot, it was just the current state of GDB master when I started.  I have not even looked at *which* commit it is.     >> >> Can you tell me what the error looks like? >>   > > With commit 7b958a48e1322880f23cdb0a1c35643dd27d3ddb and GDB built without Python support: > ERROR: tcl error sourcing /home/jacob/arena/gdb-check/build-queue-test/gdb/testsuite/../../../src/gdb/testsuite/gdb.python/py-format-string.exp. > ERROR: invoked "continue" outside of a loop Thanks. It's already been fixed: commit 05e682e3be7e3d9d63ec358dcf8943fd200545cb Author: Sandra Loosemore AuthorDate: Wed Jun 17 13:43:32 2020 -0700 Commit: Sandra Loosemore CommitDate: Wed Jun 17 13:43:32 2020 -0700 Fix TCL error in gdb.python/py-format-string.exp. 2020-06-17 Sandra Loosemore gdb/testsuite/ * gdb.python/py-format-string.exp: Move test for python support earlier, out of function body. Thanks, Pedro Alves From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 30 00:40:55 2020 Received: (at 41824) by debbugs.gnu.org; 30 Jun 2020 04:40:56 +0000 Received: from localhost ([127.0.0.1]:49954 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jq84l-00030C-JL for submit@debbugs.gnu.org; Tue, 30 Jun 2020 00:40:55 -0400 Received: from mail-ot1-f51.google.com ([209.85.210.51]:35972) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jq84h-0002zw-F9 for 41824@debbugs.gnu.org; Tue, 30 Jun 2020 00:40:54 -0400 Received: by mail-ot1-f51.google.com with SMTP id 72so17421705otc.3 for <41824@debbugs.gnu.org>; Mon, 29 Jun 2020 21:40:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=pzymAmenCY7hNFnzkL/REV4nZO9v86eNlJEesELQB3A=; b=uh0iub2C8DvaPS3y7urMPo76FjJLHxzVMaAbGqcmWV2sU55qlvJpuSMZY7PNxkMSgJ TA+r92siDqoGTQEcRJMXR0FdBa7tzdSOXHtYonAb2sOZjHnMPMykgdIWObXK6sg5hELz /fvsRvcgu7n9k5AqoyoXNEQhj8YrdVODBY+2NYP+TUV+JUxp82zbuvU1pEiM6oL1gZw3 tAliu5k9fHawZvWQOFsyNJbRlgs4RBvFSXvupdTIXt33KEdI5IAViD2rZRdWKPMt7tQd Fer9wwzwrib91q6nToXLZPBkXI913f9FozLriO5H0PHTojvAzKgPLi+Zb0nqYBh7McMw V8WQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:reply-to:user-agent :mime-version:to:cc:subject:references:in-reply-to :content-transfer-encoding; bh=pzymAmenCY7hNFnzkL/REV4nZO9v86eNlJEesELQB3A=; b=KQp8N5ikiZk6bv2xDDLeczwu4pa8dHMC8kBl6hJeix9TcCX24k3kViTpHyc9qjLrMw dm9sQjQN5I3ESWJZ/aBMiVpHwZreHw534yEomv3/ClaYcgXM56rotniUzxqYSunG+uP1 dQTw1QFQLUAy2r2xQnsJ6VFn4eRkxm2I4LcIRsJuppNBpbOZfA/+BvKpWVZwA7ChI0Np bU4GwrzUI4NpMrGVbcRL+8TXpUNdUO7kS4LrD70TyEZaz+P0kRCj7pXLINhKcNjZD9K4 UJEeA1Ew8MNNXPuuMRRnnzuRMNyFllMuCY5ARgzzM73jolM/l0bgLm5C1n6wvs1IKSY+ 7fuQ== X-Gm-Message-State: AOAM532+hv32xAk83oIH3VMl2shARLZ1KwyiISGzNvoRZrx+PCHRi60Q LeS5vgWGnV49Exx3uE3kbZQ= X-Google-Smtp-Source: ABdhPJwTsS89KAEyMmtEj1PhkAVTZ3JtecIcCCgSr1JDBIr42Q4NLdRXp9ppBVPMBouJSEuUVOOq+w== X-Received: by 2002:a9d:387:: with SMTP id f7mr7568444otf.37.1593492045684; Mon, 29 Jun 2020 21:40:45 -0700 (PDT) Received: from [192.168.2.42] (adsl-70-133-144-251.dsl.ablntx.sbcglobal.net. [70.133.144.251]) by smtp.gmail.com with ESMTPSA id w83sm445838oig.20.2020.06.29.21.40.43 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 29 Jun 2020 21:40:44 -0700 (PDT) Message-ID: <5EFAC24B.9030505@gmail.com> Date: Mon, 29 Jun 2020 23:40:43 -0500 From: Jacob Bachmeyer User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Pedro Alves Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <5EF527BE.9000806@gmail.com> <5EF678A3.9040607@gmail.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jcb62281@gmail.com Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) Pedro Alves wrote: > On 6/26/20 11:37 PM, Jacob Bachmeyer wrote: > >> Pedro Alves wrote: >> >>> And again, people run the GDB testsuite in parallel mode, so your >>> assumption that the testsuite would stop immediately anyway doesn't >>> hold. One of the parallel runtest invocations bombs >>> out, while the others keep running, so by the time you get >>> back the prompt, other testcases already executed, and you don't >>> see the ERROR in the current terminal anymore. >>> >> This is a fundamental problem with slicing up the testsuite external to DejaGnu and running multiple instances of runtest -- output that is supposed to be "last at the end of the run" can be hidden away by other ongoing subset runs. >> > > That's like arguing against parallelization, and saying that everyone > should run the testsuite in serial mode, and be OK with it taking 2 hours > instead of 30 mins to complete a run... > > The parallel mode aggregates the summaries from the different runtest > invocations as a single summary at the end, > > https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=contrib/dg-extract-results.py;h=30aa68771d491451f72d6dbd18f6d5ac339451b5;hb=HEAD > > so I don't see what problem here is unsolvable. Again, the issue is that Tcl > ERRORs aren't tallied up on the DejaGnu summary, so the aggregated summary > doesn't contain those either. > The current version of the fix for this issue (commit 61dc0cafad8845b3c668940ed2e574bd503d410f on temporary branch PR41918) generates separator lines containing at least 10 hyphens around each error when repeating the errors after the summary. The leading separator line currently begins with "ERROR: " due to limitations of the internal interfaces, but plans are to eventually fix that and collapse the separator lines between adjacent error reports if multiple Tcl errors occur in a run. Each error report begins with "ERROR: in testcase " and ends with a Tcl backtrace and a line of at least 10 hyphens. Is this something the GDB results aggregation tool can be extended to handle? -- Jacob From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 30 08:26:42 2020 Received: (at 41824) by debbugs.gnu.org; 30 Jun 2020 12:26:42 +0000 Received: from localhost ([127.0.0.1]:50392 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jqFLW-0008Da-7M for submit@debbugs.gnu.org; Tue, 30 Jun 2020 08:26:42 -0400 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:52071 helo=us-smtp-delivery-1.mimecast.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jqFLS-0008DP-7L for 41824@debbugs.gnu.org; Tue, 30 Jun 2020 08:26:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1593519997; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5IbP149u6zpJU0mrDpKfRypGG2WVl0gXnC+xeq+xjH8=; b=Rc1HlqoiHX8+9BZpQUBqvJzAfkCChKRvy+2k0s+tBt9Cgvb70E59BL6kcFBneoud7NiMb9 f8Qr1ir58QBolFg3LVRof53SzJon/Ym7bFSNpNlT/bLd3/fuNTI+RTHvYI8Egl9YaWZJ7v nL7BKfDnO4zYH/khgsubH7IFJi51muw= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-165-BENvmjKROh2vxgTfMITB8Q-1; Tue, 30 Jun 2020 08:26:30 -0400 X-MC-Unique: BENvmjKROh2vxgTfMITB8Q-1 Received: by mail-wm1-f69.google.com with SMTP id b13so18829759wme.9 for <41824@debbugs.gnu.org>; Tue, 30 Jun 2020 05:26:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=5IbP149u6zpJU0mrDpKfRypGG2WVl0gXnC+xeq+xjH8=; b=jeYQVJDHt9Ofl5ABe52LdzhdbYoDT4DV0vxym5vwBkpGD75/mZC7DIDw3D+60mAfxD CpHVm0+tv4qXWmShojp1qHUEhIin774B0ZCzswfGJEUfNWMxJHtTvAg76AhP3D9GKclh VBwCi/W8c7t7DLGpwkjRzGc7NSC0Qw9V6/xaRWwEuyawtyxQB7I1iymbjpm0CP/V50mm GXo7YL2HRGQ63Ro5QRLE1Os9jAQ7eDw2ZPdQtPybWZ0q0O+nKs6JowI5V57B3My2yzeJ qrbkTq++rpkOrghAOBwQAAbIrW9cDmmRVQak+gCfcOAEzAgIuWx4DawgHzYyUaSmvA3I UhSQ== X-Gm-Message-State: AOAM532PMP/bM2CbWk/rOv1amRqR+9ywQ3Wakx1Ix8nARA8kFlA6XZoA NMEK3cj4Awg5H9bDogkedISpKdqC4UVIN414Ub99mk7DkLY68uv+LECVVNFZgkSJhHhGvk3WmHg NOPsxT+tSlK23OHo= X-Received: by 2002:a1c:e908:: with SMTP id q8mr10907831wmc.59.1593519988782; Tue, 30 Jun 2020 05:26:28 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzGcLCKEhwULgtyJjoTh9yCiDSK7MAFmWYwAUbSmEWomhcJknB59KJG0qS/kqEUlWoWhT0EWA== X-Received: by 2002:a1c:e908:: with SMTP id q8mr10907805wmc.59.1593519988519; Tue, 30 Jun 2020 05:26:28 -0700 (PDT) Received: from ?IPv6:2001:8a0:f922:c400:56ee:75ff:fe8d:232b? ([2001:8a0:f922:c400:56ee:75ff:fe8d:232b]) by smtp.gmail.com with ESMTPSA id d81sm29925629wmc.0.2020.06.30.05.26.27 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 30 Jun 2020 05:26:27 -0700 (PDT) Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case To: jcb62281@gmail.com References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <5EF527BE.9000806@gmail.com> <5EF678A3.9040607@gmail.com> <5EFAC24B.9030505@gmail.com> From: Pedro Alves Message-ID: Date: Tue, 30 Jun 2020 13:26:25 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <5EFAC24B.9030505@gmail.com> Content-Language: en-US Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=palves@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 6/30/20 5:40 AM, Jacob Bachmeyer wrote: > Pedro Alves wrote: >> On 6/26/20 11:37 PM, Jacob Bachmeyer wrote: >>   >>> Pedro Alves wrote: >>>     >>>> And again, people run the GDB testsuite in parallel mode, so your >>>> assumption that the testsuite would stop immediately anyway doesn't >>>> hold.  One of the parallel runtest invocations bombs >>>> out, while the others keep running, so by the time you get >>>> back the prompt, other testcases already executed, and you don't >>>> see the ERROR in the current terminal anymore. >>>>       >>> This is a fundamental problem with slicing up the testsuite external to DejaGnu and running multiple instances of runtest -- output that is supposed to be "last at the end of the run" can be hidden away by other ongoing subset runs. >>>     >> >> That's like arguing against parallelization, and saying that everyone >> should run the testsuite in serial mode, and be OK with it taking 2 hours >> instead of 30 mins to complete a run... >> >> The parallel mode aggregates the summaries from the different runtest >> invocations as a single summary at the end, >> >> https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=contrib/dg-extract-results.py;h=30aa68771d491451f72d6dbd18f6d5ac339451b5;hb=HEAD >> >> so I don't see what problem here is unsolvable.  Again, the issue is that Tcl >> ERRORs aren't tallied up on the DejaGnu summary, so the aggregated summary >> doesn't contain those either. >>   > > The current version of the fix for this issue (commit 61dc0cafad8845b3c668940ed2e574bd503d410f on temporary branch PR41918) generates separator lines containing at least 10 hyphens around each error when repeating the errors after the summary.  The leading separator line currently begins with "ERROR: " due to limitations of the internal interfaces, but plans are to eventually fix that and collapse the separator lines between adjacent error reports if multiple Tcl errors occur in a run.  Each error report begins with "ERROR: in testcase " and ends with a Tcl backtrace and a line of at least 10 hyphens.  Is this something the GDB results aggregation tool can be extended to handle? Your guess is as good as mine, since I never touched that code. Thanks, Pedro Alves From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 30 19:49:24 2020 Received: (at 41824) by debbugs.gnu.org; 30 Jun 2020 23:49:24 +0000 Received: from localhost ([127.0.0.1]:51961 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jqQ0C-00027z-0K for submit@debbugs.gnu.org; Tue, 30 Jun 2020 19:49:24 -0400 Received: from mail-oo1-f68.google.com ([209.85.161.68]:41039) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jqQ07-00027k-Vy for 41824@debbugs.gnu.org; Tue, 30 Jun 2020 19:49:22 -0400 Received: by mail-oo1-f68.google.com with SMTP id z23so699863ood.8 for <41824@debbugs.gnu.org>; Tue, 30 Jun 2020 16:49:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=ILWq3qTp2iIwmvm25m09kEaPS6IV5bVrT2jscXCmjx4=; b=b/GtQLli0mRFDx7RKmidIjLmMRQxkv/6xxQBi509egqq9rxOSvrjLFcxa1lbX+xdjp m90zz3upud89u+Z2E4g/KLslaHlcwKZ2Nd7uu8zOJgI32ARtpo1VugjuX7nU4fFF6I3/ 71SDo9yDeFlKdmtduOVodst3uW5tGv4XY6t4NUAhWhqysXGse3JFNZQazjYz3XfPuBol XYp3INEUR8OAxjF69Cxc5CHkuJiZ7vZK+0IbuyFlycZi2alRe00MPpKTjkH9dz6V0D9f JyI7wxDdeNYKjEKM8yVFknHmMerIlNTxuFDIPIxpwpyPCw6nHE1AAxm2ViL54iggoRRM 0UgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:reply-to:user-agent :mime-version:to:cc:subject:references:in-reply-to :content-transfer-encoding; bh=ILWq3qTp2iIwmvm25m09kEaPS6IV5bVrT2jscXCmjx4=; b=IKtoLYuoOGbcz2p9p0qEf0W4GCalSM9v3af+UX949EiTgjERMpbaKmmza/qzfNifXJ aCiJ8U50eb77wtbbQhgNPoihVf77YTkzIs7pDR1+mD5k3Wz+clKx4z/GKR1e9o3HpmiO MokUtkkZW/cUPQvtwIrINdz0OYx7h5MfKVAlt6BeediSpcYqYoSJdoQ018tEpJ0/e+iL qtYrNimshxp2LSI4QEhNXMgM21iyTdUatH+r/9KcYPMaA/tHJ0rpFgUl5xNjdHirpZtH F7PfP2YzftNI5wS3C00OWl3UYPUK+XIgQ6k7INn7FnIYWZ537/BUuTbGO2wkmW7BU40k 0xYQ== X-Gm-Message-State: AOAM532m6qRii/9O/S1CXrCKX9URRhoX5hJhxxXxiBakA7XuVUbwadMu Nx2eDLupfZKAht02yOiu/N8= X-Google-Smtp-Source: ABdhPJxaqouFk3R+rCTsZNwpQseWWIYA7atEpp9vLD9xjE2IU8e0zNEvqfFhGNdN3riUb27nZdAMxA== X-Received: by 2002:a4a:c4c1:: with SMTP id g1mr19940141ooq.93.1593560954065; Tue, 30 Jun 2020 16:49:14 -0700 (PDT) Received: from [192.168.2.42] (adsl-70-133-144-251.dsl.ablntx.sbcglobal.net. [70.133.144.251]) by smtp.gmail.com with ESMTPSA id p18sm1092726oto.11.2020.06.30.16.49.12 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 30 Jun 2020 16:49:13 -0700 (PDT) Message-ID: <5EFBCF77.8090701@gmail.com> Date: Tue, 30 Jun 2020 18:49:11 -0500 From: Jacob Bachmeyer User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Pedro Alves Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <5EF527BE.9000806@gmail.com> <5EF678A3.9040607@gmail.com> <5EFAC24B.9030505@gmail.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 41824 Cc: Tom de Vries , 41824@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jcb62281@gmail.com Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) Pedro Alves wrote: > On 6/30/20 5:40 AM, Jacob Bachmeyer wrote: > >> Pedro Alves wrote: >> >>> On 6/26/20 11:37 PM, Jacob Bachmeyer wrote: >>> >>> >>>> Pedro Alves wrote: >>>> >>>> >>>>> And again, people run the GDB testsuite in parallel mode, so your >>>>> assumption that the testsuite would stop immediately anyway doesn't >>>>> hold. One of the parallel runtest invocations bombs >>>>> out, while the others keep running, so by the time you get >>>>> back the prompt, other testcases already executed, and you don't >>>>> see the ERROR in the current terminal anymore. >>>>> >>>>> >>>> This is a fundamental problem with slicing up the testsuite external to DejaGnu and running multiple instances of runtest -- output that is supposed to be "last at the end of the run" can be hidden away by other ongoing subset runs. >>>> >>>> >>> That's like arguing against parallelization, and saying that everyone >>> should run the testsuite in serial mode, and be OK with it taking 2 hours >>> instead of 30 mins to complete a run... >>> >>> The parallel mode aggregates the summaries from the different runtest >>> invocations as a single summary at the end, >>> >>> https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=contrib/dg-extract-results.py;h=30aa68771d491451f72d6dbd18f6d5ac339451b5;hb=HEAD >>> >>> so I don't see what problem here is unsolvable. Again, the issue is that Tcl >>> ERRORs aren't tallied up on the DejaGnu summary, so the aggregated summary >>> doesn't contain those either. >>> >>> >> The current version of the fix for this issue (commit 61dc0cafad8845b3c668940ed2e574bd503d410f on temporary branch PR41918) generates separator lines containing at least 10 hyphens around each error when repeating the errors after the summary. The leading separator line currently begins with "ERROR: " due to limitations of the internal interfaces, but plans are to eventually fix that and collapse the separator lines between adjacent error reports if multiple Tcl errors occur in a run. Each error report begins with "ERROR: in testcase " and ends with a Tcl backtrace and a line of at least 10 hyphens. Is this something the GDB results aggregation tool can be extended to handle? >> > > Your guess is as good as mine, since I never touched that code. > After briefly looking through that script, I expect that it will not be particularly difficult, although I also note that the script counts the error messages generated by DejaGnu's ::unknown procedure, so lib/framework.exp:unknown cannot be withdrawn as useless until 1.7. -- Jacob From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 30 19:55:03 2020 Received: (at 41824) by debbugs.gnu.org; 30 Jun 2020 23:55:03 +0000 Received: from localhost ([127.0.0.1]:51965 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jqQ5e-0002GQ-OS for submit@debbugs.gnu.org; Tue, 30 Jun 2020 19:55:03 -0400 Received: from mail-oi1-f178.google.com ([209.85.167.178]:39221) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jqQ5b-0002Fn-MX; Tue, 30 Jun 2020 19:55:01 -0400 Received: by mail-oi1-f178.google.com with SMTP id w17so18295719oie.6; Tue, 30 Jun 2020 16:54:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:reply-to:user-agent:mime-version:to:subject :content-transfer-encoding; bh=Foap12gIr59wZDDUy/+lYT5U6f+kcCCrw5YdW40ing0=; b=JvDJEYBCuS8d/k+VV72T5taLdSlJOHphYQoUsZO+IICajvBn0JCeHJVv97iCNpVWdg t+y5ZzcKiuMe15jSk1R5v08RtIEgNjSn8AoP7buVHnlr9pBCHwHc86QUoxXECuttoSTK IFLUacJlajq4U/3K0PLORUXwTPJGIOwSKYUSjERBzAtQz6cZsRzMpSF/Rw33CKQTa5lZ g+RHpJ/p9Y03fQrsUHbDq4oQ5HbS3hwMq+4JA1+fprruDSN1G+82eQf2oWY+tbNa/IYl SPCCFoMy+lM95A0IBTOcKid6JgtXKH+7+eOgq2n3ceAusCphEJoubJYPJA7zoA+1KpnB ETWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:reply-to:user-agent :mime-version:to:subject:content-transfer-encoding; bh=Foap12gIr59wZDDUy/+lYT5U6f+kcCCrw5YdW40ing0=; b=n9Vu0ziYLTnTYxJHTS6C/ftzDhM17ptCPJSMyEhJTfUG8nsgEDXBxdgxSzpG+YzG/l Y/RvWinVIfgGTCpUylpka49L3Qj+3sfczu4++sV3CzuESGzHSxVVY4ELgT91Rej3ggkY 6d3LJYcv/LdvsOwo4QlHo9y8p5TprOUPewtOpWi+5Nb3POiZiJRSh51bwpuDF51O+q2C SRIkSDraQQnfNeNhUYxvf5EKydoJ+9x/qfRYX/IC4L+7TNo3+tD4HWsGMICoxFEpht7i 5DC2u3DHKQCysuq+0+Mhx7j7A9wb8m71YioKat1vDUiqjZJiQMQinal8y84A3btfDbYd ozxw== X-Gm-Message-State: AOAM530s6a4v70POQw7JD1DhuHfVUGq9O4iHkiomPVwlErCOjsDFMF1a adAa240BBUTDGmPSR7krxPxhApA3 X-Google-Smtp-Source: ABdhPJz/zONAUrVBNjVE6/81B5I7VVv2O9isrTLz2RZEkILmt023LK5oU5eDTa8735VXCkOo9Q8/jw== X-Received: by 2002:aca:8cd:: with SMTP id 196mr17675099oii.148.1593561293812; Tue, 30 Jun 2020 16:54:53 -0700 (PDT) Received: from [192.168.2.42] (adsl-70-133-144-251.dsl.ablntx.sbcglobal.net. [70.133.144.251]) by smtp.gmail.com with ESMTPSA id y5sm1098239otq.75.2020.06.30.16.54.52 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 30 Jun 2020 16:54:53 -0700 (PDT) Message-ID: <5EFBD0CB.7090605@gmail.com> Date: Tue, 30 Jun 2020 18:54:51 -0500 From: Jacob Bachmeyer User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: 41824@debbugs.gnu.org, 41918@debbugs.gnu.org Subject: Branch PR41918 containing fixes for these issues is into final testing Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 41824 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jcb62281@gmail.com Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) The temporary branch PR41918 (which earlier absorbed the PR41824 temporary branch) is now merged onto the testing queue ahead of master. After these patches pass full testing, master will be fast-forwarded to include them. -- Jacob From debbugs-submit-bounces@debbugs.gnu.org Wed Jul 08 19:53:07 2020 Received: (at 41824-done) by debbugs.gnu.org; 8 Jul 2020 23:53:07 +0000 Received: from localhost ([127.0.0.1]:39409 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jtJsB-0006VA-B2 for submit@debbugs.gnu.org; Wed, 08 Jul 2020 19:53:07 -0400 Received: from mail-qt1-f171.google.com ([209.85.160.171]:40077) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jtJs9-0006Ud-I1 for 41824-done@debbugs.gnu.org; Wed, 08 Jul 2020 19:53:06 -0400 Received: by mail-qt1-f171.google.com with SMTP id w27so368957qtb.7 for <41824-done@debbugs.gnu.org>; Wed, 08 Jul 2020 16:53:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:reply-to:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=95awLH5kDoCkQqgJJBYZfv0EMbSqAOFozjA8ArH9kmA=; b=RphIJFuKgKsltLyv6whM3nSkXA0TewppdiKnthaZZr3cCwG5hVimW/Qo+mN3tY4qqk iuTXPQlwK3tLkk56gWWyGyRa58m8fz7rV3CKLeps+NnAgSkyxt0v9ZGOH9aMK2JlnHBF 8/uN+SSx7WqgCTGR5y4EDJW97XAv0yH5PjP7aJ98L8Jl7HeaRTdl0CCUWOTKUPPR6uH0 XmW1Vy6TjSknLpnbhK8RgqF5p4iGNz17+c/BNkRYb/ltbBubpD3E5tKB0bUSpid9HKdu H7L4rXeQmUZngQivf3EwvTnLy1Qtlt3lsySWpK6XFWbSOki3PDEV1nmsZn+DgV87CzgM 2oRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:reply-to:user-agent :mime-version:to:cc:subject:references:in-reply-to :content-transfer-encoding; bh=95awLH5kDoCkQqgJJBYZfv0EMbSqAOFozjA8ArH9kmA=; b=jI1plMHv52U+aQMevK3XvSl4L9NBe/oblJ9UybWlS75wl59Jpm72f/z5fNAtxSXzfe M4WdOHsfs76KrOAk3+aPmtIjxpajZpvxZ3052fqs3bEzq94CWfnrwfs5OtXIMC0bgUAV At+bZUeJRNj0bdlbtW3HdRlP6YenMGmaFJc2jp4hn2yv8/NoWDRYtvbIxqpEvuvoLwsy ozj53kmMZA+2d/QkImtT/bLe+GtX31rgos91zYloYGCmzquw78MJw2Gqqk0zgezVLmal eeg/O+ti4Xtckhbe1QbBINAl0Ajt2U+cxhhSdOrgdHqyjdFvM589026rNN47QBQ7W5MA TnDw== X-Gm-Message-State: AOAM533gB2GLqPRhYED5u0H9qoW8Tr467vPsNT3Ru0tPFhTNYq+Q5ACQ K8qn5mbwXs4Xg3nrVM5qKd0= X-Google-Smtp-Source: ABdhPJxUug4KySfBNYOb28bbHC3WmHIjDLG53/9lL1Keh/Fc440IaxafH7Mj5BcDPVZRQ4RMXy/qqw== X-Received: by 2002:ac8:6bc8:: with SMTP id b8mr61083656qtt.198.1594252379809; Wed, 08 Jul 2020 16:52:59 -0700 (PDT) Received: from [192.168.2.42] (adsl-70-133-144-251.dsl.ablntx.sbcglobal.net. [70.133.144.251]) by smtp.gmail.com with ESMTPSA id n64sm1560697qke.77.2020.07.08.16.52.57 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 08 Jul 2020 16:52:58 -0700 (PDT) Message-ID: <5F065C58.7070108@gmail.com> Date: Wed, 08 Jul 2020 18:52:56 -0500 From: Jacob Bachmeyer User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0 MIME-Version: 1.0 To: Pedro Alves Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <5EF527BE.9000806@gmail.com> <5EF678A3.9040607@gmail.com> <5EFAC24B.9030505@gmail.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.5 (/) X-Debbugs-Envelope-To: 41824-done Cc: Tom de Vries , 41824-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: jcb62281@gmail.com Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.5 (/) The bugfix branch has landed on master and will be included in the upcoming 1.6.3 release. -- Jacob From debbugs-submit-bounces@debbugs.gnu.org Thu Jul 09 08:17:17 2020 Received: (at 41824-done) by debbugs.gnu.org; 9 Jul 2020 12:17:17 +0000 Received: from localhost ([127.0.0.1]:40141 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jtVUL-0005wO-23 for submit@debbugs.gnu.org; Thu, 09 Jul 2020 08:17:17 -0400 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120]:35216 helo=us-smtp-1.mimecast.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jtVUI-0005wF-Id for 41824-done@debbugs.gnu.org; Thu, 09 Jul 2020 08:17:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1594297034; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RjdmjwMxIZbV/Kdz7AfvsohjQ+ijsymobUzV3DezJOY=; b=DbOL8GENEnJhjPYZtP4H1kKAeE8tLGUcRPzogr9P7nQisffGfx/47Rn8Wne6nGSqzJjX/V nLRcMSZ+MTyPUgGgvHbNfVIoH32sUOYMoVEkv3Hj/SYBqVkSs6tynu2mzNLeQ7xxgbHjzV 2W7wVi77xlyAOaYHc1Y8p3O/j815UKY= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-324-Ydgl16NaM1WP20z_saEHVw-1; Thu, 09 Jul 2020 08:17:11 -0400 X-MC-Unique: Ydgl16NaM1WP20z_saEHVw-1 Received: by mail-wr1-f72.google.com with SMTP id e11so1780021wrs.2 for <41824-done@debbugs.gnu.org>; Thu, 09 Jul 2020 05:17:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=RjdmjwMxIZbV/Kdz7AfvsohjQ+ijsymobUzV3DezJOY=; b=TuZ/WYiIa1tyxQAd8RUIsZ1xGEQs0DC+H+gujalrF3wMuzsxMr6hEblrd+YODzMldO 0qf/CT0mB1CZf90CIiaOy6T5iyqDoNY5lKfWin1uWm/RCuPAHKyDtA2d5LGG08CRcYbf 0O/k6oj0UxUuyewIqdvkJpbmwTt4kO5tiCSl9PvjGONAkJeWtK6aeFkndqMZosfWUMtb 1SNob1zWfo+karGgsBf1ZvjlUIpcu1i9PvtOU6eMtaYihQptPbOFFhnae1V3y58ciBxP QqJmkpcHR80kA0lkFEB0skPclv1sAQ1MJAdqSUGFFYMMbTJjLfaZ4r/Axjb/a8cd9f5N +0RQ== X-Gm-Message-State: AOAM533tDiXj/2waV4yfgzZBLiPfEm5P4J+xGx3LqGvoO0OqRM/ga7+/ Jd5oyU+T0e6uDfE5fL0cIvZdFbTYaPWN7+hVN0eg3XWzkYv8ZQoeOMcfIh5I+manzgevbrnFug0 1RzCqSzQA2a49HmJ9ZM0aWw== X-Received: by 2002:a1c:9911:: with SMTP id b17mr13443568wme.135.1594297027572; Thu, 09 Jul 2020 05:17:07 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwbt5XbYlCOfg28dcpWk4yfddF7BT9zhhuqDkCiRHKK1Y1DBAkJunfp5p3EyZkMDt/dlqfFNg== X-Received: by 2002:a1c:9911:: with SMTP id b17mr13443546wme.135.1594297027344; Thu, 09 Jul 2020 05:17:07 -0700 (PDT) Received: from ?IPv6:2001:8a0:f91a:c400:56ee:75ff:fe8d:232b? ([2001:8a0:f91a:c400:56ee:75ff:fe8d:232b]) by smtp.gmail.com with ESMTPSA id w14sm5358287wrt.55.2020.07.09.05.17.06 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 09 Jul 2020 05:17:06 -0700 (PDT) Subject: Re: bug#41824: Dejagnu's unknown proc aborts testsuite run when triggered in test-case To: jcb62281@gmail.com References: <5EE8450D.9080609@gmail.com> <5EE98D60.2090005@gmail.com> <43c02976-8b2f-3cd7-44a9-89dc03b26e38@redhat.com> <5EF29212.1050601@gmail.com> <77051534-b804-e0ea-8e3e-c192a2be63a5@redhat.com> <5EF40F6C.8010004@gmail.com> <5EF527BE.9000806@gmail.com> <5EF678A3.9040607@gmail.com> <5EFAC24B.9030505@gmail.com> <5F065C58.7070108@gmail.com> From: Pedro Alves Message-ID: Date: Thu, 9 Jul 2020 13:17:06 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 In-Reply-To: <5F065C58.7070108@gmail.com> Content-Language: en-US X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 41824-done Cc: Tom de Vries , 41824-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.0 (-) On 7/9/20 12:52 AM, Jacob Bachmeyer wrote: > The bugfix branch has landed on master and will be included in the upcoming 1.6.3 release. Thanks for all the work, and for the patience. Pedro Alves From unknown Wed Jun 18 00:26:06 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Fri, 07 Aug 2020 11:24:04 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator