GNU bug report logs - #55067
[PATCH] gnu: nomad-optimizer: Disable failing test.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 55067 in the body.
You can then email your comments to 55067 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Paul A. Patience" <paul <at> apatience.com>
To: guix-patches <at> gnu.org
Cc: "Paul A. Patience" <paul <at> apatience.com>
Subject: [PATCH] gnu: nomad-optimizer: Disable failing test.
Date: Fri, 22 Apr 2022 16:20:36 +0000

* gnu/packages/maths.scm (nomad-optimizer)[arguments]: Disable the
examples/advanced/library/PSDMads test.
---
It turns out the PSDMads test fails randomly.
I've disabled it.

 gnu/packages/maths.scm | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index f4778179d6..26488a614c 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -2008,6 +2008,9 @@ (define-public nomad-optimizer
          (add-after 'fix-sources-for-build 'fix-sources-for-tests
            (lambda _
              (substitute* "examples/CMakeLists.txt"
+               ;; This test passes only sometimes.
+               (("^ +add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/advanced/library/PSDMads\\)\n")
+                "")
                ;; examples/basic/batch/example3 is accidentally omitted.
                (("^(add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/basic/batch/example)2(\\)\n)"
                  _ prefix suffix)
@@ -2052,10 +2055,6 @@ (define-public nomad-optimizer
               "examples/advanced/library/exampleSuggestAndObserve/cache0.txt")

              (let ((builddir (string-append (getcwd) "/../build")))
-               ;; For some reason, omitting this fix causes the
-               ;; examples/advanced/library/PSDMads test to fail, even though
-               ;; it doesn't seem to reference any part of the FixedVariable
-               ;; test.
                (let ((dir "examples/advanced/library/FixedVariable"))
                  (substitute* (string-append dir "/fixedVariable.cpp")
                    (("^( +std::string sExe = ).*" _ prefix)
--
2.35.1

Message #8 received at 55067 <at> debbugs.gnu.org (full text, mbox):

From: "Paul A. Patience" <paul <at> apatience.com>
To: 55067 <at> debbugs.gnu.org
Cc: "Paul A. Patience" <paul <at> apatience.com>
Subject: [PATCH v2] gnu: nomad-optimizer: Disable failing test.
Date: Fri, 22 Apr 2022 16:30:24 +0000

* gnu/packages/maths.scm (nomad-optimizer)[arguments]: Disable the
examples/advanced/library/PSDMads test.
---
Just updating the copyright header too.

 gnu/packages/maths.scm | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index f4778179d6..190b62fca1 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -45,7 +45,7 @@
 ;;; Copyright © 2021 Gerd Heber <gerd.heber <at> gmail.com>
 ;;; Copyright © 2021 Franck Pérignon <franck.perignon <at> univ-grenoble-alpes.fr>
 ;;; Copyright © 2021 Philip McGrath <philip <at> philipmcgrath.com>
-;;; Copyright © 2021 Paul A. Patience <paul <at> apatience.com>
+;;; Copyright © 2021-2022 Paul A. Patience <paul <at> apatience.com>
 ;;; Copyright © 2021 Ivan Gankevich <i.gankevich <at> spbu.ru>
 ;;; Copyright © 2021 Jean-Baptiste Volatier <jbv <at> pm.me>
 ;;; Copyright © 2021 Guillaume Le Vaillant <glv <at> posteo.net>
@@ -2008,6 +2008,9 @@ (define-public nomad-optimizer
          (add-after 'fix-sources-for-build 'fix-sources-for-tests
            (lambda _
              (substitute* "examples/CMakeLists.txt"
+               ;; This test passes only sometimes.
+               (("^ +add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/advanced/library/PSDMads\\)\n")
+                "")
                ;; examples/basic/batch/example3 is accidentally omitted.
                (("^(add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/basic/batch/example)2(\\)\n)"
                  _ prefix suffix)
@@ -2052,10 +2055,6 @@ (define-public nomad-optimizer
               "examples/advanced/library/exampleSuggestAndObserve/cache0.txt")

              (let ((builddir (string-append (getcwd) "/../build")))
-               ;; For some reason, omitting this fix causes the
-               ;; examples/advanced/library/PSDMads test to fail, even though
-               ;; it doesn't seem to reference any part of the FixedVariable
-               ;; test.
                (let ((dir "examples/advanced/library/FixedVariable"))
                  (substitute* (string-append dir "/fixedVariable.cpp")
                    (("^( +std::string sExe = ).*" _ prefix)
--
2.35.1

Message #11 received at 55067 <at> debbugs.gnu.org (full text, mbox):

From: Maxime Devos <maximedevos <at> telenet.be>
To: "Paul A. Patience" <paul <at> apatience.com>, 55067 <at> debbugs.gnu.org
Subject: Re: [bug#55067] [PATCH] gnu: nomad-optimizer: Disable failing test.
Date: Fri, 22 Apr 2022 19:25:20 +0200

[Message part 1 (text/plain, inline)]

Paul A. Patience schreef op vr 22-04-2022 om 16:20 [+0000]:
> ---
> It turns out the PSDMads test fails randomly.

Sounds like a bug, preferably it would 100% reliably report the
underlying problem, whatever it is.

> I've disabled it.

I don't think this is a good reason to disable the test.
Disabling tests whenever they fail defeats the point of tests.

Cf. with, say, clinical trials for testing safety of new medical
treatments.  If some drug fails one of the tests (sometimes or always),
then one cannot simply ignore the test.

Greetings,
Maxime.

[signature.asc (application/pgp-signature, inline)]

Message #14 received at 55067 <at> debbugs.gnu.org (full text, mbox):

From: "Paul A. Patience" <paul <at> apatience.com>
To: Maxime Devos <maximedevos <at> telenet.be>
Cc: 55067 <at> debbugs.gnu.org
Subject: Re: [bug#55067] [PATCH] gnu: nomad-optimizer: Disable failing test.
Date: Fri, 22 Apr 2022 19:02:34 +0000

On 2022-04-22 13:25:20-04:00, Maxime Devos wrote:
>> It turns out the PSDMads test fails randomly.
>
> Sounds like a bug, preferably it would 100% reliably report the
> underlying problem, whatever it is.

I agree with you, and I have already reported the issue to the
developers (along with some other issues that I addressed in the package
definition).

>> I've disabled it.
>
> I don't think this is a good reason to disable the test.
> Disabling tests whenever they fail defeats the point of tests.

This test failure wasn't getting detected in the past due to a missing
"set -o pipefail" in the test runner, which could explain why it was
missed by the developers.
I don't know enough to fix the bug myself, but I have a suspicion it is
a bug in the test rather than in the library.

Best regards,
Paul

Message #17 received at 55067 <at> debbugs.gnu.org (full text, mbox):

From: Maxime Devos <maximedevos <at> telenet.be>
To: "Paul A. Patience" <paul <at> apatience.com>
Cc: 55067 <at> debbugs.gnu.org
Subject: Re: [bug#55067] [PATCH] gnu: nomad-optimizer: Disable failing test.
Date: Fri, 22 Apr 2022 23:37:21 +0200

[Message part 1 (text/plain, inline)]

Paul A. Patience schreef op vr 22-04-2022 om 19:02 [+0000]:
> On 2022-04-22 13:25:20-04:00, Maxime Devos wrote:
> > > It turns out the PSDMads test fails randomly.
> > 
> > Sounds like a bug, preferably it would 100% reliably report the
> > underlying problem, whatever it is.
> 
> I agree with you, and I have already reported the issue to the
> developers (along with some other issues that I addressed in the package
> definition).

Then I recommend adding a link to the upstream report (in a comment),
such that at the next update we know where to check if it has been
resolved, and to avoid giving the impression that just skipping tests
is acceptable.

Greetings,
Maxime.

[signature.asc (application/pgp-signature, inline)]

Message #20 received at 55067 <at> debbugs.gnu.org (full text, mbox):

From: "Paul A. Patience" <paul <at> apatience.com>
To: 55067 <at> debbugs.gnu.org
Cc: "Paul A. Patience" <paul <at> apatience.com>
Subject: [PATCH v3] gnu: nomad-optimizer: Disable failing test.
Date: Mon, 25 Apr 2022 19:10:10 +0000

* gnu/packages/maths.scm (nomad-optimizer)[arguments]: Disable the
examples/advanced/library/PSDMads test.
---
I've added a link to the issue I opened upstream regarding this failing test,
as suggested by Maxime Devos.

 gnu/packages/maths.scm | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/gnu/packages/maths.scm b/gnu/packages/maths.scm
index f4778179d6..448cdb4d73 100644
--- a/gnu/packages/maths.scm
+++ b/gnu/packages/maths.scm
@@ -45,7 +45,7 @@
 ;;; Copyright © 2021 Gerd Heber <gerd.heber <at> gmail.com>
 ;;; Copyright © 2021 Franck Pérignon <franck.perignon <at> univ-grenoble-alpes.fr>
 ;;; Copyright © 2021 Philip McGrath <philip <at> philipmcgrath.com>
-;;; Copyright © 2021 Paul A. Patience <paul <at> apatience.com>
+;;; Copyright © 2021-2022 Paul A. Patience <paul <at> apatience.com>
 ;;; Copyright © 2021 Ivan Gankevich <i.gankevich <at> spbu.ru>
 ;;; Copyright © 2021 Jean-Baptiste Volatier <jbv <at> pm.me>
 ;;; Copyright © 2021 Guillaume Le Vaillant <glv <at> posteo.net>
@@ -2008,6 +2008,10 @@ (define-public nomad-optimizer
          (add-after 'fix-sources-for-build 'fix-sources-for-tests
            (lambda _
              (substitute* "examples/CMakeLists.txt"
+               ;; This test passes only sometimes.
+               ;; See https://github.com/bbopt/nomad/issues/72.
+               (("^ +add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/advanced/library/PSDMads\\)\n")
+                "")
                ;; examples/basic/batch/example3 is accidentally omitted.
                (("^(add_subdirectory\\(\\$\\{CMAKE_CURRENT_SOURCE_DIR\\}/basic/batch/example)2(\\)\n)"
                  _ prefix suffix)
@@ -2052,10 +2056,6 @@ (define-public nomad-optimizer
               "examples/advanced/library/exampleSuggestAndObserve/cache0.txt")

              (let ((builddir (string-append (getcwd) "/../build")))
-               ;; For some reason, omitting this fix causes the
-               ;; examples/advanced/library/PSDMads test to fail, even though
-               ;; it doesn't seem to reference any part of the FixedVariable
-               ;; test.
                (let ((dir "examples/advanced/library/FixedVariable"))
                  (substitute* (string-append dir "/fixedVariable.cpp")
                    (("^( +std::string sExe = ).*" _ prefix)
--
2.35.1

Message #25 received at 55067-done <at> debbugs.gnu.org (full text, mbox):

From: Mathieu Othacehe <othacehe <at> gnu.org>
To: "Paul A. Patience" <paul <at> apatience.com>
Cc: 55067-done <at> debbugs.gnu.org
Subject: Re: bug#55067: [PATCH] gnu: nomad-optimizer: Disable failing test.
Date: Tue, 26 Apr 2022 10:45:31 +0200

> * gnu/packages/maths.scm (nomad-optimizer)[arguments]: Disable the
> examples/advanced/library/PSDMads test.

Pushed, thanks Paul & Maxime!

Mathieu

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.