Marius Bakke writes: > Mark H Weaver writes: > >> Hi, >> >> Marius Bakke writes: >> >>> Chris Marusich writes: >>> >>>> Hi, >>>> >>>> I've been encountering this failure off and on for a few weeks now, and >>>> I'd like to help fix it. In short, it seems like non-deterministic test >>>> failures, to me. I think we should gather data and report the issue >>>> upstream, and maybe disable the offending tests in the meantime. >>> >>> I agree. I notice many of these failing tests are for the TokuDB >>> backend, which I doubt anyone is using in Guix anyway. >>> >>> Here is a patch that disables all tests mentioned in this report. I >>> would like to push it to core-updates. Are there others? >> >> I'm concerned by how frequently and casually we simply disable failing >> tests. What is the utility of running test suites at all, if this is >> how we respond? > > I had no idea this issue was so widespread until I noticed Berlins > builders hit it more often than not. I have not been able to reproduce > these failures on my machines. So it was kind of a panic reaction, > being the person responsible for running these tests and all. > > Looking further into the changes between 10.1.37 and 10.1.38, I notice > the 'tokudb.*' tests were enabled: > > https://github.com/MariaDB/server/commit/4c490d6df63695dc97b2c808e59954e6877d3a51 > > Watching the build on Berlin in real time, I also see that the test > output grind nearly to a halt while running those. > 'tokudb.hotindex-insert-2' took 2700439 milliseconds, or 45 minutes, if > I'm reading the test output correctly. > > The default test case timeout is 40 minutes (as specified in the Guix > package), but I'm using 80 for this build (60 was insufficient). > > I suspect the problem is that the 'tokudb.*' tests put a lot of strain > on the file system, which causes these other tests to fail. It's > interesting that disabling parallel build was insufficient though. Update: Berlin built mariadb twice on core-updates with this patch: