Subject: Tracking down inconsistent failure in jenkins


Hi Erick, *

On 2019/11/09 15:15:03, Erick Erickson <[EMAIL PROTECTED]> wrote:
> How are you running the tests? Just “ant test”? If so the output is all written to stdout so I usually just redirect it somewhere , e.g. “ant test > results 2>&1”….

Yea, I run it with ant test. So the output is the log from the cmd, ok. Is there an other possibility to run the tests?
>
> That file should have the failures _and_ a “reproduce with” line for each failing test. One thing I didn’t mention is that there’s also a significant bit of randomization in the test harness. Different locales are chosen at random, different directory implementations, different timezones, etc… We had one issue that was a JVM issue that only showed up in the Turkish locale for instance that we’d never have found without the randomization. The “reproduce with” line has all that information echoed and will run the test again with all the same bits of randomization.

Ok, this sounds interesting

>
> It’s relatively rare for the test to fail reliably even if you use the “reproduce with” line because it’s, well, reproducible. When it is, you’ll see a JIRA raised something like “reproducible test failure” and/or someone will jump on it and fix it.

So just reproducible issues go into jira? In my experience, it makes sense in some case, to write an issue für a ireproducible bug. So you can collect all data on one place. Sometimes this helps to track the bugs down.
>
> Timing issues: Well, just that. Say a test creates a collection and _assumes_ (no, this isn’t a good practice) that it’ll finish in 5 seconds and it takes 6, then drives on. Oops. Other more subtle issues are just threading issues where some sequence of context switching happens to hit an unanticipated problem. etc.
>
> It’s not that we _never_ get reproducible tests, it’s that when we do someone fixes them. There are a _lot_ of tests in the full suite, so if timing-related tests fail 0.1% of the time…
>
> You can confirm this yourself pretty easily, just save the output and run the “reproduce with” line.
>

Ok, I will go now into the source, and see, what some test does and trying to get a test without errors. Last time I have had 1 error. If it continues like this, I will have 0 next time ;-)

Regards, Raphael
> Best,
> Erick
>
---------------------------------------------------------------------