Re: Turning off tests leads to a different store item

[Simon Tournier]

Hi Maxim,

On Thu, 09 Nov 2023 at 10:04, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

>> I agree.  On a side note, one of the issue is the time of some tests.
>> Sometimes, packaging is frustrating: build takes ages, then you fix some
>> tests, think it will be good, re-launch “guix build”, another test
>> failing, repeat.  It could nice to be able to cache the result of the
>> build phase.
>
> I usually end up working in the /tmp/guix-build-* failed build of a
> package for these situations; then I don't need to rebuild the whole
> thing for each test suite run, and can test changes directly without
> proper patches while working toward a fix.

Do you know all the command lines equivalent for each phase run by all
the build systems? ;-)

Well, this workflow seems appealing but it never works for me on
concrete situations.  Most of the time, my issue does not come from
“what to do from command line” but “how to do it inside the Guix
recipe”.

For example, this test suite adjustment:

          (add-before 'install 'disable-network-tests
            (lambda _
              (substitute* "test/runtests.jl"
                (("\"async.jl") "# \"async.jl")
                (("\"client.jl") "# \"client.jl"))
              (substitute* "test/aws4.jl"
                (("@testset.*HTTP.request with AWS authentication.*" all)
                 (string-append all "return\n")))
              (substitute* "test/insert_layers.jl"
                (("@testset.*Inserted final layer runs handler.*" all)
                 (string-append all "return\n")))
              (substitute* "test/multipart.jl"
                (("@testset \"Setting of Content-Type.*" all)
                 (string-append all "return\n"))
                (("@testset \"Deprecation of .*" all)
                 (string-append all "return\n")))
              (substitute* "test/websockets.jl"
                (("@testset.*External Host.*" all)
                 (string-append all "return\n")))
              (substitute* "test/messages.jl"
                (("@testset.*Read methods.*" all)
                 (string-append all "return\n"))
                (("@testset.*Body - .*" all)
                 (string-append all "return\n"))
                (("@testset.*Write to file.*" all)
                 (string-append all "return\n")))
              (substitute* "test/cookies.jl"
                (("@testset.*Set-Cookie casing.*" all)
                 (string-append all "return\n"))))))))

To have the correct replacement rules, I almost never get it right at
the very first try, and the feedback loop is poor.  In some case, the
replacement is done after 'unpack, so I can just kill the build process
and check inside /tmp/guix-build-* if the output is the expected one.

However, depending on the test suite, sometimes, it errors at the first
failing test and does not report all the failing cases.  For example, in
the previous snippet, I could have to build then check, fail and repeat
something like 11 times, adding some try-error attempts for the correct
replacement rule.

Cheers,
simon