Re: master 32e790f: Implement NTLM server for ntlm.el testing

[Michael Albinus]

fitzsim@fitzsim.org (Thomas Fitzsimmons) writes:

Hi Thomas,

> +Some optional tests require packages from GNU ELPA.  By default
> +../../elpa will be checked for these packages.  If GNU ELPA is checked
> +out somewhere else, use
> +
> +    make GNU_ELPA_DIRECTORY=/path/to/elpa ...

When I've tried it with an up-to-date checkout of emacs master, I've got

--8<---------------cut here---------------start------------->8---
# make -C test ntlm-tests
make: Entering directory '/usr/local/src/emacs/test'
make[1]: Entering directory '/usr/local/src/emacs/test'
  GEN      lisp/net/ntlm-tests.log
Running 3 tests (2021-02-19 10:07:26+0100, selector `(not (tag :unstable))')
Contacting host: localhost:8080
Test ntlm-authentication backtrace:
  signal(ert-test-failed (((should (equal (ntlm-tests--start-server-au
  ert-fail(((should (equal (ntlm-tests--start-server-authenticate-stop
  (if (unwind-protect (setq value-14 (apply fn-12 args-13)) (setq form
  (let (form-description-16) (if (unwind-protect (setq value-14 (apply
  (let ((value-14 'ert-form-evaluation-aborted-15)) (let (form-descrip
  (let* ((fn-12 #'equal) (args-13 (condition-case err (let ((signal-ho
  (closure (t) nil (let ((value-10 (gensym "ert-form-evaluation-aborte
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name ntlm-authentication :documentation "C
  ert-run-or-rerun-test(#s(ert--stats :selector (not (tag :unstable))
  ert-run-tests((not (tag :unstable)) #f(compiled-function (event-type
  ert-run-tests-batch((not (tag :unstable)))
  ert-run-tests-batch-and-exit((not (tag :unstable)))
  eval((ert-run-tests-batch-and-exit '(not (tag :unstable))) t)
  command-line-1(("-L" ":." "-L" "./../../elpa/packages/url-http-ntlm/
  command-line()
  normal-top-level()
Test ntlm-authentication condition:
    (ert-test-failed
     ((should
       (equal
        (ntlm-tests--start-server-authenticate-stop-server)
        ntlm-tests--successful-result))
      :form
      (equal "HTTP/1.1 500 Internal Server Error\12Content-type: 
text/plain\12\12un-support protocol: :NTLM" "HTTP/1.1 200 
OK\12\12Authenticated.\15\12")
      :value nil :explanation
      (arrays-of-different-length 87 33 "HTTP/1.1 500 Internal Server 
Error\12Content-type: text/plain\12\12un-support protocol: :NTLM" "HTTP/1.1 200 
OK\12\12Authenticated.\15\12" first-mismatch-at 9)))
   FAILED  1/3  ntlm-authentication (0.124960 sec)
Contacting host: localhost:8080
Test ntlm-authentication-old-compatibility-level backtrace:
  signal(ert-test-failed (((should (equal (ntlm-tests--start-server-au
  ert-fail(((should (equal (ntlm-tests--start-server-authenticate-stop
  (if (unwind-protect (setq value-21 (apply fn-19 args-20)) (setq form
  (let (form-description-23) (if (unwind-protect (setq value-21 (apply
  (let ((value-21 'ert-form-evaluation-aborted-22)) (let (form-descrip
  (let* ((fn-19 #'equal) (args-20 (condition-case err (let ((signal-ho
  (closure (t) nil (let ((value-17 (gensym "ert-form-evaluation-aborte
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name ntlm-authentication-old-compatibility
  ert-run-or-rerun-test(#s(ert--stats :selector (not ...) :tests [...
  ert-run-tests((not (tag :unstable)) #f(compiled-function (event-type
  ert-run-tests-batch((not (tag :unstable)))
  ert-run-tests-batch-and-exit((not (tag :unstable)))
  eval((ert-run-tests-batch-and-exit '(not (tag :unstable))) t)
  command-line-1(("-L" ":." "-L" "./../../elpa/packages/url-http-ntlm/
  command-line()
  normal-top-level()
Test ntlm-authentication-old-compatibility-level condition:
    (ert-test-failed
     ((should
       (equal
        (ntlm-tests--start-server-authenticate-stop-server)
        ntlm-tests--successful-result))
      :form
      (equal "HTTP/1.1 500 Internal Server Error\12Content-type: 
text/plain\12\12un-support protocol: :NTLM" "HTTP/1.1 200 
OK\12\12Authenticated.\15\12")
      :value nil :explanation
      (arrays-of-different-length 87 33 "HTTP/1.1 500 Internal Server 
Error\12Content-type: text/plain\12\12un-support protocol: :NTLM" "HTTP/1.1 200 
OK\12\12Authenticated.\15\12" first-mismatch-at 9)))
   FAILED  2/3  ntlm-authentication-old-compatibility-level (0.102966 sec)
   passed  3/3  ntlm-time-to-timestamp (0.000149 sec)

Ran 3 tests, 1 results as expected, 2 unexpected (2021-02-19 10:07:27+0100, 
0.485233 sec)

2 unexpected results:
   FAILED  ntlm-authentication
   FAILED  ntlm-authentication-old-compatibility-level
--8<---------------cut here---------------end--------------->8---

Not a big deal, after pulling and recompiling recent web-server ELPA
package it was fixed. However, it makes me nervous that a checkout of
emacs master can fail tests, w/o any reason inside the sources.

Could you pls provide more advanced checks for dependencies? For
example, instead of

 (and (featurep 'url-http-ntlm) (featurep 'web-server))

you could check a proper version of the packages?

Furthermore, when the ELPA packages are not present, we see in the make
test output:

--8<---------------cut here---------------start------------->8---
Warning (emacs): Cannot find one or more GNU ELPA packages
Warning (emacs): Skipping NTLM authentication tests
Warning (emacs): See GNU_ELPA_DIRECTORY in test/README
Running 3 tests (2021-02-19 10:26:00+0100, selector `(not (tag :unstable))')
  skipped  1/3  ntlm-authentication (0.000133 sec)
  skipped  2/3  ntlm-authentication-old-compatibility-level (0.000097 sec)
   passed  3/3  ntlm-time-to-timestamp (0.000136 sec)
--8<---------------cut here---------------end--------------->8---

Is it necessary to be such chatty? The tests are skipped (like other
tests), fine.

Best regards, Michael.