Questions regarding layout and composition of tests (bug#48598)

[J.P.]

Hi people,

I'm seeking feedback on (and perhaps tacit approval for) the following
files and their layout. AFAICT, aside from the whole

  test/lisp/foo-tests.el -> lisp/foo.el

correspondence for Make discovery, there doesn't seem to be much in the
way of rigid rules regarding naming, layout, or dependency inclusion.
(For anyone following this bug, none of those aspects has changed
dramatically in the past year or so with regard to these files).

  test/lisp/erc
  ├── erc-d
  │   ├── erc-d.el
  │   ├── erc-d-i.el
  │   ├── erc-d-self.el
  │   ├── erc-d-self-resources
  │   │   ├── basic.eld
  │   │   ├── depleted.eld
  │   │   ├── drop-a.eld
  │   │   ├── drop-b.eld
  │   │   ├── dynamic-barnet.eld
  │   │   ├── dynamic.eld
  │   │   ├── dynamic-foonet.eld
  │   │   ├── dynamic-stub.eld
  │   │   ├── eof.eld
  │   │   ├── fuzzy.eld
  │   │   ├── incremental.eld
  │   │   ├── irc-parser-tests.eld
  │   │   ├── linger.eld
  │   │   ├── linger-multi-a.eld
  │   │   ├── linger-multi-b.eld
  │   │   ├── no-block.eld
  │   │   ├── no-match.eld
  │   │   ├── nonstandard.eld
  │   │   ├── no-pong.eld
  │   │   ├── proxy-barnet.eld
  │   │   ├── proxy-foonet.eld
  │   │   ├── proxy-solo.eld
  │   │   ├── proxy-subprocess.el
  │   │   ├── timeout.eld
  │   │   └── unexpected.eld
  │   ├── erc-d-t.el
  │   └── erc-d-u.el
  ├── erc-join-tests.el
  ├── erc-networks-tests.el
  ├── erc-scenarios-common.el
  ├── erc-scenarios.el
  ├── erc-scenarios-resources
  │   ├── base
  │   │   ├── association
  │   │   │   ├── bouncer-history
  │   │   │   │   ├── barnet.eld
  │   │   │   │   └── foonet.eld
  │   │   │   ├── multi-net
  │   │   │   │   ├── barnet.eld
  │   │   │   │   └── foonet.eld
  │   │   │   ├── nick-bump
  │   │   │   │   ├── renicked-again.eld
  │   │   │   │   ├── renicked.eld
  │   │   │   │   ├── renicked-foisted-again.eld
  │   │   │   │   └── renicked-foisted.eld
  │   │   │   ├── reconnect-playback
  │   │   │   │   ├── foonet-again.eld
  │   │   │   │   └── foonet.eld
  │   │   │   └── same-network
  │   │   │       ├── chester.eld
  │   │   │       ├── tester-again.eld
  │   │   │       └── tester.eld
  │   │   ├── auth-source
  │   │   │   ├── foonet.eld
  │   │   │   └── nopass.eld
  │   │   ├── channel-buffer-revival
  │   │   │   └── foonet.eld
  │   │   ├── flood
  │   │   │   └── soju.eld
  │   │   ├── gapless-connect
  │   │   │   ├── barnet.eld
  │   │   │   ├── foonet.eld
  │   │   │   └── pass-stub.eld
  │   │   ├── mask-target-routing
  │   │   │   └── foonet.eld
  │   │   ├── network-id
  │   │   │   ├── bouncer
  │   │   │   │   ├── barnet-again.eld
  │   │   │   │   ├── barnet-drop.eld
  │   │   │   │   ├── barnet.eld
  │   │   │   │   ├── foonet-again.eld
  │   │   │   │   ├── foonet-drop.eld
  │   │   │   │   ├── foonet.eld
  │   │   │   │   └── stub-again.eld
  │   │   │   └── same-network
  │   │   │       ├── chester.eld
  │   │   │       └── tester.eld
  │   │   ├── reconnect
  │   │   │   ├── aborted-dupe.eld
  │   │   │   ├── aborted.eld
  │   │   │   ├── options-again.eld
  │   │   │   ├── options.eld
  │   │   │   ├── timer.eld
  │   │   │   └── timer-last.eld
  │   │   ├── renick
  │   │   │   ├── queries
  │   │   │   │   ├── bouncer-barnet.eld
  │   │   │   │   ├── bouncer-foonet.eld
  │   │   │   │   └── solo.eld
  │   │   │   └── self
  │   │   │       ├── auto.eld
  │   │   │       ├── manual.eld
  │   │   │       ├── qual-chester.eld
  │   │   │       └── qual-tester.eld
  │   │   └── reuse-buffers
  │   │       ├── channel-buffers
  │   │       │   ├── barnet.eld
  │   │       │   └── foonet.eld
  │   │       └── server-buffers
  │   │           ├── barnet.eld
  │   │           └── foonet.eld
  │   ├── join
  │   │   ├── legacy
  │   │   │   └── foonet.eld
  │   │   ├── network-id
  │   │   │   ├── barnet.eld
  │   │   │   ├── foonet-again.eld
  │   │   │   └── foonet.eld
  │   │   └── reconnect
  │   │       ├── foonet-again.eld
  │   │       └── foonet.eld
  │   ├── networks
  │   │   └── announced-missing
  │   │       └── foonet.eld
  │   └── services
  │       ├── auth-source
  │       │   └── libera.eld
  │       └── password
  │           └── libera.eld
  ├── erc-services-tests.el
  └── erc-tests.el
  
The files above introduce around a hundred nontrivial tests that
lengthen the overall running time for the entire Emacs suite by roughly
five minutes on EMBA [1]. Bug#49860 will likely stretch this by another
two. In spite of the cost, their utility extends to indirectly testing
the basic behavioral contracts of other libraries as well. The only
episode in the past year where these encountered any difficulty was in
the immediate wake of

  commit 0c7a7433dce1b93a685396986d3a560c9cc291f1
  Author: Miha Rihtaršič <miha@kamnitnik.top>
  Date:   Tue May 25 21:01:58 2021 +0200

  Try to not prioritise reading from lower file descriptors

  * src/process.c (wait_reading_process_output): When looping through
  fds, continue from where we left off.
  (syms_of_process): Vprocess_prioritize_lower_fds: New variable
  (bug#48118).

But that was a good thing because the only tests affected were those
intentionally demonstrating undesirable behavior.

A few of the more relevant patches are attached below, but the full set
is also available [2]. General comments and random spot checks are, of
course, always welcome.

Thanks,
J.P.


[1] https://emba.gnu.org/emacs/emacs/-/jobs/42826

    FWIW, it's more like two minutes on normal PCs and commercial cloud
    infra, like Gitlab.com's GCP runners.

[2] https://jpneverwas.gitlab.io/erc-tools/48598/patches.tar.gz

0016-Add-ERC-test-server-and-related-resources.patch
Description: Text Data

0018-SQUASH-ME-Add-user-oriented-test-scenarios-for-ERC.patch
Description: Text Data

0021-SQUASH-ME-Add-ERC-scenarios-for-identity-aware-msg-h.patch
Description: Text Data

0023-SQUASH-ME-Add-ERC-test-scenarios-for-identity-aware-.patch
Description: Text Data

0025-SQUASH-ME-Add-ERC-test-scenarios-involving-auth-sour.patch
Description: Text Data

0026-SQUASH-ME-Add-ERC-test-scenario-for-erc-cmd-JOIN.patch
Description: Text Data