Upcoming API changes in ESPResSo

[Jean-Noël Grad]

Dear users,

We have decided to extend the lifetime and support of the ESPResSo 4.2 
line. A bugfix release is currently being developed [2]. More details 
are provided at the end of this message.

We would like to share recent developments in ESPResSo that will affect 
users working with the development version of ESPResSo (python branch). 
As you know, we recently replaced the old LB and EK implementations with 
new implementations based on waLBerla. We have also replaced the old LB 
swimmer code (ENGINE feature) with a new implementation based on virtual 
sites a few weeks ago. These changes made it a lot easier to develop new 
features. For example, LB now supports per-particle gamma and 
Lees-Edwards periodic boundary conditions.

The community has expressed a growing interest for ESPResSo to support 
instantiating multiple espressomd.System objects in the same Python 
script. We are currently working on it, but it is contingent on the 
removal of all static global variables in the C++ core, which is not a 
trivial task. We have to create a C++ System class in the core and move 
all global variables into it.

As part of that effort, we are currently replacing the legacy 
`system.actors` list by a much more expressive syntax:

```python
system.electrostatics.solver = espressomd.electrostatics.P3M(...)
system.electrostatics.extension = 
espressomd.electrostatic_extensions.ICC(...)
system.magnetostatics.solver = 
espressomd.magnetostatics.DipolarDirectSumCpu(...)
system.ekcontainer.solver = espressomd.electrokinetics.EKFFT(...)
system.ekcontainer.add(espressomd.electrokinetics.EKSpecies(...))
system.lb.set(espressomd.lb.LBFluidWalberla(...))

system.electrostatics.clear()
system.magnetostatics.clear()
system.ekcontainer.clear()
system.ekcontainer.solver = None
system.lb.clear()
```

The old list-based syntax was awkward, since it mixed together unrelated 
algorithms for hydrodynamics/electrostatics/magnetostatics, which had to 
be unmixed in the core using convoluted C++ code. It also forced these 
algorithms to share a common interface, making them unnecessarily 
difficult to extend. This major API change will be rolled out on the 
python branch over the next few weeks in small, manageable PRs, starting 
with electrostatics and magnetostatics [1]. The progress will be 
reported in the proceedings of the ESPResSo meetings [3].

The remaining global variables will be dealt with as we make progress 
rewriting the rest of the ESPResSo core. We have concrete plans to 
rewrite the cell structure and integrator code in the near future, so as 
to allow e.g. running different integration schemes on a per-particle 
basis. We therefore expect more breaking API changes to come.

While we hate API changes as much as you do (we use ESPResSo too!), 
these changes are necessary to make the project sustainable. Preserving 
the old API would force us to dedicate an unreasonable amount of 
resources that could be best invested in making ESPResSo capable of 
running multiple systems in the same script.

We understand if you're uneasy about using a continuously-evolving 
software in production. We want to give you the best experience possible 
while the python branch undergoes these transformations. This is why we 
decided to walk back on our decision to stop supporting the 4.2 line. We 
are now preparing a 4.2.2 bugfix release [2] that addresses compiler 
warnings from recent compilers, and also patches the LB inertialess 
tracers bug (the patch has already been backported to the official 4.2.1 
package in Fedora and openSUSE).

Best regards,
Jean-Noël Grad

Links:
[1] https://github.com/espressomd/espresso/pull/4749
[2] https://github.com/espressomd/espresso/pull/4751
[3] 
https://github.com/espressomd/espresso/wiki/Proceedings-2023-ESPResSo-meetings