[Savannah-cvs] [615] update: mailing lists are configured from frontend directly

[ineiev]

Revision: 615
          
http://svn.savannah.gnu.org/viewvc/?view=rev&root=administration&revision=615
Author:   ineiev
Date:     2023-05-15 13:15:46 -0400 (Mon, 15 May 2023)
Log Message:
-----------
update: mailing lists are configured from frontend directly

Modified Paths:
--------------
    trunk/sviki/MailingListsInternals.mdwn
    trunk/sviki/SavannahUpdateDelays.mdwn

Modified: trunk/sviki/MailingListsInternals.mdwn
===================================================================
--- trunk/sviki/MailingListsInternals.mdwn      2023-05-13 07:11:49 UTC (rev 
614)
+++ trunk/sviki/MailingListsInternals.mdwn      2023-05-15 17:15:46 UTC (rev 
615)
@@ -1,5 +1,4 @@
-Mailing Lists Management in Savannah
-====================================
+# Mailing Lists Management in Savannah
 
 All gnu and nongnu mailing lists are processed on `lists.gnu.org`
 using the Mailman package. This machine is managed by FSF admins.
@@ -14,9 +13,8 @@
 * [[ListServer]] - manually running mailing-list related commands.
 
 
+## Overview
 
-Overview
---------
 
 Savannah provides mailing lists to hosted groups.
 Group admins can create as many mailing lists as needed through
@@ -26,22 +24,15 @@
 
 1. A group admin on Savannah creates a new mailing list on Savannah
    website.
-2. The PHP frontend code updates the mailing list information in the
-   `mail_group_list` table in the `savane` MySQL database.
-3. A cronjob on 'mgt0` reads the `mail_group_list` table and determines
-   which operations are needed (e.g. list creation, password reset).
-4. The cronjob runs local scripts (on mgt0) for each operation.
-5. Each script proxies the commands over SSH to `lists.gnu.org`
-6. On `lists.gnu.org`, the script `scannah_wrapper.pl` processes the commands
-   and runs the appropriate Mailman scripts locally on `lists.gnu.org`.
-7. The cronjob script (back on `mgt0`) sends email notification to the
-   group admins (e.g. notification about updated passwords).
+2. The PHP frontend code runs `savannah_wrapper.pl` on list@lists.gnu.org
+   via ssh; `savannah_wrapper.pl` runs appropriate Mailman scripts.
+3. The PHP frontend code updates the mailing list information in the
+   `mail_group_list` table in the `savane` MySQL database and sends
+   a notification with new mailing list password when needed.
 
 
+## Database
 
-Database
---------
-
 The `mail_group_list` table contains the following fields:
 
     mysql> select * from mail_group_list where list_name = 'sed-devel' \G
@@ -70,254 +61,80 @@
 
 The meaning of the `status` field:
 
-    Status 0: list is deleted (ie, does not exist).
-    Status 1: list is marked for creation.
-    Status 2: list is marked for reconfiguration.
-    Status 5: list has been created (ie, it exists).
+    Status 0: obsolete.
+    Status 1: obsolete.
+    Status 2: obsolete.
+    Status 5: the only one used, so actually means nothing.
 
-The `password` field means:
+The `password` field means: nothing.
 
-    NULL: default value
-    '1':  A password reset was requested for this mailing list.
+## savannah_wrapper.pl
 
-The password field should not contain any real passwords,
-though the database does contain values besides NULL and '1'
-(TODO: find out how/why):
+This script processes requests from frontend and translates them
+in Mailman script invocations.  The requests are read from STDIN
+as a set of lines in form
 
-    mysql> select count(*) from mail_group_list where password is not NULL and 
password != '1';
-    +----------+
-    | count(*) |
-    +----------+
-    |     1674 |
-    +----------+
-    1 row in set (0.00 sec)
+    VAR=VALUE
 
+The response is written to STDOUT; normal output looks like
+a similar set of
 
+    VAR=VALUE
 
-PHP web frontend code
----------------------
+It includes the version of the script and the variables
+set in the process.  Errors, if any, are reported in lines
+starting with 'Error', typically without the '=' sign.
 
-Mailing list administration is implemented in
-[./frontend/php/mail/admin/index.php](https://git.savannah.gnu.org/cgit/administration/savane.git/tree/frontend/php/mail/admin/index.php).
+The COMMAND variable is required, it may have values:
 
-The PHP code updates *only* the `mail_group_list` table in the MySQL
-database, and queues future actions by changing the `status` and
-`password` fields in the table.
+    version: Report version and generated password and quit.
+    newlist: Create a list.  The parameters used are
+      VISIBILITY, LIST_FULL_NAME, DESCRIPTION, ADMIN_MAIL, PASSWORD
+      (all are required).
+    config: Make list public or private, change its description.
+      Used parameters are VISIBILITY and DESCRIPTION (both optional).
+    change_pw: Request password reset. The parameters used are
+      LIST_NAME and PASSWORD (both required).
+    rmlist: Remove the list. The parameter used is LIST_NAME (required).
 
-No other actions are directly performed by the PHP frontend code.
+The parameters used by commands are
 
-The PHP code contains these additional comments:
+    DRY_RUN: Don't apply changes; the only valid value is 'yes'.
+    ADMIN_MAIL: Email of the administrator.
+    LIST_NAME: List name, e.g. pretest-users.
+    LIST_FULL_NAME: List name with domain, e.g. pretest-users@lists.nongnu.org.
+    DESCRIPTION: A line describing the list
+    PASSWORD: New password; when empty (which is the normal usage),
+      the script generates a new password.
+    VISIBILITY: Whether the list is 'public' or 'private'.
 
-    This frontend PHP script sets status to:
-         0 if user deletes a list before the backend ever actually created it
-         1 if user adds a list
-         2 if user reconfigures an _existing_ list (ie, status was 5)
+## Manual walk-through example - changing description
 
-    The backend sv_mailman.pl script sets status to:
-         0 when a list is actually deleted
-         5 when a list is actually created
-
-    When we create an alias, which mean someone was able, according to
-    group type restriction, to add to the group a list that was already
-    inside the database, we add the list inside the database with a status
-    of 5, so sv_mailman does not try to recreate it.
-
-    In the worst case, if two persons create the same list at the same time,
-    the field password will not contain real password, it will contain
-    '1' when the backend is supposed to reset it.
-
-
-Backend cronjob
----------------
-
-The mailing list cronjob runs on `mgt0` (in the old setup
-before the 2017 migration to newer VM, the script was on the `oldinternal` -
-outdated wiki pages might still refer to it there).
-
-In `mgt0:/etc/cron.d/savane`:
-
-    PATH=...:/opt/savannah/bin:...
-    # New list creation
-    */5 * * * *     root    sv_mailman --cron
-
-The script is `mgt0:/opt/savannah/bin/sv_mailman`.
-
-The above script is the installed version, originating from
-`mgt0:/opt/savannah/savane/backend/mail/sv_mailman.in` (and also in
-git
-<https://git.savannah.gnu.org/cgit/administration/savane.git/tree/backend/mail/sv_mailman.in>).
-If you want to modify the script, NEVER update the installed version
-in `/opt/savannah/bin`. ALAWAYS push the updated version to
-the Savannah repository, update the source version in
-`/opt/savannah/savane/backend/mail/sv_mailman.in`, then run `make
-install`.
-
-The script reads the `mail_group_list` MySQL table
-and performs the required actions based on the `status` and `password`
-fields.
-
-The script runs the following local scripts (on `mgt0`):
-
-- `/usr/sbin/newlist` - creating new mailing list
-- `/usr/sbin/config_list` - configuring a list (currently a no-op)
-- `/usr/sbin/rmlist` - removing a list
-- `/usr/lib/mailman/bin/change_pw` - resetting a mailing list password.
-
-Sadly, the paths are hard-coded in `sv_mailman` (there's even a comment
-about how bad they are).
-
-On `mgt0`, symlinks were created to the actual scripts:
-
-    agn@mgt0:~$ ls -lhog /usr/sbin/{newlist,config_list,rmlist} 
/usr/lib/mailman/bin/change_pw
-    lrwxrwxrwx 1 28 Apr 17 22:44 /usr/lib/mailman/bin/change_pw -> 
/opt/savannah/sbin/change_pw
-    lrwxrwxrwx 1 30 Apr 17 22:43 /usr/sbin/config_list -> 
/opt/savannah/sbin/config_list
-    lrwxrwxrwx 1 26 Apr 17 22:43 /usr/sbin/newlist -> 
/opt/savannah/sbin/newlist
-    lrwxrwxrwx 1 25 Apr 17 22:43 /usr/sbin/rmlist -> /opt/savannah/sbin/rmlist
-
-FIXME: remove the hard-coded PATHs from `sv_mailman`, and install an
-updated version.
-
-NOTE: These scripts are proxies to `lists.gnu.org` - they do not
-perform any local action on `mgt0`. Their names are likely left over
-from the time the entire Savannah system (including Mailman)
-was running on the same local machine.
-
-SSH Proxing to lists.gnu.org
-----------------------------
-
-The `newlist`/`rmlist`/`change_pw` scripts on `mgt0`
-proxy the needed commands to `lists.gnu.org`,
-where the Mailman package is installed.
-
-1. On `mgt0`, the script `sv_mailman` runs as root from cron-jobs.
-2. The `sv_mailman` runs one of the action scripts (e.g. `newlist`).
-   the script also generates random passwords if needed.
-3. The `newlist` script (on `mgt0`) connets with SSH to `list@lists.gnu.org`
-   (this happens as user `root@mgt0`).
-4. The corresponding pubkey in `lists.gnu.org:/home/list/.ssh/authorized_keys`
-   is defined as follows:
-
-        command="./savannah_wrapper.pl" ssh-rsa AAAAB3NzaC1yc2...
-
-5. When the `newlist` script from `mgt0` connects with SSH to `lists.gnu.org`
-   instead of getting a shell, the 
`lists.gnu.org:/home/list/savannah_wrapper.pl`
-   script is executed.
-6. This script parses limited input from STDIN, and if it contains
-   recognized commands and parameters, it runs the corresponding program
-   locally on `lists.gnu.org`.
-7. `sv_mainman` sends email notification to the administrators of the mailing
-   lists, with the newly generated random passwords.
-   (FIXME: emails are sent even if commands fail - there's no error
-   checking).
-
-Example: running the following on `mgt0` as user `root` will result in
-changing the mailing-list password on `lists.gnu.org`:
-
-    # ssh -T list@lists.gnu.org <<EOF
-    COMMAND=change_pw
-    LIST_NAME=bug-datamash
-    PASSWORD=123456
-    EOF
-
-The corresponding code in `lists.gnu.org:/home/list/savannah_wrapper.pl`
-(lots of code omitted for brevity):
-
-    ...
-    my %vars;
-    while (<>) {
-        chomp;
-        my ($variable, $value) = split('=');
-        $vars{lc($variable)} = $value;
-    }
-    ...
-    if ($vars{'command'} eq 'change_pw') {
-       (system('change_pw',
-               '-l',  $vars{'list_name'},
-               '-p', $vars{'password'},
-               '--quiet')
-        == 0) or die "change_pw: $!";
-    }
-
-TODO: document the "virtual host" hack (e.g. allowing gnu/nongnu mailing 
lists).
-
-Logs
-----
-
-On `mgt0`, the log file is `mgt0:/var/log/sv_database2system.log`:
-
-    [sv_mailman] Sun Apr  9 01:50:01 2017 - starting
-    [sv_mailman] Sun Apr  9 01:50:02 2017 - List www-zh-cn-translators 
<Fossilet> config_list.
-    [sv_mailman] Sun Apr  9 01:50:02 2017 - List www-zh-cn-translators 
<Fossilet> reconfigured.
-    [sv_mailman] Sun Apr  9 01:50:02 2017 - List www-zh-cn-translators 
password was reset.
-    [sv_mailman] Sun Apr  9 01:50:02 2017 - Mail sent to foo@gnu.org, 
bar@gnu.org, baz@gnu.org.
-    [sv_mailman] Sun Apr  9 01:50:02 2017 - work finished
-
-On `lists.gnu.org`, the log file is 
`lists.gnu.org:/home/list/savannah_wrapper.log`.
-It contains all the passed parameters (and thus passwords in clear text):
-
-    Tue Apr 18 02:52:06 2017
-    password = 12345
-    list_name = pretest-users
-    command = change_pw
-
-Manual walk-through example - changing password
------------------------------------------------
-
-When group admins request password reset for their mailing list
-on Savannah website, the PHP frontend code sets the corresponding
-`password` field in the `mail_group_list` table to `1`.
-Equivalent command:
-
-    $ mysql savane
-    mysql> UPDATE mail_group_list SET password='1' where 
list_name='pretest-users' LIMIT 1;
-
-On `mgt0` the `sv_mailman` is run periodically from cron.
-Equivalent command:
-
-    $ ssh root@mgt0
-    # cd /opt/savannah/bin
-    # ./sv_mailman --cron
-    New pretest-users password: foobar12345
-
-The log file `mgt0:/var/log/sv_database2system.log` will show:
-
-    [sv_mailman] Mon 17 Apr 2017 10:52:05 PM EDT - starting
-    [sv_mailman] Mon 17 Apr 2017 10:52:06 PM EDT - List pretest-users password 
was reset.
-    [sv_mailman] Mon 17 Apr 2017 10:52:06 PM EDT - Mail sent to 
assafgordon@gmail.com for list pretest-users.
-    [sv_mailman] Mon 17 Apr 2017 10:52:06 PM EDT - work finished
-
-The `sv_mailman` generated a new random password and executed `change_pw`.
-Equivalent command (still on `mgt0`):
-
-    $ ssh root@mgt0
-    # cd /opt/savannah/sbin
-    # ./change_pw -n pretest-users -p 123456
-    New pretest-users password: 123456
-
-The `change_pw` script connects to `lists.gnu.org`
+The frontend code connects to `lists.gnu.org`
 and transmits the commands as STDIN.
-Equivalent command (still on `mgt0`):
+Equivalent command (still on `frontend`):
 
-    # ssh -T list@lists.gnu.org <<EOF
-    COMMAND=change_pw
-    LIST_NAME=pretest-users
-    PASSWORD=123456
+    # sudo -u www-data ssh -T list@lists.gnu.org <<EOF
+    COMMAND=config
+    LIST_FULL_NAME=pretest-users@lists.nongnu.org
+    DESCRIPTION=Test mailing list
     EOF
+    version=list@lists.gnu.org 0.1
+    command=config
+    list_full_name=pretest-users@lists.nongnu.org
+    description=Test mailing list
 
-
 The script `savannah_wrapper.pl` on `lists.gnu.org`
-reads the commands from STDIN and execute the needed
+reads the commands from STDIN and executes the needed
 programs.
+
 Equivalent command (on `lists.gnu.org`):
 
     $ ssh list@list.gnu.org
-    lists:~$ which change_pw
-    /home/list/mailman/bin/change_pw
-    lists:~$ change_pw -l pretest-users -p 123456 --quiet
-    New pretest-users password: 123456
+    lists:~$ echo description='Test mailing list' \
+      | config_list -i /dev/stdin pretest-users
 
-Mailing list creation - FOR REFERENCE ONLY
-------------------------------------------
+## Mailing list creation - FOR REFERENCE ONLY
 
 **The Following section describes the original setup. It is kept for
 historical purposes**

Modified: trunk/sviki/SavannahUpdateDelays.mdwn
===================================================================
--- trunk/sviki/SavannahUpdateDelays.mdwn       2023-05-13 07:11:49 UTC (rev 
614)
+++ trunk/sviki/SavannahUpdateDelays.mdwn       2023-05-15 17:15:46 UTC (rev 
615)
@@ -19,8 +19,8 @@
   email addresses, notification settings, GnuPG and SSH keys
   are in effect with no delay.
 - Group home pages update within a few minutes.
-- Mailing list-related requests (creation, deletion, admin password
-  reset) are passed to lists.gnu.org every 5 minutes.
+- Mailing list-related requests are processed on lists.gnu.org
+  immediately.
 
 Copyright &copy; 2023 Ineiev