Re: gnupload improvements

[Sergey Poznyakoff]

Ralf Wildenhues <address@hidden> ha escrit:

> >   --symlink a b c d
> > 
> > will create two symbolic links: a -> b and c -> d.
> 
> This API looks rather unusual.

Well, I make it this way for two reasons: first, it is consistent with
another options (--rmsymlink, --delete) and secondly it allows to spare
some typing when several links need be created.  However, if you
think it is too error-prone, I can easily switch it back to a two
argument form. 

> Also, why is --rmsymlink not --delete?  (This is me wondering about the
> file format, not a question about your patch.)

They are two different commands. The --delete option causes removal of
a regular file (actually, archivation of it, but the result is the
same), whereas --rmsymlink removes a symbolic link.  

> Can we make that --symlink-regex, to avoid abbreviating?

OK

> > gnupload --to alpha.gnu.org:tar \
> >          --delete tar-1.20.90.tar.gz tar-1.20.90.tar.bz2 \
> >          --rmsymlink tar-latest.tar.gz tar-latest.tar.gz2 \
> >          -- tar-1.20.91.tar.gz
> > 
> > (double-dash in this case is needed to separate files to upload
> > from --rmsymlink arguments).         
> 
> Do we want to make guarantees about the ordering in which actions are
> taken?

The directives are created in the order of their appearance in the
command line.  The description of the upload directive file in
maintain.info says that "if you use more than one directive, the
directives are executed in the sequence they are specified in." So, it
is guaranteed, that the actions are executed in the order they are listed
in the command line.

> Please, no upper-case variables.  This isn't Fortran, and most reserved
> variables are upper-case.

OK, fixed:)

> > +  echo "$0: Reading configuration file .gnupload"
> > +  eval set -- "`sed 's/#.*$//;/^$/d' .gnupload | tr '\n' ' '` $*"
> 
> A couple of portability nits:
> - 'set x ...; shift'

Hmm, not quite sure where it can be used in the above statement?

> - \"address@hidden" instead of $*

$@ maybe, but \"address@hidden" will coalesce the actual command line arguments
into one, so that

  ./gnupload foo.tar.gz bar.tar.gz

will see one argument: "foo.tar.gz bar.tar.gz".

As far as I can tell, directive file format does not allow file names
containing white space, so the distinction between $@ and $*
is probably of little concern here.

> - not all tr implementations grok \n, and what about \r?
>   I'd probably use
>     tr '\012\015' '  '
>   as non-ASCII systems can safely be ignored.

OK

> There is no need to nest case statements here, no?

Yes, there is.  The outer case is needed to handle collect_var variable,
and the inner one to handle options.  It can be rewritten as one case
statement, but then each branch would begin with an assignment to
collect_var, e.g.: 

  case $1 in
    --to)
      collect_var=
      ...
      ;;
    --user)
      collect_var=
      ...
      ;;
    --delete)
      collect_var=delete_files
      shift
      ;;
    *) # process collect_var here
      
I am not sure which way is better.                                

> This isn't portable.  OTOH, allowing --OPT=ARG would be nice for
> other arguments like --to and --user, too.  Why not go with the
> Autoconf code?

Yes, probably.  The autoconf approach is nice, but it acts on the assumption
that an option takes only one argument.  In case of gnupload, however,
it is not always so.  Nevertheless, I'll try to figure out how combine
both approaches.

> If you agree with my comments, and answer the implied questions, I can
> make those changes; if you want to resubmit the patch, even better.  :-)

Sure, find attached a modified version :^) 

> I do with we had some testsuite exposure for gnupload though, this has
> grown so complicated that bugs are likely.  Is there test upload space
> on some of these hosts?

I plan to implement this on gnu.org.ua. I'll let you know when it is
operational.

Regards,
Sergey

diff --git a/build-aux/gnupload b/build-aux/gnupload
index 5d4289f..f75732f 100755
--- a/build-aux/gnupload
+++ b/build-aux/gnupload
@@ -1,9 +1,9 @@
 #!/bin/sh
 # Sign files and upload them.
 
-scriptversion=2009-01-30.00
+scriptversion=2009-02-17.10
 
-# Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009 Free Software Foundation
+# Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009  Free Software Foundation
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -24,100 +24,193 @@ set -e
 
 GPG='gpg --batch --no-tty'
 to=
-delete=false
+dry_run=
+symlink_files=
+delete_files=
+delete_symlinks=
+collect_var=
+dbg=
 
-usage="Usage: $0 [OPTIONS]... FILES...
+usage="Usage: $0 [OPTIONS]... [COMMAND] FILES... [[COMMAND] FILES...]
 
-Sign all FILES, and upload them to (or delete them from) selected
-destinations, according to
+Sign all FILES, and upload them to selected destinations, according to
 <http://www.gnu.org/prep/maintain/html_node/Automated-FTP-Uploads.html>.
 
+Commands:
+  --delete                 delete FILES from destination
+  --symlink                create symbolic links
+  --rmsymlink              remove symbolic links
+  --                       treat the remaining arguments as files to upload
+
 Options:
   --help                   print this help text and exit
   --to DEST                specify one destination for FILES
                            (multiple --to options are allowed)
   --user NAME              sign with key NAME
-  --delete                 delete FILES from destination instead of uploading
+  --symlink-regex[=SED-EXPR]  use SED-EXPR to create symbolic links
+  --dry-run                do nothing, show what would have been done
   --version                output version information and exit
 
+If --symlink-re is given without SED-EXPR, then the link target name
+is created by replacing the version information with \`-latest', e.g.:
+
+  foo-1.3.4.tar.gz -> foo-latest.tar.gz
+
 Recognized destinations are:
   alpha.gnu.org:DIRECTORY
   savannah.gnu.org:DIRECTORY
   savannah.nongnu.org:DIRECTORY
   ftp.gnu.org:DIRECTORY
                            build directive files and upload files by FTP
+  download.gnu.org.ua:{alpha|ftp}/DIRECTORY
+                           build directive files and upload files by SFTP
   address@hidden:DIRECTORY    upload files with scp
 
-Deletion only works for ftp.gnu.org and alpha.gnu.org (using the
-archive: directive).  Otherwise it is a no-op.  Deleting a file foo also
-deletes foo.sig; do not specify the .sig explicitly.
+If the file .gnupload exists in the current working directory, its contents
+are prepended to the actual command line options.  Use this to keep your
+defaults.  Comments (#) and empty lines in .gnupload are allowed.
 
-Simple single-target single-file examples:
-  gnupload --to alpha.gnu.org:automake automake-1.8.2b.tar.gz
-  gnupload --to ftp.gnu.org:automake automake-1.8.3.tar.gz
-  gnupload --to alpha.gnu.org:automake --delete automake-oops.tar.gz
+Examples:
+1. Upload automake-1.8.2b.tar.gz and automake-1.8.2b.tar.bz2 to two sites:
+  gnupload --to sources.redhat.com:~ftp/pub/automake \\
+           --to alpha.gnu.org:automake \\
+           automake-1.8.2b.tar.gz automake-1.8.2b.tar.bz2
 
-Multiple-target multiple-file example:
+2. Same as above, but also create symbolic links to automake-latest.tar.*:
   gnupload --to sources.redhat.com:~ftp/pub/automake \\
            --to alpha.gnu.org:automake \\
+          --symlink-regex \\
            automake-1.8.2b.tar.gz automake-1.8.2b.tar.bz2
 
-You can get the latest version of this script from savannah:
-<http://git.savannah.gnu.org/cgit/automake.git/plain/lib/gnupload>
+3. Symlink automake-1.8.2b.tar.gz to automake-latest.tar.gz and
+automake-1.8.2b.tar.bz2 to automake-latest.tar.bz2 on both sites:
+
+  gnupload --to sources.redhat.com:~ftp/pub/automake \\
+           --to alpha.gnu.org:automake \\
+          --symlink automake-1.8.2b.tar.gz automake-latest.tar.gz \\
+                    automake-1.8.2b.tar.bz2 automake-latest.tar.bz2
+
+4. Delete automake-1.8.2a.tar.gz and .bz2, remove symlink
+automake-latest.tar.gz and upload automake-1.8.2b.tar.gz:
+
+  gnupload --to sources.redhat.com:~ftp/pub/automake \\
+           --to alpha.gnu.org:automake \\
+          --delete automake-1.8.2a.tar.gz automake-1.8.2a.tar.bz2 \\
+          --rmsymlink automake-latest.tar.gz \\
+          -- \\
+           automake-1.8.2b.tar.gz automake-1.8.2b.tar.bz2
 
 Report bugs to <address@hidden>.
 Send patches to <address@hidden>."
 
+# Read local configuration file
+if test -r .gnupload; then
+  echo "$0: Reading configuration file .gnupload"
+  eval set -- "`sed 's/#.*$//;/^$/d' .gnupload | tr '\012\015' ' '` $@"
+fi
+
 while test -n "$1"; do
   case $1 in
-    --delete)
-      delete=true
-      shift
-      ;;
-    --help)
-      echo "$usage"
-      exit $?
-      ;;
-    --to)
-      if test -z "$2"; then
-       echo "$0: Missing argument for --to" 1>&2
-        exit 1
-      else
-        to="$to $2"
-        shift 2
-      fi
+  -*) collect_var=
+      case $1 in
+        --help)
+          echo "$usage"
+          exit $?
+          ;;
+        --to)
+          if test -z "$2"; then
+            echo "$0: Missing argument for --to" 1>&2
+            exit 1
+          else
+            to="$to $2"
+            shift 2
+          fi
+          ;;
+        --user)
+          if test -z "$2"; then
+            echo "$0: Missing argument for --user" 1>&2
+            exit 1
+          else
+            GPG="$GPG --local-user $2"
+            shift 2
+          fi
+          ;;
+       --delete)
+          collect_var=delete_files
+          shift
+          ;;
+       --rmsymlink)
+         collect_var=delete_symlinks
+         shift
+         ;;
+        --symlink-regex=*)
+          symlink_expr=`expr "$1" : '[^=]*=\(.*\)'`
+          shift
+          ;;
+        --symlink-regex)
+          symlink_expr='s|-[0-9][0-9\.]*\(-[0-9][0-9]*\)\{0,1\}\.|-latest.|'
+          shift
+          ;;
+        --symlink)
+          collect_var=symlink_files
+          shift
+          ;;
+        --dry-run|-n)
+          dry_run=1
+          shift
+          ;;
+        --version)
+          echo "gnupload $scriptversion"
+          exit $?
+          ;;
+       --)
+         shift
+         break
+         ;;
+        -*)
+          echo "$0: Unknown option \`$1', try \`$0 --help'" 1>&2
+          exit 1
+          ;;
+      esac
       ;;
-    --user)
-      if test -z "$2"; then
-       echo "$0: Missing argument for --user" 1>&2
-        exit 1
+  *)  if test -z "$collect_var"; then
+        break
       else
-        GPG="$GPG --local-user $2"
-        shift 2
+        eval $collect_var=\"\$$collect_var \"$1\"\"
+       shift
       fi
       ;;
-    --version)
-      echo "gnupload $scriptversion"
-      exit $?
-      ;;
-    -*)
-      echo "$0: Unknown option \`$1', try \`$0 --help'" 1>&2
-      exit 1
-      ;;
-    *)
-      break
-      ;;
   esac
 done
 
-if test $# = 0; then
-  echo "$0: No file to upload or delete" 1>&2
+dprint()
+{
+  echo "Running $*..."
+}
+
+if test -n "$dry_run"; then
+  dbg=dprint
+fi
+
+if test -z "$to"; then
+  echo "$0: Missing destination sites" >&2
   exit 1
-else
-  :
 fi
 
-if $delete; then :; else
+if test -n "$symlink_files"; then
+  x=`echo "$symlink_files" | sed 's/[^ ]//g;s/  //g'`
+  if test -n "$x"; then
+    echo "$0: Odd number of symlink arguments" >&2
+    exit 1
+  fi
+fi
+
+if test $# = 0; then
+  if test -z "${symlink_files}${delete_files}${delete_symlinks}"; then
+    echo "$0: No file to upload" 1>&2
+    exit 1
+  fi
+else
   # Make sure all files exist.  We don't want to ask
   # for the passphrase if the script will fail.
   for file
@@ -125,8 +218,15 @@ if $delete; then :; else
     if test ! -f $file; then
       echo "$0: Cannot find \`$file'" 1>&2
       exit 1
-    else
-      :
+    elif test -n "$symlink_expr"; then
+      linkname=`echo $file | sed "$symlink_expr"`
+      if test -z "$linkname"; then
+        echo "$0: symlink expression produces empty results" >&2
+        exit 1
+      elif test $linkname = $file; then
+        echo "$0: symlink expression does not alter file name" >&2
+        exit 1
+      fi
     fi
   done
 fi
@@ -146,8 +246,7 @@ read -r passphrase
 stty echo
 echo
 
-# Nothing to sign if deleting.
-if $delete; then :; else
+if test $# -ne 0; then
   for file
   do
     echo "Signing $file..."
@@ -156,59 +255,143 @@ if $delete; then :; else
   done
 fi
 
+# mkdirective DESTDIR BASE FILE STMT
+# Arguments: See upload, below
+mkdirective() {
+  stmt="$4"
+  if test -n "$3"; then
+    stmt="
+filename: $3$stmt"
+  fi
+
+  cat >${2}.directive<<EOF
+version: 1.1
+directory: $1
+comment: gnupload v. $scriptversion$stmt
+EOF
+  if test -n "$dry_run"; then
+    echo "File ${2}.directive:"
+    cat ${2}.directive
+    echo "File ${2}.directive:" | sed 's/./-/g'
+  fi
+}
+
+mksymlink() {
+  while test $# -ne 0
+  do
+    echo "symlink: $1 $2"
+    shift
+    shift
+  done
+}
+
+# upload DEST DESTDIR BASE FILE STMT FILES
+# Arguments:
+#  DEST     Destination site;
+#  DESTDIR  Destination directory;
+#  BASE     Base name for the directive file;
+#  FILE     Name of the file to distribute (may be empty);
+#  STMT     Additional statements for the directive file;
+#  FILES    List of files to upload.
+upload() {
+  dest=$1
+  destdir=$2
+  base=$3
+  file=$4
+  stmt=$5
+  files=$6
+
+  rm -f $base.directive $base.directive.asc
+  case $dest in
+    alpha.gnu.org:*)
+      mkdirective "$destdir" "$base" "$file" "$stmt"
+      echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $base.directive
+      $dbg ncftpput ftp-upload.gnu.org /incoming/alpha $files 
$base.directive.asc
+      ;;
+    ftp.gnu.org:*)
+      mkdirective "$destdir" "$base" "$file" "$stmt"
+      echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $base.directive
+      $dbg ncftpput ftp-upload.gnu.org /incoming/ftp $files $base.directive.asc
+      ;;
+    savannah.gnu.org:*)
+      if test -z "$files"; then
+        echo "$0: warning: standalone directives not applicable for $dest" >&2
+      fi
+      $dbg ncftpput savannah.gnu.org /incoming/savannah/$destdir $files
+      ;;
+    savannah.nongnu.org:*)
+      if test -z "$files"; then
+        echo "$0: warning: standalone directives not applicable for $dest" >&2
+      fi
+      $dbg ncftpput savannah.nongnu.org /incoming/savannah/$destdir $files
+      ;;
+    download.gnu.org.ua:alpha/*|download.gnu.org.ua:ftp/*)
+      mkdirective "${destdir#*/}" "$base" "$file" "$stmt"
+      echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $base.directive
+      for f in $files $base.directive.asc
+      do
+        echo put $f
+      done | $dbg sftp -b - puszcza.gnu.org.ua:/incoming/${destdir%%/*}
+      ;;
+    /*)
+      mkdirective "$destdir" "$base" "$file" "$stmt"
+      echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $base.directive
+      $dbg cp $files $base.directive.asc ${dest%%:*}
+      ;;
+    *)
+      if test -z "$files"; then
+        echo "$0: warning: standalone directives not applicable for $dest" >&2
+      fi
+      $dbg scp $files $dest
+      ;;
+  esac
+  rm -f $base.directive $base.directive.asc
+}
+
+#####
+# Process any standalone directives
+stmt=
+if test -n "$symlink_files"; then
+  stmt="$stmt
+`mksymlink $symlink_files`"
+fi
+
+for file in $delete_files
+do
+  stmt="$stmt
+archive: $file"
+done
+
+for file in $delete_symlinks
+do
+  stmt="$stmt
+rmsymlink: $file"
+done
+
+if test -n "$stmt"; then
+  for dest in $to
+  do
+    destdir=`echo $dest | sed 's/[^:]*://'`
+    upload "$dest" "$destdir" "`hostname`-$$" "" "$stmt"
+  done
+fi
+
+# Process actual uploads
 for dest in $to
 do
   for file
   do
-    # Prepare arguments.
-    if $delete; then
-      echo "Removing $file from $dest..."
-      files=  # nothing to upload if deleting
-      directive="archive: $file"
-    else
-      echo "Uploading $file to $dest..."
-      files="$file $file.sig"
-      directive="filename: "`basename -- "$file"`
-    fi
+    echo "Uploading $file to $dest..."
+    stmt=
+    files="$file $file.sig"
     destdir=`echo $dest | sed 's/[^:]*://'`
-
-    case $dest in
-      alpha.gnu.org:*)
-       rm -f $file.directive $file.directive.asc
-       cat >$file.directive<<EOF
-version: 1.1
-directory: $destdir
-$directive
-EOF
-       echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $file.directive
-        ncftpput ftp-upload.gnu.org /incoming/alpha $files $file.directive.asc
-       rm -f $file.directive $file.directive.asc
-       ;;
-      ftp.gnu.org:*)
-       rm -f $file.directive $file.directive.asc
-       cat >$file.directive<<EOF
-version: 1.1
-directory: $destdir
-$directive
-EOF
-       echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $file.directive
-        ncftpput ftp-upload.gnu.org /incoming/ftp $files $file.directive.asc
-       rm -f $file.directive $file.directive.asc
-       ;;
-      savannah.gnu.org:*)
-        # We only know how to implement delete for {ftp,alpha}.gnu.org.
-        $delete \
-        || ncftpput savannah.gnu.org /incoming/savannah/$destdir $files
-       ;;
-      savannah.nongnu.org:*)
-        $delete \
-        || ncftpput savannah.nongnu.org /incoming/savannah/$destdir $files
-       ;;
-      *)
-        $delete \
-       || scp $files $dest
-       ;;
-    esac
+    if test -n "$symlink_expr"; then
+      linkname=`echo $file | sed "$symlink_expr"`
+      stmt="$stmt
+symlink: $file $linkname
+symlink: $file.sig $linkname.sig"
+    fi
+    upload "$dest" "$destdir" "$file" "$file" "$stmt" "$files"
   done
 done