[bug#50274] [PATCH] guix: git: Adds feature to download git repository to the store.

[Sarah Morgensen]

Hello,

Thanks for the patch!  I've been wanting a feature like this.  I haven't
tested your patch, but just reading it, I have a few comments...

jgart <jgart@dismail.de> writes:

> From: Julien Lepiller <julien@lepiller.eu>
>
> * guix/git.scm (download-git-to-store): Download Git repository from
> URL at COMMIT to STORE, either under NAME or URL's basename if omitted.
> Write progress reports to LOG.  RECURSIVE? has the same effect as the
> same-named parameter of 'git-fetch'.
>
> * guix/scripts/download.scm (download-git-to-store*): Adds cli option.
> Examples:
> guix download --git-commit=v0.1.1 github.com/anaseto/gruid-tcell
> guix download -c v0.1.1 https://github.com/anaseto/gruid-tcell

These examples should probably be in the main commit message, not the
ChangeLog entry.

> ---
>  guix/git.scm              | 24 +++++++++++++++++-
>  guix/scripts/download.scm | 51 ++++++++++++++++++++++++++++++++-------
>  2 files changed, 65 insertions(+), 10 deletions(-)
>
> diff --git a/guix/git.scm b/guix/git.scm
> index 9c6f326c36..4c70782b97 100644
> --- a/guix/git.scm
> +++ b/guix/git.scm
> @@ -28,6 +28,7 @@
>    #:use-module (gcrypt hash)
>    #:use-module ((guix build utils)
>                  #:select (mkdir-p delete-file-recursively))
> +  #:use-module ((guix build git) #:select (git-fetch))
>    #:use-module (guix store)
>    #:use-module (guix utils)
>    #:use-module (guix records)
> @@ -43,6 +44,7 @@
>    #:use-module (srfi srfi-11)
>    #:use-module (srfi srfi-34)
>    #:use-module (srfi srfi-35)
> +  #:use-module (web uri)
>    #:export (%repository-cache-directory
>              honor-system-x509-certificates!
>  
> @@ -61,7 +63,9 @@
>              git-checkout-url
>              git-checkout-branch
>              git-checkout-commit
> -            git-checkout-recursive?))
> +            git-checkout-recursive?
> +
> +         download-git-to-store))

Your patch has a number of tabs in it; they should be converted to spaces.

>  
>  (define %repository-cache-directory
>    (make-parameter (string-append (cache-directory #:ensure? #f)
> @@ -614,6 +618,24 @@ objects: 'ancestor (meaning that OLD is an ancestor of 
> NEW), 'descendant, or
>                                  #:recursive? recursive?
>                                  #:log-port (current-error-port)))))
>  
> +(define* (download-git-to-store store url commit
> +                             #:optional (name (basename url))
> +                                #:key (log (current-error-port)) recursive?)
> +  "Download Git repository from URL at COMMIT to STORE, either under NAME or
> +URL's basename if omitted.  Write progress reports to LOG.  RECURSIVE? has 
> the
> +same effect as the same-named parameter of 'git-fetch'."
> +  (define uri
> +    (string->uri url))

You've defined uri but not used it anywhere.

> +
> +    (call-with-temporary-directory
> +     (lambda (temp)
> +       (let ((result
> +              (parameterize ((current-output-port log))
> +                (git-fetch url commit temp
> +                        #:recursive? recursive?))))
> +         (and result
> +              (add-to-store store name #t "sha256" temp))))))
> +

Is there a reason for not using latest-repository-commit instead of
this?  That already takes care of temporary directories, ignores .git,
and reduces duplication if you download different tags from the same
repository:

  guix download -c v0.1.3 example.com/repo

  guix download -c v2.1.0 example.com/repo

>  ;; Local Variables:
>  ;; eval: (put 'with-repository 'scheme-indent-function 2)
>  ;; End:
> diff --git a/guix/scripts/download.scm b/guix/scripts/download.scm
> index 5a91390358..6253ecaa5c 100644
> --- a/guix/scripts/download.scm
> +++ b/guix/scripts/download.scm
> @@ -26,15 +26,19 @@
>    #:use-module (guix base32)
>    #:autoload   (guix base64) (base64-encode)
>    #:use-module ((guix download) #:hide (url-fetch))
> +  #:use-module ((guix git) #:select (download-git-to-store))
>    #:use-module ((guix build download)
>                  #:select (url-fetch))
>    #:use-module ((guix progress)
>                  #:select (current-terminal-columns))
> +  #:use-module ((guix serialization)
> +             #:select (write-file))
>    #:use-module ((guix build syscalls)
>                  #:select (terminal-columns))
>    #:use-module (web uri)
>    #:use-module (ice-9 match)
>    #:use-module (srfi srfi-1)
> +  #:use-module (srfi srfi-11)
>    #:use-module (srfi srfi-14)
>    #:use-module (srfi srfi-26)
>    #:use-module (srfi srfi-37)
> @@ -76,12 +80,20 @@
>                         (ensure-valid-store-file-name (basename url))
>                         #:verify-certificate? verify-certificate?)))
>  
> +(define* (download-git-to-store* url commit #:key recursive?)
> +  (with-store store
> +    (download-git-to-store store url commit
> +                        (ensure-valid-store-file-name (basename url))
> +                        #:recursive? recursive?)))
> +
>  (define %default-options
>    ;; Alist of default option values.
>    `((format . ,bytevector->nix-base32-string)
>      (hash-algorithm . ,(hash-algorithm sha256))
>      (verify-certificate? . #t)
> -    (download-proc . ,download-to-store*)))
> +    (download-proc . ,download-to-store*)
> +    (git-download-proc . ,download-git-to-store*)
> +    (commit . #f)))
>  
>  (define (show-help)
>    (display (G_ "Usage: guix download [OPTION] URL
> @@ -100,6 +112,9 @@ and 'base16' ('hex' and 'hexadecimal' can be used as 
> well).\n"))
>                           do not validate the certificate of HTTPS servers "))
>    (format #t (G_ "
>    -o, --output=FILE      download to FILE"))

This option exists, so it would make sense for

  guix download -c v0.1 example.com/repo -o ~/src/repo

to work as well.

> +  (format #t (G_ "
> +  -c, --git-commit=COMMIT
> +                         download a Git repository"))

Would it be possible to find a way to express this option so it wouldn't
conflict with potential other repositories (SVN, hg, etc)?

Maybe something like

  guix download --git -c v0.1 example.com/repo
  guix download --svn -c v0.1 example.com/repo

But in SVN, "revisions" are used instead of "commits," so perhaps
removing that terminology would be better.  We could do

  guix download --git example.com/repo v0.1
  guix download --svn example.com/repo v0.1

Or! We could use (fake) sub-commands like so:

  guix download git example.com/repo -c v0.1

This feels nice, since it matches e.g. 'guix import go'.  See
guix/import.scm for how this works.

>    (newline)
>    (display (G_ "
>    -h, --help             display this help and exit"))
> @@ -143,6 +158,9 @@ and 'base16' ('hex' and 'hexadecimal' can be used as 
> well).\n"))
>                                (lambda* (url #:key verify-certificate?)
>                                  (download-to-file url arg))
>                                (alist-delete 'download result))))
> +     (option '(#\c "git-commit") #t #f
> +             (lambda (opt name arg result)
> +               (alist-cons 'commit arg result)))
>  
>          (option '(#\h "help") #f #f
>                  (lambda args
> @@ -182,16 +200,31 @@ and 'base16' ('hex' and 'hexadecimal' can be used as 
> well).\n"))
>                        (leave (G_ "~a: failed to parse URI~%")
>                               arg)))
>             (fetch (assq-ref opts 'download-proc))
> +           (git-fetch (assq-ref opts 'git-download-proc))
> +        (commit (assq-ref opts 'commit))
>             (path  (parameterize ((current-terminal-columns
>                                    (terminal-columns)))
> -                    (fetch (uri->string uri)
> -                           #:verify-certificate?
> -                           (assq-ref opts 'verify-certificate?))))
> -           (hash  (call-with-input-file
> -                      (or path
> -                          (leave (G_ "~a: download failed~%")
> -                                 arg))
> -                    (cute port-hash (assoc-ref opts 'hash-algorithm) <>)))
> +                 (if commit
> +                     (git-fetch (uri->string uri) commit)

You don't actually seem to use the download-git-to-store procedure you
wrote above.  An oversight?

> +                        (fetch (uri->string uri)
> +                               #:verify-certificate?
> +                               (assq-ref opts 'verify-certificate?)))))
> +           (hash  (if (or (assq-ref opts 'recursive) commit)

The 'recursive' option is used here, but not specified in options or the
help message.  I'm also not sure what it's supposed to mean here.
'Recursive' would have no meaning outside of fetching a repo, and for
fetching a repo, 'recursive' should mean that the fetch is recursive,
and would be applied above.

> +                      (let-values (((port get-hash)
> +                                    (open-hash-port
> +                                   (assoc-ref opts 'hash-algorithm))))
> +                        (write-file path port
> +                                 #:select?
> +                                 (if commit
> +                                     (lambda (file stat) (not (equal? 
> (basename file) ".git")))
> +                                     (const #t)))
> +                        (force-output port)
> +                        (get-hash))
> +                   (call-with-input-file
> +                        (or path
> +                            (leave (G_ "~a: download failed~%")
> +                                   arg))
> +                        (cute port-hash (assoc-ref opts 'hash-algorithm) 
> <>))))

Rather than special-casing repositories, perhaps you'd consider
incorporating the first 1-3 patches from [0]?  This would make it easier
to, say, add other repository formats later.  It would make all this
roughly

  (hash (and path
             (file-hash* path
                         #:algorithm
                         (assq-ref opts 'hash-algorithm))))

modulo error reporting, of course.

[0] https://issues.guix.gnu.org/50072

--
Sarah