branch master updated: doc: programming-2022: Add the latest reviews.

[Ludovic Courtès]

This is an automated email from the git hooks/post-receive script.

civodul pushed a commit to branch master
in repository maintenance.

The following commit(s) were added to refs/heads/master by this push:
     new c6b0f7c  doc: programming-2022: Add the latest reviews.
c6b0f7c is described below

commit c6b0f7c176a0b16180b3bc9b9fe2b64a668ee91b
Author: Ludovic Courtès <ludo@gnu.org>
AuthorDate: Sun Jul 3 22:14:10 2022 +0200

    doc: programming-2022: Add the latest reviews.
    
    * doc/programming-2022/programming-reviews.txt,
    doc/programming-2022/programming-response.txt: New files.
---
 doc/programming-2022/programming-response.txt | 203 ++++++++++++++++++
 doc/programming-2022/programming-reviews.txt  | 291 ++++++++++++++++++++++++++
 2 files changed, 494 insertions(+)

diff --git a/doc/programming-2022/programming-response.txt 
b/doc/programming-2022/programming-response.txt
new file mode 100644
index 0000000..5dc01b5
--- /dev/null
+++ b/doc/programming-2022/programming-response.txt
@@ -0,0 +1,203 @@
+-*- Org -*-
+
+Dear reviewers,
+
+This resubmission is my attempt at taking into account the comments and
+suggestions that you made, in particular: clarifying the scientific
+context early on, decoupling the model from the tool, and elaborating on
+the lessons learned.
+
+The reviews were detailed, which I appreciated.  I modified parts of the
+paper that appeared to be unclear or left some questions open.  Please
+find below replies to individual questions and comments.
+
+* Reviewer A
+
+Thank you for the thorough review and for your comments and suggestions.
+I tried to address most issues in the document and to answer the many
+questions below.
+
+  - You write: “when Guix has been out and [sic] about for 3-5 years
+    […]”.  Guix has been out for 10 years, as shown for example by
+    reference 12.  I have now made it explicit in the introduction.
+
+  - On the paper title: Whether of not Guix is popular, I believe the
+    title is appropriate as the paper describes a solution designed in
+    that framework.  In other words, it’s about how Guix provides
+    building blocks to build a secure supply chain—that is what I meant.
+
+  - You write “We don’t know what Guix is”—while it is nowhere near as
+    popular as Linux (but no free software package is as popular as
+    Linux), Guix is not unknown in its domain; it’s the 8th biggest
+    GNU/Linux distribution according to <https://repology.org/> and
+    there has been a number of peer-reviewed publications about it, some
+    of which are listed at <https://guix.gnu.org/en/publications/>.
+
+  - What are the “IDEAS” (sic) behind Guix?  The “Background” section
+    provides answers (Reviewers B and C appear to agree), as well as
+    references 12 and 27.  The aggressive tone of this comment is
+    regrettable.
+
+  - I added a sentence in the intro stating the threat model for Guix
+    updates upfront, as suggested.
+
+  - You write: “this seems like a rather naive statement, given that C
+    does not behave the same across systems”.  The patronizing tone is,
+    again, uncalled for.  Please define “C”; what is C, at run time, if
+    not the C library and kernel?
+
+    I have clarified this by adding: “as evidenced by more than twenty
+    years of experience with the functional deployment model [12, 18]”.
+    This is the extent of the experience these two projects have
+    gathered, in production.  If you have references to substantiate
+    that this is a “naive statement”, I can revise it and include the
+    references.
+
+  - You write: “my understanding was that there is no *possibility* of
+    there being any difference”.  The paper makes no such claim.  The
+    first sentence of the paragraph you’re referring to says that build
+    isolation _helps_ achieve bit-for-bit reproducible builds.  I added
+    a couple of sentences discussing non-deterministic build processes.
+
+  - I clarified as suggested the meaning of arrows in the caption of
+    Figure 4.
+
+  - You write: “As the paper already notes, it breaks pull-requests […]
+    Ironically, this workflow was created precisely to support the
+    creation of systems like Linux, which Guix is clearly trying to
+    support”.  I am confused as to whether this comment is about systems
+    or about workflows.
+
+    The Linux *project* uses Git for its source code repository, but it
+    does *not* use the pull-request workflow.
+
+    The Linux *kernel* is used by GNU Guix System, the standalone
+    GNU/Linux distribution.
+
+  - About the stolen laptop scenario, you write: “(a) "immediately" is
+    not always realistic; and (b) without my laptop, how do I even track
+    all the projects that I'm contributing to?”; using the same
+    condescending tone as before, you add: “this strikes me as extremely
+    simplistic to the point of being naive.”
+
+    You are right that “immediately” is unrealistic; I changed it to
+    “quickly”, with the understanding that it’s “best effort”.
+
+    How to handle compromised developer keys points at a social issue;
+    it has to do with how the project grants commit rights.  Guix has
+    about 40 authorized committers.  Like most free software projects,
+    it has a procedure to grant commit access, limiting it to
+    experienced contributors who know each other.  In that sense, it’s
+    not unreasonable to expect a committer to notify their peer; but
+    it’s true that it’s part of the social contract among committers,
+    and not something that is “enforced” or “guaranteed”.
+
+  - In the flatmap-stream scenario, the original event-stream developer
+    willingly granted access to someone who turned out to be malicious.
+    It’s of course possible for an authorized developer to be malicious,
+    but as you write, it’s beyond our threat model.
+
+  - As you note, the ‘channels.scm’ configuration file is Scheme code,
+    not data.  That file, when it exists, consists of a few lines
+    written by the user (see
+    
<https://guix.gnu.org/manual/en/html_node/Specifying-Additional-Channels.html>
+    for the format), making it an unlikely attack vector.  In any case,
+    that file is not involved in the checkout authentication mechanism.
+
+  - Regarding downgrade attacks, by virtue of preventing downgrades of
+    Guix itself, we are effectively preventing downgrades of packages.
+    I believe this concern is addressed by the paragraph that starts
+    with “Doing a non-fast-forward update”.
+
+  - You write: “Suppose, say, GitHub were to adopt this idea. You'd hit
+    1000s of commits every few seconds”.  GitHub’s source code (if
+    that’s what you mean) is proprietary so we do not know its commit
+    rate.  It’s certainly orders of magnitude below what you write
+    though.
+
+    Nixpkgs, which is a package distribution stored in Git but is one of
+    the most active distributions, receives less than 4,000 commits per
+    month, according to
+    <https://github.com/NixOS/nixpkgs/graphs/commit-activity>; Brew (in
+    the same domain) gets less than 300 commits per months according to
+    <https://github.com/Homebrew/brew/graphs/commit-activity>.  The
+    Linux kernel (perhaps the most active free software project) gets
+    less than 6,000 commits per month according to
+    <https://github.com/torvalds/linux/graphs/commit-activity>.  I added
+    these figures to this paragraph for clarity.
+
+  - You write: “I'd like to know more about the user experience if this
+    scales up.”  I expanded the last paragraph of the “Evaluation”
+    section a bit.
+
+  - You write: “A useful point of comparison to what Guix does is signed
+    commits in (say) GitHub”.  I believe Section 4 and the relevant
+    paragraph of the “Related Work” section together answer this
+    question, though I’m of course open to suggestions.
+
+    Do GitHub/Lab “verified” labels mean that commits and their parents
+    were made by authorized developers?  No.  Does GitHub/Lab supports a
+    notion of cryptographic “authorization”?  No.  Is the project user
+    account information for GitHub/Lab available in-band?  No.  Are
+    GitHub/Lab’s user OpenPGP keys available in-band?  No.
+
+  - Regarding your last question, <https://guix.gnu.org/en/security/>
+    lists the Guix security response team—people who respond to reports
+    of security vulnerabilities in Guix itself or, more commonly, in the
+    packages that Guix provides.  Other people currently on the security
+    team did not contribute to this document.  They participated in the
+    public review process of the authentication mechanism, visible at
+    <https://issues.guix.gnu.org/22883>, but were not directly involved
+    in its design and implementation, nor in the authoring of this
+    article.
+
+I am thankful for the thoroughness of your review.  However, I regret
+its aggressive tone, which I find unnecessary and detrimental to our
+work together.
+
+* Reviewer B
+
+Thank you for your comments.  I fixed the typos you reported.
+
+* Reviewer C
+
+Thank you for your comments and suggestions.  Below are answers to some
+of the comments you made:
+
+  - This new revision of the article is an attempt to more clearly
+    decouple the model from the implementation in Section 4.  The
+    introduction clarifies the context by mentioning how signed commits
+    alone are insufficient when it comes to authenticating a complete
+    checkout.
+
+  - The background section goes to great lengths explaining Guix, as you
+    write.  I deemed it necessary because the functional deployment
+    model that Guix implements is very different from the mainstream
+    model of “binary distributions”; related work such as TUF and
+    in-toto targets binary distributions and is not applicable as-is to
+    Guix.  It is hard to understand why something different from TUF is
+    necessary without first getting an overview of the functional
+    deployment model.
+
+    Furthermore, I wanted to give a holistic view of the supply chain.
+    Related work in this area takes the same approach: for instance,
+    in-toto can be thought of as a generalization of TUF.  When looking
+    at the whole supply chain, reproducible builds and “boostrapping”
+    are key aspects.
+
+  - I have clarified in Section 3 why simply checking for commit
+    signatures is of no use.  (Sections 4 and 9 already answered that
+    question, but stating it earlier is certainly helpful.)
+
+  - Regarding the pull-request workflow: the proposed mechanism does not
+    prevent its use, but it prevents delegation of merges to the server,
+    as is done when clicking on “Merge” buttons in the web interface.  I
+    have clarified it in Section 4.
+
+  - You write that “attackers may learn about the identity of
+    authorizers”.  Indeed, authorizations are public, just like the
+    repository and its history is public.  I don’t think this has any
+    impact on the effectiveness of the mechanism though.  Also, the
+    mechanism is also usable on a private repository; no information is
+    leaked since all the information used by the authentication
+    mechanism is available in the repository itself.
diff --git a/doc/programming-2022/programming-reviews.txt 
b/doc/programming-2022/programming-reviews.txt
new file mode 100644
index 0000000..fe3b50c
--- /dev/null
+++ b/doc/programming-2022/programming-reviews.txt
@@ -0,0 +1,291 @@
+From: "Programming 7.1 HotCRP" <noreply@programming71.hotcrp.com>
+Subject: [Programming 7.1] Submission #22 "Building a Secure Software Supply 
Chain..." | Notification
+To: Ludovic Courtès <ludovic.courtes@inria.fr>
+Cc: jeremy.gibbons@cs.ox.ac.uk
+Date: Sun,  3 Apr 2022 17:04:40 +0000 (UTC) (19 hours, 35 minutes ago)
+Reply-To: robert.hirschfeld@hpi.de
+
+Dear Ludovic --
+
+Thank you for your submission to The Art, Science, and Engineering of
+Programming, Volume 7 Issue 1 (Programming 7.1).
+
+      Title: Building a Secure Software Supply Chain with GNU Guix
+    Authors: Ludovic Courtès (Inria)
+       Site: https://programming71.hotcrp.com/paper/22
+
+The reviewers would like to see a "major" MINOR REVISION of your
+submission.
+
+Let me expand on the "major" MINOR REVISION by directly quoting from
+the reviewers' discussion of your submission: "This is a major
+revision on the grounds that there may be a fair amount of editing AND
+the editing is essential, but a minor revision in that we don't see
+any problem with it being done, especially because we are not asking
+for new research or results, nor is there any danger that the author
+provides us some information that negates a key result."
+
+Because of the substantial amount of editing expected to be done, I
+would like to suggest the following:
+
+#1 - Please submit a revised version of your paper by the deadline for
+ the next issue (June 1, 2022); then your paper will be reviewed for 
+inclusion in Volume 7 Issue 2 of the journal.
+
+#2 - However, if you manage to finish revisions in time, you may
+ consider submitting your revised paper already by May 1, 2022 so that 
+your submission can still be examined for inclusion in Volume 7 Issue 1.
+
+For both cases: Supplementing the revised paper, please also submit
+(1) a cover letter that outlines your changes and how you considered
+the points made by the reviewers and (2) a latex diff (using 
+https://ctan.org/pkg/latexdiff) or a similarly annotated copy that
+clearly indicates the changes you made.
+
+In addition to the detailed reviews below please pay attention to the
+author-visible note, which summarizes the reviewers' discussion and 
+contains further advice on how to improve your submission.
+
+I hope you find the reviews helpful.
+
+Note: Starting with Volume 7, The Programming Journal offers Artifact
+Evaluation (AE). You indicated during submission that you do not
+intent to submit an artifact, but since you might have reconsidered,
+our AE Chairs will contact you about that in a separate e-mail.
+
+Please do not hesitate to ask if you have any questions.
+
+Best regards,
+Robert Hirschfeld
+
+--
+Robert Hirschfeld
+The Programming Journal, Associate Editor, Volume 7
+robert.hirschfeld@hpi.uni-potsdam.de
+https://www.hpi.uni-potsdam.de/swa/
+
+-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
+
+Review #22A
+===========================================================================
+
+Paper summary (for authors)
+---------------------------
+This paper is about the role of package managers in the software supply chain 
problem. Specifically, it (rightly) notes that package managers are both a 
source of problems in supply chain security and can play a useful role in 
mitigating these problems. It describes a new Nix-like package manager, GNU 
Guix, for source distribution. The main contribution is the use of OpenPGP 
signatures as part of the repository to enable a kind of chain-of-trust 
verification. The paper talks about the de [...]
+
+Assessment and comments (for authors)
+-------------------------------------
+[The following is my review as originally written. I feel it's useful for 
authors to be able to see a reviewer's state of mind at the time of authoring. 
At the end, I have provided updated comments listing what I would like to see 
done in a revision.]
+
+# Original Review
+
+Overall I am happy to read about a paper in this space. We have not paid 
enough attention to the problems of supply-chains, and it seems necessary to 
rethink their security architecture. Guix seems to be focused on doing that in 
a specific space. The paper is pretty easy reading and the system seems to have 
"legs". I liked all that about it.
+
+I am, however, not convinced that this paper is ready for publication. The 
paper is submitted in the Engineering track, which means it provides "knowledge 
and technical skills acquired through designing and building large systems and 
through calculated application of principles in building those systems. 
Examples include measurements of artifacts’ properties, development processes 
and tools, and quality assurance methods." That doesn't really describe the 
paper as submitted. Now, I'll ad [...]
+
+What I would like to see in a revision is a clear focus on deepening the 
contribution and reducing some of the less important details. The paper 
sometimes reads like a user manual. These parts can be removed, I think, to 
make room for teaching us more of a lesson about what you have seen and 
learned. (I am also sensitive to the possibility that perhaps it is just a bit 
early to be writing this paper: maybe when Guix has been out and about for 3-5 
years, we may learn enough Engineering le [...]
+
+Below I provide various comments noted while reading the paper, so they are in 
the order in which they show up in the prose:
+
+The paper title is poorly chosen. We don't know what Guix *is*. If it were 
really well-known ("Building a Secure Software Supply Chain with Linux") that 
would be okay. Otherwise the title is unhelpful and frustrating. Also, this is 
a research publication. Therefore, the title should as much as possible focus 
on IDEAS. What is the big IDEA behind Guix, that one can read and perhaps 
replicate in a different tool? That's what the title should emphasize.
+
+I'm curious why the paper doesn't use the structured abstract asked for by the 
journal. Did the author find it too limiting?
+
+"Guix is a software deployment tool" - "Guix is first and foremost source 
code" seems like a type error. Maybe the paper means to say that it's first and 
foremost *for* source code? In general, I found several places where the paper 
could benefit from a bit of proofreading.
+
+*What is the threat model?* It takes a long time to find out — only on pg 7 do 
we get what seems to be one. This needs to be up front and center in a paper 
with "secure" in the title.
+
+"Guix packages stand alone: they provide all the user-land software they need, 
down to the C library; this guarantees they behave the same on any system." — 
this seems like a rather naive statement, given that C does not behave the same 
across systems.
+
+"For example, the command guix build -- -check hello rebuilds the hello 
package locally and prints an error if the build result differs from that 
already available." — I don't understand this. Maybe I don't know what "already 
available" refers to; the phrase is too vague to denote unambiguously. But up 
to this point in the paper, my understanding was that there is no *possibility* 
of there being any difference — that's the whole point behind a functional 
build. So one could almost ask, p [...]
+
+In figure 4, please explain what the *direction* of the arrow represents. It's 
equally reasonable to view the tail of the arrow as the older commit and the 
head as newer, which left me very confused, until I realize you mean it the 
other way around. (Never assume the meaning of a picture is obvious! It may 
be…but it could be obviously *different*.)
+
+The authorization process, as the paper says, is indeed simple to state and 
enforce. It also strikes me as far too weak.
+
+1. As the paper already notes, it breaks pull-requests; the paper waves this 
away as not being in "the needs of the Guix project". Ironically, this workflow 
was created precisely to support the creation of systems like Linux, which Guix 
is clearly trying to support (given the extensive discussion of the 
trusting-trust paper). Without a clear sense of "the needs" (i.e., the 
*limitations*) of Guix, this comment cannot really be judged and hence 
entertained.
+
+2. As the paper again notes, anyone who has commit access has full power over 
the repo *including decommissioning other users*. The paper notes the case of a 
compromised account like a "stolen laptop", "they can notify other committers 
so they immediately remove the key, thereby preventing it from being used to 
push new commits". However, (a) "immediately" is not always realistic; and (b) 
without my laptop, how do I even track all the projects that I'm contributing 
to? I have dozens and  [...]
+
+3. Furthermore, there's the more insidious case of a developer not being aware 
they are under attack. Going back to a flatmap-stream-like scenario, the 
developer may not even notice that commits are being made (in their name) to a 
repo they are no longer watching. In that case, the real developer has no 
reason to notify anyone of anything.
+
+Admittedly, this last situation somewhat falls outside the scope of the threat 
model of the paper. However, that does not mean these aren't real issues. And 
the real problem here is not that I'm falling outside the threat model, but 
rather that the permission model seems far too simplistic. For one whose whole 
foundation is security, it seems to need to be more sophisticated. I really 
needed much more discussion on this, and defense of the design, since this is 
the core of the paper.
+
+"configuration file, which contains a declarative Scheme code snippet" — this 
(and the .scm name) suggests that the configuration file is itself evaluated, 
i.e., it's not a passive s-expression (like the authorizations file). What are 
the permissions of this program? Can this not be used to create harm? Does it 
actually use raw Scheme, or a variant like Rees's Security Kernel based on 
Scheme? (Apparently not, it uses Guile. Which leaves this issue very open. Or 
is it not a problem becaus [...]
+
+Downgrade attacks: This seems to address the case of Guix itself being 
upgraded, but what about downgrade attacks on individual packages? I think the 
same logic holds, but I find it curious that the paper does not talk about this.
+
+Section 8.3 talks about the performance of authentication and its sufficiency 
against the current commit frequency. However, this is not very satisfying. It 
may be that this is sufficient *for the number of current users*. But the 
general problem here is much broader than Guix. Suppose, say, GitHub were to 
adopt this idea. You'd hit 1000s of commits every few seconds, I expect. How 
parallelizable is this? How costly is this? There needs to be some discussion 
here about deploying this ide [...]
+
+For an Engineering paper, I would have liked to hear a lot more about what was 
learned from building and deploying this system. Right now this is mostly 
limited to the evaluation section (8.5), other than a tiny bit of prose outside 
it about, e.g., performance of authentication. I would have liked to read more 
detail here, and maybe some hard numbers. I would also have liked fewer fluffy 
statements like "been adopting authentication support without hassle" 
(especially since nobody can sa [...]
+
+Related to my question in 8.3, I'd like to know more about the user experience 
if this scales up. We can certainly agree that using OpenPGP will not be 
"without hassle" when you consider the vast number of non-technical people who 
use a system like GitHub. Setting that aside, what other things will they run 
into? For instance, artists use GitHub to share Figma designs with each other. 
What happens when they try to use this system? How simple and transparent will 
it be? What is the user e [...]
+
+Finally, I am unclear from the paper on the exact functioning of Guix in an 
important way. A useful point of comparison to what Guix does is signed commits 
in (say) GitHub. The related work engages with this, but the difference is not 
as persuasive as I would like. Moreover, given the focus on source, I would 
have thought that what we would get in Guix is not just an analog to that, but 
also something like a hash chain (say a Merkle DAG) of all the source that led 
to this point. That way [...]
+
+As an aside, I'm curious — this does not need to be in the paper — why the 
Guix security page [https://guix.gnu.org/en/security/] lists three names but 
only one is an author of this paper.
+
+# Comments After Discussion
+
+In general, I would like the author to follow the usual journal response 
process. Depending on the author's experience, if they are not used to writing 
these, I encourage consultation with the journal chair. I would be happy to 
provide feedback on drafts, too.
+
+After discussion with Reviewer B, I am much more positively disposed to the 
paper. Still, I feel the presentation is not where I would it to be. Hopefully 
the only thing that needs fixing is that, not the content proper (but improving 
the presentation may yet lay bare some content issues…).
+
+While I would like to see as many things as possible in my review responded to 
and/or addressed and fixed, the issues I most care about that will keep me from 
being in favor of the paper are:
+
+- Make the threat model explicit early on. It's very difficult to evaluate a 
security-related paper without a clear sense of the threat model.
+
+- Clarify issues like "guarantees same behavior", "build result differs", etc.
+
+- Respond to and as best as possible, address the concerns about the 
authentication model.
+
+- Explain the nature of the configuration file and, if it is executable, make 
clear why that is not a problem.
+
+- Discuss downgrade attacks.
+
+- Explain authentication at scale.
+
+- In general, provide more data on actual use, given that that is the main 
thing that is attractive about the work that this paper is about.
+
+- Address loose and sloppy remarks like "without hassle": either justify them 
with hard evidence, or remote them (and indeed, acknowledge those hassles 
explicitly).
+
+I look forward to the revision.
+
+
+* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
+
+
+Review #22B
+===========================================================================
+
+Paper summary (for authors)
+---------------------------
+This paper provides an overview of how Guix supports reproducible and
+accountable builds of an operating environment from source, and then
+focuses on a particular piece of the system: how trust is established
+for the initial sources and their evolution. Sources are drawn from
+Git, and the key to establishing trust is to associate a signature
+with each commit. Trusted signatures are also stored in the
+repository, which means that a single mechanism can be used for
+authenticating changes and authors of the changes.
+
+Assessment and comments (for authors)
+-------------------------------------
+The key idea seems simple, and as the paper explains, it's not
+specific to Guix. The idea applies to any context where a Git-style
+content-addressable graph is used to communicate a history of changes
+from an agreed-on starting point. The paper addresses both the
+mechanism and the social process that is used to establish the
+starting points. At the same time, the fact that the idea is
+implemented in Guix shows the relevance of the problem and provides
+substantial evidence that it works. Although the presentation is
+clear, I am nowhere near expert enough in security and trust to vet
+the idea in this paper --- but, very much in spirit of the idea here,
+I feel inclined to trust the protocol because it has been implemented
+and vetted by others who care.
+
+The paper is well written, and I am at a loss to recommend
+improvements to the presentation. A small, self-contained model of
+`guix git authenticate` might be nice, but I don't think it's
+necessary.
+
+Typos:
+
+page 8: "computer" -> "computed"
+
+page 15: "Git 'protocols', via" - comma and italics intended?
+
+
+* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
+
+
+Review #22C
+===========================================================================
+
+Paper summary (for authors)
+---------------------------
+The paper describes an authentication mechanism built into GNU Guix to protect 
against attacks that focus on git access like the introduction of malicious 
changes and downgrade attacks. The mechanism has been deployed in production 
two years ago, and the paper briefly reports on experiences on the provided 
support.
+
+Assessment and comments (for authors)
+-------------------------------------
+Strengths
+- Supply chain attacks, in particular, git attacks, is a relevant
+  topic - Proposed mechanism seems sound and it is used in production
+
+Weaknesses
+- I liked the tutorial-like style of the paper, but the paper is not written 
in a scientific way that contextualizes the work
+- It is unclear the relevance of the contributions to state-of-the-art
+  in the verification of git updates 
+Detailed Comments:
+
+- The introduction would best describe a conceptual research question
+  rather than a research question focused on a concrete tool. As far
+  as I understand, the work focuses on studying mechanisms for
+  verifiable git updates to prevent unauthorized modifications of
+  source code. Even if the solution is integrated into GNU Guix, it
+  may help to first describe the issues the work aims to tackle
+  decoupled from GNU Guix. Having the description of the solution
+  directly explained in the context of the tool makes it difficult to
+  place the work in a broader scientific context. Since the problems
+  and solution are described for GNU Guix, it is also difficult to
+  assess how relevant the work is beyond GNU Guix. 
+- The background section focuses too much on describing GNU Guix. The
+  paper would be stronger by reducing details that are not needed for
+  understanding the contributions of the authentication mechanism
+  (e.g. section 2.3 could be removed) and adding a more detailed
+  explanation on the kind of attacks the work aims to prevent (this is
+  now briefly explained in two paragraphs in section 3). A more
+  detailed description may also clarify how the attacks can be
+  performed when the attacker is not impersonating an authorized
+  developer. The article does not focus on protecting “against cases
+  where an attacker is able to impersonate an authorized developer”,
+  but how is it possible otherwise to conduct an introduction of
+  malicious changes (an attack targeted by the work)? 
+- Section 3 could be improved if it describes what measurements exist
+  today to deal with the targeted attacks so that the paper more
+  clearly conveys shortcomings in current work that this solution can
+  overcome.  For example, as mentioned in section 9, it is possible in
+  git to cryptographically sign a commit using a GPG key. However,
+  this is criticized as being a limited form of verification but it is
+  unclear why and which of the targeted attacks could then not be
+  prevented with such support for verified git updates. 
+- The fact that the designed tool supports off-line checkout authentication is 
said to be a novelty but, unfortunately, the paper does not properly 
contextualize the work and clarifies the relevance of such off-line support.
+
+- As far as I understand, the devised authentication mechanism does
+  not work well with the pull-request-style way of working popular in
+  git repositories, which may limit the applicability of the approach
+  beyond GNU Guix. 
+- The fact that .guix-authorizations is just a file stored in the
+  repository, seems to be a weak point of the solution as attackers
+  may learn about the identity of authorizers. 
+Minor typographic comments:
+- Page 3: its salient feature are => is
+- Page 5, Niewenzhuizen et al. misses the actual citation.
+
+Comments after PC discussion
+
+Overall, I am positive about the paper but I don’t think it is ready yet for 
publication in its current form. I believe the paper needs some reorganization 
of the material and additions as mentioned in the reviews. Besides the issues 
mentioned in the list by reviewer #a, I would also like to see the following 
things incorporated into the paper:
+
+-revise the scientific context early on so that readers can understand
+ what limitations current git authentication suffers from, which
+ motivates this work. 
+-decouple the model from the tool in the paper. The abstract does explain that 
the main contribution is a model and a tool to authenticate new Git revisions, 
but those two aspects are coupled in the main body, which makes it difficult to 
assess the relevance of the “model”.
+
+-elaborate on the lesson learned from having the solution in production;  
there is not enough attention given to that aspected in the submitted version 
while it is a strength of the work.
+
+
+-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
+
+Comment @A1
+---------------------------------------------------------------------------
+Overall, we are positive about the paper but don’t think it is ready yet for 
publication in its current form. We believe the paper needs some reorganization 
of the material and additions as mentioned in the reviews. Besides the issues 
mentioned in the list by reviewer #a, we would also like to see the following 
things incorporated into the paper:
+
+- Revise the scientific context early on so that readers can understand what 
limitations current git authentication suffers from, which motivates this work.
+
+- Decouple the model from the tool in the paper. The abstract does explain 
that the main contribution is a model and a tool to authenticate new Git 
revisions, but those two aspects are coupled in the main body, which makes it 
difficult to assess the relevance of the “model”.
+
+- Elaborate on the lesson learned from having the solution in production; 
there is not enough attention given to that aspected in the submitted version 
while it is a strength of the work.
+
+
+-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
+