Embed LilyPond source files inside generated PDF (issue 225040043 by address@hidden)

[v . villenave]

Reviewers: ,

Message:
Greetings everybody,

since there have been some discussions about updating GhostScript
lately, perhaps now would be a convenient time to resurrect this
feature...

https://lists.gnu.org/archive/html/lilypond-user/2012-07/msg00105.html
https://code.google.com/p/lilypond/issues/detail?id=2643
https://lists.gnu.org/archive/html/lilypond-devel/2012-07/msg00603.html

Unlike Sam Da Mota’s patch (or what Frescobaldi offered once upon a
time), my approach doesn’t rely on external programs (pdftk is heavy and
Java-centric, and has been banned from some distros due to licensing
concerns).

Instead, I’m using internally the pdfmark EMBED feature, a bit like
Reinhold’s comment on issue 2643 (but he used ANN and a visible PDF
annotation, where as my patch produces more cleanly-attached source
files. Therefore there’s a good chance it *could* work on other OSes too
(although that’s untested as of yet).

EMBED is not supported by the old version of GhostScript we’re still
bundling, so now would definitely be a good time for an upgrade.

I also had to mess with some of the source code and create a new
ly:source-files function. It involves a mixture of Scheme and C++ code
(no smobs though), as well as one new toplevel variable. Unlike Sam’s
patch, I didn’t bother to make a list of distribution-included init
files; I’m just discarding everything in LILYPOND_DATADIR (ly/*-init.ly
etc.). I don’t believe that ly:parser-include-strings hacks will be
handled properly, however.

Cheers,
Valentin.

Description:
Embed LilyPond source files inside generated PDF

This feature has been requested by Samuel Da Mota,
who even wrote a patch in 2012.  Unlike his approach,
this attempt is fully integrated in Lily’s pipeline and
does not involve external programs (namely bash and the
Java-based pdftk).

What I’m using internally is the pdfmark EMBED feature,
unlike Reinhold’s comment on issue 2643 which involved
ANN (and therefore led to a visible modification
of the PDF).  It should be handled by most PDF viewers,
and even those who don’t should degrade gracefully.

One caveat though: EMBED is not handled by the old version
of GhostScript we’re still bundling; it requires a newer
version.

I also had to mess with some of the source code and create
a new ly:source-files function.  It involves a mixture
of Scheme and C++ code (no smobs though), as well as one
new toplevel variable. By default all files in DATADIR
will be disregarded (ly/*-init.ly etc.), but there’s a
boolean switch in case one would need them too.

Please review this at https://codereview.appspot.com/225040043/

Affected files (+82, -2 lines):
  M Documentation/changes.tely
  M Documentation/usage/running.itely
  M lily/include/sources.hh
  M lily/sources.cc
  M scm/backend-library.scm
  M scm/framework-ps.scm
  M scm/lily.scm


Index: Documentation/changes.tely
diff --git a/Documentation/changes.tely b/Documentation/changes.tely
index  
1a5208fbd07a65590c901022a740e9d7e5c1884e..af2de9bc3081b2d0b13e9984385748b4168b6dcc  
100644
--- a/Documentation/changes.tely
+++ b/Documentation/changes.tely
@@ -62,6 +62,14 @@ which scares away people.
 @end ignore

 @item
+LilyPond source files may now be embedded inside the generated PDF files.
+This experimental feature is disabled by default and may be regarded as  
unsafe,
+as PDF documents with hidden content tend to present a security risk.
+Please note that not all PDF viewers have the ability to handle embedded
+documents (if not, the PDF output will appear normally and source files
+will remain invisible).  This feature only works with the PDF backend.
+
address@hidden
 Multi-measure rests have length according to their total duration,
 under the control of @code{MultiMeasureRest.space-increment}.
 @lilypond[quote]
Index: Documentation/usage/running.itely
diff --git a/Documentation/usage/running.itely  
b/Documentation/usage/running.itely
index  
fab28a6297e5658ce0317eeeef5aea26bba77c9a..a3caf81da2bedd248617a7ffb65f4f0ea64645be  
100644
--- a/Documentation/usage/running.itely
+++ b/Documentation/usage/running.itely
@@ -548,6 +548,10 @@ compilation.
 @tab @code{#f}
 @tab Dump output signatures of each system. Used for regression testing.

address@hidden @code{embed-source-code}
address@hidden @code{#f}
address@hidden Embed the LilyPond source files inside the generated PDF 
document.
+
 @item @code{eps-box-padding}
 @tab @code{#f}
 @tab Pad left edge of the output EPS bounding box by the given amount
Index: lily/include/sources.hh
diff --git a/lily/include/sources.hh b/lily/include/sources.hh
index  
2db87cee973e26d0c3d2cecc253a3f5e990a593d..61bee6a7033d0026a4fa1e2191acb6419e79b075  
100644
--- a/lily/include/sources.hh
+++ b/lily/include/sources.hh
@@ -22,6 +22,9 @@

 #include "lily-proto.hh"
 #include "std-vector.hh"
+#include "lily-guile.hh"
+
+static SCM source_file_list;

 class Sources
 {
Index: lily/sources.cc
diff --git a/lily/sources.cc b/lily/sources.cc
index  
b42300f47a7efc081ec34686f533088d7b1cc9bc..08564a5c961c6c190b2c6d92aa154eb048cd0e8e  
100644
--- a/lily/sources.cc
+++ b/lily/sources.cc
@@ -23,6 +23,15 @@
 #include "source-file.hh"
 #include "file-name.hh"
 #include "file-path.hh"
+#include "lily-guile.hh"
+#include "main.hh"
+#include <string>
+
+static void
+init_source_file_list ()
+{
+  source_file_list = SCM_EOL;
+}

 Sources::Sources ()
 {
@@ -66,6 +75,7 @@ Sources::get_file (string file_string, string const  
&current_dir)
           file_string = file_string_o;
         }
     }
+  source_file_list = scm_cons (ly_string2scm (file_string),  
source_file_list);

   Source_file *f = new Source_file (file_string);
   add (f);
@@ -86,3 +96,34 @@ Sources::~Sources ()
     }
 }

+ADD_SCM_INIT_FUNC (init_source_file_list, init_source_file_list);
+
+LY_DEFINE (ly_source_files, "ly:source-files", 0, 1, 0, (SCM include_dist),
+           "A list of LilyPond files being processed. An optional"
+           "boolean argument allows to include or exclude default"
+           "distribution files.")
+{
+  if (include_dist != SCM_BOOL_T)
+  {
+  string dist = lilypond_datadir;
+  long int k = dist.length ();
+  string elt;
+
+  SCM lst = source_file_list;
+  SCM walk;
+  SCM *prev;
+
+  for (prev = &lst, walk = lst;
+       scm_is_pair (walk);
+       walk = SCM_CDR (walk))
+    {
+      elt = ly_scm2string (SCM_CAR (walk));
+      if (dist.compare (elt.substr (0,k)) == 0)
+       *prev = SCM_CDR (walk);
+      else
+       prev = SCM_CDRLOC (walk);
+    }
+  }
+  return scm_reverse (source_file_list);
+}
+
Index: scm/backend-library.scm
diff --git a/scm/backend-library.scm b/scm/backend-library.scm
index  
3a70c5c07d081c2efaae7ed951685249a93b0be8..800ae82f17f313efcdc66a166a366d2bbdb18413  
100644
--- a/scm/backend-library.scm
+++ b/scm/backend-library.scm
@@ -82,6 +82,7 @@
                    "-dNOPAUSE"
                    "-dBATCH"
                    "-r1200"
+                   "-P"
                    (if (ly:bigpdfs) "-dSubsetFonts=false")
                    "-sDEVICE=pdfwrite"
                    (string-append "-sOutputFile="
Index: scm/framework-ps.scm
diff --git a/scm/framework-ps.scm b/scm/framework-ps.scm
index  
9e2b7521ad38f174573ddeaf7acbc086576e568a..7cd1fc9dcc8130a289292e5173c4a2b93d5fae92  
100644
--- a/scm/framework-ps.scm
+++ b/scm/framework-ps.scm
@@ -412,10 +412,30 @@


   (display "%%BeginProlog\n" port)
+
+   (if (ly:get-option 'embed-source-code)
+       (let ((source-list (ly:source-files #f))
+             (idx -1))
+         (display "\n/pdfmark where
+{pop} {userdict /pdfmark /cleartomark load put} ifelse" port)
+         (map (lambda (fname)
+                (set! idx (1+ idx))
+                (format port "\n
+/F (~a) (r) file def
+[ /_objdef {ly~a_stream} /type /stream /OBJ pdfmark
+[ {ly~a_stream} F /PUT pdfmark
+[ {ly~a_stream} << /Type /EmbeddedFile>> /PUT pdfmark
+[ /Rect [ 0 0 0 0] /Name /DummyAnchor~a
+/FS <</F (~a) /Type /FileSpec /EF << /F {ly~a_stream} >> >>
+/Subtype /FileAttachment /EMBED pdfmark\n"
+                fname idx idx idx idx fname idx))
+          source-list)))
+
   (format
    port
-   "/lilypond-datadir where {pop} {userdict /lilypond-datadir (~a) put }  
ifelse"
+   "\n/lilypond-datadir where {pop} {userdict /lilypond-datadir (~a) put }  
ifelse"
    (ly:get-option 'datadir))
+
   (if load-fonts?
       (for-each (lambda (f)
                   (format port "\n%%BeginFont: ~a\n" (car f))
@@ -731,7 +751,7 @@ system-by-system output.  For that, use the EPS backend  
instead,

   lilypond -dbackend=eps FILE

-If have cut & pasted a lilypond fragment from a webpage, be sure
+If you have cut & pasted a lilypond fragment from a webpage, be sure
 to only remove anything before

   %% ****************************************************************
Index: scm/lily.scm
diff --git a/scm/lily.scm b/scm/lily.scm
index  
6322e0196053af7480909fc3ddbbd2d148b56e00..24fc792fd3643b0e2bf3384f451da0cb05421db4  
100644
--- a/scm/lily.scm
+++ b/scm/lily.scm
@@ -227,6 +227,9 @@ configurations.")
      #f
      "Dump output signatures of each system.  Used for
 regression testing.")
+    (embed-source-code
+     #f
+     "Embed the source files inside the generated PDF document.")
     (eps-box-padding
      #f
      "Pad left edge of the output EPS bounding box by