bug#52477: 27.1; .crx and .crx3 are zip files

[Trent W. Buck]

Michael Albinus wrote:
> "Trent W. Buck" <trentbuck@gmail.com> writes:
> > Is it reasonable for emacs to just chuck out arc-mode (and tar-mode
> > and jka-compr) and instead just use libarchive?
> > It's already used by vlc and a lot of GNOME stuff.
> >
> > libarchive already understands both .pyz and .crx3 files, at least for 
> > reading: …
>
> No. Neither .pyz nor .crx3? formats are supported by libarchive, I've just
> tested it with Nautilus, the GNOME file manager. bsdtar uses several
> archive libraries, I guess another archive library must be responsible
> for these formats.

Ah, sorry for the confusion.

On my system (Debian 11) bsdtar is provided by "Source Package: libarchive" and
only depends on libarchive13 and libc6, so
I assumed it was a reasonable test of "what can libarchive do?".

If it is using another library, I cannot see how:

    bash5$ ldd /usr/bin/bsdtar
            linux-vdso.so.1 (0x00007ffc6b1b3000)
            libarchive.so.13 => /lib/x86_64-linux-gnu/libarchive.so.13 
(0x00007fdf3c997000)
            libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdf3c7d2000)
            libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8 
(0x00007fdf3c78a000)
            libacl.so.1 => /lib/x86_64-linux-gnu/libacl.so.1 
(0x00007fdf3c77f000)
            liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 
(0x00007fdf3c757000)
            libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 
(0x00007fdf3c67c000)
            liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 
(0x00007fdf3c657000)
            libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 
(0x00007fdf3c644000)
            libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fdf3c627000)
            libxml2.so.2 => /lib/x86_64-linux-gnu/libxml2.so.2 
(0x00007fdf3c479000)
            /lib64/ld-linux-x86-64.so.2 (0x00007fdf3ca8f000)
            libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
(0x00007fdf3c457000)
            libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fdf3c44f000)
            libicuuc.so.67 => /lib/x86_64-linux-gnu/libicuuc.so.67 
(0x00007fdf3c266000)
            libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fdf3c122000)
            libicudata.so.67 => /lib/x86_64-linux-gnu/libicudata.so.67 
(0x00007fdf3a609000)
            libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 
(0x00007fdf3a43c000)
            libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 
(0x00007fdf3a422000)

    bash5$ strace -f /usr/bin/bsdtar tf *.pyz |& grep -F -e exec -e fork
    execve("/usr/bin/bsdtar", ["/usr/bin/bsdtar", "tf", "MyCoolApp.pyz"], 
0x7ffc169ca188 /* 70 vars */) = 0

Nor can I see embedded copies of libraries in
https://salsa.debian.org/debian/libarchive/-/tree/master/tar

My C-fu is not up to scratch, but Python has a thin FFI interface to 
libarchive, and
that also works fine:

    bash5$ python3
    Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
    [GCC 10.2.1 20210110] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import libarchive
    >>> with libarchive.file_reader('MyCoolApp.pyz') as z:
    ...   for i in z:
    ...     print(i.strmode, i.size, i.name, sep='\t')
    ... 
    b'-rw-r--r--'       23      __main__.py
    b'-rw-r--r--'       0       __init__.py

I had a quick look, but I can't immediately tell why nautilus would not work, as
I cannot even find where nautilus uses libarchive.

When I tested nautilus 3.38.2-1+deb11u1, it would not even open a .zip.
It only offered to run file-roller.

I found file-roller includes nautilus/nautilus-fileroller.c
which uses a hard-coded archive_mime_types[] that it will activate for.
application/x-chrome-extension (.crx3) is not in that list.

There is no MIME type for .pyz because it's just a .zip with an
optional shebang prepended ("#![^\n]*\n").
Nautilus and file-roller treat a .pyz WITHOUT a shebang
identically to a .zip.
Presumably this is due to libmagic-like MIME guessing.


It seems to me that:

  1. when given a file in pyz/crx/crx format,
     libarchive will auto-detect and use zip format.

     I think it just finds the zip archive at an offset, and
     leaves content before that offset alone.

     This would allow it to read/extract/edit existing pyz/crx/crx3,
     but not create a new one.

     This is also true for libarchive bsdtar and Info-ZIP unzip,
     although the latter emits a warning about the offset.

  2. when given a file in pyz/crx/crx format,
     emacs arc-mode will not auto-detect zip format.

  3. when given a file in pyz/crx/crx3 format,
     file-roller & nautilus will not auto-detect any format.