Re: "Too much files to mangle" error

[Thomas Schmitt]

Hi,

Mathieu Othacehe wrote:
> xorriso : UPDATE :  434919 files added in 18 seconds
> libisofs: FAILURE : HFS+ map nodes aren't implemented
> libisofs: FAILURE : Too much files to mangle, cannot guarantee unique file 
> names

Ouchers. That's in the most moldy cellar dimensions of libisofs.

The "too much files to mangle" error can occur in the ISO 9660 tree,
the HFS+ tree, the ISO 9660:1999 tree, and the Joliet tree.
In this case the HFS+ generator is so friendly to also issue a message
before it bails out with ISO_MANGLE_TOO_MUCH_FILES.

My diagnosis attempt below brings me to the theory that a directory has
more than 30720 files and that the HFS+ implementation in libisofs cannot
stand this.


> I'm trying to generate an ISO image for GNU Guix
> [...]
> Is this possible to get around this limit?

Try to suppress production of HFS+. AFAIK, guix uses grub-mkrescue, which
issues xorrisofs option -hfsplus quite unconditionally.
  http://git.savannah.gnu.org/cgit/grub.git/tree/util/grub-mkrescue.c#n718

If you cannot keep your grub-mkrescue from doing this, then i can offer
the script grub-mkrescue-sed.sh which can be submitted to grub-mkrescue
by option

  --xorriso=/...absolute.../...path.../grub-mkrescue-sed.sh

The script usage is described in the large comment at its start.
It manipulates the xorriso options from grub-mkrescue and then uses
them in a run of xorriso.

The default setting will change the xorrisofs options so that no HFS+
and no GPT will be generated. I.e. EFI will be served by a MBR partition
of type 0xef rather than a GPT partition.
It is advised to add xorrisofs option
  -partition_offset 16
to the grub-mkrescue options in order to get the EFI partition claimed
as part of the ISO filesystem and at the same time get a mountable ISO
partition. (Your quite large directory tree will be recorded twice in
the ISO, i fear. So the result will grow by 100 MB or more.)

If you export environment variable
  MKRESCUE_SED_MODE=gpt_appended
before the grub-mkrescue run with grub-mkrescue-sed.sh, then you get GPT
and unconditionally -partition_offset 16.

If you don't have the script from a xorriso or libisoburn installation,
you can download it from

  
https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/frontend/grub-mkrescue-sed.sh

(It should not be too hard to add a MKRESCUE_SED_MODE "no_hfs" to the
script or to create an own pseudo-xorriso iscript which passes all its
aruments to xorriso, except the problematic "-hfsplus".)


-------------------------------------------------------------------------
Now for a diagnosis:

HFS+ in libisofs is a contribution by Vladimir Serbinenko who was back then
maintainer of GRUB and is still to see from time to time at grub-devel
mailing list.

Mangling is a collision prevention mechanism for filesystem trees which
have a more restricted name length or character set than X/Open demands.
Names get changed by some scheme that has its natural limits.

But Vladimir seems to have re-used the error code for something different
than mangling:

  https://dev.lovelyhq.com/libburnia/libisofs/raw/master/libisofs/hfsplus.c
line 1791 has:

    if (target->hfsp_nnodes > (cat_node_size - 0x100) * 8)
      {
        iso_msg_submit(target->image->id, ISO_MANGLE_TOO_MUCH_FILES, 0,
                        "HFS+ map nodes aren't implemented");
        ret = ISO_MANGLE_TOO_MUCH_FILES;
        goto ex;
      }

This looks more like a plain overflow.

target->hfsp_nnodes is a counter which i have difficulties to understand:

      target->hfsp_levels[level].level_size++;
      target->hfsp_nnodes += target->hfsp_levels[level].level_size;

cat_node_size is set by

    target->hfsp_cat_node_size = 2 * target->opts->hfsp_block_size;
    ...
    cat_node_size = target->hfsp_cat_node_size;

grub-mkrescue sets the HFS+ blocksize to 2048.
My rough theory is that a HFS+ directory cannot have more than
   (4096 - 256) * 8 = 30720
file names. There seems to be a workaround "HFS+ map nodes". But the HFS+
code in libisofs reports not to have it implemented.


I will ask at grub-devel mailing list whether my theory is correct and
whether HFS+ is still important enough to be used for I386 and X86_64
systems. (It could be essential for POWERPC_IEEE1275.)
But except Vladimir nobody else there has an opinion about grub-mkrescue
and Vladimir does not appear often any more.


Have a nice day :)

Thomas