Re: Add option for forcing a 64bit hash on 32bit systems

[Érico Nogueira]

On Sat Dec 26, 2020 at 5:26 PM -03, Kevin Atkinson wrote:
> On Tue, 22 Dec 2020, Érico Nogueira wrote:
>
> > As seen in [1], aspell has a rather unfortunate design limitation in its
> > compiled dictionary format. This not only makes it impossible to share
> > these files between systems with different endianness, but, if compiled
> > with default settings, makes sharing impossible between 32bit and 64bit
> > systems as well.
>
> Aspell compiled dictionary formats are not really meant to be portable.
> Compiling a dictionary is now very fast, it is even faster if checks are
> disabled. One thing I will be open to is the creation of a portable text
> based format which can be compiled very quickly on startup. This will
> take some refactoring though to make it work.

Allowing compilation at run time would allow for simpler packaging,
especially when cross compiling. So this would be very nice :)

>
> > The only compatibility option offered is forcing 32bit hashes for all 
> > systems, which makes 64bit systems incapable of reading 64bit dictionaries. 
> > Since most systems nowadays (especially ones where aspell would be used) 
> > are 
> > 64bit, it feels kind of backwards to force 32bit dicts. Would it be 
> > possible, then, to force 64bit dicts on 32bit systems? Would a contribution 
> > that adds this possibility be accepted?
>
> To be clear the only thing this does is change the type the hash
> function
> used from size_t to u32int in modules/speller/default/readonly_ws.cpp:
> #ifdef USE_32_BIT_HASH_FUN
> typedef u32int hash_int_t;
> #else
> typedef size_t hash_int_t;
> #endif
> All integers used in the dictionary are 32 bit as 64 bit integers will
> be an
> overkill. The fact that a 64 bit hash function is used on a 32 bit
> integer is
> an oversight. I would rather that a 32 bit hash function is used on all
> systems. The only reason that option exists is to avoid breaking
> dictionary
> compatibility on 64bit systems. I am open to enabling 32 bit hashes by
> default on the next major version bump.
>
> Note that on Debian Aspell is compiled with --enable-32-bit-hash-fun.

I was looking into simplifying how we build the dictionaries in Void
Linux.  Since our (32-bit) ARM packages are cross compiled from 64-bit
hosts, including aspell dictionaries, I thought standardizing on a
64-bit format would be best. It might make sense, then, to clear up the
explanation in [1], since it isn't clear that 32-bit hashes are actually
preferred. Given your explanation, I will simply force the 32-bit hashes
for all platforms, which is definitely simpler.

As long as it is included in release notes, I think enabling 32-bit
hashes by default should work just fine, for what it's worth.

[1] 
http://aspell.net/man-html/Using-32_002dBit-Dictionaries-on-a-64_002dBit-System.html

>
> Kevin

Thank you for the explanation, and happy holidays!
Érico