Re: thank you for libidn 1.34 however ...

[Tim Rühsen]

Hi Dennis,


thanks for your detailed report, really appreciated !


Regarding the alloca(): Solaris surely has one (maybe a define), you
have to #include <alloca.h> in fuzz/main.c.

Will fix this for the next release - likely you can remove the whole block:

if (valgrind && *valgrind) {

...

}


> However my next issue was a sigsegv in ./fuzz/libidn_stringprep_fuzzer
thus...

Was this also on Solaris ? I am not aware of any system dependent code
where the crash happens...
will test this next week on Solaris.

> On yet another system I get some messy CFLAGS that create other problems

Looks like a hard-coded compiler flag... such is always bad (has to be
removed).
In this case it works with gcc and clang, the two compilers we build with.

> ...I may need to step up to the plate here
> and provide some online datacenter access for folks in open source
> projects to do across multiple platforms and multiple os testing ...

Always a good idea ! But there are some points to think of
1. libidn has been obsoleted by libidn2 though there is no replacement
for stringprep yet.
2. because of 1. we don't much work into libidn any more, so the goal is
use libidn2 everywhere instead of libidn
3. we have very limited time/capacity (any help appreciated, up to new
maintainers :-)). that is the main reason
we didn't test on all kinds of different systems (though we have access
to Solaris and the gcc build platform).
4. the libidn code was available for testing since >1.5 years for
everybody. If there is no feedback for a while I
expect everybody tested on their beloved platform. Of course I know not
many people do so, but hey, this is
not only free open source, it is also open development for everyone.

So if you are willing to put some work into libidn, think about libidn2
first - the API is pretty much compatible.
And of course: let me know if I can help you to step in.

With Best Regards, Tim


On 31.03.2018 20:41, Dennis Clarke wrote:
> Hello Tim :
>
>     Seems to work aas expected on Linux systems however I have not
> completed the build/test cycle on 32bit x86 nor the x86_64 platforms
> yet. I went after the 64-bit risc places first, powerpc and sparc.
> Well as you would imagine the posix Solaris world simply won't support
> that strange call "alloca()" which isn't portable at all. In fact the
> man page comes with warnings. Makes me wonder why it exists.  In any
> case :
>
> n0$ diff -c ./fuzz/main.c_backup ./fuzz/main.c
> *** ./fuzz/main.c_backup        Fri Jun 30 17:27:34 2017
> --- ./fuzz/main.c       Sat Mar 31 18:03:16 2018
> ***************
> *** 87,95 ****
>
>                 if (valgrind && *valgrind) {
>                         size_t cmdsize = strlen(valgrind) +
> strlen(argv[0]) + 32;
> !                       char *cmd = alloca(cmdsize);
> !
>                         snprintf(cmd, cmdsize, "TESTS_VALGRIND="" %s
> %s", valgrind, argv[0]);
>                         return system(cmd) != 0;
>                 }
>         }
> --- 87,95 ----
>
>                 if (valgrind && *valgrind) {
>                         size_t cmdsize = strlen(valgrind) +
> strlen(argv[0]) + 32;
> !                       char *cmd = malloc(cmdsize);
>                         snprintf(cmd, cmdsize, "TESTS_VALGRIND="" %s
> %s", valgrind, argv[0]);
> +                       free(cmd);
>                         return system(cmd) != 0;
>                 }
>         }
> n0$
>
>
> However my next issue was a sigsegv in ./fuzz/libidn_stringprep_fuzzer
> thus :
>
> n0$ dbx ./fuzz/libidn_stringprep_fuzzer
> ./time_1522519774-pid_6654-fid_libidn_stringpre.core
> Reading libidn_stringprep_fuzzer
> dbx: warning: core object name "libidn_stringpr" matches
> object name "libidn_stringprep_fuzzer" within the limit of 14.
> assuming they match
> core file header read successfully
> Reading ld.so.1
> Reading libidn.so.11.6.16
> Reading libintl.so.8.1.5
> Reading libc.so.1
> Reading libiconv.so.2.6.0
> Reading libc_psr.so.1
> program terminated by signal SEGV (no mapping at the fault address)
> Current function is _g_utf8_normalize_wc
>   798     while ((max_len < 0 || p < str + max_len) && *p)
> (dbx) where
> =>[1] _g_utf8_normalize_wc(str = (nil), max_len = -1, mode =
> G_NORMALIZE_ALL_COMPOSE), line 798 in "nfkc.c"
>   [2] stringprep_ucs4_nfkc_normalize(str = 0x100127b90, len = 5), line
> 1120 in "nfkc.c"
>   [3] stringprep_4i(ucs4 = 0x100127b90, len = 0xffffffff7fffec70,
> maxucs4len = 17U, flags = <unknown enum member 0>, profile =
> 0xffffffff7ef3d950), line 170 in "stringprep.c"
>   [4] stringprep_4zi_1(ucs4 = 0x100127b90, ucs4len = 5U, maxucs4len =
> 17U, flags = <unknown enum member 0>, profile = 0xffffffff7ef3d950),
> line 290 in "stringprep.c"
>   [5] stringprep_4zi(ucs4 = 0x100127b90, maxucs4len = 17U, flags =
> <unknown enum member 0>, profile = 0xffffffff7ef3d950), line 336 in
> "stringprep.c"
>   [6] LLVMFuzzerTestOneInput(data = 0x100127a70 "\n", size = 68U),
> line 94 in "libidn_stringprep_fuzzer.c"
>   [7] test_all_from(dirname = 0xffffffff7ffff180
> "/usr/local/build/libidn-1.34_SunOS5.10_sparcv9.001/fuzz/libidn_stringprep_fuzzer.in"),
> line 71 in "main.c"
>   [8] main(argc = 1, argv = 0xffffffff7ffff328), line 103 in "main.c"
> (dbx) quit
> n0$
>
> On yet another system I get some messy CFLAGS that create other
> problems :
>
> /opt/solarisstudio12.4/bin/cc -D_STDC_C99= -DHAVE_CONFIG_H -I. -I..
> -I../lib -I../lib -I../gl -I../gl
> -DSRCDIR=\"/usr/local/build/libidn-1.34_SunOS5.10_sparcv9base.001/fuzz\"
> -DTEST_RUN -I/usr/local/include -D_TS_ERRNO -D_POSIX_PTHREAD_SEMANTICS
> -D_LARGEFILE64_SOURCE -D_REENTRANT   -Wno-vla -m64 -xarch=sparc
> -xcache=generic -errwarn=%none -errhdr=%user -erroff=%none
> -errtags=yes -errfmt=error -errshort=full -xstrconst -xildoff
> -xmemalign=8s -xnolibmil -Xa -xcode=pic32 -xregs=no%appl -xlibmieee
> -mc -g -xs -ftrap=%none -Qy -xbuiltin=%none -xdebugformat=dwarf
> -xunroll=1 -D_TS_ERRNO -D_POSIX_PTHREAD_SEMANTICS
> -D_LARGEFILE64_SOURCE -c -o libidn_toascii_fuzzer.o
> libidn_toascii_fuzzer.c
> cc: -W option with unknown program no-vla
> gmake[2]: *** [libidn_toascii_fuzzer.o] Error 1
>
> Yet that two line code change makes no difference whatsoever on a Debian
> sid PowerPC64 system. So this is getting messy fast.
>
> I think the real issue here is not the code but the test environment. I
> have libidn-1.33 which builds and tests and works everywhere.  Today I
> have 1.34 which doesn't. However it is a "stable" release. So I am
> thinking a better test environment needs to exist and libidn isn't the
> only project that needs this. I may need to step up to the plate here
> and provide some online datacenter access for folks in open source
> projects to do across multiple platforms and multiple os testing. Just
> a thought.
>
> Dennis Clarke
> ye old UNIX/Linux programmer type

signature.asc
Description: OpenPGP digital signature