Re: bash cores if nscd disabled on Solaris LDAP sasl/gssapi client

[Serge Dussud - Sun Microsystems]

following-up on 3rd item:
[..]
> > 3.  What does the traceback look like when bash is run under gdb and
> >     allowed to fail?
> >
> > I suspect that the libraries are pre-bound to use the system's malloc,
> > and the calls to different malloc libraries are causing the core dumps.
> > Another possibility is that libc functions are using private pseudo-
> > global libc malloc interfaces, causing the libc malloc to be linked in.
> > Either way, the trace and library load address maps indicate that the
> > process is dying in the libc malloc.  One way to confirm my suspicion
> > is to start bash under gdb, set a breakpoint in malloc, and see where
> > it stops.
>
> OK, I'll try that and come back to you.
So here's a cut & paste of my gdb session, showing that bash and libc 
malloc routines are mixed (see below for possible explanation / fix):

$ gdb /usr/bin/bash
GNU gdb 6.3.50_2004-11-23-cvs
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "sparc-sun-solaris2.11"...
(no debugging symbols found)

(gdb) start
Breakpoint 1 at 0x2ece4
Starting program: /usr/bin/bash
(no debugging symbols found)
(no debugging symbols found)
(no debugging symbols found)
warning: Lowest section in /lib/libdl.so.1 is .dynamic at 00000118
(no debugging symbols found)
(no debugging symbols found)
0x0002ece4 in main ()
(gdb) break soft_build_secret_key_object
Function "soft_build_secret_key_object" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y

Breakpoint 2 (soft_build_secret_key_object) pending.
(gdb) continue
Continuing.
warning: Lowest section in /lib/libpthread.so.1 is .dynamic at 000003bc
warning: Lowest section in /lib/libthread.so.1 is .dynamic at 00000478
warning: Lowest section in /lib/librt.so.1 is .dynamic at 000001b0
Breakpoint 3 at 0xfea9fac0
Pending breakpoint "soft_build_secret_key_object" resolved

Breakpoint 3, 0xfea9fac0 in soft_build_secret_key_object ()
  from /usr/lib/security/pkcs11_softtoken.so
(gdb) break malloc
Breakpoint 4 at 0xba4a4
(gdb) continue
Continuing.

Breakpoint 4, 0x000ba4a4 in malloc ()
(gdb) where
#0  0x000ba4a4 in malloc ()
#1  0xff0d0bf0 in calloc () from /lib/libc.so.1
#2  0xfea9faec in soft_build_secret_key_object ()
  from /usr/lib/security/pkcs11_softtoken.so
#3  0xfeaa0ea4 in soft_build_object ()
  from /usr/lib/security/pkcs11_softtoken.so
#4  0xfeaa5474 in soft_add_object () from 
/usr/lib/security/pkcs11_softtoken.so
#5  0xfea97c64 in C_CreateObject () from 
/usr/lib/security/pkcs11_softtoken.so
#6  0xfe9b5de4 in init_key_uef () from /usr/lib/gss/mech_krb5.so
#7  0xfe9b4d34 in krb5_c_make_random_key () from /usr/lib/gss/mech_krb5.so
#8  0xfe9f19d8 in krb5_generate_subkey () from /usr/lib/gss/mech_krb5.so
#9  0xfe9f472c in krb5int_generate_and_save_subkey ()
  from /usr/lib/gss/mech_krb5.so
#10 0xfe9f4960 in krb5_mk_req_extended () from /usr/lib/gss/mech_krb5.so
#11 0xfe9a5098 in make_ap_req_v1 () from /usr/lib/gss/mech_krb5.so
#12 0xfe9a55c8 in new_connection () from /usr/lib/gss/mech_krb5.so
#13 0xfe9a5f8c in krb5_gss_init_sec_context () from 
/usr/lib/gss/mech_krb5.so
#14 0xfe9a2bbc in k5glue_init_sec_context () from /usr/lib/gss/mech_krb5.so
#15 0xfec84a4c in gss_init_sec_context () from /usr/lib/libgss.so.1
#16 0xfecc32f4 in gssapi_client_mech_step () from /usr/lib/sasl/gssapi.so.1
#17 0xfed173e4 in sasl_client_step () from /usr/lib/libsasl.so.1
#18 0xfed172b0 in sasl_client_start () from /usr/lib/libsasl.so.1
#19 0xfef1737c in nsldapi_sasl_do_bind () from /usr/lib/libldap.so.5
#20 0xfef17afc in ldap_sasl_interactive_bind_s () from /usr/lib/libldap.so.5
#21 0xfef6e478 in doSASLBind () from /usr/lib/libsldap.so.1
#22 0xfef6c880 in openConnection () from /usr/lib/libsldap.so.1
#23 0xfef6bcdc in makeConnection () from /usr/lib/libsldap.so.1
#24 0xfef6d234 in getConnection () from /usr/lib/libsldap.so.1
#25 0xfef6d38c in __s_api_getConnection () from /usr/lib/libsldap.so.1
#26 0xfef60048 in get_current_session () from /usr/lib/libsldap.so.1
#27 0xfef60f24 in search_state_machine () from /usr/lib/libsldap.so.1
#28 0xfef62158 in ldap_list () from /usr/lib/libsldap.so.1
#29 0xfef62264 in __ns_ldap_list () from /usr/lib/libsldap.so.1
#30 0xfefbbe88 in _nss_ldap_lookup () from /usr/lib/nss_ldap.so.1
#31 0xfefba124 in getbyuid () from /usr/lib/nss_ldap.so.1
#32 0xff0edb08 in nss_search () from /lib/libc.so.1
#33 0xff0da488 in getpwuid_r () from /lib/libc.so.1
#34 0x00030e88 in get_current_user_info ()
#35 0x000310c8 in get_current_user_info ()
#36 0x0002f178 in main ()
(gdb) continue
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0xff0e59c8 in realfree () from /lib/libc.so.1
(gdb) where
#0  0xff0e59c8 in realfree () from /lib/libc.so.1
#1  0xff0e62ac in cleanfree () from /lib/libc.so.1
#2  0xff0e53e0 in _malloc_unlocked () from /lib/libc.so.1
#3  0xff0e52c0 in malloc () from /lib/libc.so.1
#4  0xfea9d600 in get_bigint_attr_from_template ()
  from /usr/lib/security/pkcs11_softtoken.so
#5  0xfea9fe80 in soft_build_secret_key_object ()
  from /usr/lib/security/pkcs11_softtoken.so
#6  0xfeaa0ea4 in soft_build_object ()
  from /usr/lib/security/pkcs11_softtoken.so
.......



I then noticed from compilation logs that pkcs11_softtoken.so is built 
with -Bdirect ld(1) flag. I then recompile a bash using '-z interpose' 
ld(1) flag and it looks better:



$ gdb ./bash
GNU gdb 6.3.50_2004-11-23-cvs
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "sparc-sun-solaris2.11"...
(gdb) start
Breakpoint 1 at 0x2ecc4
Starting program: /var/tmp/1-23961034/bash
warning: (Internal error: pc 0x0 in read in psymtab, but not in symtab.)

warning: Lowest section in /lib/libdl.so.1 is .dynamic at 00000118
0x0002ecc4 in main ()
(gdb) break soft_build_secret_key_object
Function "soft_build_secret_key_object" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y

Breakpoint 2 (soft_build_secret_key_object) pending.
(gdb) continue
Continuing.
warning: Lowest section in /lib/libpthread.so.1 is .dynamic at 000003bc
warning: Lowest section in /lib/libthread.so.1 is .dynamic at 00000478
warning: Lowest section in /lib/librt.so.1 is .dynamic at 000001b0
Breakpoint 3 at 0xfea9fac0
Pending breakpoint "soft_build_secret_key_object" resolved

Breakpoint 3, 0xfea9fac0 in soft_build_secret_key_object ()
  from /usr/lib/security/pkcs11_softtoken.so
(gdb) break malloc
Breakpoint 4 at 0xba484
(gdb) continue
Continuing.

Breakpoint 4, 0x000ba484 in malloc ()
(gdb) where
#0  0x000ba484 in malloc ()
#1  0xfea9d600 in get_bigint_attr_from_template ()
  from /usr/lib/security/pkcs11_softtoken.so
#2  0xfea9fe80 in soft_build_secret_key_object ()
  from /usr/lib/security/pkcs11_softtoken.so
#3  0xfeaa0ea4 in soft_build_object ()
  from /usr/lib/security/pkcs11_softtoken.so
#4  0xfeaa5474 in soft_add_object () from 
/usr/lib/security/pkcs11_softtoken.so
#5  0xfea97c64 in C_CreateObject () from 
/usr/lib/security/pkcs11_softtoken.so
#6  0xfe9b5de4 in init_key_uef () from /usr/lib/gss/mech_krb5.so
#7  0xfe9b4d34 in krb5_c_make_random_key () from /usr/lib/gss/mech_krb5.so
#8  0xfe9f19d8 in krb5_generate_subkey () from /usr/lib/gss/mech_krb5.so
#9  0xfe9f472c in krb5int_generate_and_save_subkey ()
  from /usr/lib/gss/mech_krb5.so
#10 0xfe9f4960 in krb5_mk_req_extended () from /usr/lib/gss/mech_krb5.so
#11 0xfe9a5098 in make_ap_req_v1 () from /usr/lib/gss/mech_krb5.so
#12 0xfe9a55c8 in new_connection () from /usr/lib/gss/mech_krb5.so
#13 0xfe9a5f8c in krb5_gss_init_sec_context () from 
/usr/lib/gss/mech_krb5.so
#14 0xfe9a2bbc in k5glue_init_sec_context () from /usr/lib/gss/mech_krb5.so
#15 0xfec84a4c in gss_init_sec_context () from /usr/lib/libgss.so.1
#16 0xfecc32f4 in gssapi_client_mech_step () from /usr/lib/sasl/gssapi.so.1
#17 0xfed173e4 in sasl_client_step () from /usr/lib/libsasl.so.1
#18 0xfed172b0 in sasl_client_start () from /usr/lib/libsasl.so.1
#19 0xfef1737c in nsldapi_sasl_do_bind () from /usr/lib/libldap.so.5
#20 0xfef17afc in ldap_sasl_interactive_bind_s () from /usr/lib/libldap.so.5
#21 0xfef6e478 in doSASLBind () from /usr/lib/libsldap.so.1
#22 0xfef6c880 in openConnection () from /usr/lib/libsldap.so.1
#23 0xfef6bcdc in makeConnection () from /usr/lib/libsldap.so.1
#24 0xfef6d234 in getConnection () from /usr/lib/libsldap.so.1
#25 0xfef6d38c in __s_api_getConnection () from /usr/lib/libsldap.so.1
#26 0xfef60048 in get_current_session () from /usr/lib/libsldap.so.1
#27 0xfef60f24 in search_state_machine () from /usr/lib/libsldap.so.1
#28 0xfef62158 in ldap_list () from /usr/lib/libsldap.so.1
#29 0xfef62264 in __ns_ldap_list () from /usr/lib/libsldap.so.1
#30 0xfefbbe88 in _nss_ldap_lookup () from /usr/lib/nss_ldap.so.1
#31 0xfefba124 in getbyuid () from /usr/lib/nss_ldap.so.1
#32 0xff0edb08 in nss_search () from /lib/libc.so.1
#33 0xff0da488 in getpwuid_r () from /lib/libc.so.1
#34 0x00030e68 in get_current_user_info ()
#35 0x000310a8 in shell_initialize ()
#36 0x0002f158 in main ()
(gdb) continue
Continuing.

Breakpoint 4, 0x000ba484 in malloc ()
(gdb) where
#0  0x000ba484 in malloc ()
#1  0xfe9fece8 in krb5_copy_keyblock () from /usr/lib/gss/mech_krb5.so
#2  0xfe9f4768 in krb5int_generate_and_save_subkey ()
  from /usr/lib/gss/mech_krb5.so
#3  0xfe9f4960 in krb5_mk_req_extended () from /usr/lib/gss/mech_krb5.so
#4  0xfe9a5098 in make_ap_req_v1 () from /usr/lib/gss/mech_krb5.so
#5  0xfe9a55c8 in new_connection () from /usr/lib/gss/mech_krb5.so
#6  0xfe9a5f8c in krb5_gss_init_sec_context () from 
/usr/lib/gss/mech_krb5.so
#7  0xfe9a2bbc in k5glue_init_sec_context () from /usr/lib/gss/mech_krb5.so
#8  0xfec84a4c in gss_init_sec_context () from /usr/lib/libgss.so.1
#9  0xfecc32f4 in gssapi_client_mech_step () from /usr/lib/sasl/gssapi.so.1
#10 0xfed173e4 in sasl_client_step () from /usr/lib/libsasl.so.1
#11 0xfed172b0 in sasl_client_start () from /usr/lib/libsasl.so.1
#12 0xfef1737c in nsldapi_sasl_do_bind () from /usr/lib/libldap.so.5
#13 0xfef17afc in ldap_sasl_interactive_bind_s () from /usr/lib/libldap.so.5
#14 0xfef6e478 in doSASLBind () from /usr/lib/libsldap.so.1
#15 0xfef6c880 in openConnection () from /usr/lib/libsldap.so.1
#16 0xfef6bcdc in makeConnection () from /usr/lib/libsldap.so.1
#17 0xfef6d234 in getConnection () from /usr/lib/libsldap.so.1
#18 0xfef6d38c in __s_api_getConnection () from /usr/lib/libsldap.so.1
#19 0xfef60048 in get_current_session () from /usr/lib/libsldap.so.1
#20 0xfef60f24 in search_state_machine () from /usr/lib/libsldap.so.1
#21 0xfef62158 in ldap_list () from /usr/lib/libsldap.so.1
#22 0xfef62264 in __ns_ldap_list () from /usr/lib/libsldap.so.1
#23 0xfefbbe88 in _nss_ldap_lookup () from /usr/lib/nss_ldap.so.1
#24 0xfefba124 in getbyuid () from /usr/lib/nss_ldap.so.1
#25 0xff0edb08 in nss_search () from /lib/libc.so.1
#26 0xff0da488 in getpwuid_r () from /lib/libc.so.1
#27 0x00030e68 in get_current_user_info ()
#28 0x000310a8 in shell_initialize ()
#29 0x0002f158 in main ()
(gdb)  clear malloc
Deleted breakpoint 4
(gdb) continue
Continuing.

Breakpoint 3, 0xfea9fac0 in soft_build_secret_key_object ()
  from /usr/lib/security/pkcs11_softtoken.so
(gdb) clear soft_build_secret_key_object
Deleted breakpoint 3
(gdb) continue
Continuing.
bash-3.2$(gdb)  clear malloc
Deleted breakpoint 4
(gdb) continue
Continuing.

Breakpoint 3, 0xfea9fac0 in soft_build_secret_key_object ()
  from /usr/lib/security/pkcs11_softtoken.so
(gdb) clear soft_build_secret_key_object
Deleted breakpoint 3
(gdb) continue
Continuing.
bash-3.2$



From man page ld(1) (snv_99 box):

....
    -B direct | nodirect

        These options govern direct binding.  -B  direct  estab-
        lishes direct binding information by recording the rela-
        tionship between each symbol reference together with the
        dependency  that  provides  the definition. In addition,
        direct binding information is established  between  each
        symbol reference and an associated definition within the
        object being  created.  The  runtime  linker  uses  this
        information to search directly for a symbol in the asso-
        ciated object rather than to carry out a default  symbol
        search.

        Direct binding information can only  be  established  to
        dependencies  specified  with  the  link-edit. Thus, you
        should use the -z defs  option.  Objects  that  wish  to
        interpose  on  symbols  in  a direct binding environment
        should identify themselves as interposers  with  the  -z
        interpose  option. The use of -B direct enables -z lazy-
        load for all dependencies.

        The -B nodirect option prevents any  direct  binding  to
        the  interfaces offered by the object being created. The
        object being created can continue to  directly  bind  to
        external  interfaces by specifying the -z direct option.
        See Appendix D, Direct Bindings, in Linker and Libraries
        Guide.
.....

    -z interpose

        Marks the object as an interposer. At runtime, an object
        is  identified  as  an explicit interposer if the object
        has been tagged using the -z interpose option. An expli-
        cit  interposer  is  also  established when an object is
        loaded using the LD_PRELOAD environment variable. Impli-
        cit interposition can occur because of the load order of
        objects, however, this implicit interposition is unknown
        to the runtime linker. Explicit interposition can ensure
        that interposition takes place regardless of  the  order
        in which objects are loaded. Explicit interposition also
        ensures that the runtime linker searches for symbols  in
        any  explicit  interposers  when  direct bindings are in
        effect.
.....


Could it be the explanation and a possible solution ?

Serge







> Thanks,
> Serge
>
> > If it's documented that applications on Solaris may no longer link
> > with their own versions of malloc, that's fine -- I can arrange things
> > so that bash doesn't try to use it's internal malloc on Solaris 10
> > and 11.
> >
> > Chet