bug#40555: 27.0.90; out of bound array access in setup_process_coding_systems

[Matthieu Hauglustaine]

Hello,

I've experienced a EXC_BAD_ACCESS when using Emacs 27.0.90 on OS X
10.15.

The root cause appears to be an out of bound access on
proc_decode_coding_system (src/process.c:7988), in
setup_process_coding_systems() when calling setup_coding_system(). This
results in an invalid write to coding->id from
CHECK_CODING_SYSTEM_GET_ID (src/coding.c:5678). [1] for the stacktrace.

On Emacs initialization (init_process_emacs(), src/emacs.c:8234),
RLIMIT_NOFILE.rlim_cur is set to FD_SETSIZE, and the assumption seem to
be that this limit will never change for the lifetime of the
process. proc_decode_coding_system and proc_encode_coding_system are
declared with a size of FD_SETSIZE (src/process.c:311).

However, on OS X systems, the call to NSURL.getResourceValue:forKey:
(src/nsfns.c:497), when opening a file, apparently result in a call to
setrlimit with RLIMIT_NOFILE.rlim_cur > FD_SETSIZE.

Thus, when the number of FDs opened by Emacs is greater than FD_SETSIZE,
an illegal access is done when make-process is called.

I've experienced this consistently when using lsp-mode along with
lsp-java, as by default this mode will set kqueue watches on all
directories of the workspace, which can result in a lot of open
files. Fkqueue_add_watch relies on RLIMIT_NOFILE.rlim_cur to limit the
number of watches created, and not on FD_SETSIZE
(src/kqueue.c:391). Disabling file watching works as a workaround.

To reproduce, without lsp, I have been doing the following:

1. Run Emacs with lldb:

    % lldb Emacs-x86_64-10_14
    (lldb) target create "Emacs-x86_64-10_14"
    Current executable set to 
'/Applications/Emacs.app/Contents/MacOS/Emacs-x86_64-10_14' (x86_64).
    (lldb) run -Q --dump-file=/Applications/Emacs.app/Contents/MacOS/Emacs.pdmp
    Process 50395 launched: 
'/Applications/Emacs.app/Contents/MacOS/Emacs-x86_64-10_14' (x86_64)

2. Assert RLIMIT_NOFILE.rlim_cur is at FD_SETSIZE (1024):

    (lldb) expression struct rlimit { uint64_t cur, max; } $lims
    (lldb expression (void)getrlimit(8, &$lims)
    (lldb) memory read &$lims -s 8 -c 2 -fu
    0x100ec5190: 1024
    0x100ec5198: 9223372036854775807

3. Open a file. If a breakpoint on setrlimit was set, we can see a call
   is made, [2] for the stacktrace.

4. Observe RLIMIT_NOFILE.rlim_cur has been changed:

    (lldb) expression (void)getrlimit(8, &$lims)
    (lldb) memory read &$lims -s 8 -c 2 -fu
    0x100ec5190: 3328
    0x100ec5198: 9223372036854775807

5. Create > FD_SETSIZE file descriptors (I've been using
   file-notify-add-watch, as lsp-mode does):

    (require 'filenotify)
    (dotimes (_ 1900)
      (let ((dir (make-temp-file "foo" t)))
        (file-notify-add-watch dir '(change) 'prin1)))

6. Assert process has > FD_SETSIZE open files:

    % lsof -p 50395|wc -l
    1958

7. Call make-process, resulting in SEGV:

    (make-process :name "test" :buffer "test" :command '("echo" "hello"))

    * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS 
(code=1, address=0xffffffffffffffff)
        frame #0: 0x000000010007a1b5 Emacs-x86_64-10_14`setup_coding_system + 53
    Emacs-x86_64-10_14`setup_coding_system:
    ->  0x10007a1b5 <+53>: movq   %rax, (%r15)
        0x10007a1b8 <+56>: testq  %rax, %rax
        0x10007a1bb <+59>: jns    0x10007a1e2               ; <+98>
        0x10007a1bd <+61>: movq   %rbx, %rdi

I would love to help out with a patch. I'm am, however, unfamiliar with
Emacs' internals and uncertain about the best course of action.

I have the following ideas:

1. Report an error in create_process (src/process.c:2033) if in/out
   channel FDs > FD_SETSIZE.

2. Use min(FD_SETSIZE, RLIMIT_NOFILE.rlim_cur) as maxfd in
   Fkqueue_add_watch, src/kqueue.c:390. Although there is probably
   other scenarios in which the Emacs process end up having more than
   FD_SETSIZE open file descriptors?

3. Ensure RLIMIT_NOFILE.rlim_cur is at FD_SETSIZE periodically, in case
   some subroutine we don't know about changed it (where could that
   happen?  Should it be in command_loop, src/keyboard.c:1042?).

Thanks,
Matthieu


In GNU Emacs 27.0.90 (build 1, x86_64-apple-darwin18.7.0, NS appkit-1671.60 
Version 10.14.6 (Build 18G95))
of 2020-03-03 built on builder10-14.porkrind.org
Windowing system distributor 'Apple', version 10.3.1894
System Description:  Mac OS X 10.15.3

Configured using:
'configure --with-ns '--enable-locallisppath=/Library/Application
Support/Emacs/${version}/site-lisp:/Library/Application
Support/Emacs/site-lisp' --with-modules'

Configured features:
NOTIFY KQUEUE ACL GNUTLS LIBXML2 ZLIB TOOLKIT_SCROLL_BARS NS MODULES
THREADS PDUMPER

Important settings:
value of $LANG: en_FR.UTF-8
locale-coding-system: utf-8


[1] Stacktrace, disassembled code and registers on EXC_BAD_ACCESS

    * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS 
(code=EXC_I386_GPFLT)
        frame #0: 0x000000010007a1b5 Emacs-x86_64-10_14`setup_coding_system + 53
      * frame #1: 0x00000001001916c8 
Emacs-x86_64-10_14`setup_process_coding_systems + 152
        frame #2: 0x0000000100193752 Emacs-x86_64-10_14`create_process + 546
        frame #3: 0x0000000100192dcf Emacs-x86_64-10_14`Fmake_process + 2767
        frame #4: 0x000000010014b52b Emacs-x86_64-10_14`Ffuncall + 843
        frame #5: 0x000000010014b07c Emacs-x86_64-10_14`Fapply + 588
        frame #6: 0x000000010014b52b Emacs-x86_64-10_14`Ffuncall + 843
        frame #7: 0x000000010018ea47 Emacs-x86_64-10_14`exec_byte_code + 1815
        frame #8: 0x000000010014b4c9 Emacs-x86_64-10_14`Ffuncall + 745
        frame #9: 0x000000010014b07c Emacs-x86_64-10_14`Fapply + 588
        frame #10: 0x000000010014b52b Emacs-x86_64-10_14`Ffuncall + 843
        frame #11: 0x000000010018ea47 Emacs-x86_64-10_14`exec_byte_code + 1815
        frame #12: 0x000000010014b4c9 Emacs-x86_64-10_14`Ffuncall + 745
        frame #13: 0x000000010014b07c Emacs-x86_64-10_14`Fapply + 588
        frame #14: 0x000000010014b52b Emacs-x86_64-10_14`Ffuncall + 843
        frame #15: 0x000000010018ea47 Emacs-x86_64-10_14`exec_byte_code + 1815
        frame #16: 0x000000010014b4c9 Emacs-x86_64-10_14`Ffuncall + 745
        frame #17: 0x000000010014b07c Emacs-x86_64-10_14`Fapply + 588
        frame #18: 0x000000010014b52b Emacs-x86_64-10_14`Ffuncall + 843
        frame #19: 0x000000010018ea47 Emacs-x86_64-10_14`exec_byte_code + 1815
        frame #20: 0x000000010014b4c9 Emacs-x86_64-10_14`Ffuncall + 745
        frame #21: 0x000000010014b52b Emacs-x86_64-10_14`Ffuncall + 843
        frame #22: 0x000000010018ea47 Emacs-x86_64-10_14`exec_byte_code + 1815
        frame #23: 0x000000010014b4c9 Emacs-x86_64-10_14`Ffuncall + 745
        frame #24: 0x000000010018ea47 Emacs-x86_64-10_14`exec_byte_code + 1815
        frame #25: 0x000000010014b4c9 Emacs-x86_64-10_14`Ffuncall + 745
        frame #26: 0x000000010018ea47 Emacs-x86_64-10_14`exec_byte_code + 1815
        frame #27: 0x000000010014b4c9 Emacs-x86_64-10_14`Ffuncall + 745
        frame #28: 0x000000010018ea47 Emacs-x86_64-10_14`exec_byte_code + 1815
        frame #29: 0x000000010014b4c9 Emacs-x86_64-10_14`Ffuncall + 745
        frame #30: 0x000000010018ea47 Emacs-x86_64-10_14`exec_byte_code + 1815
        frame #31: 0x000000010014c355 Emacs-x86_64-10_14`funcall_lambda + 773
        frame #32: 0x000000010014b4c9 Emacs-x86_64-10_14`Ffuncall + 745
        frame #33: 0x000000010014b52b Emacs-x86_64-10_14`Ffuncall + 843
        frame #34: 0x000000010018ea47 Emacs-x86_64-10_14`exec_byte_code + 1815
        frame #35: 0x000000010014b4c9 Emacs-x86_64-10_14`Ffuncall + 745
        frame #36: 0x0000000100144b19 Emacs-x86_64-10_14`Ffuncall_interactively 
+ 73
        frame #37: 0x000000010014b52b Emacs-x86_64-10_14`Ffuncall + 843
        frame #38: 0x0000000100146008 Emacs-x86_64-10_14`Fcall_interactively + 
5336
        frame #39: 0x000000010014bf61 Emacs-x86_64-10_14`funcall_subr + 289
        frame #40: 0x000000010014b52b Emacs-x86_64-10_14`Ffuncall + 843
        frame #41: 0x000000010018ea47 Emacs-x86_64-10_14`exec_byte_code + 1815
        frame #42: 0x000000010014b4c9 Emacs-x86_64-10_14`Ffuncall + 745
        frame #43: 0x000000010014bb9c Emacs-x86_64-10_14`call1 + 44
        frame #44: 0x00000001000c3249 Emacs-x86_64-10_14`command_loop_1 + 2009
        frame #45: 0x0000000100149b87 
Emacs-x86_64-10_14`internal_condition_case + 263
        frame #46: 0x00000001000d3120 Emacs-x86_64-10_14`command_loop_2 + 48
        frame #47: 0x00000001001493ab Emacs-x86_64-10_14`internal_catch + 267
        frame #48: 0x0000000100215385 Emacs-x86_64-10_14`command_loop.cold.1 + 
69
        frame #49: 0x00000001000c2073 Emacs-x86_64-10_14`command_loop + 131
        frame #50: 0x00000001000c1fa3 Emacs-x86_64-10_14`recursive_edit_1 + 115
        frame #51: 0x00000001000c21fb Emacs-x86_64-10_14`Frecursive_edit + 347
        frame #52: 0x00000001000c0dd7 Emacs-x86_64-10_14`main + 7431
        frame #53: 0x00007fff6c9c97fd libdyld.dylib`start + 1
        frame #54: 0x00007fff6c9c97fd libdyld.dylib`start + 1

    Emacs-x86_64-10_14`setup_coding_system:
    ->  0x10007a1b5 <+53>: movq   %rax, (%r15)
    0x10007a1b8 <+56>: testq  %rax, %rax
    0x10007a1bb <+59>: jns    0x10007a1e2               ; <+98>
    0x10007a1bd <+61>: movq   %rbx, %rdi

    General Purpose Registers:
    rax = 0x0000000000000022
    rbx = 0x000000000000d0b0
    rcx = 0x0000000000000220
    rdx = 0x00000000000004bf
    rdi = 0x000000000000d0b0
    rsi = 0x000000000000d0b0
    rbp = 0x00007ffeefbfe5d0
    rsp = 0x00007ffeefbfe5a0
    r8 = 0x0000000103b235b4
    r9 = 0x00007ffeefbfe687
    r10 = 0x0000000000000018
    r11 = 0x0000000000000202
    r12 = 0x0000000100679c40  Emacs-x86_64-10_14`proc_decode_coding_system
    r13 = 0x00007ffeefbfe601
    r14 = 0x000000010067e698  Emacs-x86_64-10_14`Vcoding_system_hash_table
    r15 = 0xffffffffffffffff
    rip = 0x000000010007a1b5  Emacs-x86_64-10_14`setup_coding_system + 53
    rflags = 0x0000000000010246
    cs = 0x000000000000002b
    fs = 0x0000000000000000
    gs = 0x0000000000000000

[2] Stacktrace of call to setrlimit resulting in RLIMIT_NOFILE.rlim_cur
    greater than FD_SETSIZE

    * thread #1, queue = 'com.apple.clouddocs.xpc', stop reason = breakpoint 1.1
      * frame #0: 0x00007fff6cb0c4cd libsystem_kernel.dylib`setrlimit
        frame #1: 0x00007fff37b22162 Foundation`+[NSFileHandle initialize] + 200
        frame #2: 0x00007fff6b65d985 
libobjc.A.dylib`CALLING_SOME_+initialize_METHOD + 17
        frame #3: 0x00007fff6b65e2bc libobjc.A.dylib`initializeNonMetaClass + 
638
        frame #4: 0x00007fff6b65e991 
libobjc.A.dylib`initializeAndMaybeRelock(objc_class*, objc_object*, 
mutex_tt<false>&, bool) + 214
        frame #5: 0x00007fff6b6503db libobjc.A.dylib`lookUpImpOrForward + 969
        frame #6: 0x00007fff6b64fb99 libobjc.A.dylib`_objc_msgSend_uncached + 73
        frame #7: 0x00007fff3549e353 CoreFoundation`-[__NSSetI containsObject:] 
+ 156
        frame #8: 0x00007fff37b1ab04 
Foundation`setProtocolMetadataWithSignature + 416
        frame #9: 0x00007fff37b1a8ad Foundation`setProtocolMetdataWithMethods + 
699
        frame #10: 0x00007fff37b1a571 Foundation`setProtocolMetadata + 214
        frame #11: 0x00007fff37b1a40e Foundation`-[NSXPCInterface setProtocol:] 
+ 299
        frame #12: 0x00007fff4acee054 CloudDocs`__BRCXPCInterface_block_invoke 
+ 101
        frame #13: 0x00007fff6c97050e 
libdispatch.dylib`_dispatch_client_callout + 8
        frame #14: 0x00007fff6c971686 libdispatch.dylib`_dispatch_once_callout 
+ 20
        frame #15: 0x00007fff4acedfec CloudDocs`BRCXPCInterface + 45
        frame #16: 0x00007fff4aceca41 CloudDocs`-[BRDaemonConnection 
_setupAndResume] + 111
        frame #17: 0x00007fff4acedf9b CloudDocs`-[BRDaemonConnection 
initUsingUserLocalDaemon] + 104
        frame #18: 0x00007fff4acedef7 CloudDocs`__39+[BRDaemonConnection 
defaultConnection]_block_invoke + 83
        frame #19: 0x00007fff6c97050e 
libdispatch.dylib`_dispatch_client_callout + 8
        frame #20: 0x00007fff6c97c567 
libdispatch.dylib`_dispatch_lane_barrier_sync_invoke_and_complete + 60
        frame #21: 0x00007fff4acede6c CloudDocs`+[BRDaemonConnection 
defaultConnection] + 136
        frame #22: 0x00007fff4acfbf05 CloudDocs`-[BRFrameworkSyncedRootURLCache 
_fetchSyncedRootURLs] + 45
        frame #23: 0x00007fff4acfbeb9 
CloudDocs`__47-[BRFrameworkSyncedRootURLCache syncedRootURLs]_block_invoke + 35
        frame #24: 0x00007fff6c97050e 
libdispatch.dylib`_dispatch_client_callout + 8
        frame #25: 0x00007fff6c97c567 
libdispatch.dylib`_dispatch_lane_barrier_sync_invoke_and_complete + 60
        frame #26: 0x00007fff4acfbe5e CloudDocs`-[BRFrameworkSyncedRootURLCache 
syncedRootURLs] + 140
        frame #27: 0x00007fff4acf2d55 CloudDocs`-[NSURL(BRAdditions) 
_br_isInSyncedLocationStrictly:] + 135
        frame #28: 0x0000000100ec5eaa 
CloudDocsFileProvider`___lldb_unnamed_symbol13$$CloudDocsFileProvider + 81
        frame #29: 0x00007fff3549ffec CoreFoundation`__invoking___ + 140
        frame #30: 0x00007fff3549fe83 CoreFoundation`-[NSInvocation invoke] + 
305
        frame #31: 0x00007fff354d3d7e CoreFoundation`-[NSInvocation 
invokeWithTarget:] + 70
        frame #32: 0x00007fff379c3f82 
FileProvider`-[FPFrameworkOverridesIterator callNextOverrides] + 556
        frame #33: 0x00007fff379c3ced 
FileProvider`-[FPFrameworkOverridesIterator forwardInvocation:] + 81
        frame #34: 0x00007fff3549e889 CoreFoundation`___forwarding___ + 829
        frame #35: 0x00007fff3549e4b8 CoreFoundation`_CF_forwarding_prep_0 + 120
        frame #36: 0x00007fff379c2e23 
FileProvider`FPCFCopyAttributeValuesForItem + 292
        frame #37: 0x00007fff4de231ad 
CoreServicesInternal`CopyFromFileProvider(__CFURL const*, void const*, void 
const**, __CFError**) + 70
        frame #38: 0x00007fff4de230d6 
CoreServicesInternal`ExternalProviderPrepareValues(__CFURL const*, 
__FileCache*, __CFString const* const*, void const**, long, void const*, 
__CFError**) + 358
        frame #39: 0x00007fff4de087e4 
CoreServicesInternal`prepareValuesForBitmap(__CFURL const*, __FileCache*, 
_FilePropertyBitmap*, __CFError**) + 363
        frame #40: 0x00007fff4de04eba 
CoreServicesInternal`_FSURLCopyResourcePropertyForKeyInternal(__CFURL const*, 
__CFString const*, void*, void*, __CFError**, unsigned char) + 221
        frame #41: 0x00007fff35491958 
CoreFoundation`CFURLCopyResourcePropertyForKey + 119
        frame #42: 0x00007fff354a84ff CoreFoundation`-[NSURL 
getResourceValue:forKey:error:] + 110
        frame #43: 0x00000001001f0dcf Emacs-x86_64-10_14`ns_implicitly_set_name 
+ 287
        frame #44: 0x0000000100053531 
Emacs-x86_64-10_14`gui_consider_frame_title + 609
        frame #45: 0x00000001000289fc Emacs-x86_64-10_14`redisplay_internal + 
1324
        frame #46: 0x00000001000c6be5 Emacs-x86_64-10_14`read_char + 2213
        frame #47: 0x00000001000c47aa Emacs-x86_64-10_14`read_key_sequence + 
1722
        frame #48: 0x00000001000c2fac Emacs-x86_64-10_14`command_loop_1 + 1340
        frame #49: 0x0000000100149b87 
Emacs-x86_64-10_14`internal_condition_case + 263
        frame #50: 0x00000001000d3120 Emacs-x86_64-10_14`command_loop_2 + 48
        frame #51: 0x00000001001493ab Emacs-x86_64-10_14`internal_catch + 267
        frame #52: 0x0000000100215385 Emacs-x86_64-10_14`command_loop.cold.1 + 
69
        frame #53: 0x00000001000c2073 Emacs-x86_64-10_14`command_loop + 131
        frame #54: 0x00000001000c1fa3 Emacs-x86_64-10_14`recursive_edit_1 + 115
        frame #55: 0x00000001000c21fb Emacs-x86_64-10_14`Frecursive_edit + 347
        frame #56: 0x00000001000c0dd7 Emacs-x86_64-10_14`main + 7431
        frame #57: 0x00007fff6c9c97fd libdyld.dylib`start + 1