Re: Calling readline() a second or subsequent time and retaining state from previous calls

[Sam Habiel]

I found the solution to the second problem: Somewhere in our code we turn off 
echo on the terminal (I haven't found out where yet), but reverting that turns 
echo back on.

My apologies for the question that had nothing to do with readline.

I still have the memory leak, but will live with it for now.

--Sam

 ---- On Fri, 29 Sep 2023 14:29:05 -0400  Sam Habiel  wrote --- 
 > I fixed the double free() issue by doing rl_free_line_state  BEFORE 
 > rl_save_state. That's fixed. However, I still have a leak of 5 bytes 
 > according to LSAN, which I think is some history structure not getting 
 > freed. Unfortunately LSAN isn't showing me the trace beyond two levels, and 
 > all that says is that it happened in xalloc.c, nothing else.
 > 
 > --Sam
 > 
 >  ---- On Fri, 29 Sep 2023 11:24:53 -0400  Sam Habiel  wrote --- 
 >  > Thank you Chet.
 >  > 
 >  > I proceeded with implementing something that is split up across calls to 
 > readline. It's not perfect, and I am writing to you about the problems I 
 > faced. I totally understand if you think it's crazy and think I should 
 > abandon the effort, but I mostly got it to work.
 >  > 
 >  > The current source code is here: 
 > https://gitlab.com/shabiel/YDB/-/blob/ydb83_readline/sr_unix/readline.c (it 
 > will change a bit in the next few days, but not materially, so you can still 
 > refer to it).
 >  > 
 >  > My approach can be summarized as follows:
 >  > 
 >  > When I receive a SIGUSR1, siglongjmp to code that does the following:
 >  > - save the current state using rl_save_state
 >  > - Copy out the state->buffer and state->prompt
 >  > - NULL out rl_prompt (otherwise, it gets freed by readline on the next 
 > call)
 >  > - free state->buffer
 >  > - free state->prompt
 >  > - Put my buffer and prompt back into the state structure (but I did the 
 > allocation and they won't disappear)
 >  > - unwind the C stack
 >  > - Call readline again with the following hooks
 >  > -- startup hook: restore state using rl_restore_state, and set 
 > rl_already_prompted to 1.
 >  > -- redisplay hook: no-op: The text is already on the screen and we don't 
 > want it displayed again.
 >  > 
 >  > I compiled readline from master with -g -O3 so I can see what's going on.
 >  > 
 >  > Here are the two major issues I faced:
 >  > 1. I want to call rl_free_line_state after saving state, but doing so, 
 > after restoring state, I get a double free warning from ASAN. The undo list 
 > is attempted to be freed again:
 >  > 
 >  > ==68415==ERROR: AddressSanitizer: attempting double-free on 
 > 0x603000001ae0 in thread T0:
 >  >     #0 0x7f5e7c27edb8 in __interceptor_free 
 > (/usr/lib64/libasan.so.4+0xdbdb8)
 >  >     #1 0x7f5e763cb117 in _rl_free_undo_list ../undo.c:111
 >  >     #2 0x7f5e763cb178 in rl_free_undo_list ../undo.c:122
 >  >     #3 0x7f5e763ac32c in readline_internal_teardown ../readline.c:512
 >  >     #4 0x7f5e763ada1a in readline_internal ../readline.c:734
 >  >     #5 0x7f5e763ada1a in readline ../readline.c:387
 >  >     #6 0x7f5e76d2c7ef in readline_read_mval 
 > /home/sam/work/gitlab/YDB/sr_unix/readline.c:428
 >  >     #7 0x7f5e77fa53b7 in op_dmode 
 > /home/sam/work/gitlab/YDB/sr_port/op_dmode.c:126
 >  >     #8 0x7f5e776f0c50  (/home/sam/work/builds/YDB/libyottadb.so+0xfaac50)
 >  > 
 >  > 0x603000001ae0 is located 0 bytes inside of 32-byte region 
 > [0x603000001ae0,0x603000001b00)
 >  > freed by thread T0 here:
 >  >     #0 0x7f5e7c27edb8 in __interceptor_free 
 > (/usr/lib64/libasan.so.4+0xdbdb8)
 >  >     #1 0x7f5e763cb117 in _rl_free_undo_list ../undo.c:111
 >  > 
 >  > previously allocated by thread T0 here:
 >  >     #0 0x7f5e7c27f110 in malloc (/usr/lib64/libasan.so.4+0xdc110)
 >  > 
 >  > 2. If I bypass the readline redisplay, I cannot see any characters being 
 > typed. I tried calling rl_reset_terminal and rl_tty_set_echoing to no avail. 
 > I think it has to do with internal readline state (maybe it thinks my cursor 
 > is somewhere else), but the re-display code is so complex that it's hard for 
 > me to tell.
 >  > 
 >  > Let me know if you have any advice; and if not, that's fine too... I do 
 > know what I am doing is crazy, and maybe when we switch in the future to the 
 > callback interface, things would be easier and I won't need all of these 
 > shenanigans.
 >  > 
 >  > --Sam
 >  > 
 >  > 
 >  >  ---- On Mon, 25 Sep 2023 09:30:44 -0400  Chet Ramey  wrote --- 
 >  >  > On 9/21/23 12:34 PM, Sam Habiel wrote:
 >  >  > > Hello there,
 >  >  > > 
 >  >  > > Quick Intro: We have an interactive CLI for a language called M used 
 > in banking and healthcare... I am working on adding readline integration as 
 > a nice to have feature. We are an open source project located over here: 
 > https://gitlab.com/YottaDB/DB/YDB.
 >  >  > > 
 >  >  > > I sent an earlier message, and I thank Chet for the reply.
 >  >  > > 
 >  >  > > I am dealing with the unpleasant for anybody to handle signals. One 
 > of our signals suspends execution, completely unwinds the C stack (yes, we 
 > have lots of assembly code for that), and then reissues the prompt. An 
 > example will help:
 >  >  > > 
 >  >  > > YDB>write "fo  <<< Receive SIGUSR1
 >  >  > > 
 >  >  > > Once a SIGUSR1 is received, in the background, execution is 
 > suspended, a custom function is executed, and the entire prompt code is 
 > actually processed, but the user sees absolutely nothing of that. In the 
 > example above, the cursor will stay between f and o. Our current code just 
 > sets up all the prompt data structures from a saved structure when the 
 > signal is received.
 >  >  > > 
 >  >  > > Is this possible with readline()? I naively tried rl_save_state() 
 > and rl_restore_state(), but that doesn't seem to work.
 >  >  > > 
 >  >  > 
 >  >  > SIGUSR1 isn't one of the signals readline handles by default, but it 
 > has
 >  >  > enough API functions available so that you can do it in your 
 > application.
 >  >  > 
 >  >  > All the necessary functions are in signals.c. The basic idea is that 
 > you
 >  >  > install your signal handler as usual, call the readline functions you
 >  >  > want to restore the terminal and anything else from the handler when 
 > it's
 >  >  > safe (look at _rl_signal_handler() for examples), then restore before 
 > your
 >  >  > handler returns (look at 
 > rl_cleanup_after_signal/rl_restore_after_signal).
 >  >  > 
 >  >  > If you want to split it up across calls to readline, you'll have to 
 > save
 >  >  > and restore a lot of state. I don't know if the information in
 >  >  > struct readline_state is enough for you -- it depends on your 
 > application
 >  >  > needs -- and it's not really intended to save and restore across calls
 >  >  > to readline(). It's mostly intended as a checkpoint during a single 
 > call,
 >  >  > so the interaction between the initialization readline does on each
 >  >  > invocation and the state restored isn't well exercised.
 >  >  > 
 >  >  > If you want to go further with that approach, you'll need at least a
 >  >  > startup hook to restore the state and do a redisplay before going on
 >  >  > in the second (post-signal) call to readline.
 >  >  > 
 >  >  > Chet
 >  >  > 
 >  >  > -- 
 >  >  > ``The lyf so short, the craft so long to lerne.'' - Chaucer
 >  >  >          ``Ars longa, vita brevis'' - Hippocrates
 >  >  > Chet Ramey, UTech, CWRU    chet@case.edu    
 > http://tiswww.cwru.edu/~chet/
 >  >  > 
 >  >  > 
 >  > 
 >