Problem after removing keybinding for bind -m vi-insert '"jj": "\e\e"'

[Clark Wang]

[STEP 100] # echo $BASH_VERSION
4.4.12(4)-release
[STEP 101] # bind -m vi-insert '"jj": "\e\e"'
[STEP 102] # bind -X
"jj": "\e\e"
[STEP 103] # bind -r jj
[STEP 104] # bind -X
[STEP 105] #     <-- Here when I press j it still waits for about 1 second
to show up.