Or maybe this is the limit of what can be done in
Emacs, since there is, and always will be, overhead due to converting
C values to Lisp objects.
Not necessary, there are libraries (at least in java/rust) which provide a possibility to pass a generator which can be used to create your data structures on the fly to avoid intermittent structures. This is something that should be considered when looking for C json library.
I don't think I understand how this could help to speed up code,
please explain.
The issue here is not that the library is some code not written by the
Emacs project, because we do the same with text generated by Emacs
Lisp programs. Bugs can happen in both external libraries and in code
written by Emacs programmers.
I am a bit confused, (maybe I am talking nonsense) but if the input is already an emacs lisp string doesn't that mean that we should not decode it since it is already validated? But even if we had to convert it, we can do it once(e. g. apply decode string on the input string) instead of doing that for each string in the JSON and eventually yield performance improvement? For a reference, I have compared the performance of read-from-string with the json equivalent and it is ~3 times faster(if memory serves me right).
Thanks,
Ivan