Re: Light weight support for JSON

[Yair Lenga]

First, thanks for taking the time to read and provide your thoughts. This
is the real value of the discussion/

Second: I'm NOT trying to argue that there isn't valid use for
combining bash/curl/jq, Nor do I suggest adding JSON as first class object
to bash (Python/node/Perl/Groovy are way ahead  ...).

I hope to get feedback from other readers for the news group that may find
this approach useful.

I'll take the very common use case of using AWS CLI, which does produces
JSON response for most calls. Processing the response, while possible with
JQ, it is challenging to many junior (and intermediate) developers. In many
cases, they fall into the traps that I mentioned above - performance
(excessive forking or fork/exec), or code that is hard to read (I've seen
really bad code - combining pipes of JQ/awk/sed). I'm trying to address
those cases. Almost always they fail to properly handle objects with while
space, new-lines, etc.

To be practical, I'll try to follow the loadable extension path, and see
how much I can get thru that path. Possibly will make sense to continue
discussion with a concrete implementation. I believe the necessary commands
are:

json_read -a data-array -m meta-array -r root-obj
Parse the stdin input and into data-array with the items (as described
above), and the meta-array with helper information (length, prop list under
each node) - to help iterating.

json_write -v variable -a data-array -m meta-array -r root
The reverse - generate the JSON for associated array following '.' naming
convention

json_add -a data-array [-m meta-array] [-r root] key1=value1 key2=value2
key3=value3
Helper to add items into associative array representing JSON object,
Autodetect type, with the ability to force stringfication using format
string.


On Sun, Aug 28, 2022 at 3:22 PM John Passaro <john.a.passaro@gmail.com>
wrote:

> interfacing with an external tool absolutely seems like the correct answer
> to me. a fact worth mentioning to back that up is that `jq` exists. billed
> as a sed/awk for json, it fills all the functions you'd expect such an
> external tool to have and many many more. interfacing from curl to jq to
> bash is something i do on a near daily basis.
>
> https://stedolan.github.io/jq/
>
> On Sun, Aug 28, 2022, 09:25 Yair Lenga <yair.lenga@gmail.com> wrote:
>
>> Hi,
>>
>> Over the last few years, JSON data becomes a integral part of processing.
>> In many cases, I find myself having to automate tasks that require
>> inspection of JSON response, and in few cases, construction of JSON. So
>> far, I've taken one of two approaches:
>> * For simple parsing, using 'jq' to extract elements of the JSON
>> * For more complex tasks, switching to python or Javascript.
>>
>> Wanted to get feedback about the following "extensions" to bash that will
>> make it easier to work with simple JSON object. To emphasize, the goal is
>> NOT to "compete" with Python/Javascript (and other full scale language) -
>> just to make it easier to build bash scripts that cover the very common
>> use
>> case of submitting REST requests with curl (checking results, etc), and to
>> perform simple processing of JSON files.
>>
>> Proposal:
>> * Minimal - Lightweight "json parser" that will convert JSON files to bash
>> associative array (see below)
>> * Convert bash associative array to JSON
>>
>> To the extent possible, prefer to borrow from jsonpath syntax.
>>
>> Parsing JSON into an associative array.
>>
>> Consider the following, showing all possible JSON values (boolean, number,
>> string, object and array).
>> {
>>     "b": false,
>>     "n": 10.2,
>>     "s: "foobar",
>>      x: null,
>>     "o" : { "n": 10.2,  "s: "xyz" },
>>      "a": [
>>          { "n": 10.2,  "s: "abc", x: false },
>>          {  "n": 10.2,  "s": "def" x: true},
>>      ],
>> }
>>
>> This should be converted into the following array:
>>
>> -------------------------------------
>>
>> # Top level
>> [_length] = 6                            # Number of keys in object/array
>> [_keys] = b n s x o a            # Direct keys
>> [b] = false
>> [n] = 10.2
>> [s] = foobar
>> [x] = null
>>
>> # This is object 'o'
>> [o._length] = 2
>> [o._keys] = n s
>> [o.n] = 10.2
>> [o.s] = xyz
>>
>> # Array 'a'
>> [a._count] =  2                   # Number of elements in array
>>
>> # Element a[0] (object)
>> [a.0._length] = 3
>> [a.0._keys] = n s x
>> [a.0.n] = 10.2
>> [a.0.s] = abc
>> [a.0_x] = false
>>
>> -------------------------------------
>>
>> I hope that example above is sufficient. There are few other items that
>> are
>> worth exploring - e.g., how to store the type (specifically, separate the
>> quoted strings vs value so that "5.2" is different than 5.2, and "null" is
>> different from null.
>>
>> I will leave the second part to a different post, once I have some
>> feedback. I have some prototype that i've written in python - POC - that
>> make it possible to write things like
>>
>> declare -a foo
>> curl http://www.api.com/weather/US/10013 | readjson foo
>>
>> printf "temperature(F) : %.1f Wind(MPH)=%d" ${foo[temp_f]}, ${foo[wind]}
>>
>> Yair
>>
>