Re: Tablicious in Octave Forge [was: Re: [GSoC 2021] How should I do now with project Table datatype]

[Andrew Janke]

On 3/11/21 2:59 PM, John W. Eaton wrote:
> On 3/9/21 3:58 AM, Andrew Janke wrote:
> 
>> Even if table arrays make it in to core Octave from the GSoC work, I
>> think Tablicious could be useful as a transitional package or to
>> support older versions of Octave (which I think are still in kind-of
>> wide use?). And it also provides datetimes, categoricals, and
>> (half-assed) string arrays.
> 
> If you want to create a forge package that would be great, but I'd also
> be glad to include these classes in core Octave and can offer some help
> with that if you are interested.
>
> jwe

Oh, good! I am definitely interested. My goal for Tablicious has always
been to build this into something which I can contribute up to core
Octave, because:
  a) IMHO this stuff is quite useful for modern data-sciencey stuff and
maybe all Octave users should get it,
  b) Some of the semantics of Tablicious really need to be implemented
as core Octave language features, not just a library, and
  c) This is all stuff that is in base Matlab in recent versions, so for
feature parity and compatibility, it should end up in core Octave too.

I don't think that Tablicious is something that is ready for absorption
in to core Octave yet, because the code quality is not good enough yet
(seriously), and there are some fundamental semantic issues that need to
be worked out. In particular, I think we might need to sort out:
  * Whether and how core Octave wants to implement `missing` semantics
for its datatypes.
  * Whether my "proxy key" approach for tables is a good one, and one
that core Octave wants to adopt, and whether this should be done all in
terms of M-code functions or if there should be built-in support added
for it.
  * How Octave as a whole wants to deal with time zone definitions, the
Olson database, updates to it, interaction with OS-provided time zone
definitions, and so on.
  * Aaaaaand string arrays and the long-term plan for double-quoted
string literals in Octave. (Because handling cells in table columns is
really a pain.)

This probably means that I should actually write some decent
documentation about Tablicious's table join semantics and the "proxy
key" technique.

Also, time zone support in Tablicious is currently definitely broken,
and not good enough for inclusion in core Octave yet.

My intention for having Tablicious included in Octave Forge is just a
way of getting the word out about Tablicious to new users, since I think
Octave Forge is probably the first place most Octave users go to find
out what extension packages are available for it? Whether Tablicious
makes it in to Forge or not, my long term hope is for all this to end up
in core Octave, and the Tablicious package is just a proof-of-concept
and support-for-old-Octave versions thing.

There's one hitch: since starting the Tablicious project, I've been
exposed to MathWorks' intellectual property around the datetime class,
so I can probably no longer do any work on the datetime and maybe time
zone stuff in Tablicious, and that probably needs to be handed over to
somebody else. But that's fine: the semantics of datetime are
well-established, and don't interact in complex ways with table array
semantics; it can be cleanly separated.

In the short term, I'm going to try to figure out this new Forge package
thing, because I'd really like the extra eyeballs on my package, and I
think integration into core Octave is a ways off. Maybe the upcoming
GSoC will help.

Cheers,
Andrew