[Octave-bug-tracker] [bug #63803] Saving causes OOM, crash, and loss of already saved data

[Arun Giridhar]

URL:
  <https://savannah.gnu.org/bugs/?63803>

                 Summary: Saving causes OOM, crash, and loss of already saved
data
                 Project: GNU Octave
               Submitter: arungiridhar
               Submitted: Mon 13 Feb 2023 07:38:55 PM EST
                Category: Interpreter
                Severity: 4 - Important
                Priority: 5 - Normal
              Item Group: Segfault, Bus Error, etc.
                  Status: None
             Assigned to: None
         Originator Name: 
        Originator Email: 
             Open/Closed: Open
                 Release: dev
         Discussion Lock: Any
        Operating System: GNU/Linux
           Fixed Release: None
         Planned Release: 9.1.0 (current default)


    _______________________________________________________

Follow-up Comments:


-------------------------------------------------------
Date: Mon 13 Feb 2023 07:38:55 PM EST By: Arun Giridhar <arungiridhar>
This is on: GNU Octave Version: 9.0.0 (hg id: 3e4e74ad8fd7)

Prep work: Create a few arrays, save to file, and exit Octave:

adj = ~eye(36);
edges = reshape (1:120, 60, 2);
lst = false (5735500, 60);
ncells = 36;
nedges = 60;
whos
save -binary test.mat


Output is normal:

octave:6> whos
Variables visible from the current scope:

variables in scope: top scope

  Attr   Name         Size                     Bytes  Class
  ====   ====         ====                     =====  ===== 
         adj         36x36                      1296  logical
         edges       60x2                        960  double
         lst    5735500x60                 344130000  logical
         ncells       1x1                          8  double
         nedges       1x1                          8  double

Total is 344131418 elements using 344132272 bytes

octave:7> save -binary test.mat
octave:8> exit


Then restart Octave, load that file, then create a new large variable, and try
to save everything to the same file. I have 16 GB RAM, and I can comfortably
create a 7.5 GB array with no second thought:

load test.mat
whos
arr = false (36, 36, rows(lst));
arr(:, :, 1:2:end) = repmat(mod((1:36)' + (1:36), 2) == 1, 1, 1,
size(arr,3)/2);
arr(:, :, 2:2:end) = ~arr(:, :, 1:2:end);
whos
save -binary test.mat


Output:

octave:6> whos
Variables visible from the current scope:

variables in scope: top scope

  Attr   Name         Size                     Bytes  Class
  ====   ====         ====                     =====  ===== 
         adj         36x36                      1296  logical
         arr         36x36x5735500        7433208000  logical
         edges       60x2                        960  double
         lst    5735500x60                 344130000  logical
         ncells       1x1                          8  double
         nedges       1x1                          8  double

Total is 7777339418 elements using 7777340272 bytes

octave:7> save -binary test.mat
Killed


What happened?! Let's restart Octave and see what happened:

octave:1> load test.mat
octave:2> whos
Variables visible from the current scope:

variables in scope: top scope

  Attr   Name        Size                     Bytes  Class
  ====   ====        ====                     =====  ===== 
         adj        36x36                      1296  logical

Total is 1296 elements using 1296 bytes


All the other variables have been lost both from RAM *and from the saved
file*.

This actually happened to me in the last 30 minutes. Fortunately I lost only
12 minutes of work and was able to rerun a script that regenerated everything.
Had it been more important data this could have been BAD!

Raising importance because of the ease and surprise of data loss right in the
act of saving it, even with plenty of memory remaining.

Observations if it helps anyone localize the cause:

* Monitoring the memory usage through top, Octave somehow uses 16 GB RAM + 8
GB swap, runs out of memory, and is killed. It's not clear why the memory use
goes that high for a total of less than 8 GB worth of variables. Looks like
copy-on-write happening where it shouldn't.

* The saving works as intended if the new array "arr" is all-false. But if
it's set to a checkerboard pattern like in the case above, it crashes while
saving. Not sure why the density of the array causes a crash since it's not
even a sparse object, but it empirically does. Maybe something to do with the
way it's created? Copy-on-write problem with ranges?








    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?63803>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/