[Octave-bug-tracker] [bug #63791] movmean - out of memory or dimension too large for index type

[Rik]

Follow-up Comment #16, bug #63791 (project octave):

Analyzing the data in comment #11, Octave is probably reporting correctly that
it was unable to allocate memory (rather than index was too large).  Sometimes
a memory allocation fails because there is not enough memory, and sometimes it
fails because there is, in aggregate, enough memory but there is not a
contiguous block of the required size.  Given that 'slcidx' is 4.1GB,
'x(slcidx)' is going to be another 4.1GB.  Available RAM looks like 2.27 GB so
that is the basis of the problem.  The operating system could go in to swap
memory, but I guess in your case it is choosing not to.

There are various solutions to this

1) Average over a smaller number of samples.

Generally I would expect much smaller window lengths.  A moving average is the
equivalent of a low pass filter so often a window size as small as 10 is
enough to get rid of high frequency noise.  Imagine this is financial data,
then you might want to use N=30 to smooth data over a month.  Heck, even if
you decide that you want to get rid of the low frequency signal of seasons you
would average over 365 days which your system would support.  So, my first and
best recommendation is to examine the system you are analyzing and determine
what an appropriate value of window length would be.

2) Use filter().

You will need to review the math for FIR filters, but a moving average of
N-points can be replaced by an N-point FIR filter where each value is 1/N. 
This is what Dmitri demonstrated.  This will be more memory efficient than
movmean().

3) Change hardware or software.

This does seem to be about memory.  And memory is cheap these days.  Your
system seems to have 16 GB of memory.  I can buy 32 GB of memory for $85 on
Amazon
(https://www.amazon.com/Corsair-Vengeance-PC4-25600-Desktop-Memory/dp/B07RW6Z692/ref=sr_1_3?crid=39TXWVWGHFZOP&keywords=32+GB+RAM&qid=1676226955&sprefix=32+gb+ram%2Caps%2C187&sr=8-3).
 Sometimes, it's better not to overthink things and just upgrade your
hardware.  In a similar vein, your operating system is Windows.  If you have
access to a machine running Linux you could just try the calculation there. 
Most of the Octave developers run Linux and we aren't seeing this issue.




    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?63791>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/