Re: Excessive file system usage

[Alan Mead]

Wow, that's a lot. Do you mean that 7GB of space are needed (for, I
guess temporary files)? And you did not observe that previously?

Maybe the devs are familiar with kubernetes; I only know the name. Can
you describe the environment (e.g., OS)? And pspp version? How many
conversions have you observed this behavior?

And you're sure this isn't a kubernetes problem (like it's making
snapshots as it writes the file or something)? I ask because when I
google about this, it looks like there are sharp edges; glancing
through, these don't seem to directly and specifically address the
behavior you're seeing, but it looks like there could be these kinds of
issues with kubernetes and the PSPP devs wouldn't be able to help unless
they knew kubernetes:

https://cntnr.io/whats-eating-my-disk-docker-system-commands-explained-d778178f96f1
https://softwareengineeringdaily.com/2019/01/11/why-is-storage-on-kubernetes-is-so-hard/

-Alan


On 12/4/2019 6:40 AM, Dave Trollope wrote:
> We just moved Pspp to Kubernetes containers where we use it to extract csvs 
> from sav files. The sav files are about 1gb and each csv is about 150mb.
>
> We’ve watched the file system as it does it and over 7gb of the file system 
> is used while writing 150mb. I assume the SAVE command is doing lots of seeks 
> and insertions in the file magnifying the file system usage. Any options to 
> limit this behavior?
>
> Here is the script we are using
> GET FILE = "{}"
>
> SAVE TRANSLATE
>   /OUTFILE="{}"
>   /TYPE=CSV
>   /FIELDNAMES
>   /REPLACE
>   /KEEP={}
>   /MISSING=RECODE
>   /CELLS=LABELS.
> Cheers
> Dave
>

-- 

Alan D. Mead, Ph.D.
President, Talent Algorithms Inc.

science + technology = better workers

http://www.alanmead.org

The irony of this ... is that the Internet is
both almost-infinitely expandable, while at the
same time constrained within its own pre-defined
box. And if that makes no sense to you, just
reflect on the existence of Facebook. We have
the vastness of the internet and yet billions
of people decided to spend most of them time
within a horribly designed, fake-news emporium
of a website that sucks every possible piece of
personal information out of you so it can sell it
to others. And they see nothing wrong with that.

-- Kieren McCarthy, commenting on why we are not 
                    all using IPv6