Re: large arrays - how to store?

[Guy Harrison]

Paul Schneider wrote:

> Hello,
> 
> for my program I need a large multidimensional array the dimensions of
> which I know at compile time. Specifically it is of size 500 x 5000 x 10
> x 6. I am looking for guidance of how to treat this array on a linux
> system with 2 Gigs of memory.
> 
> 1) Should I use stack memory? I have heard that it is hard to detect
> stack overflows, however g++ doesn't give me any warnings with -Wall and
> -fstack-check gives me a seg fault. This is what I'm doing now and the
> program seems to run just fine.

Large stack is least portable and thus most problematic.
 
> 2) Should I declare it static?

Pushes the problem onto (a) the compiler and (b) the linker.
 
> 3) Should I use dynamically allocated memory?

Most portable.

> I will have to use the fourth dimension (the one with 6 elements) in
> lots of Matrix.Vector Vector.Vector products. Ironically I am currently
> using
>
blitz::TinyVector<blitz::TinyVector<blitz::TinyVector<blitz::TinyVector<double,
> 6>, 10>, 5000>, 500>, since I have read that BLAS is slow on small
> vectors and blitz seems to do a good job inlining.

Not familiar with the above.

> Is there a better (faster, safer) way of how to handle such a large
> datastructure?

You can't have fastest & safest at the same time. Safest is a container.
Fastest is access the memory directly. Exactly how that memory ought be
laid out depends on how it is to be accessed. Take a peek at mmap and if
you see mention of malloc then, without further info, you're probably
looking at std::malloc and/or new[] plus some wrappers to make the block
look like a 4 dimensional array. Check also boost - the faster/safer
tradeoff may have a solution there.