Re: [Discuss-gnuradio] Streaming IQ File Compression

[Dave NotTelling]

Juha,

     I'm worried about truncating too far because I could be close to the emitters which would cause a large SNR which (as far as I understand) also means more dynamic range.  Losing bits will chop dynamic range :(  Now, I assume that as long as I am on top of the RX gain I can ensure that I don't get massive SNR signals thus not needing the extra bits.  Please correct me if that's wrong as I'm not 100% sure that's right.

     As for the various parallel compression algorithms, I found a list at http://askubuntu.com/questions/258202/multi-core-compression-tools (about half way down) of tools.  I tried out pbzip2 earlier with `cat largeFile.sc16 | pzip2 -1 | pv > /dev/null` and found that it was only running at ~ 25 MBytes/s which is too slow.  That number is from my i7 4870HD Macbook Pro.  If I run pgzip with `cat largeFile.sc16 | pigz -1 | pv > /dev/null` I get around 120 MBytes/s.  Also, the file was stored in /dev/shm for best results.  I know that my desktop machine which will be doing the recording can run an sc16 file through at around 300-350 MBytes/s but that maxes out the entire machine.  Do you know of any tweaks that can be done to increase the speed with a slight trade off in compression?

Marcus,

     I should probably have stated the problem a little more clearly.  I would like to be able to record 100+ MHz swaths of busy spectrum.  Think 900 MHz in a busy city or LTE, WiFi, 433 MHz, etc.  Just trying to build up a large collection of signals to learn how to demodulate with.  I also hope one day to be able to pull down entire transponders of satellites :D  So, the basic point here is that there will be a lot going on in the spectrum that I am recording.  Oh, and the captures could be several minutes long.  Sometimes I find really interesting signals that look crazy in the waterfall (fosphor) plot so I want to record large files of those to poke at later.

     I'm really intrigued by your idea of storing the log magnitudes.  It will take me a bit to wrap my head around it, but it sounds really interesting :D

Thank you both very much!!

-Dave

On Sat, Jul 16, 2016 at 4:29 PM, Marcus Müller <address@hidden> wrote:

Hello Juha,

idea: if Dave's distribution of amplitudes was a little more benign than the Radar near/far problem, and he would favor full resolution when the signal is weak, but could live with a bit of degradation due to quantization when the signal is strong, what about storing a logarithm of the I and Q magnitudes instead of their linear value? That way, he could get the same weak-signal-resolution with less bits, and only suffer inaccuracies due to quantization for situations where signal is strong, anyway.

Best regards,
Marcus

On 16.07.2016 21:57, Juha Vierinen wrote:

Can you reduce the number of bits that you are using?

With radar signals, the receiver noise most of the time excites only about 8 bits out of 16. Ground clutter or meteor echoes excite nearly all of the bits occasionally, so I can't just truncate to 8 bits. In this case, bzip2 actually does a pretty good job of getting rid of the 8/16 most significant bits that are zero most of the time. Thus, I get a compression ratio close to 50% when using sc16. pbzip2 is a good tool for doing parallel compression on files. 

juha

On Sat, Jul 16, 2016 at 3:59 AM, Dave NotTelling <address@hidden> wrote:

Anyone have experience with streaming 100+ MSPS of 16 bit IQ data through a compression tool and then to disk?  I'd like to be able to take really wide band snaps for several minutes.  Currently that would take up 16 * 2 * SAMP_RATE bits per second.  So, for 200 MSPS that would end up being 800 MBytes/s.  That rate eats up a hard drive pretty quickly.  Running it through gzip by way of pigz was my first thought, but even on an 8 core Intel machine pigz just can't keep up.  I can sustain maybe 300 MBytes/s but that's it.  And I know it's not a hard disk limitation as the M.2 drive I am using can easily sustain 800 MBytes/s of uncompressed data.

Thoughts?

Thank you!

-Dave

_______________________________________________
Discuss-gnuradio mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

_______________________________________________
Discuss-gnuradio mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

_______________________________________________
Discuss-gnuradio mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio