|
| From: | Albin Stigö |
| Subject: | Re: [Discuss-gnuradio] Messaging Passing on a Budget |
| Date: | Wed, 29 May 2019 21:41:53 +0200 |
Packaging the json string as a vector of u8 integers worked like a charm, thank you for the suggestion!Memory consumption by the python flowgraph is no longer showing signs of memory leakage (at least in the 4 hours it's been running :)_______________________________________________On Fri, Mar 1, 2019 at 1:50 PM Müller, Marcus (CEL) <address@hidden> wrote:Hi Brad,
so, yes, your observation is correct: PMT symbols are/were meant to be
used as "identifiers", not as "data carriers"; the motivation behind
the hash table you find in pmt.cc is that there's only one instance of
any given PMT symbol, and thus, a simple address comparison suffices to
tell whether two PMT symbols are the same.
You'll notice that on x86 (and presumably modern ARM) string comparison
is pretty fast, and that you'd need to do *a lot* comparisons to offset
the cost of hashing a symbol once. Anyway, yes, that table grows
unboundedly.
Since your string isn't actually a "symbol", I'd recommend simply
encoding it safely (that's probably already the case), and putting it
in a uniform PMT vector of 8bit integers (u8vector).
On a different note, there's actually "unintended" (as in: I don't
intend GNU Radio to have an unbounded hash map, but it's at least what
was originally intended) memory leaks with PMTs and tag_t on the
Python/C++ boundary, and there's quite some broken concepts within PMT.
Long or medium term, we'll be replacing PMT with something that is
actually a common serialization format for usage in external software
(i.e. not for usage within the same flow graph), and ideally with the
unserialized universal container that comes with such a format. Stay
tuned. However, not happening in 3.8 or anything before.
Best regards,
Marcus
On Thu, 2019-02-28 at 14:48 -0500, Brad Hein wrote:
>
> In my gnuradio test application I’m attempting to pass a JSON formatted string from my C++ custom block, to my python flow graph containing the block.
>
> The String message are being received by my python flow graph, but there’s a memory leak. Over time, my flowgraph RSS (memory footprint) grows at about 32k/s up until the application runs out of memory.
>
> The leak didn’t start until after I retrofitted my flowgraph and custom block with polymorphic type (PMT) based message passing. I’m passing a 200 or so byte JSON string (as a symbol) from C++ block to python flow graph about 60 times per second.
>
> Sleuthing through the pmt.cc code [1] I see the String (symbol) objects are stored in a hash. I suspect what is happening is that since all of my JSON strings are unique, they’re getting added to the hash and causing the hash to grow unbounded over time.
>
> From what I can tell by reading the wiki [2], the approach I’m using is the best way to get a string from my custom block and into my python flowgraph.
>
> [1] https://github.com/gnuradio/gnuradio/blob/master/gnuradio-runtime/lib/pmt/pmt.cc
> [2] https://www.gnuradio.org/doc/doxygen/page_pmt.html
>
>
> Sample c++ block code snippet from my work function:
>
> // Calculate metrics. Occasionally (60/second) we'll get back a JSON formatted string of metrics.
> crossingData = calculateWaveCrossingMetrics(lastSampleValue,in[i]);
>
> if (crossingData.length() > 0) {
> pmt::pmt_t messageString;
> messageString = pmt::string_to_symbol(crossingData.c_str());
> message_port_pub(polymorphicMessageDestination, messageString);
>
>
> Questions:
>
> 1. Is this the best way to get a string from my C++ block into my python application?
> 2. Is there a way to pass my messages that doesn’t result in memory growing unbounded?
>
>
> Thank you
> Brad
>
> _______________________________________________
> Discuss-gnuradio mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
Discuss-gnuradio mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
| [Prev in Thread] | Current Thread | [Next in Thread] |