Re: [Discuss-gnuradio] Messaging Passing on a Budget

[CEL]

I'd recommend ZeroMQ; it's a pretty universally available socket
wrapper thing that is also "transport invariant", i.e. you can do
ZeroMQ "over the Internet" over classical TCP, just as much as for IPC
on the same machine. Unlike UDP, it can ensure messages actually reach
their target, and it's super nice for writing things like communication
systems where you have a single subscriber attach to multiple
publishers of data.
Also, IIRC, our current TCP sink/source has their own problems :/

Best regards,
Marcus
On Fri, 2019-03-01 at 23:56 -0500, Brad Hein wrote:
> Thanks for the clever insights, Marcus. I'll consider the PMT vector
> approach you mentioned, while at the same time knowing PMT is likely
> to undergo changes in the future as you mentioned, I'll also look
> into using TCP/IP to pass the needed information, perhaps ZeroMQ or a
> UDP socket even. 
> 
> On Fri, Mar 1, 2019 at 1:50 PM Müller, Marcus (CEL) <address@hidden>
> wrote:
> > Hi Brad,
> > 
> > so, yes, your observation is correct: PMT symbols are/were meant to
> > be
> > used as "identifiers", not as "data carriers"; the motivation
> > behind
> > the hash table you find in pmt.cc is that there's only one instance
> > of
> > any given PMT symbol, and thus, a simple address comparison
> > suffices to
> > tell whether two PMT symbols are the same.
> > 
> > You'll notice that on x86 (and presumably modern ARM) string
> > comparison
> > is pretty fast, and that you'd need to do *a lot* comparisons to
> > offset
> > the cost of hashing a symbol once. Anyway, yes, that table grows
> > unboundedly.
> > 
> > Since your string isn't actually a "symbol", I'd recommend simply
> > encoding it safely (that's probably already the case), and putting
> > it
> > in a uniform PMT vector of 8bit integers (u8vector).
> > 
> > On a different note, there's actually "unintended" (as in: I don't
> > intend GNU Radio to have an unbounded hash map, but it's at least
> > what
> > was originally intended) memory leaks with PMTs and tag_t on the
> > Python/C++ boundary, and there's quite some broken concepts within
> > PMT.
> > 
> > Long or medium term, we'll be replacing PMT with something that is
> > actually a common serialization format for usage in external
> > software
> > (i.e. not for usage within the same flow graph), and ideally with
> > the
> > unserialized universal container that comes with such a format.
> > Stay
> > tuned. However, not happening in 3.8 or anything before.
> > 
> > Best regards,
> > Marcus
> > 
> > On Thu, 2019-02-28 at 14:48 -0500, Brad Hein wrote:
> > >  
> > > In my gnuradio test application I’m attempting to pass a JSON
> > formatted string from my C++ custom block, to my python flow graph
> > containing the block.
> > >  
> > > The String message are being received by my python flow graph,
> > but there’s a memory leak. Over time, my flowgraph RSS (memory
> > footprint) grows at about 32k/s up until the application runs out
> > of memory.
> > >  
> > > The leak didn’t start until after I retrofitted my flowgraph and
> > custom block with polymorphic type (PMT) based message passing. I’m
> > passing a 200 or so byte JSON string (as a symbol) from C++ block
> > to python flow graph about 60 times per second.
> > >  
> > > Sleuthing through the pmt.cc code [1] I see the String (symbol)
> > objects are stored in a hash. I suspect what is happening is that
> > since all of my JSON strings are unique, they’re getting added to
> > the hash and causing the hash to grow unbounded over time.
> > >  
> > > From what I can tell by reading the wiki [2], the approach I’m
> > using is the best way to get a string from my custom block and into
> > my python flowgraph.
> > >  
> > > [1] 
> > https://github.com/gnuradio/gnuradio/blob/master/gnuradio-runtime/lib/pmt/pmt.cc
> > > [2] https://www.gnuradio.org/doc/doxygen/page_pmt.html
> > >  
> > >  
> > > Sample c++ block code snippet from my work function:
> > >  
> > > // Calculate metrics. Occasionally (60/second) we'll get back a
> > JSON formatted string of metrics.
> > > crossingData =
> > calculateWaveCrossingMetrics(lastSampleValue,in[i]);
> > >  
> > > if (crossingData.length() > 0) {
> > >   pmt::pmt_t messageString;
> > >   messageString = pmt::string_to_symbol(crossingData.c_str());
> > >   message_port_pub(polymorphicMessageDestination, messageString);
> > >  
> > >  
> > > Questions:
> > >  
> > > 1.     Is this the best way to get a string from my C++ block
> > into my python application?
> > > 2.     Is there a way to pass my messages that doesn’t result in
> > memory growing unbounded?
> > >  
> > >  
> > > Thank you
> > > Brad
> > >  
> > > _______________________________________________
> > > Discuss-gnuradio mailing list
> > > address@hidden
> > > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
> 
> _______________________________________________
> Discuss-gnuradio mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

smime.p7s
Description: S/MIME cryptographic signature