Re: [Discuss-gnuradio] Constant carrier digital transmission

[Ed Criscuolo]

On 8/16/16 11:28 PM, Inspire Me wrote:
> Hi Ed
>
> Much appreciated. You mentioned latency due to queued flags. This has
> potential to cause us issues. I was wondering if it is possible to build
> a block as a sort of switch that takes input from the standard HDLC
> Framer but executues continuously, ie once every symbol period T=
> n*(1/(9600*4), it would check to see if any items are in the input
> buffer and if not push nFlags out (n being small), boost sleep for on
> n*symbol period and repeat. I know this would be highly inefficient but
> given it will be a very basic block it would only execute for a fraction
> of the symbol time. The other option was to have a seperate thread in
> the block for sending the flags which runs independent of the Work
> function and use a semaphore to enable / disable the sending of flags..
> Your thoughts and insight would be valuable.

Again, I repeat that GnuRadio blocks are timing agnostic.  You cannot
"pace" the bitstream in real-time by the use of "sleeps".

You need to write a block that has NO bitstream inputs, and one
bitstream output. This is known as a "source" block. In addition
it should have a message port for receiving HDLC frames.  As in
the example I showed you, this new block should check the message
port for an HDLC frame, bitstuff it into the output bitstream, and
repeat this until either the message port is empty or the output
buffer is full.  If the message port is empty on the initial entry
to work, output a single FLAG worth of bits and exit, returning a
value of 8.
> Q.I thought I read that the schedular only schedules work function when
> there is input data to process. In other words if no input is available
> then the work function would not be scheduled. Is this correct ? If not
> where can I find more information on the scheduler. I have read material
> available online including (
> https://static.squarespace.com/static/543ae9afe4b0c3b808d72acd/543aee1fe4b09162d0863397/543aee20e4b09162d0863578/1380223973117/gr_scheduler_overview.pdf
> )

The scheduler will schedule a block to run if there is input available
AND the output buffer is available.  That is, all of the immediately
downstream blocks have indicated that they have consumed the data.
This is called backpressure.  In the case of a source block, there
are no inputs, and the block is run whenever the output buffer space is
available.
> Once again thank you for spending time helping.

No problem.  I believe you said you're doing a CubeSat application?
On small satellites, often the best approach to this problem is to
use an HDLC framer chip, which does all this for you and produces
a simple bitstream with flags and frames all in their proper places.
This bitstream is then straightforward to handle and send to a modulator.

One last note.  Keep the discussion on-list so others may benefit
(do a "Reply All" instead of a "Reply", otherwise it just comes to me).

@(^.^)@  Ed

> Kind Regards
> Frank
>
> On 17 August 2016 at 11:59, Ed Criscuolo <address@hidden
> <mailto:address@hidden>> wrote:
>
>     In a GnuRadio block that is derived from a "sync" block, the number of
>     input and output items are the same, so the work function is supplied
>     with a single value called "nitems".  The normal control flow of a
>     typical work function uses a loop that iterates over nitems, taking
>     one item from the input(s) buffer(s) and producing one item into the
>     output buffer. This process repeats until all input items are consumed
>     and all output buffer item spaces are full.  The work function then
>     returns a value of "nitems" to indicate to the scheduler how many items
>     were consumed and produced.
>
>     But it is perfectly legal for the loop in the work function to
>     terminate early, in which case it should return the number of items
>     _actually_ processed.  The scheduler will take care of keeping track
>     of the partial buffers and no data will be lost.  But performance
>     will be reduced.
>
>     GnuRadio blocks are timing agnostic.  They run as fast as they can,
>     and process an entire buffer's worth of items (typically 64K items)
>     per call to the work function.  This is done to improve throughput.
>     Timing only happens when your bitstream meets up with a hardware
>     device, like a USRP. But as long as the _average_ throughput is
>     fast enough to keep up with the hardware, everything is fine.
>     The amount of CPU processing "headroom" determines how far you can
>     degrade a block's average throughput without creating underruns.
>
>     This level of understanding is fundamental to writing any GnuRadio
>     blocks, so you really should take the time to play with it and get
>     to know it.
>
>     Here is a snippet of VERY OLD code for example only. It that always
>     runs "nitems" times. When idle (ie - no hdlc packets available), it
>     produces 64k bits (8k of flags) on every invocation! This results in
>     a large latency when a packet finally does come along.
>     It should be modified to break the loop on the second fifo empty
>     test and return "i" instead of "nitems"
>
>     int
>     sc_hdlc_router_source_b::work (int                       noutput_items,
>                                             gr_vector_const_void_star
>     &input_items,
>                                             gr_vector_void_star
>       &output_items)
>     {
>        unsigned char * outbuf = (unsigned char *) output_items[0];
>        int             i;
>
>        // Loop until the requested number of output stream bytes have
>     been generated
>        for(i=0; i<noutput_items; i++)
>        {
>          if (d_fifo.empty())
>            {
>              // Process a single incoming packet on the tun/tap pseudo
>     network device
>              // by bitstuffing it, encapsulating it in an MPoFR frame,
>     and pushing
>              // it into the bit_buf fifo.
>              encapsulate_incoming_packet();
>            }
>
>          // If bit_buf fifo is still empty, push a FLAG into it
>          if(d_fifo.empty())
>            {
>              push_flag();
>            }
>
>          // Output a bit from the bitbuf in the low-order bit of the
>     output byte
>          outbuf[i] = d_fifo.pop();
>        }
>
>        return noutput_items;
>     }
>
>
>     @(^.^)@  Ed
>
>
>     On 8/16/16 8:12 PM, Inspire Me wrote:
>
>         Hi Ed
>
>         Thank you for the response.
>         I like the idea of the second option. Could please expand the
>         explaination. I am new to gnuradio and don't yet fully grasp how the
>         scheduler works in detail. How do you terminate the work
>         function early
>         and ensure the single 0x7E gets streamed immediately to the
>         modulator so
>         that we don't get under runs. How quickly will the work function
>         be re
>         executed by the scheduler ? As fas as CPU is concerned, I have a
>         machine
>         with 16 cores ( 32 threads). I am not concened if one thread is
>         consumed
>         by this. Could you send me a snippet of the work function you
>         implemented.
>
>         Your assitance is much appreciated.
>
>         Kind Regards
>         Frank
>
>         On 17 August 2016 at 02:00, Ed Criscuolo <address@hidden
>         <mailto:address@hidden>
>         <mailto:address@hidden <mailto:address@hidden>>> wrote:
>
>              Hi Frank,
>
>              I'm the author of the GMSK Spacecraft Groundstation project you
>              referenced.  Been on vacation for the last 2 weeks without
>         internet,
>              so I'm just seeing this.
>
>              All in all, Marcus is giving you good advice.  At the time
>         I wrote
>              it, there were no tags or messages, so everything related to
>              PDU-like behavior had to be implemented in a single  C++
>         block. Now,
>              packets can come in as PDUs, and get converted into a
>         bitstream in a
>              single, much simpler block.
>
>              As for the issue of putting HDLC flag octets out
>         back-to-back as the
>              "idle" pattern between frames, I had tried two different
>         solutions,
>              one of which Marcus touched upon.
>
>              First approach was to simply reduce the buffer size of all
>         blocks by
>              recompiling GnuRadio with the appropriate global constant
>         reset to a
>              smaller value.  This has significant impact on overall
>         performance,
>              but because our data rate was modest (only a few hundred
>         Kbits/sec)
>              and we had some wicked fast hardware (for the times!) we
>         were able
>              to get this to work, reducing the added latency of outbound
>         packets
>              from over ten seconds to only a few seconds. This was OK
>         for our
>              particular requirements, but not for a general purpose
>         system.  It
>              was an easy but VERY brute-force solution.
>
>              Later, I modified the block to to implement a solution
>         similar to
>              Marcus' "Option A".  Every time the block's work function
>         was entered,
>              it checked for an available packet (I was using sockets,
>         but modern
>              GnuRadio would use PDUs and message queues). If a packet
>         was available,
>              it would be bitstuffed and the bitstream written to the output
>              buffer. This would continue until either no more packets were
>              available, or the
>              output buffer reached capacity.  If there were NO packets
>         available
>              when the work function was called, it would put a SINGLE
>         0x7E (that
>              is, a bitstream of eight bits) on the output buffer and
>         terminate
>              the work function early, returning only the numbers of bits
>         actually
>              produced.
>              This is perfectly allowable, but it will result in the
>         block consuming
>              a large amount of CPU when generating idle flags.  At least
>         this
>              solution only effects a single block instead of all the blocks.
>
>              If the resulting CPU usage is too high, it could be
>         reduced, at the
>              cost of adding latency, by outputting 2 or more flags at a
>         time.
>
>              Hope this helps.
>
>              @(^.^)@  Ed
>
>
>              On 8/15/16 7:00 PM, Inspire Me wrote:
>
>                  Hi Nick / Marcus
>
>                  Great discussion, obviously this issue has been solved
>         before, as
>                  mentioned in the 2009 discussion the
>         GMSKSpacecraftGroundstation
>                  project
>                  (
>         https://moo.cmcl.cs.cmu.edu/trac/cgran/wiki/GMSKSpacecraftGroundstation
>         
> <https://moo.cmcl.cs.cmu.edu/trac/cgran/wiki/GMSKSpacecraftGroundstation>
>
>         <https://moo.cmcl.cs.cmu.edu/trac/cgran/wiki/GMSKSpacecraftGroundstation
>         
> <https://moo.cmcl.cs.cmu.edu/trac/cgran/wiki/GMSKSpacecraftGroundstation>>
>
>         <https://moo.cmcl.cs.cmu.edu/trac/cgran/wiki/GMSKSpacecraftGroundstation
>         
> <https://moo.cmcl.cs.cmu.edu/trac/cgran/wiki/GMSKSpacecraftGroundstation>
>
>         <https://moo.cmcl.cs.cmu.edu/trac/cgran/wiki/GMSKSpacecraftGroundstation
>         
> <https://moo.cmcl.cs.cmu.edu/trac/cgran/wiki/GMSKSpacecraftGroundstation>>>
>                  )
>                  seems to have solved it. I acknowledge that gnuradio
>         has moved
>                  on, but I
>                  would like to see how it was solved and possibly adapt
>         it to the
>                  current
>                  gnuradio. *Is there anyway to get access to the code ?*
>         I am
>                  stuck at
>                  the moment.
>
>                  The only idea I have is to impelement a source block
>         with very small
>                  buffers to feed into the Tx Chain and some sort of
>         switch to toggle
>                  between data stream and flag stream based on a message
>         / tag
>                  from up the
>                  chain.
>
>                  Nick can you elaborate on the modification of the
>         Work() function. I
>                  thought this was scheduled only when sufficient data is
>         in the Input
>                  buffer ? Is there a way to have this execute more often
>         or when
>                  no data
>                  exists in the input buffer ? Is there a way to implement a
>                  switch block
>                  that monitors input streams and efectively switches the
>         output
>                  stream
>                  such that it is not blocked waiting for input from one
>         of the
>                  streams ?
>
>                  Once again thank you for the help
>
>                  Kind Regards
>                  Frank
>
>                  On 16 August 2016 at 03:14, Marcus Müller
>                  <address@hidden
>         <mailto:address@hidden>
>         <mailto:address@hidden <mailto:address@hidden>>
>                  <mailto:address@hidden
>         <mailto:address@hidden>
>
>                  <mailto:address@hidden
>         <mailto:address@hidden>>>> wrote:
>
>                       Hi Nick,
>
>
>                       On 15.08.2016 19:19, Nick Foster wrote:
>
>                           This is a pretty familiar problem. A lot of satcom
>                      systems require
>                           continuous transmission when in an idle state
>         so the
>                      receiver can
>                           use slow (i.e., sensitive) frequency and timing
>                      estimator loops.
>                           I'm sorry to say there's no great answer.
>
>                           But you have some options:
>                           a) modify the HDLC block so that when work()
>         is called,
>                      if nothing
>                           is in the msg queue just output some fixed
>         number of
>                      0x7Es and sit
>                           on its thumbs until the next call.
>
>                       'doh! Could've thought of that; makes a lot of
>         sense, here.
>
>                           b) modify whatever block is generating
>         messages so that
>                      it outputs
>                           0x7E frames periodically
>                           c) modify the FPGA (or whatever DUC backend you're
>                      using) to both
>                           do the modulating and send 0x7Es (modulated) when
>                      there's nothing
>                           coming in
>
>                           Each approach has its disadvantages:
>                           a) because Gnuradio buffers can be quite
>         large, you
>                      will incur a
>                           latency penalty as GR (i.e., all of the
>         buffers of all
>                      of the
>                           blocks downstream of the HDLC framer)
>         propagates your
>                      frame downstream
>
>                       Not if he adds the functionality to include a
>         tx_time tag; that
>                       wouldn't /reduce/ latency, but it would allow for
>         time-slotted
>                       transmissions!
>
>                           b) message sources don't have backpressure, so
>         this
>                      won't really
>                           work -- either you'll underflow and interrupt your
>                      sample stream
>                           or build up a huge stack of unsent data just
>         as in a)
>                           c) this is a lot of work, and requires that
>         the whole
>                      modulator,
>                           soup-to-nuts, is implemented in the FPGA.
>
>                           I'd start with approach a) and work your way
>         towards c)
>                      if your
>                           application can't handle the latency.
>
>                           --n
>
>                       Best regards,
>                       Marcus
>
>                       _______________________________________________
>                       Discuss-gnuradio mailing list
>         address@hidden <mailto:address@hidden>
>         <mailto:address@hidden <mailto:address@hidden>>
>                  <mailto:address@hidden
>         <mailto:address@hidden>
>         <mailto:address@hidden <mailto:address@hidden>>>
>         https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>         <https://lists.gnu.org/mailman/listinfo/discuss-gnuradio>
>
>         <https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>         <https://lists.gnu.org/mailman/listinfo/discuss-gnuradio>>
>
>           <https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>         <https://lists.gnu.org/mailman/listinfo/discuss-gnuradio>
>
>         <https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>         <https://lists.gnu.org/mailman/listinfo/discuss-gnuradio>>>
>
>
>
>
>                  _______________________________________________
>                  Discuss-gnuradio mailing list
>         address@hidden <mailto:address@hidden>
>         <mailto:address@hidden <mailto:address@hidden>>
>         https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>         <https://lists.gnu.org/mailman/listinfo/discuss-gnuradio>
>
>         <https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>         <https://lists.gnu.org/mailman/listinfo/discuss-gnuradio>>
>
>
>
>
>
>
>     _______________________________________________
>     Discuss-gnuradio mailing list
>     address@hidden <mailto:address@hidden>
>     https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>     <https://lists.gnu.org/mailman/listinfo/discuss-gnuradio>