GNU Radio PDU Utilities
This GNU Radio module contains tools for manipulation of PDU objects. There are blocks to translate between streams and PDUs while maintaining timing information, a number of self-explanatory blocks that emulate the behavior of some in-tree stream blocks for messages and PDUs, and some other features. This module is complimentary to the gr-timing_utils module and some of the advanced timing features require blocks there.
All blocks are written in C++, and make use of the GR_LOG API, there is no stdout output; there is minimal PMT symbol interning at runtime, and blocks should be thread safe. QA code exists for most blocks, and when bugs are fixed QA is updated to catch the errors. The general concept of this module, and usage for some of the more unique blocks are described in this document.
General Concept of PDU Conversion with the PDU Utils Module
The general concept of PDU conversion using this module is shown below:
The start and end of a burst is tagged via some means (see below) and the Tags to PDU block will emit a PDU with the PMT vector as the data between the tags. Some basic metadata is provided, and advanced options exist for timing reasons. Once turned into a PDU, the data can be processed, and if desired the data can be turned back into a stream with tx_sob and tx_eob tags for transmission with a UHD-style sink.
Advanced timing modes for RX are included that allow for coarse timing (+/- a couple symbols) which is sufficient for most communications applications. This timing is not intended for precision timing, rather as an option for relatively good timing through rate changes without too much overhead. This is particularly well suited for low-order digital signals; however, the general concept can be used for displaying all sorts of data. If End-of-burst cannot be practically tagged, it is possible to configure the Tags to PDU block to emit a fixed-length PDU and downstream processing can handle the fine details.
UHD-style receive time tags are emitted by the source and recorded by the Tag UHD Offset block which will periodically emit updated UHD-style time tags. The rate at which these are emitted is controllable; which is necessary if any arbitrary-rate-change blocks are present between the source and the Tags to PDU block. Burst tagging works as in the basic mode, and the Tags to PDU block will automatically use the UHD-style time tags to determine burst time. It is important that blocks propagate tags correctly through rate changes. Clock recovery is a step for which this can be complicated; the new (as of 3.7.11) Symbol Sync block is a good option for CR as it propagates tags correctly when the output SPS is set to 1.
Timed transmission is also supported via UHD-style tx_sob and tx_eob tags as shown below.
When the metadata dictionary key tx_time is provided with UHD-style tuple of {uint64 seconds, double fractional seconds}, this value will be added to a tx_time tag on the tx_sob sample. This allows for timed transmission with UHD-style transmit blocks.
Burst Tagging Methods
Conversion between the Streaming and Messaging APIs using the Sandia Utilities require that signals be localized temporally and spectrally in order to be further processed. There are several ways to tag bursts that work well with this type of data that are presented here.
Start and End of Burst Tagging
To identify Start-of-Burst, a very simple way is to do a basic energy detection threshold and tag according to energy level, however this is not very robust. Another straightforward method is to run an open-loop demodulator and use a Correlate Access Code - Tag block to detect the preamble and start of unique word to delineate and tag SOBs. Alternately, a correlation-based method can be used if this is already implemented for preamble detection.
End of Burst tagging can be trickier as there are several ways RF protocols indicate end-of-burst. Generally, the best way is to write a custom block that detects SOB tags, parses the RF header for length information, and automatically tags EOB's accordingly. Sometimes RF protocols will specify a detectable EOB sequence which can be detected with a second Correlate Access Code - Tag block directly. For fixed length bursts an EOB is not necessary and the block can be configured by the maximum length parameter to effectively set the EOB. Example processing flow for this type of tagging is shown below.
Broadband Energy Burst Tagging
SOB/EOB sequence detection requires knowledge of the modulation scheme and details about the time/frequency medium access scheme and the signal protocol framing, and quickly breaks down for signals in which one or more of these features are not known. The FHSS Utilities module provides a different way to isolate bursts based on broadband energy detection. The general processing flow for this model of burst detection is shown below.
In this model, time domain data representing a broadband digitized signal is passed through a processing block which makes time and frequency estimates for energy using the Discrete Fourier Transform and applies tags in the output data stream. The next block isolates these tagged start and end of bursts in time, of which there can be many simultaneously, and emits each individually as a PDU with metadata indicating the approximate center frequency estimation. The final block in this processing chain uses the center frequency estimate to decimate the block in frequency, re-estimate the center frequency, and perform fine frequency correction prior to emitting the signal data that has been time/frequency isolated.
Many such detections can happen simultaneously as would be observed in real-world congested multi-user bands such as the various unlicensed ISM bands.
Additional PDU Tagging Techniques
There are other possibilities for burst tagging that are not included within the Sandia Utilities modules and more tools may be included in the future. It is also not a requirement that PDUs be generated only on bursty signals. Many continuous signals contain frame sync data or other in-band signaling that can be used to break down signals into discrete units that are more suitable for PDU processing. It is also possible for out-of-band information to be processed and used to segment a signal into PDUs. Added functionality will be included for Streaming to Message API conversion in the future as requirements are identified.
Usage of Significant blocks
The usage of several significant blocks are described in this section. Many blocks are omitted as their behavior is straightforward or documented in the GRC XML sufficiently.
GR PDU Utils - Tags to PDU Block
Basic Usage: The Tags to PDU block accepts a stream input and produces a PDU output. The start of a PDU is indicated by a configurable GR stream tag (only the Key matters), and the end of a PDU can be defined by a configurable tag, or by a configurable maximum length; when the maximum length is reached, the PDU will be emitted. If a second start-of-burst tags is received prior to an end-of-burst tag or the maximum length being reached, the block will discard data from the first tag and reset the internal state starting with the latest start-of-burst tag. The block also accepts a Prepend vector argument, which allows for data elements to be included at the start of each PDU. This is useful for byte alignment, or when correlating against a complete or partial Unique Word, and it is desirable to have the complete UW represented in the output. The elements in this vector count toward the Max PDU Size parameter.
Optional Burst Identification Parameters: The Tags to PDU block can be configured to only accept EOB tags in discrete relationships to the SOB tag position through the EOB Alignment and EOB Offset parameters. This will ensure that valid EOB tags are only at n * EOB Alignment item indexes, and the EOB Offset can be used to slew that value if the SOB tag is not suitably located. Additionally, an Tail Size can be specified and that number of items after the EOB tag will be included in the PDU if allowable.
Advanced Timing Features: As described above, this block can be used with UHD-style time tags to provide a reasonably accurate burst timestamp (within a few symbols / bits) with relatively minimum overhead. The key for these tags can be modified but is normally rx_time and the data consist of a two element tuple of uint64 seconds followed by double fractional seconds in range [0, 1). This timing works by knowing the sample rate of the block and keeping track of the last known time-tagged sample and it's offset. Time is then propagated forward assuming the sample rate is exactly precise. As this can drift over time for a variety of reasons, it may be desirable to time-tag samples periodically upstream (e.g.: on burst detections) to improve accuracy and address clock drift, variable block ratios, or dropped samples; the Tag UHD Offset block from the gr-timing_utils module can be used to assist with this. The block also supports the ability to generate boost timestamps in seconds from unix epoch format. This is helpful for debugging but generally less accurate and may carry a greater processing penalty.
Detection Emissions: The block can be configured to emit a message every time a SOB tag is detected. This is useful when a low-latency reaction is necessary to incoming data, though it must be used with caution as it is prone to false detections. The emission is simply a uint64 PMT containing the offset of the received SOB tag.
GR PDU Utils - PDU to Bursts Block
Basic Usage: The PDU to Bursts block accepts PDUs of user-specified type and emits them as streaming data. The original intent of this block was to allow USRP based transmission of data originating from PDU-based processing from data converted to PDUs by the Tags to PDU block for half-duplex transceiver applications. As such, the block will automatically append tx_sob and tx_eob tags around streaming output data to indicate the start and end of valid data to the SDR. The block is simple to use; configuration is limited to type and behavior when new PDUs are received while the data from a current PDU is still being emitted. The data can either be appended to the current burst, dropped, or the block can throw an error ('Balk'). The latter two modes were implemented for very specific cases and generally 'Append' mode is the best choice. The number of PDUs that can queue up waiting for transmission is also configurable to bound memory usage (though the individual PDUs can be large).
Timed Transmissions: The PDU to Bursts block also supports UHD-style timed transmissions. If a PDU metadata dictionary key tx_time exists, and the value is a properly formatted UHD time tuple, a tx_time tag will be added along with the tx_sob tag to the first item in the PDU, which will be recognized as a timed transmission by downstream blocks. Late bursts will be handled according to the behavior of the downstream processing elements, and may be dropped, sent immediately, or potentially errors caused. It is also necessary to be careful with setting timestamps too far in the future as this can result in issues due to backpressure in the DSP chain.
GR PDU Utils - Tag Message Trigger Block
Overview: The Tag Message Trigger block emits PDUs based on certain input conditions observed on either stream or message inputs and supports operating with or without an arming step prior to triggering. This block is more complicated and powerful that it looks, though it has utility in many straightforward applications also. The initial intention of this block was to allow for a stream tag to emit a PDU immediately. This has been expanded upon to support several additional modes of operation which are described here.
Basic Tag Based Message Emission: The most basic mode of this block looks for a tag and emits a message (can be a PDU but does not have to) when it is identified. This was expanded on to allow the concept of a separate 'arming' tag, and an internal armed/disarmed state; the block will only emit a message when armed. If the arming key is set to PMT_NIL, the block is always armed and operates in the simple mode of emitting a message when a key is seen.
Usage of the Arming and Re-trigger Holdoff: The arming status can also be set to automatically disarm after a certain number of samples which is the Holdoff parameter. An internal counter will track samples from the arming event, and a trigger event that occurs after the arming offset + holdoff will have the same behavior as a trigger event when the block is disarmed. The holdoff value will also set the shortest interval for which the triggering action can happen, even when the block is set to always be armed. A second trigger event that is less than holdoff samples from the previous trigger event will be treated as though the block is disarmed. If the holdoff parameter is set to zero it will have no effect.
Timed PDU Mode: The Tag Message Trigger block can operate in two fundamental message modes. In 'Message' mode, the block will emit whatever PMT it has been configured to directly. The alternative, 'Timed PDU' mode allows for the block to add a tx_time metadata dictionary value that is a user definable amount of time in the future. The block will automatically track time of samples in the same way the Tags to PDU block does, and the tx_time key will be a UHD time tuple Delay Time seconds from when the trigger tag was received. This is useful for automatically transmitting a known signal whenever a qualified trigger event is detected.
Transmission Limit: A final feature that is not exposed to the GRC is the concept of transmission limits. This value can be updated by callback, and if it is not set to TX_UNLIMITED it will be decremented each time the block triggers. When it reaches zero, trigger events will be treated as though the block is disarmed.
GR PDU Utils - PDU Pack Unpack Block
Usage: The PDU Pack Unpack block is primarily used to convert between U8 PDUs representing one bit per element, and U8 PDUs representing 8 bits per element. The block can also convert bit order for packed U8 data. The usage is straightforward, specify pack/unpack/reverse and declare the bit order, however it is included in this section due to the usefulness.
This block could also be extended to support other options (higher order modulation packing, conversion to U16, etc) if necessary but a use case as not yet been identified. If this happens appropriate test code should be added to exercise such conditions.
GR PDU Utils - PDU Add Noise Block
Usage: This block can be used to add uniform random values to an input array of uint8, float, or complex data; other PDU types will be dropped with a WARNING level GR_LOG message. This is fairly straightforward; however, it is included here as the block also the non-obvious capability to scale and offset the input data PDU as well. This is done through the following logic:
out[ii] = (input[ii] + ((d_rng.ran1()*2 - 1) * d_noise_level)) * d_scale + d_offset;
Which is to say that the order of operations is to apply the random data first, then scale the data, then offset it. Generally useful for debugging and testing, and as such it has not seen extensive use so there may be some issues. Noise profiles are not supported, only uniform random data from the ran1() function within gr::random. It could be argued that these should be separate blocks entirely, but to reduce the overhead of PMT-ifying and de-PMT-ifying data it was implemented this way to allow all three operations to be done at once. Maybe the name should be changed...
GR PDU Utils - PDU Clock Recovery
Summary: This block performs clock synchronization and symbol recovery on 2-ary modulated data using algorithms from M. Ossmann’s WPCR project. The block accepts soft and unsynchronized data and uses a zero-crossing detector to effectively recover data sampled between 4 and 60 samples per symbol, though it does perform better below 16 samples per symbol. Compared to in-tree options, this block has several advantages, primarily that it operates on PDU formatted data enabling it to work within the Message Passing API. Because the block operates on PDU data, it can make use of the entire packet to aid in data synchronization improving sensitivity. Additionally, the block does not require precise configuration or tuning which results in reduced user-error and increased capability when processing signals for which exact parameters are unknown.
GR PDU Utils - PDU FIR Filter
Summary: This block is a direct analog to the in-tree Decimating FIR streaming filter. It makes use of the same underlying filterNdec function in the from the fir_filter_xxf kernel from gr::filter. The use of this block has uncovered several invalid operations due to the pointer logic used which do not manifest themselves when used with the streaming API but are a problem with the filter kernels in general. Upstream issues have been filed and workarounds built into the blocks.
These blocks do not use FFT based filters as the FFT kernels are not yet templatized. This may be added as an option to this block in the future as the FFT implementation of discrete filters is more efficient for large numbers of taps
GR PDU Utils - PDU PFB Arbitrary Resampler
Summary: This block is a direct analog to the in-tree PFB Arbitrary Resampler streaming block. It makes use of the same pfb_arb_resampler_ccf kernel from gr::filter. This block will reject non-PDU type data, and currently only works on c32 type PDUs; taps must be real valued.
GR PDU Utils - PDU Flow Controller
Summary: The GNU Radio Asynchronous Message Passing API has no concept of flow control or backpressure. A slow block in the processing chain will cause an unbounded backup of messages which can in turn result in software failures as messages are never dropped, and the publish method does not block.
This block will check the message queue size for subscribed blocks, and it will drop messages over a configurable maximum queue size. Dropping data is not always preferred, so this should only be used in situations where data loss is acceptable.
Blocks Marked for Upstreaming
Several blocks in this module are very general in nature and have been marked for upstreaming pending resolution of some process and software organization factors:
-
Message Counter
-
Message Emitter
-
Message Gate
-
Message Keep 1-in-N
-
Pack Unpack
-
PDU Align
-
PDU Binary Tools
-
PDU Clock Recovery
-
PDU Complex to Mag^2
-
PDU FIR Filter
-
PDU GMSK (rename to PDU FM Modulator)
-
PDU Logger
-
PDU PFB Resampler
-
PDU Quadrature Demod
-
PDU Split