Re: [announcement] master: SWIG -> Pybind11 (breaks OOT interface)

[Morman, Joshua]

Thank you Marcus for the merge, the introduction, and the warnings :)

For all the Ubuntu users - the GNU Radio Master PPA has been updated with builds pulled from the latest master, so it now includes the pybind11 bindings, and OOTs will need to start migrating to the new python binding structure.  Only install from this PPA if you want to test the latest master branch and get your OOTs ready for 3.9 (https://wiki.gnuradio.org/index.php/GNU_Radio_3.9_OOT_Module_Porting_Guide). Otherwise, it is better to stay with the gnuradio-releases PPA.   Here is the PPA info for master:

sudo add-apt-repository ppa:gnuradio/gnuradio-master
sudo apt-get update
sudo apt-get install gnuradio

Fedora COPR is not yet up to date, but I will let everyone know when that supports the current master branch with pybind11.

Josh

--------------------------------------------------------------------------------

From: Marcus Müller <mmueller@gnuradio.org>
Date: Thu, Jun 18, 2020 at 2:59 PM
Subject: [announcement] master: SWIG -> Pybind11 (breaks OOT interface)
To: Gnuradio Mailing List <discuss-gnuradio@gnu.org>

Hello, most prolific SDR community to wander this wide globe,

TL;DR: brace yourself. If you're on master, we'll be breaking your OOT.
You don't have to worry about anything if you're using GNU Radio 3.8.

We're close to merging the pybind branch into master – and that means
we're improving things a lot.

Formerly, we used SWIG to generate Python bindings for GNU Radio's C++
classes – a decision made much more than a decade ago[1], when it still
wasn't clear that Python would be the only language we support (SWIG can
generate wrappers for *a lot* of languages).

In what it does, SWIG is pretty impressive: give it some C++ header, and
it will generate the correct binding for python, including generating of
some mapper types, transparent handling of pointers etc. And, we've
really been using it a lot and in a lot of complex usage scenarios. I
honestly think that, together with OpenCV and LLDB, we really might be
the healthiest and largest project to use SWIG in such a wide way. I'm
very thankful SWIG exists!

Now, having said that, SWIG was always kind of a pain to work into our
source tree. The promise of "throwing in unmodified C++ headers, getting
out perfect Python bindings", far too often, turned out to be "after
massaging these headers quite a bit, then adding a bit of SWIG
directives here and there in swig .i files that we have to keep updated
whenever we change header structure"; and every time something broke our
SWIG bindings, be it a bug on our side or an unexpected change in SWIG
semantics, or be it that SWIG stubbornly refuses to use the include
paths we specify on some platforms, we sunk literal working days into
getting things fixed.

So, as awesome as SWIG's automatism are, using it on such a type-diverse
code base, with such a platform-diverse user community, was kind of
liable for a lot of time not spent on developing a better SDR framework,
but fixing tooling.

Thus, Josh took it upon himself to get this sorted out – and replace
SWIG with Pybind11. Now, Pybind11 is pretty different from SWIG. It's
actually pure C++11 code, and just compiles with standard compilers. No
extra tooling required.

It does come at the expense of you having to declare your interface
yourself. That sounds like a regression, but honestly, there's been so
much work put into making this automateable by Josh, for most blocks
that don't do anything "special", you can just let the automatism take
over. If that doesn't work, you'll typically get a compiler error,
telling you exactly what's wrong. To illustrate, I'll attach the code
generated to wrap a simple block below[2].

However, what that means is that we're also changing the way OOTs need
to wrap their C++ to Python so that they can still can work with GNU
Radio C++ objects wrapped to Python. gr_modtool has been updated to have
new stubs, and the aforementioned automatism is available, too. Small
problem: unlike compilation, the code generation does need an extra bit
of tooling, pygccxml to be specific. This is really only necessary when
you wrap a block that's not been wrapped before, not when compiling any
existing module or the GNU Radio source tree.

In terms of compile time: according to my current level of overview, for
proper multicore machines, building the many smaller C++ files tends to
be a bit faster than first running SWIG to generate 100kLOC-files and
then compile these monsters. For single- or dualcore machines, the
opposite tends to be true. It's my duty to remind you that compiling on
a RaspberryPi itself might get even more cumbersome than it currently
is, and that you should prefer the cross-building toolchains of the
operating system you're running on the embedded platform rather than
building directly on it. Especially debian makes that pretty easy.

Best regards,
and the happiest of hacking,

Marcus

[1]
https://lists.gnu.org/archive/html/discuss-gnuradio/2003-03/msg00135.html

[2]

py::class_<pub_sink,
         gr::sync_block,
         gr::block,
         gr::basic_block,
         std::shared_ptr<pub_sink>>(m, "pub_sink", D(pub_sink))
.def(py::init(&pub_sink::make),
         py::arg("itemsize"),
         py::arg("vlen"),
         py::arg("address"),
         py::arg("timeout") = 100,
         py::arg("pass_tags") = false,
         py::arg("hwm") = -1,
         py::arg("key") = "",
         D(pub_sink, make))
.def("last_endpoint", &pub_sink::last_endpoint, D(pub_sink, last_endpoint))
;