Hi all,
I've been investigating the difficulty of modularizing the message bus so that you can
choose whatever underlying messaging system you prefer (RabbitMQ,
crossbar.io, redis, ivy).
The idea is to make paparazzi easier to integrate with other projects. In my case, I seek to
interface to the bus directly from an android tablet/web browser without going through the
app server.
It took me 1.5 days to successfully get a simple simulation running entirely through redis.
Nothing else so far was tested, like replaying log files. Here's the video:
The final architecture I have in mind for this is a generic pprzbus wrapper library in C,
where you can specify one loadable plugin that implements the actual MQ system. This way,
when running and building you only need to have that one MQ dependency installed. A config
file or environment variable will then be used to select that MQ dependency.
What had to be done to do this?
- Updates to ivy-c, ivy-ocaml and paparazzi code. All makefile link dependencies, module imports,
include files were renamed from ivy to pprzbus.
- ivy-c was cloned, renamed to pprzbus and most functions related to socket handling, regex and other static
functions were removed. init, start, bind and send were reimplemented to use redis as the transport,
the pub/sub mechanisms. The ivybind function had one char * parameter added to be able to filter
messages from a specific sender.
- ivy-ocaml was cloned, renamed and linked against pprzbus and installed as pprzbus-ocaml.
- The paparazzi code had many changes related to linking -lpprzbus instead of -livy/-lglibivy.
- The paparazzi ocaml code was updated to use the Pprzbus module instead of Ivy, but the functionality
was kept the same (except for the added parameter on bind). Also, the same regex's were used to
specify the subscription. This regex is analyzed, but not actually used/executed to do the filtering.
- Hacks and tweaks in the wrapper code mostly to deal with the downsides of using the regex's as a subscription mechanism.
A better API and specifying subscriptions as basic strings help to get rid of these tweaks, but this has a larger
impact on pprz code, because all regex's have to be changed.
* The regex specifies how many arguments are returned from the message layer, which does not promote
generalization / portability. Specifically... telemetry message handling is different from gcs messages
handling.
* The regex's sometimes introduce specific filters on sender.
* The regex is executed against each received message in each process, which eventually becomes CPU
intensive and laggy.
So... now I'd like to know how to move forward with this.
If people are generally positively inclined to this change, then I can consolidate my changes, propose the API,
write the pprzbus wrapper with some plugin modules and update all the process code in paparazzi related to bind and
send. I imagine this can all be done on a new branch, so you can check it out, build and see if nothing broke.
Obviously, there are some gotcha's:
- It would be good to get some help with testing. What's most important that the apps / processes that you use
didn't end up broken. When things are ready for testing, I'd like to ask your support to build and run that through
- If you have processes in a private repo that are not part of paparazzi/paparazzi, those will need to be migrated
as well. This could have an impact on the timeline where this change can migrate into master. The changes I had to
do in paparazzi code (not build system) were very simple however, so it's mostly a boring housekeeping job.
Regards,
Gerard