Re: [Discuss-gnuradio] Software stopping of runaway realtime processes?

[Eric Blossom]

On Fri, Feb 09, 2007 at 11:17:12AM -0800, Dan Halperin wrote:
> Hey,
> 
> I'm hoping someone has run into this problem before... We have a that 
> lab we access remotely, and sometimes the students get a little 
> overzealous in e.g. their rate selection, and the machines become 
> unresponsive. I seem to be having a hard time getting them to stop using 
> realtime mode.

Yep, you can blow your foot off ;)

> 2 Questions:
> 
> 1) Can I somehow disable their ability to use realtime mode? The 
> software fails gracefully in that case, I just can't find anything 
> online about how to make the enable_realtime() call fail.

If they have access to root, it's kind of hard to keep them from
hosing themselves:

  # rm -fr /


In general, the only GNU Radio application I run as root is
tunnel.py.  That's because it opens the the tun interface, and because
I've been too lazy to code up the suggestion below.  Sounds like you
may have a higher motivation level than I do ;)

  In order to run tunnel.py (or other programs using tap/tun) without
  being root, you could implement a small setuid-root wrapper that would
  acquire the CAP_NET_ADMIN capability, drop root, then exec python
  or whatever.

  You'll probably also need a udev rule to set the permissions on
  /dev/net/tun to something that they can open without being root,
  perhaps making it rw for group usrp.  This is similar to the rule that
  we use to enable the USRP to be opened without being root.

  http://gnuradio.org/trac/wiki/UdevConfig


If they're not running as root (or holding CAP_SYS_NICE), the call 
sched_setscheduler (the system call that enables realtime) will fail.

To reduce the likelihood of people wedging the machine accidentally,
probably the easiest thing to do is to add a command line parameter
that is required to be set to enable realtime mode.  I suspect that
would reduce wedging of the machine through the choice of bad
parameters.  Once they've got a set of parameters that do work, they
could, if they thought it useful, pass the "--enable-realtime" flag.

[I wedged a couple of machines in the ORBIT testbed this way a while
back.  It took me a while to figure out what happened.  Why are these
machines crashing???  The machines were less powerful than the
machines I normally ran on, and they couldn't keep up.  Ooops...]

> 2) Can I set up some sort of watchdog process to auto-kill a process 
> that is clearly going to cause the machine to become unresponsive? I'm 
> not even sure how to identify such a process...

The traditional way to handle this problem is with a shell session,
connected via a serial link, running at higher priority than any of
your experiments.  The default real-time priority set by
gr_enable_realtime_scheduling is in the middle of the RT priorities.

You might be able to cook up some kind of "UDP packet of death" that
would kill all python processes belonging to a particular user.  
The question is whether the networking stack will get enough cycles to
deliver the packet.  The serial port trick is known to work.

FWIW, the code that enables real time scheduling is contained in
gnuradio-core/src/lib/runtime/gr_realtime.cc

Eric