I've managed to get the GUI working a little better and have tried plotting. I notice that the plot is coming up as gnuplot X11 window. I checked, and gnuplot now has a 'qt' terminal option that I've compiled. Given that my system is all set up for Qt 4.7 now, all I had to do is
./configure --enable-qt
when building gnuplot. I've run gnuplot, selected the qt terminal and run the demos. Fonts and lines have good antialiasing properties. The images scale pretty well. Large meshes can be somewhat slow (but there might be a way to control Qt graphics speed according to the documentation).
Well, not if you choose OpenGL (currently with an FLTK frame) for plotting. There are two ways of drawing graphics in a Qt window: Either via a simple, but slow QPainter API, and then you can get a bare (or multiple) OpenGL "screens" that you can draw on with just OpenGL commands (similar to the FLTK solution we have now). Drawing directly with OpenGL is much faster, even large plots should not be a problem, somewhat depending on your graphics card of course, but with a minimum overhead. Personally, I think on the large run we should drop support for gnuplot, unless gnuplot does not provide a C++ API for rendering with OpenGL and we can link against it. This would be great, too.
The advantage with gnuplot qt terminal is that it has the same look and feel as your GUI-O. The menubar might not contain what you'd like for the GUI-O plots, but there may be a way to work with that. In gnuplot X11 term is a feature where one can specify an X11 window ID as a container. That is, create some X11 app external to gnuplot to serve as a container, then set term x11 with an X window ID and gnuplot will plot into that window without all the added X stuff. qt terminal appears to have a widget ID; my first guess would be that it is a similar sort of thing. That means you should be able to create the layout for a GUI-O plot and pass in some widget for gnuplot to use.
Whatever platform GUI-O is built on will have the Qt environment all set to go so one can easily build gnuplot with qt support.
The only issue is being able to issue Octave a command to switch the terminal from x11/aqua/win to qt from the Octave command line. I know that sort of thing is done internally in order to print, but external access may be hidden from the user now. (I don't think we want Octave without GUI messing with Qt terminal.