long-term browser visualization issues
Closed this issue · 44 comments
First of all I'm not sure how much anyone actually uses the MeqTrees inspector visualization tools for actually looking at visibility data. If no-one does then changes are probably not necessary and I won't waste my time further. However I use this stuff quite a bit and as I recall commenting in some other thread, while the inspectors worked reasonably well with WSRT / VLA sized data sets, they get bogged down handling ASKAP / MeerKAT / SKA sized data sets. Largely I suspect this is because of the default behaviour of the inspectors - which is to add incoming cell data to previously stored data and then do a replot of everything. This doesn't work very well when you can have something like 350 pages of plots for a SKA -1 sized data set. The most obvious solution would be to just plot the data from the current time/frequency cell and thrown away the previously stored data, which is to a certain extent the way the individual baseline visibility plots work.
Anyway suggestions as to how this situation can be handled are welcome.
Also I've thought about having an option to plot Stokes I,Q,U,V for visibilities as a function of time. Anybody else interested in that?
@twillis449 if I recall from my py3 efforts (which I will resume after porting cubical, which is essentially ready and undergoing final tests) several of the visualizers use some combination of PyQwt and Qt4. With python 2.7 support ending soon I would say we need to first make sure that there will be a meqtrees in the long term before undertaking specific changes to the visualizers. The python 3 changes to the codebase are substantial so any conflicts will be numerous and very hard to merge.
That said I rarely use the visualizers and instead rely on CASA to plot subsets of visibilities after the pipeliner has done its thing, but maybe a ring buffer of only the last 1000 timesteps or something similar, would solve your particular problem?
Good point about PyQwt and Qt4 @bennahugo
Is there any particular reason why these plots cannot be ported to a actively maintained library like matplotlib or bokeh? @o-smirnov @twillis449 I know the Qwt libary was probably the greatest thing since apple pie when meqtrees was conceptualized, but it looks abandoned (https://github.com/PyQwt/PyQwt4), unless it has moved somewhere else. If we are going to invest serious time here it may be better to burn it all to the ground and start afresh?
The original argument for using PyQwt back around 2004/2005 was that it plotted data much faster than did the version of matplotlib available at the time. Whether or not that is still true I have no idea. When he abandoned support for PyQwt back around 2010 the original developer recommended a switch to pyqtgraph (https://github.com/pyqtgraph). @bennahugo - as you say the question is it better to start afresh or not. The C++ version of the Qwt library (now at v6,1 and compatible with Qt5) is actively maintained at https://qwt.sourceforge.io. I don't know how much time/energy would be required to develop a python 3 wrapper for Qwt6.
There is also this attempt at Py3 + Qwt6:
@ludwigschwardt Thanks - I was not aware of this port. I'll download it and test out the examples.
@ludwigschwardt interresting thanks for suggesting.
I had a quick look, their build system is very rudimentary, lacking setuptools / pip support so it complicates the install process significantly. There is no opportunity for getting a pip installable package out that fits in with a standard python workflow until this is fixed.
I found another pure python package https://pypi.org/project/PythonQwt/ which reimplements core functionality in python. I would be interested to see if the C++ bindings are actually significantly faster
than the pure python implementation with the current maturity of the standard python libraries. This needs some profiling with a VLA/MeerKAT-sized problem size.
@twillis449 This may be of use for basic high level profiling, I find it very easy to use and to visualize the hotspots in the code:
https://stackoverflow.com/questions/1896032/using-cprofile-results-with-kcachegrind
The pro of the second package is it is much better polished than the first and there is already a package on pypi. Question is whether "basic" functionality replication is enough for what we want to do in MT and Tigger @twillis449. Their samples look quite polished. The limitations are quite explicitly stated https://pythonhosted.org/PythonQwt/overview.html
The con of both packages by the looks of it is there are very few active maintainers, especially the first so question is whether we're going to run into abandonware issues again soon.
Maybe it is worth setting up a meeting @o-smirnov
As discussed on #46 the most immediate goal is getting the backend infrastructure like the pipeliner, tdls and meqserver to work on python 3. There is no viable replacement for the pipeliner (africanus and montblanc are not nearly as mature with no pipeliners available aside from the very basic crystalball). We need the ability to simulate to help understand some telescope issues.
I think I'm nearly there with a py3 version and there is a reasonable amount of test infrastructure in place to cover most of the backend simulation and calibration use cases - except perhaps the beams module, so we need to add a basic VLA case.
@bennahugo I got the package suggested by @ludwigschwardt compiled and installed in about 2 min flat in the anaconda python3 environment and all the examples there work properly. I just had to get the base packages e.g. PyQt5 etc installed via the ubuntu package manager and then follow the instructions in the README.md file. This variant is already working with Qwt6 and python3 whereas PythonQwt does not seem to have been really supported on the github repository since ca 2016. l will have a look at my Timba Plugin visualization stuff and see if I can translate then to Qwt6 / python3 with the use of https://github.com/GauiStori/PyQt-Qwt.
Alright if you're convinced this package will work then I can put in the time to package it properly - clearly we can't sudo install .py and .so files directly into /usr as these things cannot be removed later on. I prefer our python dependencies to have wheels so they can be listed as dependencies and installed and removed cleanly
Let me see how far I get. Over the lifetime of Meqtrees we've migrated from glish to python, from qt3 to qt4 and from Qwt 2 to Qwt 5 so migration is doable. :)
After working on this stuff for a couple of hours I can say that the biggest hurdle is the migration from qt4 to qt5. Lots of syntax changes and the signals / slots interface seems to have become very complicated. So far I only found one Qwt function that I use which has not been wrapped in https://github.com/GauiStori/PyQt-Qwt and I've filed an issue with the developer there.
Update: after working on this stuff for four or five days I confirm the conclusions given above. There's more work involved in moving from Pyqt4 to Pyqt5 then there is with the python QWT6 wrapper. @bennahugo I would say that one of the arguments for moving to Pyqt5 is that the underlying QT4 C++ library has not been supported by the QT Company since sometime in 2015. It does take a couple of days to get familiarity with the new pyqtSignal handling stuff plus one trips over functions that were depreciated in pyqt4 and have now disappeared from pyqt5.
I would be very careful before pouring major effort down that particular rabbit hole. If there's a quick fix, by all means let's go for it, but if it requires major coding effort, then perhaps it's time to step back and put this one to bed... the browser-based visualization has had a good innings, and has been incredibly useful for years, but can we really make it scale to MeerKAT's 2000-odd baselines? Nevermind SKA...
For our MeerKAT processing, I've pretty much abandoned the browser-server model. When your data has to sit on a remote node somewhere, and takes hours to process, relying on an X11 connection to control the operations is just too flaky. As @bennahugo says, we've reverted to running stuff from the command line (deja vu!), and visualizing the results a posteriori. Moreover, MeqTrees itself is being forced out by newer packages such as CubiCal. I can only see that as a good thing! Out with the old, in with the new, as Rick Perley likes to say (while simultaneously flagging data in AIPS on a text console using his left hand).
@twillis449 I would encourage you to rather join and contribute to https://github.com/ratt-ru/radiopadre and https://github.com/ratt-ru/ragavi, I think this is where most of our visualization efforts are going these days. There are some very neat new technologies that can be exploited to do stuff we'll never be able to do with PyQwt...
@twillis449. The question here is, is it critically necessary to port to qt5 in making the gui python 3 compatible and ensuring long term support. The claim to fame on https://github.com/GauiStori/PyQt-Qwt
is that it works with qt4 and 5. There is a debian package for python3-pyqt4 that works:
bhugo@bhugo-wks ~ [7:47:40]
> $ ipython3
Python 3.6.7 (default, Oct 22 2018, 11:32:17)
Type "copyright", "credits" or "license" for more information.
IPython 5.5.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: %paste
import sys
from PyQt4 import QtGui
def main():
app = QtGui.QApplication(sys.argv)
w = QtGui.QWidget()
w.resize(250, 150)
w.move(300, 300)
w.setWindowTitle('Simple')
w.show()
sys.exit(app.exec_())
if __name__ == '__main__':
main()
Just an update report ... I've been messing around with both PythonQwt (https://pythonhosted.org/PythonQwt/) and PyQt-Qwt https://github.com/GauiStori/PyQt-Qwt) They both seem to work reasonably well. I've decided to settle on PythonQwt because it's pure python code that works with either python2 or python3 and PyQt4 or PyQt5 and since one has the source code it will never become 'obsolete' as there's a clear sentiment in this discussion that further development with Qwt is not particularly of interest. PythonQwt only contains the most basic elements of Qwt and lacks things like zoomers, pickers etc. However ... since the MeqTrees visualization project started back when Qwt was at something like qwt 2 or 3, we have our own zoomers etc within the MeqTrees Plugins code. :) I did have a roadblock for about 2 weeks as I read too many StackOverflow articles on Mouse Events and had trouble getting mouse locations to be reported properly - didn't help that I haven't looked seriously at PyQt or PyQwt in about 8 years.Anyway I'm now over that hurdle and its now mainly an issue of handling what differences there are in syntax in PyQt5 vs earlier PyQt4 stuff.
Hey Tony, did you see my PR and description of required changes?
@gijzelaerr No, I do not believe I've seen those documents.
See comment before, there is a to-do document somewhere, forgot where. Maybe useful, maybe not. I think I was quite close.
@gijzelaerr - Hum, I can't spot anything obvious in the repositories you mention. Anyway one package I've found useful is to run pyqt4topyqt5.py (https://github.com/rferrazz/pyqt4topyqt5) for a first pass through the code. It spots all the signals that have to be turned into pyqtSignals. Unfortunately, if the signal has parameters you have to specify the type of the parameters to the pyqtSignal yourself. So far the only Qwt widget that I have needed and is not provided in PythonQwt is the QwtSlider with floating point parameters. However, that's easy enough to fake with the standard QSlider class (see e.g. https://stackoverflow.com/questions/42820380/use-float-for-qslider).
Hokey-dokey - all plotting widgets now seem to be working with python 3 / PythonQwt/ qt5. Or at least my test programs are :) Eventually some of the widgets will need to send signals to each other, but I can't test that stuff until the higher level MeqTrees interface code is up and running in python 3.
After some trial and error I found that the best way to proceed with the upgrade was
- run the program 2to3 which is supplied with python3 (at least it is in the anaconda distribution). This script replaces python2 syntax which is no longer supported in python 3 - e.g. all print 'xxx' statements get replaced with print ('xxx')
- then run the pyqt4topyqt5.py script. Amongst other things this script will find all instances of old-style pyqt3 or 4 signals and replace them with the new-style pyqtSignal equivalents. Note that you are left with the task of specifying the data types of parameters being sent with the signal.
- then replace old-style widget specifications with explicit specification of required Qt / Qwt widgets. See the PythonQwt/qwt/tests directory for examples.
4)finally, try running the script. You will almost certainly trip up over a few individual Qt widgets whose api has changed from Qt4 to Qt5.
I was talking about this file:
https://github.com/ska-sa/tigger/blob/qt5/TigGUI/todo.py
I centralized all the missing widgets, and rewrote most of the signaling.
but it looks like you are further in the meanwhile. I didn't knew about that qt convert script, cool.
keep up the good work :)
Update - as of today just about everything now working as a part of the MeqBrowser plugin system. Since python 3 syntax requirements were already optional in the latest python2.7 release I can run all the updated visualization scripts under the python2 based meqbrowser. This testing revealed numerous got-ya's etc which the stand-alone test programs did not reveal - e.g. problems with ColorBars communication with the image displays etc. pyqtSignals are also more finicky than the previous signal system. Took a bit more than a month of my time, but I really refreshed my knowledge of the Qt widget system :) Everything is running with the PythonQwt package. Hard to tell if things run faster or slower than with the old PyQwt stuff as most of the visualization stuff works with visibility data, so disk I/O would seem to be a significant limiting factor.
Geez @twillis449 that's an amazing achievement. I really feared this would be a hoary undertaking -- I should never have doubted you!
At this rate you might find yourself volunteered for missing Tigger display widget duty... ;)
Great work @twillis449. Can you start a PR with your changes for review and incorporation? This makes my life significantly easier thanks ;). Once I've found the bug that is hampering the communication between the browser and the server in python 3 mode we should have a working py3 version.
@bennahugo - OK will do so. BTW - regarding your comments back on May 29 - further testing does show that the python plotting stuff of particularly the 'stripchart' plots is indeed slower than is the case with the C++ Qwt interface. I'm still using the same underlying python numpy stuff that I used in the original python2 version of the code for the data handling, so the slowdown must come from the pure python plotting. In retrospect I can see that while the plotting can handle WSRT/ATCA/DRAO sized data sets its gasping when fed anything bigger. I think there may be a few tricks I can implement to speed thins up - we'll see, Also as regards Qwt in general, I just discovered today that the CASA casaviewer is implements using (presumably) the C++ version of Qwt!
@bennahugo I've created a branch named qt5-plugins with my updated code and have generated a pull request. I've encountered one case where I think there might be a bug in the underlying PythonQwt code but I think I've managed to dodge around it at the moment.
@o-smirnov Are you aware of anyone else except you (and occasionally me) who uses the Meqtrees solver for anything these days? I've run across a minor annoyance in moving to Qwt6 that makes it hard to plot all the various metrics that the solver produces e.g. chi_sq surfaces. I suspect that this was originally something JEN wanted way back when, and has never been used seriously by anyone else. The problem has to do with low-level QPainter code which I don't really feel like investigating.
Further investigation got me past the initial issue, but it looks like there's now a problem with the way PythonQwt plots individual symbols. Grumble.
I've now gotten a version of my QwtPlotCurveSizes code to work with python3 / qt5, which means that we should be able to plot the various Solver metrics properly.
e.g. chi_sq surfaces
Ah, the legendary "Nosebleed Plots".
Haha... thanks I will take it for a spin locally before merging.
I think the current automated pull request test infrastructure for timba is a bit inadequate (a full test is only invoked in pull requests to cattery). I will remedy it on the branch.
I think I've taken this stuff about as far as I can at the moment. There appear to be two bugs in PythonQwt that cause me problems. 1) when when I generate cross-section plots of an array of visibility data, the Qwt 'delete' function should cause them all to disappear, but somehow some of them seem to 'stick around'. This issue also affects the 'Solver' plots. 2) for some reason when one wants to display a legend, PythonQwt can complain that legend() is not a callable function, whereas its clearly defined as such in the PythonQwt code. Since the developer of the PythonQwt code base disappeared off github something like 3 1/2 years ago, we're on our own as far as finding a fix is concerned. I'll investigate when I have time. As far as getting usable plots for telescopes of the size of ASKAP. MEERKAT and SKA, the current system can't cut it, as was suggested eariier in the discussion. With SKA the visibility inspectors have ~306 tabs because of the 19000 baselines. We will just have to be selective about which baselines get plotted as is currently the case for the individual interferometer baselines.
For longer term visualization work on MEERKAT and SKA sized telescopes I think one should look at packages such as VisIt (https://wci.llnl.gov/simulation/computer-codes/visit/) or ParaView (https://www.paraview.org/). Anybody already doing that?
Re Bugs:
-
PythonQwt does not seem to properly count the number of active QwtPlotCurves so when you want to delete them all using the PythonQwt build in function to find and delete all QwtPlotCurve objects they do not all seem to go away. Solution - generate your own 'dict' of QwtPlotCurves and delete all the ones in the dict. This gets rid of all cross sections in a visibility data display.
-
PythonQwt definitely does not handle QwtLegends correctly. My solution - get rid of using QwtLegends and replace by text in a QwtPlotMarker insidea plot window. We were only using QwtLegend objects inside the Solver plots so not really a big issue. There are still some minor issues with Solver plots but I'm putting those on the back burner.
Working on this software feels similar to what it must be like working on Boeing 737MAX software. Especially display_image_qt5.py has a lot of options that I'm not sure I can completely test - I will try to go through some more of my old MeqTrees scripts to find test cases.
@o-smirnov - if we want to use the inspectors option in calico-view-ms, an SKA 197 antenna MS causes 306 plot tabs to appear!! To cut down on this load - which just causes calico-view-ms to choke up we really need some sort of option similar to what's used for the individual baseline option. Maybe an option to select one of the 197 antennas and then 'inspect' all the baselines associated with it. That would result in only 4 tabs being displayed.
@bennahugo - I agree that external visualizers are probably better for looking at visibility data being generated by MEERKAT, SKA etc. However one of the main things I've used MeqTrees for is simulation work and there the internal plots are often quite useful.
@twillis449 I finally have some time to do testing and dev. I'm running into ibus connection problems it appears that qt5 / qwt now requires an ibus connection do you have a workaround that don't require starting up the ibus deamon externally?
Secondly it looks like there is some gui threading problems:
Welcome to the MeqTrees Browser!
Please wait a second while the GUI starts up.
libpng error: IDAT: invalid distance too far back
libpng warning: iCCP: known incorrect sRGB profile
libpng error: IDAT: invalid distance too far back
WARNING: couldn't import plugin 'Timba.Plugins.quickref_plotter' (API 'QString' has already been set to version 1)
This plugin will not be available.
WARNING: couldn't import plugin 'Timba.Plugins.collections_plotter' (API 'QString' has already been set to version 1)
This plugin will not be available.
WARNING: couldn't import plugin 'Timba.Plugins.array_plotter' (API 'QString' has already been set to version 1)
This plugin will not be available.
WARNING: couldn't import plugin 'Timba.Plugins.result_plotter' (API 'QString' has already been set to version 1)
This plugin will not be available.
Binding to local socket =meqbrowser-1000:1
Binding to TCP port 5000, remote meqservers may connect with gwpeer=<host>:5000
Traceback (most recent call last):
meqbrowser.py: Fatal IO error: client killed
[xcb] Unknown sequence number while processing queue
[xcb] Most likely this is a multi-threaded client and XInitThreads has not been called
[xcb] Aborting, sorry about that.
python2.7: ../../src/xcb_io.c:259: poll_for_event: Assertion `!xcb_xlib_threads_sequence_lost' failed.
[1] 13964 abort (core dumped) env "LD_LIBRARY_PATH=/home/bhugo/workspace/install/symlinked-debug/lib"
This is a fresh virtualenv with all the packages installed including python-qwt
Nevermind grabbing a newer qt5 solved the threading problems. Ok I have it running however I get the following:
WARNING: couldn't import plugin 'Timba.Plugins.quickref_plotter' (API 'QString' has already been set to version 1)
This plugin will not be available.
WARNING: couldn't import plugin 'Timba.Plugins.collections_plotter' (API 'QString' has already been set to version 1)
This plugin will not be available.
WARNING: couldn't import plugin 'Timba.Plugins.array_plotter' (API 'QString' has already been set to version 1)
This plugin will not be available.
WARNING: couldn't import plugin 'Timba.Plugins.result_plotter' (API 'QString' has already been set to version 1)
This plugin will not be available.
Ok so far so good although I'm getting sporadic signal connection problems:
Loading /home/bhugo/workspace/DEEP2_tagged.lsm.html: Tigger sky model
QObject::connect: Incompatible sender/receiver arguments
QToolButton::clicked() --> PyQtShortcircuitSignalProxy::pysignal(PyQt_PyObject)
QObject::connect: Incompatible sender/receiver arguments
QToolButton::clicked() --> PyQtShortcircuitSignalProxy::pysignal(PyQt_PyObject)
QObject::connect: Incompatible sender/receiver arguments
QToolButton::clicked() --> PyQtShortcircuitSignalProxy::pysignal(PyQt_PyObject)
@twillis449 When I try connecting up the gain plots it says it can't find collections plotter. I'm guessing it is bacause those plugins fail to load.
@bennahugo To answer your questions ...
- to get rid of the 'API 'QString' has already been set' errors, at the beginning of meqbrowser.py I had to insert
import sip
sip.setapi('QString', 2)
sip.setapi('QVariant', 2)
sip.setapi('QDate', 2)
sip.setapi('QDateTime', 2)
sip.setapi('QTextStream', 2)
sip.setapi('QTime', 2)
sip.setapi('QUrl', 2)
This has something to do with the fact that all python 3 strings are unicode. But I had assumed that the error would disappear in a pure python3 environment - apparently not. I think you can google the error to find out more. But anyway because of this error none of the plugins etc get loaded. After I added those sip.setapi calls the plugins did load.
However - related to unicode string issues - in my chartplot_qt5.py code i have to import a module called unidecode to handle some string conversions in my python 2 test environment . Hopefully this is not required in the pure python 3 environment.
-
re: PyQtShortcircuitSignalProxy complaints - they seem to be occurring in tigger, which is @o-smirnov 's baby. But what's probably happening is that the PyQt5 signal/slot definitions are way more finicky about having exactly the same kind of data and parameters in both the signal emitters and the slot receivers than were earlier versions of PyQt.
-
libpng errors - somewhere along the line the png maintainers changed or got rid of some parameters in a png file. One of @o-smirnov icons is crashing because of this. I did repair the offending icon. I'll dig up the repaired icon for you.
Have fun. :)
I've pushed an updated version of PyApps/src/icons/treebrowser/table,png to my branch on github. The icon should now load, although there's still a warning message but will no longer crash.
@o-smirnov @bennahugo Hi Chaps - I hope you've been making some progress on the MeqTrees port to python3.
As of August 29, 2019 I'm finally ending my time at the National Research Council of Canada so I'll no longer have any formal relationship with the observatory here. And I'll be on vacation in Italy for all of September. Hopefully, I won't be taking any computing device larger than a tablet with me so if there any issues with the visualization stuff I won't be able to have a look until October. Frankly, I think that display_image_qt5 code needs to be completely re-factored but that's a big job and probably not worth doing without the motivation that the displays will actually be used by more than one or two people.
Cheers
Tony
Hey @o-smirnov @bennahugo looks like this thread has gone totally dormant. Any progress been made with the port to python3? I'm back from my holiday and my brain is starting to wake up!
@twillis449 sorry for the lack of progress on this front. I've been a bit distracted by my phd work and porting some of the other packages. When I last worked on this I couldn't get the meqserver communicating with the browser in python 3. I'm still stuck on that front but I'm hoping I will be able to fix that soon
This has been accepted and merged