h5parmpredict Invalid direction string

Maybe I'm overlooking something obvious, but I get the following error when trying to use h5parmpredict:

std exception detected: Invalid direction string: expecting array between square brackets, e.g. [a, b]

My parset is

msin = P127.ms
msin.datacolumn = DATA
msout = . 
msout.datacolumn = SUBTRACTED_DATA
msout.storagemanager = dysco

steps = [h5sub]
h5sub.type = h5parmpredict
h5sub.operation = subtract
h5sub.applycal.parmdb = phaseonlyP127.mssolsgrid_5.h5
h5sub.applycal.correction = tec000
h5sub.directions = [pointing]

There is a direction called pointing in the h5parm (otherwise it crashes that it can't find the direction). An applycal step seems to have no problems with it. Am I missing something? The h5parm was made with DDEcal's tec mode, using commit c9623fa. I'm currently running 4953170

If I look at the h5parmpredict source code, it seems that directions requires a list of list. It apparently should allow things like h5sub.directions = [[patchA, patchB], [patchC]], applying solutions of direction 1 (from the hdf5) to patchA and B (in the model). However, I don't think that actually works, because I think it would look for a direction in the h5parm file called "[patchA, patchB]". So, that looks like a bug; @tammojan ?

Maybe your case will work if you do h5sub.directions = [[pointing]], but I'm not sure. Could you try? (it might look for direction '[pointing]', with brackets, in the hdf5). If not then that's another bug. And clearly the documentation of h5parmpredict.directions is also not up-to-date.

If I try h5sub.directions = [[pointing]] then it indeed looks for a direction [pointing] and fails with

std exception detected: Direction [pointing] not found in phaseonlyP127.mssolsgrid_5.h5

Ok, thanks. I think the directions parameter has never worked ; only leaving it empty would work.

Exception is generated here:

DP3/DPPP/H5ParmPredict.cc

Line 81 in 9782b07

    
           throw std::runtime_error("Invalid direction string: expecting array between square brackets, e.g. [a, b]");

This used to be an "assert", and probably wouldn't work in a release compilation, so it might be that #159 has turned this check on. @tammojan can you look at this?

I see. I also tried leaving it empty, but that throws the error as well.

However, I don't think that actually works, because I think it would look for a direction in the h5parm file called "[patchA, patchB]". So, that looks like a bug; @tammojan ?

By default, DDECal writes directions in the form of their original specification, so if the h5parm was created with DDECal then it will have directions like [patchA,patchB].

I think the directions parameter has never worked ; only leaving it empty would work.

Not true. It works with h5pams created with DDECal (at least when I tested it).

@tikk3r What are you trying to achieve? The goal of H5ParmPredict is to subtract sources (from a skymodel) corrupted by the solutions in a parmdb. This doesn't make sense to me when you try to subtract the pointing, since you probably do not have a skymodel for that?

@tammojan This is one of a number sources that I phaseshifted to and split out (long baselines, using the helpful explode step), before being put through a self calibration script. @rvweeren there is nothing special about the DDEcal steps in the script right?

I then made a sourcedb using the clean component list written out by WSClean, which I then wanted to try to subtract from the data it was split out of.

@tikk3r There is nothing special, but you do need to predict with the beam.

Note that the H5 was generated by ddecal, but, it contains only one direction (so it's not a true dde solve, more like normal di selfcal)

I think for this case a regular predict step will do the trick:

steps = [h5sub]
h5sub.type = predict
h5sub.operation = subtract
h5sub.applycal.parmdb = phaseonlyP127.mssolsgrid_5.h5
h5sub.applycal.correction = tec000
h5sub.direction = [pointing] # Can probably be left out in this case

Great, that ran fine. I just went by the name and did not realize the normal predict could do this as well. Thanks for the help.

I think the directions parameter has never worked ; only leaving it empty would work.

Not true. It works with h5pams created with DDECal (at least when I tested it).

Ah okay, thanks. The documentation of h5parmpredict is nevertheless a bit poor on the direction property: could you fix that @tammojan , and include some of the info from this discussion here? If even I can't figure out from the source code how it works I think some explanation will be helpful ;).

Thanks, I improved the docs a bit.