Image domain gridding

The image-domain gridder (IDG) is a new, fast gridder that makes w-term correction and a-term correction computationally very cheap. It performs extremely well on gpus.

To use IDG, the IDG library needs to be available during compilation. WSClean 2.5 is compatible with IDG version 0.2. See installation for more information.

Considerations

IDG supports the same cleaning and data selections options that WSClean offers in normal mode (without IDG – so e.g. Cotton-Schwab, multi-frequency multi-scale cleaning, auto-masking, etc.). However there are a few points which require a bit of care to use IDG:

  • Baseline-dependent averaging does not work together with IDG and should not be used together.

  • Imaging multiple polarizations independently (without joining polarizations) is not possible with IDG, see the discussion on polarization settings below.

  • IDG can only be used on measurement sets that have four polarizations (e.g. XX/XY/YX/YY).

Command-line options

There are several IDG related WSClean parameters:

-use-idg

This option turns on the IDG gridder. If IDG is not available, WSClean will show an error.

-idg-mode [cpu/gpu/hybrid]

IDG mode to use. The default is CPU, which works on any computer. GPU and Hybrid both make use of the GPU and require the availability of GPU libraries when compiling IDG (see the IDG manual). The “pure” GPU mode is however not recommended to be used, as it does not have all the features that the hybrid mode has, hence hybrid is the recommended method when GPUs are available.

Gridding with the beam

IDG allows gridding with a time-variable beam – currently (as of 2.6), this includes the LOFAR, AARTFAAC and MWA beam. This would increase the accuracy of with which sources are deconvolved (in theory, but it depends of course on the accuracy of the beam model). The option to do this is -grid-with-beam, and optionally, if the differential beam needs to be applied (which is common; see the LOFAR beam page for more info) one would also add -use-differential-lofar-beam.

In summary; IDG with full beam correction:

wsclean -use-idg -grid-with-beam ...

IDG with differential beam correction:

wsclean -use-idg -grid-with-beam -use-differential-lofar-beam ...

For more information about the differential lofar beam, see primary beam correction. Note that you do not have to add -apply-primary-beam: if you don’t specify this parameter, WSClean will grid with the beam and output both the “flat noise” image and the pb corrected image.

NB: In earlier versions, specifying both -apply-primary-beam and -use-idg would apply the wrong beam!

Polarization settings for IDG

IDG always images all Stokes parameters. Therefore, it makes sense to set the polarizations to iquv (with -pol iquv) when using the IDG, as otherwise the other polarizations are imaged and then thrown away. IDG does however require joined or linked polarization cleaning: it is not possible to clean the polarizations individually with IDG in a single run. If this is really desired it can be done by running wsclean 4 times, imaging one polarization at a time, however most science cases will want to use one of the other options described below.

There are several distinct cases of interest:

  • Total intensity science, i.e., not interested in QUV: only image I with -pol I.

  • Total intensity science, but “nice to have QUV” for e.g. sensivity analysis (V is useful for that): image all polarizations with -pol IQUV and link the deconvolution on polarization I with -link-polarization i.

  • Interested in rotation measure synthesis, not directly in I and V but nice to have: image all polarizations with -pol IQUV and link the deconvolution on polarization Q and U with -link-polarization qu (possibly with squared deconvolution etc., see [polarized cleaning page](PolarizedCleaning)).

  • Interested in all stokes parameter, cleaning each polarization in a joined way: -pol IQUV -join-polarizations.

Some further explanation on this: IDG in combination with -pol iquv -join-polarizations can be the best choice in some cases, but if one is only interested in Stokes I, it has the downside that QUV are involved in the cleaning of I, and in case these are mostly empty, the net effect is that the noise in the clean-component finding step is increased slightly. Since [WSClean 2.6](Changelog-2.6) it is possible to “link” polarizations. Its use is explained on the [polarized cleaning page](PolarizedCleaning). To overcome the downside described above, the recommended setting for imaging with IDG is to use polarization linking on I, similar to:

wsclean -link-polarization i -use-idg [..]

This will clean Stokes I fully, and clean the components found in I also from the other polarizations. However, it will not clean structure from QUV that is not found in I.

Performance

On my 4-core home machine, the CPU version is not as fast as the wstacking gridder. However, it can apply a-terms and uses less memory, which can be advantageous in some cases. In contrast to the CPU gridder, the GPU gridder is considerably faster (up to an order of magnitude), but only for large images (>6k or so).

The exact performance benefits depend heavily on image size, bandwidth and number of cores available. Hence if performance is important, I recommend to make a careful comparison for the particular test case you are interested in.

IDG has been tested and shown to work on MWA data. It performs well, but does require a lot of memory, caused by the wide field-of-view of the MWA and therefore high w-terms.

Advanced a-term corrections

WSClean+IDG allows a combination of several direction-dependent corrections to be applied, including TEC screens, diagonal gain correction and position shifts (“dldm screens”). These are discussed on the [a-term correction page](ATermCorrection).

Information for older IDG versions

Before WSClean version 2.9, IDG’s memory usage was highly dependant on the number of channels in the set. IDG can be made to use considerably less memory by splitting the bandwidth using -channels-out and wide-band deconvolution (see making image cube and wideband deconvolution). For example, splitting the bandwidth in 4 output channels has allowed imaging one of the MWA sets on a 32 GB machine:

wsclean -grid-with-beam -beam-aterm-update 10 -channels-out 4 -join-channels -link-polarizations i -use-idg -size 1536 1536 -scale 1amin -niter 1000000 -auto-threshold 0.5 -auto-mask 4 -multiscale -mgain 0.8 observation.ms

This should no longer be necessary for WSClean 2.9 and later. In those versions, IDG should honour the requested memory settings and available memory. If you do expect memory issues, you can tweak the memory usage using the -mem and -absmem parameters of WSClean.