5.5 More editing of uv data

5.5.1 General remarks on, and tools for, editing

There are many programs which aid in the processing, display, and editing of uv data. Summaries of this software may be listed on your terminal with:

> ABOUT UV  C R

to list all uv-related software.

> ABOUT EDITING  C R

to list all editing software.

> ABOUT PLOT  C R

to list all plotting software.

and are also in Chapter 13 of this CookBook. Type

> DOCRT -1 ; EXPLAIN taskname  C R

to print information about task taskname.

to get more information about any of the tasks mentioned below. The discussion below assumes that you have deduced that there are suspect samples in your data set and that you want to remove them. Read §4.3.5 before investing large amounts of time in editing even at this stage.

There are facilities in CALIB, RFLAG, FLAGR, CLIP, CORER, UVMLN, FLGIT, DEFLG, and SNFLG to flag uv data in AIPS based on deviations from specified norms. There is also the task UVFLG to flag and unflag by antenna-IF or by correlator. The task UVPLT plots various combinations of uv data; see §6.3.1. The task WIPER makes a similar plot on the TV and allows you to wipe away offending data. The task UVFND is also recommended for printing out suspicious portions of the database; see §6.2.1. Note that CLIP examines the data correlator by correlator, but UVFND normally converts the data to Stokes components (using the same criteria as UVMAP) before checking that the amplitudes and/or phases are in range. To examine the correlators individually, use STOKES ’CORR’ in UVFND, or to flag the data based on their values after conversion to true Stokes use STOKES = ’IQUV’ in CLIP. Task FINDR is a companion to FLAGR intended to assist you in determining what is normal within your data.

CLIP is also useful for flagging discrepant data (e.g., due to interference or malfunctions) on the basis of their deviations from the visibility predicted by a set of Clean components. Tasks OOSUB or UVSUB will subtract the Fourier transform of a set of Clean components from visibility data. You may then use UVPLT to display the residual uv data set and CLIP to flag abnormally high points. You may wish to be cautious, and run UVFND to display such points before running an automatic CLIP task — be especially careful not to CLIP away evidence for real extended structure near the center of your uv plane! Before re-imaging, you must copy, with TACOP, the flag table produced by CLIP to the data set used for imaging. Note that IMAGR’s workfile is also a uv data set from which the current Clean component model has been subtracted. It may also be used with UVPLT to help you to diagnose problems. If you run CLIP on it, you will need to append the resulting FG table to one on the imaging data set; use TAPPE.

FFT is another useful tool for finding suspicious data. Transform your image back into the (u,v) plane by running FFT and then display the results on the TV. Use image read-back verbs like CURVALUE and IMPOS (§6.4.5) to find the u and v values for abnormally high cells. Then use UVFND with OPCODE UVBXto print the data surrounding these cells and UVFLG to delete any bad data. This method is particularly effective when applied to residual images from Clean. (You can instruct IMAGR to put out a residual image by setting BMAJ <0. CCRES can also make a residual image from a single-field normal image.)

In 31DEC16, the process described above has been improved upon considerably. Task UFLAG makes a uv-plane grid (with pill-box convolution) of your visibility data selcted by adverbs much like those used in IMAGR. It shows you the vector and scalar averaged grids in amplitude and the vector averaged grid in phase and offers a variety of editing options. You may choose to mark an entire cell as “bad” or look at the visibilities contributing to cells you select and choose to mark some of them (rather than all) as bad. There are even automatic routines to do this last operation over cells meeting criteria that you set. The entire process is decribed in an AIPS Memo.4

There is a task called FLAGR which goes through a data set determining what are normal rmses and weights and then flagging those that deviate excessively including clipping all those that have amplitudes or weights outside specified normal ranges. FLAGR is intended for use eventually in pipeline data-reduction procedures, but at present should be considered experimental, but potentially very valuable. Before calibration, try it on your calibration sources using default values for most adverbs plus SOLINT set to 2.5 times the basic integration time and VECTOR set to 0. Task FINDR is a companion intended to determine what is normal in the data and then to print those values and return selected adverb values to AIPS for use by procedures.

There is a task for automatic flagging called RFLAG. Since RFI tends to be highly variable over time and/or frequency, RFLAG accumulates statistics about this variation which it plots along with recommended clip levels. Running it a second time, with DOPLOT0, causes the clip levels to be applied to make a large flag table. This appears to be a very succesful program with EVLA wide-band data (§4.3.13). Task REFLG may then be used to compress and extend the flag table.

TVFLG, SPFLG, FTFLG, WIPER, IBLED and EDITR are TV-based, interactive editors. TVFLG is most suitable for data sets with large numbers of baselines, e.g., the VLA, but it can be used usefully for VLBI data experiments with 10 or more antennas. TVFLG allows you a global overview of your data and can display the data for all baselines simultaneously as a function of time. This task is documented extensively in §O.1.6 of the CookBook. SPFLG is a very useful task for data with a significant number of spectral channels. It is effective in examining data for frequency-dependent errors and interference and can be an effective data editor for interferometers with a small number of baselines; see §10.2.2 and §8.1. FTFLG is like SPFLG, but combines all baselines into one plot. This is a much faster way to look for global RFI, but it must flag all baselines with each flag generated. IBLED has a different philosophy; it plots one baseline at a time in a graphical rather than gray-scale (image) fashion. It is able to average data over time, spectral channels, and/or IFs to make a more manageable amount of data and to measure the “decorrelation index” which is a measure of how variable the phase is over the averaging intervals. The capability of averaging IFs and displaying decorrelation may be of special interest for VLBI data sets. Otherwise, IBLED has been replaced in 15APR98 by EDITR, which also uses the graphics planes rather than gray-scale images but which can plot multiple baselines to a chosen antenna and can display two data sets at the same time. This is obviously more useful for smaller arrays — e.g., VLBI, MERLIN, and the Australia Telescope. This task is described below.

The editing task WIPER should be used with caution. It makes a plot like UVPLT of almost any parameter of a uv dataset against any other parameter. The plot is displayed on the TV and you may “wipe away” any points you do not like one point at a time or many at a time with a “fat brush.” The task will be very useful for fields with a well-behaved visibility function seen with good signal-to-noise. It may also be useful with data sets from which a fairly good IMAGR model has been subtracted with UVSUB. It allows you to plot and edit any reasonable STOKES polarizations in one execution and displays which baselines enter into each plotted point.

In 31DEC16, the new task FGCNT may be used to count the number of correlators flagged by a particular flag table. Many of the tasks above can generate very large flag tables and can, if the choice of parameters was not right, flag way too much data. This task provides a way to check on this.

5.5.2 Baseline-based uv-data editing — EDITR

EDITR is a very effective editing tool from the beginning of data analysis on data sets with modest numbers of antennę. Since it can display two data sets at the same time for comparison purposes, EDITR may also be used to good purpose with larger data sets during the self-calibration and imaging stage. The visibility amplitude or phase or the amplitude of the visibility with a running vector average subtracted may be displayed. The data for the selected baseline are shown in an edit window at the bottom of the display. Optionally, a second observable (e.g., phase) from the selected baseline is shown in the same color in a window directly above the edit window. This option is controlled by the DOTWO adverb. Data for 0 to 10 other baselines to the selected antenna may be displayed in a different color in windows above these. A second uv data set may also be displayed along with the first. These data are not used for editing but may help you to select the data to be deleted. A “normal” choice for the second data set would be the residuals after Cleaning or UVSUB. A menu-like control interface is available to select the data antenna and time range to be edited and to select various forms of editing. Instructions, explanations, informative messages, and the results of various functions appear in the standard AIPS message window. When prompted for information, such as an antenna number, type it into your normal AIPS input window (which is where the prompt message should have appeared).

EDITR is for editing continuum data from one or more IFs. Multiple spectral channels may be averaged on input with the vector average used for display and editing; multiple IFs are kept separate. The data may also be averaged over time as they are read into memory. This is useful for improved signal-to-noise, but will cause the data flags to be less selective in time. The program will allocate sufficient dynamic memory to hold the selected data. If you have a very large data set and a modest computer, it would probably improve efficiency to limit the time range and run the task more than once to cover all time ranges.

To run it, enter:

> TASK EDITR ; INP  C R

to select the task and review the inputs.

> INDI n1 ; GETN ctn1  C R

to select the ’TB’ sorted uv single- or multi-source data set.

> DOCAL FALSE  C R

to apply no calibration. The SN or CL table from previous calibrations can be applied.

> FLAGVER fg1  C R

to apply flag table fg1 to the data on input.

> OUTFGVER 0  C R

to write a new flag table containing version fg1 and all new flag commands generated.

> SOLINT = Δt  C R

to have the data averaged over a time interval Δt minutes. If you do not want averaging, set this parameter to a small value; the default is 16000 = 0.01 second. Editing times are recorded with an offset of SOLINT/2 which may cause confusion when no averaging was actually done.

> DETIME T  C R

to set the initial scan length estimate to T minutes (which can be changed later interactively) and to set the interval regarded as a break in the regular time sequence of the data. Setting this parameter suitably helps the program do a better display, but its exact value is not critical.

> CLR2NAME  C R

to display only one data set.

> DOTWO TRUE  C R

to display a second observable from the main baseline.

> CROWDED TRUE  C R

to allow all IFs and all polarizations to be displayed and edited at one time.

> DO3COL TRUE  C R

to use color to separate IFs and polarizations when more than one is displayed and edited at one time.

> INP  C R

to review the other parameters, which we assume here to be set to their null values.

> GO  C R

to run the task.

You can average the data over spectral channels (the default will average all channels present). IFs are edited separately; the default will include all IFs after which you can choose the one to edit interactively.

Since the display used by EDITR is very similar to the one used by EDITA displayed in §4.3.11, we do not include a figure here; see Figure 4.2. The upper left corner of the display is reserved for displays of the selected data sample during editing while the bottom left corner is used for status information including flagging options. Menus, discussed below, appear down the left and right sides of the screen. The data are displayed in a stack of plots in the center of the screen. At the bottom are the data from the selected baseline in the primary observable; then the data from the primary baseline in a second observable (if DOTWO is true), and finally the data in the primary observable from 0–10 other baselines to the primary antenna. Data which have been flagged are shown in a different color. The data are plotted on a linear axis vertically, while the horizontal axis is monotonic but irregular in time. Tick marks are plotted at integer hours and the time interval of the edit area is indicated by times at the left and right ends of the axis. The time range displayed in all plots may be selected interactively and editing may therefore be done in crowded full time-range plots or in well separated short time-range plots. Surrounding the plot are various annotations describing the data plotted and the status of the various flags which control which data will be deleted on the next flagging command. If a second data set was specified, then data from that file are displayed in a different color in the same plot areas used for the primary data set.

The interactive session is driven by a menu which is displayed on the same screen as the data. Move the cursor to the desired operation (noting that the currently selected one is highlighted in a different color on many TVs) and press button A, B, or C to select the operation. Press button D for a short explanation of the selected operation. The right-hand column contains options to select which data are displayed and to select which data are flagged on the next flag command. The menus are changed to adapt to the input data in order to avoid, for example, offering options to select IF in a one-IF data set. The left-hand column contains 7 interactive modes for editing the data plus options to set the display ranges and scan averaging length, to turn on error bars in plotting samples, to review, alter, and re-apply the existing flag commands, to defer or force a TV display, to switch to entering commands from the keyboard instead of the menu, and to exit with or without applying the current flag commands.

The right-hand menu can contain

NEXT CORRELATOR

To switch to viewing the next correlator, switching to the other polarization and, if needed, incrementing the IF.

SWITCH POLARIZ

To switch to viewing and editing the other polarization, cycles through both if CROWDED was true.

SWITCH ALL POL

To switch functions from applying to one polarization to applying to both polarizations or vice versa.

ENTER IF

To select which IF is viewed and edited. When CROWDED is true, zero means all.

SWITCH ALL IF

To switch functions from applying to one IF, to applying to a range of IFs, to applying to all IFs.

SWITCH ALL TIME

To switch FLAG ABOVE and FLAG BELOW between all times and the time range of the frame.

ROTATE ALL ANT

To rotate functions from applying to (a) one baseline, (b) all baselines to the main antenna, and (c) all baselines.

SWITCH ALL SOURC

To switch between flagging only the current source and flagging all sources.

ENTER ANTENNA

To select the main antenna, baselines to which are displayed on the screen.

ENTER OTHER ANT

To select up to 11 other antennas to define the baselines to be displayed; enter 11 numbers, 0’s are then ignored (to plot 5 enter the 5 plus 6 0’s). The first one is used for the edit area.

NEXT BASELINE

To advance the list of other antennas, selecting the next one for the edit area.

NEXT ANTENNA

To select a new main antenna, one higher than the current main antenna. The “others” will also be adjusted if appropriate.

PLOT ALL TIMES

To display all data for the selected baselines.

SELECT FRAME

To select a window into the current data interactively.

NEXT FRAME

To select the next time range window of the same size as the current frame.

PREVIOUS FRAME

To select the previous time range window of the same size as the current frame.

SHOW AMPLITUDE

To display and edit amplitudes.

SHOW PHASE

To display and edit phases.

SHOW DIFF AMPL

To display and edit the amplitudes of the vector difference between the sample and its running mean.

SHOW ALSO AMPL

To display amplitudes of the edit baseline for reference with the phase or difference amplitude edit window.

SHOW ALSO PHASE

To display phases of the edit baseline for reference with the amplitude or difference amplitude edit window.

SHOW ALSO DAMP

To display difference amplitudes of the edit baseline for reference with the phase or amplitude edit window.

TV ZOOM

To alter the display zoom used while in the flag functions.

OFF ZOOM

To turn off any zooming.

2ND UV OFF

To disable the display of the 2nd uv data set.

2ND UV ON

To enable the display of the 2nd uv data set.

The data displayed are of a single polarization, single IF, and 1–11 baselines to a single antenna. If CROWDED is true, then you may also choose to display and edit both polarizations and/or all IFs at the same time and input adverb DO3COL determines if polarizations and IFs are shown in different colors. The NEXT CORRELATOR cycles through all polarizations and IFs, show one at a time. The SWITCH POLARIZATION option switches the displayed polarization, the ENTER IF option prompts you if necessary for a new IF number, the ENTER ANTENNA option prompts you for a new primary antenna number, and the ENTER OTHER ANT prompts you for up to 11 other antenna numbers to select the main editing baseline and up to 10 secondary baselines to the primary antenna. (Note that you have to type in 11 numbers, but zeros are then ignored.) A flag command can apply to one or both polarizations and to one, some, or all IFs. It can apply to one baseline, to all baselines to the primary antenna, or to all baselines. The FLAG ABOVE and FLAG BELOW commands can apply only to the time range displayed in the data “frame” or they can apply to the full time range in the data set. The SWITCH ALL POL, SWITCH ALL IF, ROTATE ALL ANT and SWITCH ALL TIME options control these choices and the current state of these switches is displayed at the lower left of the TV screen. The task is able to zoom the display during interactive editing operations if you should need magnification to see what you are doing. The TV ZOOM and OFF ZOOM options let you control this. In larger data sets, however, a more useful display is obtained by interactively selecting a narrower time range with the SELECT FRAME option. To step forward and back through the frames, use the NEXT FRAME and PREVIOUS FRAME options, respectively. To display uv data amplitude, select SHOW AMPLITUDE and to display uv data phase, select SHOW PHASE. You may also display the difference between the current data sample and a running vector average of the data centered on the current sample and extending no more than plus or minus the “scan length” divided by two. To display the amplitude of the vector difference, select SHOW DIFF AMPL. Such displays are particularly sensitive to short-term problems while ignoring longer-term changes due to source structure. Since it takes time to compute things for, and display, the second data set, you may wish to turn it off part of the time. The 2ND UV OFF and 2ND UV ON options control this choice.

The left-hand menu can contain

FLAG TIME

To delete one time at a time.

FLAG TIME RANGE

To delete one or more time ranges.

FLAG BELOW

To delete all displayed times with data below a cutoff value.

FLAG ABOVE

To delete all displayed times with data above a cutoff value.

FLAG AREA

To delete one or more areas in the data-value vs time plane.

FLAG POINT

To delete one sample at a time using both horizontal and vertical cursor position.

FLAG QUICKLY

To delete samples using only mouse clicks

ENTER AMPL RNG

To select the display range for amplitude plots. Use 0 - 1 for zero to maximum, 00 for minimum to maximum.

ENTER PHASE RNG

To select the display range for phase plots.

ENTER DAMP RNG

To select the display range for plots of the amplitude of the visibility minus a running vector average visibility.

PLOT ERROR BARS

To plot error bars based on data weights.

SET SCAN LENGTH

To set the averaging time used to determine the running average in seconds.

LIST FLAGS

To list all flags now in the Flag Command table.

UNDO FLAGS

To undo one of the flag operations in the FC table

REDO FLAGS

To reapply all remaining flags after one or more have been undone

SET REASON

To set the “reason” string to be put in the uv-data flag table.

USE EXPERT MODE

To control the task from the keyboard instead of the menu.

HOLD TV LOAD

To stop updating the TV display with every change of parameter; change several, then select

DO TV LOAD

To update the TV display now and with each change of display parameter.

REPLOT

To do the current plot over again, recomputing the differences from the running mean if appropriate.

EXIT

To exit EDITR, moving the FC table to a uv-data FG table.

ABORT

To exit EDITR, deleting the FC table.

The first seven items select interactive flagging modes to delete all selected data at a single time, over a range of times, over all values below or above a specified value, or within a range of times and values (respectively). When one of these options is invoked, the screen zooms (if set to do so), a line or box appears in the editing window, and a display of the sample (source, time, value) under the cursor appears at the upper left. Follow the instructions in the message window to select and edit data. Note that this is a very good way to look at your data values even if you do not want to delete anything. The FLAG QUICKLY method is very efficient, but it requires caution in its use. Whenever the left mouse button is depressed, the sample closest to the cursor position is flagged. The next three options set the range of amplitudes, phases, and difference amplitudes displayed. These default to the full range in the data (separately and differently for each baseline) and can be set back to default by entering 0 0. The SET SCAN LENGTH option prompts you for a “scan” length in seconds used as the averaging interval for computing the running mean used in the difference displays. A longer scan length takes longer to compute, but is likely to be less noisy and more meaningful as an editing tool. If you are not using the difference display, set the scan length to a short interval. The running mean is not carried between sources and, as a result, is not normally carried across actual scan boundaries.

When you execute a flagging option, one or more lines are written to a flag command (FC) table attached to the input data set. If EDITR dies abnormally, this FC table can even be used in a later session. To list all of the flagging commands now in the table select the LIST FLAGS option. If you decide that you no longer want one of these flags, select UNDO FLAGS and enter the number (from LIST FLAGS) of the undesirable flag command. More than one flag command may apply to the same datum. After undoing flags, it is probably a good idea to select REDO FLAGS first to undo all remaining flags and then to reapply them to the data to make sure that everything is consistent. When the flag commands in the FC table are entered into a normal flag table, a 24-character “reason” is attached which is both descriptive and can even be used in UVFLG when removing entries in the FG table. The SET REASON command prompts you for the reason to be attached to subsequent flag commands. The default reason is the task name, time and date. Normally, EDITR updates the display whenever anything is changed. If you are about to change more than one display parameter (i.e., polarization, IF, antenna, other antennę, frame) before doing more editing, select HOLD TV LOAD to defer the display update until you select the DO TV LOAD option. If the display appears not to be current, select the REPLOT option. Finally, you may exit the program with the EXIT or ABORT options. The former applies your editing to a flag (FG) table attached to the input data set, while the latter discards any editing commands you may have generated.

Note that value-dependent flagging (FLAG BELOW, FLAG ABOVE, and FLAG AREA) use the values currently plotted to make a list of value-independent flag commands, namely a single time for the specified antennę, IFs, polarizations, etc. When a value-dependent flag operation is undone with UNDO FLAGS or redone with REDO FLAGS, it is these value-independent flags which are undone or redone. You may have to undo more commands and then repeat flag commands to get the results you could have gotten by doing the now desired value-dependent command in the first place. You need also to be careful with the ROTATE ALL ANT setting with these value-dependent commands. If one baseline is set, then the commands only apply to the current baseline. If one antenna is set, the commands apply to all baselines to the current main antenna, while if all antennas is set, the commands apply to all baselines. The first two set a clip level, below or above which data are deleted, based on the value of the observable in each baseline independently. The FLAG AREA command, however, only looks at the values of the observable in the main edit baseline and flags those samples from all applicable baselines.

Be careful when choosing EXIT versus ABORT. The former applies the flag commands to a flag table attached to the input uv data, the latter causes the flag commands to disappear without a trace. After EXIT, of course, one may use, edit, or ignore the output flag (FG) table. For single-source files, it may be necessary to run SPLIT to apply the FG table to the data since only some tasks know how to apply FG tables (those with FLAGVER as an adverb).

The colors used by EDITR are those of the various graphics planes when it begins to run. You may change them with the AIPS verb GWRITE to more desirable colors. The planes are:

   Plane     Default RGB      Use  
     1     1.00  1.00  0.00   Main editing and secondary windows  
     2     0.06  1.00  0.00   Comparison baseline data windows  
     3     1.00  0.67  1.00   Menu highlight  
     4     0.00  1.00  1.00   Edit and frame window boundaries  
     5     1.00  0.18  0.18   Flagged data in all windows  
     6     0.60  0.60  1.00   Menu foreground  
     7     1.00  0.80  0.40   Second uv data set if present

You may wish to change the colors to ones that you can see better.