; PCFLG ;--------------------------------------------------------------- ;! interactive flagging of Pulse-cal data in channel-TB using the TV ;# TASK UV TV-APPL CALIBRATION EDITING vlbi INTERACTIVE ;----------------------------------------------------------------------- ;; Copyright (C) 2016-2017 ;; Associated Universities, Inc. Washington DC, USA. ;; ;; This program is free software; you can redistribute it and/or ;; modify it under the terms of the GNU General Public License as ;; published by the Free Software Foundation; either version 2 of ;; the License, or (at your option) any later version. ;; ;; This program is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; ;; You should have received a copy of the GNU General Public ;; License along with this program; if not, write to the Free ;; Software Foundation, Inc., 675 Massachusetts Ave, Cambridge, ;; MA 02139, USA. ;; ;; Correspondence concerning AIPS should be addressed as follows: ;; Internet email: aipsmail@nrao.edu. ;; Postal address: AIPS Project Office ;; National Radio Astronomy Observatory ;; 520 Edgemont Road ;; Charlottesville, VA 22903-2475 USA ;----------------------------------------------------------------------- PCFLG LLLLLLLLLLLLUUUUUUUUUUUU CCCCCCCCCCCCCCCCCCCCCCCCCCCCC PCFLG: edit pulse-cal data using the TV display and cursor INNAME UV data (name). INCLASS UV data (class). INSEQ 0.0 9999.0 UV data (seq. #). 0 => high INDISK Disk unit #. 0 => any INVERS 0.0 PC table version number DOCAT -1.0 2.0 Catalog work file ? IN2SEQ 0.0 Sequence number of work file IN2DISK Disk number of work file DOHIST -10.0 1.0 Record flags in history file SOURCES Source list CALCODE Calibrator code ' '=>all TIMERANG Time range to include BIF 0.0 100.0 Lowest IF number 0=1 EIF 0.0 100.0 Highest IF number ANTENNAS Antennas to include SUBARRAY Subarray to consider DOCENTER > 0 -> center the grey-scale in the TV, else avoid menu BADDISK Disks to avoid for scratch and for master grid file. ---------------------------------------------------------------- PCFLG Type: Task Use: Grids pulse-cal data to form an image with pulse-cal number on the horizontal axis (essentially a frequency axis) and time on the vertical axis. After making this "master" grid, PCFLG displays the grid on the TV smoothing the data in time and showing every n'th pixel in channel as needed to fit the display. Then, PCFLG displays its interactive options: fiddle colors and contrast, change zoom, set windows (and reload the new subimage with less time smoothing), and select cells to flag. PCFLG can fit the phases for delays and display either the input data or the residual data (selection can be made interactively). After some flagging, the input data may be fit over again to adjust the residual values. **** When viewing phases, select VIEW RESID to see phases which can be interpreted by humans. VIEW INPUT phases often go through a large part of a turn between channels. After significant flagging, you will probably want to REDO DELAY. Note that PCFLG flags the pulse-cal data in a new PC table, not the visibility data. This last is, of course, the main function. The grid file may be cataloged and kept for more than one session of interactive editing. All commands to flag data are written to a special FC extension to the master grid file. As a result, they are not lost when the TV hangs up, the power fails, the program aborts, or the user exits the program willingly. Commands in the FC file may be deleted and their effects on the master grid file undone. The actual flagging of the pulse-cal table is done at the end of all interactive operations at the users' command. Adverbs: INNAME.....UV file name (name). Standard defaults. INCLASS....UV file name (class). Standard defaults. INSEQ......UV file name (seq. #). 0 => highest. INDISK.....Disk unit #. 0 => any. DOCAT......True (>0) means to use a cataloged file for the master grid. It creates a new one or uses an old one, depending on the value of IN2SEQ. If DOCAT = 2 and IN2SEQ = 0, then the program returns to the user immediately, prepares the cataloged master grid file, and then exits. The program must be rerun with an appropriate, non-zero value of IN2SEQ to flag the data. This option allows the user to do other things while creating a large master grid file. IN2DISK is used for cataloged master grid files with BADDISK used for scratch and, if IN2DISK = 0, cataloged master grid files. IN2SEQ.....Sequence number of cataloged master grid file. 0 => create new one, else use existing one of seq number = IN2SEQ. NOTE: to use this option, all of the adverbs must have the same value as when the old grid file was created. IN2DISK....Disk number of cataloged work file. 0 => use any disk for pre-existing file and any disk not listed in BADDISK (independent of DOCAT) to create a new cataloged file. Ignored when DOCAT <= 0. DOHIST.....> 0 => record task execution and flagging info in the history file (this can be a lot!). <= 0.0 means to omit the flagging info and <= -9.5 means to omit the execution information as well. SOURCES....List of sources to be gridded. ' '=> all; if any starts with a '-' then all except ANY source named. CALCODE....Sources may be selected on the basis of calibrator code: ' ' => any calibrator code selected '* ' => any non blank code (cal. only) '-CAL' => blank codes only (no calibrators) anything else = calibrator code to select. TIMERANG...Time range of the data to be gridded. In order: Start day, hour, min., sec, end day, hour, min., sec. in days relative to reference date. BIF........Lowest IF to grid. IFs are now gridded along with spectral channels in one longer horizontal axis. If this is undesirable, run PCFLG 1 IF at a time. This is only a little more painful than the previous treatment of each IF as a separate cube in the grid. EIF........Highest IF to grid. 0 => highest. ANTENNAS...A list of the antennas to grid. If any number is negative then all antennas listed are NOT desired and all others are. All 0 => grid all. DOCENTER...> 0 => display the grey-scale approximately at the center of the TV window even if this makes it overlap the menu. <= 0 => display the grey-scale to the right of most of the menu if at all possible. BADDISK....Disk number(s) to avoid for scratch files AND, if IN2DISK is zero, for the master disk file (independent of DOCAT). ---------------------------------------------------------------- PCFLG has three main steps: (1) Grid the data to a master grid with channels/IFs on the x axis and time on the y axis. The cell size in time is set by the data and data are not averaged in time. This master grid saves the real and imaginary parts of the pulse-cal data separately along with a flag number for each sample. Each data sample is accompanied by the real and imaginary values of the input data corrected by the fit delay and center phase. Each data sample also contains its frequency so that new fits may be found and revised residuals computed. The master grid will not be completely regular in its times. No more than 1 blank row will appear between scans and different sources cannot be in the same row. Thus the y axes in the master grid (and in the TV grid) will be somewhat irregular. (2) The interactive session(s) in which the user selects data to be flagged and alters the TV display of the data to assist in the process. (3) The actual flagging of the data and writing of the history file. A new PC table is written containing all input PC data but with certain samples magic blanked. All steps may be done in a single execution. However, if DOCAT is used, then they may be done in several sessions. Most operations on the master grid are relatively efficient and reference only small parts of that file. However, some read and write the whole file (all times, Stokes, antennas, etc.) to ensure that the operation is truly complete. Breaking your data set up into smaller pieces will speed such operations greatly without compromising the editing. This is now less inportant as computers have become more capable. The interactive session is driven by a menu which is displayed on the TV graphics screen. Move the cursor to the desired operation (noting that the currently selected one is highlighted in a different color on many TVs) and press any button to select the operation. The first (left-most column) of choices is: -------------- | OFFZOOM | turn off any zoom magnification | OFFTRANS | turn of any black & white enhancement | OFFCOLOR | turn of any pseudo-coloring | TVFIDDLE | as in AIPS, allows zoom, pseudo-color contours or black and white enhancement | TVTRANSF | black and white enhancement as in AIPS | TVPSEUDO | many pseudo-colorings as in AIPS | DO WEDGE ? | switches choice of displaying a step wedge | LOAD xxxx | switch TV load transfer function to xxxx | LIST FLAGS | list selected range of flag commands | UNDO FLAGS | remove flags by number from FC table and from the master grid | REDO FLAGS | reapply all flags to master grid | REDO DELAY | recompute the fits for delays and center phases and recompute the corrected (residual) PC values | VIEW RESID | to switch from viewing the input data to viewing the residual data (only phase differs) | VIEW INPUT | to switch from viewing the residual data to viewing the input data (only phase differs) | DO LABEL ? | turns on/off axis labeling -------------- Note: when a flag is undone, all cells in the master grid which were first flagged by that command are restored to use. Flag commands done after the one that was undone may also, however, have applied to some of those cells. To check this and correct any improperly unflagged pixels, use the REDO FLAGS option. This option even redoes CLIP operations! After an UNDO or REDO FLAGS operation, the TV is automatically reLOADed if needed. Note that the UNDO operation is one that reads and writes the full master grid. The load to the TV for all non-phase displays may be done with all standard transfer functions: LINear, LOG, SQRT, LOG2 (more extreme log). The menu shows the next one in the list through which you may cycle. The TV is reloaded immediately when a new transfer function is selected. Only one of the VIEW yyyyy options will appear, depending on which form you are currently viewing. Note that the two forms differ only in phase but that that affects vector RMS and amplitude of the vector difference as well as the phase display types. Column 2 offers type-in controls of the TV display and controls of which data are to be flagged. In general, the master grid may be too large to display on the TV screen in its entirety. The program begins by loading every n'th channel and time smoothing by m time intervals in order to fit the full image on the screen. However, you may select a sub-window in order to see the data in more detail. You may also control the range of intensities displayed (like the adverb PIXRANGE in TVLOD inside AIPS). The averaging time to smooth the data for the TV display may be chosen, as may the averaging time for the "scan average" used in some of the displays. Which correlators are to be flagged by the next flagging command may be typed in. All of the standard STOKES values, plus any 4-bit mask may be entered (see HELP file above). The antenna number may be typed in. Flagging may be done only for the current antenna and IF and source, or it may be done for all antennas and/or IFs and/or sources. Note that these controls affect the next LOADs to the TV or the flagging commands prepared after the parameter is changed. When the menu of options is displayed at the top of the TV, the current selections are shown along the bottom. If some will change on the next load, they are shown with an asterisk following. ---------------------- | ENTER BLC | Type in a BLC on the terminal | ENTER TRC | Type in a TRC on the terminal | ENTER AMP PIXRANGE | Type in the intensity range to be used for loading amplitude images to the TV | ENTER PHS PIXRANGE | Type in the phase range to be used for loading phase images to the TV | ENTER RMS PIXRANGE | Type in the intensity range to be used for loading images of the rms to the TV | ENTER R/M PIXRANGE | Type in the value range to be used for loading rms/mean images to the TV | ENTER SMOOTH TIME | To enter the time smoothing length in units of the master grid cell size | ENTER SCAN TIME | To enter the time averaging length for the "scan average" in units of the master grid cell size | ENTER ANTENNA | To enter a desired antenna using the terminal | ENTER STOKES FLAG | To type in the 4-character string which will control which correlators (polarizations) are flagged. Note: this will apply only to subsequent flagging commands. It should be changed whenever a different Stokes is displayed. | ENTER CH SMOOTH | To type in the FWHM of a Gaussian smoothing in spectral channels in the data type being loaded. | SWITCH SOURCE FLAG | To switch between having all sources flagged by the current flag commands and having only those sources included in this execution of PCFLG flagged. The former is desirable when a time range encompasses all of 2 calibrator scans. | SWITCH ALLANT FLAG | To switch between flagging only the current antenna and flagging all antennas. | SWITCH ALL-IF FLAG | To rotate the flag all IFs status from one IF to a range of IFs, to all IFs. Applies to subsequent flag commands. ---------------------- The all-IF flag remains true if the input data set has no more than one IF and the all-source flag remains true if the file has only one source. An extra word should be said about the "scan average". This is used solely for displaying the difference of the data at time T and the average of the data at times near T. This average is computed with a "rolling buffer". Thus for a scan average time of 30 seconds and data at 10 second intervals, the average for a set of 7 points is as follows: time: average of times: 00 00 10 20 10 00 10 20 20 10 20 30 30 20 30 40 40 30 40 50 50 40 50 60 60 40 50 60 The third column of options is used to control which data are displayed and to cause the TV display to be updated. The master grid must be converted from complex to amplitude or phase for display. Using either scalar or vector averaging, it may be converted to the rms of the amplitude or the rms divided by the mean of the amplitude. It may also be converted to the amplitude of the vector difference between the current observation and the "scan average" as defined above or the absolute value of the difference in amplitude with the scalar-average amplitude or the absolute value of the difference in phase with the vector scan average. This column has the options: ---------------------- | DISPLAY AMPLITUDE | To display amplitudes on the TV | DISPLAY PHASE | To display phases on the TV | DISPLAY RMS | To display scalar amplitude rms on the TV | DISPLAY RMS/MEAN | To display scalar amplitude rms/mean on the TV | DISPLAY VECT RMS | To display vector amplitude rms on the TV | DISPLAY VRMS/VAVG | To display vector amplitude rms/mean on the TV | DISPLAY AMP V DIFF | To display the amplitude of the difference between the data and a running (vector) "scan average" | DISPLAY AMPL DIFF | To display the abs(difference) of the amplitude of the data and a running scalar average of the amplitudes in the "scan" | DISPLAY PHASE DIFF | To display the abs(difference) of the phase of the data and the phase of a running (vector) "scan average" | DISPLAY STOKES xx | To switch to Stokes type xx (where xx can be RR, LL, XX, YY) - cross-hands do not occur in PC table data. | OFF WINDOW + LOAD | Reset the window to the full image and reload the TV | SET WINDOW + LOAD | Interactive window setting (like TVWINDOW) followed by reloading the TV. | LOAD LAST ANTENNA | Reload TV with the current parameters and the previous antenna in sequence. | LOAD NEXT ANTENNA | Reload TV with the current parameters and the next antenna in sequence. | LOAD LAST PIECE | Load the previous overlapping piece of the data | LOAD NEXT PIECE | Load the next overlapping piece of the data | LOAD | Reload TV with the current parameters. ---------------------- SET WINDOW + LOAD is "smarter" than TVWINDOW and will not let you set a window larger than the basic image. Therefore, if you wish to include all pixels on some axis, move the TV cursor outside the image in that direction. The selected window will be shown. LOAD LAST PIECE and LOAD NEXT PIECE appear only when there are too many times to fit on the TV screen at the current smoothing parameter. It lets you load one piece at a time in sequence. The pieces will overlap somewhat. The fourth column is used to select the type of flagging to be done. During flagging, a TV graphics plane is used to display the current pixel much like CURVALUE in AIPS. Buttons A and B do the flagging (except A switches corners for the area and time range modes). Button C also does the flagging, but the program then returns to the main menu rather than prompting for more flagging selections. Button D exits back to the menu without doing any additional flagging. Another graphics plane is used to show the current area/time/antenna being flagged. All flagging commands can create zero, one, two, or more entries in the flagging list; hit button D at any time. There are also two clipping modes, an interactive one and one in which the user enters the clip limits from the terminal. In both, the current image computed for the TV (with user-set windows and data type, but not any other windows or alternate pixels etc. required to fit the image on the TV) is examined for pixels which fall outside the allowed intensity range. Flagging commands are prepared and the master file blanked for all such pixels. In the interactive mode, buttons A and B switch between setting the lower and upper clip limits, button C causes the clipping to occur followed by a return to the main menu, and button D exits to the menu with no flagging. The options are ------------------- | FLAG PIXEL | To flag single pixels | FLAG/CONFIRM | To flag single pixels, but request a yes or no on the terminal before proceding | FLAG AREA | To flag a rectangular area in Channel-T | FLAG TIME RANGE | To flag all channels for a range of times | FLAG CHANNEL-DT | To flag a channel for a range of times | FLAG TIME | To flag all channels for a specific time | FLAG A CHANNEL | To flag all times for a specific channel | CLIP BY SET #S | To enter from the terminal a clipping range for the current mode and then clip | CLIP INTERACTIV | To enter with the cursor and LUTs a clipping range for the current mode and then clip the data. | CLIP BY FORM | To clip selected antennas using the "method" and clipping range of some previous clip operation. ------------------- The last operation allows you to apply a clipping method already used on one antenna to other antennas and/or Stokes. CLIP BY FORM asks for a command number (use LIST FLAGS) and applies its display type (amp, phase, rms, rms/mean), averaging interval and clip levels to a range of antennas and Stokes (as entered from the terminal). To terminate the operation, doing nothing, enter a letter instead of one of the requested antenna numbers. To omit a Stokes, reply, if requested for a flag pattern, with a blank line. You may watch the operation being carried out on the TV as it proceeds. The right-most column has only the option: -------- | EXIT | Go resume AIPS and enter the flags in the data -------- Before the flags are entered in the data, PCFLG asks the user whether or not he actually wishes to do this. You must respond yes or no. When the menu is displayed, status line(s) are also displayed on the TV to show the averaging time, corners, antenna/IF, Stokes, etc. of the currently displayed data. If these will change on the next load, an asterisk is show after the values. The building of the master file can be a lengthy process for large uv data sets. If you are going to run PCFLG more than once - WITH THE SAME PARAMETERS - then use the DOCAT and IN2SEQ options to catalog the master file during the first execution and to restart with that file in later executions. Note that DOCAT = 2 with IN2SEQ = 0, allows immediate resumption of AIPS since no interactive flagging will be done. PCFLG keeps track of the source name associated with each row of data. When averaging to build the master grid and to build the displayed grids, PCFLG will not average data from different sources and will inform you that it has omitted data if it has had to do so for this reason. For multi-source files, the source name is displayed during the CURVALUE-like sections.