; RFLAG ;--------------------------------------------------------------- ;! Flags data set based on time and freq rms in fringe visibilities ;# Task UV EDITING VLA ;----------------------------------------------------------------------- ;; Copyright (C) 2011-2013 ;; Associated Universities, Inc. Washington DC, USA. ;; ;; This program is free software; you can redistribute it and/or ;; modify it under the terms of the GNU General Public License as ;; published by the Free Software Foundation; either version 2 of ;; the License, or (at your option) any later version. ;; ;; This program is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; ;; You should have received a copy of the GNU General Public ;; License along with this program; if not, write to the Free ;; Software Foundation, Inc., 675 Massachusetts Ave, Cambridge, ;; MA 02139, USA. ;; ;; Correspondence concerning AIPS should be addressed as follows: ;; Internet email: aipsmail@nrao.edu. ;; Postal address: AIPS Project Office ;; National Radio Astronomy Observatory ;; 520 Edgemont Road ;; Charlottesville, VA 22903-2475 USA ;----------------------------------------------------------------------- RFLAG LLLLLLLLLLLLUUUUUUUUUUUU CCCCCCCCCCCCCCCCCCCCCCCCCCCCC RFLAG Flags data sets based on the cross-hand visibilities INNAME Input UV file name (name) INCLASS Input UV file name (class) INSEQ 0.0 9999.0 Input UV file name (seq. #) INDISK 0.0 9.0 Input UV file disk unit # SOURCES Source name QUAL -10.0 Calibrator qualifier -1=>all CALCODE Calibrator code ' '=>all TIMERANG Time range to use SELBAND Bandwidth to select (kHz) SELFREQ Frequency to select (MHz) FREQID Freq. ID to select. SUBARRAY 0.0 1000.0 Sub-array, 0=>all DOCALIB -1.0 101.0 > 0 calibrate data & weights > 99 do NOT calibrate weights GAINUSE CL (or SN) table to apply DOPOL -1.0 10.0 If >0.5 correct polarization. PDVER PD table to apply (DOPOL>0) BLVER BL table to apply. FLAGVER Flag table version DOBAND -1.0 10.0 If >0.5 apply bandpass cal. Method used depends on value of DOBAND (see HELP file). BPVER Bandpass table version SMOOTH Spectral smoothing function Dangerous here - see help. OUTFGVER 0.0 Output flag table: 0 new STOKES Stokes type ' ' => 'RLLR' UVRANGE Restrict baselines in klambda BIF Begin IF to process this time EIF End IF to process this time DCHANSEL Array of start and stop chan numbers, plus a channel increment and IF omitted from all operations. ICHANSEL Array of start and stop chan numbers, plus a channel increment and IF to be used to select channels to sum to find spectral rms. AVGCHAN 0.0 16384.0 Width of median window in the spectral flagging mode DOSCALE -1.0 1.0 > 0 -> determine relative gains first DOPLOT -16.0 16.0 > 0 -> plot, else flag data (see help) < 0 -> flag data, plot after DOFLAG > 0 -> flag when DOPLOT > 0 FPARM (1) > 0 rolling T buffer to find time rms F(1) times (2) Normal time interval between samples (sec) IMPORTANT TO BE ~RIGHT (3) Maximum rms allowed in Jy in time rms (4) if not 0, do robust rms spectrally and flag channels deviating > F(4) (5) NEW - see HELP # channels taken from each spectral window (6) NEW - see HELP expand flagged region (7) flag whole spectrum if fraction flagged > F(7) (8) flag F(8) channels between flag groups (9) scale mean rms to compute NOISE, 0 -> 5. (10) scale mean deviation to compute SCUTOFF. 0 -> 5 (11) Flag all baselines if a channel > F(11) flagged (12) Flag all baselines to an antenna if a channel > F(12) flagged (13) Flag any visibility amp > F(13) before time rms 0 -> 1.E6 (14) Flag times with fewer than F(14) baselines (15) Flag baseline if any vis > FPARM(15) (16) Flag both ants if any vis > FPARM(16) (20) >= 1 => flag only center time in bin VPARM (1) Time rms upper limit to histogram (Jy) 0 -> 2 (2) Spectral rms upper limit to histogram (Jy) (3) Plot only lower V(3) portion of plots 0 -> 1 (4,5) Min/max time box to be plotted (6) Max Y time histogram (7) Max Y time cumulative (8,9) Min/max spectral box to be plotted (10) Max Y of spectral plot (11) Max Y cumulative spect. (12,13) min/max of average rms versus channel plot (14,15) min/max of average deviation vs channel plot NOISE $ If FPARM(3) < 0, NOISE(i) is the max allowed in time rms for IF i (Jy) Returned when DOPLOT not 0 SCUTOFF $ If FPARM(4) < 0, SCUTOFF(i) is max allowed deviation in a channel for IF i (Jy) Returned when DOPLOT not 0 DOTV -1.0 1.0 > 0 Do plot on the TV, else make a plot file GRCHAN 0.0 7.0 Graphics channel 0 => several LTYPE -10.0 10.0 Type of labeling: 1 border, 2 no ticks, 3 standard, 4 rel to center, 5 rel to subim cen 6 pixels, 7-10 as 3-6 with only tick labels <0 -> no date/time FUNCTYPE 'LG' plot log of counts else plot is fully linear XYRATIO 0.0 X/Y ratio 0 -> fit TV or 1 PL NBOXES 0.0 20000.0 Number histogram boxes DOOUTPUT -1.0 3.0 > 0, write OUTTEXT w weights OUTTEXT Text file of channel weights ---------------------------------------------------------------- RFLAG Task: This task examines SELECTED visibility spectra looking for RFI primarily. It examines the rms versus time over short time intervals a polarization and a spectral channel at a time. Those exceeding user-specified threshold(s) are flagged in all polarizations. Optionally, the robust average and rms of the real and imaginary parts of the visibility at each time are found in each IF individually. Those channels deviating from this average by more than a user-specified cutoff in Jy are also flagged in all polarizations. To aid in setting the cutoff values, RFLAG can begin by computing histograms of the time rms and of the spectral deviations. It can also compute means and rmses of the time rms and of the deviations from the spectral fits as a function of spectral channel and IF. The computations done for the latter set adverb values (NOISE, SCUTOFF) which may be used for the actual flagging. The histograms, cumulative histograms, and spectral functions may be plotted. Following the plots, the program can go on to flag the data immediately or it can leave that operation for a later execution. Note that the cutoffs are all in Jy. That may make particular values suitable only for sources within a modest range of fluxes. RFLAG may then need to be used multiple times, once for strong calibrators, once for modest calibrators, and one more time for weak target sources. It may be better to use, e.g. 'RLLR' Stokes or 'QU' for this purpose than I which is likely to have larger signals perhaps made "noisier" by delay rocking and other gain-like errors. Stokes V may suffer from errors in calibration causing spurious values way larger than the true V polarization (if any). Note one gotcha: if one of RR and LL is known to be bad, it has been traditional to flag it and both cross hands. If you use only cross-hand polarizations, times previously flagged in this manner will not be examined. STOKES='HALF' may be needed but with cutoff values different from when using STOKES='RLLR'. Adverbs: INNAME.....Input UV file name (name). Standard defaults. INCLASS....Input UV file name (class). Standard defaults. INSEQ......Input UV file name (seq. #). 0 => highest. INDISK.....Disk drive # of input UV file. 0 => any. SOURCES....Source to be copied. ' '=> all; if any starts with a '-' then all except ANY source named. QUAL.......Qualifier of source to be copied. -1 => all. CALCODE....Calibrator code of sources to copy. ' '=> all. TIMERANG...Time range of the data to be copied. In order: Start day, hour, min. sec, end day, hour, min. sec. Days relative to ref. date. SELBAND....Bandwidth of data to be selected. If more than one IF is present SELBAND is the width of the first IF required. Units = kHz. For data which contain multiple bandwidths/frequencies the task will insist that some form of selection be made by frequency or bandwidth. SELFREQ....Frequency of data to be selected. If more than one IF is present SELFREQ is the frequency of the first IF required. Units = MHz. FREQID.....Frequency identifier to select (you may determine which is applicable from the OPTYPE='SCAN' listing produced by LISTR). If either SELBAND or SELFREQ are set, their values override that of FREQID. However, setting SELBAND and SELFREQ may result in an ambiguity. In that case, the task will request that you use FREQID. SUBARRAY...Sub-array number to copy. 0=>all. DOCALIB....If true (>0), calibrate the data using information in the specified Cal (CL) table for multi-source or SN table for single-source data. Also calibrate the weights unless DOCALIB > 99 (use this for old non-physical weights). GAINUSE....version number of the CL table to apply to multi-source files or the SN table for single source files. 0 => highest. DOPOL......If > 0 then correct data for instrumental polarization as represented in the AN or PD table. This correction is only useful if PCAL has been run or feed polarization parameters have been otherwise obtained. See HELP DOPOL for available correction modes: 1 is normal, 2 and 3 are for VLBI. 1-3 use a PD table if available; 6, 7, 8 are the same but use the AN (continuum solution) even if a PD table is present. PDVER......PD table to apply if PCAL was run with SPECTRAL true and 0 < DOPOL < 6. <= 0 => highest. FLAGVER....specifies the version of the flagging table to be applied. 0 => highest numbered table. <0 => no flagging to be applied. Keep track of your FG table versions to be certain that you are using all of the flags that you intend to use. DOBAND.....If true (>0) then correct the data for the shape of the antenna bandpasses using the BP table specified by BPVER. The correction has five modes: (a) if DOBAND=1 all entries for an antenna in the table are averaged together before correcting the data. (b) if DOBAND=2 the entry nearest in time (including solution weights) is used to correct the data. (c) if DOBAND=3 the table entries are interpolated in time (using solution weights) and the data are then corrected. (d) if DOBAND=4 the entry nearest in time (ignoring solution weights) is used to correct the data. (e) if DOBAND=5 the table entries are interpolated in time (ignoring solution weights) and the data are then corrected. IMAGR uses DOBAND as the nearest integer; 0.1 is therefore "false". BPVER......Specifies the version of the BP table to be applied 0 => highest numbered table. <0 => no bandpass correction to be applied. SMOOTH.....Specifies the type of spectral smoothing to be applied to a uv database . The default is not to apply any smoothing. NOTE: SMOOTH IS VERY DANGEROUS IN THIS CONTEXT. IF ONE USES IT AND FLAGS SOME CHANNELS THE VALUES IN UNFLAGGED CHANNELS CHANGE MAKING RFLAG APPEAR TO HAVE NOT FLAGGED ALL THAT IT SHOULD HAVE. If you realy need SMOOTH to remove ringing from narrow-band signals, run SPLAT on the data to apply the SMOOTH and then set SMOOTH=0 for later work. The elements of SMOOTH are as follows: SMOOTH(1) = type of smoothing to apply: 0 => no smoothing To smooth before applying bandpass calibration 1 => Hanning, 2 => Gaussian, 3 => Boxcar, 4 => Sinc To smooth after applying bandpass calibration 5 => Hanning, 6 => Gaussian, 7 => Boxcar, 8 => Sinc SMOOTH(2) = the "diameter" of the function, i.e. width between first nulls of Hanning triangle and sinc function, FWHM of Gaussian, width of Boxcar. Defaults (if < 0.1) are 4, 2, 2 and 3 channels for SMOOTH(1) = 1 - 4 and 5 - 8, resp. SMOOTH(3) = the diameter over which the convolving function has value - in channels. Defaults: 1,3,1,4 times SMOOTH(2) used when input SMOOTH(3) < net SMOOTH(2). OUTFGVER...Output flag table version. 0 -> new version and the contents of FLAGVER are copied to it initially. n = pre-existing file -> new flags are appended to the OUTFGVER but FLAGVER is not copied. OUTFGVER may point to the same file as FLAGVER. In that case, FLAGVER and any new flags are put in a temporary flag table and finally the temporary flag table replaces the input FLAGVER. STOKES.....Type of Stokes parameters to examine. All Stokes will be flagged whenever one of the examined ones is deemed out of bounds. ' ' => 'RLLR' UVRANGE....Restrict the examination of data to the specified range of baseline lengths in klambda at the reference frequency. UVRANGE(2) <= UVRANGE(1) => UVRANGE(2)=1.E12 BIF........First IF included in operation EIF........Last IF included in operation Plots sum the histograms over included channels and IFs but the time rms is done one spectral channel and IF at a time for computing the rms and for flagging and the spectral fitting is done one IF at a time. DCHANSEL...Array of start, stop, and increment channel numbers plus an IF used for channel DESELECTION. Any group of channels specified will not be examined for time or spectral rms and will not be flagged except when the full spectrum is flagged. Up to 20 sets of channels/IF may be entered. The first having DCHANSEL(2,i) <= 0 terminates the list. DCHANSEL(4,i) is the IF number, with <= 0 meaning all. If an IF has no DCHANSEL set for it, then all channels in the IF are included in the rms and flagging oprations. ICHANSEL.. Array of start, stop, and increment channel numbers plus an IF used for channel selection in the averaging to compute a spectral rms. Up to 20 sets if channels/IF may be entered. The first having ICHANSEL(2,i) <= 0 terminates the list. ICHANSEL(4,i) is the IF number, with <= 0 meaning all IFs. If an IF has no ICHANSEL set for it, then ALL channels will be used. Note that this is a non-standard default appropriate to the sliding median window method now used. AVGCHAN....The spectral mode uses a median window AVGCHAN spectral channels wide where AVGCHAN is forced to be odd. The default is one or two less than the full width of the spectrum. The value at the center of the window is compared to the median value of the window and values more than FPARM(4) or SCUTOFF will be flagged. DOSCALE....> 0 -> Read the data through to find separate amplitude gains for each baseline and IF and apply those to the data when plotting and flagging. The median amplitude of the channels in each spectral window at each sample is taken as the amplitude to be averaged over all times. <= 0 -> do not determine such gains and do not apply them. DOPLOT.....If DOPLOT > 0, histograms of the time rms and of the deviations from the spectral fit are computed. Also computed are the mean time rms and the mean deviation from the spectral fit in each spectral channel. These are then used to set the adverbs NOISE and SCUTOFF. If 16 > DOPLOT > 0, it is broken into a bit pattern and plots corresponding to the bits that have value are generated. bit 1 (1,3,5,...) => Plot the time rms and the spectral deviations as histograms. bit 2 (2,3,6,...) => Plot the cummulative histograms (fraction of data above x) bit 3 (4,5,6,...) => Plot the spectrum of time rms (average and deviation in each channel) versus channel bit 4 (8,9,10,..) => Plot the spectrm of deviation from the spectral fit (average and rms) versus channel = 0 => Do not do the plots and do not change the input values of NOISE and SCUTOFF. DOPLOT < 0 => flag the data with FPARM and possibly NOISE and SCUTOFF. Then plot the data using the new flag table with abs(DOPLOT) providing the plot control as above. DOFLAG.....If DOFLAG > 0 or DOPLOT <= 0, compute the time rmses and optionally spectral fits with deviations and generate flag table entries for the data in excess of the cutoffs. This means that one can (a) compute NOISE and SCUTOFF and then use them (or use FPARM(3) and/or FPARM(4)) (DOPLOT > 0, DOFLAG > 0); (b) use the input NOISE and SCUTOFF (or FPARM(3) and/or FPARM(4)) without changing them (DOPLOT <= 0, and any DOFLAG); (c) compute NOISE and SCUTOFF and generate plots (16 > DOPLOT > 0 and DOFLAG <= 0) with no flagging; and (d) compute NOISE and SCUTOFF and generate plots (DOPLOT >= 16 and DOFLAG <= 0) with no flagging; (e) use the input NOISE and SCUTOFF (or FPARM(3) and/or FPARM(4)) without changing them (DOPLOT < 0 and any DOFLAG) to flag the data follwoed by data plots generated with the new flag table. FPARM......(1) Number of times to be included in the time rms. We recommend an odd number and < 3 -> 3. (2) Normal interval between samples in seconds. The program will not do intervals longer than 2 * FPARM(1) * FPARM(2). 0 -> 10 secs IT IS IMPORTANT TO HAVE THIS ABOUT RIGHT. If it is too short, in particular, then data samples that belong in the same bin (time interval) may end up in different bins. This happens most often when data have been time averaged after selective flagging. Set it to somewhat less than the averaging time - if observations at XINT are averaged over N times, set FPARM(2) = (N-0.5) * XINT. (3) Flag all data having a time rms > FPARM(3) Jy. 0 -> 1.e6. If FPARM(3) < 0.0, use NOISE(i) as the clip level for IF i instead of FPARM(3). If ABS(FPARM(3)) >= 1.E6, the computation of time rms is not done under any circumstances, so no time rms flags, plots, and changed output NOISE values occur. (4) If FPARM(4) not 0, also do a median-window operation over the spectral channels in each IF. Flag those channels which differ from the median by more than FPARM(4) Jy when FPARM(4)>0. If FPARM(4)<0, do this flagging using IF-dependent values SCUTOFF(i) instead of FPARM(4). The computation is done separately for the real and imaginary parts. The width of the window is set by AVGCHAN. 0 -> do not do the spectral computation. (5) The center channel plus FPARM(5) channels on each side of the central channel are evaluated from each spectral median-filter window. That window then moves 1+2*FPARM(5) for the next evaluation - speeding the spectral analysis process. 0 -> AVGCHAN/40, -1 -> 0. If AVGCHAN is say 101, then the default would take 5 channels at a time centered in each window and the next window would start 5 channels later than the previous one. (6) After the time and spectral flagging is evaluated for each time, expand each region of flagged channels by FPARM(6) channels in each direction. This occurs before the application of the options in FPARM(7), FPARM(8), FPARM(11), and FPARM(12). (7) If the fraction of channels to be flagged (and previously flagged) in a given time and IF sample exceeds FPARM(7), flag the whole thing. Note that this looks at the results of the time rms and the spectral rms (if any) plus the FPARM(8) test. It is done both before and after the FPARM(10) test. 0->1 (8) Flag up to FPARM(8) "good" channels located between groups of flagged channels. Thus if we are to flag, for example, channels 12-14, 16-20, 23-24, and 28-30 and FPARM(7)=2, then we will flag 12-24 and 28-30. (9) When DOPLOT > 0, the program sums the time rmses in each channel separately. It then computes the mean and rms of these time rmses in each channel and finds the median average channel rms over each IF and its variance. Adverb NOISE(i) is set to FPARM(9) * (median + variance). 0 -> 5.0 which seems to be a good value. (10) When DOPLOT > 0, the program sums the deviations from the spectral fit (if FPARM(4) not 0.0) in each channel separately. It then computes the mean rms of these deviations in each channel and finds the median average channel deviation over each IF and its variance. Adverb SCUTOFF(i) is set to FPARM(10) * (median + variance). 0 -> 5.0 which may be a good value. (11) If a channel is flagged at one time in > FPARM(11) fraction of the baselines, flag it in all baselines at that time. 0 -> 1 => don't do this. Note that autocorrelations are allowed, so the maximum number of "baselines" is (Mant * (Mant+1)) / 2 where Mant is the maximum antenna number. Thus the EVLA with Mant = 28 has 406 "baselines" rather than 351 when antenna 28 is in the array. (12) If a channel is flagged at one time in > FPARM(12) fraction of the baselines to a particular antenna, flag it in all baselines to that antenna at that time. 0 -> 1 => don't do this. Note that autocorrelations are allowed, so the maximum number of "baselines" to an antenna is the maximum antenna number. (13) Examine the data before doing any other checks and flag all samples > FPARM(13) Jy. 0 -> 1.E8. (14) Examine the data before doing any other checks and flag all baselines when there are fewer than FPARM(14) baselines with valid data, channel by channel, IF by IF. This is called a quack operation but applies at any time, not just the start and end of scans. (15) Examine the data before doing other checks and, if any channel has amplitude > FPARM(15), flag all channels in that baseline and IF. (16) Examine the data before doing any other checks and, if any channel has amplitude > FPARM(16), flag all channels in that IF for every baseline involving the two antennas of the subject baseline. (20) >= 1.0 => flag only the center time in a time bin when the time rms is excessive. This is not recommended when actually doing time rms flagging. If FPARM(3) is very large so that only spectral flagging is actually being done, this option may speed the performance to the spectral code. VPARM......Plot controls when DOPLOT > 0: (1) Highest rms to histogram in the time-based rms's. 0 -> 2 Jy. (2) Highest deviation to histogram in the spectral histogram. 0 -> 2 Jy (3) Plot only the lowest FPARM(10) portion of each plot < 0.001 -> 1.0 Since you are usually interested only in the lower portions, this option is quite useful. (4) Minimum time rms histogram box to plot (5) Maximum time rms histogram box to plot If 0 < VPARM(4) < VPARM(5), the specified range of boxes will be plotted. Otherwise, the lowest box > 0 to the highest box > 0 is the range plotted. (6) > 0 -> Maximum value on Y axis for time rms histogram before VPARM(3). (7) > 0 -> Maximum value on Y axis for time rms cumulative histogram before VPARM(3). (8) Minimum spectral deviation histogram box to plot (9) Maximum spectral deviation histogram box to plot If 0 < VPARM(8) < VPARM(9), the specified range of boxes will be plotted. Otherwise, the lowest box > 0 to the highest box > 0 is the range plotted. (10) Maximum value on Y axis of spectral deviation normal histogram before VPARM(3). (11) > 0 -> Maximum value on Y axis for spectral deviation cumulative histogram before VPARM(3). (12,13) Min and max of plot of mean time rms as a function of channel. (14,15) Min and max of plot of mean spectral deviation as a function of channel. NOISE......DOPLOT <= 0 : If FPARM(3) < 0, NOISE(i) will be used as the maximum allowed time rms for IF i. Units are Jy. DOPLOT > 0 : Value of median time rms plus median rms of rmses times FPARM(9) in each IF. NOISE is a returned adverb value not an input and is plotted as a straight line in each IF when bit 3 of DOPLOT is on (=4,5,6,..). NOISE may then be applied (if FPARM(3) < 0) in the same execution of RFLAG if DOFLAG > 0. SCUTOFF....DOPLOT <= 0 : If FPARM(4) < 0, SCUTOFF(i) will be used as the maximum allowed deviation from the spectral fit for IF i. Units are Jy. DOPLOT > 0 : Value of median spectral deviation plus median rms of devaitions times FPARM(10) in each IF. SCUTOFF is a returned adverb value not an input and is plotted as a straight line in each IF when bit 4 of DOPLOT is on (=8,9,10,..). SCUTOFF may then be applied in the same eecution of RFLAG to generate flags (if FPARM(4) < 0) if DOFLAG > 0. DOTV.......> 0 => plot directly on the TV device, otherwise make a plot file for later display on one or more devices (including the TV if desired). GRCHAN......Graphics channel (1 - 7) to use for line drawing. 0 => use multiple graphics planes for the various parts of the graphical drawing. LTYPE.......Labelling type: 1 = border, 2 = no ticks, 3 or 7 = standard, 4 or 8 = relative to ref. pixel, 5 or 9 = relative to subimage (BLC, TRC) center, 6 or 10 = pixels. 7-10 all labels other than tick numbers and axis type are omitted. Less than 0 is the same except that the plot file version number and create time are omitted. FUNCTYPE....Histogram transfer function: 'LG' => log else linear. X axis (rms in Jy and deviation in Jy) is linear. XYRATIO.....Scale the X axis longer than the Y by XYRATIO. If DOTV > 0, 0 -> fit to the TV window If DOTV <= 0, 0 -> 1. NBOXES......Number of cells for histogram. 20 <= nboxes <= 20000 < 20 -> 200. DOOUTPUT....If DOPLOT > 0 and OUTTEXT is not blank and DOOUTPUT > 0, write a text file with channel dependent weights. DOOUT = 1 => use time rmses only DOOUT = 2 => use freq rmses only DOUUT = 3 or more => use both. OUTTEXT.....See DOOUTPUT above, gives path to desired text file. ----------------------------------------------------------------