; FLGIT ;----------------------------------------------------------------------- ;! flags data based on the rms of the spectrum ;# TASK SPECTRAL UV CALIBRATION EDITING ;----------------------------------------------------------------------- ;; Copyright (C) 1998-2000, 2003, 2006-2007, 2010 ;; Associated Universities, Inc. Washington DC, USA. ;; ;; This program is free software; you can redistribute it and/or ;; modify it under the terms of the GNU General Public License as ;; published by the Free Software Foundation; either version 2 of ;; the License, or (at your option) any later version. ;; ;; This program is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; ;; You should have received a copy of the GNU General Public ;; License along with this program; if not, write to the Free ;; Software Foundation, Inc., 675 Massachusetts Ave, Cambridge, ;; MA 02139, USA. ;; ;; Correspondence concerning AIPS should be addressed as follows: ;; Internet email: aipsmail@nrao.edu. ;; Postal address: AIPS Project Office ;; National Radio Astronomy Observatory ;; 520 Edgemont Road ;; Charlottesville, VA 22903-2475 USA ;----------------------------------------------------------------------- FLGIT LLLLLLLLLLLLUUUUUUUUUUUU CCCCCCCCCCCCCCCCCCCCCCCCCCCCC FLGIT Flags data based on the rms of uv-data spectra INNAME Input UV file name (name) INCLASS Input UV file name (class) INSEQ 0.0 9999.0 Input UV file name (seq. #) INDISK 0.0 9.0 Input UV file disk unit # OUTNAME Output UV file name (name) OUTCLASS Output UV file name (class) OUTSEQ -1.0 9999.0 Output UV file name (seq. #) OUTDISK 0.0 9.0 Output UV file disk unit #. OUTFGVER -1.0 If >=0 write FG table instead SOURCES Source list QUAL -10.0 Source qualifier -1=>all CALCODE Calibrator code ' '=>all TIMERANG Time range to purge SELBAND Bandwidth to select (kHz) SELFREQ Frequency to select (MHz) FREQID Freq. ID to select. BCHAN 0.0 9999. First channel selected. ECHAN 0.0 9999. Last channel selected. BIF 0.0 100.0 Lowest IF number 0=>all EIF 0.0 100.0 Highest IF number 0=>all SUBARRAY 0.0 1000.0 Subarray, 0=>all DOCALIB -1.0 101.0 > 0 calibrate data & weights > 99 do NOT calibrate weights GAINUSE CL (or SN) table to apply DOPOL -1.0 10.0 If >0 correct polarization. PDVER PD table to apply (DOPOL>0) BLVER BL table to apply. FLAGVER Flag table version DOBAND -1.0 10.0 If >0 apply bandpass cal. Method used depends on value of DOBAND (see HELP file). BPVER Bandpass table version SMOOTH Smoothing function. See HELP SMOOTH for details. OPCODE 'MWFL' median window else fit linear function APARM Cutoff parameters: 1. Clip 2. Flag in BL regions 3. Flag in signal regions 4. Flag flux > A(4) * RMS 5. Flag Re/Im > A(5)*RMS 6. VPOL clip level 7. Width of median window 8. Number median pts averaged 9. Flag all samples with |u| < APARM(9) lambda 10. > 0 => flag cross if parallel bad BPARM 1. Flag BPARM(1) channels around a bad channel 2. Flag all channels if more than BPARM(2) are bad ICHANSEL Select channels to fit: NOTE this is start,end,increment and IF for each region ORDER 0.0 1.0 Order of fit line (0 -> DC) BADDISK 0.0 9999.0 Disks to avoid for scratch ---------------------------------------------------------------- FLGIT Task: This task applies the calibration to spectral-line data and then makes and subtracts a linear fit to the real and imaginary spectra using a specified set of the spectral channels or subtracts a median window filtered version of the spectrum. If the residual exceeds a noise-based cutoff the channel is flagged. The user specifies the threshold appropriate for a 10 sec integration and the test is done taking into account the weight of the data point. Different cutoff levels are applied to channels specified by BOX and channels not included in BOX. Be conservative (i.e. 6 - 8 times the theoretical sigma is a good default choice). Multiple sources may be processed in a single run and calibration and editing, if present, should be applied, particularly for the linear baseline method. It is important to have a good enough bandpass calibration so that the linear fit makes some sense. A relatively narrow median window filter may reduce the need for bandpass calibration. You may now choose between writing out the edited data in a new data set or writing a flag table. If a lot of isolated channels are flagged, an FG table may actually exceed the size of the initial UV data set. Adverbs: INNAME.....Input UV file name (name). Standard defaults. INCLASS....Input UV file name (class). Standard defaults. INSEQ......Input UV file name (seq. #). 0 => highest. INDISK.....Disk drive # of input UV file. 0 => any. OUTNAME....Output UV file name (name). Standard defaults. OUTCLASS...Output UV file name (class). Standard defaults. OUTSEQ.....Output UV file name (seq. #). 0 => highest unique OUTDISK....Disk drive # of output UV file. 0 => highest w room OUTFGVER...< 0 => write a new UV data set >= 0 => write a FG table attached to the input file of version OUTFGVER. If OUTFGVER is = 0 or greater than FGmax (the previously highest FG version number), then a new FG table will be created for the new flags with version FGmax+1. This new table will also contain the flags applied on input (if any) from FG version FLAGVER. If OUTFGVER specifies a pre-existing FG version, then the input flags are not copied even if OUTFGVER and FLAGVER are not equal. SOURCES....Source list. The task loops over all sources specified. '*' = all; a "-" before a source name means all except ANY source named. QUAL.......Only sources with a source qualifier number in the SU table matching QUAL will be used if QUAL is not -1. CALCODE....Sources may be selected on the basis of the calibrator code given in the SU table. ' ' => any calibrator code selected '* ' => any non blank code (cal. only) '-CAL' => blank codes only (no calibrators) anything else = calibrator code to select. NB: The CALCODE test is applied in addition to the other tests, i.e. SOURCES and QUAL, in the selection of sources to process. TIMERANG...Time range of the data to be copied. In order: Start day, hour, min. sec, day, hour, min. sec. Days relative to reference date. SELBAND....Bandwidth of data to be selected. If more than one IF is present SELBAND is the width of the first IF required. Units = kHz. For data which contain multiple bandwidths/frequencies the task will insist that some form of selection be made by frequency or bandwidth. SELFREQ....Frequency of data to be selected. If more than one IF is present SELFREQ is the frequency of the first IF required. Units = MHz. FREQID.....Frequency identifier to select (you may determine which is applicable from the OPTYPE='SCAN' listing produced by LISTR). If either SELBAND or SELFREQ are set, their values override that of FREQID. However, setting SELBAND and SELFREQ may result in an ambiguity. In that case, the task will request that you use FREQID. BCHAN......First frequency channel selected. 0 => 1. ECHAN......Last channel selected. 0 => all BIF........First IF to copy. 0=>all. EIF........Highest IF to copy. 0=>all higher than BIF SUBARRAY...Subarray number to copy. 0=>all. DOCALIB....If true (>0), calibrate the data using information in the specified Cal (CL) table for multi-source or SN table for single-source data. Also calibrate the weights unless DOCALIB > 99 (use this for old non-physical weights). GAINUSE....version number of the CL table to apply to multi-source files or the SN table for single-source files. 0 => highest. DOPOL......If > 0 then correct data for instrumental polarization as represented in the AN or PD table. This correction is only useful if PCAL has been run or feed polarization parameters have been otherwise obtained. See HELP DOPOL for available correction modes: 1 is normal, 2 and 3 are for VLBI. 1-3 use a PD table if available; 6, 7, 8 are the same but use the AN (continuum solution) even if a PD table is present. PDVER......PD table to apply if PCAL was run with SPECTRAL true and 0 < DOPOL < 6. <= 0 => highest. BLVER......Version number of the baseline based calibration (BL) table to appply. <0 => apply no BL table, 0 => highest. FLAGVER....specifies the version of the flagging table to be applied to the data on input and, if FLGIT generates any flags, also written. > 0 => apply version FLAGVER. < 0 or > max => apply none; = 0 apply maximum FG version DOBAND.....If true (>0) then correct the data for the shape of the antenna bandpasses using the BP table specified by BPVER. The correction has five modes: (a) if DOBAND=1 all entries for an antenna in the table are averaged together before correcting the data. (b) if DOBAND=2 the entry nearest in time (including solution weights) is used to correct the data. (c) if DOBAND=3 the table entries are interpolated in time (using solution weights) and the data are then corrected. (d) if DOBAND=4 the entry nearest in time (ignoring solution weights) is used to correct the data. (e) if DOBAND=5 the table entries are interpolated in time (ignoring solution weights) and the data are then corrected. BPVER......Specifies the version of the BP table to be applied 0 => highest numbered table. <0 => no bandpass correction to be applied. SMOOTH.....Specifies the type of spectral smoothing to be applied to a uv database . The default is not to apply any smoothing. The elements of SMOOTH are as follows: SMOOTH(1) = type of smoothing to apply: 0 => no smoothing To smooth before applying bandpass calibration 1 => Hanning, 2 => Gaussian, 3 => Boxcar, 4 => Sinc To smooth after applying bandpass calibration 5 => Hanning, 6 => Gaussian, 7 => Boxcar, 8 => Sinc SMOOTH(2) = the "diameter" of the function, i.e. width between first nulls of Hanning triangle and sinc function, FWHM of Gaussian, width of Boxcar. Defaults (if < 0.1) are 4, 2, 2 and 3 channels for SMOOTH(1) = 1 - 4 and 5 - 8, resp. SMOOTH(3) = the diameter over which the convolving function has value - in channels. Defaults: 1,3,1,4 times SMOOTH(2) used when input SMOOTH(3) < net SMOOTH(2). OPCODE.....'MWFL' => use the median window filter method; otherwise use the spectral baseline method. APARM......Clipping and flagging parameters: APARM(1) Maximum amplitude allowed in the spectrum before fitting. Any channel exceeding this is flagged in advance. APARM(2) Max. residual flux allowed for channels within the baseline fitting regions for unity weight. This is adjusted by the actual weights to account for S/N. This should be 6-8 times the rms noise for a single channel in a 10 second integration. Any channel exceeding this in amplitude after the spectral baseline is subtracted will be flagged on output. APARM(3) Max residual allowed in the non-baseline fitting regions of the spectrum. Any channel exceeding this in amplitude after the spectral baseline is subtracted will be flagged on output. APARM(4) Flag all channels having residual flux > APARM(4) * RMS. 0 => 6. APARM(5) Flag all channels having Real or Imaginary parts > APARM(5) * RMS. 0 => 5. APARM(6) Maximum amplitude allowed in V polarization; any channel exceeding this is flagged in advance of the baseline fitting or median filtering. 0 -> infinite APARM(7) The width of the median window in channels. 0 -> 5. An odd number is recommended, APARM(8) The number of channels in the median window which are averaged to form the "median". 0 -> 1, an odd number < APARM(7) is recommended. APARM(9) All samples having abs (u) < APARM(9) will be flagged before fitting. APARM(10) > 0 => flag cross-hand polarizations if parallel are flagged. BPARM......1. Extend the flag of a channel +- BPARM(1) channels around the bad channel. Thus if BPARM(1) = 5 and channel 43 is bad, channels 38 through 48 will be flagged. 2. If more than BPARM(2) channels are bad in any polarization/IF, flag all channels in that polarization/IF. 0 -> a large number of channels. ICHANSEL...Select up to 20 groups of channels/IF(s) to fit as sets of (Start,end,inc,IF), i.e., ICHANSEL = 6,37,1,0, 92,123,1,0 for two regions applyingto all IFs. The first group for which ICHANSEL(2,i) <= 0 ends the list. Defaults: Any IF having no group assigned to it, gets a group including all channels from BCHAN through ECHAN. ICHANSEL(1,j) and ICHANSEL(2,j) are restricted to be within the range BCHAN through ECHAN ICHANSEL(4,j) <= 0 => this group applies to all IFs. ORDER......Normally the fit should be done with a first order polynomial, namely a DC term and a slope. If only one box is used, the slope may not be adequately defined and a simple DC term for the real and for the imaginary parts would be more reliable. BADDISK....A list of disks on which scratch files are not to be placed. ---------------------------------------------------------------- FLGIT: Task which subtracts continuum from channels in UV-plane to determine entries in a flag table from scatter of residuals. Documenter: Juan M. Uson (juson@nrao.edu) PURPOSE FLGIT fits and removes the continuum emission in the UV-plane. FLGIT fits to the real and imaginary parts of the visibility. The residuals are checked in order to generate flags if the scatter is excessive. This is done in order to clean-up the data prior to the generation of BP tables to be used for calibration. The fit is performed using the specified channels. FLAGGING of the data on the basis on discrepancy in the fit is performed using FLUX: the maximum error allowed (Jy). This is very useful for removing narrow band interference. The number specified is the limit per unit weight. This is adjusted by 1/sqrt[visibility weight] to correct for integration time differences. If FILLM has been run so that the weight of the data does NOT take into account the system temperature, this weight is normally (tint/10 sec). FLGIT will assume this to be the case. If not, you might have to adjust FLUX accordingly although in the case of the VLA, the Tsys values are only OK as relative values for any given antenna over the course of the run and are not absolutely calibrated (they can be off by factors of up to three depending on the actual frequency used). Channel selection for fitting and flagging is done with NBOXES pairs of channel numbers given in BOX e.g.: NBOXES=3 ; BOX=2, 6, 9, 12, 14, 14 specifies that channels 2, 3, 4, 5, 6, 9, 10, 11, 12, and 14 are used for the fit and then examined for the rms of the residual. Note that the flagging is applied to all channels of the visibility point being examined (e.g., baseline 7 with 8, Stokes LL, all channels for a given time stamp). This is done in order to obey the assumptions underlying the calibration of the data. Notes on STRATEGY: FLGIT should be used to clean-up RFI from the multi- source data prior to generating the BP tables which will be used for the calibration. However, running the task efficiently requires an initial BP table as otherwise the bandpass shape will determine the residuals to the fit to a large extent. This initial BP table can be generated with SOLINT=-1 (in BPASS). If RFI is pervasive, even this initial table can be so contaminated that it will induce large residuals in most visibility points. In that case, start with a high value for FLUX in order to eliminate the worst part of the data, generate a new BP table and iterate as needed. FLGIT will report to the message file on visibilities present as well as on the percentage that it flags. It also reports on how many times any given channel triggered a flag. Notice that any one visibility might be flagged on triggers by several channels. Use the output to decide on threshold level and trigger channels as discussed in the next paragraph. If a channel suffers heavily from RFI, it might cause a great number of flags. If you need that channel to do the science, you might have to accept the flagging of a large number of visibilities in order to generate an interference-free bandpass table. However, if that particular frequency is not needed for the research, it might be best to exclude it from the fit and flagging. As long as the data have been Hanning-smoothed (on-line or off-line), the RFI should not spread its effect to neighboring channels and all that will be lost will be that particular one, which will have to be ignored in the spectral cube. Other tasks can be used to help with a contaminated, but needed, channel. (See for example UVNOU). FLGIT will work on single-source data but it is best to use UVLIN on such data. REFERENCES See Cornwell, Uson and Haddad (Astron. Astrophys. 258, 583; 1992) for a detailed discussion of UVLIN, which includes estimates of errors.