; FLAGR
;---------------------------------------------------------------
;! Edit data based on internal RMS, amplitudes, weights
;# TASK CALIBRATION OOP EDITING UV
;-----------------------------------------------------------------------
;;  Copyright (C) 2004-2007, 2010-2011
;;  Associated Universities, Inc. Washington DC, USA.
;;
;;  This program is free software; you can redistribute it and/or
;;  modify it under the terms of the GNU General Public License as
;;  published by the Free Software Foundation; either version 2 of
;;  the License, or (at your option) any later version.
;;
;;  This program is distributed in the hope that it will be useful,
;;  but WITHOUT ANY WARRANTY; without even the implied warranty of
;;  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;  GNU General Public License for more details.
;;
;;  You should have received a copy of the GNU General Public
;;  License along with this program; if not, write to the Free
;;  Software Foundation, Inc., 675 Massachusetts Ave, Cambridge,
;;  MA 02139, USA.
;;
;;  Correspondence concerning AIPS should be addressed as follows:
;;         Internet email: aipsmail@nrao.edu.
;;         Postal address: AIPS Project Office
;;                         National Radio Astronomy Observatory
;;                         520 Edgemont Road
;;                         Charlottesville, VA 22903-2475 USA
;-----------------------------------------------------------------------
FLAGR     LLLLLLLLLLLLUUUUUUUUUUUU CCCCCCCCCCCCCCCCCCCCCCCCCCCCC
FLAGR    Use rms and amplitudes in uv data set to edit the data
INNAME                             Input UV data
INCLASS                            Input UV data (class)
INSEQ                              Input UV data (seq. #)
INDISK                             Input UV data disk drive #
SOURCES                            Source (pointings) list
QUAL            -10.0              Source qualifier -1=>all
CALCODE                            Calibrator code '    '=>all
TIMERANG                           Time range to process.
SELBAND                            Bandwidth to select (kHz)
SELFREQ                            Frequency to select (MHz)
FREQID                             Freq. ID to select.
BIF               0.0      100.0   Lowest IF number 0=>all
EIF               0.0      100.0   Highest IF number 0=>all
BCHAN             0.0     2048.0   Lowest channel number 0=>all
ECHAN             0.0     2048.0   Highest channel number 0=>all
SUBARRAY          0.0     1000.0   Subarray, 0=>all
DOCALIB          -1.0      101.0   > 0 calibrate data & weights
                                   > 99 do NOT calibrate weights
GAINUSE                            CL/SN table to apply
DOPOL            -1.0       10.0   If >0 correct polarization.
PDVER                              PD table to apply (DOPOL>0)
BLVER                              BL table to apply.
FLAGVER                            Flag table version
OUTFGVER          0.0              Output FG table version
DOBAND           -1.0       10.0   If >0 apply bandpass cal.
                                   Method used depends on value
                                   of DOBAND (see HELP file).
BPVER                              Bandpass table version
SMOOTH                             Smoothing function. See
                                   HELP SMOOTH for details.
UVRANGE         0.                 Min & max baseline (klambda)
SOLINT                             Time interval (sec)
SCANLENG                           Scan length (sec)
REFANT                             Reference antenna for VDIF
OPTYPE                             Type of operation on the rms
                                   'RFI ' 'HIST' 'ANTE' 'GAIN'
                                   'VDIF' 'VRFI'
                                   'TIME' is default
DOCAT            -1.0        1.0   > 0 => keep the XX file
VECTOR           -1.0        1.0   > 0 -> vector averaging
                                   <= 0 -> use amplitudes only
DOROBUST         -1.0        1.0   > 0 -> robust spectral avg
BPARM                              (1) Fraction allowable ??
                                       correlators
                                   (2) Fraction allowable ??
                                       baselines
                                   (3) rms cutoff (Jy) for ???
                                   (4) Amp. factor for cutoff
                                   (5) rms cutoff (Jy) for bad
                                   (6) Amp. factor for cutoff
                                   (7) Min hist level Jy for ???
                                   (8) Min hist level Jy for bad
                                   (9) > 0 save 1-sample SOLINTs
                                   (10) best left <= 0.
CPARM                              (1) Question if amp<CPARM(1)
                                   (2) Question if amp>CPARM(2)
                                   (3) Flag if amp < CPARM(3)
                                   (4) Flag if amp > CPARM(4)
                                   (5) min amp rms allowed
                                   (6) min rms rms allowed
                                   (7) Question if amp >
                                       CPARM(7)*rms away from
                                       mean amplitude
                                   (8) Question if rms >
                                       CPARM(8)*(rms of rms's)
                                       away from mean rms
                                   (9) Flag if amp >
                                       CPARM(8)*rms away from
                                       mean amplitude.
                                   (10) Flag if rms >
                                       CPARM(10)*(rms of rms's)
                                       away from mean rms.
DPARM                              (1) Question if wt<DPARM(1)
                                   (2) Question if wt>DPARM(2)
                                   (3) Flag if wt < DPARM(3)
                                   (4) Flag if wt > DPARM(4)
                                   (5) Question if closure
                                       fraction > DPARM(5)
                                   (6) Flag if closure error
                                       fraction > DPARM(6)
STOKES                             Correlators examined
                                   ' ' => RR/LL, XX/YY, I
DOSTOKES         -1.0        4.0   -1 flag only bad corr.
                                   1 flag cross if parallel bad
                                   2 flag parallel if cross bad
                                   3 both 1 and 2
                                   4 flag all Stokes if one bad
                                   0 -> 3
DOIFS            -1.0        1.0   > 0 -> flag all IFs if one
                                   is bad
DOALL            -1.0        1.0   > Flag all channels, else
                                   flag BCHAN - ECHAN
DOCRT            -3.0     132.0    > 0 -> use the terminal,
                                   > 72 => terminal width
                                   0 -> no printing
                                   else use the line printer
OUTPRINT
                                   Printer disk file to save
PRTLEV                             > 0 => debug messages
BADDISK           -1.0      1000.0 Disks to avoid for scratch.
----------------------------------------------------------------
FLAGR
Type:  Task
Use:   This task computes the visibility amplitude and rms
       fluctuations in each baseline and correlator over a specified
       range of spectral channels and time.  It can then convert the
       baseline-based quantities into apparent antenna-based
       quantities.  In some of the OPTYPEs these are then compared
       with user-specified cutoffs.  If a large enough fraction of the
       data are "questionable" then all of those data are flagged.  If
       a given baseline/correlator or antenna/correlator is "bad", it
       will be flagged.  A time-based OPTYPE saves the amplitudes and
       rmses and then reviews all of them over time, flagging
       discrepant ones on a source-by-source and antenna-by-antenna
       basis.  Another time-based OPTYPE compares the complex
       visibilities on an antenna basis with the average of those in a
       range of time surrounding.
Adverbs:
  INNAME.....Input UV data file
  INCLASS....Input UV data file (class).      Standard defaults.
  INSEQ......Input UV data file (seq. #).     0 => highest.
  INDISK.....Input UV data file disk drive #. 0 => any.
  SOURCES....List of sources (pointings) to be processed.
             '*' or blank = all; a "-" before a source name means all
             except ANY source named.
  QUAL.......Only sources with a source qualifier number in the SU
             table matching QUAL will be used if QUAL is not -1.
  CALCODE...."sources" may be selected on the basis of the calibrator
             code given in the SU table.
                  '    ' => any calibrator code selected
                  '*   ' => any non blank code (cal. only)
                  '-CAL' => blank codes only (no calibrators)
             NB: The CALCODE test is applied in addition to the other
             tests, i.e. SOURCES and TIMERANG, in the selection of
             sources to process.  CALCODE affects only the selection
             of calibrators.
  TIMERANG...Time range of the data to be processed. In order: Start
             day, hour, min. sec, end day, hour, min. sec. Days
             relative to reference date.
  SELBAND....Bandwidth of data to be selected. If more than one IF is
             present SELBAND is the width of the first IF required.
             Units = kHz. For data which contain multiple
             bandwidths/frequencies the task will insist that some
             form of selection be made by frequency or bandwidth.
  SELFREQ....Frequency of data to be selected. If more than one IF is
             present SELFREQ is the frequency of the first IF
             required. Units = MHz.
  FREQID.....Frequency identifier to select (you may determine which
             is applicable from the OPTYPE='SCAN' listing produced by
             LISTR). If either SELBAND or SELFREQ are set, their
             values overide that of FREQID.  However, setting SELBAND
             and SELFREQ may result in an ambiguity.  In that case,
             the task will request that you use FREQID.
  DOCALIB....If true (>0), calibrate the data using information in the
             specified Cal (CL) table for multi-source or SN table for
             single-source data.  Also calibrate the weights unless
             DOCALIB > 99 (use this for old non-physical weights).
  BIF........First IF to select. 0=>all.
  EIF........Highest IF to select. 0=>all higher than BIF
  BCHAN......First channel to select. 0=>all.
  ECHAN......Highest channel to select.  The spectral channels are
             used as separate samples of the visibility to provide
             better statistics in determining the rms.
  SUBARRAY...Subarray number to consider. 0=>1.
  GAINUSE....CL table version number to apply.  0=> highest.
  DOPOL......If > 0.5 then correct data for instrumental polarization
             as represented in the AN or PD table.  This correction is
             only useful if PCAL has been run or feed polarization
             parameters have been otherwise obtained.  See HELP DOPOL
             for available correction modes: 1 is normal, 2 and 3 are
             for VLBI.  1-3 use a PD table if available; 6, 7, 8 are
             the same but use the AN (continuum solution) even if a PD
             table is present.
  PDVER......PD table to apply if PCAL was run with SPECTRAL true and
             0 < DOPOL < 6.  <= 0 => highest.
  BLVER......Version number of the baseline based calibration (BL)
             table to appply. <0 => apply no BL table,  0 => highest.
  FLAGVER....Specifies the version of the flagging table to be
             applied.  0 => highest numbered table.  <0 => no flagging
             to be applied.
  OUTFGVER...Flag table version to be used on output for both single-
             and multi-source data sets.  If OUTFGVER is <= 0 or
             greater than FGmax (the previously highest FG version
             number), then a new FG table will be created for the new
             flags with version FGmax+1.  This new table will also
             contain the flags applied on input (if any) from FG
             version FLAGVER.  If OUTFGVER specifies a pre-existing FG
             version, then the input flags are not copied even if
             OUTFGVER and FLAGVER are not equal.
  DOBAND.....If true (>0) then correct the data for the shape of the
             antenna bandpasses using the BP table specified by BPVER.
             The correction has five modes:
             (a) if DOBAND=1 all entries for an antenna in the table
             are averaged together before correcting the data.
             (b) if DOBAND=2 the entry nearest in time (including
             solution weights) is used to correct the data.
             (c) if DOBAND=3 the table entries are interpolated in
             time (using solution weights) and the data are then
             corrected.
             (d) if DOBAND=4 the entry nearest in time (ignoring
             solution weights) is used to correct the data.
             (e) if DOBAND=5 the table entries are interpolated in
             time (ignoring solution weights) and the data are then
             corrected.
  BPVER......Specifies the version of the BP table to be applied
  SMOOTH.....Specifies the type of spectral smoothing to be applied to
             a uv database . The default is not to apply any smoothing.
             The elements of SMOOTH are as follows:
             SMOOTH(1) = type of smoothing to apply: 0 => no smoothing
               To smooth before applying bandpass calibration
                 1 => Hanning, 2 => Gaussian, 3 => Boxcar, 4 => Sinc
               To smooth after applying bandpass calibration
                 5 => Hanning, 6 => Gaussian, 7 => Boxcar, 8 => Sinc
             SMOOTH(2) = the "diameter" of the function, i.e. width
               between first nulls of Hanning triangle and sinc
               function, FWHM of Gaussian, width of Boxcar. Defaults
               (if < 0.1) are 4, 2, 2 and 3 channels for SMOOTH(1) =
               1 - 4 and 5 - 8, resp.
             SMOOTH(3) = the diameter over which the convolving
               function has value - in channels.  Defaults: 1,3,1,4
               times SMOOTH(2) used when input SMOOTH(3) < net
               SMOOTH(2).
  UVRANGE....(Minimum,Maximum) baseline (kilo-lambda) to process.
  SOLINT.....The interval over which the rms is determined.
             0 => 60 seconds.
  SCANLENG...Length of "scan" for VDIF and VRFI in seconds.  The
             vector average of the visibilities over the scan is
             differenced from the individual visibility to make the
             "vector difference".
  REFANT.....Reference antenna suggestion for VDIF.
  OPTYPE.....Choice of operation to be performed:
             'RFI ' -> acts similar to task RFI, flag all data for
                       times when some fraction of baselines and
                       correlators are excessively noisy, flag all of
                       a correlator for times when some fraction of
                       baselines are too noisy, and flag individual
                       baselines at times when they are too noisy.
             'HIST' -> like RFI above except that the histogram of all
                       baselines for each correlator also yields a
                       cutoff for those baselines wildly different
                       from their fellows in addition to the cutoffs
                       used in RFI.
             'ANTE' -> Like RFI except that the baseline RMSes are
                       converted to antenna RMSes before being checked
                       for excess values.  A robust method is used
                       and closure failure statistics maintained.
                       Data with excess closure failures are flagged
                       along with those having amplitudes, rmses, and
                       weights out of range.
             'TIME' -> (default)
                       Like ANTE it computes the antenna amplitudes
                       and rmses in each SOLINT, but it simply saves
                       them for later.  A robust method is used and
                       closure failure statistics maintained.  When
                       all data have been read, the antenna-based
                       results are analyzed, source by source, antenna
                       by antenna.  Regression is used to compute the
                       mean antenna amplitude and mean rms and the
                       rmses in these parameters.  Outliers are then
                       flagged including those with excess closure
                       failures.
             'GAIN' -> Like TIME except that the source amplitudes are
                       scaled by their average (over all data on that
                       source) and then all sources are examined
                       together.  This may be needed when there is not
                       much data on some of the sources making too few
                       SOLINTs to determine robust averages on a
                       source by source basis.  One should do this
                       only with calibrator sources of similar flux.
             'VDIF' -> It computes the baseline reals, imaginaries,
                       and weights in each SOLINT and converts then to
                       antenna-based values.  A robust method is used
                       and closure failure statistics maintained.  It
                       saves them for later.  When all data have been
                       read, the antenna-based results are analyzed
                       source by source, correlator by correlator.
                       First the visbility at each time is differenced
                       with the average of the visibilities
                       surrounding it.  The weights, amplitudes, and
                       amplitudes of the differences are then used to
                       flag the data as are the closure failure rates.
             'VRFI' -> It computes the baseline reals, imaginaries,
                       and weights in each SOLINT and saves as many as
                       it can for later.  When the source changes, the
                       buffer is full, or all data have been read, the
                       baseline-based results are analyzed correlator
                       by correlator.  First the visbility at each
                       time is differenced with the average of the
                       visibilities surrounding it.  The weights,
                       amplitudes, and amplitudes of the differences
                       are then used to flag the data.
  DOCAT......> 0 => keep the XX extension file generated by the TIME
             and VDIF OPTYPEs.  Otherwise delete it on exit.
  VECTOR     <= 0 -> all averaging and rms's are computed using
             amplitudes only.  Otherwise, the real and imaginary parts
             of the visibility are used.  The former will be
             insensitive to phase variations, but have biased noise
             statistics.
  DOROBUST...> 0 => Do robust averaging of the spectral channels in
                    each time and baseline.  Normal averaging is done
                    over time.
             <= 0 => Normal averaging (all samples included) is done
                     over all channels as well as time.
  BPARM......Control parameters
                (1) If the fraction of all correlators regarded as
                questionable exceeds BPARM(1), flag all data for this
                integration period.  0 -> 0.51     All OPTYPEs.
                (2) If the fraction of baselines regarded as
                questionable exceeds BPARM(2), mark the particualr
                correlator as questionable and flag it (or all
                correlators - see BPARM(1) above).  0 -> 0.25
                All OPTYPEs.
                (3) The maximum allowable sine/cosine rms per baseline
                and correlator to mark it as questionable.  No default.
                For VDIF, the maximum allowable amplitude of the
                vector difference to mark questionable.
                Not used in OPTYPE='TIME'.  Jy.
                (4) Source amplitude coefficient for determining
                questionable baselines.  The maximum allowed rms or
                amplitude of the vector difference is given by:
                   sqrt (BPARM(3)**2 + (avg_amp*BPARM(4))**2)
                Note: the value of BPARM(4) is a compromise between
                false detection of interference on strong sources
                and missing interference on weak sources as the
                interference may mimic a strong source.  0 -> 0.1.
                Not used in OPTYPE='TIME'.
                (5) The maximum allowable sine/cosine rms per baseline
                and correlator to flag that baseline.  0 -> BPARM(3).
                For VDIF, the maximum allowable amplitude of the
                vector difference to mark as bad.
                Not used in OPTYPE='TIME'.   Jy.
                (6) Source amplitude coefficient like BPARM(4) but
                used to flag individual baselines rather than just
                mark them as questionable.  0 -> BPARM(4).
                Not used in OPTYPE='TIME'.
                (7) HIST only: Set the histogram flag level for
                questionable data to MAX (BPARM(7), histogram value).
                Thus, if the data are particularly noise free do not
                delete some anyway.   Jy.   No default.
                (8) HIST only: Set the histogram flag level for
                bad data to MAX (BPARM(8), histogram value).  Thus, if
                the data are particularly noise free do not delete
                some anyway.    Jy.         0 -> BPARM(7)
                (9) > 0 => do not flag correlators having only 1
                sample in the integration period; <=0 => such orphans
                are deleted.  RFI, HIST, ANTE, and TIME now include
                such orphans in the previous integration if possible
                and if all correlators are orphaned.  If previous
                editing reduces some correlators to single samples,
                but not others, then the single-sample ones are
                flagged under control of BPARM(9).
                (10) unused
  CPARM......More control parameters: 1-4 for All OPTYPEs
                (1) Treat data with amplitude <= CPARM(1) as
                questionable.   Jy.
                (2) Treat data with amplitude > CPARM(2) as
                questionable.   Jy.
                (3) Treat data with amplitude <= CPARM(3) as bad.  Jy.
                (4) Treat data with amplitude > CPARM(4) as bad.  Jy.
                (5) 'TIME', 'GAIN' only: Limit the true rms in the
                amplitudes to be >= CPARM(5).   Jy.
                (6) 'TIME', 'GAIN' only: Limit the true rms in the
                rmses to be >= CPARM(6).   Jy.
                (7) 'TIME', 'GAIN' only: Question data more than
                CPARM(7) times the true rms away from the true mean.
                0 -> 3.5
                (8) 'TIME', 'GAIN' only: Question data with rms more
                than CPARM(8) times the true rms of the rms's over all
                antennas, IFs, and polarizations away from the true
                mean of those rms's.   0 -> 8.
                (9) 'TIME', 'gain' only: Flag data more than CPARM(9)
                times the true rms away from the true mean.  0 -> 7
                (10) 'TIME', 'GAIN' only: Flag data with rms more than
                CPARM(10) times the true rms of the rms's over all
                antennas, IFs, and polarizations away from the true
                mean of those rms's.   0 -> 16.
  DPARM......(1) Question data with weight <= DPARM(1) in 1/Jy^2.
             (2) Question data with weight > DPARM(2) in 1/Jy^2.
             (3) Flag data with weight <= DPARM(1) in 1/Jy^2.
             (4) Flag data with weight > DPARM(2) in 1/Jy^2.
             (5) The antenna amplitudes are tested for closure error.
             If the difference of the baseline amplitude and the
             product of the antenna amplitudes exceeds 3 times the rms
             difference, a closure error is counted.  For VDIF, the
             antenna-based complex solution is compared with the data
             and if the amplitude of the difference exceeds 2.5 times
             the rms of the differences, a closure error is counted.
             If the fraction of baselines involving an entenna with
             closure error is > DPARM(5) then the data for that
             antenna are questioned.   0 -> 0.5
             (6) If the fraction of baselines with closure error is
             > DPARM(6), then that antenna is flagged.  0 -> 1 (i.e.
             no flagging).
  STOKES.....Which correlators will be examined: 'FULL', 'HALF',
             'RR', 'LL', 'RRLL', 'RLLR', 'XX', 'YY', 'XXYY', 'XYYX' or
             a "bit pattern" '1111', '1110', etc.  Note that input
             correlators are examined no Stokes conversion will take
             place.  Only limited forms of the Stokes axis are
             supported other than through the bit pattern form.
  DOSTOKES...-1 => Flag only the bad correlator, no other
             +1 => Flag both cross-hand polarizations when one
                   parallel-hand is bad
             +2 => Flag both parallel-hand polarizations when one
                   cross-hand is bad
             +3 => +1 and +2
             +4 => flag all correlators when one is bad
             0 => +3
  DOIFS......> 0 => flag all IFs if one is bad; else flag only the bad
             IF.
  DOALL......> 0 => flag all spectral channels; else flag only
             channels BCHAN through ECHAN.
  DOCRT......Zero means write the flag table without comment.
             False (< 0) use the line printer if OUTPRINT = ' ' else
                   write named OUTPRINT file only.  When OUTPRINT is
                   not blank, DOCRT=-2 suppresses the page-feed
                   character on page headers and DOCRT=-3 suppresses
                   page headers and most other header information.
             True  (> 0) use the terminal interactively.  The task will
                   use the actual terminal width as a display limit
                   unless 72 < DOCRT < width.  In that case, the display
                   limit will be DOCRT characters.
  OUTPRINT...Disk file name in which to save the line printer output.
             ' ' => use scratch and print immediately for interactive
             jobs - batch jobs use OUTPRINT = 'PRTFIL:BATCHjjj.nnn'
             (jjj= job #, nnn = user #).  When OUTPRINT is not blank,
             multiple outputs are concatenated, and the file is not
             actually printed.
  PRTLEV.... Set > 0 to get various debug level messages.  Attached to
             the histogram finder and to the antenna-based solver.
             PRTLEV=3 gets failure information, =4 gets the results of
             the histogram and antenna based functions.  In the
             finding of antenna-base complex "gains", PRTLEV=4 will
             show results of the fit and PRTLEV=5 will show the input
             data and the rms at each iteration.
  BADDISK....This array contains the numbers of disks on which it is
             desired that scratch files not be located.
----------------------------------------------------------------
See FINDR for a task to find typical values to guide you in using
FLAGR.  FINDR even returns typical values in adverb ARRAY2.

Explanation of the methods used


All OPTYPEs (so far)

start by averaging the calibrated and flagged data over a period of
SOLINT seconds.  Data with a time difference <= SOLINT are included.
Thus, a SOLINT of 60 with records every 10 seconds will include 7
records; to get only 6 set SOLINT to something > 50 and < 60 such as
55.  The data from each included antenna pair, IF, and polarization
are kept separate, but the spectral channels from BCHAN through ECHAN
are averaged together.

For OPTYPEs RFI, HIST, ANTE, GAIN, and TIME: During the averaging, the
apparent rms is found including all data (no outlier rejection).  The
rms and amplitude are averaged by vector means unless BPARM(10) > 0,
in which case scalar amplitudes are used.

For OPTYPEs VDIF and VRFI: The averaging is done to determine the
"individual" records to be compared with the "scan" averages of such
records in a region of time centered on the individual records.
Typically, with decent signal-to-noise, one will set SOLINT for VDIF
to less than the time between records.  The "scan" averaging is done
over the scan interval using parameter SCANLENG and is done by finding
the median real and imaginary parts which should be less sensitive to
outliers.

If PRTLEV=1, the task prints messages about "bad" samples.  If
PRTLEV=2, it also prints messages about "questionable" samples.
Even with PRTLEV=0, messages about antenna-independent flags are
printed.  A summary of generated flags is printed at the end.  Note
that DOCRT=0, suppresses all printing although the flag counts will go
to the message file in that case.

At this point, the OPTYPEs do somewhat different things:

===============
OPTYPE 'RFI '

The task loops over all antenna pairs, IFs, and polarizations counting
the "questionable" data.  A sample is questionable if

Sig(A1,A2,I,P)^2  >  BP(3)^2 + (BP(4) * Amp(A1,A2,I,P))^2          or
Amp(A1,A2,I,P)    <= CP(1)                                         or
Amp(A1,A2,I,P)    >  CP(2)                                         or
Wt(A1,A2,I,P)     <= DP(1)                                         or
Wt(A1,A2,I,P)     >  DP(2)

where Sig(A1,A2,I,P) is the rms of the antenna pair A1 and A2 at IF I,
polarization P, Amp is the average amplitude of the sample, Wt is the
average weight of the sample, BP(*) is BPARM(*), CP(*) is CPARM(*),
and DP(*) is DPARM(*).  Looping over all IFs and polarizations, the
task counts the number of antenna pairs that are questionable.  If the
fraction of questionable pairs exceeds BPARM(2), then that
IF/polarization is "bad".  If the number of correlators (1 IF with 1
polarization all antenna pairs) that are bad divided by the total
number of correlators exceeds BPARM(1), then all data for that period
are "bad".  The task then makes flags for these two types of badness
and can print information about them.

The task then loops over all remaining antenna pairs and flags any
that are "clearly bad", i.e. that have

Sig(A1,A2,I,P)^2  >  BP(5)^2 + (BP(6) * Amp(A1,A2,I,P))^2         or
Amp(A1,A2,I,P)    <= CP(3)                                        or
Amp(A1,A2,I,P)    >  CP(4)                                        or
Wt(A1,A2,I,P)     <= DP(3)                                        or
Wt(A1,A2,I,P)     >  DP(4)


===============
OPTYPE 'HIST'

The task loops over all antenna pairs, IFs, and polarizations counting
the "questionable" data.  A sample is questionable if

Sig(A1,A2,I,P)^2  >  BP(3)^2 + (BP(4) * Amp(A1,A2,I,P))^2         or
Sig(A1,A2,I,P)^2  >  Max (BP(7), HistClip(I,P))^2                 or
Amp(A1,A2,I,P)    <= CP(1)                                        or
Amp(A1,A2,I,P)    >  CP(2)                                        or
Wt(A1,A2,I,P)     <= DP(1)                                        or
Wt(A1,A2,I,P)     >  DP(2)

where Sig(A1,A2,I,P) is the rms of the antenna pair A1 and A2 at IF I,
polarization P, Amp is the average amplitude of the sample, BP(*) is
BPARM(*), CP(*) is CPARM(*), and DP(*) is DPARM(*).  HistClip is
determined from the histogram of rms's for each antenna pair and is
set to delete <= 6 per cent of the rms's or all rms's more than 3
sigma from the mean rms.  BPARM(7) keeps this from getting too small
when the data are nearly perfect.  Looping over all IFs and
polarizations, the task counts the number of antenna pairs that are
questionable.  If the fraction of questionable pairs exceeds BPARM(2),
then that IF/polarization is "bad".  If the number of correlators (1
IF with 1 polarization all antenna pairs) that are bad divided by the
total number of correlators exceeds BPARM(1), then all data for that
period are "bad".  The task then makes flags for these two types of
badness and can print information about them.

The task then loops over all remaining antenna pairs and flags any
that are "clearly bad", i.e. that have

Sig(A1,A2,I,P)^2  >  BP(5)^2 + (BP(6) * Amp(A1,A2,I,P))^2         or
Sig(A1,A2,I,P)^2  >  Max (BP(8), HistClip(I,P))^2                 or
Amp(A1,A2,I,P)    <= CP(3)                                        or
Amp(A1,A2,I,P)    >  CP(4)                                        or
Wt(A1,A2,I,P)     <= DP(3)                                        or
Wt(A1,A2,I,P)     >  DP(4)


===============
OPTYPE 'ANTE'

The task converts the average amplitudes, rms's, and weights of the
antenna pairs into antenna-based values using the assumption that

Amp(A1,A2,I,P) = SQRT (Amp(A1,I,P) * Amp(A2,I,P))
Sig(A1,A2,I,P) = SQRT (Sig(A1,I,P) * Sig(A2,I,P))
Wt(A1,A2,I,P)  = SQRT (Wt(A1,I,P)  * Wt(A2,I,P))

It uses robust methods (outliers ignored) to compute the antenna-based
values and then counts the fraction [Fr(A,I,P)] of baselines for each
antenna that are more than 3 times the rms away from the final
solution.

The task then loops over all antennas, IFs, and polarizations counting
the "questionable" data.  A sample is questionable if

Sig(A,I,P)^2  >  BP(3)^2 + (BP(4) * Amp(A,I,P))^2                 or
Amp(A,I,P)    <= CP(1)                                            or
Amp(A,I,P)    >  CP(2)                                            or
Wt(A,I,P)     <= DP(1)                                            or
Wt(A,I,P)     >  DP(2)                                            or
Fr(A)         >  DP(5)

where Sig(A,I,P) is the rms of antenna A at IF I, polarization P,
Amp is the average amplitude of the sample, BP(*) is BPARM(*), CP(*)
is CPARM(*), and DP(*) is DPARM(*).  Looping over all IFs and
polarizations, the task counts the number of antennas that are
questionable.  If the fraction of questionable antennas exceeds
BPARM(2), then that IF/polarization is "bad".  If the number of
correlators (1 IF with 1 polarization all antennas) that are bad
divided by the total number of correlators exceeds BPARM(1), then all
data for that period are "bad".  The task then makes flags for these
two types of badness and can print information about them.

The task then loops over all remaining antenna pairs and flags any
that are "clearly bad", i.e. that have

Sig(A,I,P)^2  >  BP(5)^2 + (BP(6) * Amp(A,I,P))^2                 or
Amp(A,I,P)    <= CP(3)                                            or
Amp(A,I,P)    >  CP(4)                                            or
Wt(A,I,P)     <= DP(3)                                            or
Wt(A,I,P)     >  DP(4)                                            or
Fr(A)         >  DP(6)


===============
OPTYPE 'TIME'     The default.

The task converts the average amplitudes, rms's, and weights of the
antenna pairs into antenna-based values using the assumption that

Amp(A1,A2,I,P) = SQRT (Amp(A1,I,P) * Amp(A2,I,P))
Sig(A1,A2,I,P) = SQRT (Sig(A1,I,P) * Sig(A2,I,P))
Wt(A1,A2,I,P)  = SQRT (Wt(A1,I,P)  * Wt(A2,I,P))

It uses robust methods (outliers ignored) to compute the antenna-based
values and then counts the fraction [Fr(A,I,P)] of baselines for each
antenna that are more than 3 times the rms away from the final
solution.

The values of Amp(A,I,P), Sig(A,I,P), Wt(A,I,P) and Fr(A,I,P) are then
written to an XX table (which may be saved by setting DOCAT > 0).
When all of the data have been read and averaged and the antenna-based
results written to the XX table, these results are read back into
memory.

One source at a time, the task determines the robust average amplitude
and robust average rms (along with the uncertainties in these robust
averages) for each antenna, IF, and polarization.  By "robust"
average, we mean an average determined by repeated passes through the
data with the averaging process discarding outlier points.  The robust
average of these robust averages is then determined and printed in the
message window.  The robust average is computed only for those
amplitudes between CPARM(1) and CPARM(2).

For each source and one antenna, IF, and polarization at a time, the
task loops over time counting the "questionable" data.  A sample is
questionable if

Wt(A,I,P,t)     <= DP(1)                                            or
Wt(A,I,P,t)     >  DP(2)                                            or
Fr(A,I,P,t)     >  DP(5)                                            or
Amp(A,I,P,t)    <= CP(1)                                            or
Amp(A,I,P,t)    >  CP(2)                                            or
ABS(Amp(A,I,P,t)-AvAmp(A,I,P)) > CP(7)*MAX (CP(5), AvAmprms(A,I,P)) or
Sig(A,I,P,t)-AvSig(A,I,P) > CP(8) * AvAvRms

where Sig(A,I,P,t) is the rms of antenna A at IF I, polarization P,
time t; Amp(A,I,P,t) is the average amplitude of that sample,
AvAmp(A,I,P) is the robust average over time of that antenna, IF, and
polarization, AvAmprms(A,I,P) is the uncertainty in that average over
time, AvSig(A,I,P) is the robust average over time of the rms's of
that antenna, IF, and polarization, and AvAvRms is the robust average
over all antennas, IFs, and polarizations of AvSig, CP(*) is CPARM(*),
and DP(*) is DPARM(*).  Looping over all IFs and polarizations, the
task counts the number of antennas that are questionable at the time
in question.  If the fraction of questionable antennas exceeds
BPARM(2), then that IF/polarization is "bad".  If the number of
correlators (1 IF with 1 polarization all antennas) that are bad
divided by the total number of correlators exceeds BPARM(1), then all
data for that period are "bad".  The task then makes flags for these
two types of badness and can print information about them.

The task then loops over all remaining antennas and flags any
that are "clearly bad", i.e. that have

Wt(A,I,P,t)     <= DP(3)                                            or
Wt(A,I,P,t)     >  DP(4)                                            or
Fr(A,I,P,t)     >  DP(6)                                            or
Amp(A,I,P,t)    <= CP(3)                                            or
Amp(A,I,P,t)    >  CP(4)                                            or
ABS(Amp(A,I,P,t)-AvAmp(A,I,P)) > CP(9)*MAX (CP(5), AvAmprms(A,I,P)) or
Sig(A,I,P,t)-AvSig(A,I,P) > CP(10) * AvAvRms


===============
OPTYPE 'GAIN'

The task converts the average amplitudes, rms's, and weights of the
antenna pairs into antenna-based values using the assumption that

Amp(A1,A2,I,P) = SQRT (Amp(A1,I,P) * Amp(A2,I,P))
Sig(A1,A2,I,P) = SQRT (Sig(A1,I,P) * Sig(A2,I,P))
Wt(A1,A2,I,P)  = SQRT (Wt(A1,I,P)  * Wt(A2,I,P))

It uses robust methods (outliers ignored) to compute the antenna-based
values and then counts the fraction [Fr(A,I,P)] of baselines for each
antenna that are more than 3 times the rms away from the final
solution.

The values of Amp(A,I,P), Sig(A,I,P), Wt(A,I,P) and Fr(A,I,P) are then
written to an XX table (which may be saved by setting DOCAT > 0).
When all of the data have been read and averaged and the antenna-based
results written to the XX table, these results are read back into
memory.

The task determines the relative Flux(s) of each source s averaging
ALL values of Amp(A,I,P,t) by source.  The "gains" are then
Gn(A,I,P,t) = Amp(A,I,P,t) / Flux(s)

Using all sources, the task determines the robust average gain and
robust average rms (along with the uncertainties in these robust
averages) for each antenna, IF, and polarization.  By "robust"
average, we mean an average determined by repeated passes through the
data with the averaging process discarding outlier points.  The robust
average of these robust averages is then determined and printed in the
message window.  The robust average is computed only for those
amplitudes between CPARM(1) and CPARM(2).

One antenna, IF, and polarization at a time, the task loops over time
counting the "questionable" data.  A sample is questionable if

Wt(A,I,P,t)     <= DP(1)                                          or
Wt(A,I,P,t)     >  DP(2)                                          or
Fr(A,I,P,t)     >  DP(5)                                          or
Amp(A,I,P,t)    <= CP(1)                                          or
Amp(A,I,P,t)    >  CP(2)                                          or
ABS(Gn(A,I,P,t)-AvGn(A,I,P)) > CP(7)*MAX (CP(5), AvGnrms(A,I,P))  or
Sig(A,I,P,t)-AvSig(A,I,P) > CP(8) * AvAvRms

where Sig(A,I,P,t) is the rms of antenna A at IF I, polarization P,
time t; Amp(A,I,P,t) is the average amplitude of that sample;
Gn(A,I,P,t) is the average gain of that sample; AvGn(A,I,P) is the
robust average over time of that antenna, IF, and polarization,
AvGnrms(A,I,P) is the uncertainty in that average over time;
AvSig(A,I,P) is the robust average over time of the rms's of that
antenna, IF, and polarization, and AvAvRms is the robust average over
all antennas, IFs, and polarizations of AvSig, CP(*) is CPARM(*), and
DP(*) is DPARM(*).  Looping over all IFs and polarizations, the task
counts the number of antennas that are questionable at the time in
question.  If the fraction of questionable antennas exceeds BPARM(2),
then that IF/polarization is "bad".  If the number of correlators (1
IF with 1 polarization all antennas) that are bad divided by the total
number of correlators exceeds BPARM(1), then all data for that period
are "bad".  The task then makes flags for these two types of badness
and can print information about them.

The task then loops over all remaining antennas and flags any that are
"clearly bad", i.e. that have

Wt(A,I,P,t)     <= DP(3)                                          or
Wt(A,I,P,t)     >  DP(4)                                          or
Fr(A,I,P,t)     >  DP(6)                                          or
Amp(A,I,P,t)    <= CP(3)                                          or
Amp(A,I,P,t)    >  CP(4)                                          or
ABS(Gn(A,I,P,t)-AvGn(A,I,P)) > CP(9)*MAX (CP(5), AvGnrms(A,I,P))
Sig(A,I,P,t)-AvSig(A,I,P) > CP(10) * AvAvRms


===============
OPTYPE 'VDIF'

The task converts the average weights and complex visibilities of the
antenna pairs into antenna-based values using the assumption that

Amp(A1,A2,I,P) = SQRT (Amp(A1,I,P) * Amp(A2,I,P))
Phs(A1,A2,I,P) = Phs(A1,I,P) - Phs(A2,I,P)
Wt(A1,A2,I,P)  = SQRT (Wt(A1,I,P)  * Wt(A2,I,P))

It uses robust methods (outliers ignored) to compute the antenna-based
values and then counts the fraction [Fr(A,I,P)] of baselines for each
antenna that are more than 2.5 times the rms away from the final
solution.

The values of Amp(A,I,P), Phs(A,I,P) (as real and imaginary),
Wt(A,I,P), Fr(A,I,P), and reference antennas (I,P) are then written to
an XX table (which may be saved by setting DOCAT > 0).  When all of
the data have been read and averaged and the antenna-based results
written to the XX table, these results are read back into memory.  If
needed, the data are put on a consistent reference antenna basis.

One source at a time, the task then loops over each antenna, IF, and
polarization converting the time sequence of reals and imaginaries
into time sequences of amplitudes and amplitudes of the "vector
difference" between the current antenna visibility and the vector
average of antenna visibilities within SCANLENG/2 of the current
time.  Note that, in this case, the averages are generated by finding
the median value of the reals and the median value of the imaginaries.
This should be somewhat less succeptible to outliers than straight
averaging and better than robust averaging as well for the small
number of samples which will be normal.

For each source and one antenna, IF, and polarization at a time, the
task loops over time counting the "questionable" data.  A sample is
questionable if

Wt(A,I,P,t)     <= DP(1)                                            or
Wt(A,I,P,t)     >  DP(2)                                            or
Fr(A,I,P,t)     >  DP(5)                                            or
Amp(A,I,P,t)    <= CP(1)                                            or
Amp(A,I,P,t)    >  CP(2)                                            or
Dif(A,I,P,t)    >  BP(3) + (BP(4) * Amp(A,I,P))

where Dif(A,I,P,t) is the amplitude of the vector difference of
antenna A at IF I, polarization P, time t with the average of the data
within SCANLENG/2 of t; Amp(A,I,P,t) is the average amplitude of that
sample, BP(*) is BPARM(*), CP(*) is CPARM(*), and DP(*) is DPARM(*).
Looping over all IFs and polarizations, the task counts the number of
antennas that are questionable at the time in question.  If the
fraction of questionable antennas exceeds BPARM(2), then that
IF/polarization is "bad".  If the number of correlators (1 IF with 1
polarization all antennas) that are bad divided by the total number of
correlators exceeds BPARM(1), then all data for that period are "bad".
The task then makes flags for these two types of badness and can print
information about them.

The task then loops over all remaining antennas and flags any
that are "clearly bad", i.e. that have

Wt(A,I,P,t)     <= DP(3)                                            or
Wt(A,I,P,t)     >  DP(4)                                            or
Fr(A,I,P,t)     >  DP(6)                                            or
Amp(A,I,P,t)    <= CP(3)                                            or
Amp(A,I,P,t)    >  CP(4)                                            or
Dif(A,I,P)      >  BP(5) + (BP(6) * Amp(A,I,P))


===============
OPTYPE 'VRFI'

The task accumulates SOLINT averages in memory until the source
changes, the data ends, or the number of times that it can hold is
exhausted.  For each time, the task then loops over each antenna pair,
IF, and polarization finding the amplitude of the "vector difference"
between the current visibility and the vector average of visibilities
for that antenna pair within SCANLENG/2 of the current time.  Note
that, in this case, the averages are generated by finding the median
value of the reals and the median value of the imaginaries.   This
should be somewhat less succeptible to outliers than straight
averaging and better than robust averaging as well for the small
number of samples which will be normal.

The task loops over all antenna pairs, IFs, and polarizations counting
the "questionable" data.  A sample is questionable if

Dif(A1,A2,I,P)    >  BP(3) + (BP(4) * Amp(A1,A2,I,P))              or
Amp(A1,A2,I,P)    <= CP(1)                                         or
Amp(A1,A2,I,P)    >  CP(2)                                         or
Wt(A1,A2,I,P)     <= DP(1)                                         or
Wt(A1,A2,I,P)     >  DP(2)

where Dif(A1,A2,I,P) is the amplitude of the vector difference of the
antenna pair A1 and A2 at IF I, polarization P, Amp is the average
amplitude of the sample, Wt is the average weight of the sample, BP(*)
is BPARM(*), CP(*) is CPARM(*), and DP(*) is DPARM(*).  Looping over
all IFs and polarizations, the task counts the number of antenna pairs
that are questionable.  If the fraction of questionable pairs exceeds
BPARM(2), then that IF/polarization is "bad".  If the number of
correlators (1 IF with 1 polarization all antenna pairs) that are bad
divided by the total number of correlators exceeds BPARM(1), then all
data for that period are "bad".  The task then makes flags for these
two types of badness and can print information about them.

The task then loops over all remaining antenna pairs and flags any
that are "clearly bad", i.e. that have

Dif(A1,A2,I,P)    >  BP(5) + (BP(6) * Amp(A1,A2,I,P))             or
Amp(A1,A2,I,P)    <= CP(3)                                        or
Amp(A1,A2,I,P)    >  CP(4)                                        or
Wt(A1,A2,I,P)     <= DP(3)                                        or
Wt(A1,A2,I,P)     >  DP(4)

The task then loops for the new source and/or a new buffer load as
appropriate until the data are exhausted.
