; REWAY
;---------------------------------------------------------------
;! computes weights based in rms in spectra
;# Task UV VLA
;-----------------------------------------------------------------------
;;  Copyright (C) 2010
;;  Associated Universities, Inc. Washington DC, USA.
;;
;;  This program is free software; you can redistribute it and/or
;;  modify it under the terms of the GNU General Public License as
;;  published by the Free Software Foundation; either version 2 of
;;  the License, or (at your option) any later version.
;;
;;  This program is distributed in the hope that it will be useful,
;;  but WITHOUT ANY WARRANTY; without even the implied warranty of
;;  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;  GNU General Public License for more details.
;;
;;  You should have received a copy of the GNU General Public
;;  License along with this program; if not, write to the Free
;;  Software Foundation, Inc., 675 Massachusetts Ave, Cambridge,
;;  MA 02139, USA.
;;
;;  Correspondence concerning AIPS should be addressed as follows:
;;         Internet email: aipsmail@nrao.edu.
;;         Postal address: AIPS Project Office
;;                         National Radio Astronomy Observatory
;;                         520 Edgemont Road
;;                         Charlottesville, VA 22903-2475 USA
;-----------------------------------------------------------------------
REWAY     LLLLLLLLLLLLUUUUUUUUUUUU CCCCCCCCCCCCCCCCCCCCCCCCCCCCC
REWAY     Computes data weights based solely on rms in spectra
INNAME                             Input UV file name (name)
INCLASS                            Input UV file name (class)
INSEQ             0.0     9999.0   Input UV file name (seq. #)
INDISK            0.0        9.0   Input UV file disk unit #
SOURCES                            Source name
QUAL            -10.0              Calibrator qualifier -1=>all
CALCODE                            Calibrator code '    '=>all
TIMERANG                           Time range to use
SELBAND                            Bandwidth to select (kHz)
SELFREQ                            Frequency to select (MHz)
FREQID                             Freq. ID to select.
SUBARRAY          0.0     1000.0   Sub-array, 0=>all
DOCALIB          -1.0      101.0   > 0 calibrate data & weights
                                   > 99 do NOT calibrate weights
GAINUSE                            CL (or SN) table to apply
DOPOL            -1.0       10.0   If >0.5 correct polarization.
PDVER                              PD table to apply (DOPOL>0)
BLVER                              BL table to apply.
FLAGVER                            Flag table version
DOBAND           -1.0       10.0   If >0.5 apply bandpass cal.
                                   Method used depends on value
                                   of DOBAND (see HELP file).
BPVER                              Bandpass table version
SMOOTH                             Smoothing function. See
                                   HELP SMOOTH for details.
DOACOR                             Include autocorrelations?
OUTNAME                            Output UV file name (name)
OUTCLASS                           Output UV file name (class)
OUTSEQ           -1.0     9999.0   Output UV file name (seq. #)
OUTDISK           0.0        9.0   Output UV file disk unit #.
ICHANSEL                           Array of start and stop chan
                                   numbers, plus a channel
                                   increment and IF to be used
                                   to select channels to sum to
                                   find RMS.  If all 0, range
                                   set to inner 75% of band.
OUTFGVER         -1.0              Output flag table: 0 none
APARM                              (1) > 0 rolling T buffer to
                                      find baseline weights
                                      A(1) times
                                       = 0 find baseline weights
                                      with no averaging
                                       < 0 find antenna weights
                                      averaged on |A(1)| sec
                                   (2) Max time in buffer if
                                      A(1) > 0 in sec  0 -> 600
                                   (3) Smooth rmses with func
                                      width = A(3) sec.
                                   (4) Type func: 0 Gaussian,
                                      1 EXP, 2 boxcar
                                   (5) ignore pts > A(5) sigma
                                      in smoothing.  < 1 -> 4
                                      can lead to flagging
                                   (6) ignore pts > A(6) sigma
                                      overall in smoothing
                                      can lead to flagging
                                   (7) flag data w wt < A(7)
                                   (8) flag data w wt > A(8)
                                   (9) > 0 => ignore incoming
                                       data weights
                                   (10) > 0 => keep WT table
                                       as extension file
OPTYPE                             'MEDI' use median window
                                   solver for mean/rms else use
                                   "robust" methods
----------------------------------------------------------------
REWAY
Task:  This task will copy selected portions of a data set (much like
       SPLIT) and change the data weights by computing the uncertainty
       using the multiple spectral channels in each IF and
       polarization.  Note, there must be enough spectral channels to
       make this computation meaningful.  Furthermore, all possible
       calibrations should be applied so that, in particular, any
       delay and bandpass shape are removed.  Those calibrations will
       greatly reduce the apparent rms.  EVLA weights are not
       calibrated when DOCALIB is true.  Therefore, for the EVLA, the
       full gain calibration should be made so that the weights
       computed here are in Jy^(-2) on the same scale for all
       baselines.

       The task offers the option to ignore the input data weights
       (except for flagging).  If they are not ignored, the the robust
       rms methods use the weights in the rms computation and the
       "rms" that is stored in the WT table, smoothed, and used in
       flagging is the actual rms scaled by square root of the average
       data weight.  When the rmses are converted to data weights they
       are used to scale the input weights.

       Output files are always uncompressed so as to avoid losing any
       weights which might be different in different IFs.  Input files
       may be compressed or uncompressed.
Adverbs:
  INNAME.....Input UV file name (name).      Standard defaults.
  INCLASS....Input UV file name (class).     Standard defaults.
  INSEQ......Input UV file name (seq. #).    0 => highest.
  INDISK.....Disk drive # of input UV file.  0 => any.
  SOURCES....Source to be copied.   '  '=> all; if any starts with a
             '-' then all except ANY source named.
  QUAL.......Qualifier of source to be copied. -1 => all.
  CALCODE....Calibrator code of sources to copy. ' '=> all.
  STOKES.....Specifies which STOKES parameters are written in the
             output data set:  ' ' => 'FULL'
               'I','Q','U','V', 'IV', 'IQU', 'IQUV'
               'RR','LL', 'RL', 'LR', 'RRLL', 'RLLR', 'RLRL'
               'XX','YY', 'XY', 'YX', 'XXYY', 'XYYX', 'XYXY'
             'HALF', 'CROS', and 'FULL' have sensible interpretations
             depending on the Stokes present in the data.  The last in
             each of the 3 rows above == 'FULL'.  Note that many
             combinations of polarizations in the input and values
             above are not supported.
  TIMERANG...Time range of the data to be copied. In order: Start day,
             hour, min. sec, end day, hour, min. sec. Days relative to
             ref. date.
  SELBAND....Bandwidth of data to be selected. If more than one IF is
             present SELBAND is the width of the first IF required.
             Units = kHz. For data which contain multiple
             bandwidths/frequencies the task will insist that some form
             of selection be made by frequency or bandwidth.
  SELFREQ....Frequency of data to be selected. If more than one IF is
             present SELFREQ is the frequency of the first IF required.
             Units = MHz.
  FREQID.....Frequency identifier to select (you may determine which is
             applicable from the OPTYPE='SCAN' listing produced by
             LISTR). If either SELBAND or SELFREQ are set, their values
             override that of FREQID.  However, setting SELBAND and
             SELFREQ may result in an ambiguity.  In that case, the task
             will request that you use FREQID.
  SUBARRAY...Sub-array number to copy. 0=>all.
  DOCALIB....If true (>0), calibrate the data using information in the
             specified Cal (CL) table for multi-source or SN table for
             single-source data.  Also calibrate the weights unless
             DOCALIB > 99 (use this for old non-physical weights).
  GAINUSE....version number of the CL table to apply to multi-source
             files or the SN table for single source files.
             0 => highest.
  DOPOL......If > 0 then correct data for instrumental polarization as
             represented in the AN or PD table.  This correction is
             only useful if PCAL has been run or feed polarization
             parameters have been otherwise obtained.  See HELP DOPOL
             for available correction modes: 1 is normal, 2 and 3 are
             for VLBI.  1-3 use a PD table if available; 6, 7, 8 are
             the same but use the AN (continuum solution) even if a PD
             table is present.
  PDVER......PD table to apply if PCAL was run with SPECTRAL true and
             0 < DOPOL < 6.  <= 0 => highest.
  FLAGVER....specifies the version of the flagging table to be applied.
              0 => highest numbered table.
             <0 => no flagging to be applied.
  DOBAND.....If true (>0) then correct the data for the shape of the
             antenna bandpasses using the BP table specified by BPVER.
             The correction has five modes:
             (a) if DOBAND=1 all entries for an antenna in the table
             are averaged together before correcting the data.
             (b) if DOBAND=2 the entry nearest in time (including
             solution weights) is used to correct the data.
             (c) if DOBAND=3 the table entries are interpolated in
             time (using solution weights) and the data are then
             corrected.
             (d) if DOBAND=4 the entry nearest in time (ignoring
             solution weights) is used to correct the data.
             (e) if DOBAND=5 the table entries are interpolated in
             time (ignoring solution weights) and the data are then
             corrected.
             IMAGR uses DOBAND as the nearest integer; 0.1 is therefore
             "false".
  BPVER......Specifies the version of the BP table to be applied
                0 => highest numbered table.
               <0 => no bandpass correction to be applied.
  SMOOTH.....Specifies the type of spectral smoothing to be applied to
             a uv database . The default is not to apply any smoothing.
             The elements of SMOOTH are as follows:
             SMOOTH(1) = type of smoothing to apply: 0 => no smoothing
               To smooth before applying bandpass calibration
                 1 => Hanning, 2 => Gaussian, 3 => Boxcar, 4 => Sinc
               To smooth after applying bandpass calibration
                 5 => Hanning, 6 => Gaussian, 7 => Boxcar, 8 => Sinc
             SMOOTH(2) = the "diameter" of the function, i.e. width
               between first nulls of Hanning triangle and sinc
               function, FWHM of Gaussian, width of Boxcar. Defaults
               (if < 0.1) are 4, 2, 2 and 3 channels for SMOOTH(1) =
               1 - 4 and 5 - 8, resp.
             SMOOTH(3) = the diameter over which the convolving
               function has value - in channels.  Defaults: 1,3,1,4
               times SMOOTH(2) used when input SMOOTH(3) < net
               SMOOTH(2).
  DOACOR.....> 0 => include autocorrelations as well as cross
             correlation data.
  OUTNAME....Output UV file name (name).     Standard defaults.
  OUTCLASS...Output UV file name (class).    Standard defaults.
  OUTSEQ.....Output UV file name (seq. #).   0 => highest unique
  OUTDISK....Disk drive # of output UV file. 0 => highest with space
             for the file.
  ICHANSEL.. Array of start, stop, and increment channel numbers plus
             an IF used for channel selection in the averaging to
             compute an rms.  Up to 20 sets if channels/IF may be
             entered.  The first having ICHANSEL(2,i) <= 0 terminates
             the list.  ICHANSEL(4,i) is the IF number, with <= 0
             meaning all IFs.  If an IF has no ICHANSEL set for it,
             then the inner 75% of that IF is used.
  OUTFGVER...If solutions are flagged (based on APARM(5)-APARM(8)), a
             flag table may be written.  If OUTFGVER <= 0, this will
             not happen.  If > 0, a new FG table is made and the
             FLAGVER FG table copied to it.  If OUTFGVER points at an
             existing flag table other than FLAGVER, the new flags are
             written to that table.  If OUTFGVER points at FLAGVER,
             the program will fail so it is changed to make a new FG
             table instead with the contents of FLAGVER copied to it.
  APARM......(1) If = 0, the rmses are found on a per baseline per
                 IF and per Stokes basis and applied to the data as
                 weights = rms^(-2) directly after smoothing and
                 clipping controlled by APARM(3-8).
                 If < 0, the above rmses are averaged over -APARM(1)
                 seconds, converted to antenna-based rmses per IF per
                 Stokes.  They are then smoothed and clipped and
                 converted to baseline weights = 1/(rms1 * rms2).
                 If > 0, the data are accumlated in a buffer APARM(1)
                 time samples long, robust rmses are found for each
                 baseline, IF, and Stokes and are then turned into
                 baseline-based weights and saved in the WT table.
                 After optional smoothing, thw WT table is applied to
                 the data.
                 NO DEFAULT.
             (2) If APARM(1) > 0, do not average over times longer
                 than APARM(8) seconds.  0 -> 600.
             (3) The rmses found on an antenna or a baseline basis
                 may be smoothed with a function of width APARM(2).
                 If APARM(3) <= 0, then no further smoothing or
                 clipping of rmses is done.  0 means 0 for APARM(1) =
                 0, 0 -> 1 sec for the other modes.
             (4) Type of function: 0 -> Gaussian, width is FWHM,
                 1 -> Exponential, width is FWHM, 2 -> boxcar, width
                 is full support.  <= 0, > 2 => 0
             (5) When smoothing rmses, only those rmses within
                 APARM(5) * SIGMA of the average are included.  The
                 average and sigma for this limit are on a per
                 baseline, per IF, per polarization basis.
                 0 -> 4; use large number to avoid clipping.
                 If the smoothed rms contains no points, the data
                 sample will be flagged.
             (6) The mean and rms over all baselines, IFs, and
                 polarizations may also be used to limit the rmses
                 used when smoothing.  Include only rmses withing
                 APARM(6) * Overall_SIGMA  of the overall mean rms.
                 0 -> 10000.  If the smoothed rms contains no points,
                 the data sample will be flagged.  This effectively
                 sets defaults for APARM(7) and APARM(8).
             (7) Flag data with weight < APARM(7).
             (8) Flag data with weight > APARM(8); 0 -> 10^10
                 The flagging is only applied if smoothing is also
                 done (APARM(2) > 0 and APARM(2) > APARM(1) when
                 APARM(1) < 0).
             (9) > 0 => ignore incoming data weights.  Otherwise the
                 non-median methods use the data weights in
                 determining the rmses and both scale the output
                 weights with the input weights.
             (10) The rmses are worked on using a non-standard WT
                 table.  To keep this table, set APARM(10) > 0.
                 Note that the WT table is an extension file with a
                 table format like most other extension files.  It
                 contains rmses not weights (despite its name) and may
                 be plotted with TAPLT, printed with PRTAB.
  OPTYPE.....'MEDI' uses a median operation to do the statistics on
                    the visibility values.
             'MEDR' uses a median operation plus robust methods to do
                    the statistics (expensive, probably not worth it)
             other  uses simple statistics but with a robust aspect to
                    gradually discard outliers.  MEDI is modestly
                    faster than the robust method, but is more likely
                    to be fooled by real signal one would like to
                    ignore.
----------------------------------------------------------------

APARM(1) appears to be confusing.  There are 3 algorithms:

APARM(1) = 0 is simple - each spectrum has a robust rms determined
over the specified channels and a weight = 1/rms^2 is stored.  Thus
the weight is independent in each polarization, IF, baseline, and
time.

APARM(1) < 0 is more complicated.  Each rms is found as above.  Then
the RMSes are averaged over an interval -APARM(1) seconds.  Then the
rmses are turned into antenna-based rmses (separately for each
polarization and IF) and stored in a temporary WT table.  Optionally,
the WT table may be smoothed, clipped, and data flags found.  Then the
data are re-read and the weights and flags in the WT table are applied
to the output file.

APARM(1) > 0 is even more complicated and may require very large
amounts of dynamic memory.  All data for APARM(1) times are read into
memory.  Then the rms for each baseline, IF, and Stokes is determined
by a robust average over all spectral channels and times in the
buffer.  These rmses are turned into antenna-based weights for the
central time in the buffer and stored in the WT file.  Then the first
record in the buffer is discarded and replaced with the next time and
the process is repeated.  APARM(1) must be odd so that the midpooint
of the buffer is well determined.  Note that the first (APARM(1)+1)/2
times and the last (APARM(1)+1)/2 times in a scan will have the same
value so that the number of times entering in each weight will be
constant.

For example, let APARM(1)=+9.  The first 9 times are read into memory
and the rmses and antenna-based weights are found and stored in the WT
table.  These are written to the WT table for times 1 through 5.  Then
time 1 is discarded and time 10 read.  The process repeats and the
time and weights for time 6 are written in the WT table.  Then time 2
is deleted and time 11 read and so forth.  When the end of the scan is
reached, the rolling buffer is cleared out much like at the start and
a new scan is processed.  Scan boundaries are found when the source
name changes or when APARM(2) seconds have passed from the first time
currently in the rolling buffer.

The WT table so written can then be smoothed, clipped, flagged etc
and the data are re-read and weighted as above.


