As of Fri Jul 19 10:08:13 2024

REWAY: Computes data weights based solely on rms in spectra


INNAME                             Input UV file name (name)
INCLASS                            Input UV file name (class)
INSEQ             0.0     9999.0   Input UV file name (seq. #)
INDISK            0.0        9.0   Input UV file disk unit #
SOURCES                            Source name
QUAL            -10.0              Calibrator qualifier -1=>all
CALCODE                            Calibrator code '    '=>all
TIMERANG                           Time range to use
SELBAND                            Bandwidth to select (kHz)
SELFREQ                            Frequency to select (MHz)
FREQID                             Freq. ID to select.
SUBARRAY          0.0     1000.0   Sub-array, 0=>all
DOCALIB          -1.0      101.0   > 0 calibrate data & weights
                                   > 99 do NOT calibrate weights
GAINUSE                            CL (or SN) table to apply
DOPOL            -1.0       10.0   If >0.5 correct polarization.
PDVER                              PD table to apply (DOPOL>0)
BLVER                              BL table to apply.
FLAGVER                            Flag table version
DOBAND           -1.0       10.0   If >0.5 apply bandpass cal.
                                   Method used depends on value
                                   of DOBAND (see HELP file).
BPVER                              Bandpass table version
SMOOTH                             Smoothing function. See
                                   HELP SMOOTH for details.
DOACOR                             Include autocorrelations?
OUTNAME                            Output UV file name (name)
OUTCLASS                           Output UV file name (class)
OUTSEQ           -1.0     9999.0   Output UV file name (seq. #)
OUTDISK           0.0        9.0   Output UV file disk unit #.
ICHANSEL                           Array of start and stop chan
                                   numbers, plus a channel
                                   increment and IF to be used
                                   to select channels to sum to
                                   find RMS.  If all 0, range
                                   set to inner 75 percent of band.
OUTFGVER         -1.0              Output flag table: 0 none
APARM                              (1) > 0 rolling T buffer to
                                      find baseline weights
                                      A(1) times
                                       = 0 find baseline weights
                                      with no averaging
                                       < 0 find antenna weights
                                      averaged on |A(1)| sec
                                   (2) Max time in buffer if
                                      A(1) > 0 in sec  0 -> 600
                                   (3) Smooth rmses with func
                                      width = A(3) sec.
                                   (4) Type func: 0 Gaussian,
                                      1 EXP, 2 boxcar
                                   (5) ignore pts > A(5) sigma
                                      in smoothing.  < 1 -> 4
                                      can lead to flagging
                                   (6) ignore pts > A(6) sigma
                                      overall in smoothing
                                      can lead to flagging
                                   (7) flag data w wt < A(7)
                                   (8) flag data w wt > A(8)
                                   (9) > 0 => ignore incoming
                                       data weights
                                       > 1.5 => ignore INTTIME
                                   (10) > 0 => keep WT table
                                       as extension file
BPARM                              (1) Set weight to zero if <
                                       B(1) samples in window
                                   (2) > 0 compute statistics
                                       of weights, 1 give a
                                       summary and outliers,
                                       2 give details when
                                       APARM(1) < 0
                                   (3) Outliers are BPARM(3)
                                       sigma from the average
                                       0 -> 5
                                   (4) > 0 => substitute cross-
                                       hand weights into all
OUTTEXT                            Not blank - write details of
                                   statistics in this text file
OPTYPE                             'MEDI' use median window
                                   solver for mean/rms else use
                                   "robust" methods
FQCENTER                           >= 0 -> center frequency axis
BADDISK                            Disk drive #'s to avoid


Task:  This task will copy selected portions of a data set (much like
       SPLIT) and change the data weights by computing the uncertainty
       using the multiple spectral channels in each IF and
       polarization.  Note, there must be enough spectral channels to
       make this computation meaningful.  Furthermore, all possible
       calibrations should be applied so that, in particular, any
       delay and bandpass shape are removed.  Those calibrations will
       greatly reduce the apparent rms.  EVLA weights are not
       calibrated when DOCALIB is true until after application of
       TYAPL or REWAY.  Therefore, for the EVLA, the full gain
       calibration should be made so that the weights computed here
       are in Jy^(-2) on the same scale for all baselines.

       The task offers the option to ignore the input data weights
       (except for flagging).  If they are not ignored, then the
       robust rms methods use the weights in the rms computation and
       the "rms" that is stored in the WT table, smoothed, and used in
       flagging is the actual rms scaled by square root of the average
       data weight.  When the rmses are converted to data weights they
       are used to scale the input weights.

       Output files are always uncompressed so as to avoid losing any
       weights which might be different in different IFs.  Input files
       may be compressed or uncompressed.

       The task computes the minimum, maximum, mean, and rms of the
       weights by antennas (APARM(3)<0) or baselines (APARM(1)>= 0)
       separated also by IF and Stokes.  These statistics may be
       displayed in the message window and written to a text file.
       First the averages over all antennas or baselines in each of
       the 4 are displayed (again the minimum, maximum, mean, and rms
       of the parameter).  Then seriously discrepant points from these
       averages are displayed.  The full details may also be displayed
       although the full details of the baseline dependent ones go
       only to the output text file.  To put antenna-based weights on
       the same scale as baseline-based ones, the weight shown is that
       which would be applied to an autocorrelation (1/sigma^4).

       There are quite a large number of "knobs" that can be changed
       in this task to give dofferent results.  Task ANBPL may be a
       good way to check the results you obtain.  One experienced user
       recommends the following:
          APARM = 31 0 300 0 0 0 1.E-7 1000
       which uses a 31-sample buffer to obtain robust rmses, does not
       limit averaging time (usually scan boundaries will be the
       limit), smooths the rmses over a significant time to dampen out
       excessive variation using a default Gaussian function, and
       flags data with excessively small and large weights.
          BPARM(1) = 20,0
       requires a useful number of samples before an rms is computed.

  INNAME.....Input UV file name (name).      Standard defaults.
  INCLASS....Input UV file name (class).     Standard defaults.
  INSEQ......Input UV file name (seq. #).    0 => highest.
  INDISK.....Disk drive # of input UV file.  0 => any.
  SOURCES....Source to be copied.   '  '=> all; if any starts with a
             '-' then all except ANY source named.
  QUAL.......Qualifier of source to be copied. -1 => all.
  CALCODE....Calibrator code of sources to copy. ' '=> all.
  STOKES.....Specifies which STOKES parameters are written in the
             output data set:  ' ' => 'FULL'
               'I','Q','U','V', 'IV', 'IQU', 'IQUV'
               'RR','LL', 'RL', 'LR', 'RRLL', 'RLLR', 'RLRL'
               'VV','HH', 'VH', 'HV', 'VVHH', 'VHHV', 'VHVH'
             'HALF', 'CROS', and 'FULL' have sensible interpretations
             depending on the Stokes present in the data.  The last in
             each of the 3 rows above == 'FULL'.  Note that many
             combinations of polarizations in the input and values
             above are not supported.
  TIMERANG...Time range of the data to be copied. In order: Start day,
             hour, min. sec, end day, hour, min. sec. Days relative to
             ref. date.
  SELBAND....Bandwidth of data to be selected. If more than one IF is
             present SELBAND is the width of the first IF required.
             Units = kHz. For data which contain multiple
             bandwidths/frequencies the task will insist that some form
             of selection be made by frequency or bandwidth.
  SELFREQ....Frequency of data to be selected. If more than one IF is
             present SELFREQ is the frequency of the first IF required.
             Units = MHz.
  FREQID.....Frequency identifier to select (you may determine which is
             applicable from the OPTYPE='SCAN' listing produced by
             LISTR). If either SELBAND or SELFREQ are set, their values
             override that of FREQID.  However, setting SELBAND and
             SELFREQ may result in an ambiguity.  In that case, the task
             will request that you use FREQID.
  SUBARRAY...Sub-array number to copy. 0=>all.
  DOCALIB....If true (>0), calibrate the data using information in the
             specified Cal (CL) table for multi-source or SN table for
             single-source data.  Also calibrate the weights unless
             DOCALIB > 99 (use this for old non-physical weights).
  GAINUSE....version number of the CL table to apply to multi-source
             files or the SN table for single source files.
             0 => highest.
  DOPOL......If > 0 then correct data for instrumental polarization as
             represented in the AN or PD table.  This correction is
             only useful if PCAL has been run or feed polarization
             parameters have been otherwise obtained.  See HELP DOPOL
             for available correction modes: 1 is normal, 2 and 3 are
             for VLBI.  1-3 use a PD table if available; 6, 7, 8 are
             the same but use the AN (continuum solution) even if a PD
             table is present.
  PDVER......PD table to apply if PCAL was run with SPECTRAL true and
             0 < DOPOL < 6.  <= 0 => highest.
  FLAGVER....specifies the version of the flagging table to be applied.
              0 => highest numbered table.
             <0 => no flagging to be applied.
  DOBAND.....If true (>0) then correct the data for the shape of the
             antenna bandpasses using the BP table specified by BPVER.
             The correction has five modes:
             (a) if DOBAND=1 all entries for an antenna in the table
             are averaged together before correcting the data.
             (b) if DOBAND=2 the entry nearest in time (including
             solution weights) is used to correct the data.
             (c) if DOBAND=3 the table entries are interpolated in
             time (using solution weights) and the data are then
             (d) if DOBAND=4 the entry nearest in time (ignoring
             solution weights) is used to correct the data.
             (e) if DOBAND=5 the table entries are interpolated in
             time (ignoring solution weights) and the data are then
             IMAGR uses DOBAND as the nearest integer; 0.1 is therefore
  BPVER......Specifies the version of the BP table to be applied
                0 => highest numbered table.
               <0 => no bandpass correction to be applied.
  SMOOTH.....Specifies the type of spectral smoothing to be applied to
             a uv database . The default is not to apply any smoothing.
             The elements of SMOOTH are as follows:
             SMOOTH(1) = type of smoothing to apply: 0 => no smoothing
               To smooth before applying bandpass calibration
                 1 => Hanning, 2 => Gaussian, 3 => Boxcar, 4 => Sinc
               To smooth after applying bandpass calibration
                 5 => Hanning, 6 => Gaussian, 7 => Boxcar, 8 => Sinc
             SMOOTH(2) = the "diameter" of the function, i.e. width
               between first nulls of Hanning triangle and sinc
               function, FWHM of Gaussian, width of Boxcar. Defaults
               (if < 0.1) are 4, 2, 2 and 3 channels for SMOOTH(1) =
               1 - 4 and 5 - 8, resp.
             SMOOTH(3) = the diameter over which the convolving
               function has value - in channels.  Defaults: 1,3,1,4
               times SMOOTH(2) used when input SMOOTH(3) < net
  DOACOR.....> 0 => include autocorrelations as well as cross
             correlation data.
  OUTNAME....Output UV file name (name).     Standard defaults.
  OUTCLASS...Output UV file name (class).    Standard defaults.
  OUTSEQ.....Output UV file name (seq. #).   0 => highest unique
  OUTDISK....Disk drive # of output UV file. 0 => highest with space
             for the file.
  ICHANSEL.. Array of start, stop, and increment channel numbers plus
             an IF used for channel selection in the averaging to
             compute an rms.  Up to 20 sets if channels/IF may be
             entered.  The first having ICHANSEL(2,i) <= 0 terminates
             the list.  ICHANSEL(4,i) is the IF number, with <= 0
             meaning all IFs.  If an IF has no ICHANSEL set for it,
             then the inner 75 percent of that IF is used.
             For instance, if you wished to exclude channels 1 - 10
             and 121 - 128 because of bandpass effects, and channels
             56 - 80 of IF 1 but not IF 2 because of interference,
             then you would set ICHANSEL = 11,55,1,1, 81,121,1,1,
             11,121,1,2.  If you only wished to use every other
             channel from the second IF then you would set ICHANSEL =
             11,55,1,1, 81,121,1,1, 11,121,2,2.  if there are actually
             4 IFs, then IFs 3 and 4 would use channels 17 through
             112.  To set the channel range for all IFs to 14, 115
             enter ICHANSEL = 14,125.  To set the channel range for
             all IFs to 14-115 except IF 3 set ICHANSEL=14,115,1,0,
             23 45,1,3, 64,101,1,3 where the all IF part must come
             before the parts that partially override it.
  OUTFGVER...If solutions are flagged (based on APARM(5)-APARM(8)), a
             flag table may be written.  If OUTFGVER <= 0, this will
             not happen.  If > 0, a new FG table is made and the
             FLAGVER FG table copied to it.  If OUTFGVER points at an
             existing flag table other than FLAGVER, the new flags are
             written to that table.  If OUTFGVER points at FLAGVER,
             the program will fail so it is changed to make a new FG
             table instead with the contents of FLAGVER copied to it.
  APARM......(1) If = 0, the rmses are found on a per baseline per
                 IF and per Stokes basis and applied to the data as
                 weights = rms^(-2) directly after smoothing and
                 clipping controlled by APARM(3-8).
                 If < 0, the above rmses are averaged over -APARM(1)
                 seconds, converted to antenna-based rmses per IF per
                 Stokes.  They are then smoothed and clipped and
                 converted to baseline weights = 1/(rms1 * rms2).
                 If > 0, the data are accumlated in a buffer APARM(1)
                 time samples long, robust rmses are found for each
                 baseline, IF, and Stokes and are then turned into
                 baseline-based weights and saved in the WT table.
                 After optional smoothing, thw WT table is applied to
                 the data.
                 NO DEFAULT.
             (2) If APARM(1) > 0, do not average over times longer
                 than APARM(2) seconds.  0 -> 600.
             (3) The rmses found on an antenna or a baseline basis
                 may be smoothed with a function of width APARM(3).
                 If APARM(3) <= 0, then no further smoothing or
                 clipping of rmses is done.
                 <=0 means 0 for APARM(1) = 0,
                 <=0 -> 1 sec for the other modes.
             (4) Type of function: 0 -> Gaussian, width is FWHM,
                 1 -> Exponential, width is FWHM, 2 -> boxcar, width
                 is full support.  <=0 or >2 => 0
             (5) When smoothing rmses, only those rmses within
                 APARM(5) * SIGMA of the average are included.  The
                 average and sigma for this limit are on a per
                 baseline, per IF, per polarization basis.
                 0 -> 4; use large number to avoid clipping.
                 If the smoothed rms contains no points, the data
                 sample will be flagged.
             (6) The mean and rms over all baselines, IFs, and
                 polarizations may also be used to limit the rmses
                 used when smoothing.  Include only rmses withing
                 APARM(6) * Overall_SIGMA  of the overall mean rms.
                 0 -> 10000.  If the smoothed rms contains no points,
                 the data sample will be flagged.  This effectively
                 sets defaults for APARM(7) and APARM(8).
             (7) Flag data with weight < APARM(7).
             (8) Flag data with weight > APARM(8); 0 -> 10^10
                 The flagging is only applied if smoothing is also
                 done (APARM(2) > 0 and APARM(2) > APARM(1) when
                 APARM(1) < 0).
             (9) <= 0 The non-median methods use the data weights in
                      determining the rmses and both scale the output
                      weights with the input weights.
                 > 0  As above, but take incoming weights to be the
                      value of the INTTIME random parameter
                 > 1.5 As above but take the incoming weights to be
                      1.0 (i.e.ignore incoming weights entirely)
             (10) The rmses are worked on using a non-standard WT
                 table.  To keep this table, set APARM(10) > 0.
                 Note that the WT table is an extension file with a
                 table format like most other extension files.  It
                 contains rmses not weights (despite its name) and may
                 be plotted with TAPLT, printed with PRTAB.
  BPARM......(1) Set the weight to zero when the number of samples
                 entering the rms computation is < BPARM(1).  0 means
                 0 as in no limit.  The maximum number of samples in a
                 computation is the number of spectral channels in the
                 given spectral window (IF and polarization) times the
                 number of times in the window.  The actual number of
                 samples after any previous flagging is examined.
             (2) If > 0, keep statistics of the weights by antenna or
                 baseline and IF.  If = 1, print a summary of these
                 including those with extreme values of weight or
                 weight rms.  If = 2, print the full details when the
                 weights are antenna based (APARM(1) < 0).  Full
                 details will go to non-blank OUTTEXT.
             (3) The statistics averaged over all antennas or
                 baselines are printed first.  Then those antennas or
                 baselines for which the statistics differ from these
                 averages by more than BPARM(3) times the rms for the
                 same parameter will be printed.  All 4 statistics
                 (minimum, maximum, mean, rms weight) will be printed
                 but the offending one(s) will be followed by an *.
             (4) Weights are normally found separately for each
                 polarization.  However, for data with high S/N and
                 lots of source structure, the weights in the
                 parallel-hand correlations may be set low.  If
                 BPARM(4) > 0, the cross-hand polarization weights are
                 averaged and used for all 4 polarizations.
  OUTTEXT....Text file to be written with the summaries, outliers, and
             details of the statistics.  Written if not blank and
             BPARM(2) > 0.
  OPTYPE.....'MEDI' uses a median operation to do the statistics on
                    the visibility values.
             'MEDR' uses a median operation plus robust methods to do
                    the statistics (expensive, probably not worth it)
             other  uses simple statistics but with a robust aspect to
                    gradually discard outliers.  MEDI is modestly
                    faster than the robust method, but is more likely
                    to be fooled by real signal one would like to
  FQCENTER...>  0 => Change frequency axis reference pixel to
                     Nchan / 2 + 1
             else => do not change reference pixel
  BADDISK....Disk numbers on which scratch files are not to
             be placed.


APARM(1) appears to be confusing.  There are 3 algorithms:

APARM(1) = 0 is simple - each spectrum has a robust rms determined
over the specified channels and a weight = 1/rms^2 is stored.  Thus
the weight is independent in each polarization, IF, baseline, and

APARM(1) < 0 is more complicated.  Each rms is found as above.  Then
the RMSes are averaged over an interval -APARM(1) seconds.  Then the
rmses are turned into antenna-based rmses (separately for each
polarization and IF) and stored in a temporary WT table.  Optionally,
the WT table may be smoothed, clipped, and data flags found.  Then the
data are re-read and the weights and flags in the WT table are applied
to the output file.

APARM(1) > 0 is even more complicated and may require very large
amounts of dynamic memory.  All data for APARM(1) times are read into
memory.  Then the rms for each baseline, IF, and Stokes is determined
by a robust average over all spectral channels and times in the
buffer.  These rmses are turned into antenna-based weights for the
central time in the buffer and stored in the WT file.  Then the first
record in the buffer is discarded and replaced with the next time and
the process is repeated.  APARM(1) must be odd so that the midpooint
of the buffer is well determined.  Note that the first (APARM(1)+1)/2
times and the last (APARM(1)+1)/2 times in a scan will have the same
value so that the number of times entering in each weight will be

For example, let APARM(1)=+9.  The first 9 times are read into memory
and the rmses and antenna-based weights are found and stored in the WT
table.  These are written to the WT table for times 1 through 5.  Then
time 1 is discarded and time 10 read.  The process repeats and the
time and weights for time 6 are written in the WT table.  Then time 2
is deleted and time 11 read and so forth.  When the end of the scan is
reached, the rolling buffer is cleared out much like at the start and
a new scan is processed.  Scan boundaries are found when the source
name changes or when APARM(2) seconds have passed from the first time
currently in the rolling buffer.

The WT table so written can then be smoothed, clipped, flagged etc
and the data are re-read and weighted as above.

There are quite a large number of "knobs" that can be changed in this
task to give different results.  Task ANBPL may be a good way to check
the results you obtain.  One experienced user recommends the following:
       APARM = 31 0 300 0 0 0 1.E-7 1000
which uses a 31-sample buffer to obtain robust rmses, does not limit
averaging time (usually scan boundaries will be the limit), smooths
the rmses over a significant time to dampen out excessive variation
using a default Gaussian function, and flags data with excessively
small and large weights.
       BPARM(1) = 20,0
requires a useful number of samples before an rms is computed.