AIPS HELP file for REWAY in 31DEC24
As of Fri Apr 26 14:47:07 2024
REWAY: Computes data weights based solely on rms in spectra
INPUTS
INNAME Input UV file name (name)
INCLASS Input UV file name (class)
INSEQ 0.0 9999.0 Input UV file name (seq. #)
INDISK 0.0 9.0 Input UV file disk unit #
SOURCES Source name
QUAL -10.0 Calibrator qualifier -1=>all
CALCODE Calibrator code ' '=>all
TIMERANG Time range to use
SELBAND Bandwidth to select (kHz)
SELFREQ Frequency to select (MHz)
FREQID Freq. ID to select.
SUBARRAY 0.0 1000.0 Sub-array, 0=>all
DOCALIB -1.0 101.0 > 0 calibrate data & weights
> 99 do NOT calibrate weights
GAINUSE CL (or SN) table to apply
DOPOL -1.0 10.0 If >0.5 correct polarization.
PDVER PD table to apply (DOPOL>0)
BLVER BL table to apply.
FLAGVER Flag table version
DOBAND -1.0 10.0 If >0.5 apply bandpass cal.
Method used depends on value
of DOBAND (see HELP file).
BPVER Bandpass table version
SMOOTH Smoothing function. See
HELP SMOOTH for details.
DOACOR Include autocorrelations?
OUTNAME Output UV file name (name)
OUTCLASS Output UV file name (class)
OUTSEQ -1.0 9999.0 Output UV file name (seq. #)
OUTDISK 0.0 9.0 Output UV file disk unit #.
ICHANSEL Array of start and stop chan
numbers, plus a channel
increment and IF to be used
to select channels to sum to
find RMS. If all 0, range
set to inner 75 percent of band.
OUTFGVER -1.0 Output flag table: 0 none
APARM (1) > 0 rolling T buffer to
find baseline weights
A(1) times
= 0 find baseline weights
with no averaging
< 0 find antenna weights
averaged on |A(1)| sec
(2) Max time in buffer if
A(1) > 0 in sec 0 -> 600
(3) Smooth rmses with func
width = A(3) sec.
(4) Type func: 0 Gaussian,
1 EXP, 2 boxcar
(5) ignore pts > A(5) sigma
in smoothing. < 1 -> 4
can lead to flagging
(6) ignore pts > A(6) sigma
overall in smoothing
can lead to flagging
(7) flag data w wt < A(7)
(8) flag data w wt > A(8)
(9) > 0 => ignore incoming
data weights
> 1.5 => ignore INTTIME
also
(10) > 0 => keep WT table
as extension file
BPARM (1) Set weight to zero if <
B(1) samples in window
(2) > 0 compute statistics
of weights, 1 give a
summary and outliers,
2 give details when
APARM(1) < 0
(3) Outliers are BPARM(3)
sigma from the average
0 -> 5
(4) > 0 => substitute cross-
hand weights into all
OUTTEXT Not blank - write details of
statistics in this text file
OPTYPE 'MEDI' use median window
solver for mean/rms else use
"robust" methods
FQCENTER >= 0 -> center frequency axis
BADDISK Disk drive #'s to avoid
HELP SECTION
REWAY
Task: This task will copy selected portions of a data set (much like
SPLIT) and change the data weights by computing the uncertainty
using the multiple spectral channels in each IF and
polarization. Note, there must be enough spectral channels to
make this computation meaningful. Furthermore, all possible
calibrations should be applied so that, in particular, any
delay and bandpass shape are removed. Those calibrations will
greatly reduce the apparent rms. EVLA weights are not
calibrated when DOCALIB is true until after application of
TYAPL or REWAY. Therefore, for the EVLA, the full gain
calibration should be made so that the weights computed here
are in Jy^(-2) on the same scale for all baselines.
The task offers the option to ignore the input data weights
(except for flagging). If they are not ignored, then the
robust rms methods use the weights in the rms computation and
the "rms" that is stored in the WT table, smoothed, and used in
flagging is the actual rms scaled by square root of the average
data weight. When the rmses are converted to data weights they
are used to scale the input weights.
Output files are always uncompressed so as to avoid losing any
weights which might be different in different IFs. Input files
may be compressed or uncompressed.
The task computes the minimum, maximum, mean, and rms of the
weights by antennas (APARM(3)<0) or baselines (APARM(1)>= 0)
separated also by IF and Stokes. These statistics may be
displayed in the message window and written to a text file.
First the averages over all antennas or baselines in each of
the 4 are displayed (again the minimum, maximum, mean, and rms
of the parameter). Then seriously discrepant points from these
averages are displayed. The full details may also be displayed
although the full details of the baseline dependent ones go
only to the output text file. To put antenna-based weights on
the same scale as baseline-based ones, the weight shown is that
which would be applied to an autocorrelation (1/sigma^4).
There are quite a large number of "knobs" that can be changed
in this task to give dofferent results. Task ANBPL may be a
good way to check the results you obtain. One experienced user
recommends the following:
APARM = 31 0 300 0 0 0 1.E-7 1000
which uses a 31-sample buffer to obtain robust rmses, does not
limit averaging time (usually scan boundaries will be the
limit), smooths the rmses over a significant time to dampen out
excessive variation using a default Gaussian function, and
flags data with excessively small and large weights.
BPARM(1) = 20,0
requires a useful number of samples before an rms is computed.
Adverbs:
INNAME.....Input UV file name (name). Standard defaults.
INCLASS....Input UV file name (class). Standard defaults.
INSEQ......Input UV file name (seq. #). 0 => highest.
INDISK.....Disk drive # of input UV file. 0 => any.
SOURCES....Source to be copied. ' '=> all; if any starts with a
'-' then all except ANY source named.
QUAL.......Qualifier of source to be copied. -1 => all.
CALCODE....Calibrator code of sources to copy. ' '=> all.
STOKES.....Specifies which STOKES parameters are written in the
output data set: ' ' => 'FULL'
'I','Q','U','V', 'IV', 'IQU', 'IQUV'
'RR','LL', 'RL', 'LR', 'RRLL', 'RLLR', 'RLRL'
'VV','HH', 'VH', 'HV', 'VVHH', 'VHHV', 'VHVH'
'HALF', 'CROS', and 'FULL' have sensible interpretations
depending on the Stokes present in the data. The last in
each of the 3 rows above == 'FULL'. Note that many
combinations of polarizations in the input and values
above are not supported.
TIMERANG...Time range of the data to be copied. In order: Start day,
hour, min. sec, end day, hour, min. sec. Days relative to
ref. date.
SELBAND....Bandwidth of data to be selected. If more than one IF is
present SELBAND is the width of the first IF required.
Units = kHz. For data which contain multiple
bandwidths/frequencies the task will insist that some form
of selection be made by frequency or bandwidth.
SELFREQ....Frequency of data to be selected. If more than one IF is
present SELFREQ is the frequency of the first IF required.
Units = MHz.
FREQID.....Frequency identifier to select (you may determine which is
applicable from the OPTYPE='SCAN' listing produced by
LISTR). If either SELBAND or SELFREQ are set, their values
override that of FREQID. However, setting SELBAND and
SELFREQ may result in an ambiguity. In that case, the task
will request that you use FREQID.
SUBARRAY...Sub-array number to copy. 0=>all.
DOCALIB....If true (>0), calibrate the data using information in the
specified Cal (CL) table for multi-source or SN table for
single-source data. Also calibrate the weights unless
DOCALIB > 99 (use this for old non-physical weights).
GAINUSE....version number of the CL table to apply to multi-source
files or the SN table for single source files.
0 => highest.
DOPOL......If > 0 then correct data for instrumental polarization as
represented in the AN or PD table. This correction is
only useful if PCAL has been run or feed polarization
parameters have been otherwise obtained. See HELP DOPOL
for available correction modes: 1 is normal, 2 and 3 are
for VLBI. 1-3 use a PD table if available; 6, 7, 8 are
the same but use the AN (continuum solution) even if a PD
table is present.
PDVER......PD table to apply if PCAL was run with SPECTRAL true and
0 < DOPOL < 6. <= 0 => highest.
FLAGVER....specifies the version of the flagging table to be applied.
0 => highest numbered table.
<0 => no flagging to be applied.
DOBAND.....If true (>0) then correct the data for the shape of the
antenna bandpasses using the BP table specified by BPVER.
The correction has five modes:
(a) if DOBAND=1 all entries for an antenna in the table
are averaged together before correcting the data.
(b) if DOBAND=2 the entry nearest in time (including
solution weights) is used to correct the data.
(c) if DOBAND=3 the table entries are interpolated in
time (using solution weights) and the data are then
corrected.
(d) if DOBAND=4 the entry nearest in time (ignoring
solution weights) is used to correct the data.
(e) if DOBAND=5 the table entries are interpolated in
time (ignoring solution weights) and the data are then
corrected.
IMAGR uses DOBAND as the nearest integer; 0.1 is therefore
"false".
BPVER......Specifies the version of the BP table to be applied
0 => highest numbered table.
<0 => no bandpass correction to be applied.
SMOOTH.....Specifies the type of spectral smoothing to be applied to
a uv database . The default is not to apply any smoothing.
The elements of SMOOTH are as follows:
SMOOTH(1) = type of smoothing to apply: 0 => no smoothing
To smooth before applying bandpass calibration
1 => Hanning, 2 => Gaussian, 3 => Boxcar, 4 => Sinc
To smooth after applying bandpass calibration
5 => Hanning, 6 => Gaussian, 7 => Boxcar, 8 => Sinc
SMOOTH(2) = the "diameter" of the function, i.e. width
between first nulls of Hanning triangle and sinc
function, FWHM of Gaussian, width of Boxcar. Defaults
(if < 0.1) are 4, 2, 2 and 3 channels for SMOOTH(1) =
1 - 4 and 5 - 8, resp.
SMOOTH(3) = the diameter over which the convolving
function has value - in channels. Defaults: 1,3,1,4
times SMOOTH(2) used when input SMOOTH(3) < net
SMOOTH(2).
DOACOR.....> 0 => include autocorrelations as well as cross
correlation data.
OUTNAME....Output UV file name (name). Standard defaults.
OUTCLASS...Output UV file name (class). Standard defaults.
OUTSEQ.....Output UV file name (seq. #). 0 => highest unique
OUTDISK....Disk drive # of output UV file. 0 => highest with space
for the file.
ICHANSEL.. Array of start, stop, and increment channel numbers plus
an IF used for channel selection in the averaging to
compute an rms. Up to 20 sets if channels/IF may be
entered. The first having ICHANSEL(2,i) <= 0 terminates
the list. ICHANSEL(4,i) is the IF number, with <= 0
meaning all IFs. If an IF has no ICHANSEL set for it,
then the inner 75 percent of that IF is used.
For instance, if you wished to exclude channels 1 - 10
and 121 - 128 because of bandpass effects, and channels
56 - 80 of IF 1 but not IF 2 because of interference,
then you would set ICHANSEL = 11,55,1,1, 81,121,1,1,
11,121,1,2. If you only wished to use every other
channel from the second IF then you would set ICHANSEL =
11,55,1,1, 81,121,1,1, 11,121,2,2. if there are actually
4 IFs, then IFs 3 and 4 would use channels 17 through
112. To set the channel range for all IFs to 14, 115
enter ICHANSEL = 14,125. To set the channel range for
all IFs to 14-115 except IF 3 set ICHANSEL=14,115,1,0,
23 45,1,3, 64,101,1,3 where the all IF part must come
before the parts that partially override it.
OUTFGVER...If solutions are flagged (based on APARM(5)-APARM(8)), a
flag table may be written. If OUTFGVER <= 0, this will
not happen. If > 0, a new FG table is made and the
FLAGVER FG table copied to it. If OUTFGVER points at an
existing flag table other than FLAGVER, the new flags are
written to that table. If OUTFGVER points at FLAGVER,
the program will fail so it is changed to make a new FG
table instead with the contents of FLAGVER copied to it.
APARM......(1) If = 0, the rmses are found on a per baseline per
IF and per Stokes basis and applied to the data as
weights = rms^(-2) directly after smoothing and
clipping controlled by APARM(3-8).
If < 0, the above rmses are averaged over -APARM(1)
seconds, converted to antenna-based rmses per IF per
Stokes. They are then smoothed and clipped and
converted to baseline weights = 1/(rms1 * rms2).
If > 0, the data are accumlated in a buffer APARM(1)
time samples long, robust rmses are found for each
baseline, IF, and Stokes and are then turned into
baseline-based weights and saved in the WT table.
After optional smoothing, thw WT table is applied to
the data.
NO DEFAULT.
(2) If APARM(1) > 0, do not average over times longer
than APARM(2) seconds. 0 -> 600.
(3) The rmses found on an antenna or a baseline basis
may be smoothed with a function of width APARM(3).
If APARM(3) <= 0, then no further smoothing or
clipping of rmses is done.
<=0 means 0 for APARM(1) = 0,
<=0 -> 1 sec for the other modes.
(4) Type of function: 0 -> Gaussian, width is FWHM,
1 -> Exponential, width is FWHM, 2 -> boxcar, width
is full support. <=0 or >2 => 0
(5) When smoothing rmses, only those rmses within
APARM(5) * SIGMA of the average are included. The
average and sigma for this limit are on a per
baseline, per IF, per polarization basis.
0 -> 4; use large number to avoid clipping.
If the smoothed rms contains no points, the data
sample will be flagged.
(6) The mean and rms over all baselines, IFs, and
polarizations may also be used to limit the rmses
used when smoothing. Include only rmses withing
APARM(6) * Overall_SIGMA of the overall mean rms.
0 -> 10000. If the smoothed rms contains no points,
the data sample will be flagged. This effectively
sets defaults for APARM(7) and APARM(8).
(7) Flag data with weight < APARM(7).
(8) Flag data with weight > APARM(8); 0 -> 10^10
The flagging is only applied if smoothing is also
done (APARM(2) > 0 and APARM(2) > APARM(1) when
APARM(1) < 0).
(9) <= 0 The non-median methods use the data weights in
determining the rmses and both scale the output
weights with the input weights.
> 0 As above, but take incoming weights to be the
value of the INTTIME random parameter
> 1.5 As above but take the incoming weights to be
1.0 (i.e.ignore incoming weights entirely)
(10) The rmses are worked on using a non-standard WT
table. To keep this table, set APARM(10) > 0.
Note that the WT table is an extension file with a
table format like most other extension files. It
contains rmses not weights (despite its name) and may
be plotted with TAPLT, printed with PRTAB.
BPARM......(1) Set the weight to zero when the number of samples
entering the rms computation is < BPARM(1). 0 means
0 as in no limit. The maximum number of samples in a
computation is the number of spectral channels in the
given spectral window (IF and polarization) times the
number of times in the window. The actual number of
samples after any previous flagging is examined.
(2) If > 0, keep statistics of the weights by antenna or
baseline and IF. If = 1, print a summary of these
including those with extreme values of weight or
weight rms. If = 2, print the full details when the
weights are antenna based (APARM(1) < 0). Full
details will go to non-blank OUTTEXT.
(3) The statistics averaged over all antennas or
baselines are printed first. Then those antennas or
baselines for which the statistics differ from these
averages by more than BPARM(3) times the rms for the
same parameter will be printed. All 4 statistics
(minimum, maximum, mean, rms weight) will be printed
but the offending one(s) will be followed by an *.
(4) Weights are normally found separately for each
polarization. However, for data with high S/N and
lots of source structure, the weights in the
parallel-hand correlations may be set low. If
BPARM(4) > 0, the cross-hand polarization weights are
averaged and used for all 4 polarizations.
OUTTEXT....Text file to be written with the summaries, outliers, and
details of the statistics. Written if not blank and
BPARM(2) > 0.
OPTYPE.....'MEDI' uses a median operation to do the statistics on
the visibility values.
'MEDR' uses a median operation plus robust methods to do
the statistics (expensive, probably not worth it)
other uses simple statistics but with a robust aspect to
gradually discard outliers. MEDI is modestly
faster than the robust method, but is more likely
to be fooled by real signal one would like to
ignore.
FQCENTER...> 0 => Change frequency axis reference pixel to
Nchan / 2 + 1
else => do not change reference pixel
BADDISK....Disk numbers on which scratch files are not to
be placed.
EXPLAIN SECTION
APARM(1) appears to be confusing. There are 3 algorithms:
APARM(1) = 0 is simple - each spectrum has a robust rms determined
over the specified channels and a weight = 1/rms^2 is stored. Thus
the weight is independent in each polarization, IF, baseline, and
time.
APARM(1) < 0 is more complicated. Each rms is found as above. Then
the RMSes are averaged over an interval -APARM(1) seconds. Then the
rmses are turned into antenna-based rmses (separately for each
polarization and IF) and stored in a temporary WT table. Optionally,
the WT table may be smoothed, clipped, and data flags found. Then the
data are re-read and the weights and flags in the WT table are applied
to the output file.
APARM(1) > 0 is even more complicated and may require very large
amounts of dynamic memory. All data for APARM(1) times are read into
memory. Then the rms for each baseline, IF, and Stokes is determined
by a robust average over all spectral channels and times in the
buffer. These rmses are turned into antenna-based weights for the
central time in the buffer and stored in the WT file. Then the first
record in the buffer is discarded and replaced with the next time and
the process is repeated. APARM(1) must be odd so that the midpooint
of the buffer is well determined. Note that the first (APARM(1)+1)/2
times and the last (APARM(1)+1)/2 times in a scan will have the same
value so that the number of times entering in each weight will be
constant.
For example, let APARM(1)=+9. The first 9 times are read into memory
and the rmses and antenna-based weights are found and stored in the WT
table. These are written to the WT table for times 1 through 5. Then
time 1 is discarded and time 10 read. The process repeats and the
time and weights for time 6 are written in the WT table. Then time 2
is deleted and time 11 read and so forth. When the end of the scan is
reached, the rolling buffer is cleared out much like at the start and
a new scan is processed. Scan boundaries are found when the source
name changes or when APARM(2) seconds have passed from the first time
currently in the rolling buffer.
The WT table so written can then be smoothed, clipped, flagged etc
and the data are re-read and weighted as above.
There are quite a large number of "knobs" that can be changed in this
task to give different results. Task ANBPL may be a good way to check
the results you obtain. One experienced user recommends the following:
APARM = 31 0 300 0 0 0 1.E-7 1000
which uses a 31-sample buffer to obtain robust rmses, does not limit
averaging time (usually scan boundaries will be the limit), smooths
the rmses over a significant time to dampen out excessive variation
using a default Gaussian function, and flags data with excessively
small and large weights.
BPARM(1) = 20,0
requires a useful number of samples before an rms is computed.