; REWAY ;--------------------------------------------------------------- ;! computes weights based in rms in spectra ;# Task UV VLA ;----------------------------------------------------------------------- ;; Copyright (C) 2010-2011, 2013-2014, 2016 ;; Associated Universities, Inc. Washington DC, USA. ;; ;; This program is free software; you can redistribute it and/or ;; modify it under the terms of the GNU General Public License as ;; published by the Free Software Foundation; either version 2 of ;; the License, or (at your option) any later version. ;; ;; This program is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; ;; You should have received a copy of the GNU General Public ;; License along with this program; if not, write to the Free ;; Software Foundation, Inc., 675 Massachusetts Ave, Cambridge, ;; MA 02139, USA. ;; ;; Correspondence concerning AIPS should be addressed as follows: ;; Internet email: aipsmail@nrao.edu. ;; Postal address: AIPS Project Office ;; National Radio Astronomy Observatory ;; 520 Edgemont Road ;; Charlottesville, VA 22903-2475 USA ;----------------------------------------------------------------------- REWAY LLLLLLLLLLLLUUUUUUUUUUUU CCCCCCCCCCCCCCCCCCCCCCCCCCCCC REWAY Computes data weights based solely on rms in spectra INNAME Input UV file name (name) INCLASS Input UV file name (class) INSEQ 0.0 9999.0 Input UV file name (seq. #) INDISK 0.0 9.0 Input UV file disk unit # SOURCES Source name QUAL -10.0 Calibrator qualifier -1=>all CALCODE Calibrator code ' '=>all TIMERANG Time range to use SELBAND Bandwidth to select (kHz) SELFREQ Frequency to select (MHz) FREQID Freq. ID to select. SUBARRAY 0.0 1000.0 Sub-array, 0=>all DOCALIB -1.0 101.0 > 0 calibrate data & weights > 99 do NOT calibrate weights GAINUSE CL (or SN) table to apply DOPOL -1.0 10.0 If >0.5 correct polarization. PDVER PD table to apply (DOPOL>0) BLVER BL table to apply. FLAGVER Flag table version DOBAND -1.0 10.0 If >0.5 apply bandpass cal. Method used depends on value of DOBAND (see HELP file). BPVER Bandpass table version SMOOTH Smoothing function. See HELP SMOOTH for details. DOACOR Include autocorrelations? OUTNAME Output UV file name (name) OUTCLASS Output UV file name (class) OUTSEQ -1.0 9999.0 Output UV file name (seq. #) OUTDISK 0.0 9.0 Output UV file disk unit #. ICHANSEL Array of start and stop chan numbers, plus a channel increment and IF to be used to select channels to sum to find RMS. If all 0, range set to inner 75% of band. OUTFGVER -1.0 Output flag table: 0 none APARM (1) > 0 rolling T buffer to find baseline weights A(1) times = 0 find baseline weights with no averaging < 0 find antenna weights averaged on |A(1)| sec (2) Max time in buffer if A(1) > 0 in sec 0 -> 600 (3) Smooth rmses with func width = A(3) sec. (4) Type func: 0 Gaussian, 1 EXP, 2 boxcar (5) ignore pts > A(5) sigma in smoothing. < 1 -> 4 can lead to flagging (6) ignore pts > A(6) sigma overall in smoothing can lead to flagging (7) flag data w wt < A(7) (8) flag data w wt > A(8) (9) > 0 => ignore incoming data weights > 1.5 => ignore INTTIME also (10) > 0 => keep WT table as extension file BPARM (1) Set weight to zero if < B(1) samples in window (2) > 0 compute statistics of weights, 1 give a summary and outliers, 2 give details when APARM(1) < 0 (3) Outliers are BPARM(3) sigma from the average 0 -> 5 OUTTEXT Not blank - write details of statistics in this text file OPTYPE 'MEDI' use median window solver for mean/rms else use "robust" methods FQCENTER >= 0 -> center frequency axis BADDISK Disk drive #'s to avoid ---------------------------------------------------------------- REWAY Task: This task will copy selected portions of a data set (much like SPLIT) and change the data weights by computing the uncertainty using the multiple spectral channels in each IF and polarization. Note, there must be enough spectral channels to make this computation meaningful. Furthermore, all possible calibrations should be applied so that, in particular, any delay and bandpass shape are removed. Those calibrations will greatly reduce the apparent rms. EVLA weights are not calibrated when DOCALIB is true until after application of TYAPL or REWAY. Therefore, for the EVLA, the full gain calibration should be made so that the weights computed here are in Jy^(-2) on the same scale for all baselines. The task offers the option to ignore the input data weights (except for flagging). If they are not ignored, then the robust rms methods use the weights in the rms computation and the "rms" that is stored in the WT table, smoothed, and used in flagging is the actual rms scaled by square root of the average data weight. When the rmses are converted to data weights they are used to scale the input weights. Output files are always uncompressed so as to avoid losing any weights which might be different in different IFs. Input files may be compressed or uncompressed. The task computes the minimum, maximum, mean, and rms of the weights by antennas (APARM(3)<0) or baselines (APARM(1)>= 0) separated also by IF and Stokes. These statistics may be displayed in the message window and written to a text file. First the averages over all antennas or baselines in each of the 4 are displayed (again the minimum, maximum, mean, and rms of the parameter). Then seriously discrepant points from these averages are displayed. The full details may also be displayed although the full details of the baseline dependent ones go only to the output text file. To put antenna-based weights on the same scale as baseline-based ones, the weight shown is that which would be applied to an autocorrelation (1/sigma^4). There are quite a large number of "knobs" that can be changed in this task to give dofferent results. Task ANBPL may be a good way to check the results you obtain. One experienced user recommends the following: APARM = 31 0 300 0 0 0 1.E-7 1000 which uses a 31-sample buffer to obtain robust rmses, does not limit averaging time (usually scan boundaries will be the limit), smooths the rmses over a significant time to dampen out excessive variation using a default Gaussian function, and flags data with excessively small and large weights. BPARM(1) = 20,0 requires a useful number of samples before an rms is computed. Adverbs: INNAME.....Input UV file name (name). Standard defaults. INCLASS....Input UV file name (class). Standard defaults. INSEQ......Input UV file name (seq. #). 0 => highest. INDISK.....Disk drive # of input UV file. 0 => any. SOURCES....Source to be copied. ' '=> all; if any starts with a '-' then all except ANY source named. QUAL.......Qualifier of source to be copied. -1 => all. CALCODE....Calibrator code of sources to copy. ' '=> all. STOKES.....Specifies which STOKES parameters are written in the output data set: ' ' => 'FULL' 'I','Q','U','V', 'IV', 'IQU', 'IQUV' 'RR','LL', 'RL', 'LR', 'RRLL', 'RLLR', 'RLRL' 'XX','YY', 'XY', 'YX', 'XXYY', 'XYYX', 'XYXY' 'HALF', 'CROS', and 'FULL' have sensible interpretations depending on the Stokes present in the data. The last in each of the 3 rows above == 'FULL'. Note that many combinations of polarizations in the input and values above are not supported. TIMERANG...Time range of the data to be copied. In order: Start day, hour, min. sec, end day, hour, min. sec. Days relative to ref. date. SELBAND....Bandwidth of data to be selected. If more than one IF is present SELBAND is the width of the first IF required. Units = kHz. For data which contain multiple bandwidths/frequencies the task will insist that some form of selection be made by frequency or bandwidth. SELFREQ....Frequency of data to be selected. If more than one IF is present SELFREQ is the frequency of the first IF required. Units = MHz. FREQID.....Frequency identifier to select (you may determine which is applicable from the OPTYPE='SCAN' listing produced by LISTR). If either SELBAND or SELFREQ are set, their values override that of FREQID. However, setting SELBAND and SELFREQ may result in an ambiguity. In that case, the task will request that you use FREQID. SUBARRAY...Sub-array number to copy. 0=>all. DOCALIB....If true (>0), calibrate the data using information in the specified Cal (CL) table for multi-source or SN table for single-source data. Also calibrate the weights unless DOCALIB > 99 (use this for old non-physical weights). GAINUSE....version number of the CL table to apply to multi-source files or the SN table for single source files. 0 => highest. DOPOL......If > 0 then correct data for instrumental polarization as represented in the AN or PD table. This correction is only useful if PCAL has been run or feed polarization parameters have been otherwise obtained. See HELP DOPOL for available correction modes: 1 is normal, 2 and 3 are for VLBI. 1-3 use a PD table if available; 6, 7, 8 are the same but use the AN (continuum solution) even if a PD table is present. PDVER......PD table to apply if PCAL was run with SPECTRAL true and 0 < DOPOL < 6. <= 0 => highest. FLAGVER....specifies the version of the flagging table to be applied. 0 => highest numbered table. <0 => no flagging to be applied. DOBAND.....If true (>0) then correct the data for the shape of the antenna bandpasses using the BP table specified by BPVER. The correction has five modes: (a) if DOBAND=1 all entries for an antenna in the table are averaged together before correcting the data. (b) if DOBAND=2 the entry nearest in time (including solution weights) is used to correct the data. (c) if DOBAND=3 the table entries are interpolated in time (using solution weights) and the data are then corrected. (d) if DOBAND=4 the entry nearest in time (ignoring solution weights) is used to correct the data. (e) if DOBAND=5 the table entries are interpolated in time (ignoring solution weights) and the data are then corrected. IMAGR uses DOBAND as the nearest integer; 0.1 is therefore "false". BPVER......Specifies the version of the BP table to be applied 0 => highest numbered table. <0 => no bandpass correction to be applied. SMOOTH.....Specifies the type of spectral smoothing to be applied to a uv database . The default is not to apply any smoothing. The elements of SMOOTH are as follows: SMOOTH(1) = type of smoothing to apply: 0 => no smoothing To smooth before applying bandpass calibration 1 => Hanning, 2 => Gaussian, 3 => Boxcar, 4 => Sinc To smooth after applying bandpass calibration 5 => Hanning, 6 => Gaussian, 7 => Boxcar, 8 => Sinc SMOOTH(2) = the "diameter" of the function, i.e. width between first nulls of Hanning triangle and sinc function, FWHM of Gaussian, width of Boxcar. Defaults (if < 0.1) are 4, 2, 2 and 3 channels for SMOOTH(1) = 1 - 4 and 5 - 8, resp. SMOOTH(3) = the diameter over which the convolving function has value - in channels. Defaults: 1,3,1,4 times SMOOTH(2) used when input SMOOTH(3) < net SMOOTH(2). DOACOR.....> 0 => include autocorrelations as well as cross correlation data. OUTNAME....Output UV file name (name). Standard defaults. OUTCLASS...Output UV file name (class). Standard defaults. OUTSEQ.....Output UV file name (seq. #). 0 => highest unique OUTDISK....Disk drive # of output UV file. 0 => highest with space for the file. ICHANSEL.. Array of start, stop, and increment channel numbers plus an IF used for channel selection in the averaging to compute an rms. Up to 20 sets if channels/IF may be entered. The first having ICHANSEL(2,i) <= 0 terminates the list. ICHANSEL(4,i) is the IF number, with <= 0 meaning all IFs. If an IF has no ICHANSEL set for it, then the inner 75% of that IF is used. OUTFGVER...If solutions are flagged (based on APARM(5)-APARM(8)), a flag table may be written. If OUTFGVER <= 0, this will not happen. If > 0, a new FG table is made and the FLAGVER FG table copied to it. If OUTFGVER points at an existing flag table other than FLAGVER, the new flags are written to that table. If OUTFGVER points at FLAGVER, the program will fail so it is changed to make a new FG table instead with the contents of FLAGVER copied to it. APARM......(1) If = 0, the rmses are found on a per baseline per IF and per Stokes basis and applied to the data as weights = rms^(-2) directly after smoothing and clipping controlled by APARM(3-8). If < 0, the above rmses are averaged over -APARM(1) seconds, converted to antenna-based rmses per IF per Stokes. They are then smoothed and clipped and converted to baseline weights = 1/(rms1 * rms2). If > 0, the data are accumlated in a buffer APARM(1) time samples long, robust rmses are found for each baseline, IF, and Stokes and are then turned into baseline-based weights and saved in the WT table. After optional smoothing, thw WT table is applied to the data. NO DEFAULT. (2) If APARM(1) > 0, do not average over times longer than APARM(2) seconds. 0 -> 600. (3) The rmses found on an antenna or a baseline basis may be smoothed with a function of width APARM(3). If APARM(3) <= 0, then no further smoothing or clipping of rmses is done. <=0 means 0 for APARM(1) = 0, <=0 -> 1 sec for the other modes. (4) Type of function: 0 -> Gaussian, width is FWHM, 1 -> Exponential, width is FWHM, 2 -> boxcar, width is full support. <=0 or >2 => 0 (5) When smoothing rmses, only those rmses within APARM(5) * SIGMA of the average are included. The average and sigma for this limit are on a per baseline, per IF, per polarization basis. 0 -> 4; use large number to avoid clipping. If the smoothed rms contains no points, the data sample will be flagged. (6) The mean and rms over all baselines, IFs, and polarizations may also be used to limit the rmses used when smoothing. Include only rmses withing APARM(6) * Overall_SIGMA of the overall mean rms. 0 -> 10000. If the smoothed rms contains no points, the data sample will be flagged. This effectively sets defaults for APARM(7) and APARM(8). (7) Flag data with weight < APARM(7). (8) Flag data with weight > APARM(8); 0 -> 10^10 The flagging is only applied if smoothing is also done (APARM(2) > 0 and APARM(2) > APARM(1) when APARM(1) < 0). (9) <= 0 The non-median methods use the data weights in determining the rmses and both scale the output weights with the input weights. > 0 As above, but take incoming weights to be the value of the INTTIME random parameter > 1.5 As above but take the incoming weights to be 1.0 (i.e.ignore incoming weights entirely) (10) The rmses are worked on using a non-standard WT table. To keep this table, set APARM(10) > 0. Note that the WT table is an extension file with a table format like most other extension files. It contains rmses not weights (despite its name) and may be plotted with TAPLT, printed with PRTAB. BPARM......(1) Set the weight to zero when the number of samples entering the rms computation is < BPARM(1). 0 means 0 as in no limit. The maximum number of samples in a computation is the number of spectral channels in the given spectral window (IF and polarization) times the number of times in the window. The actual number of samples after any previous flagging is examined. (2) If > 0, keep statistics of the weights by antenna or baseline and IF. If = 1, print a summary of these including those with extreme values of weight or weight rms. If = 2, print the full details when the weights are antenna based (APARM(1) < 0). Full details will go to non-blank OUTTEXT. (3) The statistics averaged over all antennas or baselines are printed first. Then those antennas or baselines for which the statistics differ from these averages by more than BPARM(3) times the rms for the same parameter will be printed. All 4 statistics (minimum, maximum, mean, rms weight) will be printed but the offending one(s) will be followed by an *. OUTTEXT....Text file to be written with the summaries, outliers, and details of the statistics. Written if not blank and BPARM(2) > 0. OPTYPE.....'MEDI' uses a median operation to do the statistics on the visibility values. 'MEDR' uses a median operation plus robust methods to do the statistics (expensive, probably not worth it) other uses simple statistics but with a robust aspect to gradually discard outliers. MEDI is modestly faster than the robust method, but is more likely to be fooled by real signal one would like to ignore. FQCENTER...> 0 => Change frequency axis reference pixel to Nchan / 2 + 1 else => do not change reference pixel BADDISK....Disk numbers on which scratch files are not to be placed. ---------------------------------------------------------------- APARM(1) appears to be confusing. There are 3 algorithms: APARM(1) = 0 is simple - each spectrum has a robust rms determined over the specified channels and a weight = 1/rms^2 is stored. Thus the weight is independent in each polarization, IF, baseline, and time. APARM(1) < 0 is more complicated. Each rms is found as above. Then the RMSes are averaged over an interval -APARM(1) seconds. Then the rmses are turned into antenna-based rmses (separately for each polarization and IF) and stored in a temporary WT table. Optionally, the WT table may be smoothed, clipped, and data flags found. Then the data are re-read and the weights and flags in the WT table are applied to the output file. APARM(1) > 0 is even more complicated and may require very large amounts of dynamic memory. All data for APARM(1) times are read into memory. Then the rms for each baseline, IF, and Stokes is determined by a robust average over all spectral channels and times in the buffer. These rmses are turned into antenna-based weights for the central time in the buffer and stored in the WT file. Then the first record in the buffer is discarded and replaced with the next time and the process is repeated. APARM(1) must be odd so that the midpooint of the buffer is well determined. Note that the first (APARM(1)+1)/2 times and the last (APARM(1)+1)/2 times in a scan will have the same value so that the number of times entering in each weight will be constant. For example, let APARM(1)=+9. The first 9 times are read into memory and the rmses and antenna-based weights are found and stored in the WT table. These are written to the WT table for times 1 through 5. Then time 1 is discarded and time 10 read. The process repeats and the time and weights for time 6 are written in the WT table. Then time 2 is deleted and time 11 read and so forth. When the end of the scan is reached, the rolling buffer is cleared out much like at the start and a new scan is processed. Scan boundaries are found when the source name changes or when APARM(2) seconds have passed from the first time currently in the rolling buffer. The WT table so written can then be smoothed, clipped, flagged etc and the data are re-read and weighted as above. There are quite a large number of "knobs" that can be changed in this task to give different results. Task ANBPL may be a good way to check the results you obtain. One experienced user recommends the following: APARM = 31 0 300 0 0 0 1.E-7 1000 which uses a 31-sample buffer to obtain robust rmses, does not limit averaging time (usually scan boundaries will be the limit), smooths the rmses over a significant time to dampen out excessive variation using a default Gaussian function, and flags data with excessively small and large weights. BPARM(1) = 20,0 requires a useful number of samples before an rms is computed.