; UVFLG ;--------------------------------------------------------------- ;! Flags UV-data ;# TASK UV CALIBRATION EDITING ;----------------------------------------------------------------------- ;; Copyright (C) 1995-1996, 1998, 2000-2001, 2003-2004, 2006-2009 ;; Associated Universities, Inc. Washington DC, USA. ;; ;; This program is free software; you can redistribute it and/or ;; modify it under the terms of the GNU General Public License as ;; published by the Free Software Foundation; either version 2 of ;; the License, or (at your option) any later version. ;; ;; This program is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; ;; You should have received a copy of the GNU General Public ;; License along with this program; if not, write to the Free ;; Software Foundation, Inc., 675 Massachusetts Ave, Cambridge, ;; MA 02139, USA. ;; ;; Correspondence concerning AIPS should be addressed as follows: ;; Internet email: aipsmail@nrao.edu. ;; Postal address: AIPS Project Office ;; National Radio Astronomy Observatory ;; 520 Edgemont Road ;; Charlottesville, VA 22903-2475 USA ;----------------------------------------------------------------------- UVFLG LLLLLLLLLLLLUUUUUUUUUUUU CCCCCCCCCCCCCCCCCCCCCCCCCCCCC UVFLG: Task to Flag selected UV data on disk. INNAME UV data (name). INCLASS UV data (class). INSEQ 0.0 9999.0 UV data (seq. #). 0 => high INDISK 0.0 9.0 Disk unit #. 0 => any INTEXT Text file name. SOURCES Sources selected/deselected SUBARRAY Subarray number 0=>all. Elevation is flagged: 0=>1 SELBAND Bandwidth to flag (kHz) SELFREQ Frequency to flag (MHz) FREQID Freq. ID to flag. -1=>all. 0=> first Freq. ID. TIMERANG Time: start day,hr,min,sec stop day,hr,min,sec. BCHAN -1.0 9999.0 First frequency channel ECHAN -1.0 9999.0 Last channel 0=>all BIF -1.0 9999.0 First IF 0=>1 EIF -1.0 9999.0 Last IF 0=>all ANTENNAS Antennas BASELINE Baselines with ANTENNAS STOKES Stokes' parameters OUTFGVER Flag table version number Used w single-source too APARM 1:flag elevations>APAR(1) deg 2:flag elevations1 0 => do not flag due to this reason 4:>0 => Print source, time, baseline, U,V of expected zero fringe rate (U=0) 0 => do not print 5: > 0 flag for shadowing 6: flag for cross-talk (see help) OPCODE 'FLAG','UFLG', 'REAS', 'WILD' REASON Reason (24 char.) DOHIST -3.0 1.0 > 0 -> add to history ---------------------------------------------------------------- UVFLG Type: Task Use: Data can be flagged or unflagged by modifying a flag (FG) table. This is done for both multi- and single-source data sets. If many flagging criteria are to be supplied at once, they can be entered in a text file which is then read by UVFLG. There are three ways to remove one or more flags from the FG table. OPCODE='REAS' compares only the reason given in the FG table with adverb REASON. If they match, then the flag record is deleted. OPCODE = 'WILD' is like REAS except that REASON may be typed with wild-card characters and full wild-card matching of the flag reason will be done. OPCODE='UFLG' uses all adverbs including REASON (as does 'FLAG') but UFLG allows for special values of adverbs which say that a record does not have to match that parameter in order to be deleted. For example, if BIF=-1, any value of the start IF in a flag record will be deleted if the other adverbs match. If BIF is anything else, the start IF in the flag record must match the value of BIF in order for the flag record to be deleted. UFLG is rather demanding. You may also use task TAFLG. Set OPCODE='>', APARM=0; BPARM=1,0; and then set BCOUNT and ECOUNT to the range of FG table row numbers you wish to flag. See EXPLAIN for more examples of UFLG. Adverbs: USERID.....User number. 0 => current, 32000 => any user. INNAME.....UV file name (name). Standard defaults. INCLASS....UV file name (class). Standard defaults. INSEQ......UV file name (seq. #). 0 => highest. INDISK.....Disk unit #. 0 => any. INTEXT.....Name of the text file containing flagging instructions. Adverb values are entered in a text file with descriptions of data to be flagged separated by a '/'. There can be up to 40000 selection criteria in the edit file. See EXPLAIN UVFLG. If INTEXT is used, then adverbs such as TIMERANG, BIF, EIF.. are ignored. Only flagging by elevation and pulse cal works together with INTEXT. SOURCES....List of sources in a raw data file to be selected. If any names begins with a '-' all sources listed are deselected (i.e. all others are). Special rule for UFLG: SOURCES = '-', ... means ignore source name when deciding to unflag. SUBARRAY...The subarray desired for the operation. 0=> all. 0 => 1 for PC table flagging. SELBAND....Bandwidth of data to be flagged. If more than one IF is present SELBAND is the width of the first IF required. Units = kHz, 0=> all SELFREQ....Frequency of data to be flagged. If more than one IF is present SELFREQ is the frequency of the first IF required. Units = MHz, 0=> all FREQID.....Frequency identifier to flag (you may determine which is applicable from the OPTYPE='SCAN' listing produced by LISTR. If either SELBAND or SELFREQ are set their values overide that of FREQID. However setting SELBAND and SELFREQ may occasionally result in an ambiguity, in which case the task will request that you use FREQID. 0=> program will select the first in the FQ table, -1=> flag all FREQID values. Note this useage is slightly different from that in other tasks. -1 => ignore FREQID on UFLG TIMERANG...The specified time range. 1 = Start IAT day (day 0 = first day in data base) 2 = Start IAT hour 3 = Start IAT minute 4 = Start IAT second 5 = Stop IAT day (day 0 = first day in data base) 6 = Stop IAT hour 7 = Stop IAT minute 8 = Stop IAT second Stop IAT = 0 => Stop IAT = Start IAT Both = 0 => all times stop IAT = start IAT not 0, flag +/- 0.5 sec of the stated time Times are NO LONGER altered from the user input except as stated above. Remember that some displays of time are integerized, i.e., a displayed 17 could be anywhere from 16.5 to 17.5. You must now supply any desired rounding down (Start IAT) and up (Stop IAT) if you enter both times. Start IAT < 0 => ignore Start IAT on UFLG, unchanged on FLAG Stop IAT < 0 => ignore Stop IAT on UFLG, can be trouble on FLAG BCHAN......The first frequency channel selected. -1 => ignore BCHAN on UFLG, 0 -> 1 ECHAN......The last frequency channel selected. -1 => ignore ECHAN on UFLG, <=0 -> 0 on FLAG (means max) BIF........The first IF selected; -1 => ignore on UFLG, 0 -> 1 EIF........The last IF selected; -1 => ignore on UFLG, 0 -> max ANTENNAS...A list of the antennas to be flagged. If any number is negative then all antennas listed are NOT to be flagged and all others are. Applies to elevation flagging as a list of antennas to be flagged. All 0 means do not care on UFLG - it will UFLG flags with specific antennas expressed in the flag. BASELINE...If any elements of BASELINE are non zero then all baselines between antennas named in ANTENNAS and those named in BASELINE are specified. If ANTENNAS contains a negative value then all baselines NOT specified are selected. See EXPLAIN UVFLG for examples. Does not apply when doing elevation flagging. STOKES.....Specifies the Stokes types to be flagged Recognized values are 'I', 'Q', 'U', 'V', 'IQU', 'IQUV', 'IV', 'RR', 'LL', 'LR', 'RL', 'HALF'(=RR,LL), 'FULL' (=RR,LL,RL,LR) 'CROS'(=RL,LR), ' '=>all stokes' types. Alternatively, the "mask" to be applied can be recognised e.g. "1001" for flag RR and LR. On UFLG, STOKES = '----' means do not pay attention to the Stokes-level flagging pattern. OUTFGVER...Specifies the version of the flagging table into which the flags are written (0 => highest, or 1 if no FG table exists) for all data sets. APARM......Flagging may be done on one of elevation, shadowing, pulse-cal issues, or the data specified in the other adverbs: Single and multisource flagging (unflagging) on elevation is possible using APARM(1) and APARM(2). These specify the range in elevation in degrees over which data are to be flagged (unflagged). APARM(1) >= APARM(2) => no flagging on elevation. If APARM(1) = 0 and APARM(2) > 0, APARM(1) is set to -90. Adverbs BCHAN, ECHAN, BIF, EIF, and BASELINE do not apply. Normal values for APARM would be something like APARM = -90,10. If elevation flagging is not requested, you may choose to flag data based on shadowing of one antenna by another in the array. APARM(5) gives the minimum baseline in meters before shadowing is thought to occur (e.g. 25 for the VLA). If APARM(5) > 0, you may also set APARM(6) to the minimum baseline in meters to delete the single baseline which may cause shadowuing if it shortens more but not the other baselines to the (pre)-shadowed antenna. This would be meant to omit possible cross-talk when one antenna starts to see another in its near-in sidelobes. Spurious fringes appear due to pulse cal injection, if fringe rate close to zero. These spurious fringes are self averaged at fringe rate stopping procedure if fringe rate is not close to zero. Time interval required to suppress the spurious fringes is calculated by Kogan at VLBA Test Memo No 58, 1998. This time interval depends on suppression factor (APARM(3)) The greater the suppression factor the longer the flagging time interval will be. The desired suppression factor is determined by APARM(3)>1. The value of the suppression factor should be selected experimentally considering pcal tones output less than rms of noise. APARM(4)>0 allows to print source, time, baseline, and U,V of expected zero fringe rate (U=0) at the first line for each scan. The following lines (for each scan) are the flagging intervals for each IF. All other flag selections are applied simultaneously. For example, setting a specific time range will cause only data below the elevation cutoff in that time range to be flagged (or unflagged). Elevation flagging can be combined to similar effect with antenna, baseline, Stokes, IF and channel selection. If all APARM=0 the flagging(unflagging) does not depend on elevation or shadowing, i.e. all data satisfying the other conditions are flagged or unflagged. OPCODE.....Tells whether to flag or unflag the data. 'FLAG' => flag the data, 'UFLG' => unflag with adverbs including REASON Note that the adverbs must match the flag or be set to special "ignore this adverb" values. Defaults are not applied for UFLG. 'REAS' => unflag all that match REASON ignoring all other adverbs 'WILD' => unflag all that match REASON ignoring all other adverbs and interpreting REASON as containing wild-card characters other => 'FLAG' REASON.....Reason for flagging the data, up to 24 characters. ' ' on 'UFLG' means any reason; ' ' not allowed on 'REAS' If APARM(1) and APARM(2) are set and REASON is left blank then 'ELEVATION RANGE EXCLUDED' is used as a REASON If APARM(3) is set and REASON is left blank then 'small fringe rate' is used as a REASON On OPCODE 'WILD' this adverb is handled under special "wildcard" rules. These rules determine which test strings (e.g. reasons in the FG table) match the user-specified string. Except for '*', '?', and trailing blanks (when an * is present), all characters specified in REASON must exactly match the corresponding characters in the test string. A '?' in REASON matches any single character in the test string. An '*' in REASON matches 0 or more characters in the test string. Thus, REASON = '*' matches any test string. reason = 'A?B*D' matches 'ASBD', 'A BXYZD D', and 'A3B45678901D', but does not match 'ABCD' or 'ASBCDE'. DOHIST.....True means add the individual flags in the history file, else just record the use of UVFLG. (True is > 0) ---------------------------------------------------------------- UVFLG: Task to Flag selected UV data on disk. Documentor: E.B.Fomalont Related Programs: CLIP, PRTUV, UVPLT, UVCOP, PRTAB PURPOSE UVFLG is a general-purpose editing task. The user specifies a time range, an antenna or correlator and a polarization channel etc. Entries are made in the flagging ('FG') table. The contents of the FG file can be examined with AIPS task PRTAB and plotted with FGPLT. As of AIPS version 15JUL95 UVFLG will accept two different formats of the ascii files. The new format described below was introduced to allow the ascii flagging files generated by the VLBA monitor system to be read into AIPS FG tables with no user editing being required. Previously users had to modify the day numbers to be 0-relative and change the antenna names to their equivalent numbers read from the AN table. UVFLG can read both the new and the old format files in a manner transparent to the users. The time syntax in AIPS is different than that in the DEC-10. Time zero is defined as midnight of the first day in the data base and four numbers are needed for the start time and end time of any edit. These times are inserted in TIMERANG. If both the Start and Stop IAT times are zero, then all data in the file are flagged. All data within 0.5 seconds of the time range specified will be flagged if the two times are equal (or the stop time given is zero while the start time is not 0). TIMES ARE NO LONGER ADJUSTED BY UVFLG. YOU MUST PROVIDE ANY ROUNDING DOWN AND UP TO ACCOUNT FOR THE ACCURACY OF THE DISPLAYS FROM WHICH YOU DETERMINED THE TIMES TO BE FLAGGED. Remember that the numbers entered into AIPS are floating point, so you can enter, particularly seconds, in decimal fraction form. INTEXT If more than one set of data are to be flagged a text file can be read. If INTEXT is used, then adverbs such as TIMERANG, BIF, EIF.. are ignored. Only flagging by elevation and pulse cal works together with INTEXT. Examples: INTEXT='disk$res:[username]flag.dat' (VMS) INTEXT='myarea:flag.dat' (Unix) where MYAREA is an environment variable set before starting AIPS: %setenv MYAREA /mnt/username This table will be read using the Caltech KEYIN routine and is free format; a "!" denotes that the rest of the line is a comment, a " /" denotes the end of the entry. Adverbs are entered with the same name as in AIPS but with no minimum match. The '=' sign is optional and all expressions must be enclosed in parentheses '(' and ')'. Elements of an array MUST be separated by commas. Only one entry should be made on a given line of the editing table although a given entry may span several lines. Adverb values are initialized to their null values before each entry in the table is read. Up to 40000 editing entries may be made for single source files; there is no limit for multi-source files. Note however, that if multiple baselines are specified, each will count as one of the 40000 entries. The adverbs which may be entered in this table are: SOURCES, SUBARRAY, TIMERANG, BCHAN, ECHAN, BIF, EIF, ANTENNAS, BASELINE, STOKES, OPCODE, REASON, and FREQID. In addition, a parameter TIMEOFF can be given which is the number of seconds to add to the timerange to convert it to IAT times. The current value to correct UTC to IAT is about +24.0 sec. In addition, a parameter DTIMRANG can be given which is the number of seconds to increase TIMERANG from each side i.e to subtract DTIMRANG from the left border of TIMERANG and add DTIMRANG to the right border of TIMERANG. NEW FORMAT: UVFLG will now also accept adverbs ANT_NAME and BAS_NAME in place of ANTENNAS and BASELINE. The original adverbs are still valid though. In the following example entry in a RUN table all data on day 1 between 01:00:00 and 01:05:00 IAT involving antenna 7 and FREQID 3 is to be flagged: ! Flag Antenna 7 1/1:0:0 to 1/1:5:0; all IFs and frequencies, ! any vis. number in a single source data file: TIMERANG=1,1,0,0, 1,1,5,0 ANTENNAS=7 FREQID=3 / NEW FORMAT: in the new format this entry might be typed as: TIMERANG=242,1,0,0, 242,1,5,0 ANT_NAME='KP' FREQID=3 / where KP is antenna 7 in the data, 242 is the day number of the observations. In the following example, for a multi-source data file, Antenna 10, left circular polarization, is flagged for all times on sources 3CXYZ and 0000-900 because the antenna is the wrong color: ANTENNAS=10 SOURCES='3CXYZ','0000-900' STOKES 'LL' REASON='ANTENNA WRONG COLOR' / Note: commas are not allowed between ADVERBS and there should also be at least one blank before each "/" used to delimit an entry. SOURCES For multi-source data sets a list of sources may be specified. If any of the names listed begin with a "-" then all sources listed are deselected, i.e. the described operation will be performed on all OTHER sources. The "-" sign will be ignored in the consideration of the source name. ANTENNAS, BASELINE Antennas to be flagged are specified in array ANTENNAS. If any antenna number is negative the absolute value will be used and all antennas listed will be deselected, i.e. the descriped operation will be done on all OTHER antennas. NEW FORMAT: The adverb ANT_NAME can replace ANTENNAS in the new format. Following ANT_NAME will be the left-justified name (<= 8 characters) and placed in quotes. If using ANT_NAME there can be no deselection as described above. Only one ANT_NAME can be specified per entry. Baselines are specified with the BASELINE adverb. If BASELINE contains non zero elements then all baselines between all antennas specified in ANTENNAS and those specified in BASELINE are selected. If ANTENNAS contains a negative entry then all the specified operation will be carried out on all baselines OTHER than those explicitly named. NEW FORMAT: The adverb BAS_NAME can replace BASELINE in the new format. Following BAS_NAME will be the left-justified name (<= 8 characters) and placed in quotes. Examples: If ANTENNAS = 1,2,3 and BASELINE = 4,5 then baselines 1-4, 2-4, 3-4, 1-5, 2-5 and 3-5 are selected. If ANTENNAS = -1,2,3 and BASELINE = 4,5 then ALL baselines EXCEPT 1-4, 2-4, 3-4, 1-5, 2-5 and 3-5 are selected. STOKES The Stokes type of the data to be flagged can be specified as on of the following: 'I', 'Q', 'U', 'V', 'IQU', 'IQUV', 'IV', 'RR', 'LL', 'LR', 'RL', 'HALF'(=RR,LL) and 'FULL' (=RR,LL,RL,LR), ' '=>all stokes' types. A given polarization for antenna flagging can be specified by 'RR' or 'LL'. Note that specification of 'RL' or 'LR' will flag 'RL' AND 'RR' or 'LR' AND 'LL' respectively. To flag a particular fringe e.g.'RL', specify this using the appropriate mask - in this case '0010'. The order of the mask bits is RR, LL, RL, LR. OPCODE If OPCODE='FLAG' the specified data is to be flagged either by entries in the flagging table. If OPCODE='UFLG' then the flagging table entries are set to be deselected if they match the specified entry. NOTE change: UFLG now unflags only records that match the adverbs exactly, although many have magic values which implies that they can be ignored. Previously (before July 2008) if there were ANY overlap in the specifications of data to be unflagged and an entry in the flagging table then the entire table entry is flagged (deselected, turned off). Thus, if some range of the specifications in the table entry are to remain flagged, these portions should be reflagged. UFLG is now much more demanding. OPCODEs 'REAS' and 'WILD' (REAS with wild-card conventions) do UFLG examining only the reason recording with the flag. Any that match are deselected. REASON Entries made in a flagging table can be labeled with a reason that the data was flagged. OPCODE='REAS' can then be used to deselect entries in the flagging table with this reason, ignoring all other adverbs, thereby effectively unflagging the data. The entry may be up to 24 characters long. REASON can also be used with OPCODE='UFLG' and the other adverbs to unflag data selectively. Added July 2008: UFLG example Consider the following flag table (an edited PRTAB output from a VLBA flag table): So Su FQ Ant timerange IF Ch flgs Reason 1 1 1 6 0/14:09:58.0 1 1 1111 System idle 0 0/14:10:02.6 4 16 1 1 1 5 0/14:09:58.6 1 1 1111 Subreflector error 0 0/14:10:04.5 4 16 1 1 1 7 0/14:09:58.6 1 1 1111 Antenna off source 0 0/14:10:08.5 4 16 1 1 1 11 0/14:09:58.6 1 1 1111 Subreflector error 0 0/14:10:04.6 4 16 1 1 1 1 0/14:09:58.6 1 1 1111 Subreflector error 0 0/14:10:04.5 4 16 1 1 -1 13 0/14:10:11.7 1 1 1111 MIN ELEVATION 10 DEG 0 0/14:11:25.1 0 0 2 1 1 9 0/14:11:28.6 1 1 1111 Source change in progres 0 0/14:11:36.5 4 16 2 1 1 1 0/14:11:28.6 1 1 1111 Antenna off source 0 0/14:11:37.5 4 16 2 1 1 6 0/14:11:28.6 1 1 1111 Subreflector error 0 0/14:11:34.6 4 16 2 1 1 4 0/14:11:49.2 1 1 1111 Antenna off source 0 0/14:11:54.2 4 16 1 1 1 3 0/14:14:58.6 1 1 1111 Source change in progres 0 0/14:15:05.5 4 16 1 1 1 7 0/14:14:58.6 1 1 1111 Source change in progres 0 0/14:15:06.5 4 16 1 1 1 8 0/14:14:58.6 1 1 1111 Subreflector error 0 0/14:15:04.5 4 16 1 1 1 4 0/14:15:21.2 1 1 1111 Antenna off source 0 0/14:15:27.2 4 16 2 1 1 6 0/14:16:28.6 1 1 1111 Antenna off source 0 0/14:16:38.5 4 16 2 1 1 3 0/14:16:28.6 1 1 1111 Subreflector error 0 0/14:16:34.5 4 16 2 1 1 3 0/14:16:28.6 1 1 1111 Source change in progres 0 0/14:16:35.5 4 16 2 1 1 9 0/14:16:47.5 1 1 1111 Antenna off source 0 0/14:16:53.5 4 16 1 1 1 2 0/14:19:58.3 1 1 1111 Antenna off source 0 0/14:20:04.3 4 16 1 1 1 5 0/14:19:58.6 1 1 1111 Antenna off source 0 0/14:20:09.5 4 16 1 1 1 6 0/14:19:58.6 1 1 1111 Antenna off source 0 0/14:20:09.5 4 16 1 1 1 2 0/14:20:04.3 1 1 1111 Antenna off source 0 0/14:20:28.3 4 16 2 1 1 1 0/14:21:28.6 1 1 1111 Source change in progres 0 0/14:21:35.5 4 16 2 1 1 7 0/14:21:28.6 1 1 1111 Subreflector error 0 0/14:21:35.6 4 16 2 1 1 5 0/14:21:28.6 1 1 1111 Antenna off source 0 0/14:21:38.5 4 16 2 1 -1 9 0/14:21:34.4 1 1 1111 MIN ELEVATION 10 DEG 0 0/14:24:57.7 0 0 1 1 1 11 0/14:24:58.6 1 1 1111 Source change in progres 0 0/14:25:05.5 4 16 1 1 1 5 0/14:24:58.6 1 1 1111 Subreflector error 0 0/14:25:04.5 4 16 1 1 1 6 0/14:24:58.6 1 1 1111 Antenna off source 0 0/14:25:09.5 4 16 1 1 1 9 0/14:25:17.5 1 1 1111 Antenna off source 0 0/14:25:23.5 4 16 2 1 1 5 0/14:26:28.6 1 1 1111 Source change in progres 0 0/14:26:35.5 4 16 2 1 1 6 0/14:26:28.6 1 1 1111 Antenna off source 0 0/14:26:38.5 4 16 2 1 1 8 0/14:26:28.6 1 1 1111 Subreflector error 0 0/14:26:34.5 4 16 2 1 1 11 0/14:28:43.5 1 1 1111 Antenna off source 0 0/14:28:49.5 4 16 4 1 1 3 0/15:26:08.5 1 1 1111 Source change in progres 0 0/15:26:15.5 4 16 4 1 1 8 0/15:26:24.5 1 1 1111 Ellipsoid posn error 0 0/15:26:29.5 4 16 To deselect the Ellipsoid flag(s) without deleting the others: 1. The easiest thing to do is OPCODE = 'WILD'; REASON = 'Ellipsoid* Note the lack of close quote on REASON and lack of any command after it on the line. This command will deselect all antennas that have this reason (and all sources etc etc). 2. To limit the deselection to antenna 8 with this reason OPCODE = 'UFLG'; ANTEN=8,0; BASELI=0 SOURCES = '-',' '; STOKES='----'; SUBARRAY=1; FREQID=1; TIMER=-1; BIF=-1; EIF=-1; BCHAN=1; ECHAN=16 REASON = 'Ellipsoid posn error where REASON, BCHAN, ECHAN, SUBARRAY, FREQID, ANTEN, and BASELINE are given valid values that must match and SOURCE, STOKES, BIF, EIF and TIMERANG are given "I don't care values" and do not have to match. The STOKES, BIF, and EIF could be specified, but the SOURCE would require listing all sources in the data to be specified and the TIMERANGE would be hard to manage without the -1 code.