; VLBAPIPE - Lorant Sjouwerman, NRAO - DEC 04 02 ; Ver 1.0 : released to public in July 2002 ; Ver 2.1 : released in MNJ in November 2002 ; Ver 3.2 : rewrite, released in MNJ in December 2002 ;--------------------------------------------------------------- ;! applies amplitude and phase calibration procs to VLBA data ;# RUN POPS VLBI UTILITY CALIBRATION ;--------------------------------------------------------------- ;; Copyright (C) 2001-2003 ;; Associated Universities, Inc. Washington DC, USA. ;; ;; This program is free software; you can redistribute it and/or ;; modify it under the terms of the GNU General Public License as ;; published by the Free Software Foundation; either version 2 of ;; the License, or (at your option) any later version. ;; ;; This program is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; ;; You should have received a copy of the GNU General Public ;; License along with this program; if not, write to the Free ;; Software Foundation, Inc., 675 Massachusetts Ave, Cambridge, ;; MA 02139, USA. ;; ;; Correspondence concerning AIPS should be addressed as follows: ;; Internet email: aipsmail@nrao.edu. ;; Postal address: AIPS Project Office ;; National Radio Astronomy Observatory ;; 520 Edgemont Road ;; Charlottesville, VA 22903-2475 USA ;--------------------------------------------------------------- VLBAPIPE LLLLLLLLLLLLUUUUUUUUUUUU CCCCCCCCCCCCCCCCCCCCCCCCCCCCC VLBAPIPE: Applies amplitude and phase calibration to VLBA data ! To run this procedure you MUST type RUN VLBAUTIL and RUN VLBAPIPE first to define the procedures in AIPS. This is 'the' input dialog to the VLBA pipeline. There are NO DEFAULTS! Note the special use of almost all variables ! -> READ THE HELP/EXPLAIN FILE FOR 'CLEAN' STARTING CONDITIONS AND SPECIFIC USE OF ALL THE VARIABLES. When you run VLBAPIPE with DOALL > 0 the procedure will ask about loading tapes. You will have to type 0 to confirm that the tape has been loaded into the drive. When it is done with loading data it will run to completion on its own. - WHEN IT GOES, IT GOES !!! - ----------------------------- DOALL Tape unit #, < 1 to skip load -> READ THE HELP/EXPLAIN FILE NCOUNT Number of distribution tapes. APARM Number of files on each tape. DOUVCOMP Compress data? Use -1, 0 or 1 ----------------------------- OUTNAME VLBA program name (eg BZ099A) Restarts use this as INNAME! First 6=outclass cal'd data. OUTDISK Working disk with ample space ----------------------------- OPTYPE For CONT, PSEU or SPEC (LINE) ----------------------------- CLINT CL table interval in minutes. REFANT Reference antenna #, or use 0 SORT and eg. SORT 'PT' if unknown Specify either REFANT or SORT ----------------------------- INFILE 1st IONEX file name for TECOR NFILES Number of TECOR files to use; NFILES < 1: don't run TECOR. ----------------------------- TIMERANG A good band-pass scan, or use scan# in TIMERANG(1)+rest=0. CALSOUR List ALL Fring fit & CAL sour FIRST is bandpass calibrator in TIMERANG specified above. If CALSOUR(2)='*', then will take every source in SUtable Put all strong sources here. SOURCES Source * PAIRS * to calibrate START each pair with Ph-Ref. 2nd='*': all non-calibrators are phase-referenced to 1st. SOURCES ONLY USED FOR PH-REFS ----------------------------- SOLINT Time interval for fringe-fit, BPARM or : SOLINT per frequency-ID. Use either SOLINT /or/ BPARM INTERPOL Interpolation: '2PT', 'SIMP', 'AMBG', or 'CUBE' (in CLCAL) ----------------------------- IMSIZE Image size for 'even' sources FACTOR others are FACTORx128 square IMSIZE ONLY USED FOR SOURCES IMAGING CALSOUR USES FACTOR ----------------------------- DOTV Plot type: >0 then plot on TV =0 make PL file, <0 make PL files and send to printer. Recommend =0, discourage =-1 BADDISK Disks to avoid for scratch VLBAPIPE is defined in the standard AIPS runfile area. ---------------------------------------------------------------- VLBAPIPE Type: Procedure Use: VLBAPIPE is the procedure that applies the VLBA calibration procedures (VLBAUTIL) for a-priori amplitude and phase corrections, and also performs fringe fitting on single strong sources and phase-references weak sources. Simple visibility flagging, bandpass calibration, and ionospheric (total electron content) corrections are applied before fringe fitting and averaging. It may attempt to self-cal the calibrators and will image the targets. ==================================================================== READ THE HELP/EXPLAIN FILE THOROUGHLY BEFORE USING SINCE NONE OF THE AIPS INPUTS ARE USED IN THE USUAL WAY. ==================================================================== NOTE: currently this procedure is intended for simple VLBA-only experiments! More info on http://www.aoc.nrao.edu/vlba/html/vlbahome/observer.html under 'Calibration and Imaging' of NRAO's VLBA web page. $---------------------------------------------------------------------------- Type RUN VLBAUTIL and RUN VLBAPIPE to define the VLBAPIPE procedures; it is wise to type RESTORE 0 (before the 'run' commands above) or DEFAULT (after the 'run' commands, and setting TASK 'VLBAPIPE') to set the default task values. Default values should be MEANINGLESS here. Note that a 'RESTORE 0' after RUN VLBAPIPE deletes all the procedures! The procedure is run by typing : (GO) VLBAPIPE The procedure will check the inputs (as far as it can) to avoid later interruptions, and then proceed directly, if the data is already on disk, or after the last tape has been loaded if the data is not on disk. Some manual operation (loading tapes and typing a zero) is needed. CLEAN STARTING CONDITIONS I prefer to use one project per user number, as otherwise you may get into troubles if different projects use the same source names. And I use a different user number than for my other non-VLBA projects. This means I can type RESTORE 0 without it affecting my tget's and catalogs that I use for my other non-VLBA projects. But this is up to you. It is not recommended to run two pipelines with the same user number simultaneously, nor overlapping at any time during the processes. Clean up all files resulting from previous attempts of running the procedure, except for the original UVDATA and perhaps, when restarting a specific frequency-ID, also not the F(X)POL files. When restarting from disk, make sure the UV data has no extra SN, CL, FG etc tables. Use the procedure P_RESTART (which is defined in VLBAPIPE) to do this. When you start the procedure on a new data set which includes sources that have the same name in a previous run, then RENAME the images from the prevoius run. IMAGR otherwise attempts to overwrite the earlier images with the same source name and dies. If you wish to flag some additional data (because the procedure did not flag all the bad data automatically), make sure you put the flags in FG table number 1. The procedure uses the CALIBRATION TRANSFER FILEs that come with the data on the VLBA data distribution tape. Occasionally a VLA antenna, and/or the Effelsberg or Green Bank telescopes also participated. For now that means that if you want these antennas also to contribute to the calibrated data set, you would want to load all the data with VLBALOAD, and include the external calibration of the non-VLBA antennas in the old-fashioned way with ANTAB. Write the external calibration system temperatures and gain curves in TYVER=1 and GCVER=1 (and BLVER=1). It is a VERY good idea to run TASAV before you do this to keep the original TY, GC, BL and also FG tables number 1 in case of disasters. After you loaded the data with VLBALOAD and put the external calibration and flagging information for the non-VLBA antennas in the appropriate tables number one, start the VLBAPIPEline with DOALL=0 and OUTNAME and OUTDISK pointing to the file VLBALOAD created (thus not INNAME, INDISK). Make sure there is enough disk space available on OUTDISK, about 4 to 5 times the expected compressed UVDATA set. For spectral line probably you need much more because it will make spectral line image cubes. DOALL is supposed to set the starting point for the procedure. A positive number is used to start with loading the data from tape, and using INTAPE=DOALL for the tape drive. A DOALL=0 means, the data is loaded to disk (and has name 'OUTNAME'.UVDATA.1 on 'OUTDISK' -see below), but nothing else has been done to it. DOALL=-1, -2, -3, etc is used to restart at frequency-ID 1, 2, 3, etc. and continue to higher numbers, if they exist. The files 'OUTNAME'.FQ-(i).1 or 'OUTNAME'.FPOL(i).1 (for a single freq. UVDATA or FXPOL) should exist! Restarting at a given point may be fragile and may break the procedure. * testing: If < 1000, then ONLY do Freq-ID = (-1*DOALL)-1000; ie. -1004->4 NCOUNT is the number of distribution tapes to be loaded. The number of files per tape is specified in APARM. APARM is the number of files to be loaded from each distribution tape - files at the beginning of a tape cannot be skipped. For more control use VLBALOAD. Make sure that APARM lists the number of files per tape in the right order ;-), that the unused APARMs are set to zero, and the tapes are loaded into the tape drive in the correct order. DOUVCOMP determines whether all UV data will be compressed (DOUVCOMP=1), not compressed at all (DOUVCOMP=-1), or whether you want the UV data loaded in uncompressed multi-source form with FITLD, and compressed in the calibrated single source files after SPLIT (using DOUVCOMP=0). OUTNAME is used to create files (e.g. from tape), AND to search for as an INNAME for files that are already on disk (which will be OUTDISK). OUTNAME='VLBA-experiment name' (eg BZ99Z) is a good unique name to use. All final calibrated single source UV files will use the first 6 characters of OUTNAME as their OUTCLASS, so for multiple passes make OUTNAME unique in the first six characters. OUTDISK is the disk that is used for ALL files produced by the procedure. It may use other disks for scratch, which can be regulated with the BADDISK parameter below. OUTDISK is the working disk; it needs to be spacey - clean up if needed (before you start vlbapipe). OPTYPE should reflect the type of observation and requested output files: 'CONT' for plain continuum observations using the PC (pulse-cal) table calibration; all data is averaged over IFs and channels and continuum images are made from the averaged data. 'PSEU' for pseudo-continuum observations using the PC table calibration the data is NOT averaged over IFs nor over channels, but the resulting images are continuum images - used for large FOVs 'SPEC' for spectral line observations - there should not be a PC-table! Manual pulse-cal is performed on the bandpass calibrator in the timerange given (see below). Data is unaveraged and images are spectral line cubes of the phase referenced target sources ONLY. (see below in CALSOUR for strong spectral line sources that can be fringed - essentially you have to make your own cube later) Calibrator sources will be made continuum images even in 'SPEC'! 'LINE' is the same as 'SPEC'. ********************************************************************** NOTE that the imaging is only done with the intention to check whether the calibration has worked and whether maybe some more data needs to be flagged (in FG#1). Always re-image your sources for your scientific use (imaging, analysis) - these won't be good enough! ********************************************************************** CLINT is the CL table interval (minutes) used throughout the whole experiment For VLBA data you would generally choose a CLINT between 0.1-2 minutes. REFANT or SORT is used to select the reference antenna. Because the antenna number for REFANT is usually unknown before loading, SORT can be used to supply the two letter code; e.g. SORT='PT' for Pie Town. Use either REFANT or SORT, not both! Set REFANT to zero if you are using SORT; blank SORT if using REFANT. SORT Use either REFANT or SORT, not both; see REFANT above. INFILE The name of the file containing a model for the total electron content (TEC) in the ionosphere which is used by TECOR to correct delays caused by the ionosphere. It is recommended to run TECOR for ALL frequencies up to 15 GHz. The filename should be given in the usual AIPS style as a logical directory name followed by a colon and the name of the file. The entire name will be converted to upper-case. The file should contain TEC maps spanning the time range covered by the input CL table and should be in IONEX format. If NFILES > 1 then the name MUST be in standard format CCCCdddC.yyC where C can be any character, ddd is the day number, and yy is the year. Also the name given in INFILE must be the first day. The IONEX/TECOR files can be downloaded through an anonymous ftp to cddisa.gsfc.nasa.gov and following the directory structure: /gps/products/ionex// (take doy from your *.sum). Also note that if your observation starts before 01:00 UT, then you need the IONEX/TECOR for the preceding day too, and point INFILE to this first file. For observations ending after 23:00 UT, you need the next day's file as well, and in both cases add one to NFILES. TECOR is not run and INFILE is ignored for NFILES < 1, e.g. for 43 GHz data. For more details, see EXPLAIN TECOR. NFILES is the number of IONEX files to use, starting at INFILE. Note that if NFILES > 1 then the INFILE must be in a standard format and the file in INFILE must be the first day of those to be loaded. TIMERANG is a time range on which ALL antennas have good data on the bandpass calibrator (see below, CALSOUR(1)). This interval is used for pulse-cal, either with the PC table, or manually if there is none (as in spectral line data). For multiple frequency data it is VERY nice to have TIMERANGE to span the whole time range in which you flip through all the observed frequencies - otherwise the procedure will choke. Alternatively, if you know the scan number, you can use the scan number in TIMERANGE=66,0 to get the time range associated with scan 66. CALSOUR is the parameter to name all the sources you want to be fringe fitted e.g. your calibrator sources, strong (target), or survey sources. The first source MUST BE named, and will be used for pulse-cal and bandpass calibration. Use a simple source with good signal to noise and with all antennas present (note: phase-reference calibrators may be good too) in the TIMERANGE specified above. If you want all sources fringe fitted then you may use CALSOUR(2)='*' but remember to also specify CALSOUR(1) to get the pulse-cal and bandpass calibration. If you want to attempt a self-cal for stronger sources, CALSOUR(2)='*' is not allowed, so you have to specify your calibrator sources (up to 30). The source names that start with an additional minus will be self-cal'd, but be warned that this is a frequent point of failure which may break the procedure. The sources that are fringe fitted will be imaged in (pseudo) continuum mode, so if your spectral line target is strong (e.g. when looking for absorption) you will only get a continuum image (not UV data set) which is used to asses the calibration - Re-image for the spectral line cube. SOURCES holds the phase-referencing scheme in PAIRS with the phase-reference calibrator in the odd index. Each phase-reference target is in an even index, and calibrated with the preceding phase-reference calibrator. There is a maximum of 15 pairs, a total of 30 source fields. If your are not phase-referencing any source, then leave SOURCES blank. Do not repeat sources that are in CALSOUR that are not actually used in the phase-referencing scheme. If all targets are to be referenced to only one source, use SOURCES = 'PRCAL', '*' to get all non-calibrators phase referenced to 'PRCAL' - sources in CALSOUR are left alone, as is. E.g. SOURCES='PRCAL1','TARG_1','PRCAL2','TARG_2','PRCAL1','TARG_3' will reference target source 'TARG_1' and 'TARG_3' to phase-reference source 'PRCAL1' and target source 'TARG_2' will be phase-referenced to source 'PRCAL2'. All phase reference sources 'PRCAL1', etc MUST also be present in the CALSOUR list, so they will be fringe fitted. SOLINT or BPARM is used to specify the length of the fringe fit intervals. They should be given in minutes. If SOLINT > 0, the whole experiment uses the same fringe fit interval, regardless of frequency-ID. If you set SOLINT=-1 then you may choose different fringe fit intervals for each frequency-ID (e.g. 3 minutes for 5 GHz and 0.5 minutes for 43 GHz) with BPARM. But you need to know the order (usually order of observing) and you need to specify a number (maximum 10 values) for each separate frequency-ID in BPARM. SOLINT is also used in the self-cal of CALSOUR. BPARM Use either SOLINT or BPARM, not both; see SOLINT above. INTERPOL is the interpolation all sources. The options are '2PT', 'SIMP', 'AMBG', or 'CUBE' as defined by CLCAL. ***************************************************************************** The procedure will also image all the sources, but note that the images are produced in order to check the calibration. By no means, are the choices made in the imaging stage represent the best possible imaging that could be performed with this data. Those selections should be made by the individual observer, in pursuit or their own scientific goals. The images should allow you to get an idea about the success of the calibration, and to find data that needs flagging before restarting it. ***************************************************************************** IMSIZE is used to define the output image sizes. You will want a big field for your phase-referenced target sources with a minimum IMSIZE of 512 pixels and up to 8192. A larger image is slower, but accounts for possible position inaccuracies if it is known to less accuracy than 500/Frequency(GHz) mas. Sources listed in CALSOUR are automatically imaged with a size of 128 by 128 unless FACTOR is set. Pixels automatically scale with frequency: 2.25/Frequency(GHz) mas. FACTOR can increase the image size of CALSOURs. It is assumed that your calibrators are well-known, so they do not need to be imaged in more than 128 pixels. However you may want to increase this for complicated sources. If FACTOR is used then the resulting image size for the sources listed in CALSOUR is FACTOR*128. The maximum FACTOR is 64. DOTV is used to direct all the 'instructive' plots that the procedure makes to show you how the calibration is progressing. A DOTV < 0 value, which is highly DISCOURAGED, will divert plots directly to the printer. A positive value for DOTV send the plots to the TV and thus is interactive. This can be slow and the user should be present the entire time the procedure is running. DOTV=0 is recommended, this makes PLot tables. Afterwards all the PLot tables can be cycled through with the procedure P_ALLPLOT, maybe P_ALLPRINT, which are defined in the VLBAPIPE runfile. BADDISK is the well-known parameter specifying the disks to avoid for scratch. $--------------------------------------------------------------------------- VLBAPIPE: Procedure that attempts to calibrate VLBA data blindly. Documentor: Lorant Sjouwerman, lsjouwerman@aoc.nrao.edu. Related Programs: All VLBAUTIL procs, TECOR, BPASS, UVMLN, SPLIT, IMAGR,.. CLIPM, POSSM, VPLOT and many more VLBAPIPE applies VLBAUTIL procedures to simple VLBA-only experiments, with simple flagging and Ionospheric Total Electron Content correction before bandpass calibration. Sources are split after calibration, and strong and phase-reference sources are self-called and target sources are imaged as a first order quality check. ==================================================================== READ THE HELP/EXPLAIN FILE THOROUGHLY BEFORE USING SINCE NONE OF THE AIPS INPUTS ARE USED IN THE USUAL WAY. ==================================================================== ----------------------------------------------------------------------------