General information on the VLBA, how to propose for observations with the VLBA, how to schedule VLBA observations, and how to reduce VLBA data can be obtained through the VLBA and AIPS homepages. Two extensive tutorials on VLBI data reduction in AIPS have been prepared, complete with data sets, introductory material about AIPS, and detailed instructions. They are simple VLBA project including self-calibration and spectral-line VLBA project plus astrometry. Users new to, or rusty in, VLBI data reduction are encouraged to try these tutorials. Appendix C and Chapter 9 of the AIPS CookBook are also recommended.This web document is intended to provide information on the VLBA data calibration pipeline. This is a set of AIPS procedures found in a single `RUN file' which executes an automated process to provide a best effort a-priori calibration of the data. It includes instrumental calibration as well as some automated flagging of bad data points and atmospheric phase calibration (i.e. "fringe-fitting"). In order to asses the quality of the data calibration, plots and maps are produced but not distributed.
Topics in this document include:
|Pipeline ideology||What is the VLBA data calibration pipeline?|
|Copyrights, etc.||Restrictions on the use of the VLBA data calibration pipeline|
|Current status||Current capabilities and limitations of the VLBA data calibration pipeline|
|Scripts & releases||Updates of the VLBA data calibration pipeline scripts|
|How to pipeline||How to pipeline the data for your project|
|Multiple phase centers||Pipelining secondary phase-stopping center data|
|Last but not least||The acknowledgments section|
The VLBA data calibration pipeline is a 'run file' named VLBARUN in AIPS which is primarily based on the AIPS VLBAUTIL procedures. The VLBAUTIL procedures are described in Appendix C of the AIPS Cookbook. The VLBA data calibration pipeline performs calibration tasks and makes lots of plots and some simple images to find out whether the calibration has worked. The plots may be made available on an html page of the user's choosing to enable easy review. The VLBA data calibration pipeline is not intended to produce high fidelity images. The VLBA data calibration pipeline will be available to AIPS users as part of the AIPS distribution and the Midnight Job.
At the moment the VLBA data calibration pipeline works reasonably well for simple VLBA-only experiments correlated in Socorro, at frequencies between about 1 and 15 GHz, both for continuum and spectral line experiments. It has been successfully run on higher and lower frequencies, but we're not advertising that for general use,
At the moment the VLBA data calibration pipeline works according to the flow chart as shown here in ps or in gif.
The VLBA data calibration pipeline is an effort to streamline and simplify customary calibration of data from the Very Long Baseline Array (VLBA). The VLBA is operated by the National Radio Astronomy Observatory (NRAO) as a facility of the National Science Foundation (NSF) operated under cooperative agreement by Associated Universities, Inc. (AUI).
The Astronomical Imaging Processing System (AIPS) is NRAO's older, but still current, software package for calibration, data analysis, image display, plotting, and a variety of ancillary tasks on (mostly interferometric, radio-) astronomical data. The VLBA data calibration pipeline is an NRAO product (like AIPS itself), comes as an AIPS 'run file' (with an associated 'help file'), and therefore assumes the terms and conditions applicable to using AIPS; this is mostly the GNU General Public License (GNU GPL). AIPS is copyright (c) 1995 through 2019 by AUI.
To make use of the VLBA data calibration pipeline, you must have a relatively current installation of AIPS on your computer (AIPS version 31DEC19 or later), preferably run the AIPS 'Midnight Job' (MNJ), and follow the instructions as provided in the VLBA data calibration pipeline 'help file'. The `MNJ' is a script, usually in your home area, named do_daily.hostname, where hostname is the name of your computer or, perhaps, localhost. If you do not run the MNJ, you may be restricted to using an older version. Please report any bugs you may find after making sure you are using the latest versions (the bug may already have been fixed). Some options described here exist only in the 31DEC20 version of AIPS.
Note that the VLBA data calibration pipeline comes as is, without warranty or guarantee that it works (although we try to make it work)! [That is, we do not assume any responsibility for whatever happens (probably nothing malignant) in case you decide to use the scripts.]
When you have calibrated your data using the VLBA data calibration pipeline, it would be appreciated if you would mention the VLBA pipeline next to an acknowledgment to NRAO in the publication of your scientific results; e.g. "The VLBA data were calibrated using NRAO's "VLBA data calibration pipeline" in AIPS.
The current capabilities of VLBARUN are summarized and then described in detail below. Following that is a list of matters that VLBARUN cannot handle.
|Some general operations that the VLBA data calibration pipeline does or can do:|
|Specific operations performed by VLBARUN:|
|Wish list for future implementation, roughly in order of priority:|
|Of course any other suggestions for the wish list are very welcome as well.|
|The VLBA data calibration pipeline is not supposed to handle, or cannot do:|
VLBARUN received numerous corrections and additions in the 31DEC19 version of AIPS, so we recommend that you obtain this version of the script. The 31DEC19 script might work with older versions of AIPS although certain options (and corrections) in RFLAG will not be present. We do strongly recommend keeping your version of AIPS relatively current by installing the new development release in January and then running the MNJ periodically. Conservative sites might wish to stay a year behind, but start that next year. Some otions described below exist only in the 31DEC20 and later versions of AIPS.
To set up the AIPS environment in your terminal process, source the AIPS LOGIN.[C]SH file found in the AIPS_ROOT directory. Then type $CDTST to find a large number of environment variables including $RUNSYS. If your version of AIPS is up to date, then the RUN file VLBARUN is found in $RUNSYS and its help file in $HLPFIL. You may use these distributed versions simply by invoking them in your AIPS session. If you want to modify them, copy them to some directory of yours and edit away. Then, before starting AIPS, set an environment variable such as MYAREA to point at your disk directory. Then in AIPS enter VERSION = 'MYAREA'. For more information about the use of private disk areas, See section 3.10 (page 3-16) of the AIPS Cookbook.
You may also get your version of VLBARUN from the links below. Put the two files in your disk directory and set the VERSION adverb to use them.
For both the MNJ and newly fetched files, you have to type RUN VLBAUTIL and RUN VLBARUN (in that order and on separate command line) in your AIPS window to make sure you are using the latest 'run files'. If you forget this, either you might end up using an older version, or it will complain about not finding or recognizing the command. The same will happen if you forget to place the 'help file' or 'run files' in the appropriate directories (or misplace them).
After defining the procedure in AIPS, read the 'help file' for its use (type EXPLAIN VLBARUN) and type INPUTS VLBARUN to review its current parameter settings. Invoke the VLBA data calibration pipeline by simply typing VLBARUN; it will check the inputs for possible problems, and if none are found, it will proceed with the operations listed above in that order. Do not forget to check the plots and images it has made, as they may point to specific problems with your data set (dead IF channels, RFI, failing fringe-fitting, etc). If something is definitely wrong, usually the problems are solved by flagging the bad data (in flag/FG table one!) and rerunning the VLBA data calibration pipeline. Other common problems are that you selected a time range for your bandpass calibrator in which it has too few visibilities (e.g. when it is late on source), in which the source is below the horizon for one or more antennas, or the time range does not correspond to the scan on your bandpass calibrator - change either or both. For other problems contact designated AIP. The table below allows you to download the procedures and help file. To view the help file in your browser with links to other help files, click on VLBARUN.HLP
|Current VLBA pipeline run file script:||VLBARUN.001||Put this in your $MYAREA directory|
|Current VLBA pipeline help file text:||VLBARUN.HLP||Put this in your $MYAREA directory|
|Current VLBAUTIL procedures:||VLBAUTIL.001||You may need this too (in $MYAREA)|
A sample html file from a run at L band is shown in file 19OCT291452/vlbarun.html Note that the html file, message file, and plots are placed in a subdirectory labeled with the date and local time. Note also that the results are generally okay but the data from SC really need closer scrutiny. There are two calibration sources and one target which is not detected in the data from individual days. Note that the full message file from the run is available, on a separate page in your browser, by clicking on the link at the top of the vlbarun.html file.
The inputs to VLBARUN:
|-- Load the data from disk --|
|DATAIN||Disk file name|
|DOUVCOMP||Compress data? Use -1, 0 or 1|
|OUTNAME||File name (name)|
|OUTDISK||Working disk with ample space|
|-- OR existing file --|
|INNAME||Input file name|
|INCLASS||Input file class|
|INSEQ||Input file sequence number|
|INDISK||Disk number for input file|
|OPTYPE||For CONT, PSEU or SPEC (LINE)|
|CLINT||CL table interval in minutes.|
|CHREFANT||Reference antenna NAME|
|TIMERANG||Time range of a good bandpass scan, or use|
|scan# in TIMERANG(1)+rest=0|
|INVERS||PC table to use|
|-1 => don't use Pulse cals, do manual phase cal|
|CALSOUR||List ALL Fringe fit & CAL sour|
|FIRST is bandpass calib|
|SOURCES||ONLY FOR PHASE REFERENCING:|
|Source *PAIRS* to calibrate, START each pair with phase referencing calibrator|
|If 2nd='*': all non-CALSOUR are phase-referenced to 1st|
|INFILE||Set to DELZN correction file|
|TECRFILE||Local copy of TEC file; '' => download automatically|
|EOPSFILE||Local copy of EOP file; '' => download automatically|
|SOLINT||Time interval for fringe-fit|
|IMSIZE||Image size for target SOURCES; =-1 for no images|
|FACTOR||CALSOUR IMSIZE (FACTORx128)|
|DOCLIP||> 0 -> use CLIP on all sources: gives clip levels|
|DOFLAG||> 0 -> use RFLAG on all sources - clip level is|
|max (100, DOFLAG) if a clip limit > 100 Jy is desired|
|TYSMOLEV||> 0 -> clip level in TYSMO|
|ELEVLIM||> 0 -> flag elevation < ELEVLIM degrees|
|INTEXT||Input flag commands|
|SELFCAL||> 0 -> do self-cal with IMAGR and FRING on CALSOUR|
|SMRATES||(1) > 0 do SNSMO on rates with SMRATES(1) boxcar (hours)|
|(2) MWF for clipping rates of width SMRATES(2) (hours)|
|(3) flag rates > SMRATES(3) (mHz) from median|
|this edit and smooth on rates is done after the long FRING operations|
|Hint: when setting directory and e-mail, lower-case letters|
|can be retained by NOT using the close quote.|
|DOPLOT||<= 0 no plots; =1 some plots; >1 huge number of plots|
|OUTFILE||Directory for html and plot files: if you want an html file with|
|plots please set the dir where the html and plot files can be put.|
|OUTTEXT||E-MAIL ADDRESS: if you want an e-mail when the script is complete.|
|BADDISK||Disks to avoid for scratch|
The DiFX correlator is capable of correlating a single antenna pointing position at numerous phase-stopping positions within the beam. These data are written into one "primary" file containing the target's central phase-stopping position plus all the calibration data which are taken with only one phase stopping position. "Secondary" FITS-IDI files are written for each of the additional phase-stopping positions. A pipeline RUN file called VLBAMPHC has been developed in 31DEC19 to use the output of VLBARUN on the primary file to assist in calibrating the secondary files.
Many of the inputs to VLBAMPHC are very similar to those of VLBARUN. The main difference is that one specifies the primary data set (assumed to be on disk) in the IN2NAME et al. adverbs. The steps done by VLBAMPHC are then:
|Specific operations performed by VLBAMPHC:|
VLBAMPHC run and help files are distributed with AIPS and updated by the MNJ. The help file may be viewed in your browser with links to other AIPS symbols by clicking on VLBAMPHC.HLP. Copies of the run and help files are also available for download from the following links.
|Current VLBA secondary pipeline run file script:||VLBAMPHC.001||Put this in your $MYAREA directory|
|Current VLBA pipeline secondary help file text:||VLBAMPHC.HLP||Put this in your $MYAREA directory|
Again, when you have calibrated your data using the VLBA data calibration pipeline, it would be appreciated if you would mention the VLBA pipeline next to an acknowledgment to NRAO in the publication of your scientific results; e.g. "The VLBA data were calibrated using NRAO's VLBA data calibration pipeline in AIPS." Thanks!
We on our side acknowledge the efforts of the Cast of AIPS, in particular Eric Greisen, for making and maintaining the AIPS-tasks and environment needed to reduce VLBI data in AIPS over the years and Bill Cotton for writing many of the tasks used on VLBI data. Chris Flatters started collecting the tasks into the VLBAUTIL procedures. This project was continued and completed by Amy Mioduszewski. Lorant Sjouwerman then developed the VLBAPIPE pipeline script making heavy use of the VLBAUTIL procedures. Amy Mioduszewski then developed VLBARUN using VLBAPIPE as a starting point. Most recently Eric Greisen cleaned up minor problems in VLBARUN and added a variety of options. He also developed the secondary pipeline VLBAMPHC
Comments, suggestions, and bug reports should be sent to the designated AIP.
Pipeline ideology / Copyrights, etc. / Current status / Scripts & releases / How to use / Multiple phase stopping centers / Acknowledgments
VLBA Home Page
The VLBA data calibration pipeline processing webpage originated June 5th, 2002 by Loránt Sjouwerman.
October 2019 version written and maintained by Eric Greisen firstname.lastname@example.org.