AIPS HELP file for VLBARFI in 31DEC22
As of Sat May 21 13:59:32 2022
VLBARFI: Applies amplitude and phase calibration to VLBA data !
To run this procedure you MUST type RUN VLBAUTIL and
RUN VLBARFI first to define the procedures in AIPS.
-- Load the data from disk --
DATAIN Disk file name
DOUVCOMP Compress data? Use -1, 0 or 1
OUTNAME File name (name)
OUTDISK Working disk with ample space
-- OR existing file --
INNAME Input file name
INCLASS Input file class
INSEQ Input file sequence number
INDISK Disk number for input file
CLINT CL table interval in minutes.
CHREFANT Reference antenna NAME
TIMERANG A good bandpass scan, or use
scan# in TIMERANG(1)+rest=0
CALSOUR List ALL Fring fit & CAL sour
FIRST is bandpass calib
TECRFILE Local copy of TEC file
'' => download autmatically
'-' => don;t do this corr.
EOPSFILE Local copy of EOP file
'' => download autmatically
'-' => don;t do this corr.
DOBAND >= 0 -> compute and apply a
DOCLIP not 0 -> use CLIP on all
sources: gives clip levels
before amp cal & BPASS
DOFLAG > 0 -> make RFLAG plots on
all sources - clip level is
max (100, DOFLAG) if a
clip limit > 100 Jy is
desired (e.g. masers)
TYSMOLEV > 0 -> clip level in TYSMO
ELEVLIM > 0 -> flag elevation <
INTEXT Input flag commands
Hint: when setting directory and e-mail lower-case
letters can be retained by NOT using the close quote.
DOPLOT <= 0 no plots;=1 basic plots
= 2 instrumental phase cal
and CL after ACSCL, APCAL
= 3 POSSM plot each cal scan
>3.5 many more plots
OUTFILE DIRECTORY FOR HTML AND PLOT
FILES: if you want an html
file with plots please set
the dir where the html and
plot files can be put.
SYSOUT File name for extra copy of
the VBRFI text file. Full
path must be given.
OUTTEXT E-MAIL ADDRESS: if you want
an e-mail when the script
is complete, set the
BADDISK Disks to avoid for scratch
Q_DEBUG > 0 -> pause before most
tasks and VLBAUTIL procs
showing the inputs and
allowing VLBARFI to stop
Use: VLBARFI is the procedure that uses the VLBA calibration
procedures (VLBAUTIL) to calibrate VLBA data. See the explain
file for a detailed description of the procedure.
NOTE: currently this procedure is intended for simple experiments!
Type RUN VLBAUTIL and RUN VLBARFI to define the VLBARFI
procedures. Do not do this again unless you lose the procedures
somehow. HELP PROCS will show the names of all defined
The procedure is run by typing : VLBARFI
The procedure will check the inputs (as far as it can) to avoid
later interruptions, and then proceed directly, if the data are
already on disk. The procedure will clear the message file when
it starts, so print any messages you need to save. This insures
that the printed message file at the end contains only the
messages from this run.
CLEAN STARTING CONDITIONS
It is best to use one project per user number, as otherwise you
may get into trouble if different projects use the same source
names. t is not recommended to run two pipelines with the same
user number simultaneously, although with different POPS numbers
it should work. Overlapping VLBARFIs at any time during the
processes may also have issues.
Clean up all files resulting from previous attempts of running
the procedure, except for the original UVDATA and perhaps, when
restarting a specific frequency-ID, also not the F(X)POL files.
When restarting from disk, make sure the UV data has no extra
SN, CL, PL files. Note that you may want to keep flag (FG)
tables attached to the input file so that they will be applied
during all stages of the pipeline run. Use the procedure
Q_RESTART (which is defined in VLBARFI) to do this. Note that
Q_RESTART does not delete FG tables. NEVER DELET FG TABLE 1,
the on-line flags.
If you have any antennas that do not have Tsys or Gain curves in
the data, please load them with ANTAB into TY and GC tables #1.
It is a VERY good idea to run TASAV before you do this to keep
the original TY, GC, BL and also FG tables number 1 in case of
Make sure there is enough disk space available on OUTDISK.
DATAIN......48-character name of the disk file from which to read a
FITS file. It must be in the form
where is the remote computer name, is the
environment variable (logical name) for the disk area in
which the file named is stored. is
usually omitted when the file is local to the current
computer. If DATAIN is not found, the task will try
DATAIN with the character 1 appended.
Beginning 2003-Oct-16 FITLD can read more than one disk
file at a time. In that case, they must all have the
same name except that the last letter(s) are the
sequence number 1 through NCOUNT with no leading zeros.
DATAIN must give the base name only, omitting the
sequence number. Then either
DATAIN = 'MYDATA:file_
DATAIN = 'MYDATA:file_1
can access files in $MYDATA named file_1, file_2,
file_3, etc. Note that this is similar to the pieces of
a FITS pieced UV-table data set, but only one UV-table
data set may be read at a time.
DOUVCOMP....If true (> 0.5) then output data will be compressed
which saves disk space at the expense of losing some
OUTNAME.....The name for the output files. The output files will all
be given the class UVDATA and assigned distinct sequence
numbers as needed.
OUTDISK.....The disk drive number for the output data if loading from
disk. Choose a disk with a large amount of space. If
data are already loaded into AIPS (i.e. you are using
INNAME, INCLASS etc) then OUTDISK is ignored and INDISK
INNAME......Input UV file name (name) that is already loaded.
Note that you CANNOT use both DATAIN/OUTNAME/OUTDISK
INCLASS.....Input UV file name (class) that is already loaded.
INSEQ.......Input UV file name (seq. #) that is already loaded.
INDISK......Disk drive # of input UV file. NOTE: this is also
used as OUTDISK.
CLINT.......Calibration table interval in minutes. This should
normally be in the range 0.25 to 1.0. If <=0 then
0.25 is assumed.
CHREFANT....Name of reference antenna. If not set, a sensible
antenna in the southwest will be tried (FD, PT, LA or
TIMERANG....Time range on which ALL antennas have good data on the
bandpass calibrator (see below, CALSOUR(1)). Used for
instrumental phase calibration by manually running
FRING on this scan. If you know the scan number, you
can use the scan number e.g., set TIMERANGE=66,0 to
get the time range associated with scan 66.
If all 0, then VLBARFI will set the TIMERANG to that
of the only scan if the NX table hasonly 1 row. If
there is more than one row, VLBARFI Will try to find a
good scan by running BSCAN on the bandpass calibrator.
CALSOUR.....All the calibrators should be listed here. If none are
listed and the source table has only one row, the
source name in that row will be used. IMPORTANT: the
FIRST source is the source used for instrumental phase and
bandpass calibration. If TIMERANGE was set, it must be a
scan on CALSOUR(1). If you want all sources treated as
calibration sources, you may use CALSOUR(2)='*' but
remember to also specify CALSOUR(1). The sources in
CALSOUR will have RFLAG run separately (DOFLAG>0) and
will have separate POSSM plots made for each scan and
antenna (4 to a page) if DOPLOT>2.5. All sources listed
in CALSOUR must be present in the data set.
TECRFILE....Input file. If ''=> file will be automatically
downloaded. If there is a problem with automatic
downloads (this can be a problem with some firewalls)
the file can be can be downloaded manually. See EXPLAIN
VLBATECR for details on how to download the file(s).
The file should contain TEC maps spanning the time
range covered by the input CL table and should be in
IONEX format. If NFILES > 1 then the name MUST be in
standard format CCCCdddC.yyC where C can be any
character, ddd is the day number, and yy is the year.
Also the name given in INFILE must be the first day.
If TECRFILE='-', skip this correction which is also
skipped for frequencies above 12 GHz.
EOPSFILE....Input file. If ''=> file will be automatically
downloaded. If there is a problem with automatic
downloads (this can be a problem with some firewalls)
the file can be can be picked up from:
If EOPSFILE='-', skip this correction because the data
are too new for the USNO file.
DOBAND......> or = 0 - run BPASS and apply the resulting BP table.
< 0 -> no bandpass function (when really noisy)
DOCLIP......If DOCLIP(1) not 0, clip parallel hand data at
ABS(DOCLIP(1)) Jy and cross-hand data at ABS(DOCLIP(2))
Jy. If DOCLIP(2)=0 no cross-hand clipping is done.
Parallel hand data are flagged when cross-hand data are
and vice versa. CLIP is run before amplitude calibration,
just before BPASS is run.
DOFLAG......> 0 says to run RFLAG once for each CALSOUR, the first
time to plot and set the flagging levels. No flag table
is written. If you are observing a source with strong
lines, the value of the pre-clip in Jy (FPARM(13)) is set
to max (100, DOFLAG). The plot files will appear in
glorious color in the output html page.
TYSMOLEV....> 0 => run TYSMO on the TY table. The input TY values are
clipped between below TYSMOLEV/20 and above TYSMOLEV.
Then the data are median window filtered over a 10 minute
interval and values > TYSMOLEV/10 from the median are
flagged. Then a 15-minute box filter is used to replace
all flagged values with a smoothed value from a nearby
time. Do NOT do this unless you think the TY table needs
regularizing (and many do).
ELEVLIM.....> 0 => Flag all data below ELEVLIM degrees
INTEXT......If not blank, text file with flagging commands to be
read by UVFLG.
Adverb values are entered in a text file with
descriptions of data to be flagged separated by a '/'.
There can be up to 40000 selection criteria in the edit
file. See EXPLAIN UVFLG.
DOPLOT......Make diagnostic plots to judge quality of procedure
<=0 => no plots
> 0 and < 0.5 => some plots but convert is known
not to work
1 => Plot BPASS solutions, RFLAG and VBRFI plots
2 => Plot Intrumental phase cal, CL table after ACSCL
3 => POSSM for each scan and antenna after calibration
for each CALSOUR.
4 => Also SN and CL table after ACCOR and after ACSCL
OUTFILE.....Directory of html and plots files. If you want the
diagnostic plots written to disk and an html file
created to make the plots easier to examine please set
directory here. This is limited to 35 characters
(including final /). Environmental variables are
allowed and must end with a ':'. Mixed environmental
and regular are not allowed, e.g.,
A subdirectory will actually be used so this directory
may contain previous subdirectories. It should not
contain any PostScript files. A text file containing
all of the messages will be saved in the subdirectory
with the name messages.txt.
SYSOUT......Full path name of an extra copy of the text file written
by VBRFI. E.G., SYSOUT = '$MYVBRFI/21MAY19/RepC.txt_7
Note use of environment variable, subdirectory for date,
and ID of band and subband number. The intent of this
option is to provide an archival copy.
OUTTEXT.....E-mail address for notifications when script is done or
BADDISK.....A list of disk numbers to be avoided when creating scratch
Q_DEBUG.....> 0 -> pause before most tasks and VLBAUTIL procs showing
the inputs and allowing VLBARFI to stop. Simply hit
carriage return (enter) to continue, type 0 to stop
VLBARFI: Procedure that attempts to calibrate VLBI RFI data runs
automatically including output of statistical measures.
Documentor: Eric Greisen
Related Programs: All VLBAUTIL procs and many more.
VLBARFI is based on VLBARUN written by Amy Mioduszewski, NRAO.
VLBARFI applies VLBAUTIL procedures to VLBA RFI measuring data sets.
It calibrates the data sufficient to do statistics on the
autocorrelation data. Those statistics are written to a data area
designed for same.
VLBARFI attempts to make sensible choices if inputs are left to
default. The only inputs that must be set are:
DATAIN or INNAME (INCLASS, INSEQ and INDISK) -> so the dataset is
CALSOUR(1) -> so VLBARFI knows what calibrator to use as the
bandpass/instrumental phase calibrator
The observation is treated as spectral line and some calibrations
done by VLBARUN are omitted as not affecting the autocorrelation
More detail about defaults
VLBARFI tries to pick sensible defaults if inputs are left blank. Below
describes how it makes these decisions.
CLINT=0 -> set to 0.25 minutes
CHREFANT='' -> Chooses the first antenna present from the southwest VLBA
antennas in the order of FD, PT, LA and KP.
TIMERANGE=0 -> Runs the task BSCAN on CALSOUR(1) to select scan with the
highest signal to noise with all the antennas present. If
INVERS>=0 then BSCAN also checks if there are PC table
entries for that scan.
VLBARFI offers the option to create diagnostic plots. If DOPLOT is set
to 0 no plots are made. For 0<DOPLOT<1.5 (i.e. DOPLOT=1) the following plots
-POSSM plot of bandpass solution
-SNPLT of amplitudes in CL table after ACSCL
-SNPLT of amplitudes in CL table after APCAL
-RFLAG of rms spectrum of time and spectral deviations
-VBRFI of mean and rms/mean for each antenna
For DOPLOT>1.5, IN ADDITION to the above plots the following are produced
-POSSM of calibrated autocorrelations, each antenna and scan
For DOPLOT>2.5, IN ADDITION to the above plots the following are produced
-SNPLT of amplitudes in SN table after ACSCL
-SNPLT of amplitudes in SN table after ACCOR
-SNPLT of amplitudes in CL table after ACCOR
-SNPLT of amplitudes in SN table after APCAL
Output of plots to disk, creation of html file and e-mail message
VLBARFI will create .gif files on disk in a subdirectory of the
directory defined by OUTFILE of the plots that are requested. It then
makes a vlbarfi.html file that includes headings for the plots and at
the bottom a list of the CL tables and what calibration is added to
each. This vlbarfi.html file is intended to make examination of the
diagnostic plots easier, so the user can quickly judge whether the
pipeline worked. A link to the message file from the run is also
provided in the vlbarfi.html file.
Note that .gif files are not made if 0.0 < DOPLOT < 0.5 since "convert"
(used to convert PostScript to gif) is said not to work. This seems
common on Macs unfortunately. The PostScript files from basic
calibration are kept in the output subdirectory, but not those POSSM
on the calibrated autocorrelation scans.
It is STRONGLY recommended that the directory defined by OUTFILE be
empty of PostScript files although it may contain date-stamped
subdirectories. VLBARFI creates and deletes PostScript files in this
directory so it is safer if there aren't any such files for VLBARFI to
potentially clobber or delete. VLBARFI makes a subdirectory in
OUTFILE named by the date and time and the final gif files and
vlbarfi.html will reside in the subdirectory. VLBARFI checks if there
is an existing vlbarfi.html file and will not run if there is. That
should happen only if you restart VLBARFI within the same minute.
It is also recommended that if OUTFILE is used then an e-mail address
be provided in OUTTEXT. The e-mail will state whether VLBARFI is done
(subject: VLBARFI DONE) or failed (subject: VLBARFI FAILED). The
FAILED message only catches failure due to a problem with how VLBARFI
was set up, if the pipeline fails because a task fails that is not
captured and no e-mail message is sent. If OUTFILE is set, the DONE
message includes the URI to the vlbarfi.html file assuming that the
machine you are viewing it on is the "localhost". So if you are on
that machine, you can copy that URI ("file:///etc") into your web
browser and look at the plots. If you are not on the machine where
the plots were created then just copying the file:///... URI will not
CONSIDERATIONS FOR MAC OS X AND OTHER OS THAT MIGHT NOT INCLUDE COMMON
LINUX COMMAND-LINE TOOLS -- the output of plots to display using .gif
and html files requires the command-line command "convert" or "magic".
If you do not have "convert" or "magic" then you cannot use the
OUTFILE input. There are packages you can install on your machine
that will give you access to linux command-line tools like convert.
What does VLBARFI do?
Below are the basic calibration steps that VLBARFI performs. This does not
include all the plotting etc..
0) checks that the inputs are correct as well as it can, sets some
defaults. Prints the INPUTS and runs LISTR with OPTYPE='SCAN' and
places them in the output messages file.
1) run VLBALOAD - if DATAIN is specified, loads the data from a FITS
or FITS-IDI disk file. Merges duplicate table files
Runs VLBAFIX to sort the data if needed and make an
index table. If there are multiple frequency IDs,
VLBARFI stops and tells the user to separate them into
separate data sets.
2) run VLBATECR - corrects ionosphere if the observing frequency is
less than 12 GHz. This procedure uses Global
Positioning System (GPS) models of the electron
content in the ionosphere to correct the dispersive
delays caused by the ionosphere. This correction
is particularly important for phase referencing
experiments at low frequency.
3) run VLBAEOPS - corrects Earth Orientation Parameters (EOP) used in
correlation. VLBI correlators must use
measurements of the EOPs while correlating. These
change slowly with time and therefore the EOPs used
by the correlator must be continually updated.
Even though the EOPs that a correlator uses are the
the most accurate at the time of correlation,
frequently after the correlation the EOPs are
improved. This procedure inserts a correction for
the difference between the EOPs used in correlation
and the current best calculation of the EOPs. This
is particularly important for phase referencing.
4) run UVFLG - If INTEXT is not blank, it is used to add flag
commands to FG table 1.
5) run TYSMO - If TYSMOLEV > 0, apply TYSMO to the TY table making
a corrected second version. TYSMO begins by
flagging all Tsys values less than TYSMOLEV/20 and
greater than TYSMOLEV. It then does a median window
filter with a width of 10 minutes and a clip level
of TYSMOLEV/10. Finally a boxcar of width 15 minutes
computes a smoothed version and all clipped values
are replaced with smoothed values.
6) run UVFLG - If ELEVLIM > 0, run UVFLG to flag all data for which
one of the antennas has a source elevation less than
7) run VLBACCOR - sampler corrections. This procedure corrects the
amplitudes in cross- correlation spectra due to
errors in sampler thresholds using measurements of
8) run VLBAPCOR or VLBAMPCL - correct instrumental delays if there is
more than one IF. These delays or "instrumental
single-band delays" are caused by the passage of
the signal through the electronics of the baseband
converters and can be corrected with either the
pulse cals (VLBAPCOR) or by running FRING on one
9) run CLIP - OPTIONAL: If DOPCLIP(1) < 0, CLIP is run on the data
BEFORE any amplitude calibration using ABS(DOCLIP(1))
as the parallel-hand upper limit.
10) run VLBABPSS - Runs BPASS to calibrate the bandpass response
functions for each antenna.
11) run VLBAAMP - Runs ACSCL which applies the BP table and previous
calibration tables and corrects the
cross-correlation spectra using auto- correlations.
For more information see VLBA Scientific Memo #37
(Craig Walker). Then preforms a-priori amplitude
gain calibration using the gain curves and system
12) run VLBAPANG - correct paralactic angle. The RCP and LCP feeds on
alt-az antennas will rotate in position angle with
respect to the source during the course of the
observation (all VLBA and VLA antennas are alt-az).
Since this rotation is a simple geometric effect,
it can be corrected by adjusting the phases without
looking at the data. You must do this correction
for polarization and phase referencing experiments.
13) run SPLAT - apply calibration and select out the autocorrelation
14) run VBRFI - to compute the mean and rms/mean spectra averaging
over each scan average and polarization. The spectra
are plotted if DOPLOT>0 and the detailed numbers are
written to a text file if OUTFILE is specified. That
text file will be linked at the top of the output