4.1 Copying data into AIPS multi-source disk files

There are several ways to write VLA data to AIPS multi-source uv data sets on disk. They include:

  1. For VLA data from the archive, use FILLM to read one or more disk files; see 3.10.4. The VLA format was changed on January 1, 1988, but all older data were translated and archived in the modern format. On July 1, 2007, the ModComps were replaced with modern computers and the format had an essential change made to it. Use FILLM to read data from the post-ModComp era. Archive data are obtained from https://archive.nrao.edu/archive/e2earchive.jsp
  2. For an AIPS multi-source data set written to a FITS tape or FITS disk during an earlier AIPS session, use UVLOD or FITLD to read the tape.
  3. For single-source data sets that are already on disk and are very similar in structure, use UV2MS on one of them to create a multi-source data set, and then on each of the others to append them to that multi-source data set. Each of the input data sets should have the same number of polarizations, IFs, spectral channels, and “random parameters.” UV2MS also makes no corrections for differences in observed source positions or frequencies. After all are appended, use UVSRT to put the data in time-baseline order and INDXR to make an index and initial (null) calibration file.
  4. For single-source data sets that are already on disk and are not sufficiently similar in structure for the method above, use MULTI on each single-source file to convert to multi-source format. Then use DBCON to concatenate the individual multi-source files into one big multi-source file. Finally use UVSRT, if needed, to put the data in time-baseline order and INDXR to make an index and initial (null) calibration file.
  5. Data from the Australia Telescope may be loaded from disk files into AIPS using the task ATLOD which is now included with AIPS.

Data from other telescopes can be read into AIPS only if they are written in AIPS-like FITS files already or if you have a special format-translation program for that telescope. The VLBA correlator produces a format which is translated by the standard AIPS task FITLD; see 4.1.2. A translation task for the Westerbork Synthesis Telescope (WSLOD) is available from the Dutch, but is not distributed by the NRAO with the normal AIPS system.

4.1.1 Reading from VLA archive files using FILLM

The NRAO Archive makes available, among other things, data from the EVLA, which began observing in January 2010, and from the old VLA which ceased observing a few days into 2010. To load data from the EVLA into AIPS consult Appendix E for information about BDF2AIPS and other options. The following decribes how to read old VLA data into AIPS.

The Archive now serves VLA data in the form of one or more “MOdComp” format disk files. To load these into AIPS, enter

> TASK FILLM’ ; INP  C R

to review the inputs needed.

> DATAIN ’MYDATA:AG230_’  C R

to read from the disk area pointed at by the logical MYDATA and the data from program ID AG230.

> NFILES 0  C R

to start with file AG230_1. If your first file is e.g.AG230_4, set NFILES = 3.

> NCOUNT 4  C R

to read four data files AG230_1 through AG230_4.

> OUTNA ’ ’  C R

to take the default output file name.

> OUTDI 3  C R

to write the data to disk 3 (one with enough space).

> DOUVCOMP -1  C R

to write visibilities in uncompressed format. VLA files are small by modern standards, so saving space is not worth the costs.

> DOWEIGHT 1  C R

Data weights will depend on the “nominal sensitivity” and should be calibrated along with the visibility amplitudes (DOCALIB = 1).

> CPARM 0  C R

to do no averaging of the data in FILLM.

> CPARM(6) 1  C R

to select VLA sub-array 1.

> CPARM(7) 2000  C R

to have observations within 2 MHz be regarded as being at the same frequency.

> CPARM(8) 1  C R

to use a 2-minute interval for the CL table; default is 5 min.

> CPARM(9) 0.25  C R

to use a 15-second interval for the TY table; default is the input data interval.

> DPARM 0  C R

to have no selection by specific frequency.

> REFDATE yyyymmdd  C R

to specify the year, month, and day of the reference date. This should be the first date in the data set (or earlier). All times in AIPS will be measured with respect to that date and must be positive. The default is the first date included by the data selection adverbs, which may not be the desired one. Note that REFDATE is only a reference point; it does not affect which data are loaded from the the files.

> TIMERANG db , hb , mb , sb , de, he , me , se  C R

 

to specify the beginning day, hour, minute, and second and ending day, hour, minute, and second (wrt REFDATE) of the data to be included. The default is to include all times.

> INP  C R

to review the inputs.

> GO  C R

to run the program when you’re satisfied with inputs.

There are numerous adverbs including BAND, QUAL, CALCODE, VLAOBS. and VLAMODE to limit what data were loaded from magnetic tapes which could hold data from multiple projects. These adverbs still function, but are of little use today. Note that the values given above are illustrative and should not be copied verbatim in most cases.

Be careful when choosing the averaging time with CPARM(1). If you have a large data set, setting this time too low will make an unnecessarily large output file; this may waste disk space and slow the execution of subsequent programs. Setting it too high can, however, (1) smear bad data into good, limiting the ability to recognize and precisely remove bad data, (2) smear features of the image that are far from the phase center, and (3) limit the dynamic range that can be obtained using self-calibration. If you need a different (usually shorter) averaging time for the calibrator sources than for your program sources, use CPARM(10) to specify the averaging time for calibrators. See Lectures 12 and 13 in Synthesis Imaging in Radio Astronomy1 for general guidance about the choice of averaging time given the size of the required field of view and the observing bandwidth.

CPARM(2) controls a number of mostly esoteric options. If your data include the Sun or planets, you must set CPARM(2) = 16 to avoid having each scan on the moving source assigned a different name. The adverb DOWEIGHT = 1 has the same affect as CPARM(2) = 8 and both select the use of the nominal sensitivity to scale the data weights. When this is done, the weights will be 1∕σ2 as they should for imaging, with σ in “Jy” in the same uncalibrated scale as the fringe visibilities. Having selected this option, you should apply any amplitude calibration to the weights as well as the visibilities. If you store the data in compressed form, only one weight may be retained with each sample. Any differences between polarizations and/or IFs in that sample will be lost. Uncompressed data require less cpu, but more real, to read but 2 to 3 times as much disk space to store.

CPARM(2)=2048 allows you to load data as correlation coefficients, which can be scaled to visibilities later with TYAPL (4.1.1.1). CPARM(3) controls which on-line flags are applied by FILLM, which now always writes an OF table containing information about these flags. That information can be viewed with PRTOF and applied selectively to the data at a later time with OFLAG.

FILLM writes a weather (WX) table to the output file. At the same time, it uses “canned” VLA antenna gain curves and a balance of the current with a seasonal model weather data to estimate opacity and gain corrections to be written into the first calibration (CL) table. These functions are controlled by adverbs CALIN and BPARM and may be turned off, although the default is to make the corrections. In subsequent tasks, set DOCALIB = 1 to use these initial calibration data. If, for some reason, the data weights do not depend on the nominal sensitivity, use DOCALIB=100 to apply calibration.

Where possible, FILLM will try to place all data in one file. However, in many cases this is not possible. For instance so-called “channel 0” data from a spectral-line observation will be placed in a separate file from its associated line data. Similarly, scans which have differing numbers of frequency channels will also be placed into separate files. Another case is observations made in mode LP, i.e., one IF-pair is set to L band, the other to P band. In this case the two bands will be split into separate files. Yet another case arises when there are observations of different bandwidths. All of this should be relatively transparent to the user.

FILLM and many AIPS tasks are able to handle multiple, logically different, frequencies within a multi-source data set. FILLM does this by assigning an FQ number to each observation and associating a line of information about that frequency in the FQ file associated with the data set. Users should note that this concept can become quite complicated and that not all tasks can handle it in full generality. In fact, most tasks can only process one FQ number at a time. Polarization calibration works only on one FQ at a time since the antenna file format allows for only one set of instrumental polarization parameters. Therefore, it is strongly advised that you fill continuum experiments which involve multiple frequencies into separate data sets. FILLM will separate bands automatically, but you will have to force any remaining separation. To do this, (a) use the QUAL adverb in FILLM, assuming that you have used separate qualifiers in OBSERVE for each frequency pair; (b) use the DPARM adverb array in FILLM to specify the desired frequencies precisely; or (c) use the UVCOP task to separate a multiple FQ data set into its constituent parts. Note that the first two options require multiple executions of FILLM, while the third option requires more disk space.

Spectral-line users and continuum observers using different frequencies in the same band should be aware of the FQ entry tolerance. Each frequency in a uv file will be assigned an FQ number as it is read from disk by FILLM. For spectral-line users, the observing frequency will normally change as a function of time due to Doppler tracking of the Earth’s rotation, or switching between sources or between spectral lines; in general, this will cause different scans to have different FQ numbers. FILLM assigns an FQ number to a scan based on the FQ tolerance adverb CPARM(7) which defines the maximum change of frequency allowed before a new FQ number is allocated. If CPARM(7) < 0, the the same FQ number is assigned to all data in spectral-line data sets. If CPARM(7) is positive, a scan’s will be assigned to an existing FQ number if

∥ν      - ν      ∥ < CP ARM (7)
  current   firstFQ
where νfirstFQ is the frequency of the first sample to which the particular FQ number was assigned. If no match is found, then a new FQ number is created and assigned and another line added to the FQ table file. Alternatively, if CPARM(7) is zero, then the FQ tolerance is assumed to be half of the maximum frequency difference caused by observing in directions 180 degrees apart (i.e.Δν = 10-4 × ν).

An example: if an observer observes the 1612, 1665 and 1667 MHz OH masers in VY CMa and NML Cygnus, then presumably he would like his data to have 3 FQ numbers, one associated with each OH transition. However, running FILLM with CPARM(7) set to 0 would produce 6 FQ numbers because the frequency difference between the masers in VY CMa and NML Cygnus is greater than the calculated tolerance of 160 kHz. Therefore, in order to ensure that only 3 FQ numbers are assigned, he should set CPARM(7) to 1000 kHz. Setting CPARM(7) < 0 would result in all data having the same FQ number, which is clearly undesirable.

For most continuum experiments the FQ number will be constant throughout the database. Normally any change in frequency should be given a new FQ number. To achieve this, FILLM treats CPARM(7) differently for continuum. If CPARM(7)0.0, then FILLM assumes a value of 100 kHz. A positive value of CPARM(7) is treated as a tolerance in kHz as in the spectral line case.

Note: If your uv database contains several frequency identifiers, you should go through the calibration steps for each FQ code separately.

FILLM can still read from magnetic tape. Set DATIN to blanks, mount your tape (adverb INTAPE), index the tape with PRTTP, and use the adverbs to limit the data loaded to that portion of your project in which you are interested.

If FILLM is executing correctly, your message terminal will report the number of your observing program, the VLA archive format revision number, and then the names of the sources as they are found in the data files. Once FILLM has completed, you can find the database on disk using:

> INDI 0 ; UCAT  C R

This should produce a listing such as:

Catalog on disk  3  
Cat Usid Mapname      Class  Seq  Pt     Last access      Stat  
  1  103 25/11/88    .X BAND.   1 UV 05-FEB-1994 12:34:16

You might then examine the header information for the disk data set by:

> INDI 3 ; GETN 1 ; IMH  C R

This should produce a listing like:

 Image=MULTI     (UV)         Filename=25/11/88    .X BAND.   1  
 Telescope=VLA                Receiver=VLA  
 Observer=AC238               User #=  103  
 Observ. date=25-NOV-1988     Map date=05-FEB-1994  
 # visibilities    191317     Sort order  TB  
 Rand axes: UU-L-SIN  VV-L-SIN  WW-L-SIN  BASELINE  TIME1  
            SOURCE  FREQSEL WEIGHT  SCALE  
 --------------------------------------------------------------  
 Type    Pixels   Coord value  at Pixel    Coord incr   Rotat  
 COMPLEX      1   1.0000000E+00    1.00 1.0000000E+00    0.00  
 STOKES       4  -1.0000000E+00    1.00-1.0000000E+00    0.00  
 IF           2   1.0000000E+00    1.00 1.0000000E+00    0.00  
 FREQ         1   8.4110000E+09    1.00 1.2500000E+07    0.00  
 RA           1    00 00 00.000    1.00      3600.000    0.00  
 DEC          1    00 00 00.000    1.00      3600.000    0.00  
 --------------------------------------------------------------  
 Maximum version number of extension files of type HI is   1  
 Maximum version number of extension files of type AN is   1  
 Maximum version number of extension files of type NX is   1  
 Maximum version number of extension files of type SU is   1  
 Maximum version number of extension files of type FQ is   1  
 Maximum version number of extension files of type WX is   1  
 Maximum version number of extension files of type CL is   1  
 Keyword = ’CORRMODE’  value = ’        ’  
 Keyword = ’VLAIFS  ’  value = ’ABCD    ’

This header identifies the file as a multi-source file (Image=MULTI) with 191317 floating-point visibilities in time-baseline (TB) order. There are two entries on the IF axis. These correspond to the old VLA’s “AC” and “BD” IF-pairs respectively. The description of the frequency (FREQ) axis shows that the first IF (“AC”) is at 8411 MHz and has 12.5 MHz bandwidth. The parameters of the second IF-pair (“BD”) are determined from the data in the FQ table file and cannot be read directly from this header; these values are shown in the SCANlisting from LISTR. The header shown above indicates that the data are in compressed format since the number of pixels on the COMPLEX axis is 1 and the WEIGHT and SCALE random parameters are present. Uncompressed data does not use these random parameters and has 3 pixels on the COMPLEX axis.

The term “IF” can be confusing. At the VLA, IFs “A” and “C” correspond to right-hand and left-hand circularly polarized (RHC and LHC) signals, respectively, and are normally for the same frequency in an observing band. Such pairs, if at the same frequency, are considered to be one “IF” in AIPS. An observation which was made in spectral line mode “2AC” is considered at the VLA to have two “IFs” whereas within AIPS this would be filled as one “IF” with two polarizations if they were both observed with the same frequency, the same number of channels, and the same channel separation. If these conditions do not hold, then they are filled into separate uv files, each with a single IF and a single polarization. The term “sub-array” is also confusing. At the VLA — and in task FILLM — sub-array means the subset of the 27 antennas actually used to observe your sources. (The VLA allows up to 5 simultaneous sub-arrays in this sense.) In the rest of AIPS, sub-array refers to sets of antennas used together at the same time. If observations from separate times (e.g., separate array configurations) are concatenated into the same file, then AIPS will regard the separate sets of antennas as different “sub-arrays” whether or not the same physical antennas occur within more than one of these sub-arrays.

If your experiment contains data from several bands FILLM will place the data from each band in separate data sets. Also, if you observed with several sets of frequencies or bandwidths in a given observing run these will be assigned different FQ numbers by FILLM. You can determine which frequencies correspond to which FQ numbers from the SCANlisting provided by LISTR. Line data are divided into the “channel 0” (central 34 of the of the observing band averaged) and the spectra. Data observed in the “LP” mode (or any other two-band mode) will be broken into separate data sets, one for each band.

4.1.1.1 Editing and applying nominal sensitivities to VLA data

FILLM scales the correlation coefficients by the instantaneous measured “nominal sensitivities,” producing data approximately in deci-Jy. The VLA nominal sensitivities are stored in the TY table as “system temperatures” (Tsys). For calibration purposes, it is best to have the nominal sensitivities applied, but it may be better to use a clipped and/or time-smoothed version of those sensitivities. If you want to do this, load the Tsys data into the TY table with the highest time resolution possible by setting CPARM(9)=0 in FILLM. FILLM can also be told not to apply the nominal sensitivities and therefor produce correlation coefficients by setting CPARM(2)=2048, but this is not strictly necessary. In order to smooth and clip the TY table use the task TYSMO. If you have done editing such as QUACK, it may help to copy the data with UVCOP, applying your flag table not only to the visibilities but also to the TY table (UVCOPPRM(6)=3) before running TYSMO to remove questionable values at the start of scans. Alternatively, SNEDT will apply the data flags to the table allowing you to write a new, cleaned-up version of the table. Then a TY table may be applied (and/or removed) from a data set with TYAPL:

> TASK TYAPL’ ; INP  C R

to review the inputs needed.

> INDI n ; GETN m  C R

to select the correct data set.

> FREQID 1  C R

to select FQ number 1.

> INVERS 1  C R

TY table to remove from data, will only work if data are not already correlation coefficients.

> IN2VERS 2  C R

smoothed TY table to apply to data, will only work if data is in correlation coefficient form — either initially or after removal of INVERS.

> INP  C R

to re-check all the inputs parameters.

> GO  C R

to start the task.

EVLA users (see Appendix E) should have an SY table which contains system gain and temperature data. These data should be applied to the visibility data in order to correct for measured gain changes and to convert the data weights from a simple count of the integration time into more meaningful values. Tasks TYSMO and SNEDT may be used to clean up the SY data and then TYAPL may be used to apply the gain and system temperature data.

4.1.2 Reading data from FITS files with FITLD

FITLD is used to read FITS-format disk files (and tapes) into AIPS. It recognizes images, single- and multi-source uv data sets, and the special FITS uv-data tables produced by the VLBA and DiFX correlators (“FITS-IDI” format). In particular, VLA data sets that have been read into AIPS previously with FILLM and then saved to tape (or pseudo-tape disk) files with FITTP and FITAB can be recovered for further processing with task FITLD. (The older task UVLOD will also work with uv data sets in FITS format, but it cannot handle image or FITS-IDI format files.)

A multi-source data file with all of its tables can be read from a FITS tape by:

> TASK FITLD’ ; INP  C R

to review the inputs needed.

> INTAPE n  C R

to specify the tape drive for input from tape.

> DATAIN filename  C R

if the input is from a FITS disk file (see 3.10.3).

> DOUVCOMP FALSE  C R

to write visibilities in uncompressed format.

> OUTNA ’ ’  C R

take default (previous AIPS) name.

> OUTCL ’ ’  C R

take default (previous AIPS) class.

> OUTSEQ 0  C R

take default (previous AIPS) sequence #.

> OUTDI 3  C R

to write the data to disk 3 (one with enough space).

> INP  C R

to review the inputs (several apply only to VLBA format files).

> GO  C R

to run the program when you’re satisfied with inputs.

FITLD is the equivalent of FILLM, but for output from the VLBA, rather than the VLA, correlator. The data-selection adverbs SOURCES, QUAL, CALCODE, and TIMERANG and the table-control adverbs CLINT and FQTOL are used, for VLBA-format data only, in FITLD in ways similar to the data-selection and control adverbs of FILLM. See Chapter 9 for more specific information.