AIPS NRAO AIPS HELP file for VLBARUN in 31DEC19



As of Thu Dec 12 4:23:23 2019


VLBARUN: Applies amplitude and phase calibration to VLBA data !

INPUTS


          To run this procedure you MUST type RUN VLBAUTIL and
          RUN VLBARUN first to define the procedures in AIPS.

                                   -- Load the data from disk --
DATAIN                             Disk file name
DOUVCOMP                           Compress data? Use -1, 0 or 1
OUTNAME                            File name (name)
OUTDISK                            Working disk with ample space
                                   -- OR existing file --
INNAME                             Input file name
INCLASS                            Input file class
INSEQ                              Input file sequence number
INDISK                             Disk number for input file
                                   -----------------------------
OPTYPE                             For CONT, PSEU or SPEC (LINE)
                                   -----------------------------
CLINT                              CL table interval in minutes.
CHREFANT                           Reference antenna NAME
TIMERANG                           A good bandpass scan, or use
                                     scan# in TIMERANG(1)+rest=0
INVER             -1.0   46655.0   PC table to use
                                   -1 => don't use Pulse cals
                                       do manual phase cal
CALSOUR                            List ALL Fring fit & CAL sour
                                     FIRST is bandpass calib
SOURCES                            ONLY FOR PHASE REFERENCING:
                                     Source *PAIRS* to calibrate
                                     START each pair with phase
                                     referencing calibrator, if
                                     2nd='*': all non-CALSOUR
                                     are phase-referenced to 1st
INFILE                             Set to DELZN correction file
TECRFILE                           Local copy of TEC file
                                   '' => download autmatically
EOPSFILE                           Local copy of EOP file
                                   '' => download autmatically
                                   -----------------------------
SOLINT                             Time interval for fringe-fit
                                   -----------------------------
IMSIZE                             Image size for target SOURCES
                                     =-1 for no images
FACTOR                             CALSOUR IMSIZE (FACTORx128)
                                   -----------------------------
DOCLIP                             > 0 -> use CLIP on all
                                   sources: gives clip levels
DOFLAG                             > 0 -> use RFLAG on all
                                   sources - clip level is
                                   max (100, DOFLAG) if a
                                   clip limit > 100 Jy is
                                   desired (e.g. masers)
TYSMOLEV                           > 0 -> clip level in TYSMO
ELEVLIM                            > 0 -> flag elevation <
                                          ELEVLIM
INTEXT                             Input flag commands
          Hint: when setting directory and e-mail lower-case
          letters can be retained by NOT using the close quote.
DOPLOT                             <= 0 no plots;=1 some plots
                                   >1 huge number of plots
OUTFILE                            DIRECTORY FOR HTML AND PLOT
                                     FILES: if you want an html
                                     file with plots please set
                                     the dir where the html and
                                     plot files can be put.
OUTTEXT                            E-MAIL ADDRESS: if you want
                                     an e-mail when the script
                                     is complete, set the
                                     address here.
BADDISK                            Disks to avoid for scratch

HELP SECTION

VLBARUN
Type: Procedure
Use:  VLBARUN is the procedure that uses the VLBA calibration
      procedures (VLBAUTIL) to calibrate VLBA data.  See the explain
      file for a detailed description of the procedure.

NOTE: currently this procedure is intended for simple experiments!

      Type RUN VLBAUTIL and RUN VLBARUN to define the VLBARUN
      procedures.  Do not do this again unless you lose the procedures
      somehow.  HELP PROCS will show the names of all defined
      procedures.

      The procedure is run by typing : VLBARUN

      The procedure will check the inputs (as far as it can) to avoid
      later interruptions, and then proceed directly, if the data are
      already on disk.  The procedure will clear the message file when
      it starts, so print any messages you need to save.  This insures
      that the printed message file at the end contains only the
      messages from this run.

CLEAN STARTING CONDITIONS
      It is best to use one project per user number, as otherwise you
      may get into trouble if different projects use the same source
      names.  t is not recommended to run two pipelines with the same
      user number simultaneously, although with different POPS numbers
      it should work. Overlapping VLBARUNs at any time during the
      processes may also have issues.

      Clean up all files resulting from previous attempts of running
      the procedure, except for the original UVDATA and perhaps, when
      restarting a specific frequency-ID, also not the F(X)POL files.
      When restarting from disk, make sure the UV data has no extra
      SN, CL, PL files.  Note that you may want to keep flag (FG)
      tables attached to the input file so that they will be applied
      during all stages of the pipeline run.  Use the procedure
      P_RESTART (which is defined in VLBARUN) to do this.  Note that
      P_RESTART does not delete FG tables.  NEVER DELET FG TABLE 1,
      the on-line flags.

      If you have any antennas that do not have Tsys or Gain curves in
      the data, please load them with ANTAB into TY and GC tables #1.

      It is a VERY good idea to run TASAV before you do this to keep
      the original TY, GC, BL and also FG tables number 1 in case of
      disasters.

      Make sure there is enough disk space available on OUTDISK, about
      4 to 5 times the expected compressed UVDATA set. For spectral
      line probably you need much more because it will make spectral
      line image cubes.

  DATAIN......48-character name of the disk file from which to read a
              FITS file.  It must be in the form
                  :
              or
                  :::
              where  is the remote computer name,  is the
              environment variable (logical name) for the disk area in
              which the file named  is stored.   is
              usually omitted when the file is local to the current
              computer.  If DATAIN is not found, the task will try
              DATAIN with the character 1 appended.
              Beginning 2003-Oct-16 FITLD can read more than one disk
              file at a time.  In that case, they must all have the
              same name except that the last letter(s) are the
              sequence number 1 through NCOUNT with no leading zeros.
              DATAIN must give the base name only, omitting the
              sequence number.  Then either
                    DATAIN = 'MYDATA:file_
              or
                    DATAIN = 'MYDATA:file_1
              can access files in $MYDATA named file_1, file_2,
              file_3, etc.  Note that this is similar to the pieces of
              a FITS pieced UV-table data set, but only one UV-table
              data set may be read at a time.
  DOUVCOMP....If true (> 0.5) then output data will be compressed
              which saves disk space at the expense of losing some
              weighting information.
  OUTNAME.....The name for the output files. The output files will all
              be given the class UVDATA and assigned distinct sequence
              numbers as needed.
  OUTDISK.....The disk drive number for the output data if loading from
              disk. Choose a disk with a large amount of space.  If
              data are already loaded into AIPS (i.e. you are using
              INNAME, INCLASS etc) then OUTDISK is ignored and INDISK
              is used.
  INNAME......Input UV file name (name) that is already loaded.
              Note that you CANNOT use both DATAIN/OUTNAME/OUTDISK
              and INNAME/INCLASS/INSEQ/INDISK
  INCLASS.....Input UV file name (class) that is already loaded.
  INSEQ.......Input UV file name (seq. #) that is already loaded.
  INDISK......Disk drive # of input UV file.  NOTE: this is also
              used as OUTDISK.
  OPTYPE......The type of observation and requested output files:
              ''     = 'CONT' for channels < 65; 'SPEC' otherwise
              'CONT' =  Plain continuum observations, uses pulse cals
                        unless INVER = -1
              'PSEU' = Pseudo-continuum observations, uses pulse cals
                       unless INVER = -1.  The data is NOT averaged
                       over channels, but the resulting images are
                       continuum images - used for large FOVs
              'SPEC' = Spectral line observations.  Manual pulse-cal
                       is performed.  Data is unaveraged and images
                       are spectral line cubes of the phase referenced
                       target sources ONLY.
              'LINE' = Same as 'SPEC'.
  CLINT.......Calibration table interval in minutes. This should
              normally be in the range 0.25 to 1.0.  If <=0 then
              0.25 is assumed.
  CHREFANT....Name of reference antenna.  If not set, a sensible
              antenna in the southwest will be tried (FD, PT, LA or
              KP).
  TIMERANG....Time range on which ALL antennas have good data on the
              bandpass calibrator (see below, CALSOUR(1)).  Used for
              instrumental phase calibration by either the pulse-cals
              or manually by running FRING on this scan.
              If you know the scan number, you can use the scan number
              e.g., set TIMERANGE=66,0 to get the time range
              associated with scan 66.
              If all 0, then VLBARUN will try to find a good scan by
              running BSCAN on the bandpass calibrator.
  INVER.......PC table to use, if -1 do manual phase cal
  CALSOUR.....All the calibrators should be listed here.  All these
              sources will be FRING fit.  IMPORTANT: the FIRST source
              is the source used for instrumental phase and bandpass
              calibration.  If TIMERANGE was set, it must be a scan on
              CALSOUR(1).  If you want all sources fringe fitted then
              you may use CALSOUR(2)='*' but remember to also specify
              CALSOUR(1).  These sources will be imaged in (pseudo)
              continuum mode.
  SOURCES.....For phase referencing.  List PAIRS of
              phase-referencing/target sources.  The phase calibrator
              should be the first in the pair, i.e., with the odd
              index.  Each target should be the second in the pair,
              i.e., have an even index.  All the phase calibrators
              must be in the CALSOUR list as well.  If your are not
              phase-referencing any source, then leave SOURCES blank.
              If SOURCES = 'PRCAL', '*' then all non-calibrators are
              phase referenced to 'PRCAL'.
              There is a maximum of 15 pairs, a total of 30 source
              fields.
  SOLINT......The solution interval (min.) for FRING fit.  0 => 1
              minute  If SOLINT > Scan/2 (in Multisource) SOLINT =
              Scan.
  INFILE......If you want to apply a DELZN corrections file, set
              INFILE to the filename.  See CLCOR explain file for more
              details, the file format is the OPCODE='ATMO' format
              (which is the same as DELZN produces).
  TECRFILE....Input file.  If ''=> file will be automatically
              downloaded.  If there is a problem with automatic
              downloads (this can be a problem with some firewalls)
              the file can be can be downloaded manually.  See EXPLAIN
              VLBATECR for details on how to download the file(s).
              The file should contain TEC maps spanning the time
              range covered by the input CL table and should be in
              IONEX format.  If NFILES > 1 then the name MUST be in
              standard format CCCCdddC.yyC where C can be any
              character, ddd is the day number, and yy is the year.
              Also the name given in INFILE must be the first day.
  EOPSFILE....Input file.  If ''=> file will be automatically
              downloaded.  If there is a problem with automatic
              downloads (this can be a problem with some firewalls)
              the file can be can be picked up from:
                https://gemini.gsfc.nasa.gov/solve_save/usno_finals.erp
  IMSIZE......Output target image sizes.  Minimum IMSIZE of 512 up to
              8192 pixels.  Sources listed in CALSOUR are automatically
              imaged with a size of 128 by 128 unless FACTOR is set.
              Pixel size is automatically set by frequency:
              2.25/Frequency(GHz) mas.
              IMSIZE=-1 do not make images.
  FACTOR......scaled image size of CALSOURs.  Image size of sources
              listed in CALSOUR is FACTOR*128.  The maximum FACTOR is
              64.
  DOCLIP......If DOCLIP(1) > 0, clip parallel hand data at DOCLIP(1)
              Jy and cross-hand data at DOCLIP(2) Jy.  If DOCLIP(2)=0
              no cross-hand clipping is done.  Parallel hand data are
              flagged when cross-hand data are and vice versa.
  DOFLAG......> 0 says to run RFLAG twice for each source, the first
              time to set the flagging levels and the second time to
              apply them.  Plot files are made in each pass but do not
              appear in the output html page (if any).  If you are
              observing a source with strong lines, the value of the
              pre-clip in Jy (FPARM(13)) is set to max (100, DOFLAG).
              <= 0 -> use any flag tables present with the data file
              but do not create any new ones.  Warning these RFLAG
              flag tables can get large.
  TYSMOLEV....> 0 => run TYSMO on the TY table.  The input TY values are
              clipped between below TYSMOLEV/20 and above TYSMOLEV.
              Then the data are median window filtered over a 10 minute
              interval and values > TYSMOLEV/10 from the median are
              flagged.  Then a 15-minute box filter is used to replace
              all flagged values with a smoothed value from a nearby
              time.  Do NOT do this unless you think the TY table needs
              regularizing (and many do).
  ELEVLIM.....> 0 => Flag all data below ELEVLIM degrees
  INTEXT......If not blank, text file with flagging commands to be
              read by UVFLG.
              Adverb values are entered in a text file with
              descriptions of data to be flagged separated by a '/'.
              There can be up to 40000 selection criteria in the edit
              file.  See EXPLAIN UVFLG.
  DOPLOT......Make diagnostic plots to judge quality of procedure
              results.
                 <=0 => no plots
                 1 => some plots
                 2 => lots of plots (could be hundreds)
  OUTFILE.....Directory of html and plots files.  If you want the
              diagnostic plots written to disk and an html file
              created to make the plots easier to examine please set
              directory here.  This is limited to 37 characters
              (including final /).  Environmental variables are
              allowed and must end with a ':'.  Mixed environmental
              and regular are not allowed, e.g.,
                OUTFILE='/home/computer/BM394/pipeline/
                OUTFILE='FITS:'
              are allowed.
              BUT:
                OUTFILE='FITS:BM394/
              is not.
              A subdirectory will actually be used so this directory
              may contain previous subdirectories.  It should not
              contain any PostScript files.  A text file containing
              all of the messages will be saved in the subdirectory
              with the name messages.txt.
  OUTTEXT.....E-mail address for notifications when script is done or
              failed.
  BADDISK.....A list of disk numbers to be avoided when creating scratch
              files.

EXPLAIN SECTION

VLBARUN:           Procedure that attempts to calibrate VLBI data blindly.
Documentor:        Amy Mioduszewski, amiodusz@nrao.edu.
Related Programs:  All VLBAUTIL procs, BPASS, SPLIT, IMAGR,..
                                  POSSM, VPLOT and many many more

VLBARUN applies VLBAUTIL procedures to simple VLBI experiments.
VLBARUN attempts to make sensible choices if inputs are left to
default.  The only inputs that must be set are:
   DATAIN or INNAME (INCLASS, INSEQ and INDISK) -> so the dataset is
           defined
   CALSOUR(1) -> so VLBARUN knows what calibrator to use as the
           bandpass/instrumental phase calibrator
   SOURCES -> if the experiment is phase referenced

More detail about defaults

VLBARUN tries to pick sensible defaults if inputs are left blank.  Below
describes how it makes these decisions.
  OPTYPE=''   -> VLBARUN checks the total number of channels and decides from
                 that whether dataset is continuum (64 or less) or spectral
                 line (>65).
  CLINT=0     -> set to 0.25 minutes
  CHREFANT='' -> Chooses the first antenna present from the southwest VLBA
                 antennas in the order of FD, PT, LA and KP.
  TIMERANGE=0 -> Runs the task BSCAN on CALSOUR(1) to select scan with the
                 highest signal to noise with all the antennas present. If
                 INVER>=0 then BSCAN also checks if there are PC table
                 entries for that scan.

Diagnostic plots

VLBARUN offers the option to create diagnostic plots.  If DOPLOT is set
to 0 no plots are made.  For 0<DOPLOT<1.5 (i.e. DOPLOT=1) the following plots
are produced:
    -POSSM plot of bandpass solution
    -SNPLT of amplitudes in CL table after ACSCL
    -SNPLT of amplitudes in CL table after APCAL
    -POSSM plots of amp and phase after instrumental phase calibration
    -SNPLT of phase, delay, rate and SNR in CL table after FRING
    -VPLOT of phase of data for phase reference sources with all calibration
     applied
    -RFLAG of rms spectrum of time and spectral deviations
    -KNTR (contour) plots of the images produced, if images are requested

For DOPLOT>1.5, IN ADDITION to the above plots the following are produced
as well:
    -SNPLT of amplitudes in SN table after ACSCL
    -SNPLT of amplitudes in SN table after ACCOR
    -SNPLT of amplitudes in CL table after ACCOR
    -SNPLT of amplitudes in SN table after APCAL
    -SNPLT of phase, delay, rate and SNR in SN table after FRING
    -SNPLT of amplitude in CL table after FRING

Output of plots to disk, creation of html file and e-mail message

VLBARUN will create .gif files on disk in a subdirectory of the
directory defined by OUTFILE of the plots that are requested.  It then
makes a vlbarun.html file that includes headings for the plots and at
the bottom a list of the CL tables and what calibration is added to
each.  This vlbarun.html file is intended to make examination of the
diagnostic plots easier, so the user can quickly judge whether the
pipeline worked.  A link to the message file from the run is also
provided in the vlbarun.html file.

It is STRONGLY recommended that the directory defined by OUTFILE be
empty of PostScript files alsthough it may contain date-stamped
subdirectories.  VLBARUN creates and deletes PostScript files in this
directory so it is safer if there aren't any such files for VLBARUN to
potentially clobber or delete.  VLBARUN makes a subdirectory in
OUTFILE named by the date and time and the final gif files and
vlbarun.html will reside in the subdirectory.  VLBARUN checks if there
is an existing vlbarun.html file and will not run if there is.  That
should happen only if you restart VLBARUN within the same minute.

It is also recommended that if OUTFILE is used then an e-mail address
be provided in OUTTEXT.  The e-mail will state whether VLBARUN is done
(subject: VLBARUN DONE) or failed (subject: VLBARUN FAILED).  The
FAILED message only catches failure do to a problem with how VLBARUN
was set up, if the pipeline fails because a task fails that is not
captured and no e-mail message is sent.  If OUTFILE is set, the DONE
message includes the URI to the vlbarun.html file assuming that the
machine you are viewing it on is the "localhost".  So if you are on
that machine, you can copy that URI ("file:///etc") into your web
browser and look at the plots.  If you are not on the machine where
the plots were created then just copying the file:///... URI will not
work.

CONSIDERATIONS FOR MAC OS X AND OTHER OS THAT MIGHT NOT INCLUDE COMMON
LINUX COMMAND-LINE TOOLS  -- the output of plots to display using .gif
and html files requires the command-line command "convert" or "magic".
If you do not have "convert" or "magic" then you cannot use the
OUTFILE input.  There are packages you can install on your machine
that will give you access to linux command-line tools like convert.

What does VLBARUN do?

Below are the basic calibration steps that VLBARUN performs.  This does not
include all the plotting etc..
 0) checks that the inputs are correct as well as it can, sets some
    defaults.
 1) run VLBALOAD - if DATAIN is specified, loads the data from a FITS
                   or FITS-IDI disk file.  Merges duplicate table files
                   Runs VLBAFIX to sort the data if needed and make an
                   index table.  If there are multiple frequency IDs,
                   VLBARUN stops and tells the user to separate them into
                   separate data sets.
 2) run VLBATECR - corrects ionosphere if the observing frequency is
                   less than 12 GHz.  This procedure uses Global
                   Positioning System (GPS) models of the electron
                   content in the ionosphere to correct the dispersive
                   delays caused by the ionosphere.  This correction
                   is particularly important for phase referencing
                   experiments at low frequency.
 3) run VLBAEOPS - corrects Earth Orientation Parameters (EOP) used in
                   correlation.  VLBI correlators must use
                   measurements of the EOPs while correlating. These
                   change slowly with time and therefore the EOPs used
                   by the correlator must be continually updated.
                   Even though the EOPs that a correlator uses are the
                   the most accurate at the time of correlation,
                   frequently after the correlation the EOPs are
                   improved.  This procedure inserts a correction for
                   the difference between the EOPs used in correlation
                   and the current best calculation of the EOPs.  This
                   is particularly important for phase referencing.
 4) run UVFLG    - If INTEXT is not blank, it is used to add flag
                   commands to FG table 1.
 5) run TYSMO    - If TYSMOLEV > 0, apply TYSMO to the TY table making
                   a corrected second version.  TYSMO begins by
                   flagging all Tsys values less than TYSMOLEV/20 and
                   greater than TYSMOLEV.  It then does a median window
                   filter with a width of 10 minutes and a clip level
                   of TYSMOLEV/10.  Finally a boxcar of width 15 minutes
                   computes a smoothed version and all clipped values
                   are replaced with smoothed values.
 6) run UVFLG    - If ELEVLIM > 0, run UVFLG to flag all data for which
                   one of the antennas has a source elevation less than
                   ELEVLIM degrees.
 7) run VLBACCOR - sampler corrections.  This procedure corrects the
                   amplitudes in cross- correlation spectra due to
                   errors in sampler thresholds using measurements of
                   auto-correlation spectra
 8) run VLBAPCOR or VLBAMPCL - correct instrumental delays if there is
                   more than one IF.  These delays or "instrumental
                   single-band delays" are caused by the passage of
                   the signal through the electronics of the baseband
                   converters and can be corrected with either the
                   pulse cals (VLBAPCOR) or by running FRING on one
                   scan (VLBAMPCL).
 9) run VLBABPSS - Runs BPASS to calibrate the bandpass response
                   functions for each antenna.
10) run VLBAAMP  - Runs ACSCL which applies the BP table and previous
                   calibration tables and corrects the
                   cross-correlation spectra using auto- correlations.
                   For more information see VLBA Scientific Memo #37
                   (Craig Walker). Then preforms a-priori amplitude
                   gain calibration using the gain curves and system
                   temperatures.
11) run VLBAPANG - correct paralactic angle.  The RCP and LCP feeds on
                   alt-az antennas will rotate in position angle with
                   respect to the source during the course of the
                   observation (all VLBA and VLA antennas are alt-az).
                   Since this rotation is a simple geometric effect,
                   it can be corrected by adjusting the phases without
                   looking at the data. You must do this correction
                   for polarization and phase referencing experiments.
12) run CLCOR - OPTIONAL, if INFILE is not blank will run CLCOR with
                   OPCODE='ATMO' in order to apply a DELZN output file
                   specified in INFILE.
13) run FRING - final fringe fit of the data.  This step removes the
                   global frequency- and time-dependent phase errors.
14) run RFLAG - optionally, runs RFLAG twice on each source, the first
                   time to get flag levels and the second to apply
                   them.  Plot files are generated.
15) run SPLIT - applies final calibration and splits into single
                   source files.  If OPTYPE 'CONT' then frequencies
                   are averaged.
16) run IMAGR - OPTIONAL, if IMSIZE >= 0 then produce images of all the
                   sources using the IMAGR autoboxing feature.  If
                   OPTYPE='SPEC', separate image cubes are made for
                   each IF.

AIPS