; VLBAPIPE - Lorant Sjouwerman, NRAO - DEC 04 02
; Ver 1.0 : released to public in July 2002
; Ver 2.1 : released in MNJ in November 2002
; Ver 3.2 : rewrite, released in MNJ in December 2002
;---------------------------------------------------------------
;! applies amplitude and phase calibration procs to VLBA data
;# RUN POPS VLBI UTILITY CALIBRATION
;---------------------------------------------------------------
;;  Copyright (C) 2001-2003
;;  Associated Universities, Inc. Washington DC, USA.
;;
;;  This program is free software; you can redistribute it and/or
;;  modify it under the terms of the GNU General Public License as
;;  published by the Free Software Foundation; either version 2 of
;;  the License, or (at your option) any later version.
;;
;;  This program is distributed in the hope that it will be useful,
;;  but WITHOUT ANY WARRANTY; without even the implied warranty of
;;  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;  GNU General Public License for more details.
;;
;;  You should have received a copy of the GNU General Public
;;  License along with this program; if not, write to the Free
;;  Software Foundation, Inc., 675 Massachusetts Ave, Cambridge,
;;  MA 02139, USA.
;;
;;  Correspondence concerning AIPS should be addressed as follows:
;;         Internet email: aipsmail@nrao.edu.
;;         Postal address: AIPS Project Office
;;                         National Radio Astronomy Observatory
;;                         520 Edgemont Road
;;                         Charlottesville, VA 22903-2475 USA
;---------------------------------------------------------------
VLBAPIPE  LLLLLLLLLLLLUUUUUUUUUUUU CCCCCCCCCCCCCCCCCCCCCCCCCCCCC
VLBAPIPE: Applies amplitude and phase calibration to VLBA data !

          To run this procedure you MUST type RUN VLBAUTIL and
          RUN VLBAPIPE first to define the procedures in AIPS.

          This is 'the' input dialog to the VLBA pipeline.
          There are NO DEFAULTS! Note the special use of
          almost all variables !
            -> READ THE HELP/EXPLAIN FILE FOR 'CLEAN' STARTING
               CONDITIONS AND SPECIFIC USE OF ALL THE VARIABLES.

          When you run VLBAPIPE with DOALL > 0 the procedure
          will ask about loading tapes.  You will have to type
          0 <return> to confirm that the tape has been loaded
          into the drive.  When it is done with loading data
          it will run to completion on its own.
                   - WHEN IT GOES, IT GOES !!! -

                                   -----------------------------
DOALL                              Tape unit #, < 1 to skip load
                                   -> READ THE HELP/EXPLAIN FILE
NCOUNT                             Number of distribution tapes.
APARM                              Number of files on each tape.
DOUVCOMP                           Compress data? Use -1, 0 or 1
                                   -----------------------------
OUTNAME                            VLBA program name (eg BZ099A)
                                    Restarts use this as INNAME!
                                    First 6=outclass cal'd data.
OUTDISK                            Working disk with ample space
                                   -----------------------------
OPTYPE                             For CONT, PSEU or SPEC (LINE)
                                   -----------------------------
CLINT                              CL table interval in minutes.
REFANT                             Reference antenna #, or use 0
SORT                                and eg. SORT 'PT' if unknown
                                   Specify either REFANT or SORT
                                   -----------------------------
INFILE                             1st IONEX file name for TECOR
NFILES                             Number of TECOR files to use;
                                    NFILES < 1: don't run TECOR.
                                   -----------------------------
TIMERANG                           A good band-pass scan, or use
                                    scan# in TIMERANG(1)+rest=0.
CALSOUR                            List ALL Fring fit & CAL sour
                                    FIRST is bandpass calibrator
                                    in TIMERANG specified above.
                                    If CALSOUR(2)='*', then will
                                    take every source in SUtable
                                    Put all strong sources here.
SOURCES                            Source * PAIRS * to calibrate
                                    START each pair with Ph-Ref.
                                    2nd='*': all non-calibrators
                                    are phase-referenced to 1st.
                                   SOURCES ONLY USED FOR PH-REFS
                                   -----------------------------
SOLINT                             Time interval for fringe-fit,
BPARM                              or : SOLINT per frequency-ID.
                                    Use either SOLINT /or/ BPARM
INTERPOL                           Interpolation: '2PT', 'SIMP',
                                    'AMBG', or 'CUBE' (in CLCAL)
                                   -----------------------------
IMSIZE                             Image size for 'even' sources
FACTOR                              others are FACTORx128 square
                                   IMSIZE ONLY USED FOR SOURCES
                                   IMAGING CALSOUR USES FACTOR
                                   -----------------------------
DOTV                               Plot type: >0 then plot on TV
                                    =0 make PL file, <0 make PL
                                    files and send to printer.
                                    Recommend =0, discourage =-1
BADDISK                            Disks to avoid for scratch

          VLBAPIPE is defined in the standard AIPS runfile area.

----------------------------------------------------------------

VLBAPIPE
Type: Procedure
Use:  VLBAPIPE is the procedure that applies the VLBA calibration procedures
      (VLBAUTIL) for a-priori amplitude and phase corrections, and also
      performs fringe fitting on single strong sources and phase-references
      weak sources.  Simple visibility flagging, bandpass calibration, and
      ionospheric (total electron content) corrections are applied before
      fringe fitting and averaging.  It may attempt to self-cal the
      calibrators and will image the targets.

    ====================================================================
    READ THE HELP/EXPLAIN FILE THOROUGHLY BEFORE USING SINCE NONE OF THE
                 AIPS INPUTS ARE USED IN THE USUAL WAY.
    ====================================================================

NOTE: currently this procedure is intended for simple VLBA-only experiments!

      More info on http://www.aoc.nrao.edu/vlba/html/vlbahome/observer.html
      under 'Calibration and Imaging' of NRAO's VLBA web page.

$----------------------------------------------------------------------------

      Type RUN VLBAUTIL and RUN VLBAPIPE to define the VLBAPIPE procedures;
      it is wise to type RESTORE 0 (before the 'run' commands above) or
      DEFAULT (after the 'run' commands, and setting TASK 'VLBAPIPE') to
      set the default task values. Default values should be MEANINGLESS here.
      Note that a 'RESTORE 0' after RUN VLBAPIPE deletes all the procedures!

      The procedure is run by typing : (GO) VLBAPIPE
      The procedure will check the inputs (as far as it can) to avoid later
      interruptions, and then proceed directly, if the data is already on
      disk, or after the last tape has been loaded if the data is not on disk.
      Some manual operation (loading tapes and typing a zero) is needed.

CLEAN STARTING CONDITIONS
      I prefer to use one project per user number, as otherwise you may get
      into troubles if different projects use the same source names. And I
      use a different user number than for my other non-VLBA projects. This
      means I can type RESTORE 0 without it affecting my tget's and catalogs
      that I use for my other non-VLBA projects. But this is up to you.
      It is not recommended to run two pipelines with the same user number
      simultaneously, nor overlapping at any time during the processes.

      Clean up all files resulting from previous attempts of running the
      procedure, except for the original UVDATA and perhaps, when restarting
      a specific frequency-ID, also not the F(X)POL files. When restarting
      from disk, make sure the UV data has no extra SN, CL, FG etc tables.
      Use the procedure P_RESTART (which is defined in VLBAPIPE) to do this.

      When you start the procedure on a new data set which includes sources
      that have the same name in a previous run, then RENAME the images from
      the prevoius run. IMAGR otherwise attempts to overwrite the earlier
      images with the same source name and dies. If you  wish to flag some
      additional data (because the procedure did not flag all the bad data
      automatically), make sure you put the flags in FG table number 1.

      The procedure uses the CALIBRATION TRANSFER FILEs that come with the
      data on the VLBA data distribution tape. Occasionally a VLA antenna,
      and/or the Effelsberg or Green Bank telescopes also participated. For
      now that means that if you want these antennas also to contribute to
      the calibrated data set, you would want to load all the data with
      VLBALOAD, and include the external calibration of the non-VLBA antennas
      in the old-fashioned way with ANTAB. Write the external calibration
      system temperatures and gain curves in TYVER=1 and GCVER=1 (and BLVER=1).
      It is a VERY good idea to run TASAV before you do this to keep the
      original TY, GC, BL and also FG tables number 1 in case of disasters.
      After you loaded the data with VLBALOAD and put the external calibration
      and flagging information for the non-VLBA antennas in the appropriate
      tables number one, start the VLBAPIPEline with DOALL=0 and OUTNAME and
      OUTDISK pointing to the file VLBALOAD created (thus not INNAME, INDISK).

      Make sure there is enough disk space available on OUTDISK, about 4 to
      5 times the expected compressed UVDATA set. For spectral line probably
      you need much more because it will make spectral line image cubes.

DOALL is supposed to set the starting point for the procedure. A positive
      number is used to start with loading the data from tape, and using
      INTAPE=DOALL for the tape drive. A DOALL=0 means, the data is loaded
      to disk (and has name 'OUTNAME'.UVDATA.1 on 'OUTDISK' -see below), but
      nothing else has been done to it. DOALL=-1, -2, -3, etc is used to
      restart at frequency-ID 1, 2, 3, etc. and continue to higher numbers,
      if they exist.  The files 'OUTNAME'.FQ-(i).1 or 'OUTNAME'.FPOL(i).1
      (for a single freq. UVDATA or FXPOL) should exist!  Restarting at a
      given point may be fragile and may break the procedure.
* testing:    If < 1000, then ONLY do Freq-ID = (-1*DOALL)-1000; ie. -1004->4

NCOUNT is the number of distribution tapes to be loaded.  The number of
      files per tape is specified in APARM.

APARM is the number of files to be loaded from each distribution tape
      - files at the beginning of a tape cannot be skipped.  For more
      control use VLBALOAD.  Make sure that APARM lists the number of
      files per tape in the right order ;-), that the unused APARMs are
      set to zero, and the tapes are loaded into the tape drive in the
      correct order.

DOUVCOMP determines whether all UV data will be compressed (DOUVCOMP=1),
      not compressed at all (DOUVCOMP=-1), or whether you want the UV data
      loaded in uncompressed multi-source form with FITLD, and compressed
      in the calibrated single source files after SPLIT (using DOUVCOMP=0).

OUTNAME is used to create files (e.g. from tape), AND to search for as an
      INNAME for files that are already on disk (which will be OUTDISK).
      OUTNAME='VLBA-experiment name' (eg BZ99Z) is a good unique name to
      use.  All final calibrated single source UV files will use the
      first 6 characters of OUTNAME as their OUTCLASS, so for multiple
      passes make OUTNAME unique in the first six characters.

OUTDISK is the disk that is used for ALL files produced by the procedure.
      It may use other disks for scratch, which can be regulated with the
      BADDISK parameter below.  OUTDISK is the working disk; it needs to
      be spacey - clean up if needed (before you start vlbapipe).

OPTYPE should reflect the type of observation and requested output files:
      'CONT' for plain continuum observations using the PC (pulse-cal)
             table calibration; all data is averaged over IFs and channels
             and continuum images are made from the averaged data.
      'PSEU' for pseudo-continuum observations using the PC table calibration
             the data is NOT averaged over IFs nor over channels, but the
             resulting images are continuum images - used for large FOVs
      'SPEC' for spectral line observations - there should not be a PC-table!
             Manual pulse-cal is performed on the bandpass calibrator in the
             timerange given (see below). Data is unaveraged and images are
             spectral line cubes of the phase referenced target sources ONLY.
             (see below in CALSOUR for strong spectral line sources that can
              be fringed - essentially you have to make your own cube later)
             Calibrator sources will be made continuum images even in 'SPEC'!
      'LINE' is the same as 'SPEC'.
      **********************************************************************
      NOTE that the imaging is only done with the intention to check whether
           the calibration has worked and whether maybe some more data needs
           to be flagged (in FG#1). Always re-image your sources for your
           scientific use (imaging, analysis) - these won't be good enough!
      **********************************************************************

CLINT is the CL table interval (minutes) used throughout the whole experiment
      For VLBA data you would generally choose a CLINT between 0.1-2 minutes.

REFANT or SORT is used to select the reference antenna. Because the antenna
      number for REFANT is usually unknown before loading, SORT can be used
      to supply the two letter code; e.g. SORT='PT' for Pie Town.  Use
      either REFANT or SORT, not both! Set REFANT to zero if you are using
      SORT; blank SORT if using REFANT.

SORT  Use either REFANT or SORT, not both; see REFANT above.

INFILE The name of the file containing a model for the total electron content
      (TEC) in the ionosphere which is used by TECOR to correct delays caused
      by the ionosphere.  It is recommended to run TECOR for ALL frequencies
      up to 15 GHz.  The filename should be given in the usual AIPS
      style as a logical directory name followed by a colon and the name of
      the file. The entire name will be converted to upper-case. The file
      should contain TEC maps spanning the time range covered by the input CL
      table and should be in IONEX format.  If NFILES > 1 then the name MUST
      be in standard format CCCCdddC.yyC where C can be any character, ddd
      is the day number, and yy is the year.  Also the name given in INFILE
      must be the first day.
      The IONEX/TECOR files can be downloaded through an anonymous ftp to
      cddisa.gsfc.nasa.gov and following the directory structure:
      /gps/products/ionex/<year>/<day-of-year> (take doy from your *.sum).
      Also note that if your observation starts before 01:00 UT, then you
      need the IONEX/TECOR for the preceding day too, and point INFILE to
      this first file. For observations ending after 23:00 UT, you need the
      next day's file as well, and in both cases add one to NFILES. TECOR is
      not run and INFILE is ignored for NFILES < 1, e.g. for 43 GHz data.
      For more details, see EXPLAIN TECOR.

NFILES is the number of IONEX files to use, starting at INFILE.  Note that
      if NFILES > 1 then the INFILE must be in a standard format and the
      file in INFILE must be the first day of those to be loaded.

TIMERANG is a time range on which ALL antennas have good data on the bandpass
      calibrator (see below, CALSOUR(1)). This interval is used for pulse-cal,
      either with the PC table, or manually if there is none (as in spectral
      line data). For multiple frequency data it is VERY nice to have
      TIMERANGE to span the whole time range in which you flip through all
      the observed frequencies - otherwise the procedure will choke.
      Alternatively, if you know the scan number, you can use the scan number
      in TIMERANGE=66,0 to get the time range associated with scan 66.

CALSOUR is the parameter to name all the sources you want to be fringe fitted
      e.g. your calibrator sources, strong (target), or survey sources. The
      first source MUST BE named, and will be used for pulse-cal and bandpass
      calibration. Use a simple source with good signal to noise and with all
      antennas present (note: phase-reference calibrators may be good too)
      in the TIMERANGE specified above. If you want all sources fringe fitted
      then you may use CALSOUR(2)='*' but remember to also specify CALSOUR(1)
      to get the pulse-cal and bandpass calibration. If you want to attempt a
      self-cal for stronger sources, CALSOUR(2)='*' is not allowed, so you
      have to specify your calibrator sources (up to 30). The source names
      that start with an additional minus will be self-cal'd, but be warned
      that this is a frequent point of failure which may break the procedure.
      The sources that are fringe fitted will be imaged in (pseudo) continuum
      mode, so if your spectral line target is strong (e.g. when looking for
      absorption) you will only get a continuum image (not UV data set) which
      is used to asses the calibration - Re-image for the spectral line cube.

SOURCES holds the phase-referencing scheme in PAIRS with the phase-reference
      calibrator in the odd index. Each phase-reference target is in an even
      index, and calibrated with the preceding phase-reference calibrator.
      There is a maximum of 15 pairs, a total of 30 source fields.
      If your are not phase-referencing any source, then leave SOURCES blank.
      Do not repeat sources that are in CALSOUR that are not actually used in
      the phase-referencing scheme. If all targets are to be referenced to
      only one source, use SOURCES = 'PRCAL', '*' to get all non-calibrators
      phase referenced to 'PRCAL' - sources in CALSOUR are left alone, as is.
      E.g. SOURCES='PRCAL1','TARG_1','PRCAL2','TARG_2','PRCAL1','TARG_3' will
      reference target source 'TARG_1' and 'TARG_3' to phase-reference source
      'PRCAL1' and target source 'TARG_2' will be phase-referenced to source
      'PRCAL2'.  All phase reference sources 'PRCAL1', etc MUST also be
      present in the CALSOUR list, so they will be fringe fitted.

SOLINT or BPARM is used to specify the length of the fringe fit intervals.
      They should be given in minutes. If SOLINT > 0, the whole experiment
      uses the same fringe fit interval, regardless of frequency-ID. If you
      set SOLINT=-1 then you may choose different fringe fit intervals for
      each frequency-ID (e.g. 3 minutes for 5 GHz and 0.5 minutes for 43 GHz)
      with BPARM. But you need to know the order (usually order of observing)
      and you need to specify a number (maximum 10 values) for each separate
      frequency-ID in BPARM. SOLINT is also used in the self-cal of CALSOUR.

BPARM Use either SOLINT or BPARM, not both; see SOLINT above.

INTERPOL is the interpolation all sources. The options are '2PT', 'SIMP',
      'AMBG', or 'CUBE' as defined by CLCAL.

*****************************************************************************
 The procedure will also image all the sources, but note that the images are
 produced in order to check the calibration. By no means, are the choices
 made in the imaging stage represent the best possible imaging that could
 be performed with this data.  Those selections should be made by the
 individual observer, in pursuit or their own scientific goals.  The images
 should allow you to get an idea about the success of the calibration, and
 to find data that needs flagging before restarting it.
*****************************************************************************

IMSIZE is used to define the output image sizes.  You will want a big field
      for your phase-referenced target sources with a minimum IMSIZE of 512
      pixels and up to 8192. A larger image is slower, but accounts for
      possible position inaccuracies if it is known to less accuracy than
      500/Frequency(GHz) mas.  Sources listed in CALSOUR are automatically
      imaged with a size of 128 by 128 unless FACTOR is set.
      Pixels automatically scale with frequency: 2.25/Frequency(GHz) mas.

FACTOR can increase the image size of CALSOURs. It is assumed that your
      calibrators are well-known, so they do not need to be imaged in more
      than 128 pixels.  However you may want to increase this for complicated
      sources.  If FACTOR is used then the resulting image size for the
      sources listed in CALSOUR is FACTOR*128.  The maximum FACTOR is 64.

DOTV is used to direct all the 'instructive' plots that the procedure
     makes to show you how the calibration is progressing. A DOTV < 0 value,
     which is highly DISCOURAGED, will divert plots directly to the printer.
     A positive value for DOTV send the plots to the TV and thus is
     interactive.  This can be slow and the user should be present the
     entire time the procedure is running.  DOTV=0 is recommended, this makes
     PLot tables. Afterwards all the PLot tables can be cycled through with
     the procedure P_ALLPLOT, maybe P_ALLPRINT, which are defined in the
     VLBAPIPE runfile.

BADDISK is the well-known parameter specifying the disks to avoid for scratch.

$---------------------------------------------------------------------------
VLBAPIPE:           Procedure that attempts to calibrate VLBA data blindly.
Documentor:         Lorant Sjouwerman, lsjouwerman@aoc.nrao.edu.
Related Programs:   All VLBAUTIL procs, TECOR, BPASS, UVMLN, SPLIT, IMAGR,..
                                        CLIPM, POSSM, VPLOT and many more

VLBAPIPE applies VLBAUTIL procedures to simple VLBA-only experiments, with
simple flagging and Ionospheric Total Electron Content correction before
bandpass calibration.  Sources are split after calibration, and strong and
phase-reference sources are self-called and target sources are imaged as a
first order quality check.

    ====================================================================
    READ THE HELP/EXPLAIN FILE THOROUGHLY BEFORE USING SINCE NONE OF THE
                 AIPS INPUTS ARE USED IN THE USUAL WAY.
    ====================================================================

----------------------------------------------------------------------------
