; VLBAARCH
;---------------------------------------------------------------
;! Procedure to archive VLBA correlator data
;# RUN VLBI UTILITY
;-----------------------------------------------------------------------
;;  Copyright (C) 2000-2004, 2008
;;  Associated Universities, Inc. Washington DC, USA.
;;
;;  This program is free software; you can redistribute it and/or
;;  modify it under the terms of the GNU General Public License as
;;  published by the Free Software Foundation; either version 2 of
;;  the License, or (at your option) any later version.
;;
;;  This program is distributed in the hope that it will be useful,
;;  but WITHOUT ANY WARRANTY; without even the implied warranty of
;;  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;;  GNU General Public License for more details.
;;
;;  You should have received a copy of the GNU General Public
;;  License along with this program; if not, write to the Free
;;  Software Foundation, Inc., 675 Massachusetts Ave, Cambridge,
;;  MA 02139, USA.
;;
;;  Correspondence concerning AIPS should be addressed as follows:
;;         Internet email: aipsmail@nrao.edu.
;;         Postal address: AIPS Project Office
;;                         National Radio Astronomy Observatory
;;                         520 Edgemont Road
;;                         Charlottesville, VA 22903-2475 USA
;-----------------------------------------------------------------------
VLBAARCH  LLLLLLLLLLLLUUUUUUUUUUUU CCCCCCCCCCCCCCCCCCCCCCCCCCCCC
VLBAARCH: Procedure to archive VLBA correlator data.
INTAPE          0.0         9.0    Tape drive #. Use 0 for disk!
DATAIN                             Disk file name, jobs location
                                      -> specify the name FORMAT
FORMAT          0.0         1.0    Disk file name format:
                                    0 -> job number + 0001.FITS
                                    1 -> FITLD style, which is
                                      [<logical>:]<file name>[#]
NCOUNT          0.0      9999.0    Number of VLBA tapes to load
                                     OR number of disk files to
                                     load -> then leave APARM=0
APARM                              No of FILES PER TAPE to load
NFILES          0.0      9999.0    Number of files to skip, e.g.
                                     519 to start @ 5200001.FITS
                                     ONLY for disk or FIRST TAPE
OUTNAME                            Output filename, 9 characters
                                    Use <exp name> - or multiple
                                    passes : <exp name>[ pass#]
                                    e.g., "BX999Z P1" (= 9 char)
OUTDISK         0.0        15.0    Disk drive with ample space
SUBARRAY        0.0        10.0    Set <> 0 ONLY if you do have
                                    subarrays, otherwise leave 0
                                    set subarray to #>=2 of subs
BANDPOL                            For non-standard polarization
                                    * USE ONLY FOR ODD SETUPS *
DATAOUT                            Archive directory or logical:

          VLBAARCH combines VLBALOAD, VLBAFIX and FITAB @01JAN04
          To use these procedures type 'RUN VLBAARCH'.
          For disk space requirements see 'EXPLAIN VLBAARCH'
          Do NOT run simultaneous VLBAARCH's in the same userno!

----------------------------------------------------------------
VLBAARCH
VLBAARCH:           Procedure to archive VLBA correlator data.

Type: Procedure
Use:  VLBAARCH is a procedure that loads VLBA data with weights
      above 0.1 (versus 0.7 for normal VLBALOAD) from tape or
      disk and may be used for any data from the VLBA correlator
      VLBAARCH will perform VLBAFIX, VLBAFPOL and use FITAB to
      write the output to FITS disk files. It clears the catalog
      (i.e. deletes or 'zaps' files that are not needed anymore)
      Type RUN VLBAARCH (after VERSION 'MYAREA' where MYAREA is
      a defined logical where both VLBAARCH.001 and VLBAARCH.HLP
      reside) to make the VLBAARCH procedure available.
      Note that VLBAARCH permanently flags data with weights
      less than 0.7 (which is fine for almost all data sets),
      does not apply any compression, attaches a CL table with a
      default interval of 15 seconds (0.25 minutes), and now is

      THE DEFAULT PROCEDURE TO ARCHIVE VLBA USER DATA FROM JAN04

      VLBAARCH redefines ordinary VLBALOAD, VLBAFIX and VLBAFPOL
      To use VLBAUTIL's VLBALOAD, VLBAFIX and VLBAFPOL you have
      to restore them with 'RUN VLBAUTIL' (and vice versa) again

Documentor:         Lorant Sjouwerman, NRAO - version 1: 01JAN04
Related Programs:   VLBAUTIL (VLBALOAD, VLBAFIX, VLBAFPOL) FITAB

Adverbs:
  INTAPE......The input tape drive number. Use 0 for disk files.
              Do not mount the tape, but do let it settle for a
              while after loading the tape. It will be ejected
              when the procedure has ended - if it does not come
              out by it self, something went wrong in the script
              Multiple tapes (NCOUNT>1): archiving starts after
              loading the first batch of tapes, ejecting each
              after it has finished loading it and requiring you
              to load the next tape until archiving starts after
              loading (and ends with ejecting) the last tape.

  Use DATAIN and FORMAT for disk files - also see NFILES below

  DATAIN......Name of the disk file or the logical or directory
              where the indicvidual correlator jobs can be found
              (for jobs, DATAIN uses the standardized job names)

  FORMAT......Switch to select the disk file name convention.

  NCOUNT......The number of tapes to read (use APARM) or the
              number of files to read from disk.

  APARM.......Leave 0 for disk loads - else this is the number
              of files to read in order for each of the tapes.
              Read the number of files on a VLBA correlator tape
              from the listing inserted in the tape case(s).

  NFILES......Number of files to skip on the tape or disk before
              reading data. For example, 1 => start of next file,
              -1 => start of previous file, 0 => no tape movement
              at all, i.e., start where the tape is now. If you
              have more than one tape NFILES ONLY APPLIES to the
              first tape - all remaining will use NFILES = 0 !
              (continue at the start of the next tape is probably
              what you want when loading VLBA distribution tapes)

              NFILES is also used for disk files, in particular
              for disk files named after job numbers. E.g, use
              NFILES = 519 to start with job number 520, in which
              case the file will be named 5200001.FITS. NCOUNT>1
              will result in the next job number (521) to be read
              if FORMAT is set to zero (0). If FORMAT is non-zero
              then disk file names assume the FITLD default, i.e.
              a literal file name to which a number is appended.
              See DATAIN in EXPLAIN FITLD for more on this.

  OUTNAME.....The name for the output files, usually the EXP-code
              and 'EXP-code P1' etc for multiple passes. The name
              must be 9 characters or less to avoid overflow. In
              the past, before Aug04, longer names such as a full
              'EXP-code PASS1' etc have been used, but we needed
              more characters in the description to distinguish
              files/frequency-id that are very close in frequency

  OUTDISK.....The disk drive number for the output data. Choose
              a disk with a large amount of space for processing.
              If TkB is the total number of kilobytes from all
              distribution tapes or disk files, then you will use
              about 1.5*TkB for a single frequency file with the
              correct polarization labeling (single or all polzn)
              and about 3-4*TkB for multi-frequency, dual polzn.
              In the latter case, VLBAARCH will use other disks,
              i.e. IN/OUTDISK parameters are set to zero for the
              used archive VLBAFIX/FPOL/FITS(=FITAB) procedures.

  SUBARRAY....Set to number of subarrays if more than one - one
              (sub)array is not a subarray -> leave 0 for one.

  BANDPOL.....Usually BANDPOL is determined correctly, but if you
              you know it is going to be wrong, specify it here.
              These are usually very odd cases, e.g., 6IFs RLLLRR
              and BANDPOL is only used in case the script cannot
              determine it - so it is safe to use in a data set
              with multiple determinable BANDPOLs and one odd one

  DATAOUT.....To specify where the files should go (usually just
              a logical, e.g., DA01: for the first data disk).
              All output names should be unique if configured as:
              DATAOUT=OUTNAME+_#ST#IF+BW[K/M]HZ#CH+FREQ+GHZ+ID,
              where #ST stands for number of 'STOKES' parameters
              for unknown, or #POL stands for total polarization
              (R=RR, L=LL, H=half, ie RR+LL, F=full, X=XX, Y=YY)
              #IF=number of IFs times BW=the bandwidth per IF in
              KHZ or MHZ, #CH is number of channels per IF, and
              FREQ is the frequency at the reference pixel in GHz.
              Furthermore there is a freq-id/sequence appended
              to DATAOUT. When TkB is the total number of kBytes
              from all distribution tapes or disk files, then you
              will need about 1.5*TkB here for all archive files.
              Note: if DATAOUT is on OUTDISK you need ~5*TkB max.
              The total DATAOUT length should be <=48 characters,
              i.e., DATAOUT given should be 4 char +":" or less.
              Thus DATAOUT='HOME:' and 'DA01:' etc are OK options


WHEN TO USE VLBAARCH

VLBAARCH loads the data from one or more VLBA export tapes or disk files
uncompressed into AIPS, it splits off the different frequencies, possibly
corrects for polarization labeling and writes out the FITS files uncompressed
to disk with standard names. When it ends successfully, no trace will be left
on the AIPS disks. This is not a calibration pipeline but a standard way to
archive VLBA data in user friendly and recognizable disk files.

Do not use VLBAARCH if you need to apply data selection criteria or otherwise
need more control over e.g. FITLD. For this redefine the VLBAUTIL procedures
with RUN VLBAUTIL (see 'EXPLAIN VLBAUTIL') or use the individual AIPS tasks.

VLBAARCH "step-by-step" : (see also 'EXPLAIN VLBAUTIL')
> load all the raw correlator data from tape or disk uncompressed into AIPS
  using VLBALOAD (only keep weights > 0.7; may create more than one file)
- merge GC, TY, and PC tables for each of the files if needed
- split out dual frequency observed S/X and 90/50 data sets if needed
- split out different frequency-ID's (usually different observing bands)
- reindex all files with INDXR using a CL-interval of 0.25 (15 seconds)
> correct all data sets with VLBAFIX for sub-arrays, sort order, polarization
- sort the data in the right TB sort order with MSORT if needed
- arrange in subarrays using USUBA
- correct for polarization labeling (VLBAFPOL)
> write each of the data sets to an uncompressed disk fits file using FITAB
- file names are usually like <exp-name[pass number]>=?POL?IF?MHZ?CH_?GHZxx
  where ?POL = F(ull)POL, H(alf)POL (ie RR+LL), RPOL or LPOL (or # STokes)
  where ?IF?MHZ = # IF's per POL, each ?MHZ wide, devided in # CH channels
  and where ?GHZ is the observing frequency of this frequency-ID (xx) file
  [older versions (from jun03-dec03) have <exp>.FQ-#, <exp>.FXPOL, etc.]
> remove all the data from the AIPS disks (and cross your fingers!)

Then the files should appear in NRAO's (VLBA) data archive as one or more
multi-source, single frequency-ID FITS files. You can grab these if you can
(ie, they are public or you know your project key), and they should be okay
for you to start calibrating (eg with VLBACALA, VLBAPIPE or ACCOR), presuming
you have examined your data and considered editing bad data before starting.
The exact data handling one can get from the HIstory file (using AIPS' PRTHI).

----------------------------------------------------------------
