; VLBAARCH ;--------------------------------------------------------------- ;! Procedure to archive VLBA correlator data ;# RUN VLBI UTILITY ;----------------------------------------------------------------------- ;; Copyright (C) 2000-2004, 2008 ;; Associated Universities, Inc. Washington DC, USA. ;; ;; This program is free software; you can redistribute it and/or ;; modify it under the terms of the GNU General Public License as ;; published by the Free Software Foundation; either version 2 of ;; the License, or (at your option) any later version. ;; ;; This program is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; ;; You should have received a copy of the GNU General Public ;; License along with this program; if not, write to the Free ;; Software Foundation, Inc., 675 Massachusetts Ave, Cambridge, ;; MA 02139, USA. ;; ;; Correspondence concerning AIPS should be addressed as follows: ;; Internet email: aipsmail@nrao.edu. ;; Postal address: AIPS Project Office ;; National Radio Astronomy Observatory ;; 520 Edgemont Road ;; Charlottesville, VA 22903-2475 USA ;----------------------------------------------------------------------- VLBAARCH LLLLLLLLLLLLUUUUUUUUUUUU CCCCCCCCCCCCCCCCCCCCCCCCCCCCC VLBAARCH: Procedure to archive VLBA correlator data. INTAPE 0.0 9.0 Tape drive #. Use 0 for disk! DATAIN Disk file name, jobs location -> specify the name FORMAT FORMAT 0.0 1.0 Disk file name format: 0 -> job number + 0001.FITS 1 -> FITLD style, which is [:][#] NCOUNT 0.0 9999.0 Number of VLBA tapes to load OR number of disk files to load -> then leave APARM=0 APARM No of FILES PER TAPE to load NFILES 0.0 9999.0 Number of files to skip, e.g. 519 to start @ 5200001.FITS ONLY for disk or FIRST TAPE OUTNAME Output filename, 9 characters Use - or multiple passes : [ pass#] e.g., "BX999Z P1" (= 9 char) OUTDISK 0.0 15.0 Disk drive with ample space SUBARRAY 0.0 10.0 Set <> 0 ONLY if you do have subarrays, otherwise leave 0 set subarray to #>=2 of subs BANDPOL For non-standard polarization * USE ONLY FOR ODD SETUPS * DATAOUT Archive directory or logical: VLBAARCH combines VLBALOAD, VLBAFIX and FITAB @01JAN04 To use these procedures type 'RUN VLBAARCH'. For disk space requirements see 'EXPLAIN VLBAARCH' Do NOT run simultaneous VLBAARCH's in the same userno! ---------------------------------------------------------------- VLBAARCH VLBAARCH: Procedure to archive VLBA correlator data. Type: Procedure Use: VLBAARCH is a procedure that loads VLBA data with weights above 0.1 (versus 0.7 for normal VLBALOAD) from tape or disk and may be used for any data from the VLBA correlator VLBAARCH will perform VLBAFIX, VLBAFPOL and use FITAB to write the output to FITS disk files. It clears the catalog (i.e. deletes or 'zaps' files that are not needed anymore) Type RUN VLBAARCH (after VERSION 'MYAREA' where MYAREA is a defined logical where both VLBAARCH.001 and VLBAARCH.HLP reside) to make the VLBAARCH procedure available. Note that VLBAARCH permanently flags data with weights less than 0.7 (which is fine for almost all data sets), does not apply any compression, attaches a CL table with a default interval of 15 seconds (0.25 minutes), and now is THE DEFAULT PROCEDURE TO ARCHIVE VLBA USER DATA FROM JAN04 VLBAARCH redefines ordinary VLBALOAD, VLBAFIX and VLBAFPOL To use VLBAUTIL's VLBALOAD, VLBAFIX and VLBAFPOL you have to restore them with 'RUN VLBAUTIL' (and vice versa) again Documentor: Lorant Sjouwerman, NRAO - version 1: 01JAN04 Related Programs: VLBAUTIL (VLBALOAD, VLBAFIX, VLBAFPOL) FITAB Adverbs: INTAPE......The input tape drive number. Use 0 for disk files. Do not mount the tape, but do let it settle for a while after loading the tape. It will be ejected when the procedure has ended - if it does not come out by it self, something went wrong in the script Multiple tapes (NCOUNT>1): archiving starts after loading the first batch of tapes, ejecting each after it has finished loading it and requiring you to load the next tape until archiving starts after loading (and ends with ejecting) the last tape. Use DATAIN and FORMAT for disk files - also see NFILES below DATAIN......Name of the disk file or the logical or directory where the indicvidual correlator jobs can be found (for jobs, DATAIN uses the standardized job names) FORMAT......Switch to select the disk file name convention. NCOUNT......The number of tapes to read (use APARM) or the number of files to read from disk. APARM.......Leave 0 for disk loads - else this is the number of files to read in order for each of the tapes. Read the number of files on a VLBA correlator tape from the listing inserted in the tape case(s). NFILES......Number of files to skip on the tape or disk before reading data. For example, 1 => start of next file, -1 => start of previous file, 0 => no tape movement at all, i.e., start where the tape is now. If you have more than one tape NFILES ONLY APPLIES to the first tape - all remaining will use NFILES = 0 ! (continue at the start of the next tape is probably what you want when loading VLBA distribution tapes) NFILES is also used for disk files, in particular for disk files named after job numbers. E.g, use NFILES = 519 to start with job number 520, in which case the file will be named 5200001.FITS. NCOUNT>1 will result in the next job number (521) to be read if FORMAT is set to zero (0). If FORMAT is non-zero then disk file names assume the FITLD default, i.e. a literal file name to which a number is appended. See DATAIN in EXPLAIN FITLD for more on this. OUTNAME.....The name for the output files, usually the EXP-code and 'EXP-code P1' etc for multiple passes. The name must be 9 characters or less to avoid overflow. In the past, before Aug04, longer names such as a full 'EXP-code PASS1' etc have been used, but we needed more characters in the description to distinguish files/frequency-id that are very close in frequency OUTDISK.....The disk drive number for the output data. Choose a disk with a large amount of space for processing. If TkB is the total number of kilobytes from all distribution tapes or disk files, then you will use about 1.5*TkB for a single frequency file with the correct polarization labeling (single or all polzn) and about 3-4*TkB for multi-frequency, dual polzn. In the latter case, VLBAARCH will use other disks, i.e. IN/OUTDISK parameters are set to zero for the used archive VLBAFIX/FPOL/FITS(=FITAB) procedures. SUBARRAY....Set to number of subarrays if more than one - one (sub)array is not a subarray -> leave 0 for one. BANDPOL.....Usually BANDPOL is determined correctly, but if you you know it is going to be wrong, specify it here. These are usually very odd cases, e.g., 6IFs RLLLRR and BANDPOL is only used in case the script cannot determine it - so it is safe to use in a data set with multiple determinable BANDPOLs and one odd one DATAOUT.....To specify where the files should go (usually just a logical, e.g., DA01: for the first data disk). All output names should be unique if configured as: DATAOUT=OUTNAME+_#ST#IF+BW[K/M]HZ#CH+FREQ+GHZ+ID, where #ST stands for number of 'STOKES' parameters for unknown, or #POL stands for total polarization (R=RR, L=LL, H=half, ie RR+LL, F=full, X=XX, Y=YY) #IF=number of IFs times BW=the bandwidth per IF in KHZ or MHZ, #CH is number of channels per IF, and FREQ is the frequency at the reference pixel in GHz. Furthermore there is a freq-id/sequence appended to DATAOUT. When TkB is the total number of kBytes from all distribution tapes or disk files, then you will need about 1.5*TkB here for all archive files. Note: if DATAOUT is on OUTDISK you need ~5*TkB max. The total DATAOUT length should be <=48 characters, i.e., DATAOUT given should be 4 char +":" or less. Thus DATAOUT='HOME:' and 'DA01:' etc are OK options WHEN TO USE VLBAARCH VLBAARCH loads the data from one or more VLBA export tapes or disk files uncompressed into AIPS, it splits off the different frequencies, possibly corrects for polarization labeling and writes out the FITS files uncompressed to disk with standard names. When it ends successfully, no trace will be left on the AIPS disks. This is not a calibration pipeline but a standard way to archive VLBA data in user friendly and recognizable disk files. Do not use VLBAARCH if you need to apply data selection criteria or otherwise need more control over e.g. FITLD. For this redefine the VLBAUTIL procedures with RUN VLBAUTIL (see 'EXPLAIN VLBAUTIL') or use the individual AIPS tasks. VLBAARCH "step-by-step" : (see also 'EXPLAIN VLBAUTIL') > load all the raw correlator data from tape or disk uncompressed into AIPS using VLBALOAD (only keep weights > 0.7; may create more than one file) - merge GC, TY, and PC tables for each of the files if needed - split out dual frequency observed S/X and 90/50 data sets if needed - split out different frequency-ID's (usually different observing bands) - reindex all files with INDXR using a CL-interval of 0.25 (15 seconds) > correct all data sets with VLBAFIX for sub-arrays, sort order, polarization - sort the data in the right TB sort order with MSORT if needed - arrange in subarrays using USUBA - correct for polarization labeling (VLBAFPOL) > write each of the data sets to an uncompressed disk fits file using FITAB - file names are usually like =?POL?IF?MHZ?CH_?GHZxx where ?POL = F(ull)POL, H(alf)POL (ie RR+LL), RPOL or LPOL (or # STokes) where ?IF?MHZ = # IF's per POL, each ?MHZ wide, devided in # CH channels and where ?GHZ is the observing frequency of this frequency-ID (xx) file [older versions (from jun03-dec03) have .FQ-#, .FXPOL, etc.] > remove all the data from the AIPS disks (and cross your fingers!) Then the files should appear in NRAO's (VLBA) data archive as one or more multi-source, single frequency-ID FITS files. You can grab these if you can (ie, they are public or you know your project key), and they should be okay for you to start calibrating (eg with VLBACALA, VLBAPIPE or ACCOR), presuming you have examined your data and considered editing bad data before starting. The exact data handling one can get from the HIstory file (using AIPS' PRTHI). ----------------------------------------------------------------