; PFT ;--------------------------------------------------------------- ;! The Perley-Feigelson Test; see PFTLOAD.RUN, PFTEXEC.RUN ;# PROCEDURE OBSOLETE ;----------------------------------------------------------------------- ;; Copyright (C) 1995 ;; Associated Universities, Inc. Washington DC, USA. ;; ;; This program is free software; you can redistribute it and/or ;; modify it under the terms of the GNU General Public License as ;; published by the Free Software Foundation; either version 2 of ;; the License, or (at your option) any later version. ;; ;; This program is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; ;; You should have received a copy of the GNU General Public ;; License along with this program; if not, write to the Free ;; Software Foundation, Inc., 675 Massachusetts Ave, Cambridge, ;; MA 02139, USA. ;; ;; Correspondence concerning AIPS should be addressed as follows: ;; Internet email: aipsmail@nrao.edu. ;; Postal address: AIPS Project Office ;; National Radio Astronomy Observatory ;; 520 Edgemont Road ;; Charlottesville, VA 22903-2475 USA ;----------------------------------------------------------------------- ;--------------------------------------------------------------- PFT LLLLLLLLLLLLUUUUUUUUUUUU CCCCCCCCCCCCCCCCCCCCCCCCCCCCC PFT : The Perley-Feigelson Test; see PFTLOAD.RUN, PFTEXEC.RUN TCODE INIT,TEST,READ, or WRIT TMODE T or M TMASK 1.0 127.0 test selection bit mask INNAME OUTNAME INDISK 1.0 9.0 Input disk drive #. IN2DISK 1.0 9.0 Input disk drive #. OUTDISK 1.0 9.0 Output image disk drive # INTAPE 1.0 9.0 Input tape drive #. OUTTAPE 1.0 9.0 Output image tape drive # BADDISK (1,3 good on CVAX) VERSION ---------------------------------------------------------------- PFT Type: RUN file. Use: PFT performs the Perley-Feigelson Test of an AIPS system. See RUN files PFTLOAD.RUN and PFTEXEC.RUN. The purpose of the inputs file is to perform bounds checks on the parameters and to enable the use of TPUT and TGET. Adverbs: TCODE String*4 INIT,TEST,READ, or WRIT TMODE String*1 T or M TMASK Scalar test selection bit mask (1-127) INNAME String*12 OUTNAME String*6 INDISK Scalar Input disk drive #. (1-9) IN2DISK Scalar Input disk drive #. (1-9) OUTDISK Scalar Output image disk drive # (1-9) INTAPE Scalar Input tape drive #. (1-9) OUTTAPE Scalar Output image tape drive # (1-9) BADDISK Array (1,3 good on CVAX) VERSION String*40 OLD,NEW,OLDPSAP,NEWPSAP,LOCAL (also private dirs; use | to separate) PFT runs in four different modes: (TCODE) INIT = starts from the original master UV dataset, and (re)computes master files (does ALLDEST). The "mask" TMASK controls which steps execute. TEST = re-does the INIT computation but stores the new files under different names and compares against masters. The "mask" TMASK controls which steps execute. TEST steps use either the TEST run output of the previous step (TMODE=T) or the INIT run output (TMODE=M). WRIT = write out all master files to tape. READ = read in all master files from tape. A binary mask controls which steps of the test execute: TMASK +1=>SORT, +2=>UVMAP, +4=>APCLN, +8=>ASCAL, +16=>MXmap, +32=>MXclean, +64=>VM For example, TMASK=54 would execute UVMAP,APCLN,MXmap,and MXclean. Input data files for test steps are selected by the TMODE adverb which has two possible values: 'T' or 'M'. ---------------------------------------------------------------- Explanation of PFT: (DCW, Feb/Mar85) PFT exercises the "Dirty Dozen" of AIPS tasks using D-array 20cm data of Virgo-A which was acquired in June 1983 by Rick Perley. This data set was used by Eric Feigelson in 1984 as a test case for the PSAIPS project. The PFT problem is intended to be used to certify the correct execution of all AIPS systems. (It is small enough to run in less than one day on all machines which will ever run AIPS.) Its execution time is significant and easily measurable on VAX+AP systems, and approximate timings can also be obtained for machines 10-100x faster, like the Cray. In order to properly test the performance of these machines which are much faster than a VAX+AP, we will need to construct clones of PFT with datasets having sizes of the order of 10x and 100x larger than PFT. ============================================================= Steps to execute the Perley-Feigelson Test on a new computer: 1. Read the first three files from the tape to the appropriate directories of the system. These files contain 80-character ASCII records (1 record per block); the three files are separated by tapemarks. The first file is this HELP file; it should be copied to HLPFIL:PFT.HLP. The second and third files are the RUN files which perform the test; they should be copied to RUNFIL:PFTLOAD.RUN and RUNFIL:PFTEXEC.RUN, respectively. Techniques for copying card image files from tape to disk vary from system to system. A collection of recipes appear in a later section of this file. 2. Log in to AIPS under the user number which will own the data files and the SAVE/GET files associated with the test. It is probably convenient to clear your AIPS message file before proceeding: PRNUM=-1;CLRMSG (or else PRTMSG) 3. Now do the POPS command "RUN PFTLOAD". This initiates the compilation process, which can take as long as 15 minutes on a heavily loaded VAX-11/780. You may want to execute the POPS command "PRTMSG" afterwards to document the compilation of the test procedures. 4. The next step is to read the data files from the tape. These are the 4th thru Nth files. The first one (the 4th physical file on the tape) is the original visibility data set of the PFT. Subsequent files are the "master" versions of files computed from the original dataset. The read process consists of the following steps: CLRMSG (prepare for tape read steps) GET PFTINIT Specify INDISK, IN2DISK, OUTDISK, INTAPE, and OUTTAPE. Also specify BADDISK as needed TCODE='READ'; TPUT PFT (TPUT saves adverb values) MOUNT the tape RUN PFTEXEC DISMOUNT the tape PRTMSG (documents tape read process) The READ mode of PFT will rewind the tape and will skip over the three text files. It will then execute UVLOD and IMLOD in the appropriate pattern to read the UV and map files to disk. At the end of the process it rewinds the tape and prints a directory of the disk files. NOTE: PFTEXEC will insist on having non-default values for INTAPE, OUTTAPE, INDISK, INDISK2, and OUTDISK. 5. Now it is time to actually perform the test. The 'TEST' mode recomputes the various files. All map results are compared to the master versions which were read from tape in the previous step. There are two ways to perform the calculations: with each step computed from the result of the previous test calculation in the sequence, or else from the result of the previous MASTER computation in the sequence. The first mode (TMODE='T') tests how well the system under test is able to reproduce a complex sequence of calculations beginning from a designated starting point. That is, it measures the accumulation of error in the system. The second mode (TMODE='M') tests the steps of the sequence separately so that errors created in each step can be isolated. Both tests are important, and both should be performed to fully certify the scientific adequacy of an AIPS installation, but it is generally best to use TMODE='M' as the initial (and primary) test of all systems. The steps are: CLRMSG (prepare for test results) GET PFTINIT TGET PFT TCODE='TEST';TMODE='M';TMASK=127 TPUT PFT (saves the adverb values) RUN PFTEXEC PRTMSG (print out the test results) The message file from PFTEXEC will be more than 2000 lines long. It is intended to be the major item documenting the test procedures and results. Making a second run with TMODE='T' is generally advisable to evaluate the accumulation of error. This process also checks the manipulations of visibilities, antenna files, and gain files; these processes are not really tested in the 'M' mode because AIPS currently has no comparison operators for files of these types. The 'T' mode tests them by their effect on subsequent map results. One can also rerun the tests with a different value for the VERSION adverb. Generally this is most interesting in the comparison of the NEWPSAP library with the AP library. One can also use this option to test experimental versions of specific tasks. 6. It is also possible to test task FITTP. The basic idea is to run PFT in the 'WRIT' mode to produce a tape, and then to read that tape on an AIPS system to verify that it yields correct results. Note that the combination of the 'READ' and 'TEST' modes effectively verify the correctness of IMLOD and UVLOD on the current system. It follows that generating a new tape with the 'WRIT' mode and then executing 'READ' and 'TEST' on it will effectively test task FITTP. Better yet, take the tape to another AIPS and run PFT on it! This verifies the intercommunication of the systems. The steps are: CLRMSG GET PFTINIT TGET PFT TCODE='WRIT';TMODE='T 7. Finally, one wants to know the accounting statistics from the test runs. The accounting listing can be correlated with the message file listings in order to analyze the performance of the tested system. It would be nice if all AIPS tasks would put the accounting information into the message file when they exit, but currently they don't. See the PRTACC discussion below to find out how to produce the desired listing. ============================================================= How to list the accounting data after PFT runs (DCW, 19Mar85) The accounting data is in the 'AC' file. It is listed by a utility program called PRTACC, whose executable image is in the "LOAD" directory (e.g. UMA0:[AIPS.15APR85.LOAD] on the CVAX machine at this time). For VMS the operations are: RUN LOAD:PRTACC (using logical symbol LOAD) SOPT (PRTACC opcode; must be upper case!) (blanks ==> show all tasks) 0 (0 ==> show all POPS #'s) xxxxyyyy (range of user #'s, can be xxxxxxxx) 850101 (starting date yymmdd) 0. (minimum CPU time) TFFF (flags, selects simple listing) GO (prepare the commanded listing) QUIT (dispose listing to printer & quit) The listing should come out at this point. Note that PRTACC has exclusive access to the accounting file until the QUIT is issued, and so various tasks may complain while they are waiting to get access to the file. The listing produced by PRTACC will have one line for each execution of any task or program AIPS. The columns of the listing give the taskname, POPS#, user#, start-time, real-time, CPU-time, and "IOCNT". Asterisks mark any run that aborted. PRTACC is a machine independent program; this means it can be run on all AIPS systems and will accept the same user responses shown above, although the "RUN" command will be different. The IOCNT field of the listing does not have the same meaning on all machines. ============================================================ Regarding "cloning": It is obvious that PFTLOAD and PFTEXEC are cloneable. First, a clone of PFT could process another dataset; this would be done by changing only a few adverb values here and there. Second, the general pattern of executing and testing the tasks in succession will work for a wide variety of AIPS tasks and data types. The first step of the clone process is to substitute a new test name string for "PFT" everywhere in both PFTLOAD and PFTEXEC. ============================================================== Interpretation of Test Results This section will be added later and will discuss issues of numerical precision. It will also address various issues associated with benchmarking. We need some beta-testing before this section can be constructed.