[AIPS] [NRAO] AIPS Benchmarks
(Dirty Dozen Tasks)


Most recent change on $Date: 2013/12/06 16:41:35 $ (UT) [Y2K test]


Current Record AIPSMark(00)

320
(Xeon 2 dual 64-bit processor, Speed 3.0 GHz, Memory 4 Gb, Cache 4096 Kb, RAID0 file system;
31DEC05 compiled with g77 2.95 dynamic link on RedHat Enterprise at CfA)
313
(Xeon 2 dual 64-bit processor, Speed 3.0 GHz, Memory 2 Gb, Cache 4096 Kb, RAID0 file system;
31DEC06 compiled with Intel ifort on RedHat Enterprise at CfA)

NOTE: a new benchmark test, called Y2K, has been developed. It is an adjustment of the existing DDT with an altered set of tasks (UVMAP and MX replaced by IMAGR), and a 10-fold change in the size of the problems. Its goal is to take on the order of an hour to run on fairly new desktop hardware. However, it now takes 3:43 on a 3.06 GHz dual-cpu Intel Xeon with the Intel binary installation. It takes about 2:05 on the new record holder.

The Y2K and DDT (Dirty Dozen) suites are reliable ways of determining two things:

The new AIPS Memo 104 describes the new test and gives initial performance results. It was developed on the 31DEC00 version of AIPS. See the 31DEC05 and 31DEC06 pages for information on obtaining and maintaining your very own copy of the current releases. For more details on the earlier DDT, see AIPS Memo 85 which has the information on the last iteration of the DDT test. This was modified for the 15JAN94 release of AIPS (this is not the most recent version; see the AIPS Home Page for info on recent releases). There are other documents available too.

The AIPSMark00 is defined as:

Am(00) = (40000 ÷ TY2K(L))

where TY2K(L) is the total run time in seconds of the Y2K LARGE test. This run time was defined initially by Glendenning and Hunt as the time in seconds from RUN DDTEXEC through PRINTING ANSWERS, ERRORS, OTHER IMPORTANT MESSAGES). The multiplier 40000 was chosen to make the new AIPSMark00 about the same as the old AIPSMark93 on many machines.

The AIPSMark93 is defined as:

Am(93) = (4000 ÷ TDDT(L))

where TDDT(L) is the total run time in seconds for the LARGE DDT suite as defined by Glendenning and Hunt. Chris Flatters devised a formula for predicting a computer's AIPSMarks based on its measured SPECfp(95) and SPECint(95) performance. An extensive list of SPEC results is maintained on the Web by John DiMarco at the University of Toronto. Chris finds that the following relation gives a fair, ball-park estimate of the AIPSmark that can be expected on a machine with known SPEC(95) benchmarks.

AIPSMark93 = 0.898 × SPECfp(95) + 0.11 × SPECint(95) - 1.665

How do I run Y2K and DDT?

Install the latest version of AIPS; see for example 31DEC06. Start running AIPS and enter EXPLAIN Y2K and EXPLAIN DDT. These are available on the web for the impatient as Y2K.HLP and DDT.HLP. The process consists of several steps:

Where can I get the Y2K and DDT data files?

Don't panic; 31DEC10 versions of the Y2K and DDT data files are available (remember to right click for Save as or Save link as) via anonymous ftp. For manual use of ftp, the address for anonymous ftp is ftp.aoc.nrao.edu and the data files are located in directory pub/software/aips/FITS/31DEC10. The older Y2K data files are available (remember to right click for Save as or Save link as) via anonymous ftp and the older DDT data files are also available via anonymous ftp. But please look at the file sizes (use the ls -l or dir command in your ftp program) before pulling these across. Some of them are large: the aggregate size of all 33 DDT files is about 65 Megabytes while the Y2K files add up to 217 Megabytes. The RUN and HELP files needed to compile and execute the tests are provided with the 31DEC00 and later versions of AIPS. Note that DDT does not run on modern version of AIPS since MX and other tasks involved have been removed from the system.

DISCLAIMER

The results published in these pages do NOT constitute any sort of an endorsement by NRAO or Associated Universities, Inc. (NRAO's parent organization) of any particular vendor. The AIPS web pages at NRAO/CV do not and will not contain links to any hardware vendors' web pages.

In addition, many of the results reported here were relayed to us by others, and while all of these were carried out in a uniform way to the best of our knowledge, there will inevitably be variations introduced by factors such as changes in the AIPS code over time, system loads, different disk controllers (e.g. IDE vs. SCSI), and so on. The intent here is to share AIPS users' experiences with various platforms among the Astronomical Community.

New Y2K HUGE test

As reported above, the Y2K LARGE test can now run in as little as 5.0 minutes on some computers. Therefore, we have put together a HUGE version of the Y2K test. It uses Carilli/Perley Cynus-A X-Band data from D, C, B, and 4 frequencies of A configurations of the VLA. The 2674490 visbilities are Cleaned in four resolutions by IMAGR before the self-cal (replacing APCLN since it does not do multiple resolutions). The self-cal is amplitude and phase with the square root of the normal weights. The following Clean is also four resolutions done with IMAGR. The VTESS (and the one in LARGE) use the trick of setting FLUX = -F/10 where F is about the correct flux. With this trick VTESS is quite convergent rather than being completely erratic.

The Mac OS/X 2 GHz dual-headed machine listed at the top of this page does the HUGE test in 3519 seconds (2753 seconds of cpu). My Dell 1.3-GHz Linux box takes 9090 seconds (6416 seconds of cpu). The UV data sets and images for HUGE are available along with the other 31DEC04 data sets.

Initial test results with Y2K

The test results from AIPS Memo 104 are summarized in a separate table.

Recent Mac OS/X results

    Machine   Machine         Compiler    DDTL      Y2KL     Y2KL cpu  AMark
    run on    compiled on     type        sec       sec      min        2000
    -------   -----------     --------   -------   --------  --------  -----

1      4         4  .           A GNU    121-124     1126     16.28      35
2.     4         4              GNU      115-120     1060     15.14      38
3.     4         4 (b)          IBM       81- 83      748      9.58      53
5.     4         5 (gen)        IBM       77- 79      748      9.55      53
7.     4         4 (gen)        IBM       78- 83      759      9.57      53

11.    5         5              A GNU     69- 73      582      9.29      69
12.    5         5              GNU       68- 70      558      7.77      72
13.    5         5 (a)          IBM       33- 36      310      4.60     129
13.    5         5 (b)          IBM       32- 38      305      4.42     131
15.    5         5 (gen)        IBM       33- 34      300      4.46     133
16.    5         5 (gen -O5)    IBM       34- 39   313-322  4.66-4.82  >124

The G5 used is hakone a 2.0 GHz, dual-headed machine with 8 Gbyte ram,
       3 Terabytes disk!
The G4 used is pippin a 1.33 GHz, single-headed, rack-mounted machine
       with 1 Gbyte of ram.

Putting aips on a "disk image" slows things down (e.g. 47 sec DDT on
the G5 with the IBM compiler, 90 sec on G4)

IBM options (a) are -qthreaded -qunroll=auto -qtune=auto -qarch=auto
            (b) are -qtune=auto -qarch=auto
    all including (gen) use -O3.

The A GNU is the Apple-provided version of the GNU compilers while GNU
is one installed by Wes (~ 3.4.1).

Hot Boxes per DDT

This table is an attempt to show the best results for the most advanced hardware on which we or others have been able to run the DDT and AIPS. It is incomplete, but will be filled in as time permits.

Hardware Operating System Compiler AIPS Version Am(93) Flatters' prediction Notes
Intel Pentium III Xeon 1.7GHz Red Hat Linux 7.1 gcc 2.95.3 31DEC01 85.0 ? 1 Gbyte memory, dual processor, two 10,000-RPM 70G SCSI disks.
Intel Pentium 4 1.5 GHz Linux 2.2 (RH7.0) g77/gcc 2.95 31DEC00 69.0 ? Single processor, 384 MBytes memory, 3x 15,000-RPM 18GB drives (two striped as /usr, one as /data). Speed parameter set to 55 after first getting Y2K rating of 55.2.
Alpha Digital/Compaq XP1000 Red Hat Linux 6.2 DEC Fortran 31DEC00 58.8 ? 512 MB memory, 4MB Cache, 18.2G SCSI disk
Alpha Digital/Compaq DS-20 OSF/1 4.0D DEC Fortran 15OCT99 33.61 ? 2 Gbytes memory, RAID disks, guest Compaq/CSA system.
SGI Origin 2000 Irix 6.4 SGI 15APR99 30.3 16.4/cpu? 4 300-MHz R12K CPUs, speed parameter set to 30.
AMD Athlon 600MHz Linux 2.2.13 gcc 2.95.2 31DEC00 27.2 ? SuSE 6.3, 512 MBytes memory, 2 GBytes swap, 2 EIDE hdparm-tuned disks.
Intel Pentium III 800 Linux 2.2 (RH6.1) g77-egcs 2.91.66 31DEC00 25.6 ? Single processor, 128 MBytes memory, hdparm-tuned 7200 RPM EIDE disk (AIPSMark only 14 without tuning).
HP C360 (PA-8500 chip) HP-UX 10.20 HP f77 (not f90) 15APR99 24.6 ? 512 Mbytes main memory, Ultra-SCSI and SCSI Fast/Wide disks.
Intel Pentium III 550 Xeon Linux 2.2 (RH6.0) g77/egcs 1.1 15OCT99 21.6 ? Single processor, 768 MBytes memory, 4 x U-2 SCSI 10,000 RPM disks
Sun Sparc Ultra 60 SunOS 5.6 Sunsoft 15APR98 18.4 ? 13.9 without /tmp trick
speed parameter 13.5
Sun Sparc Ultra 30/300 SunOS 5 Sunsoft 15APR98 15.0 16.1 10.0 without /tmp trick
Aspen Systems Alpha 21164/433 MHz Linux 2.1.57 DEC Fortran 15APR97 9.0 16.1 Binaries built statically under OSF/1 4.0, run in emulation mode.
IBM RS/6000 SP2 AIX 4 xlf/cc 15APR98 4.7 7.1 machine not empty, compiler options non-optimal

Recent DDT Benchmark Results

A full table of results is available along with the initial 1993 results.

Pointers to sections ordered by manufacturer.

DEC ALPHA AXP (Digital Unix, OSF/1; and Linux)
Hewlett Packard (HP-UX 10)
IBM Power PC (AIX)
Intel/Linux and others, including Pentium III, Athlon, P-Pro, P-II, P-I, i486, i386
Silicon Graphics
Sun SPARC Ultra including the /tmp trick, the Weitek Upgrade

Patrick P. Murphy
Eric W. Greisen