[AIPS] [NRAO] AIPS Benchmarks
(Dirty Dozen Tasks)


Most recent change on $Date: 2023/08/24 20:26:45 $ (UT) [Y2K test]


Current Record AIPSMark(00)

1379
Mac Studio (2022) M1 Max, 64 Gbyte memory, 10 core cpu, 32 core GPU, 16 core Neural Engine, 2TB internal SSD, MACARM port.
1379
Mac mini M1, 16 Gbyte memory, 500 Gbyte Macintosh flash storage, Mac OS 13.5, MACARM port, gfortran 12.2.
1000
Same machine, MACINT port running under the Rosetta system.
769
Dell desktop manufactured 20200805 ("primate") RedHat 7.

NOTE: a benchmark test, called Y2K, has been developed. It is an adjustment of the existing DDT with an altered set of tasks (UVMAP and MX replaced by IMAGR), and a 10-fold change in the size of the problems. Its goal was to take on the order of an hour to run on fairly new desktop hardware. However, it now takes all of 29 seconds to run on the new record holder.

The Y2K and DDT (Dirty Dozen) suites are reliable ways of determining two things:

The AIPS Memo 104 describes the new test and gives initial performance results. It was developed on the 31DEC00 version of AIPS. See the 31DEC22 and 31DEC23 pages for information on obtaining and maintaining your very own copy of the current releases. For more details on the earlier DDT, see AIPS Memo 85 which has the information on the last iteration of the DDT test. This was modified for the 15JAN94 release of AIPS (this is not the most recent version; see the AIPS Home Page for info on recent releases). There are other documents available too. Note that the DDT test is no longer available; it uses the task MX which is itself no longer available.

The AIPSMark00 is defined as:

Am(00) = (40000 ÷ TY2K(L))

where TY2K(L) is the total run time in seconds of the Y2K LARGE test. This run time was defined initially by Glendenning and Hunt as the time in seconds from RUN DDTEXEC through PRINTING ANSWERS, ERRORS, OTHER IMPORTANT MESSAGES). The multiplier 40000 was chosen to make the new AIPSMark00 about the same as the old AIPSMark93 on many machines.

The AIPSMark93 is defined as:

Am(93) = (4000 ÷ TDDT(L))

where TDDT(L) is the total run time in seconds for the LARGE DDT suite as defined by Glendenning and Hunt. Chris Flatters devised a formula for predicting a computer's AIPSMarks based on its measured SPECfp(95) and SPECint(95) performance. An extensive list of SPEC results is maintained on the Web by John DiMarco at the University of Toronto. Chris finds that the following relation gives a fair, ball-park estimate of the AIPSmark that can be expected on a machine with known SPEC(95) benchmarks.

AIPSMark93 = 0.898 × SPECfp(95) + 0.11 × SPECint(95) - 1.665

How do I run Y2K?

Install the latest version of AIPS; see for example 31DEC22. Start running AIPS and enter EXPLAIN Y2K. This is available on the web for the impatient as Y2K.HLP. Note that DDT is no longer available since it uses tasks that have been removed from AIPS. The Y2K process consists of several steps:

Where can I get the Y2K data files?

Don't panic; 31DEC22 versions of the Y2K data files are available (remember to right click for Save as or Save link as) via anonymous ftp. For manual use of ftp, the address for anonymous ftp is ftp.aoc.nrao.edu and the data files are located in directory pub/software/aips/FITS/31DEC22. The older Y2K data files are available (remember to right click for Save as or Save link as) via anonymous ftp. But please look at the file sizes (use the ls -l or dir command in your ftp program) before pulling these across. Some of them are large: the aggregate size of the 19 Y2K (large) files is 319 Megabytes. The RUN and HELP files needed to compile and execute the tests are provided with the 31DEC00 and later versions of AIPS. Note that DDT does not run on modern version of AIPS since MX and other tasks involved have been removed from the system.

DISCLAIMER

The results published in these pages do NOT constitute any sort of an endorsement by NRAO or Associated Universities, Inc. (NRAO's parent organization) of any particular vendor. The AIPS web pages at NRAO/Socorro do not and will not contain links to any hardware vendors' web pages.

In addition, many of the results reported here were relayed to us by others, and while all of these were carried out in a uniform way to the best of our knowledge, there will inevitably be variations introduced by factors such as changes in the AIPS code over time, system loads, different disk controllers (e.g. IDE vs. SCSI), and so on. The intent here is to share AIPS users' experiences with various platforms among the Astronomical Community.

New Y2K HUGE test

As reported above, the Y2K LARGE test can now run in as little as 29 seconds on some computers. Therefore, we have put together a HUGE version of the Y2K test. It uses Carilli/Perley Cynus-A X-Band data from D, C, B, and 4 frequencies of A configurations of the VLA. The 2674490 visbilities are Cleaned in four resolutions by IMAGR before the self-cal (replacing APCLN since it does not do multiple resolutions). The self-cal is amplitude and phase with the square root of the normal weights. The following Clean is also four resolutions done with IMAGR. The VTESS (and the one in LARGE) use the trick of setting FLUX = -F/10 where F is about the correct flux. With this trick VTESS is quite convergent rather than being completely erratic. The 42 files of Y2K HUGE add up to 1.2 Gbytes.

The use of versions of Clean in these tests is somewhat problematical. When there are numerous image pixels at the same value as seen by a human, there will likely be one a single bit higher than the others as seem by the computer, Which pixel that is will depend on slight differences in the round-off and other algorithms of the host arithmetic unit. And, once Clean has chosen a different pixel, the algorithm will head off in a different direction. This explains why, with data of high signal-to-noise, the peak error in Cleans is usually no better than 9 bits. The Huge test saves the 4 residual images from the initial Clean and they usually have an rms difference around 6 bits and a peak difference around 0.3 - 3 bits. Nonetheless, the Huge test is useful. On the top-speed Mac mini M1 machine, Huge takee 390 seconds real time. On my latest Dell desktop, Huge takes twice as long (803 seconds).

Initial test results with Y2K

The test results from AIPS Memo 104 are summarized in a separate table.

Ancient Mac OS/X results

    Machine   Machine         Compiler    DDTL      Y2KL     Y2KL cpu  AMark
    run on    compiled on     type        sec       sec      min        2000
    -------   -----------     --------   -------   --------  --------  -----

1      4         4  .           A GNU    121-124     1126     16.28      35
2.     4         4              GNU      115-120     1060     15.14      38
3.     4         4 (b)          IBM       81- 83      748      9.58      53
5.     4         5 (gen)        IBM       77- 79      748      9.55      53
7.     4         4 (gen)        IBM       78- 83      759      9.57      53

11.    5         5              A GNU     69- 73      582      9.29      69
12.    5         5              GNU       68- 70      558      7.77      72
13.    5         5 (a)          IBM       33- 36      310      4.60     129
13.    5         5 (b)          IBM       32- 38      305      4.42     131
15.    5         5 (gen)        IBM       33- 34      300      4.46     133
16.    5         5 (gen -O5)    IBM       34- 39   313-322  4.66-4.82  >124

The G5 used is hakone a 2.0 GHz, dual-headed machine with 8 Gbyte ram,
       3 Terabytes disk!
The G4 used is pippin a 1.33 GHz, single-headed, rack-mounted machine
       with 1 Gbyte of ram.

Putting aips on a "disk image" slows things down (e.g. 47 sec DDT on
the G5 with the IBM compiler, 90 sec on G4)

IBM options (a) are -qthreaded -qunroll=auto -qtune=auto -qarch=auto
            (b) are -qtune=auto -qarch=auto
    all including (gen) use -O3.

The A GNU is the Apple-provided version of the GNU compilers while GNU
is one installed by Wes (~ 3.4.1).

Really old boxes per DDT

This table is an attempt to show the best results for the most advanced hardware on which we or others have been able to run the DDT and AIPS back in the day. It is obsolete.

Hardware Operating System Compiler AIPS Version Am(93) Flatters' prediction Notes
Intel Pentium III Xeon 1.7GHz Red Hat Linux 7.1 gcc 2.95.3 31DEC01 85.0 ? 1 Gbyte memory, dual processor, two 10,000-RPM 70G SCSI disks.
Intel Pentium 4 1.5 GHz Linux 2.2 (RH7.0) g77/gcc 2.95 31DEC00 69.0 ? Single processor, 384 MBytes memory, 3x 15,000-RPM 18GB drives (two striped as /usr, one as /data). Speed parameter set to 55 after first getting Y2K rating of 55.2.
Alpha Digital/Compaq XP1000 Red Hat Linux 6.2 DEC Fortran 31DEC00 58.8 ? 512 MB memory, 4MB Cache, 18.2G SCSI disk
Alpha Digital/Compaq DS-20 OSF/1 4.0D DEC Fortran 15OCT99 33.61 ? 2 Gbytes memory, RAID disks, guest Compaq/CSA system.
SGI Origin 2000 Irix 6.4 SGI 15APR99 30.3 16.4/cpu? 4 300-MHz R12K CPUs, speed parameter set to 30.
AMD Athlon 600MHz Linux 2.2.13 gcc 2.95.2 31DEC00 27.2 ? SuSE 6.3, 512 MBytes memory, 2 GBytes swap, 2 EIDE hdparm-tuned disks.
Intel Pentium III 800 Linux 2.2 (RH6.1) g77-egcs 2.91.66 31DEC00 25.6 ? Single processor, 128 MBytes memory, hdparm-tuned 7200 RPM EIDE disk (AIPSMark only 14 without tuning).
HP C360 (PA-8500 chip) HP-UX 10.20 HP f77 (not f90) 15APR99 24.6 ? 512 Mbytes main memory, Ultra-SCSI and SCSI Fast/Wide disks.
Intel Pentium III 550 Xeon Linux 2.2 (RH6.0) g77/egcs 1.1 15OCT99 21.6 ? Single processor, 768 MBytes memory, 4 x U-2 SCSI 10,000 RPM disks
Sun Sparc Ultra 60 SunOS 5.6 Sunsoft 15APR98 18.4 ? 13.9 without /tmp trick
speed parameter 13.5
Sun Sparc Ultra 30/300 SunOS 5 Sunsoft 15APR98 15.0 16.1 10.0 without /tmp trick
Aspen Systems Alpha 21164/433 MHz Linux 2.1.57 DEC Fortran 15APR97 9.0 16.1 Binaries built statically under OSF/1 4.0, run in emulation mode.
IBM RS/6000 SP2 AIX 4 xlf/cc 15APR98 4.7 7.1 machine not empty, compiler options non-optimal

Recent DDT Benchmark Results

A full table of results is available along with the initial 1993 results.

Pointers to sections ordered by manufacturer.

DEC ALPHA AXP (Digital Unix, OSF/1; and Linux)
Hewlett Packard (HP-UX 10)
IBM Power PC (AIX)
Intel/Linux and others, including Pentium III, Athlon, P-Pro, P-II, P-I, i486, i386
Silicon Graphics
Sun SPARC Ultra including the /tmp trick, the Weitek Upgrade

Patrick P. Murphy
Eric W. Greisen