AIPS Benchmarks
(Dirty Dozen Tasks)

---

Most recent change on $Date: 2008/06/24 19:19:49 $ (UT) [Y2K test]

NOTE: a new benchmark test, called Y2K, has been developed. It is an adjustment of the existing DDT with an altered set of tasks (UVMAP and MX replaced by IMAGR), and a 10-fold change in the size of the problems. Its goal is to take on the order of an hour to run on fairly new desktop hardware. However, it now takes 3:43 on a 3.06 GHz dual-cpu Intel Xeon with the Intel binary installation. It takes about 2:05 on the new record holder.

The Y2K and DDT (Dirty Dozen) suites are reliable ways of determining two things:

• Does a given AIPS installation give accurate results, and
• How fast does it do it?

The new AIPS Memo 104 describes the new test and gives initial performance results. It was developed on the 31DEC00 version of AIPS. See the 31DEC05 and 31DEC06 pages for information on obtaining and maintaining your very own copy of the current releases. For more details on the earlier DDT, see AIPS Memo 85 which has the information on the last iteration of the DDT test. This was modified for the 15JAN94 release of AIPS (this is not the most recent version; see the AIPS Home Page for info on recent releases). There are other documents available too.

The AIPSMark⁰⁰ is defined as:

A_m⁽⁰⁰⁾ = (40000 ÷ T_Y2K(L))

where T_Y2K(L) is the total run time in seconds of the Y2K LARGE test. This run time was defined initially by Glendenning and Hunt as the time in seconds from RUN DDTEXEC through PRINTING ANSWERS, ERRORS, OTHER IMPORTANT MESSAGES). The multiplier 40000 was chosen to make the new AIPSMark⁰⁰ about the same as the old AIPSMark⁹³ on many machines.

The AIPSMark⁹³ is defined as:

A_m⁽⁹³⁾ = (4000 ÷ T_DDT(L))

where T_DDT(L) is the total run time in seconds for the LARGE DDT suite as defined by Glendenning and Hunt. Chris Flatters devised a formula for predicting a computer's AIPSMarks based on its measured SPECfp(95) and SPECint(95) performance. An extensive list of SPEC results is maintained on the Web by John DiMarco at the University of Toronto. Chris finds that the following relation gives a fair, ball-park estimate of the AIPSmark that can be expected on a machine with known SPEC(95) benchmarks.

AIPSMark⁹³ = 0.898 × SPECfp(95) + 0.11 × SPECint(95) - 1.665

How do I run Y2K and DDT?

Install the latest version of AIPS; see for example 31DEC06. Start running AIPS and enter EXPLAIN Y2K and EXPLAIN DDT. These are available on the web for the impatient as Y2K.HLP and DDT.HLP. The process consists of several steps:

• Install AIPS
• Copy the relevant data files (see below) to your computer.
• Compile Y2K and/or DDT (RUN Y2KLOAD).
• Set adverbs in Y2K to load the data and load them.
• Set adverbs in Y2K to run the test and then actually run the test.
• Examine the printfile for timing and agreement information

Where can I get the Y2K and DDT data files?

Don't panic; 31DEC04 versions of the Y2K and DDT data files are available (remember to right click for Save as or Save link as) via anonymous ftp. The older Y2K data files are available (remember to right click for Save as or Save link as) via anonymous ftp and the older DDT data files are also available via anonymous ftp. But please look at the file sizes (use the ls -l or dir command in your ftp program) before pulling these across. Some of them are large: the aggregate size of all 33 DDT files is about 65 Megabytes while the Y2K files add up to 217 Megabytes. The RUN and HELP files needed to compile and execute the tests are provided with the 31DEC00 and later versions of AIPS.

DISCLAIMER

The results published in these pages do NOT constitute any sort of an endorsement by NRAO or Associated Universities, Inc. (NRAO's parent organization) of any particular vendor. The AIPS web pages at NRAO/CV do not and will not contain links to any hardware vendors' web pages.

In addition, many of the results reported here were relayed to us by others, and while all of these were carried out in a uniform way to the best of our knowledge, there will inevitably be variations introduced by factors such as changes in the AIPS code over time, system loads, different disk controllers (e.g. IDE vs. SCSI), and so on. The intent here is to share AIPS users' experiences with various platforms among the Astronomical Community.

New Y2K HUGE test

As reported above, the Y2K LARGE test can now run in as little as 5.0 minutes on some computers. Therefore, we have put together a HUGE version of the Y2K test. It uses Carilli/Perley Cynus-A X-Band data from D, C, B, and 4 frequencies of A configurations of the VLA. The 2674490 visbilities are Cleaned in four resolutions by IMAGR before the self-cal (replacing APCLN since it does not do multiple resolutions). The self-cal is amplitude and phase with the square root of the normal weights. The following Clean is also four resolutions done with IMAGR. The VTESS (and the one in LARGE) use the trick of setting FLUX = -F/10 where F is about the correct flux. With this trick VTESS is quite convergent rather than being completely erratic.

The Mac OS/X 2 GHz dual-headed machine listed at the top of this page does the HUGE test in 3519 seconds (2753 seconds of cpu). My Dell 1.3-GHz Linux box takes 9090 seconds (6416 seconds of cpu). The UV data sets and images for HUGE are available along with the other 31DEC04 data sets.

Initial test results with Y2K

The test results from AIPS Memo 104 are summarized in a separate table.

Recent Mac OS/X results

    Machine   Machine         Compiler    DDTL      Y2KL     Y2KL cpu  AMark
    run on    compiled on     type        sec       sec      min        2000
    -------   -----------     --------   -------   --------  --------  -----

1      4         4  .           A GNU    121-124     1126     16.28      35
2.     4         4              GNU      115-120     1060     15.14      38
3.     4         4 (b)          IBM       81- 83      748      9.58      53
5.     4         5 (gen)        IBM       77- 79      748      9.55      53
7.     4         4 (gen)        IBM       78- 83      759      9.57      53

11.    5         5              A GNU     69- 73      582      9.29      69
12.    5         5              GNU       68- 70      558      7.77      72
13.    5         5 (a)          IBM       33- 36      310      4.60     129
13.    5         5 (b)          IBM       32- 38      305      4.42     131
15.    5         5 (gen)        IBM       33- 34      300      4.46     133
16.    5         5 (gen -O5)    IBM       34- 39   313-322  4.66-4.82  >124

The G5 used is hakone a 2.0 GHz, dual-headed machine with 8 Gbyte ram,
       3 Terabytes disk!
The G4 used is pippin a 1.33 GHz, single-headed, rack-mounted machine
       with 1 Gbyte of ram.

Putting aips on a "disk image" slows things down (e.g. 47 sec DDT on
the G5 with the IBM compiler, 90 sec on G4)

IBM options (a) are -qthreaded -qunroll=auto -qtune=auto -qarch=auto
            (b) are -qtune=auto -qarch=auto
    all including (gen) use -O3.

The A GNU is the Apple-provided version of the GNU compilers while GNU
is one installed by Wes (~ 3.4.1).

Hot Boxes per DDT

This table is an attempt to show the best results for the most advanced hardware on which we or others have been able to run the DDT and AIPS. It is incomplete, but will be filled in as time permits.

Recent DDT Benchmark Results

A full table of results is available along with the initial 1993 results.

Pointers to sections ordered by manufacturer.

DEC ALPHA AXP (Digital Unix, OSF/1; and Linux)

Hewlett Packard (HP-UX 10)

IBM Power PC (AIX)

Intel/Linux and others, including Pentium III, Athlon, P-Pro, P-II, P-I, i486, i386

Silicon Graphics

Sun SPARC Ultra including the /tmp trick, the Weitek Upgrade