SpeedFAQ--How We Conduct Speed Tests

Return to Macs Only!'s Current News Page.
Copyright 2010 by Bill Fox All rights reserved.
Last Updated: May 9, 2010


Our complete test suite consists of a number of "canned" benchmarks and several so-called "real world" tests. They are simple enough for anyone to repeat on their own Mac for comparison. We conduct our tests in a manner that maximizes consistency and comparability. (Note: With Apple's switch to Intel CPUs, we will be shifting our benchmark applications to versions with universal binaries. They ones available so far are noted below. In some cases, this shift will affect the scores on Power PC CPUs.)

General Setup:

  1. No application is running at the same time as the test application.
  2. No Finder windows are open except those created by the test.
  3. There are no Startup Items in the Accounts System Preferences pane.
  4. Networking is turned off in the Network System Preferences pane by deselecting each port in the network port configuration window.
  5. No haxies or applications that create menu icons are installed.
  6. External Hard drives are detached.
  7. Disk Utility is run on the main drive to correct permissions and conduct repairs.
  8. No optical media are left in the optical drive.
  9. Automatic login is enabled.
  10. All documents, etc. are removed from the desktop.

General Test Procedure:

  1. Each test (or test suite in the case of some canned benchmarks) is run immediately after a startup to avoid memory caching issues.
  2. Each manually-timed test is recorded to the nearest whole second using a digital timer from Radio Shack.
  3. Each test is repeated three times.
  4. The results are averaged and reported to the nearest full second or full score point.

Canned Benchmarks--Ready-made applications for performing speed tests.

1. Altivec Fractal (Carbon) v1.3 measures the floating point calculation speed of a processor in millions of floating operations per seconds (GFLOPS). Higher is faster.

Settings: default

Procedure: Double-click application icon.

2. Cinebench 10, a Universal Application, measures the time to render an image and the time to run through a set of images using OpenGL with hardware acceleration and assigns a performance score. Higher is faster.

Settings: default

Procedure: Click on the Rendering test start button for either single or multiple CPUs as contained in the hardware. Afterwards, click on OpenGL Hardware Lighting test start button.

3. Let1kWindowsBloom 1.0, a Carbon application for Mac OS X that records the time in seconds to open an close 1000 Finder windows. Lower is faster.

Settings: n/a

Procedure: Double-click application icon.

4. Xbench v1.3, a Universal Application, times the speed of many hardware processes and reports the results as individual, collected and an overall scores. We report the collected CPU, Threading, Memory, Quartz Graphics, OpenGL Graphics, User Interface Graphics and Hard Disk scores. Higher is faster.

Settings: n/a

Procedure: Double-click application icon.

5. OpenGL Extensions Viewer 3.22, a Universal Application, times the speed of OpenGL specifications 1.1 through 3.0. Apple has only fully implemented specification 2.1 so that is the one used in our tests. Higher is faster.

Settings under the Tests tab:

  • Check specification 2.1, uncheck all others
  • Display Mode: 2560 x 1600 x 32 bpp
  • Check fullscreen
  • Framebuffer: Standard
  • Check Multisample with slider at 2
  • Check Anisotropy with slider at 4
  • All others unchecked
Procedure: Click the Test button.

"Real World" Tests--These essentially manually replicate parts of the various canned benchmark applications by doing several ordinary things on the computer and timing how long it takes. Extra care must be taken to ensure that they are setup, run and recorded exactly the same each time.

1. Startup time is measured in seconds from the chime of a startup (or "cold" boot), not a restart, until a complete desktop is formed, usually either the appearance of the drive images or the standard menu icons which ever is last. Note that haxies, other applications that place icons in the menu bar and extra hard drives can affect this test and need to be uninstalled, disabled or detached. Lower is faster

Settings: n/a

Procedure: Press the "On" button, start the timer on the chime and stop the timer when the desktop is complete.

2. Shutdown time is measured in seconds from clicking the Shutdown button in the dialog box that appears after selecting Shutdown in the Apple menu until the screen goes black. Lower is faster.

Settings: n/a

Procedure: Select Shutdown from the Apple menu, simultaneously click Shutdown in the ensuing dialog box and start the timer. Then, stop the timer exactly when the screen goes black.

3. Launch Classic Mode time is measured in seconds from clicking the start button until the full classic startup window disappears. Lower is faster.

Settings: n/a

Procedure: Launch System Preferences and click the Classic pane. Wait for the folder search to finish. Simultaneously click the Start button and start the timer. Stop the timer exactly when the classic mode startup window disappears.

4a. Duplicate a 700 MB Folder is measure in seconds from selecting "Duplicate" in the Finder's File menu until the copy window closes. The folder is a Mac OS 9.0 system folder with 3600 files in it. Lower is faster.

Settings: n/a

Procedure: Select the folder, move the mouse over "Duplicate" from the Finder's File pull-down menu, start timer when the mouse is released and stop timer when the copy window disappears.

4b. Duplicate a 500 MB Folder is measure in seconds from selecting "Duplicate" in the Finder's File menu until the copy window closes. The folder contains 20 copies of the iTunes 6.0.3 25MB .mkpg installer. Lower is faster.

Settings: n/a

Procedure: Select the folder, move the mouse over "Duplicate" from the Finder's File pull-down menu, start timer when the mouse is released and stop timer when the copy window disappears.

5. Scroll 144-page Document in Acrobat Reader (v8.1.), now a Universal Application, is measured in seconds from the time scrolling the document starts to the time it is completed. The file being scrolled is the pdf file for the Virtual PC 4.0 Manual. Lower is faster.

Settings: n/a

Procedure: Open the file with Acrobat reader and leave it at the default actual (100%) size, click and hold the scroll down arrow and start the timer simultaneously and stop the timer when the bottom of the document is reached.

6. Encode MP3 in iTunes is measured in seconds from clicking import to completing the encoding of the file. The file is the first track (10:23) on a CD of Beethoven's 6th Symphony. Lower is faster.

Settings:

  • Import Using: MP3 Encoder
  • Setting: Higher Quality (192 kbps)
  • Play songs while importing: Unchecked
  • Create file names with track number: Checked
  • Use error correction when reading Audio CDs: Unchecked

Procedure: Select the first CD track, click the "Import" button on iTunes and simultaneously start the timer and stop the timer when encoding is complete.

7. Export QuickTime Movie in iMovie is measured in seconds from selecting the location to save the file until the compressor window closes. The file is the original iMovie Tutorial clips. Lower is faster.

Settings: n/a

Procedure: Open the iMovie Tutorial with iMovie, stack the 6 film clips side by side, select "Share" in the File pull-down menu, select QuickTime in the ensuing window, choose "Full Quality DV" in the "Compress movie for:" pop-up window, click the "Share" button, in the resulting dialog box simlutaneously click the "Save" button and start the timer and stop the time when the compressor window closes.

3D Game Tests--The 3D game tests use built-in benchmark algorithms to time the average, maximum and minimum frame rates (frames processed per second) of a test set of scenes, usually termed a demo, rather than a standalone speed test program. As such, they are actually more like the canned benchmarks. Four of the games have Universal Binaries and run natively on Intel-based Macs. Call of Duty does not have Universal Binaries so it runs under Rosetta on Intel-based Macs.

1. Quake III Arena (v132b & UB-Universal): The v132b is not Universal and runs on Rosetta in Intel-based Macs. The UB version is a Universal Application we use for Intel-based Macs. The speed is measured in frames per second using a built-in demo timer. We use settings that match those we play with. They represent a balance between performance and visual experience. Higher is faster.

Settings:

  • Video Mode (resolution): 1024x768
  • Color Depth: 32 bit
  • Fullscreen: On
  • Lighting: Vertex
  • Geometric Detail: High
  • Texture Detail: Last setting before highest
  • Texture Quality: Default
  • Texture Filter: Bilinear
  • Screen Size: Maximum
  • Sound Quality: High

Prodecure: To use the built-in demo timer, at the main options screen choose DEMOS, bring down the console (shift~), type "timedemo 1" with no "s and a space before the 1, close the console (shift~), and run the FOUR.DM_68 demo that comes with the game by clicking on PLAY. When the demo finishes, bring down the console and note the frames per second.

2. Halo (v1.5.2 & 2.0.2-Universal): Version 1.5.2 is the PowerPC version and v2.0.2 is the latest Intel-based version with Universal Binaries that we use on Intel-based Macs. Speed is measured in frames per second using the built-in demo timing mode. We use settings that match those we play with. They represent a balance between performance and visual experience. Higher is faster.

Settings:

  • Hardware Shaders: Advanced Pixel Shaders
  • Detail Objects: Checked
  • Model Reflections: Checked
  • FSAA: Off since v2.0.2 and 2x Sampling before
  • Lens Flare: Low
  • Model Detail: Medium
  • Resolution: 1024x768
  • Framerate: No VSync
  • Specular: Yes
  • Shadows: Yes
  • Decals: Yes
  • Particles: Low
  • Texture Quality: Medium
  • Sound Quality: Low
  • Sound Variety: Medium

Procedure: Check "Run Time Demo" under Game Options and click OK. The results are written to a text file called timedemo.txt in the Halo game folder.

3. Call of Duty (1.5): The speed is measured in frames per second using the built-in demo timer. We use settings that match those we play with. They represent a balance between performance and visual experience. Higher is faster.

Settings:

  • Video Mode (Resolution): 1024x768
  • Full Screen: Yes
  • Character Textures: Normal
  • General Textures: High
  • Texture Filter: Trilinear
  • Texture Quality: 32 bit
  • Nvidia Distance Fog: Yes
  • Sound Quality: 44khz
  • Sound Provider: Miles Fast 2D Positional Audio
  • Wall Marks: On
  • Ejecting Brass: On
  • Dynamic Lights: Everything
  • World Dynamic Light Quality: Nicest
  • Model Detail: Normal
  • Sync Every Frame: No
  • Show Blood: Yes

Procedure: To use the built-in demo timer, at the main options screen bring down the console (shift ~), type "timedemo 1" without the "s and with a space before the 1, hit the return key, type "demo timedemo1.dm_2" with a space after demo but no other spaces, hit the return key again and the demo will run. When the demo finishes bring down the console (shift ~) again and note the frames per second.

4. Doom 3 (1.3.1-Universal): The speed is measured in frames per second using the built-in demo timer. We use settings that match those we play with on a given Mac (MQ for the iMac G5, 24" iMac Core 2 Duo and MacBook Pro and HQ for the Power Mac G5 Quad). They represent a balance between performance and visual experience. Higher is faster.

Settings:

MQ

-System Options-

  • Video Quality: Medium Quality
  • Screen size: 640x480 for iMac G5 & 800x600 for MacBook Pro & 24" iMac Core 2 Duo
  • Fullscreen: Yes

-Advanced Options-

  • High Quality Special Effects: Yes
  • Enable Shadows: No
  • Enable Specular: Yes
  • Enable Bump Map: Yes
  • Vertical Sync: No
  • Antialiasing: Off

HQ

-System-

  • Video Quality: High Quality
  • Screen size: 1024x768
  • Fullscreen: Yes

-Advanced-

  • High Quality Special Effects: Yes
  • Enable Shadows: No
  • Enable Specular: Yes
  • Enable Bump Map: Yes
  • Vertical Sync: No
  • Antialiasing: Off

Procedure: If you change the settings, you need click the "Apply" button, then quit and relaunch the application. To use the built-in demo timer, at the main options screen bring down the console (control-option ~), type "timedemo demo1 1" without the "s and with the space before demo1 and before the final 1, hit the return key and the demo will run. When the demo finishes a dialog box appears and the frames per second are displayed. The demo needs to be run 4 times since the first run is affected by an incomplete pre-cache and will be slower.

5. Quake 4 (1.2-Universal): The speed is measured in frames per second using the built-in demo timer with v1.2 and later. (We don't use v1.3 because of multiplayer and demo issues.) We use settings that match those we play with on a given Mac. They represent a balance between performance and visual experience. Higher is faster.

Settings:

-System Options-

  • Video Quality: Medium Quality
  • Screen size: 1024x640
  • Aspect ratio: Widescreen [16:9]
  • Fullscreen: Yes

-Advanced Options-

  • High Quality Special Effects: No
  • Enable Shadows: No
  • Enable Specular: Yes
  • Enable Bump Map: Yes
  • Vertical Sync: No
  • Antialiasing: Off
  • Multiple CPU/Core: Yes

Procedure: If you change the settings, you need click the "Apply" button--some require that you quit and relaunch the application. To use the built-in demo timer, at the main options screen bring down the console (control-option ~), type

"playnettimedemo id_demo001"

without the "s and with the space before id_demo001 --Special Note: in some cases the full file path needs to be typed in as:

"playnettimedemo Benchmarking/demos/id_demo001.netdemo"

--then hit the return key and the demo will run. When the demo finishes the console appears again and the frames per second are displayed. The demo needs to be run 4 times since the first run is slightly affected by an incomplete pre-cache and will be slower than the other three. When done, either type "quit" and press the "return" key or press the F9 key and then the "esc" key after the game loads to get back to the settings screen--for some reason one is unable to just raise the console.

 



Copyright © 1995-2010 by Bill Fox
All rights reserved.

MY LAWYER MADE ME POST THIS: 8-/
The Macs Only! web site is for informational purposes only. No one associated with Macs Only! assumes any responsibility for its accuracy. The information is subject to change without notice. Any use of, or actions taken based upon any of the information contained on this web site is done entirely at your own risk. Mention of any products or services is for informational purposes only and constitutes neither an endorsement nor a recommendation. Macs Only! and those associated with Macs Only! assume no responsibility with regard to the selection, performance or use of these products.

OUR PRIVACY POLICY

Apple, Apple Logos, Macintosh, and Mac OS Logos are registered trademarks of Apple Computer, Inc. All other trademarks mentioned belong to their respective owners.

[Back to top of page]