In this document we measure the performance of a few operating systems for some important key performance indicators, such as startup time, data transfer speeds, etc.

The operating systems we test in this document are:
- Mac OS
- Windows
- Ubuntu
- Axiom

Axiom is an operating system that can run on any Unix kernel; for our tests we use the latest production version that uses the Linux kernel. Axiom is faster than any other operating system for all tests, so it serves as a useful benchmark.

We explain the methodology followed in these tests below, so that anyone can reproduce them with high accuracy. The tests are clear and easy to perform.

We do the tests on 2 different hardware platforms, the Tassig Box and the Macbook Air.

What are we going to test and why

System startup

We measure the following:

Startup time
Shutdown time
Restart time

These 3 components help us to measure how efficient and well-designed a system is. A “bloated” system will typically perform a lot of lengthy and resource-intensive actions. A “clean” system will do less, and will do it faster.

The startup process can be roughly split in the following successive steps:
1) BIOS step
2) kernel step
3) userland step

The BIOS step takes the same amount of time regardless of the OS, and depends only on the machine. The 2 next steps are OS-dependent:
The kernel step is essentially about loading and setting up software drivers for the machine.
The userland step is about starting system services, the user interface, users profiles, etc.

Data transfer speed

Wifi download speed. This measures the quality of the wifi drivers.

USB hard drive transfer speed. This measures the quality of the USB and disk drivers

Application startup

Time to start applications. This measures how well the kernel is designed to load and start applications. In this test we use LibreOffice and Google Chrome brower as example applications to test. They are both big and feature-full applications.

The following mechanisms and components take part in the startup of applications:
- loading of executables and libraries from permanent storage to RAM
- set up and startup of the executables by the kernel

Different operating systems do these things differently and will show different performance characteristics.

Methodology

General considerations and limitations

To measure Mac OS performance there are some limitations:
- generally, only Mac OS can run on Mac hardware so we cannot run Windows and Ubuntu on Mac hardware and compare performance of both operating systems directly.
- Mac OS does not run on other hardware than Mac, so we can’t compare on other hardware either.

To resolve this, there is a solution: we can run the operating system Axiom on Mac hardware and performance measurements can then be interpolated, because Axiom can also run on other hardware where Windows also runs.

We do 5 measurements for each test and report the average. Theses tests are easily reproduceable and there is little variance in the results.

Here are the 2 main setups for performance measurements and the description of the methology for each measurement case:

Machine 1: A Tassig Box Computer

In this setup we run every OS (Windows, Ubuntu, Axiom, but not Mac OS) directly on the same computer hardware. We choose the Tassig box for that, which is inexpensive commodity hardware with the following specifications:
- Intel CPU N3710
- 4GB of RAM
- 64GB SSD

Startup time

Press the power blue button, wait until the interface appears and the mouse is responsive.

Shutdown time

Press the shutdown button in the user interface, wait until the machine is powered off.

Restart time

Press the restart button, wait until the interface reappears and the mouse is responsive

Wifi download speed

This test measures the transfer rate on the local area network via wifi, in the following conditions:
- an http server is placed on the local area network. Such http server must be capable of high transfer rates. For example, the popular apache can be used.
- a file of approximately 1GB containing pseudo-random data is downloaded via the http protocol. Any software client can be used that can achieve high download speed. For example, a web browser, or the command line tool “wget”. To get a file with random data, any file containing compressed data can be used. For example, a file obtaining via “gzip”.
- report the average transfer rate per second.

USB SSD drive transfer speed

This test measures the capabilities of the USB and disk drivers.

- connect 2 USB drives to the machine using the USB 2.0 ports
- transfer a file of approximately 1GB file between the drives. This file needs to contain pseudo-random data, such as data obtained via compressing a file via gzip.
- we used fast external SSD drives from Tassig
- both drives are formatted in FAT32, so we can use the same format for all operating systems under testing
- report the average transfer rate per second.

Time to start applications

- Start the Chrome browser until you can type something in the address bar.
- Start Libreoffice Writer until you can write something in the new document

Machine 2: A Macbook Air

In this setup we run the operating systems on the Macbook Air. This setup cannot run Microsoft Windows or Ubuntu, but can run the following:
- Mac OS X directly from the internal SSD (default setup with a new Macbook Air)
- Axiom OS from a SSD connected via USB (Axiom to Go).

The 2 settings aren’t strictly comparable, and it would be better to run Axiom from the internal SSD, but because of practical limitations we didn’t do that.

The Macbook Air comes with the following specifications:

- Intel i5-3427U
- 8GB RAM
- 128GB SSD

Startup time

For Mac OS X: Press the power button, wait until the interface appears and the mouse is responsive.

For Axiom: We run Axiom from a USB drive. We choose the “Axiom to Go” drive for this, which is an external SSD where Axiom is installed. Any computer, including Macs, can run from the “Axiom to Go” drive. First plug the Axiom to Go drive. Then press the power button and then maintain the left “Option” key pressed to show the boot menu, boot on the USB drive. Then wait until the interface appears and the mouse is responsive.

Shutdown time

Both Mac OS X and Axiom: Press the shutdown software button in the user interface, wait until the machine is powered off.

Restart time

Press the restart button in the user interface, wait until the interface reappears and the mouse is responsive

Wifi download speed

Same procedure as with the Tassig Box above.

USB SSD drive transfer speed

Connect 2 USB drives on the machine, transfer a 1GB file between the drives. To do this we use a fast SSD external drive from Tassig. The drives need to be connected with a USB hub because the Macbook Air only has 2 USB ports.
Aside from that, this is the same procedure as with the Tassig Box above.

Time to start applications

- Start the Chrome browser until you can type something in the address bar.
- Start Libreoffice Writer until you can write something in the new document

Tests results

The tests show that Windows 10, Ubuntu are roughly comparable in performance. Mac OS tends to perform worse on all measurements. Axiom performs better than any other system on most measurements.

Interpretation

The startup, shutdown and restart tests show that Windows, Mac and Ubuntu are significantly “bloated”. The situation used to be worst in previous versions of Windows, as most Windows users would have witnessed. Windows 10 seems to have brought significant improvements to the Windows line. In particular, it seems to have caught up with Mac OS on these tests, which was always relatively fast.

Nevertheless, these systems are significantly slower than Ubuntu, which offers an intermediate level of speed.

Ubuntu, despite using the same kernel as Axiom, performs significantly worse than Axiom, which is the benchmark in terms of speed.

An interesting situation is that the Axiom system is much faster than Mac OS on Apple’s own hardware, namely the Macbook Air. In our tests, we did not even run Axiom from the internal storage of the Macbook Air, but from a USB drive.

For data transfer speeds Windows, Ubuntu and Axiom offer quite similar transfer speeds.
Mac OS performs once again significantly worse than Axiom on its own hardware.

Under applications startup time, we see for that cold starts are much faster on Axiom than on Windows and Mac OS, compared to warm starts. The reason is not obvious.

Warm starts are only slightly faster on Axiom compared to Windows and Mac OS, and tend to be quite fast, even for big applications. It seems that the CPU is the bottleneck for this task and that the parts that are kernel specific are quite insignificant compared to the parts that are kernel independent, therefore we observe roughly the same performance characteristics on all systems.

We can conclude that there is still a lot of room for improvement for Windows and Mac OS until they can catch up with Ubuntu. Overall, Axiom performs better than any other systems on most measurements.