Test Plan

In this document we describe a performance test plan of ArmoniK. The purpose is to measure the performance of the different functionalities of ArmoniK (task submissions, processing of tasks, ...). We will perform the following tests:

• Measure the execution time of different numbers of submitted tasks and different size of the tasks.
• The strong scaling.

The purpose of these tests is to have performance measurements based on ArmoniK versions. As this is the first iteration of the performance tests, we will establish our baseline based on a specific version (see hereafter).

Products

Stress test

This is a stress test which runs a number of tasks with a specific I/O sizes and task workloads. It uses ArmoniK C# extension which copies the DLLs of the application as a zip and installs them on the workers.

Bench

This test is similar to StressTest but it doesn't use ArmoniK C# extension. The application is already installed on the workers.

HtcMock

With this test we can have an idea of how ArmoniK can handle the applications with subtasks and aggregation tasks.

Functionalities

We want to test in this first iteration, the three main functionalities of ArmoniK described below.

The submission of the tasks

With these tests, we want to measure the performances of tasks submissions od ArmoniK. This is the first step of the task orchestration which consists in submitting the tasks and the input data to be processed by ArmoniK workers.

The processing of the tasks

In addition to the tasks submission, these tests will allow us to measure the ArmoniK ability to process different numbers of tasks with different workload durations.

The retrieving of the results

These tests will allow us to determine ArmoniK performances to retrieve the results data after tasks processing. This step consists in the recovery of the output data after the processing of the tasks and it is the last step of the task orchestration.

Tools

Bash scripts

The scripts start a warming up run, launch a batch of runs with the same parameters, and store performance data in files.

Python

• Clean the files of the performance data and store them in readable JSON files.
• Compute the median and the mean of each batch of runs

JSON

We store the results of the performance tests in JSON files which we will be stored in a database.

Tests environment

These tests are dependent of the configuration of the underlying infrastructure and are a prerequirement of every test execution used as comparison. This method could be reused as reproducer for anyone desiring to execute our tests.

The versions of the different ArmoniK components and the third-party tools are defined in versions.tfvars.json

Exploited ressources

The estimated results

Faster task processing by increasing the number of pods.

Tests to do

Throughput scalability tests

• 1000 tasks on 100 pods, 1000 pods and 10000 pods
• 5000 tasks on 100 pods, 1000 pods and 10000 pods
• 10K tasks on 100 pods, 1000 pods and 10000 pods
• 100K tasks on 100 pods, 1000 pods and 10000 pods
• 1M tasks on 100 pods, 1000 pods and 10000 pods

with the following parameters:

• Tasks duration: 1 ms
• Input payload size: 8B
• Output payload size: 8B

Stress Test

Bench

HtcMock

Submission scalability tests

• Number of pods : 1000
• Tasks duration : 500 ms
• Output payload size : 8B

Stress Test

Other tests

This document will be completed in the future with more tests to target more detailed and deeper ArmoniK's features and components.