.. sample documentation master file, created by sphinx-quickstart on Mon Apr 16 21:22:43 2012. You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. HPCBench Documentation ====================== .. module:: hpcbench HPCBench is a Python package that allows you to specify and execute benchmarks. It provides: * an API to describe how to execute benchmarks utilities and gather metrics. * A way to describe benchmark campaigns in YAML format. * command line utilities to execute your campaigns, and post-process generated metrics in various ways: * Merging metrics from different campaigns * Exporting metrics in CSV format * Exporting data to Elasticsearch * PDF report generation HPCBench assumes that the various benchmark tools/binaries are managed elsewhere and does not provide features for building and maintaining benchmark software. NB: This Python package is still in pre-alpha stage, and not suitable for production. Installation ============ .. highlight:: bash **HPCBench** is in the `Python Package Index `_. Installation with pip --------------------- We recommend using `pip `_ to install hpcbench on all platforms:: $ python -m pip install hpcbench To upgrade HPCBench with pip:: $ python -m pip install --upgrade hpcbench Dependencies ------------ HPCBench supports both python 2.7 and 3.4+. Overview ======== CLI --- The main interface through which **HPCbench** is used is a set of command line utilities: * ben-sh: Execute a tests campaign on your workstation * ben-csv: Extract metrics of an existing campaign in csv format * ben-umb: Extract metrics of an existing campaign * ben-elastic: Push campaign data to Elasticsearch * ben-nett: Execute a tests campaign on a cluster * ben-merge: Merge campaign output directories * ben-tpl: Generate HPCBench plugin scaffolds, see :ref:`usage ` for more information on plugin generation. **ben-sh** and **ben-nett** expect a YAML file describing the campaign to execute. The structure of this YAML file is detailed in the :doc:`campaign file reference `. Campaign YAML description ------------------------- .. toctree:: :maxdepth: 2 :numbered: campaign.rst standard_benchmark.rst API --- The purpose of the HPCBench API is to provide a consistent and unified layer: * to execute, and parse results of existing benchmarks utilities (Linpack, IOR, ...) * to use extracted metrics to build figures Both system benchmarks (e.g. STREAM, linpack), as well as software benchmarks should be implemented using this API. Most users parametrize benchmarks in the above-mentioned campaign YAML file. More advanced users will want to implement their own benchmarks based on the API. For more information check the :doc:`module reference ` Getting Started =============== As of now, only a few benchmarks are supported. This section assumes that you have at least installed the ``sysbench`` utility on your workstation. Launch a campaign on your workstation ------------------------------------- Create your first campaign YAML file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Create a ``local-campaign.yaml`` file with the following content:: benchmarks: '*': test: type: sysbench Launch the benchmark on your workstation ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Execute the following command:: $ ben-sh local-campaign.yaml This will create a ``hpcbench-`` in the directory with the benchmark's results. Although the user is not meant to be manually checking results inside the output directory at this point take a look at ``hpcbench-//*/test/metrics.json``. You will find that this file contains the collected metrics data from sysbench. The raw logs and stdout's can be found further down the directory tree. **Note**: Do not manually edit files inside the output directory. HPCBench offers a number of utilities to export and post-process the collected results. Launch a campaign on a set of nodes ----------------------------------- The YAML config file is getting a little more complex. For instance create the following ``remote-campaign.yaml``:: network: nodes: - localhost benchmarks: '*': test: type: sysbench You can add computer nodes to the ``nodes`` section. Launch the benchmark ~~~~~~~~~~~~~~~~~~~~ Use the ``ben-nett`` utility to execute the campaign on every nodes. It uses SSH to submit jobs so you have to make sure you can access those nodes without passphrase. For this you could use the ``ssh_config_file`` key in YAML to specify a custom configuration (see) :doc:`campaign file reference `):: $ ben-nett remote-campaign.yaml .. _ben-tpl-usage: How to create a new benchmark Python module? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It is possible to create your own benchmark based on a tool that has not been so far supported by HPCBench. This is done by generating an external plugin scaffold using ``ben-tpl`` and implementing the benchmark execution and metrics parsing inside the generated classes. Here is the basic workflow: 1. First create a default JSON file describing the Python module: ``ben-tpl benchmark -g config.json`` 2. Update fields in ``config.json`` 3. Generate the Python module template in the current directory: ``ben-tpl benchmark config.json`` 4. Edit the ``benchmark.py`` file 5. When ready you can install the module, with ``pip`` for instance. Development Guide ================= Prerequisites ------------- Docker ~~~~~~ `Elasticsearch `_ is required to execute some of the unit-tests. The easiest way to accomplish this, is to use Docker containers. Quick'n'dirty Docker installation:: $ curl -fsSL get.docker.com -o get-docker.sh $ sh get-docker.sh $ curl -L https://github.com/docker/compose/releases/download/1.15.0/docker-compose-`uname -s`-`uname -m` > /usr/local/bin/docker-compose $ chmod +x /usr/local/bin/docker-compose Post-installation instructions to use Docker without root privileges (logout/login) required:: $ sudo groupadd docker $ sudo usermod -aG docker $USER Test your docker installation with:: $ docker run --rm hello-world Build instructions ------------------ .. highlight:: bash Grab the source code:: $ git clone https://github.com/BlueBrain/hpcbench.git $ cd hpcbench We suggest you use a dedicated virtual environment. For that you can use either `virtualenv `_ package or `pyenv `_, which is even better. With ``pyenv``:: $ pyenv virtualenv hpcbench $ pyenv local hpcbench Alternatively, with ``virtualenv``:: $ virtualenv .env $ . .env/bin/activate Then:: $ pip install tox $ tox -e py27 ``tox`` is configured to test HPCBench against different Python versions. To test against a specific python version you can supply it the ``-e`` parameter:: $ tox -e py27 -- $ tox -e py36 -- The ``--tests`` parameter can be used to run only one specific unit test module or class:: $ tox -e py36 -- --tests tests/test_driver.py Elasticsearch ~~~~~~~~~~~~~ To start an Elasticsearch container, you can use the ``misc/dc`` script wrapper on top of ``docker-compose``:: $ misc/dc up -d It will start an Elasticsearch container listening on port 9200 and a Kibana instance listening on port 5612. Let's now try to ping Elasticsearch:: $ curl localhost:9200 { "name" : "jQ-BcoF", "cluster_name" : "elasticsearch", "cluster_uuid" : "yGP7_Q2gSU2HmHpnQB-jzg", "version" : { "number" : "5.5.1", "build_hash" : "19c13d0", "build_date" : "2017-07-18T20:44:24.823Z", "build_snapshot" : false, "lucene_version" : "6.6.0" }, "tagline" : "You Know, for Search" } You can also access Kibana at http://localhost:5601 The ``Dev Tools`` is one of the most handy Elasticsearch client for humans. Testing it all out ~~~~~~~~~~~~~~~~~~ Unit-tests assume that Elasticsearch is running on localhost. You can define the ``UT_ELASTICSEARCH_HOST`` environment variable to specify another location:: $ ELASTICSEARCH_HOST=server01:9200 tox How to integrate a new benchmark utility in the HPCBench repository? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1. First make sure you can properly build the project and tests pass successfully. It may be tempting to skip this part, but please don't. 2. Create a dedicated Git branch. 3. Create a new Python module in ``hpcbench/benchmark`` directory named after the utility to integrate. 4. In this new module, implement ``hpcbench.api.Benchmark`` and ``hpcbench.MetricsExtractor`` classes. 5. Register the new module in ``setup.py`` ``[hpcbench.benchmarks]`` entrypoint so that it can be found by HPCBench. 6. Create a dedicated unit test class in `tests/benchmark/` directory. The purpose of this test is to make sure that: * your Benchmark class is properly defined, and usable by HPCBench. * your metric extractor is properly working, without having to launch the utility itself. 7. To properly test your metrics extractors, some outputs of the benchmark utility will be added to the repository. For every category of your benchmark, create a file title ``tests/benchmark/..stdout`` with the benchmark utility's output. These files will be automatically used. Do not hesitate to take inspiration from the ``tests/benchmark/test_sysbench.py`` test module. 8. Run the test-suites until it passes:: $ tox 9. Submit a `pull-request `_ LICENSE ======= This software is released under MIT License. Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search`