This page contains the full documentation for the Python-interface of the IOHexperimenter. The Python-version of IOHexperimenter is built using pybind11, and desiged so most functions closely match their C++ equivalents, with a few added ease-of-use settings.
Note that these instructions are for versions >= 0.3.10. The minimum required Python version is 3.6.
You can install the experimenter directly from pip:
pip install ioh>=0.3.10
To verify the installation, you can use:
pip show ioh
Create a function object#
The most basic way to use the IOHexperimenter is to just access the problems from our predefined problem suites. For example one of the 25 Pseudo-Boolean problems or any of the single-objective BBOB-problems from COCO.
To use these problems, you can import the associated module as follows:
from ioh import problem
For all of these problems, you can then access the docstrings to get more information about precise usage.
These modules can then be used exactly as their c++ equivalent. However, for convenience, some wrapper functions are provided, such as the get_problem function. To get a 5-dimensional sphere function, with instance number 1, you can use the following:
from ioh import get_problem, ProblemClass f = get_problem("Sphere", 1, 5, ProblemClass.BBOB)
Instead of using the name of the function, you can also use their function number within their respecitve suite:
f1 = get_problem(1, 1, 5, ProblemClass.PBO)
With these problem-objects, the state, meta_data and contrainsts can easily be accessed:
And the problem can be evaluated by calling it:
This will then immediately update the state-variables:
Running an algorithm#
To showcase how to use these functions in combination with an optimization algorithm, we define a simple random-search example wich accepts an argument of class IOH_function.
import numpy as np np.random.seed(42)
def random_search(func, budget = 10): for i in range(budget): x = np.random.uniform(func.bounds.lb, func.bounds.ub) func(x)
To record data, we need to add a logger to the problem. We can do this easily by importing the logger module:
from ioh import logger
This logger can then be initialized with the directory in which to store the logs:
#Create default logger which writes to folder 'temp' l = logger.Analyzer(folder_name="temp")
This can then be attached to the problem:
f = get_problem(1, 1, 5, ProblemClass.BBOB)
Now, we can run the algorithm and store data
To ensure all data is written, we should flush the logger after running our experiments. This happens automatically when the python process terminates, but when running in an interactive environment, such as a jupyter notebook, this should be done explicitly.
Note that we should call the
reset method on a problem. By calling the
reset method, the problem’s internal state is automatically reset, and the data for the attached loggers will be flushed to the output streams. This function should also be used when performing multiple runs of the same problem in a loop, for example:
f5 = get_problem(5, 1, 5) l2 = logger.Analyzer(folder_name="temp2") f5.attach_logger(l2) for run in range(10): random_search(f5) f5.reset() l2.close()
If we want to track parameters of the algorithm, these variables need to be available on a object, i.e. as member variables on a class instance. In order to do this for the random search algorithm, we can do this be wrapping the function in a class.
class RandomSearch: def __init__(self, budget): #Note that we should re-initialize all dynamic variables if we want to run the same algorithm multiple times self.budget = budget # A parameter static over the course of an optimization run of an algorithm self.algorithm_id = np.random.randint(100) # A dynamic parameter updated by the algorithm self.a_tracked_parameter = None def __call__(self, func): self.f_opt = np.Inf self.x_opt = None for i in range(self.budget): x = np.random.uniform(func.bounds.lb, func.bounds.ub) # Update the tracked parameter self.a_tracked_parameter = i ** 10 f = func(x) if f < self.f_opt: self.f_opt = f self.x_opt = x return self.f_opt, self.x_opt @property def a_property(self): return np.random.randint(100) def reset(self): self.algorithm_id = np.random.randint(100)
A full example is below, we first instantiate the algorithm instance, after which we create a new logger. Then we update the logger instance, telling it to track the
algorithm_id parameter when we start a new run with a problem. The variable
a_tracked_parameter is updated before every problem evaluation, so we want the logger to store this variable at every problem evaluation. We can also do this for properties, using the same method.
algorithm = RandomSearch(10) l3 = logger.Analyzer(folder_name="temp3") # Track variables static over the course of an algorithm run l3.add_run_attributes(algorithm, ["algorithm_id"]) # Track dynamic variables, updated during the run of an algorithm l3.watch(algorithm, ["a_property", "a_tracked_parameter"]) # attach a problem to the logger p2 = get_problem(2, 2, 5) p2.attach_logger(l3) for run in range(10): algorithm(p2) # run the algorithm on the problem p2.reset() # reset the problem and logger state algorithm.reset() # update the algorithm_id parameter
Alternatively, we can use the
Experiment class, to more easily run the benchmarking experiment with multiple functions.
from ioh import Experiment
This can be initialized using a problem class by providing lists of function ids (or names), dimensions, instance ids and a number of independent repetitions as follows:
exp = Experiment( RandomSearch(10), # instance of optimization algorithm , # list of problem id's [1, 2], # list of problem instances , # list of problem dimensions problem_class = ProblemClass.BBOB, # the problem type, function ids should correspond to problems of this type njobs = 1, # the number of parrellel jobs for running this experiment reps = 2, # the number of repetitions for each (id x instance x dim) logged_attributes = [ # list of the tracked variables, must be available on the algorithm instance (RandomSearch) "a_property", "a_tracked_parameter" ] )
To start analyzing this data, simply upload the zip ‘data’-folder and upload it to the IOHanalyzer!
For a more fully flegded tutorial, which goes a more into detail on adding custom problems/loggers and dealing with constraints, please find the file example.ipynb in our github repository.