logger¶

Namespaces¶

Types¶

Analyzer¶

using ioh::logger::Analyzer = analyzer::Analyzer ¶: The Analyzer logger.

Default¶

using Default = Store¶: Default logger is the in-memory one.

Properties¶

using ioh::logger::Properties = std::vector<std::reference_wrapper<Property>>¶: convenience typedef for a vector of properties

Triggers¶

using ioh::logger::Triggers = std::vector<std::reference_wrapper<Trigger>>¶: convenience typedef for a vector of triggers

Classes¶

Combine¶

class Combine : public ioh::Logger ¶

An logger that can hold several loggers and call them all.

Useful if one want to use several loggers without having to manage them separately.

Example:

FIXME update this example
    BBOB_suite bench({1},{2},{3});
using Input = BBOB_suite::Input_type;

    ecdf_logger<Input> log_ecdf(0,6e7,20, 0,100,20);
    csv_logger<Input> log_csv;

// Combine them all
    LoggerCombine<Input> loggers(log_ecdf);
    loggers.add(log_csv);

// Now you can single call.
    loggers.track_suite(bench);
// etc.

Note

: Loggers are guaranteed to be called in the order they are added.

Unnamed Group

inline virtual void attach_suite(const std::string &suite_name) override¶: Logger interface.

inline virtual void attach_problem(const problem::MetaData &problem) override¶

Starts a new session for the given problem/instance/dimension/run.

This is called automatically by the Problem (or Suite) to which the Logger is attached, each time a change occurs (be it a new run, or a new problem instance).

Warning

If you override this function, do not forget to call the base class’ one.

inline virtual void log(const logger::Info &logger_info) override¶: Check if the logger should be triggered and if so, call call(log_info).

inline virtual void call(const logger::Info&) override¶: Main entry point, called everytime a Trigger is true.

inline virtual void reset() override¶

Optional actions when the logger is detached from a suite/problem or the problem is reset.

Useful if you want to flush data, for instance, and/or start a new run of the linked algorithm.

Warning

You most probably don’t want to call this directly, but should call the Problem::reset() instead, which will call this anyway.

Warning

If you override this function, do not forget to call the base class’ one.

Public Functions

inline explicit Combine(Logger &logger)¶: Takes at least one mandatory logger, because an empty instance would be illogical.

inline explicit Combine(std::vector<std::reference_wrapper<Logger>> loggers)¶

Handle several loggers at once, but you have to pass pointers.

Note

you can use initializer lists to instantiate the given std::vector:

LoggerCombine loggers({log_ecdf, log_csv});

inline void append(Logger &logger)¶: Add another logger to the list.

Protected Attributes

std::vector<std::reference_wrapper<Logger>> _loggers¶: Store the managed loggers.

EAH¶

class EAH : public ioh::Logger ¶

A logger that stores bi-dimensional error/evaluations discretized attainment matrices.

A matrix is stored for each triplet (problem,dimension,instance), everything being identified by its index. Attainment matrices are boolean 2D tables, in which a 1 indicates that an error target (row index) have been attained at an evaluation target (column index).

The whole set of matrices can be accessed by the get accessor. A single one can be accessed by the at accessor. Scales can be get back by *_range accessors, which give access to the corresponding min and max functions (

using namespace ioh::problem;
using namespace ioh::logger;
unsigned int sample_size = 100;
unsigned int buckets = 20;
ioh::suite::BBOB suite({1, 2}, {1, 2}, {2, 10});
ioh::logger::EAH logger(0, 6e7, buckets, 0, sample_size, buckets);
suite.attach_logger(logger);
for (const auto &p : suite) {
for (auto r = 0; r < 2; r++) {
for (auto s = 0; s < sample_size; ++s) {
            (*p)(ioh::common::random::real(p->meta_data().n_variables));
        } // s in sample_size
        p->reset(); // New run
    } // r in runs
} // p in suite

FlatFile¶

class FlatFile : public ioh::logger::Watcher ¶

A logger that stores some information in a single, tabular-like, file.

Each line displays the problem metadata and the watched properties. This format displays a lot of redundant information, but is very easy to parse.

logger::FlatFile(
    {trigger::always},
    {watch::evaluations, watch::transformed_y},
"my_experiment.dat",
"./today/"
);

Subclassed by ioh::logger::analyzer::Analyzer

Public Functions

inline FlatFile(std::vector<std::reference_wrapper<Trigger>> triggers, std::vector<std::reference_wrapper<Property>> properties, const std::string &filename = "IOH.dat", const fs::path &output_directory = fs::current_path(), const std::string &separator = "\t", const std::string &comment = "# ", const std::string &no_value = "None", const std::string &end_of_line = "\n", const bool repeat_header = false, const bool store_positions = false, const std::vector<std::string> &common_header_titles = {"suite_name", "problem_name", "problem_id", "problem_instance", "optimization_type", "dimension", "run"})¶

The logger should at least track one logger::Property, or else it makes no sense to use it.

Parameters:

triggers – When to fire a log event.
properties – What to log.
filename – File name in which to write the logged properties.
output_directory – Directory in which to put the data file.
separator – The string separating fields.
comment – The string indicating a comment.
no_value – The string indicating that a watched property does not exists in this context.
end_of_line – The string to use when all fields have been written.
repeat_header – If true, the commented header is printed for each new run.
store_positions – Whether to store x positions in the logged data
common_header_titles – Seven strings to print in the header for the common problem meta data (property names are automatically printed after).

inline virtual void attach_problem(const problem::MetaData &problem) override¶

Starts a new session for the given problem/instance/dimension/run.

This is called automatically by the Problem (or Suite) to which the Logger is attached, each time a change occurs (be it a new run, or a new problem instance).

Warning

If you override this function, do not forget to call the base class’ one.

inline virtual void call(const Info &log_info) override¶: Main entry point, called everytime a Trigger is true.

inline void set_full_precision(const bool full_precision)¶

Write values with round-trip precision ({:.17g}) instead of {:.10f}.

Applies to the properties that use the default double format (DEFAULT_DOUBLE_FORMAT) and to the x positions. Properties with an explicit format, such as evaluations, keep it. Default: false, which keeps the output of earlier versions, where values below 5e-11 are written as 0. Set it before the first problem is attached.

inline bool full_precision() const¶: Whether values are written with round-trip precision.

inline virtual fs::path output_directory() const¶: Accessor for output directory.

inline std::string filename() const¶: Accessor for filename.

inline virtual void close() override¶: close data file

inline virtual ~FlatFile()¶

Protected Functions

inline void open_stream(const std::string &filename, const fs::path &output_directory)¶: Open a file.

Protected Attributes

const std::string sep_¶: Seperator.

const std::string com_¶: Comment character.

const std::string eol_¶: EOL character.

const std::string nan_¶: NAN string.

const std::string common_header_¶: Common header.

const bool repeat_header_¶: Repeat header for every run?

const bool store_positions_¶: Store x positions?

bool requires_header_¶: Requires header?

const bool log_meta_data_¶: Log meta data?

fs::path output_directory_¶: Output directory.

std::string filename_¶: Filename.

std::ofstream out_¶: Output stream.

bool full_precision_ = false¶: Write doubles with round-trip precision (“{:.17g}”) instead of “{:.10f}”?

size_t current_run_¶: Current run.

std::string current_meta_data_¶: Current meta data.

Private Functions

inline void cache_meta_data()¶

Caches the meta data logged at every row.

If log_meta_data is false, it will store an empty string. Otherwise, the string ends with a separator.

Property¶

class Property¶

Interface for callbacks toward variable to be logged.

A class that inherit this interface should be able to indicate if the managed variable is not available at the time of the current logger’s call.

Subclassed by ioh::watch::CurrentBestY, ioh::watch::CurrentY, ioh::watch::Evaluations, ioh::watch::Penalty, ioh::watch::Pointer< T >, ioh::watch::PointerReference< T >, ioh::watch::RawY, ioh::watch::RawYBest, ioh::watch::Reference< T >, ioh::watch::TransformedY, ioh::watch::TransformedYBest, ioh::watch::Violation

Public Functions

inline Property(const std::string &name, const std::string &format = DEFAULT_DOUBLE_FORMAT)¶: Constructor.

virtual std::optional<double> operator()(const logger::Info &log_info) const = 0¶

Returns the current value of the managed variable.

The call interface should return an optional, which should be set to std::nullopt if the managed variable cannot be accessed (for instance if it does not exists during the call scope, for example if it’s a dynamic algorithm parameter that is not currently configured).

Parameters:: log_info – The current problem state data.
Returns:: An optional that holds a double if the variable is available, std::nullopt else.

inline std::string name() const¶: Configured name accessor.

inline std::string format() const¶: Configured format accessor.

inline void set_name(const std::string &name)¶: Accessor for name.

virtual ~Property() = default¶: Destructor.

inline virtual std::string call_to_string(const logger::Info &log_info, const std::string &nan = "") const¶

Method to parse the log data into a string.

Parameters:

log_info – The current problem state data.
nan – The value to log when there is no data.

Returns:

a string representation of the properties’ data

Protected Attributes

std::string name_¶: Unique name for the logged property.

const std::string format_¶: format specification for fmt

Store¶

class Store : public ioh::logger::Watcher ¶

A logger that stores all the possible information in-memory, in nested maps.

The order of the entry keys is: suite_name > problem_id > n_variables > instance > run_id > evaluation > property_name > property_value In JSON-like format: {suite_name: {problem_id: {n_variables: {instance: {run_id: {evaluation: {property_name : property_value}}}}}}}

You would access a given property value like:

logger::Store store({log::TransformedY()});
// [Attach to a problem and run...]

auto prop = store.data["None"][1][10][2][0][1234]["transformed_y"];
// Or alternatively:
logger::Store::Cursor cur("None",1,10,2,0,1234);
const std::string prop_name = TransformedY().name();
logger::Store::Value prop = store.data(cur)[prop_name];

// Then test if the property has a value:
if(prop) {
    std::cout << prop_name << ": " << prop << std::endl;
} else {
    std::cout << prop_name << " does not exists in this context" << std::endl;
}

Note

If track_suite has never been called, the default suite name is Store::default_suite (“None”);

Data structure types

Convenience naming for the underlying nested data structure.

using Value = std::optional<double>¶: value

using Attributes = std::map<std::string, Value>¶: name => value

using Run = std::map<size_t, Attributes>¶: evaluations => Attributes

using Runs = std::map<size_t, Run>¶: run id => Run

using Instances = std::map<int, Runs>¶: instance id => runs

using Dimensions = std::map<int, Instances>¶: nb of dim => instances

using Problems = std::map<int, Dimensions>¶: pb id => dimension

using Suites = std::map<std::string, Problems>¶: suite name => problems

Public Functions

inline Suites data()¶: Direct accessor to the data structure.

inline Attributes data(const Cursor &current)¶: Access a map of property values with a Cursor.

inline Value at(const Cursor &current, const std::string &property_name)¶: Access a property value with a Cursor and the property name.

inline Value at(const Cursor &current, const Property &property)¶: Access a property value with a Cursor and the Property itself.

inline Store(std::vector<std::reference_wrapper<logger::Trigger>> triggers, std::vector<std::reference_wrapper<logger::Property>> Attributes)¶: The logger::Store should at least track one logger::Property, or else it makes no sense to use it.

inline virtual void attach_problem(const problem::MetaData &problem) override¶

Track a problem/instance/dimension and/or create a new run.

Problem, instance and dimension are given in the argument. Run and evaluations number are managed internally.

Note

A new run is created everytime this is called.

inline virtual void attach_suite(const std::string &suite_name) override¶: Set the current suite name.

Note

If this is never called, the suite name defaults to Store::default_suite (“None”).

inline virtual void call(const logger::Info &log_info) override¶: Atomic log action.

Public Static Attributes

static const std::string default_suite = "None"¶: When attached directly to a Problem (out of a Suites), use the following key for the Suites map.

Protected Functions

inline Attributes &current_attributes()¶: Accessor to the current attributes map.

Protected Attributes

Suites _data¶: The main data structure in which log events are stored.

Cursor _current¶: The current Cursor.

struct Cursor¶

A set of keys leading to a set of property values within the data structure.

Public Functions

inline Cursor(const std::string &suite_name = default_suite, int problem_id = 0, int dimension = 0, int instance_id = 0, size_t run_id = 0, size_t evaluation_id = 0)¶

Construct a new Cursor object.

Parameters:

suite_name – Suite name
problem_id – problem id
dimension – Dimension
instance_id – Problem instance
run_id – run id
evaluation_id – evaluation number

Public Members

std::string suite¶: Suite name.

int pb¶: problem id

int dim¶: Dimension.

int instance¶: Problem instance.

size_t run¶: run id

size_t evaluation¶: evaluation number

Watcher¶

class Watcher : public ioh::Logger ¶

Interface for a Logger to which the user can add properties to be watched.

Subclassed by ioh::logger::FlatFile, ioh::logger::Store

Public Functions

inline Watcher(std::vector<std::reference_wrapper<logger::Trigger>> when, std::vector<std::reference_wrapper<logger::Property>> what)¶: Use this constructor if you just need to trigger a log event if ANY of the triggers are fired.

Warning

Do not pass references to members of your derived class, as this would pass a reference to an unitiliazed member, the base class constructor being always called first. Instead, use the empty Watcher() constructor and then add the properties manually in your own constructor.

inline Watcher(trigger::Set &when, std::vector<std::reference_wrapper<logger::Property>> what)¶

Use this constructor if you want to combine triggers differently.

You can for instance combine your triggers in a trigger::All, so that the log event would be triggered only if ALL triggers are fired at once.

inline Watcher()¶: Use this constructor if you just need the interface without triggers or properties, for instance if you define default triggers/properties in your own constructor.

Note

If you manage your own default triggers/properties, use _properties.insert_or_assign(name, std::ref(tp)); instead of classical access operators.

inline virtual void watch(logger::Property &property)¶: Adds a property to be logged.

Structs¶

Info¶

struct Info¶

Information about the current log.

Note

The properties for bests values holds a state since the first start or the last call to reset.

Note

“transformed” indicates that a monotonic transformation is applied, which is specific to the currently configured problem instance.

Note

If there is an improvement, then fields holding “best” values will have the same value than the others.

Public Functions

template<typename T> inline void update(const problem::State<T, double> &state, const problem::ConstraintSet<T> &constraintset)¶

update the log info based on the constraint and state of the problem

Template Parameters:

T – the type of the problem

Parameters:

state – the state of the problem
constraintset – the set of constraints for the problem

template<typename T> inline void allocate(const problem::Solution<T, double> &opt, const problem::ConstraintSet<T> &constraintset)¶

allocate static (during a run) values for the log info

Template Parameters:

T – the type of the problem

Parameters:

opt – the optimum of the problem
constraintset – the set of constraints for the problem

Public Members

size_t evaluations¶: Number of evaluations of the objective function so far.

double raw_y¶: The current value.

double raw_y_best¶: The current best internal objective function value (since the last reset).

double transformed_y¶: The current transformed objective function value.

double transformed_y_best¶: The current best transformed objective function value (since the last reset).

double y¶: The current transformed objective function value with constraints applied.

double y_best¶: The current best transformed objective function value with constraints applied (since the last reset).

std::vector<double> x¶: Current search space variables.

std::vector<double> violations¶: Constraint violations, first element is total violation by ContraintSet.

std::vector<double> penalties¶: Applied penalties by constraint, first element is total penalty applied by ContraintSet.

problem::Solution<double, double> optimum¶: Optimum to the current problem instance, with the corresponding transformed objective function value.

bool has_improved¶: Single objective check whether state-update has caused an improvement.

Trigger¶

struct Trigger¶

Interface for classes triggering a log event.

Subclassed by ioh::trigger::Always, ioh::trigger::At, ioh::trigger::During, ioh::trigger::Each, ioh::trigger::OnDeltaImprovement, ioh::trigger::OnImprovement, ioh::trigger::OnViolation, ioh::trigger::Set

Public Functions

virtual bool operator()(const logger::Info &log_info, const problem::MetaData &pb_info) = 0¶

Returns:: true if a log event is to be triggered given the passed state.

inline virtual void reset()¶

Reset any internal state.

Useful if, for instance, the trigger maintain its own “best value so far” (

Functions¶

EAF¶

EAF()¶

A logger that store the 2D quality/time attainment fronts for each runs.

An attainment front is the set of Pareto-optimal quality/time targets attained during a given run. That is, a set of points that are non-dominated: either the quality or the time is better than the other points in set.

The set of attainment fronts samples a 2D distribution of probability, which is a generalization of performance distribution of the algorithm(s) observed by the logger(s). Taking the convex subset of the projections of the non-dominated points for a given quality target would recover the sample of the expected runtime empirical cumulative density function.

The underlying empirical attainment function levelsets can be computed with stat::Levels.

More information in the following publication:

        @incollection{LopPaqStu09emaa,
          editor = { Thomas Bartz-Beielstein  and  Marco Chiarandini  and  Lu{\'\i}s Paquete  and  Mike Preuss },
          year = 2010,
          address = {Berlin, Germany},
          publisher = {Springer},
          booktitle = {Experimental Methods for the Analysis of Optimization Algorithms},
          author = { Manuel L{\'o}pez-Ib{\'a}{\~n}ez  and  Lu{\'\i}s Paquete  and  Thomas St{\"u}tzle },
          title = {Exploratory Analysis of Stochastic Local Search Algorithms in Biobjective Optimization},
          pages = {209--222},
          doi = {10.1007/978-3-642-02538-9_9},
          abstract = {This chapter introduces two Perl programs that
                      implement graphical tools for exploring the performance of stochastic local search algorithms
                      for biobjective optimization problems. These tools are based on the concept of the empirical attainment
                      function (EAF), which describes the probabilistic distribution of the outcomes obtained by a
                      stochastic algorithm in the objective space. In particular, we consider the visualization of
                      attainment surfaces and differences between the first-order EAFs of the outcomes of two
                      algorithms. This visualization allows us to identify certain algorithmic behaviors in a graphical way.
                      We explain the use of these visualization tools and illustrate them with examples arising from
                      practice.}
        }
      /
    class EAF : public Logger {
    public:
        using Runs       = std::map<size_t     , eaf::Front>;
        using Instances  = std::map<int        , Runs      >;
        using Dimensions = std::map<int        , Instances >;
        using Problems   = std::map<int        , Dimensions>;
        using Suites     = std::map<std::string, Problems  >;
        inline static const std::string default_suite = "None";

        [[nodiscard]] const Suites& data() const {return _data;}

        struct Cursor {
            std::string suite;
            int pb;
            int dim;
            int ins;
            size_t run;
            Cursor(std::string suite_name = default_suite,
                    int problem_id = 0,
                    int dimension = 0,
                    int instance_id = 0,
                    size_t run_id = 0)
            : suite(suite_name),
              pb(problem_id),
              dim(dimension),
              ins(instance_id),
              run(run_id)
            { }
        };

        [[nodiscard]] const eaf::Front& data(const Cursor& cur) const
        {
#ifndef NDEBUG
            assert(_data.count(cur.suite) > 0);
            assert(_data.at(cur.suite).count(cur.pb) > 0);
            assert(_data.at(cur.suite).at(cur.pb).count(cur.dim) > 0);
            assert(_data.at(cur.suite).at(cur.pb).at(cur.dim).count(cur.ins) > 0);
            assert(_data.at(cur.suite).at(cur.pb).at(cur.dim).at(cur.ins).count(cur.run) > 0);
#endif
            return _data.at(cur.suite).at(cur.pb).at(cur.dim).at(cur.ins).at(cur.run);

        }

    public:

attach_problem¶

void ioh::logger::attach_problem(const problem::MetaData &problem) override¶: Attach to a problem.

attach_suite¶

virtual void ioh::logger::attach_suite(const std::string &suite_name) override¶: Set the current suite name.

Note

If this is never called, the suite name defaults to Store::default_suite (“None”).

call¶

void ioh::logger::call(const logger::Info &log_info) override¶: Process a log event.

current_front¶

eaf::Front &ioh::logger::current_front()¶: Accessor to the current front.

optimization_type¶

common::OptimizationType ioh::logger::optimization_type() const¶: Getter for the optimizetion type.

reset¶

void ioh::logger::reset() override¶: Reset the state.

Variables¶

DEFAULT_DOUBLE_FORMAT¶

std::string ioh::logger::DEFAULT_DOUBLE_FORMAT = "{:.10f}"¶: Default format for storing doubles.

FULL_PRECISION_DOUBLE_FORMAT¶

const std::string ioh::logger::FULL_PRECISION_DOUBLE_FORMAT = "{:.17g}"¶: Format for storing doubles with round-trip precision (used by FlatFile::set_full_precision)