Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Introduction

Has it ever happened to you when you run experiments, change some things, run it again, then, few hours or days later, you have worst performances than before, but you don't remember what you changed? Your search for hours before finding it, or even worse, never find it again?

This is why I created Deepboard, a modern experiment tracking tool to help you organise your deep learning / machine learning research. We currently find ourselves overwhelmed by a lot of experiment tracking tools such as Weights & Biases, ClearML, MLFlow, Tensorboard, etc., so why a new one? I believe these tools can be split into two main categories: full ML lifecycle management and lightweight experiment tracking. Popular proprietary tools such as Weights & Biases and ClearML and the open source MLFlow are designed to manage the full ML lifecycle. On the other side, tensorboard is a lightweight experiment tracking tool as it focuses on logging and visualizing experiment results, and comparing them. Tensorboard have been introduced quite some time ago, and while it is still a great tool, I believe it lacks some useful features. Deepboard aims to replace tensorboard as a lightweight experiment tracking tool, while providing a modern and user-friendly interface, and some useful features missing in the classic Tensorboard.

With deepboard, all your logs are stored locally in a single database file, similar to tensorboard that stores the logs in binary files. This mean you do not have to be connected to internet, or send your data to a cloud service, everything is local. Deepboard uses a sqlite3 database, which makes its format transparent easy to work with. In addition, it comes with a public api to read the logs programmatically, which is not the case with tensorboard. It also comes with a beautiful and modern web interface to visualize and compare your experiments.

However, where deepboard really separates itself from others is with its unique designed philosophy around reproducibility. Deepboard is designed not only to track and visualize your experiments, but also to allow you to reproduce the results of any previously run experiment.

Reproducibility

In this tutorial, we’ll learn how to use the deepboard tool. Before diving in, we’ll first discuss the best practices for using it effectively. Since the main purpose of deepboard is to ensure the reproducibility of previously run and reported experiments, adopting a few coding habits will help you make full use of its features.

Deterministic philosophy

The main idea is that an experiment with the same configuration should always run the same code. For example, let's say we first run a baseline experiment and get some results. Then, we can add a new feature that improves our performances ... or degrades them. In the last case, we might want to revert back to the previous version of the code. If we naively implemented the new feature to always run, we would not be able to reproduce the previous results. Of course, we could comment out the new feature, but this does not scale well when we have many features to try. Instead, we should implement the new feature in an optional fashion, and activate it when a certain parameter is set. For example, we could test the new feature when passing a flag to our program with a command line argument. When the flag is not present, we run our baseline code.

Let's look at a simple code example:

main.py — Baseline implementation :

def experiment():
    # We start by loading the data.
    data = load_data()
    # Then, we create the model.
    model = Model()
    # Finally, we train and evaluate the model.
    return train_and_evaluate(model, data)

Run it with:

python main.py

Next, we implement a new feature that will be activated when passing a flag: main.py — New feature implementation :

# Note that we would usually parse the cli, then pass the flag to the experiment function.
def experiment(new_feature: bool = False):
    # Let's imagine our new feature is model-specific.
    # We start by loading the data.
    data = load_data()
    # Then, we create the model.
    if new_feature:
        model = ModelWithNewFeature()
    else:
        model = Model()
    # Finally, we train and evaluate the model.
    return train_and_evaluate(model, data)

Run it with:

python main.py --new_feature
# We can also run the original experiment without the new feature.
python main.py

By following this coding habit, you can easily reproduce previously run experiments by simply passing the same configuration. Note that most deep learning algorithms are nondeterministic by nature, which can lead to variations in the final performance. However, by using this methodology, you can ensure that it is, in fact, the same code that is being run, and that any differences with the previous run is due to the inherent randomness, not due to code changes. If you prefer even more reproducibility, nothing stops you from setting seeds and make the algorithms more deterministic.

Configuration management

Since the configuration of each experiment is a crucial aspect in the deepboard way, we determined two main types of configurations: configuration file and command line arguments. We recommend to put most parameters in the configuration file, and use command line arguments only for parameters that are likely to change often.

For new features, we recommend adding a new parameter in the configuration file, and add a default value for it when it is not present.

Mistakes can happen

Despite our best efforts, mistakes can still happen, and we might change the code without making it optional through a parameter. This will render prior experiments unreproducible since not the same code will be run with the same configuration. To mitigate this risk, deepboard automatically saves the git commit hash and the diff at each experiment run. This makes it easier to revert back to a previous version of the code if needed.

While it is possible to revert back to a previous commit, we recommend not to rely on this feature, and consider it as a safety net.

Getting Started

First, install the package with the GUI with pip:

pip install deepboard[full]

The deepboard package is divided into two main components: the experiment tracking one called ResultTable, and the GUI application. You will interact with the ResultTable api in your code to log your experiments, and then use the GUI to visualize and analyze them.

Creating a new experiment

from deepboard.resultTable import ResultTable

# Step 1: Create / load the result table
rTable = ResultTable()

# Step 2: Create a new run
run = rTable.new_run("My Experiment")

# Step 3: Write the final results
run.write_result(accuracy=0.98, loss= 0.0257)

Here, we first create a new ResultTable instance. If no database exist at the specified path, a new result table, a database, will be created. Then, we create a new entry in the table with the new run command. We can run our experiment, then write the final results to the result table. By default, it creates a file named result_table.db. You can visualize the results in the GUI by running and opening in your web browser.

deepboard result_table.db

Experiment uniqueness and debug mode

Note that if you try to run the previous script again, it will raise an error saying that the experiment has already been run with runID 1. This is because each experiment must be unique in the result table. To avoid getting this constraint, you can run in debug mode, which will give the same runID (-1) to every run, and each run will overwrite the previous one. You can simply call the new_debug_run method instead of new_run, and the rest of the api remains the same, so we can use the same code as before.

run = rTable.new_debug_run("My Experiment")

The Run ID

As every run must be unique, each run is given a unique id called the RunID. You can retrieve this id by accessing the run_id attribute of the LogWriter object (the run variable in the previous examples). Since this id is unique, you can use it for whatever you want, for example, you could save checkpoints to a directory, where each checkpoint file or directory is named with the corresponding RunID.

Note:
When an experiment fails (an exception is raised), the ResultTable will detect it, and the run won't be considered when checking uniqueness. This means that you can re-run failed experiments without changing anything, and the run will be accepted as a new unique experiment.

It is useful to log final results of experiments to compare them later, but we often want to log intermediate results during training. This topic is explored in the next chapter.

Logging

With the result table, we can also log information during runs. For example, we can log the loss and accuracy at each step or epoch. We can also log images, raw text, and even html artefacts for more flexibility. It even automatically detects matplotlib figures and logs them. All of these results can be visualized in the GUI, and are all stored in a single file: the result table database.

Log scalars

You can log scalar values such as loss, accuracy, or any other metric use the add_scalar method of the LogWriter object. Here is an example of logging loss and accuracy at each epoch:

from deepboard.resultTable import ResultTable

rTable = ResultTable("results.db")

# Create a new LogWriter object named run.
run = rTable.new_debug_run("My Experiment")

loss = 1.
accuracy = 0.
for epoch in range(10):
    for step in range(100):
        global_step = epoch * 100 + step
        loss *= 0.9
        accuracy = 1 - loss
        run.add_scalar("Step/loss", loss, step=global_step, epoch=epoch)
    
    run.add_scalar("Train/loss", loss, step=global_step, epoch=epoch)
    run.add_scalar("Train/accuracy", accuracy, step=global_step, epoch=epoch)

run.write_result(accuracy=accuracy)

Let's explore the code above. We simulate three epochs and 100 steps per epoch. At each step, we log the loss and accuracy using the add_scalar method. The first argument is the tag, which is a string composed of two parts separated by a slash. The first part is the category (e.g., "Train"), and the second part is the name of the metric. You will see in the GUI that metrics of the same category are grouped together when comparing runs, and metrics with the same name, but in different categories are found in the same figure in the single view. The second argument is the value to log, and the last two arguments are optional: step is the global step number, and epoch is the current epoch number. If you do not provide the global step, it will be automatically incremented at each call to add_scalar with a specific tag.

Log artefacts (images, html fragments, text, etc)

You can also log various type of artefacts, which share a similar api. Here, we will make an example where we log an image, and a html hello world fragment to show the concept.

from PIL import Image
import numpy as np

...
# Add this in the previous loop

# We create a random image
img = (np.random.randn(256, 256, 3) * 128).astype(np.uint8)
img = Image.fromarray(img)
run.add_image(img, tag="Random Image", step=global_step, epoch=epoch, flush=True)

fragment = "<h1>Hello World</h1><p>This is a html fragment logged from Deepboard.</p>"
run.add_fragment(fragment, tag="Dummy text", step=global_step, epoch=epoch, flush=True)

Here, we introduce two new parameters: tag and flush. The tag parameter is optional for these methods, and is used to categorize artefacts. Adding tags to artefacts allows you to define more precise filters in the GUI. The flush parameter will be explained later in the "Automatic cache flushing" section.

Matplotlib figures

When you plot figures using matplotlib, the LogWriter object will automatically detect them and log them for you in the result table. Here is an example:

import matplotlib.pyplot as plt

...
# Add this in the previous loop

plt.plot([1, 2, 3], [1, 4, 9])
plt.title("Sample Plot")

run.write_results(accuracy=0.98)

Figures are detected when a method is called on the LogWriter object, such as add_scalar, add_image, etc. If you plot a figure and never call any method after, the figure will not be logged. Fortunately, you can force the figures to be detected by calling the detect_and_log_figures method. This method support similar parameters as the other artefact logging methods, such as tag, step, epoch, and flush.

Example:

plt.plot([1, 2, 3], [1, 4, 9])
plt.title("Sample Plot")
run.detect_and_log_figures()

Tip:
If you want to add a tag to your figures, you can call the detect_and_log_figures method with the tag parameter right after plotting. This will prevent automatic detection and will assign your tag to the figure.

Automatic memory flushing

By default, the LogWriter object keeps in memory recently logged data, and only writes them to the database once in a while. This is done to improve performances. The parameters that controls the number of logged data kept in memory is determined when creating the new run with the flush_each parameter. If you log less than this parameter value, you may not see the logged data in the GUI. To tackle this, you can set the flush parameter to True when logging data, which will force flushing everything in memory to the database. As you will write to disk, note that this operation is slow, so use it sparingly.

In addition, calling write_result method automatically flushes everything in memory to the database. This method should always be called at the end of your experiment.

Note:
Each run has a state: "running", "finished" or "failed". When creating a new run, its state is set to "running". When you call the write_result method, the state is set to "finished". If an exception occurs during your experiment, the state is set to "failed". In order to get the right state, and flushes all the logs, make sure to call the write_result method at the end of your experiment.

IMPORTANT:
Always call the write_result method at the end of your experiment to ensure that all logged data is saved to the database.

Parametrization and Configuration

We said earlier that each non-debug experiment must be unique. By default, the uniqueness is defined by the combination of multiple elements:

  • The configuration file
  • The command line arguments
  • The tag
  • Comment

Configuration file and command line arguments

You can provide a configuration file path, and command line arguments passed by the user, when creating a new run. Any changes in these two elements are detected, and considered when checking for uniqueness. This means that you can run multiple time the same script, but change the configuration file or command line arguments, and the ResultTable will accept them as different runs.

Example:

from deepboard.resultTable import ResultTable

rTable = ResultTable("experiments.db")
run = rTable.new_run(
    config_path="config.yaml",
    cli=dict(learning_rate=0.001, batch_size=32)
)

Run Tag

You can provide a tag when creating a new run. The tag can be any string. I recommend using tags to separate between different sub-experiments. Tags are also considered when checking for uniqueness. For example, you can run the same experiment with the same configuration file and command line arguments, but with different tags, and the ResultTable will accept them as different runs.

Comment

Finally, you can also provide a comment when creating a new run. The comment can be used to textually describe what you changed in this run, in case the ResultTable refuse to create a new run because it believes it not unique.

For example:

run1 = rTable.new_run(
    config_path="config.yaml",
    cli=dict(learning_rate=0.001, batch_size=32)
)
run2 = rTable.new_run(
    config_path="config.yaml",
    cli=dict(learning_rate=0.001, batch_size=32),
    comment="This will run because of the comment"
)

In the previous example, even if both run are the same, the second run will be accepted because of the comment. However, usually you should add a comment for a run that is different, and use it to describe what you changed in this run.

Hyperparameters

You can log hyperparameters that you change between runs to the result table using the add_hparams method on the LogWriter object. You can provide any arguments as key-value pairs, and they will be stored in the database.

For example:

run = rTable.new_run(
    config_path="config.yaml",
    cli=dict(learning_rate=0.001, batch_size=32)
)
run.add_hparams(learning_rate=0.001, batch_size=32, optimizer="adam")

With the previous example, you will see in the GUI that new columns were added to the table with those hyperparameters. In addition, a new tab will be created containing all those hyperparameters for easier visualization.

Advanced Features

TODO

Multiple run repetitions

TODO

Commit flags

You can prevent the ResultTable from accepting a new run if there is uncommitted changes in the git repository. You can do this by setting the nocommit_action when loading the ResultTable. The possible values are:

  • NoCommitAction.NOP: Ignore, do nothing
  • NoCommitAction.WARN: Print a warning message if there are uncommitted changes (default behavior)
  • NoCommitAction.RAISE: Raise an error and prevent the new run from being added

Custom unique columns

TODO

Class: ResultTable

Description: This class represents all the results. There are a lot of method to interact with the resultTable (the database).

All actions performed by the GUI (DeepBoard) are available by public methods to get a programmatic access.

How to use:

  • First, specify a path the result table. If the db was not created, it will be created automatically.

  • Then, create a run with all the specific parameters describing the run. A unique run_id will be generated. Note that each run must be unique. This security allows more reproducible runs. If one run perform better than the others, you can run the code again with all the parameters in the result table and you should get the same results.

  • If you simply want to test your code, you can create a debug run. It won't create a permanent entry in the table. You will still be able to see the logged scalars in the GUI with the run_id -1 that is reserved for debug runs. This run will be overwritten by the next debug run.

  • Finally, you can interact with the table with the different available methods.

Method: get_config()

get_config(self, run_id: int) -> str

Description: Load the configuration file of a given run id

Parameters:

  • run_id: The run id

Return:

  • The path to the configuration file

Method: get_experiment_raw()

get_experiment_raw(self, run_id: int) -> Dict[str, Any]

Description: Load the row of an experiment. It will return a dictionary with the keys being the column names and the values the actual values. Note that this does not perform any other operations than fetch in the database. This means that it will also show columns that were hidden.

Parameters:

  • run_id: The run id to fetch

Return:

  • The raw row of an experiment

Method: get_file_hash()

get_file_hash(configfile: Union[str, IO[str]], hash_algo: str = 'sha256') -> str

Description: Returns the hash of the file at file_path using the specified hashing algorithm.

Method: get_hidden_runs()

get_hidden_runs(self) -> List[int]

Description: Get the list of all hidden run ids.

Return:

  • A list of run ids associated to hidden runs.

Method: get_image_by_id()

get_image_by_id(self, image_id: int) -> Optional[PIL.Image.Image]

Description: If the image_id is valid, it will return the image as a PIL Image object.

Parameters:

  • image_id: The id of the image to fetch.

Return:

  • The image as a PIL Image object or None if the image_id is not valid.

Method: get_results()

get_results(self, run_id: Optional[int] = None, show_hidden: bool = False) -> deepboard.resultTable.resultTable.TableTuple

Description: This function will build the result table and return it as a list. It will also return the column names and their unique id. It will not return the columns that were hidden and will format the table to respect the column order. By default, it does not include hidden runs, but they can be included by setting the show_hidden. You can also get a single row by passing a run_id to the method.

Parameters:

  • run_id: the run id. If none is specified, it fetches all results
  • show_hidden: Show hidden runs.

Return:

  • A list of columns names, a list of column ids, a list of whether they are hparam or not, and a list of rows

Method: hide_column()

hide_column(self, column: str)

Description: Hide a column in the result table.

Parameters:

  • column: The column name to hide.

Method: hide_run()

hide_run(self, run_id: int)

Description: Instead of deleting runs and lose information, you can hide it. It will not be visible in the default view of the result Table, however, it can be unhidden if it was a mistake.

Parameters:

  • run_id: The run id to hide

Method: load_run()

load_run(self, run_id) -> deepboard.resultTable.logwritter.LogWriter

Description: Load a specific run's LogWriter in read-only mode.

Parameters:

  • run_id: The run id

Return:

  • The logWriter bound to the run

Method: new_debug_run()

new_debug_run(self, experiment_name: str, configfile: Union[str, pathlib._local.PurePath, IO[str], NoneType] = None, cli: Optional[dict] = None, comment: Optional[str] = None, tag: str = '', flush_each: int = 10, keep_each: int = 1, auto_log_plt: bool = True, disable: bool = False) -> deepboard.resultTable.logwritter.LogWriter

Description: Create a new DEBUG socket to log the results. The results will be entered in the result table, but as the runID -1. This means that everytime you run the same code, it will overwrite the previous one. This is useful to avoid adding too many rows to the table when testing the code or debugging.

Note: It will not log the git diff or git hash

Parameters:

  • experiment_name: The name of the current experiment
  • configfile: The path to the configuration path
  • cli: The cli arguments
  • comment: The comment, if any
  • tag: The tag of the run
  • flush_each: Every how many logs does the logger save them to the database?
  • keep_each: If the training has a lot of steps, it might be preferable to not log every step to save space and speed up the process. This parameter controls every how many step we store the log. 1 means we save at every steps. 10 would mean that we drop 9 steps to save 1.
  • auto_log_plt: If True, automatically detect if matplotlib figures were generated and log them. Note that it checks only when a method on the socket is called. It is better to log them manually because you can set the appropriate step, epoch and split.
  • disable: If true, disable the logwriter, meaning that nothing will be written to the database.

Return:

  • The log writer

Method: new_run()

new_run(self, experiment_name: str, configfile: Union[str, pathlib._local.PurePath, IO[str], NoneType] = None, cli: Optional[dict] = None, comment: Optional[str] = None, tag: str = '', flush_each: int = 10, keep_each: int = 1, auto_log_plt: bool = True, disable: bool = False) -> deepboard.resultTable.logwritter.LogWriter

Description: Create a new logwritter object bound to a run entry in the table. Think of it as a socket.

Parameters:

  • experiment_name: The name of the current experiment
  • configfile: The path to the configuration path, or the file-like object containing the configuration
  • cli: The cli arguments
  • comment: The comment, if any
  • tag: The tag of the run
  • flush_each: Every how many logs does the logger save them to the database?
  • keep_each: If the training has a lot of steps, it might be preferable to not log every step to save space and speed up the process. This parameter controls every how many step we store the log. 1 means we save at every steps. 10 would mean that we drop 9 steps to save 1.
  • auto_log_plt: If True, automatically detect if matplotlib figures were generated and log them. Note that it checks only when a method on the socket is called. It is better to log them manually because you can set the appropriate step, epoch and split.
  • disable: If true, disable the logwriter, meaning that nothing will be written to the database.

Return:

  • The log writer

Method: set_column_alias()

set_column_alias(self, columns: Dict[str, str])

Description: Set the alias of the column in the result table.

Parameters:

  • columns: A dict of column name and their alias. The alias is the name displayed in the table.

Method: set_column_order()

set_column_order(self, columns: Dict[str, Optional[int]])

Description: Set the order of the column in the result table. If order is None, it will be set to NULL

Parameters:

  • columns: A dict of column name and their order. The order is the index of the column in the table. If the order is None, it will be set to NULL and be hidden

Method: show_column()

show_column(self, column: str, order: int = -1)

Description: Show a column in the result table. If order is -1, it will be set to the last column.

Method: show_run()

show_run(self, run_id: int)

Description: This method unhide a run that has been hidden. It undo the operation performed by hide_run.

Parameters:

  • run_id: The run id to show

Method: to_pd()

to_pd(self, get_hidden: bool = False) -> pandas.core.frame.DataFrame

Description: Export the table to a pandas dataframe.

Parameters:

  • get_hidden: If True, it will include the hidden runs.

Return:

  • The table as a pandas dataframe.

Class: LogWriter

Description: This class makes an object that is bound to a run row in the result table. This means that everything that is logged through this object is added into the result table and this object can be used to interact with a specific run. This object is single use. This means that once the final results are written, the object becomes read-only.

You should not instantiate this class directly, but use the ResultTable class to create it instead.

Method: add_fragment()

add_fragment(self, content: str, step: Optional[int] = None, tag: Optional[str] = None, epoch: Optional[int] = None, flush: bool = False)

Description: Add a html fragment to the resultTable

scalar steps.

Parameters:

  • content: Must be a string containing valid HTML content.
  • step: The global step at which the image was generated. If None, the maximum step is taken from all global
  • tag: A tag describing the fragment.
  • epoch: The epoch at which the image was generated. If None, no epoch is saved.
  • flush: If True, flush all data in memory to the database.

Method: add_hparams()

add_hparams(self, param_dict: Optional[dict[str, Any]] = None, **kwargs)

Description: Add hyperparameters to the result table

Parameters:

  • param_dict: The hyperparameters to add as a dict.
  • kwargs: The hyperparameters to save

Method: add_image()

add_image(self, image: Union[bytes, PIL.Image.Image], step: Optional[int] = None, tag: Optional[str] = None, epoch: Optional[int] = None, flush: bool = False)

Description: Add an image to the resultTable

steps.

Parameters:

  • image: Must be png bytes or a PIL Image object.
  • step: The global step at which the image was generated. If None, the maximum step is taken from all global
  • tag: A tag describing the image.
  • epoch: The epoch at which the image was generated. If None, no epoch is saved.
  • flush: If True, flush all data in memory to the database.

Method: add_scalar()

add_scalar(self, tag: str, scalar_value: Union[float, int], step: Optional[int] = None, epoch: Optional[int] = None, walltime: Optional[float] = None, flush: bool = False)

Description: Add a scalar to the resultTable

Parameters:

  • tag: The tag, formatted as: 'split/name' or simply 'split'
  • scalar_value: The value
  • step: The global step. If none, the one calculated is used
  • epoch: The epoch. If None, none is saved
  • walltime: Override the wall time with this
  • flush: Force flush all the scalars in memory

Method: add_text()

add_text(self, text: str, step: Optional[int] = None, tag: Optional[str] = None, epoch: Optional[int] = None, flush: bool = False)

Description: Add a text sample to the resultTable

scalar steps.

Parameters:

  • text: Must be a string
  • step: The global step at which the image was generated. If None, the maximum step is taken from all global
  • tag: A tag describing the text.
  • epoch: The epoch at which the image was generated. If None, no epoch is saved.
  • flush: If True, flush all data in memory to the database.

Method: detect_and_log_figures()

detect_and_log_figures(self, step: Optional[int] = None, tag: Optional[str] = None, epoch: Optional[int] = None, flush: bool = False, close_figs: bool = False)

Description: Detect matplotlib figures that are currently open and log them to the result table. (Save them as png).

steps.

Parameters:

  • step: The global step at which the image was generated. If None, the maximum step is taken from all global
  • tag: A tag describing the figures.
  • epoch: The epoch at which the images were generated. If None, no epoch is saved.
  • flush: If True, flush all data in memory to the database.
  • close_figs: If True, close all figures after detection.

Method: get_figures()

get_figures(self, id: Optional[int] = None, step: Optional[int] = None, tag: Optional[str] = None, epoch: Optional[int] = None, repetition: Optional[int] = None)

Description: Return all figures logged in the run with the given step, tag and/or epoch.

Parameters:

  • id: The id of the figure to read. If None, all figures are returned.
  • step: The step at which the figure was generated. If None, all figures are returned.
  • tag: The tag describing the figures. If None, all tags are returned.
  • epoch: The epoch at which the figures were generated. If None, all epochs are returned.
  • repetition: The repetition of the figures. If None, all figures are returned.

Method: get_fragment()

get_fragment(self, id: Optional[int] = None, step: Optional[int] = None, tag: Optional[str] = None, epoch: Optional[int] = None, repetition: Optional[int] = None)

Description: Return all html fragments logged in the run with the given id, step, tag and/or epoch.

Parameters:

  • id: The id of the html fragment to read
  • step: The step at which the html fragment was generated. If None, all html fragment are returned.
  • tag: The tag describing the fragment. If None, all tags are returned.
  • epoch: The epoch at which the html fragments were generated. If None, all epochs are returned.
  • repetition: The repetition of the run. If None, all html fragment are returned.

Return:

  • A list of html fragment

Method: get_hparams()

get_hparams(self) -> Dict[str, Any]

Description: Get the hyperparameters of the current run

Return:

  • A dict of hyperparameters

Method: get_images()

get_images(self, id: Optional[int] = None, step: Optional[int] = None, tag: Optional[str] = None, epoch: Optional[int] = None, repetition: Optional[int] = None) -> List[dict]

Description: Return all images logged in the run with the given step, tag and/or epoch.

Parameters:

  • id: The id of the image to read
  • step: The step at which the image was generated. If None, all images are returned.
  • tag: The tag in which the images were generated. If None, all tags are returned.
  • epoch: The epoch at which the images were generated. If None, all epochs are returned.
  • repetition: The repetition of the images. If None, all images are returned.

Return:

  • A list of image bytes

Method: get_repetitions()

get_repetitions(self) -> List[int]

Description: Get the all the repetitions ids of the current run

Return:

  • A list of repetitions ids

Method: get_scalar()

get_scalar(self, tag) -> List[deepboard.resultTable.scalar.Scalar]

Description: Read a scalar from the resultTable with the given tag

Parameters:

  • tag: The tag to read formatted as: 'split/name' or simply 'split'.

Return:

  • A list of Scalars items

Method: get_text()

get_text(self, id: Optional[int] = None, step: Optional[int] = None, tag: Optional[str] = None, epoch: Optional[int] = None, repetition: Optional[int] = None)

Description: Return all text samples logged in the run with the given id, step, tag and/or epoch.

Parameters:

  • id: The id of the text sample to read
  • step: The step at which the text was generated. If None, all text samples are returned.
  • tag: The tag describing the text. If None, all tags are returned.
  • epoch: The epoch at which the texts were generated. If None, all epochs are returned.
  • repetition: The repetition of the run. If None, all text samples are returned.

Return:

  • A list of text samples

Method: new_repetition()

new_repetition(self)

Description: Create a new repetition of the current run. This is useful if you want to log multiple repetitions of the same run. This is a mutating method, meaning that you can call it at the end of the training loop before the next full training loop is run again.

Method: set_color()

set_color(self, color: Optional[str] = None)

Description: Set the color of the run

Parameters:

  • color: The color to set (hex format without #, e.g. 'ff0000' for red)

Method: set_note()

set_note(self, note: str)

Description: Update the note of the run

Parameters:

  • note: The note to set (will overwrite the previous one)

Method: set_status()

set_status(self, status: Literal['running', 'finished', 'failed'])

Description: Manually set the status of the run

Parameters:

  • status: The status to set

Method: set_tag()

set_tag(self, tag: str)

Description: Update the tag of the run

Parameters:

  • tag: The tag to set (will overwrite the previous one)

Method: write_result()

write_result(self, **kwargs)

Description: Log the results of the run to the table, then disable the logger. This means that the logger will be read-only after this operation. If you run multiple iterations, consider writing the results only once all the runs are finished. You can aggregate the different metrics before passing them.

Parameters:

  • kwargs: The metrics to save

Enum: NoCommitAction

Description: How to notify the user when there are changes that are not committed and a new run is started not in DEBUG mode

  • NOP: No action
  • WARN: Show a warning
  • RAISE: Raise an exception
    NOP = NOP
    WARN = WARN
    RAISE = RAISE