Config System

The original motivation for this package was to design a system that would read arguments from files, the terminal, or directly as python objects (eg. in a jupyter notebook) and would structure so that the arguments are always used in the code where they need to be. This is accomplished chiefly by a hierarchical structure in the config much like a tree where each branch corresponds to the a dict or list of arguments (or other branches).

When reading (aka pulling) arguments from the config, if an argument is not found in the current branch, it will automatically defer to the higher branches (aka parent branch) as well, which allows users to define more or less “global” arguments depending on which node actually contains the argument.

It should be noted that despite the hierarchical structure of the config object, it can ultimately always be exported into a simple yaml file - so it should not contain any values that are primitives (str, bool, int, float, None) aside from the branches that behave either as a dict or list. Using the registries, the config can implicitly contain specifications for arbitrarily complex components such as objects and functions (see Registry for more details and an example).

Probably, the next most important feature of the config object is that a config object can be updated with other configs, and thereby “inherit” arguments from other config files. This inheritance behaves analogously to python’s class inheritance in that each config can have arbitrarily many parents and the full inheritance tree is linearized using the “C3” linearization algorithm (so no cycles are permitted). The configs are updated in reverse order of precedence (so that the higher precedence config file can override arguments in the lower precedence files).

Generally, arguments are read from the config object using pull() and individually updated or set using push().Both pull() and push() supports deep gets and sets, which means you can get and set arguments arbitrarily deep in the config hierarchy using “.” to “dereference” (aka “go into”) a branch. When pulling, additional default values can be provided to process if the key is not found. This is especially useful in conjunction with another feature called aliasing, where arguments can reference each other (see the example below).

For example, given the config object that is loaded from the this yaml file (registered as myconfig):

favorites:
    games: [Innovation, Triumph and Tragedy, Inis, Nations]
    language: Python
    activity: <>games.0

wallpaper:
    color: red

jacket:
    size: 30

nights: 2
trip:
    - location: London
      nights: 3
    - location: Berlin
    - location: Moscow
      nights: 4

app:
    price: 1.99
    color: <>wallpaper.color

When this yaml file is loaded (eg. config = omnifig.get_config('myconfig')), we could use it like so:

assert config.pull('favorites.language') == 'Python'
assert config.pull('favorites.0') == 'Innvoation'
assert config.pull('app.color') == 'red'
assert config.pull('app.publisher', 'unknown') == 'unknown'
assert config.pull('jacket.color', '<>wallpaper.color') == 'red'
assert config.pull('favorites.activity') == 'Innovation'
assert config.pull('jacket.price', '<>price', '<>total_cost', 'too much') == 'too much'
assert config.pull('trip.0.location') == 'London'
assert config.pull('trip.1.nights', 4) == 2

While this example should give you a sense for what kind of features the config system offers, a comprehensive list of features follows.

Keys

In addition to the behavior described above, the keys (or indices) in a config branch have the following features (where {} refers to any value):

• push()/pull() '_{}' - protected argument → not visible to child branches when they defer to parents

• push()/pull() '__{}' - volatile argument → is not exported when saving config to yaml (can be used for non-yamlifiable data)

• push()/pull() ({1},{2}, ...) - deep key → [{1}][{2}]

• push()/pull() '{1}.{2}' - deep key as str → ['{1}']['{2}']

• push()/pull() '{1}.{2}' - deep key through list → ['{1}'][{2}] (where {2} is an int and self['{1}'] is a list)

• push() '{1}.{2}' where '{1}' is missing - deep push → automatically creates a new branch '{1}' in config and then pushes '{2}' to that new branch

Values

The values of arguments also have a few special features worth noting:

• '<>{}' - local alias → use value of key {} starting search for the key here

• '<o>{}' - origin alias → use value of key {} starting search for the key at origin (this only makes a difference when chaining aliases, origin refers to the branch where pull() was called)

• _x_ - remove key if encountered (during update) → remove corresponding key it it appears in the config being updated

• __x__ - cut deferring chain of key → behave as though this key didn’t exist (and don’t defer to parent)

Code

class ConfigType(parent=None, printer=None, prefix=None, safe_mode=False, project=None, data=None)

Bases: omnibelt.transactions.Transactionable

The abstract super class of config objects.

The most important methods:

• push() - set a parameter in the config

• pull() - get a parameter in the config

• sub() - get a sub branch of this config

• seq() - iterate through all contents

• update() - update a config with a different config

• export() - save the config object as a yaml file

Another important property of the config object is that it acts as a tree where each node can hold parameters. If a parameter cannot be found at one node, it will search up the tree for the parameter.

Config objects also allow for “deep” gets/sets, which means you can get and set parameters not just in the current node, but any number of nodes deeper in the tree by passing a list/tuple of keys or keys separated by “.” (hereafter called the “address” of the parameter).

Note: that all parameters must not contain “.” and should generally be valid python identifiers (strings with no white space that don’t start with a number).

__deepcopy__(memodict={})

__copy__()

copy()

shallow copy of the config object

pythonize()

classmethod convert(data, recurse)

used by configurize to turn a nested python object into a config object

sub(item)

Used to get a subbranch of the overall config :param item: address of the branch to return :return: config object at the address

update(other)

Used to merge two config nodes (and their children) together

This method must be implemented by child classes depending on how the contents of the node is stored

Parameters

other – config node to overwrite self with

Returns

None

pull(item, *defaults, silent=False, ref=False, no_parent=False, as_iter=False, raw=False)

Top-level function to get parameters from the config object (including automatically creating components)

Parameters

• item – address of the parameter to get

• defaults – default values to use if item is not found

• silent – suppress printing message that this parameter was pulled

• ref – if the parameter is a component that has already been created, get a reference to the created component instead of creating a new instance

• no_parent – don’t default to a parent node if the item is not found here

• as_iter – return an iterator over the selected value (only works if the value is a dict/list)

Returns

processed value of the parameter (or default if item is not found, or raises a MissingConfigError if not found)

pull_self(name=None, silent=False, as_iter=False, raw=False, ref=False)

Process self as a value being pulled.

Parameters

• name – Name given to self for printed message

• silent – suppress printing message

• as_iter – Return self as an iterator (has same effect as calling seq())

Returns

the processed value of self

push(key, val, *_skip, silent=False, overwrite=True, no_parent=True, force_root=False, process=True)

Set key with val in the config object, but pulls key first so that val is only set if it is not found or overwrite is set to True. It will return the current value of key after possibly setting with val.

Parameters

• key – key to check/set (can be list or ‘.’ separated string)

• val – data to possibly write into the config object

• _skip – soak up all other positional arguments to make sure the remaining are keyword only

• silent – Do not print messages

• overwrite – If key is already set, overwrite with (configurized) ‘val’

• no_parent – Do not check parent object if not found in self

• force_root – Push key to the root config object

Returns

current val of key (updated if written)

export(path=None)

Convert all data to raw data (using dict/list) and save as yaml file to path if provided. Also returns yamlified data.

Parameters

path – path to save data in this config (data is not saved to disk if not provided)

Returns

raw “yamlified” data

seq()

Returns an iterator over the contents of this config object where elements are lazily processed during iteration (see ConfigIter for details).

Returns

iterator over all arguments in self

replace_vals(replacements)

set_silent(silent=True)

Sets whether pushes and pulls on this config object should be printed out to stdout

silence(silent=True)

silenced(setting=True)

Returns a context manager to silence this config object

is_root()

Check if this config object has a parent for defaults

set_parent(parent)

Sets the parent config object to be checked when a parameter is not found in self

get_parent()

Get parent (returns None if this is the root)

set_process_id(name=None)

Set the unique ID to include when printing out pulls from this object

get_root()

Gets the root config object (returns self if self is the root)

_set_prefix(prefix)

_reset_prefix()

_get_hidden_prefix()

set_project(project)

get_project()

set_safe_mode(safe_mode)

in_safe_mode()

purge_volatile()

Recursively remove any items where the key starts with “__”

Must be implemented by the child class

Returns

None

_single_set(key, val)

_missing_key(key, *context)

class ConfigDict(parent=None, printer=None, prefix=None, safe_mode=False, project=None, data=None)

Bases: omnifig.config.ConfigType, omnibelt.basic_containers.tdict

Dict like node in the config.

Keys should all be valid python attributes (strings with no whitespace, and not starting with a number).

NOTE: avoid setting keys that start with more than one underscore (especially ‘__obj’) (unless you really know what you are doing)

__deepcopy__(memodict={})

__copy__()

copy()

replace_vals(replacements)

classmethod convert(data, recurse)

update(other)

Merge self with another dict-like object

Parameters

other – must be dict like

Returns

None

purge_volatile()

Recursively remove any items where the key starts with “__”

Returns

None

class EmptyElement

Bases: object

class ConfigList(*args, empty_fill_value=<class 'omnifig.config.EmptyElement'>, **kwargs)

Bases: omnifig.config.ConfigType, omnibelt.basic_containers.tlist

List like node in the config.

__deepcopy__(memodict={})

__copy__()

replace_vals(replacements)

purge_volatile()

Recursively remove any items where the key starts with “__”

Returns

None

_str_to_int(item)

Convert the input items to indices of the list

update(other)

Overwrite self with the provided list other

_single_set(key, val)

push(first, *rest, overwrite=True, **kwargs)

When pushing to a list, if you don’t provide an index, the value is automatically pushed to the end of the list

Parameters

• first – if no additional args are provided in rest, then this is used as the value and the key is the end of the list, otherwise this is used as key and the first element in rest is the value

• rest – optional second argument to specify the key, rather than defaulting to the end of the list

• kwargs – same keyword args as for the ConfigDict

Returns

same as for ConfigDict

append(item)

extend(item)

class ConfigIter(origin, elements=None, auto_pull=True, include_key=None, reversed=False)

Bases: object

Iterate through a list of parameters, processing each item lazily, ie. only when it is iterated over (with next())

view()

Returns the next object without processing the item, may throw a StopIteration exception

step()

set_auto_pull(auto=True)

set_reversed(reversed=True)

has_next()

__iter__()

configurize_nones(s, recurse)

Turns strings into None, if they match the expected patterns