Quickstart

A project only requires a yaml file called .fig.yml or similar (see documentation), however, it is also suggested to create a directory called config to contain any config yaml files that should automatically be registered when the project is loaded. Usually, when loading a project, that requires running some python files, relative path to the top level source file to run should be specified in the project info file (.fig.yml) under the key src. Below is an example of a simple omni-fig project with all the suggested bells and whistles:

project/ - project root name (can be anything)
│
├── config/ - any yaml config files that should automatically be registered
│   ├── dir1/
│   │   ├── myconfig2.yaml
│   │   └── ... - any configs will be registered with the relative path as prefix ("dir1/")
│   │
│   ├── myconfig1.yaml
│   └── debug.yaml - config to be automatically used in debug mode
│
├── src/ - any python source files
│   ├── __init__.py - python file to be called to load project as a package into omnifig.projects
│   ├── script1.py - any additional source files
│   └── ...
│
├── .fig.yml - project info file
└── ...

For the example above, .fig.yml should contain something like:

name: myproject
package: src

To specify that src/ contains the code neccessary load the project.

Inside the python package src/ you can register any Components, Modifiers, Scripts, or configs needed for the project. For example, src/__init__.py might look like:

import omnifig as fig

@fig.Component('myconverter') # registers a new component (any class or function to be specified in the config)
class Converter:
    def __init__(self, A): # when creating a component, the input is the config object at the corresponding node
        self.rates = A.pull('rates', {})

    def to_usd(self, value, currency):
        if currency in self.rates:
            return value / self.rates[currency]
        return value

@fig.AutoModifier('sketchy') # registers a new automodifier (used to dynamically modify components)
class Sketchy:
    def __init__(self, A):
        super().__init__(A) # AutoModifiers become subclasses of the Component they modify

        self.fudge_the_numbers = A.pull('fudge_the_numbers', True)

    def to_usd(self, value, currency):
        value = super().to_usd(value, currency)
        if self.fudge_the_numbers:
            return value * 0.9
        return value


@fig.Script('myscript', description='Does something awesome') # registers a new script called "myscript"
def run_train_model(A): # config object containing all necessary config info
    print('Running myscript!')

    arg1 = A.pull('arg1') # gets the value corresponding to "arg1" in the config

    # pull the value corresponding to the key "arg2" starting from the node at "some.deep"
    # defaults to "[default value]" if that fails
    arg2 = A.pull('some.deep.arg2', '[default value]')

    # set (and get) arg2 to "myvalue", unless it already exists
    # also this will automatically create the node "other_branch" if it doesn't already exist
    arg3 = A.push('other_branch.arg3', 'myvalue', overwrite=False)

    # when a node (eg. "converter") contains the key "_type" (and optionally "_mod") it is treated as a component
    A.push('converter._type', 'myconverter', overwrite=False)

    # values can be lists/dicts (even nested)

    budget, unit = A.pull('mymoney', [1000000000, 'Zimbabwe-dollars'])

    converter = A.pull('converter', None) # when pulling components, they objects are automatically created

    if converter is not None:
        budget = converter.to_usd(budget, unit)
    else:
        raise Exception('No converter to confirm budget')

    # ... maybe do something interesting with all that money

    msg = "I'm {}a millionaire".format('' if budget > 1e6 else 'not ')
    print(msg)

    return msg # anything this script should return

Any function or class that should be specified in the config should be registered as a Component. When “pulling” a component (a config node that contains the “_type” key), the config system will automatically get the corresponding class/function and run it (returning the created instance/output). You can also define and register Modifiers to dynamically specify modifications that you want to make to the components in the config (using the “_mod” key in the same node as “_type”).

It is highly recommended that you create a profile info yaml file and set the environment variable FIG_PROFILE to the full path to that profile info file. For example, the profile might contain:

name: mycomputer

projects:
    myproject: /path/to/myproject # path to the "myproject" directory mentioned above

As you create new projects, you can add those to the profile info file so they can loaded from anywhere. By default, only the project in the current working direcory is loaded (and any “related” projects thereof), however that can also be changed in the profile info file (see the documentation).

With this setup, you should be able to run all of the below (from the terminal inside myproject/):

# execute myscript without any config files or arguments
fig myscript

# execute myscript in debug mode ("-d") and with config file "dir1/myconfig2"
fig -d myscript dir1/myconfig2

# execute myscript with "myconfig1" as config updated by command line argument
fig myscript myconfig1 --arg1 cmdline

# execute myscript with merged config file and command line arguments
python script1.py myconfig1 dir1/myconfig2 --some.deep.arg2 10.2

# execute myscript in debug mode with merged config and command line argument
python script1.py -d myconfig1 dir1/myconfig2 --converter._mod.sketchy 1 --arg1

It might be worth taking a look at the resulting config object looks like for each of these commands (and depending on what information is saved in the corresponding config files in myproject/config/. Note that you can use -d to switch to debug mode (see documentation for more info).

You might also load and run scripts in this project from a jupyter notebook (or a python console) using:

import omnifig as fig

fig.initialize('myproject') # load profile and project

A = fig.get_config('dir1/myconfig2', 'config1') # positional arguments can be names of registered config files
out1 = fig.run('myscript', A)

B = fig.get_config('config1', arg1=[1,2,3]) # keyword arguments are much like command line arguments
out2 = fig.run('myscript', B, debug=True) # meta arguments (such as "debug") can be set using keyword args in run()

C = fig.get_config(arg1='something', arg2='another thing')
C.update(B)
C.push('arg1', 'something else') # the config object can be modified with push()/update()
out3 = fig.run('myscript', C)

# quick_run effectively combines get_config and
out4 = fig.quick_run('myscript', 'config1', use_gpu=True)

While this example should give you a basic idea for what a project might look like, this only touches on the basics of what you can do with omni-fig. I strongly recommend you check out the documentation. for more information, additionally there are some examples of real projects that use omni-fig such as omnilearn and No-Nonsense-News .

Top-level interface

By default, these functions provide all the basic and most important features of this package. However, they usually delegate all the real heavy-lifting to the “current” project or profile, so for more fine-grain control over exactly how your code gets organized and additional customization, take a look at the full documentation (particularly the sections about profiles and projects).

get_current_project()

Get the current project, assuming a profile is loaded, otherwise returns None

get_project(ident=None)

Checks the profile to return (and possibly load) a project given the name or path ident

entry(script_name=None)

Recommended entry point when running a script from the terminal. This is also the entry point for the fig command.

This collects the command line arguments in sys.argv and overrides the given script with script_name if it is provided

Parameters

script_name – script to be run (may be set with arguments) (overrides other arguments if provided)

Returns

None

main(*argv, script_name=None)

Runs the desired script using the provided argv which are treated as command line arguments

Before running the script, this function initializes omni-fig using initialize(), and then cleans up after running using cleanup().

Parameters

• argv – raw arguments as if passed in through the terminal

• script_name – name of registered script to be run (may be set with arguments) (overrides other arguments if provided)

Returns

output of script that is run

run(script_name, config, **meta)

Runs the specified script registered with script_name using the current project.

Parameters

• script_name – must be registered in the current project or defaults to the profile

• config – config object passed to the script

• meta – any meta rules that modify the way the script is run

Returns

output of the script, raises MissingScriptError if the script is not found

quick_run(script_name, *parents, **args)

Convenience function to run a simple script without a given config object, instead the config is entirely created using the provided parents and args.

Parameters

• script_name – name of registered script that is to be run

• parents – any names of registered configs to load

• args – any additional arguments to be provided manually

Returns

script output

initialize(*projects, **overrides)

Initializes omni-fig by running the “princeps” file (if one exists), loading the profile, and any active projects. Additionally loads the project in the current working directory (by default).

Generally, this function should be run before running any scripts, as it should register all necessary scripts, components, and configs when loading a project. It is automatically called when running the main() function (ie. running through the terminal). However, when starting scripts from other environments (such as in a jupyter notebook), this should be called manually after importing omnifig.

Parameters

• projects – additional projects that should be initialized

• overrides – settings to be checked before defaulting to os.environ or global settings

Returns

None

cleanup(**overrides)

Cleans up the projects and profile, which by default just updates the project/profile info yaml file if new information was added to the project/profile.

Generally, this should be run after running any desired scripts.

Parameters

overrides – settings to check before defaulting to global settings or os.environ

Returns

None

get_config(*contents, **parameters)

Process the provided info using the current project into a config object. :param contents: usually a list of parent configs to be merged :param parameters: any manual parameters to include in the config object :return: config object

create_component(config)

Create a component using the current project :param config: Must contain a “_type” parameter with the name of a registered component :return: the created component

quick_create(_type, *parents, **parameters)

Creates a component without an explicit config object. Effectively combines get_config() and create_component() :param _type: :param parents: :param parameters: :return:

register_script(name, fn, description=None, use_config=False)

Manually register a new script to the current project

register_component(name, fn, description=None)

Manually register a new component to the current project

register_modifier(name, fn, description=None, expects_config=False)

Manually register a new modifier to the current project

register_config(name, path)

Manually register a new config file to the current project

register_config_dir(path, recursive=True, prefix=None, joiner='/')

Manually register a new config directory to the current project

has_script(name)

find_script(name)

view_scripts()

has_component(name)

find_component(name)

view_components()

has_modifier(name)

find_modifier(name)

view_modifiers()

has_config(name)

find_config(name)

view_configs()