Initialization

It is strongly encouraged to initialize omni-fig before running any scripts. When calling a script from the terminal using the fig command, omni-fig will initialize automatically, but when running in a separate environment (such as in a jupyter notebook), it is suggested to call omnifig.initialize() after importing the package. More info about initialize() can be found in Running Scripts.

The initialization process:

1. runs the initial “princeps” file (if one is specified)

2. loads the profile (if one exists)

3. loads any active projects listed in the profile

4. recursively loads all related projects

5. loads the profile in the current working directory (if there is one and it hasn’t been loaded yet)

Princeps File

The princeps file is an optional startup file that can be run to change global settings in omni-fig, to change the profile type, to register new project types, or to make other advanced customizations in how omni-fig runs.

The princeps file is specified by defining an environment variable FIG_PRINCEPS_PATH which contains the absolute path to a python file.

For most use cases it should not be necessary to specify a princeps file, and running one can also be disabled all together - so it is generally discouraged to use a princeps file unless absolutely necessary.

Profiles

Generally, every OS image or file system should use it’s own profile. The profile is specified by defining an environment variable FIG_PROFILE which contains the absolute path to a yaml file.

While using a profile is completely optional, it is highly recommended as the profile is the primary way to specify the location of all of your projects that you may want to load. Additionally, the profile can be used to change the global settings of omni-fig to tailor the behavior to the specific machine/OS. Since the profile is meant to act globally for the whole file system, all paths should be absolute.

The most important contents of the profile file (all of which are optional):

• projects - dictionary from project name to absolute path to the project directory

• active_project - list of names of projects that should automatically be loaded (the paths to these projects must be in the projects table

• config_paths - list of absolute paths to config (yaml) files or directories that should be registered when loading

• src_paths - list of absolute paths to python files that should be run when loading

• default_ptype - name of the registered project type that should be used to load projects (default: default)

• global_settings - dictionary of setting names and values to be set during loading

• autoload_local - bool to specify whether the project in the current working directory should be loaded (default: True)

class Profile(**kwargs)

Bases: omnifig.organization.Workspace

Generally all paths that the Profile deals with should be absolute paths as the profile operates system wide

__init__(**kwargs)

_process(raw)

Processes the data from a yaml file and saves it to self :param raw: data from a yaml file :return: None

static is_valid_project_path(path)

Check if a path points to a project directory

initialize(loc=None)

Steps to execute during initialization including: updating global settings, loading configs, running any specified source files, loading active projects, and possibly the local project.

Parameters

loc – absolute path to current working directory (default comes from os.getcwd())

Returns

None

update_global_settings()

Updates global settings with items in self.global_settings

load_active_projects(load_related=True)

Load active projects in self.active_projects in order, and potentially all related projects as well

Parameters

• load_related – load directly related projects

• all_related – recursively load all related projects

Returns

None

get_project(ident=None, load_related=True, all_related=False)

Gets project if already loaded, otherwise tries to find the project path, and then loads the missing project

Parameters

• ident – project name or path

• load_related – also load related projects (before this one) (dont recursively load related)

• all_related – recursively load all related projects (trumps load_related)

Returns

project object

get_project_type(name)

Gets the project type registered with the given name

load_project(ident, load_related=True, all_related=False)

Parameters

• ident – path or name of project (if name, then it must be profile.projects)

• load_related – also load related projects (before this one) (dont recursively load related)

• all_related – recursively load all related projects (trumps load_related)

Returns

loaded project (or raises NoValidProjectError if none is found)

resolve_project_path(ident)

Map/complete/fix given identifier to the project path

Parameters

ident – name or path to project

Returns

correct project path or None

clear_loaded_projects()

Clear all loaded projects from memory (this does not affect registered components or configs)

contains_project(ident)

track_project_info(name, path)

Add project info to projects table self.projects

track_project(project)

Add project to projects table self.projects

is_tracked(project)

Check if a project is contained in projects table self.projects

include_project(project, track=True)

Include a project instance in loaded projects table managed by this project and optionally track a project persistently.

add_active_project(project)

Add a project to the list of active projects

get_active_projects()

Get a list of all projects specified as “active”

set_current_project(project=None)

Set the current project

get_current_project()

Get the current project (usually loaded last and local),

find_artifact(atype, name)

Search for a registered artifact from the name either in a loaded project or in the profile (global) registries

Parameters

• atype – component, modifier, script, or config

• name – registered artifact name, can use prefix to specify project (separated with “:”)

Returns

artifact entry (namedtuple)

has_artifact(atype, name)

Check if a registered artifact of type atype with name name exists

Parameters

• atype – component, modifier, script, or config

• name – registered artifact name, can use prefix to specify project (separated with “:”)

Returns

True iff an entry for name exists

cleanup()

Saves project data if some has changed, and possibly also updates the project files of all loaded projects if they have changed.

Projects

For omni-fig to recognize a directory as a project, it must contain a yaml file named .fig.yml (or similarly, see code for all options). This yaml file should contain all project specific information (similar to profiles) that may be useful for running code or interacting with other projects. The most important information contained in the project yaml file:

• related - a list of project names that should be loaded before loading this one (all the paths to all related projects must be found in the profile’s projects table)

• src - the relative path to the python file that should be run in order to fully load all components of this project

• name - while not enforced, it is strongly encouraged that project info files contain their own name (ideally the same as is used in the profile’s projects table)

• project_type - the registered name of the project type to use when instantiating this project

• ptype_src_file - a python file to run before trying to load this project (for example, as it might define and register the desired project type)

• py_info - relative path to a python file that defines project meta data (this is particularly useful for projects that are packages, as the meta data is potentially already specified in various places, like requirements.txt) (omni-fig itself is a good example, see this project’s .fig.yml file and the associated omnifig/_info.py file)

The directory that contains the project info file (.fig.yml) is defined as the “project directory”. All paths in the info file should be relative to the project directory. By default, if there is a directory called config or configs in the project directory, then that directory will automatically be registered as a config directory (ie. all yaml files inside will be registered while preserving the folder hierarchy) - see the unit tests (in test/ for an example).

When a project is loaded, first the desired type is identified. As a result, you can subclass the Project class and override the behavior of project objects. Note that this is a fairly advanced featured and should be used only when absolutely necessary (atm I’m not sure why I added this feature in the first place).

class Cerifiable

Bases: object

__certify__(A, **kwargs)

class Workspace(silent=False, **kwargs)

Bases: omnifig.containers.Container

__init__(silent=False, **kwargs)

_process(raw)

initialize()

This loads the project, primarily by registering any specified config files, importing specified packages, and finally running any provided source files

Returns

None

load_configs(paths=[])

Registers all specified config files and directories

load_src(srcs=[], packages=[])

Imports all specified packages and runs the specified python files

meta_rules()

meta_rules_fns()

reset_registries()

Clears all registries

register_artifact(atype, name, info)

General function to register an artifact of type atype with name name

register_script(name, fn, description=None, use_config=False)

Function to register a script

Parameters

• name – name of script

• fn – script function (usually a callable that expects the config object)

• use_config – True if the config should be passed as only arg when calling the script function, otherise it will automatically pull all arguments in the script function signature

• description – a short description of what the script does

Returns

register_component(name, fn, description=None)

fn takes a single input - a Config object The config object is guaranteed to have at least one entry with key “_type” and the value is the same as the registered name of the component.

Parameters

• name – str (should be unique)

• fn – callable accepting one arg (a Config object) (these should usually be classes)

• description – description of what this component is about

register_modifier(name, fn, description=None, expects_config=False)

fn takes as input a component and a Config object.

Parameters

• name – str (should be unique)

• fn – callable accepting one arg (the “create_fn” of a registered component) (these should usually be classes)

• description – description of what this modifier is about

register_config(name, path)

Register a file as a named config

Parameters

• name – str (should be unique)

• path – full path to the config

register_config_dir(path, recursive=False, prefix=None, joiner='/')

Registers all yaml files found in the given directory (possibly recursively)

When recusively checking all directories inside, the internal folder hierarchy is preserved in the name of the config registered, so for example if the given path points to a directory that contains a directory a and two files f1.yaml and f2.yaml:

Contents of path and corresponding registered names:

• f1.yaml => f1

• f2.yaml => f2

• a/f3.yaml => a/f3

• a/b/f4.yaml => a/b/f3

If a prefix is provided, it is appended to the beginning of the registered names

Parameters

• path – path to root directory to search through

• recursive – search recursively through sub-directories for more config yaml files

• prefix – prefix for names of configs found herein

• joiner – string to merge directories when recursively searching (default /)

Returns

None

has_artifact(atype, name)

Check if this workspace has an artifact of type atype registered with name name

find_artifact(atype, name)

Find a registered artifact from the type and name

Parameters

• atype – component, modifier, script, or config

• name – registered name

Returns

artifact entry, throws UnknownArtifactError if atype is not recognized,

and the corresponding artifact missing error if the name cannot be found

view_artifacts(atype)

Return a shallow copy of the full registry of type atype

has_script(name)

find_script(name)

view_scripts()

has_component(name)

find_component(name)

view_components()

has_modifier(name)

find_modifier(name)

view_modifiers()

has_config(name)

find_config(name)

view_configs()

run(script_name=None, config=None, **meta_args)

This actually runs the script given the config object.

Before starting the script, all meta rules are executed in order of priority (low to high) as they may change the config or script behavior, then the run mode is created, which is then called to execute the script specified in the config object (or manually overridden using script_name)

Parameters

• script_name – registered script name to run (overrides what is specified in config)

• config – config object (usually created with get_config() (see Config System)

• meta_args – Any additional meta arguments to include before running

Returns

script output

create_component(info)

Creates the component specified in info (checks component registry using info.pull(‘_type’), and modifier registry for info.pull(‘_mod’))

_mod can be a list, in which case they will be applied in the given order, eg:

let mods = [A, B, C]

component <- C(B(A(component)))

_mod can also be a dict, in which case the keys should be the mod names and the values the order (low to high). So for the same behavior as above, a _mod could also be {A:0, B:1, C:2}

NOTE: generally, modifiers should be ordered from more specific to more general

Parameters

info – should be a config object with attribute “_type” (and optionally “_mod”)

Returns

component created using the provided config (info)

process_argv(argv=(), script_name=None)

Parses the command line arguments to identify the meta arguments, script name (optionally overridden using script_name), and config args

From that, this builds the config and meta config object.

Parameters

• argv – list of all command line arguments to parse in order

• script_name – optional script name to override any script specified in argv

Returns

config object (containing meta config under _meta)

_load_config_from_path(path, process=True)

Load the yaml file and transform data to a config object

Generally, get_config should be used instead of this method

Parameters

• path – must be the full path to a yaml file

• process – if False, the loaded yaml data is passed without converting to a config object

Returns

loaded data from path (usually as a config object)

_merge_configs(configs, parent_defaults=True)

configs should be ordered from oldest to newest (ie. parents first, children last)

This is an internal method used by get_config() and should generally not be called manually.

_process_single_config(data, process=True, parents=None, tree=None, me='')

This loads the data (if a path or name is provided) and then checks for parents and loads those as well

Generally, get_config should be used instead of this method

Parameters

• data – config name or path or raw data (dict/list) or config object

• process – configurize loaded data

• parents – if None, no parents are loaded, otherwise it must be a dict where the keys are the absolute paths to the config (yaml) file and values are the loaded data

Returns

loaded data (as a config object or raw)

create_config(*contents, **parameters)

Top level function for users. This is the best way to load/create a config object.

All parent config (registered names or paths) that should be loaded must precede any manual entries, and will be loaded in reverse order (like python class inheritance).

If the key _history_key is specified and not None, a flattened list of all parents of this config is pushed to the given key.

Parameters

• contents – registered configs or paths or manual entries (like in terminal)

• parameters – specify parameters manually as key value pairs

Returns

config object

class Project(profile=None, **kwargs)

Bases: omnifig.organization.Workspace

Projects are used to group code into packets with specific config files that should be loaded all together. A project must contain a yaml file named .fig.yml in the root directory of the project (aka the “project directory”), and all paths in that yaml file should be relative to the project directory.

Generally there are two kinds of projects: “packages” and “loose” projects. Package projects are meant to be installed and used as a library, while loose projects may just be a series of python files.

This class may also be subclassed to change the behavior of projects (such as changing the loading), in fact, any subclasses of this class can automatically be registered when providing a name for the project type in the class definition.

classmethod __init_subclass__(name=None)

Subclasses can automatically be registered if a ptype for the registry is provided

__init__(profile=None, **kwargs)

get_profile()

get_path()

Gets the path to the project directory

get_related()

Returns a list of project names of all projects that should be loaded prior to this one

static check_project_type(raw)

Based on raw project info, check if this project expects a certain project type, if so optionally provide a path to the source file of the project type.

Parameters

raw – raw project info

Returns

None or tuple with project type identifier and path to source file (or None)

_process(raw)

Given the raw info loaded from a yaml file, this function checks integrates the information into the project object (self).

Parameters

raw – dictionary of info (usually loaded from a yaml)

initialize()

register_artifact(atype, name, info, include_global=True)

has_artifact(atype, name, check_global=True)

find_artifact(atype, name, check_global=True)

has_config(name)

find_config(name)

view_artifacts(atype)

Meta Rules

Meta rules allows changing any script’s behavior before it is run, primarily by making changes in the loaded config or changing the run mode that will be used for execution. Meta rules can be activated from the command line using the registered code (usually a single letter) preceded by a single “-”. A meta rule can also be provided with additional required arguments (ie. strings), but the number of arguments must be specified when registering. If a meta rule is activated it should appear in the meta config (under the branch _meta in the config object).

Before a script is executed, all registered meta rules are executed in order of priority (low to high). Since all meta rules are always executed, each rule is expected to check the meta config object whether it has been activated and act accordingly. Note that every meta rule is given the loaded config (and separately the meta config) and returns the config after making whatever changes are desired.

Meta_Rule(name, priority=0, code=None, expects_args=0, description=None)

Decorator used to register meta rules.

Parameters

• name (str) – Name of the meta rule (and key in meta config for any required arguments)

• priority (int) – priority of this meta rule (the higher it is the earlier it is run relative to other meta rules)

• code (Optional[str]) – Command line code (usually a single letter) preceded by “-” to activate this meta rule

• expects_args (int) – number of command line arguments required when activating

• description (Optional[str]) – short description of what this rule does for the help message

Returns

decorator function to register the meta rule

register_meta_rule(name, fn, priority=0, code=None, expects_args=0, description=None)

Meta Rules are expected to be a callable of the form:

config <= fn(meta, config)

where meta is the meta config, and the output should be the processed config

Parameters

• name – name of the new rule

• fn – callable to enforce rule

• priority – priority of this rule (rules are enforced by priority high to low)

• code – (optional) single letter code used to activate rule from the terminal

• expects_args – number of arguments expected by this meta rule through the terminal

• description – one line description of what the rule does

Returns

None

Help Rule

As a particularly useful example of how meta rules can be used, the “help rule” implements the help messages for the fig command, which includes printing out a list of all registered scripts (after the relevant projects are loaded) as well as descriptions (if they are provided).

Note that by subclassing and reregistering this rule, the help message and behavior can easily be augmented.

help_message(meta, config)

When activated, this rule prints out help message for the fig command, which includes a list of all registered scripts, meta rules, and configs that have been loaded.

Parameters

• meta – meta config object

• config – config object

Returns

[system exit, with code 0]

Run Modes

The run mode is a component that is created using the meta config object. Aside from any user defined functionality, the run mode responsible for identifying the script that should be run (by default saved to script_name in the meta config), find the corresponding function (usually using self.get_script_info()), execute it with the provided config, and finally return the output.

class Run_Mode(A)

Bases: object

Run modes are used to actually execute the script specified by the user in the run command.

It is recommended to register all run_modes with a run_mode/ prefix, but not required.

__init__(A)

static get_script_info(script_name)

Given the name of the registered script, this returns the corresponding entry in the registry

run(meta, config)

When called this should actually execute the registered script whose name is specified under meta.script_name.

Parameters

• meta – meta config - contains script_name

• config – config for script

Returns

output of script

Debug Mode

The debug mode serves as a good example for how run modes can be used. During development of new scripts and components it can be invaluable to use a debugger (such as in pycharm) to step through the code and see exactly where bugs might be lurking. Alternatively, when running in the terminal, you can still debug your script using the ipdb.post_mortem() debugger.

The debug mode is activated using a meta rule (-d => debug), which then changes the run mode to the debug run mode (registered under run_mode/debug. Finally, the debugger also automatically updates the config to include a debug config (registered as debug)

debug_rule(meta, config)

When activated, this rule changes the run mode to run_mode/debug and updates the config to include

Parameters

• meta – meta config object

• config – full config object

Returns

config object updated with debug config (if available)

class Debug_Mode(A)

Bases: omnifig.modes.Run_Mode

Behaves just like the default run mode, excapt if the script raises an exception, either the ipdb post_mortem() debugger is activated, or if this is already running in a debugger (checked using sys.gettrace() is not None) then ipdb is not activated.

run(meta, config)

Calls ipdb to activate debugger if an exception is raised when running the script.

Utilities

While this doesn’t include all of the utilities used for organization and managing projects and scripts, this should give you a sense for where some of the functionality from behind the scenes actually originates.

Generally, it should not be necessary for a user to call any of these utilities, but they may be useful to add or change the behavior of omni-fig.

include_files(*paths)

Executes all provided paths to python files that have not already been run.

Parameters

paths – paths to python files to be executed (if not already)

Returns

None

include_package(*packages)

Imports packages based on their names

Parameters

packages – list of package names to be imported

Returns

None

register_project_type(name, cls)

Project types allow users to customize the behavior of project objects

Parameters

• name – identifier of this project type

• cls – project type class

Returns

None

get_project_type(name)

Gets the project type associated with that name, otherwise returns None

view_project_types()

Returns a copy of the full project type registry

set_profile_cls(cls)

Set the class used when loading the profile

get_profile_cls()

Get the class used when loading the profile

set_profile(profile)

Set the loaded profile object

load_profile(**overrides)

Load the profile with the yaml file with the path found with the environment variable FIG_PROFILE

Parameters

overrides – Any additional overrides to use instead of checking the environment variables

Returns

loaded profile object

get_profile(**overrides)

Returns current profile (which gets loaded if there is None)

class set_current_project(project=None)

Bases: object

Context manager to change set the current project in the context

clear_current_project()

Unset the current project (setting it to None)