Configuration

en_core.configuration

This module implements functionality to manage configurations in applications and services. Most applications and services will use a hierarchical structure for configuration to allow configuration values to be overwritten by configuration overrides. The Config class allows applications and services to represent that hierarchy.

Imagine a service that will run in the cloud as a serverless application in AWS Lambda. We normally implement this application as a Flask application, so we can test it locally, and then deploy it to Lambda using Zappa.

The application has multiple environments and allows for configuration overrides. The idea is to have a default configuration with the most appropriate values. This default configuration can be overwritten by a separate configuration file. The override will reside locally in the development environment, and in AWS S3 when running in cloud environments. Finally, we will like to support environment variables that can override some of the configuration values.

The application can expose a config.py module that provides the configuration. The following is an example of such config.py:

# In config.py
import os

import en_core.configuration as c
import en_core.configuration.local as cl
import en_core.configuration.s3 as cs3

# Default configuration definition
DEFAULT_CONFIGURATION = {
    'max_threads': 10,
    'dbconnect': 'file:///data.sqlite
}

_config = None

def get_config():
    global _config
    if not _config:
        # Environment variables are case insensitive in Windows
        # but not Linux
        #
        env_config = c.CaseInsensitiveMapping(os.environ)   
        environment = os.environ.get('ENVIRONMENT')
        if environment.lower() == 'local':
            override_file = os.environ.get('CONFIG_OVERRIDE')
            source = cl.LocalJsonFile(override_file)
        else:
            bucket = os.environ.get('CONFIG_BUCKET')
            key = os.environ.get('CONFIG_KEY')
            source = cs3.S3JsonFile(bucket, key)
        override_config = c.CaseInsensitiveFileMapping(source)
        default_config = c.CaseInsensitiveMapping(DEFAULT_CONFIGURATION)
        _config = c.Config(env_config, override_config, default_config)
    return _config

# In application code.
import config
configuration = config.get_config()
connect_str = configuration.get('DBCONNECT')

In the example above, the environment is specified first; therefore, it will override other configuration values with the same keys. If the key is not found in the environment, it will look in the override file. Finally, it will look in the default configuration if the key is not found.

class CaseInsensitiveFileMapping(source)

This class extends CaseInsensitiveMapping to initialize the values using a source. The source object must implement a .read() method that returns a mapping.

Parameters: source (Source) – Object that provides the initial mapping returned from the .read() method.

import en_core.configuration as config
import en_core.configuration.s3 as s3c

source = s3c.S3JsonFile('my-bucket', 'path/to/config.json')
mapping = config.CaseInsensitiveFileMapping(source)
mapping.get('existing')         # Returns value from the file.
mapping.get('ANOTHER_EXISTING') # Returns value from the file.
mapping.get('INEXISTENT')       # Returns None.

class CaseInsensitiveMapping(mapping)

This class implements the collections.abc.Mapping interface treating keys as case insensitive. The class is initialized with other mapping object used as a source for the values.

Parameters: mapping (Mapping) – Mapping object containing the initial values of the map.

import en_core.configuration as config

original_mapping = {
    'value1: 1,
    'value2: 'second value',
    'value_3': 3
}
mapping = config.CaseInsensitiveMapping(original_mapping)
mapping.get('VALUE1')       # -> 1
mapping.get('Value2')       # -> 'second value'
mapping.get('vAlUe_3)       # -> 3
mapping.get('INEXISTENT)    # -> None

The mapping returns all the keys in upper case, so it can be used for configuration because some applications require configuration keys to be all upper case.

# Continue example
mapping.keys()  # -> ['VALUE1', 'VALUE2', 'VALUE_3']

class Config(*mappings)

This class represents the configuration of an application and service. The class is initialized with a list of mappings (dictionary like objects) where the configuration parameters are stored. The look up priority is done in the order in which the mappings are added; therefore, mappings added in-front will override values of subsequent mappings.

The class supports the full collections.abc.Mapping interface, which makes it suitable as a configuration object from most application frameworks.

JSON Sources

en_core.configuration.json

Base classes for JSON based sources.

class JsonFile

Base class for JSON files used as sources of configurations. Derived classes must implement _open() that returns an input stream with the file contents.

read(): Reads the file and returns a dictionary representation of its contents.

Local Sources

en_core.configuration.local

This module implements functionality that allow using local files as sources for configuration.

class LocalJsonFile(file_name, _open_func=None)

Implements the configuration Source interface to read the configuration from a JSON file in the local file system. The class is initialized with the path pointing to the file containing the configuration.

Parameters: file_name (str) – Name of the bucket where the configuration is stored.

AWS S3 Sources

en_core.configuration.s3

This module provides functionality to read configurations from AWS S3. Applications and services using this module are required to add ptaws as a dependency since this library doesn’t include it.

class S3JsonFile(bucket, file_obj, _client=None)

Implements the configuration Source interface to read the configuration from a JSON file in S3. The class is initialized with the S3 bucket and file key pointing to the file containing the configuration.

Parameters

bucket (str) – Name of the bucket where the configuration is stored.
file_obj (str) – Key to the JSON configuration file