Migrations¶

Database migration tools are essential for managing incremental database changes.

Edgy, built on top of SQLAlchemy core, includes a powerful internal migration tool.

This tool simplifies the management of models and their corresponding migrations.

Inspired by Flask-Migrations, Edgy's migration tool is framework-agnostic, making it usable anywhere.

Important¶

Before proceeding, familiarize yourself with Edgy's application discovery methods.

The following examples will use the --app and environment variable approach (see Discovery), but auto-discovery (see Auto Discovery) is equally valid.

Project Structure for this Document¶

For clarity, we'll use the following project structure in our examples:

.
├── README.md
├── .gitignore
└── myproject
    ├── __init__.py
    ├── apps
    │   ├── __init__.py
    │   └── accounts
    │       ├── __init__.py
    │       ├── tests.py
    │       └── v1
    │           ├── __init__.py
    │           ├── schemas.py
    │           ├── urls.py
    │           └── views.py
    ├── configs
    │   ├── __init__.py
    │   ├── development
    │   │   ├── __init__.py
    │   │   └── settings.py
    │   ├── settings.py
    │   └── testing
    │       ├── __init__.py
    │       └── settings.py
    ├── main.py
    ├── serve.py
    ├── utils.py
    ├── tests
    │   ├── __init__.py
    │   └── test_app.py
    └── urls.py

Migration Object¶

Edgy requires a Migration object to manage migrations consistently and cleanly, similar to Django migrations.

This Migration class is framework-independent. Edgy ensures it integrates with any desired framework upon creation.

This flexibility makes Edgy uniquely adaptable to frameworks like Esmerald, Starlette, FastAPI, and Sanic.

from edgy import Instance, monkay

monkay.set_instance(Instance(registry=registry, app=None))

Parameters¶

The Instance object accepts the following parameters:

registry: The model registry. It must be an instance of edgy.Registry or an AssertionError is raised.
app: An optional application instance.

Migration Settings¶

Migrations now utilize Edgy settings. Configuration options are located in edgy/conf/global_settings.py.

Key settings include:

multi_schema (bool / regex string / regex pattern): (Default: False). Enables multi-schema migrations. True for all schemas, a regex for specific schemas.
ignore_schema_pattern (None / regex string / regex pattern): (Default: "information_schema"). When using multi-schema migrations, ignore schemas matching this regex pattern.
migrate_databases (tuple): (Default: (None,)). Specifies databases to migrate.
migration_directory (str): (Default: "migrations"). Path to the Alembic migration folder. Overridable per command via -d or --directory parameter.

alembic_ctx_kwargs (dict): Extra arguments for Alembic. Default:

{
    "compare_type": True,
    "render_as_batch": True,
}

Usage¶

Using the Instance class is straightforward. For advanced usage, see the LRU cache technique in Tips and Tricks.

We'll use a utils.py file to store database and registry information.

my_project/utils.py

from functools import lru_cache


@lru_cache()
def get_db_connection():
    from edgy import Registry

    # use echo=True for getting the connection infos printed, extra kwargs are passed to main database
    return Registry("postgresql+asyncpg://user:pass@localhost:5432/my_database", echo=True)

This ensures object creation only once.

Now, use the Migration object in your application.

Using Esmerald¶

my_project/main.py

#!/usr/bin/env python
import os
import sys
from pathlib import Path

from esmerald import Esmerald, Include
from my_project.utils import get_db_connection


def build_path():
    """
    Builds the path of the project and project root.
    """
    Path(__file__).resolve().parent.parent
    SITE_ROOT = os.path.dirname(os.path.realpath(__file__))

    if SITE_ROOT not in sys.path:
        sys.path.append(SITE_ROOT)
        sys.path.append(os.path.join(SITE_ROOT, "apps"))


def get_application():
    """
    Encapsulating in methods can be useful for controlling the import order but is optional.
    """
    # first call build_path
    build_path()
    # because edgy tries to load settings eagerly
    from edgy import monkay, Instance

    registry = get_db_connection()

    # ensure the settings are loaded
    monkay.evaluate_settings(ignore_import_errors=False)

    app = registry.asgi(
        Esmerald(
            routes=[Include(namespace="my_project.urls")],
        )
    )

    monkay.set_instance(Instance(registry=registry, app=app))
    return app


app = get_application()

Using FastAPI¶

Edgy's framework-agnostic nature allows its use in FastAPI applications.

my_project/main.py

#!/usr/bin/env python
import os
import sys
from pathlib import Path

from fastapi import FastAPI
from my_project.utils import get_db_connection


def build_path():
    """
    Builds the path of the project and project root.
    """
    Path(__file__).resolve().parent.parent
    SITE_ROOT = os.path.dirname(os.path.realpath(__file__))

    if SITE_ROOT not in sys.path:
        sys.path.append(SITE_ROOT)
        sys.path.append(os.path.join(SITE_ROOT, "apps"))


def get_application():
    """
    Encapsulating in methods can be useful for controlling the import order but is optional.
    """
    # first call build_path
    build_path()
    # because edgy tries to load settings eagerly
    from edgy import Instance, monkay

    registry = get_db_connection()

    # ensure the settings are loaded
    monkay.evaluate_settings(ignore_import_errors=False)

    app = registry.asgi(FastAPI(__name__))

    monkay.set_instance(Instance(registry=registry, app=app))
    return app


app = get_application()

Using Starlette¶

Similarly, Edgy works with Starlette.

my_project/main.py

#!/usr/bin/env python
import os
import sys
from pathlib import Path

from starlette.applications import Starlette
from my_project.utils import get_db_connection


def build_path():
    """
    Builds the path of the project and project root.
    """
    Path(__file__).resolve().parent.parent
    SITE_ROOT = os.path.dirname(os.path.realpath(__file__))

    if SITE_ROOT not in sys.path:
        sys.path.append(SITE_ROOT)
        sys.path.append(os.path.join(SITE_ROOT, "apps"))


def get_application():
    """
    Encapsulating in methods can be useful for controlling the import order but is optional.
    """
    # first call build_path
    build_path()
    # because edgy tries to load settings eagerly
    from edgy import monkay, Instance

    registry = get_db_connection()
    # ensure the settings are loaded
    monkay.evaluate_settings(ignore_import_errors=False)

    app = registry.asgi(Starlette())

    monkay.set_instance(Instance(registry=registry, app=app))
    return app


app = get_application()

Using Other Frameworks¶

Edgy's design requires no framework-specific parameters, allowing integration with frameworks like Quart, Ella, or Sanic.

Example¶

Consider an application with the following structure:

.
├── README.md
├── .gitignore
└── myproject
    ├── __init__.py
    ├── apps
    │   ├── __init__.py
    │   └── accounts
    │       ├── __init__.py
    │       ├── tests.py
    │       ├── models.py
    │       └── v1
    │           ├── __init__.py
    │           ├── schemas.py
    │           ├── urls.py
    │           └── views.py
    ├── configs
    │   ├── __init__.py
    │   ├── development
    │   │   ├── __init__.py
    │   │   └── settings.py
    │   ├── settings.py
    │   └── testing
    │       ├── __init__.py
    │       └── settings.py
    ├── main.py
    ├── serve.py
    ├── utils.py
    ├── tests
    │   ├── __init__.py
    │   └── test_app.py
    └── urls.py

Focus on accounts/models.py, where models for the accounts application are placed.

myproject/apps/accounts/models.py

from datetime import datetime

from my_project.utils import get_db_connection

import edgy

registry = get_db_connection()


class User(edgy.Model):
    """
    Base model for a user
    """

    first_name: str = edgy.CharField(max_length=150)
    last_name: str = edgy.CharField(max_length=150)
    username: str = edgy.CharField(max_length=150, unique=True)
    email: str = edgy.EmailField(max_length=120, unique=True)
    password: str = edgy.CharField(max_length=128)
    last_login: datetime = edgy.DateTimeField(null=True)
    is_active: bool = edgy.BooleanField(default=True)
    is_staff: bool = edgy.BooleanField(default=False)
    is_superuser: bool = edgy.BooleanField(default=False)

    class Meta:
        registry = registry

Use preloads to load the model file:

myproject/configs/settings.py

from typing import Optional, Union

from edgy import EdgySettings


class MyMigrationSettings(EdgySettings):
    # here we notify about the models import path
    preloads: list[str] = ["myproject.apps.accounts.models"]
    # here we can set the databases which should be used in migrations, by default (None,)
    migrate_databases: Union[list[Union[str, None]], tuple[Union[str, None], ...]] = (None,)

Set migrate_databases if additional databases are used.

Generating and Working with Migrations¶

Ensure you've read the Usage section and have everything set up.

Edgy's internal client, edgy, manages the migration process.

Refer to the project structure at the beginning of this document.

Note

The provided structure is for demonstration purposes; you can use any structure.

Danger

Migrations can be generated anywhere, but be mindful of paths and dependencies. It's recommended to place them at the project root.

Assuming your application is in my_project/main.py, follow these steps.

Environment Variables¶

Edgy uses the following environment variables for migrations:

EDGY_DATABASE: Restricts migrations to the specified database metadata. Use a whitespace for the main database. Special mode when used with EDGY_DATABASE_URL.
EDGY_DATABASE_URL: Two modes:
1. EDGY_DATABASE is empty: Retrieves metadata via URL. Default database used if no match, with differing URL.
2. EDGY_DATABASE is not empty: Uses metadata of the named database with a different URL.

Use the migrate_databases setting instead of environment variables.

Warning

Spaces can be invisible. Verify EDGY_DATABASE for spaces or whitespace.

Tip

Change MAIN_DATABASE_NAME in env.py for a different main database name.

Initialize the Migrations Folder¶

To begin, generate the migrations folder.

# code is in myproject.main
edgy init
# or specify an entrypoint module explicitly
# edgy --app myproject.main_test init

The discovery mechanism automatically locates the entrypoint, but you can also provide it explicitly using --app.

The optional --app parameter specifies the application's location in module_app format, a necessity due to Edgy's framework-agnostic nature.

Edgy requires the module to automatically set the instance (see Connections), enabling it to determine the registry and application object.

The location where you execute the init command determines where the migrations folder is created.

For example, my_project.main_test indicates your application is in myproject/main_test.py, and the migration folder will be placed in the current directory.

After generating the migrations, the project structure will resemble this:

.
└── README.md
└── .gitignore
├── migrations
│   ├── alembic.ini
│   ├── env.py
│   ├── README
│   ├── script.py.mako
│   └── versions
└── myproject
    ├── __init__.py
    ├── apps
    │   ├── __init__.py
    │   └── accounts
    │       ├── __init__.py
    │       ├── tests.py
    │       └── v1
    │           ├── __init__.py
    │           ├── schemas.py
    │           ├── urls.py
    │           └── views.py
    ├── configs
    │   ├── __init__.py
    │   ├── development
    │   │   ├── __init__.py
    │   │   └── settings.py
    │   ├── settings.py
    │   └── testing
    │       ├── __init__.py
    │       └── settings.py
    ├── main.py
    ├── serve.py
    ├── utils.py
    ├── tests
    │   ├── __init__.py
    │   └── test_app.py
    └── urls.py

The migrations folder and its contents are automatically generated and tailored to Edgy's requirements.

Templates¶

Edgy offers various migration templates to customize the generation process.

default (Default): Uses hashed database names and is compatible with Flask-Migrate multi-database migrations.
plain: Uses plain database names (extra databases must be valid identifiers) and is compatible with Flask-Migrate. Extra database names are restricted to Python identifiers.
url: Uses database URLs for hashing, suitable for non-local database environments. Requires adapting env.py due to incompatibility with Flask-Migrate. URL parameters used for hashing are f"{url.username}@{url.hostname}:{url.port}/{url.database}".
sequential: Uses a sequence of numbers for migration filenames (e.g., 0001_<SOMETHING>).

Use these templates with:

edgy init -t plain

List all available templates:

edgy list_templates

You can also use templates from the filesystem:

Example how to use the singledb template from tests

edgy --app myproject.main init -t tests/cli/custom_singledb

Templates are starting points and may require customization.

Generate the First Migrations¶

Generate your first migration.

Assume your accounts application models are in models.py. Define a User model:

my_project/apps/accounts/models.py

from datetime import datetime

from my_project.utils import get_db_connection

import edgy

registry = get_db_connection()


class User(edgy.Model):
    """
    Base model for a user
    """

    first_name: str = edgy.CharField(max_length=150)
    last_name: str = edgy.CharField(max_length=150)
    username: str = edgy.CharField(max_length=150, unique=True)
    email: str = edgy.EmailField(max_length=120, unique=True)
    password: str = edgy.CharField(max_length=128)
    last_login: datetime = edgy.DateTimeField(null=True)
    is_active: bool = edgy.BooleanField(default=True)
    is_staff: bool = edgy.BooleanField(default=False)
    is_superuser: bool = edgy.BooleanField(default=False)

    class Meta:
        registry = registry

Ensure the models are accessible for discovery. For Esmerald, add the User model to my_project/apps/accounts/__init__.py:

my_project/apps/accounts/__init__.py

from .models import User

Note

Edgy, being framework-agnostic, doesn't use hard-coded detection like Django's INSTALLED_APPS. Use preloads and imports to load models.

Generate the migration:

$ edgy makemigrations

The new migration will be in migrations/versions/:

.
└── README.md
└── .gitignore
├── migrations
│   ├── alembic.ini
│   ├── env.py
│   ├── README
│   ├── script.py.mako
│   └── versions
│       └── d3725dd11eef_.py
└── myproject
    ...

Add a message to the migration:

$ edgy makemigrations -m "Initial migrations"

.
└── README.md
└── .gitignore
├── migrations
│   ├── alembic.ini
│   ├── env.py
│   ├── README
│   ├── script.py.mako
│   └── versions
│       └── d3725dd11eef_initial_migrations.py
└── myproject
    ...

Migrate Your Database¶

Apply the migrations:

$ edgy migrate

Change Models and Generate Migrations¶

Modify your models and generate new migrations:

Generate new migrations:

$ edgy makemigrations

Apply them:

$ edgy migrate

More Migration Commands¶

Access available commands with --help:

Edgy Command-Line¶

List available Edgy options:

$ edgy --help

View options for a specific command (e.g., merge):

$ edgy merge --help

Usage: edgy merge [OPTIONS] [REVISIONS]...

  Merge two revisions together, creating a new revision file

Options:
  --rev-id TEXT         Specify a hardcoded revision id instead of generating
                        one
  --branch-label TEXT   Specify a branch label to apply to the new revision
  -m, --message TEXT    Merge revision message
  -d, --directory TEXT  Migration script directory (default is "migrations")
  --help                Show this message and exit.

This applies to all Edgy commands.

References¶

Edgy's command-line interface is user-friendly.

Edgy migrations use Alembic, so commands are similar, with two exceptions:

makemigrations: Calls Alembic's migrate.
migrate: Calls Alembic's upgrade.

Edgy uses more intuitive names.

Migrate to new non-nullable fields¶

Sometimes you want to add fields to a model which are required afterwards in the database. Here are some ways to archive this.

With explicit server_default (`allow_auto_compute_server_defaults=False`)¶

This is a bit more work and requires a supported field (all single-column fields and some multiple-column fields like CompositeField). It works as follows:

Add a column with a server_default which is used by the migrations.
Create the migration and migrate.
Remove the server_default and create another migration.

Here is a basic example:

Create the field with a server_default

class CustomModel(edgy.Model):
    active: bool = edgy.fields.BooleanField(server_default=sqlalchemy.text("true"))
    ...

Generate the migrations and migrate
```
edgy makemigration
edgy migrate
```

Remove the server_default

class CustomModel(edgy.Model):
    active: bool = edgy.fields.BooleanField()
    ...

Generate the migrations without the server_default and migrate
```
edgy makemigration
edgy migrate
```

With implicit server_default (`allow_auto_compute_server_defaults=True` (default))¶

This is the easiest way; it only works with fields which allow auto_compute_server_default, which are the most. Notable exceptions are Relationship fields and FileFields.

You just add a default... and that was it.

Create the field with a default

class CustomModel(edgy.Model):
    active: bool = edgy.fields.BooleanField(default=True)
    ...

Generate the migrations and migrate
```
edgy makemigration
edgy migrate
```

In case of allow_auto_compute_server_defaults=False you can enable the auto-compute of a server_default by passing auto_compute_server_default=True to the field. The first step would be here:

class CustomModel(edgy.Model):
    active: bool = edgy.fields.BooleanField(default=True, auto_compute_server_default=True)
    ...

To disable the behaviour for one field you can either pass auto_compute_server_default=False or server_default=None to the field.

With null-field¶

Null-field is a feature to make fields nullable for one makemigration/revision. You can either specify model:field_name or just :field_name for automatic detection of models. Non-existing models are ignored, and only models in registry.models are migrated. In the migration file, you will find a construct monkay.instance.registry.apply_default_force_nullable_fields(...). The model_defaults argument can be used to provide one-time defaults that overwrite all other defaults. You can also pass callables, which are executed in context of the extract_column_values method and have all of the context variables available.

Let's see how to implement the last example with null-field and we add also ContentTypes. 1. Add the field with the default (and no server-default).

class CustomModel(edgy.Model):
    active: bool = edgy.fields.BooleanField(default=True, server_default=None)
    ...

2. Apply null-field to CustomModel:active and also for all models with active content_type.

edgy makemigration --nf CustomModel:active --nf :content_type
edgy migrate

3. Now create a cleanup migration.

edgy makemigration
edgy migrate

Tip

In case you mess up the null-fields, you can also fix them manually in the script file. You can also specify custom defaults for fields.

Multi-Database Migrations¶

Edgy supports multi-database migrations. Continue using single-database migrations or update env.py and existing migrations.

Migrate from Flask-Migrate¶

Flask-Migrate was the basis for the deprecated Migrate object. Use edgy.Instance and migration settings.

edgy.Instance takes (registry, app=None) as arguments, unlike Flask-Migrate's (app, database). Settings are now in the Edgy settings object.

Migrate `env.py`¶

Replace env.py with Edgy's default. Adjust migrations if necessary.

Migrate from Single-Database Migrations¶

Adapt old migrations for multi-database support:

Add an engine_name parameter to upgrade/downgrade functions, defaulting to ''.
Prevent execution if engine_name is not empty.

For different default databases, add the database to extra and prevent execution for other names.

Example:

def downgrade():
    ...

Becomes:

def downgrade(engine_name: str = ""):
    if engine_name != "":  # or dbname you want
        return

Multi-Schema Migrations¶

Enable multi-schema migrations by setting multi_schema in Migration Settings. Filter schemas using schema parameters.

Migrations in Libraries and Middleware¶

Edgy has not only an interface for main applications but also for libraries. We can use Edgy even in an ASGI middleware when the main project is Django.

To integrate, there are two ways:

Extensions¶

Add an extension which, when included in Edgy settings extensions, injects the model into the current registry.

Pros:

Reuses the registry and database.
Migrations contain the injected models.

Cons:

Requires Edgy as the main application.
Only one registry is supported.
Not completely independent. Affected by settings.

Automigrations¶

Provide an extra registry with the automigrate_config parameter filled with an EdgySettings object/string to the config.

Pros:

Can use its own registry and database. Completely independent from the main application.
Ideal for ASGI middleware.
In the best case, no user interaction is required.

Cons:

Requires DDL access on the database it is using. In the case of the offline mode of Alembic, all libraries must be accessed manually via edgy migrate -d librarypath/migrations.
May need to be disabled via allow_automigrations=False in Edgy settings in case of missing DDL permissions.

What to Use¶

The optimal way is to provide the user the extension way and provide a fallback way with automigrations which reuses the extension way to inject into a registry.

import edgy
from monkay import ExtensionProtocol
from edgy import Registry, EdgySettings


class User(edgy.Model):
    age: int = edgy.IntegerField(gte=18)
    is_active: bool = edgy.BooleanField(default=True)


class AddUserExtension(ExtensionProtocol):
    name = "add_user"

    def apply(self, monkay_instance):
        User.add_to_registry(monkay_instance.registry)


class LibraryConfig(EdgySettings):
    extensions = [AddUserExtension()]


async def create_custom_registry():
    return Registry("DB_URL", automigrate_config=LibraryConfig)


def get_application(): ...


app = create_custom_registry().asgi(get_application())

This way, the user is free to decide which way to use. If this is not enough, they can also directly attach the models to a registry and provide their own migration settings. But this is similar to automigrations just without extensions.

import edgy
from monkay import ExtensionProtocol
from edgy import Registry, EdgySettings


class User(edgy.Model):
    age: int = edgy.IntegerField(gte=18)
    is_active: bool = edgy.BooleanField(default=True)


class AddUserExtension(ExtensionProtocol):
    name = "add_user"

    def apply(self, monkay_instance):
        User.add_to_registry(monkay_instance.registry)


class LibraryConfig(EdgySettings):
    extensions = [AddUserExtension()]


class Config(EdgySettings):
    extensions = [AddUserExtension()]


async def create_custom_registry():
    return Registry("DB_URL", automigrate_config=LibraryConfig)


def get_application():
    edgy.monkay.settings = Config
    return Registry("DB_URL").asgi(...)


app = get_application()

In special environments without DDL change permissions, you need to disable the automigrations via configuration and extract the migrations with --sql.

import edgy
from monkay import ExtensionProtocol
from edgy import Registry, EdgySettings


class User(edgy.Model):
    age: int = edgy.IntegerField(gte=18)
    is_active: bool = edgy.BooleanField(default=True)


class AddUserExtension(ExtensionProtocol):
    name = "add_user"

    def apply(self, monkay_instance):
        User.add_to_registry(monkay_instance.registry)


class LibraryConfig(EdgySettings):
    extensions = [AddUserExtension()]


class Config(EdgySettings):
    allow_automigrations = False


async def create_custom_registry():
    return Registry("DB_URL", automigrate_config=LibraryConfig)


def get_application():
    edgy.monkay.settings = Config
    return Registry("DB_URL").asgi(...)


app = create_custom_registry().asgi(get_application())

# get sql migrations with
# edgy -d library/migrations_folder migrate --sql
# this is also the way for downgrades as automigrations does only work for upgrades

Offline Mode¶

Sometimes, without DDL access, we need the offline mode. Offline means the database structure is only read, not modified, and the migrations are output as SQL scripts for the user to provide to the DBA.

Here we need the Environment Variables and add to edgy migrate --sql to get the SQL scripts for migrations one-by-one.

This can be quite time-intensive, especially if libraries need their own tables.

You may consider in this case to use the Extension way of integration or to use a different database like SQLite for the library registries which do not have the restrictions.

Migrations¶

Important¶

Project Structure for this Document¶

Migration Object¶

Parameters¶

Migration Settings¶

Usage¶

Using Esmerald¶

Using FastAPI¶

Using Starlette¶

Using Other Frameworks¶

Example¶

Generating and Working with Migrations¶

Environment Variables¶

Initialize the Migrations Folder¶

Templates¶

Generate the First Migrations¶

Migrate Your Database¶

Change Models and Generate Migrations¶

More Migration Commands¶

Edgy Command-Line¶

References¶

Migrate to new non-nullable fields¶

With explicit server_default (allow_auto_compute_server_defaults=False)¶

With implicit server_default (allow_auto_compute_server_defaults=True (default))¶

With null-field¶

Multi-Database Migrations¶

Migrate from Flask-Migrate¶

Migrate env.py¶

Migrate from Single-Database Migrations¶

Multi-Schema Migrations¶

Migrations in Libraries and Middleware¶

Extensions¶

Automigrations¶

What to Use¶

Offline Mode¶

With explicit server_default (`allow_auto_compute_server_defaults=False`)¶

With implicit server_default (`allow_auto_compute_server_defaults=True` (default))¶

Migrate `env.py`¶