Migrations¶
Database migration tools are essential for managing incremental database changes.
Edgy, built on top of SQLAlchemy core, includes a powerful internal migration tool.
This tool simplifies the management of models and their corresponding migrations.
Inspired by Flask-Migrations, Edgy's migration tool is framework-agnostic, making it usable anywhere.
Important¶
Before proceeding, familiarize yourself with Edgy's application discovery methods.
The following examples will use the --app
and environment variable approach (see Discovery), but auto-discovery (see Auto Discovery) is equally valid.
Project Structure for this Document¶
For clarity, we'll use the following project structure in our examples:
.
├── README.md
├── .gitignore
└── myproject
├── __init__.py
├── apps
│ ├── __init__.py
│ └── accounts
│ ├── __init__.py
│ ├── tests.py
│ └── v1
│ ├── __init__.py
│ ├── schemas.py
│ ├── urls.py
│ └── views.py
├── configs
│ ├── __init__.py
│ ├── development
│ │ ├── __init__.py
│ │ └── settings.py
│ ├── settings.py
│ └── testing
│ ├── __init__.py
│ └── settings.py
├── main.py
├── serve.py
├── utils.py
├── tests
│ ├── __init__.py
│ └── test_app.py
└── urls.py
Migration Object¶
Edgy requires a Migration
object to manage migrations consistently and cleanly, similar to Django migrations.
This Migration
class is framework-independent. Edgy ensures it integrates with any desired framework upon creation.
This flexibility makes Edgy uniquely adaptable to frameworks like Esmerald, Starlette, FastAPI, and Sanic.
from edgy import Instance, monkay
monkay.set_instance(Instance(registry=registry, app=None))
Parameters¶
The Instance
object accepts the following parameters:
- registry: The model registry. It must be an instance of
edgy.Registry
or anAssertionError
is raised. - app: An optional application instance.
Migration Settings¶
Migrations now utilize Edgy settings. Configuration options are located in edgy/conf/global_settings.py
.
Key settings include:
multi_schema
(bool / regex string / regex pattern): (Default:False
). Enables multi-schema migrations.True
for all schemas, a regex for specific schemas.ignore_schema_pattern
(None / regex string / regex pattern): (Default:"information_schema"
). When using multi-schema migrations, ignore schemas matching this regex pattern.migrate_databases
(tuple): (Default:(None,)
). Specifies databases to migrate.migration_directory
(str): (Default:"migrations"
). Path to the Alembic migration folder. Overridable per command via-d
or--directory
parameter.-
alembic_ctx_kwargs
(dict): Extra arguments for Alembic. Default:{ "compare_type": True, "render_as_batch": True, }
Usage¶
Using the Instance
class is straightforward. For advanced usage, see the LRU cache technique in Tips and Tricks.
We'll use a utils.py
file to store database and registry information.
from functools import lru_cache
@lru_cache()
def get_db_connection():
from edgy import Registry
# use echo=True for getting the connection infos printed, extra kwargs are passed to main database
return Registry("postgresql+asyncpg://user:pass@localhost:5432/my_database", echo=True)
This ensures object creation only once.
Now, use the Migration
object in your application.
Using Esmerald¶
#!/usr/bin/env python
import os
import sys
from pathlib import Path
from esmerald import Esmerald, Include
from my_project.utils import get_db_connection
def build_path():
"""
Builds the path of the project and project root.
"""
Path(__file__).resolve().parent.parent
SITE_ROOT = os.path.dirname(os.path.realpath(__file__))
if SITE_ROOT not in sys.path:
sys.path.append(SITE_ROOT)
sys.path.append(os.path.join(SITE_ROOT, "apps"))
def get_application():
"""
Encapsulating in methods can be useful for controlling the import order but is optional.
"""
# first call build_path
build_path()
# because edgy tries to load settings eagerly
from edgy import monkay, Instance
registry = get_db_connection()
# ensure the settings are loaded
monkay.evaluate_settings(ignore_import_errors=False)
app = registry.asgi(
Esmerald(
routes=[Include(namespace="my_project.urls")],
)
)
monkay.set_instance(Instance(registry=registry, app=app))
return app
app = get_application()
Using FastAPI¶
Edgy's framework-agnostic nature allows its use in FastAPI applications.
#!/usr/bin/env python
import os
import sys
from pathlib import Path
from fastapi import FastAPI
from my_project.utils import get_db_connection
def build_path():
"""
Builds the path of the project and project root.
"""
Path(__file__).resolve().parent.parent
SITE_ROOT = os.path.dirname(os.path.realpath(__file__))
if SITE_ROOT not in sys.path:
sys.path.append(SITE_ROOT)
sys.path.append(os.path.join(SITE_ROOT, "apps"))
def get_application():
"""
Encapsulating in methods can be useful for controlling the import order but is optional.
"""
# first call build_path
build_path()
# because edgy tries to load settings eagerly
from edgy import Instance, monkay
registry = get_db_connection()
# ensure the settings are loaded
monkay.evaluate_settings(ignore_import_errors=False)
app = registry.asgi(FastAPI(__name__))
monkay.set_instance(Instance(registry=registry, app=app))
return app
app = get_application()
Using Starlette¶
Similarly, Edgy works with Starlette.
#!/usr/bin/env python
import os
import sys
from pathlib import Path
from starlette.applications import Starlette
from my_project.utils import get_db_connection
def build_path():
"""
Builds the path of the project and project root.
"""
Path(__file__).resolve().parent.parent
SITE_ROOT = os.path.dirname(os.path.realpath(__file__))
if SITE_ROOT not in sys.path:
sys.path.append(SITE_ROOT)
sys.path.append(os.path.join(SITE_ROOT, "apps"))
def get_application():
"""
Encapsulating in methods can be useful for controlling the import order but is optional.
"""
# first call build_path
build_path()
# because edgy tries to load settings eagerly
from edgy import monkay, Instance
registry = get_db_connection()
# ensure the settings are loaded
monkay.evaluate_settings(ignore_import_errors=False)
app = registry.asgi(Starlette())
monkay.set_instance(Instance(registry=registry, app=app))
return app
app = get_application()
Using Other Frameworks¶
Edgy's design requires no framework-specific parameters, allowing integration with frameworks like Quart, Ella, or Sanic.
Example¶
Consider an application with the following structure:
.
├── README.md
├── .gitignore
└── myproject
├── __init__.py
├── apps
│ ├── __init__.py
│ └── accounts
│ ├── __init__.py
│ ├── tests.py
│ ├── models.py
│ └── v1
│ ├── __init__.py
│ ├── schemas.py
│ ├── urls.py
│ └── views.py
├── configs
│ ├── __init__.py
│ ├── development
│ │ ├── __init__.py
│ │ └── settings.py
│ ├── settings.py
│ └── testing
│ ├── __init__.py
│ └── settings.py
├── main.py
├── serve.py
├── utils.py
├── tests
│ ├── __init__.py
│ └── test_app.py
└── urls.py
Focus on accounts/models.py
, where models for the accounts
application are placed.
from datetime import datetime
from my_project.utils import get_db_connection
import edgy
registry = get_db_connection()
class User(edgy.Model):
"""
Base model for a user
"""
first_name: str = edgy.CharField(max_length=150)
last_name: str = edgy.CharField(max_length=150)
username: str = edgy.CharField(max_length=150, unique=True)
email: str = edgy.EmailField(max_length=120, unique=True)
password: str = edgy.CharField(max_length=128)
last_login: datetime = edgy.DateTimeField(null=True)
is_active: bool = edgy.BooleanField(default=True)
is_staff: bool = edgy.BooleanField(default=False)
is_superuser: bool = edgy.BooleanField(default=False)
class Meta:
registry = registry
Use preloads
to load the model file:
from typing import Optional, Union
from edgy import EdgySettings
class MyMigrationSettings(EdgySettings):
# here we notify about the models import path
preloads: list[str] = ["myproject.apps.accounts.models"]
# here we can set the databases which should be used in migrations, by default (None,)
migrate_databases: Union[list[Union[str, None]], tuple[Union[str, None], ...]] = (None,)
Set migrate_databases
if additional databases are used.
Generating and Working with Migrations¶
Ensure you've read the Usage section and have everything set up.
Edgy's internal client, edgy
, manages the migration process.
Refer to the project structure at the beginning of this document.
Note
The provided structure is for demonstration purposes; you can use any structure.
Danger
Migrations can be generated anywhere, but be mindful of paths and dependencies. It's recommended to place them at the project root.
Assuming your application is in my_project/main.py
, follow these steps.
Environment Variables¶
Edgy uses the following environment variables for migrations:
- EDGY_DATABASE: Restricts migrations to the specified database metadata. Use a whitespace for the main database. Special mode when used with EDGY_DATABASE_URL.
- EDGY_DATABASE_URL: Two modes:
- EDGY_DATABASE is empty: Retrieves metadata via URL. Default database used if no match, with differing URL.
- EDGY_DATABASE is not empty: Uses metadata of the named database with a different URL.
Use the migrate_databases
setting instead of environment variables.
Warning
Spaces can be invisible. Verify EDGY_DATABASE for spaces or whitespace.
Tip
Change MAIN_DATABASE_NAME
in env.py
for a different main database name.
Initialize the Migrations Folder¶
To begin, generate the migrations folder.
# code is in myproject.main
edgy init
# or specify an entrypoint module explicitly
# edgy --app myproject.main_test init
The discovery mechanism automatically locates the entrypoint, but you can also provide it explicitly using --app
.
The optional --app
parameter specifies the application's location in module_app
format, a necessity due to Edgy's framework-agnostic nature.
Edgy requires the module to automatically set the instance (see Connections), enabling it to determine the registry and application object.
The location where you execute the init
command determines where the migrations folder is created.
For example, my_project.main_test
indicates your application is in myproject/main_test.py
, and the migration folder will be placed in the current directory.
After generating the migrations, the project structure will resemble this:
.
└── README.md
└── .gitignore
├── migrations
│ ├── alembic.ini
│ ├── env.py
│ ├── README
│ ├── script.py.mako
│ └── versions
└── myproject
├── __init__.py
├── apps
│ ├── __init__.py
│ └── accounts
│ ├── __init__.py
│ ├── tests.py
│ └── v1
│ ├── __init__.py
│ ├── schemas.py
│ ├── urls.py
│ └── views.py
├── configs
│ ├── __init__.py
│ ├── development
│ │ ├── __init__.py
│ │ └── settings.py
│ ├── settings.py
│ └── testing
│ ├── __init__.py
│ └── settings.py
├── main.py
├── serve.py
├── utils.py
├── tests
│ ├── __init__.py
│ └── test_app.py
└── urls.py
The migrations folder and its contents are automatically generated and tailored to Edgy's requirements.
Templates¶
Edgy offers various migration templates to customize the generation process.
default
(Default): Uses hashed database names and is compatible with Flask-Migrate multi-database migrations.plain
: Uses plain database names (extra databases must be valid identifiers) and is compatible with Flask-Migrate. Extra database names are restricted to Python identifiers.url
: Uses database URLs for hashing, suitable for non-local database environments. Requires adaptingenv.py
due to incompatibility with Flask-Migrate. URL parameters used for hashing aref"{url.username}@{url.hostname}:{url.port}/{url.database}"
.sequential
: Uses a sequence of numbers for migration filenames (e.g.,0001_<SOMETHING>
).
Use these templates with:
edgy init -t plain
List all available templates:
edgy list_templates
You can also use templates from the filesystem:
edgy --app myproject.main init -t tests/cli/custom_singledb
Templates are starting points and may require customization.
Generate the First Migrations¶
Generate your first migration.
Assume your accounts
application models are in models.py
. Define a User
model:
from datetime import datetime
from my_project.utils import get_db_connection
import edgy
registry = get_db_connection()
class User(edgy.Model):
"""
Base model for a user
"""
first_name: str = edgy.CharField(max_length=150)
last_name: str = edgy.CharField(max_length=150)
username: str = edgy.CharField(max_length=150, unique=True)
email: str = edgy.EmailField(max_length=120, unique=True)
password: str = edgy.CharField(max_length=128)
last_login: datetime = edgy.DateTimeField(null=True)
is_active: bool = edgy.BooleanField(default=True)
is_staff: bool = edgy.BooleanField(default=False)
is_superuser: bool = edgy.BooleanField(default=False)
class Meta:
registry = registry
Ensure the models are accessible for discovery. For Esmerald, add the User
model to my_project/apps/accounts/__init__.py
:
from .models import User
Note
Edgy, being framework-agnostic, doesn't use hard-coded detection like Django's INSTALLED_APPS
. Use preloads
and imports to load models.
Generate the migration:
$ edgy makemigrations
The new migration will be in migrations/versions/
:
.
└── README.md
└── .gitignore
├── migrations
│ ├── alembic.ini
│ ├── env.py
│ ├── README
│ ├── script.py.mako
│ └── versions
│ └── d3725dd11eef_.py
└── myproject
...
Add a message to the migration:
$ edgy makemigrations -m "Initial migrations"
.
└── README.md
└── .gitignore
├── migrations
│ ├── alembic.ini
│ ├── env.py
│ ├── README
│ ├── script.py.mako
│ └── versions
│ └── d3725dd11eef_initial_migrations.py
└── myproject
...
Migrate Your Database¶
Apply the migrations:
$ edgy migrate
Change Models and Generate Migrations¶
Modify your models and generate new migrations:
Generate new migrations:
$ edgy makemigrations
Apply them:
$ edgy migrate
More Migration Commands¶
Access available commands with --help
:
Edgy Command-Line¶
List available Edgy options:
$ edgy --help
View options for a specific command (e.g., merge
):
$ edgy merge --help
Usage: edgy merge [OPTIONS] [REVISIONS]...
Merge two revisions together, creating a new revision file
Options:
--rev-id TEXT Specify a hardcoded revision id instead of generating
one
--branch-label TEXT Specify a branch label to apply to the new revision
-m, --message TEXT Merge revision message
-d, --directory TEXT Migration script directory (default is "migrations")
--help Show this message and exit.
This applies to all Edgy commands.
References¶
Edgy's command-line interface is user-friendly.
Edgy migrations use Alembic, so commands are similar, with two exceptions:
makemigrations
: Calls Alembic'smigrate
.migrate
: Calls Alembic'supgrade
.
Edgy uses more intuitive names.
Migrate to new non-nullable fields¶
Sometimes you want to add fields to a model which are required afterwards in the database. Here are some ways to archive this.
With explicit server_default (allow_auto_compute_server_defaults=False
)¶
This is a bit more work and requires a supported field (all single-column fields and some multiple-column fields like CompositeField). It works as follows:
- Add a column with a server_default which is used by the migrations.
- Create the migration and migrate.
- Remove the server_default and create another migration.
Here is a basic example:
- Create the field with a server_default
class CustomModel(edgy.Model): active: bool = edgy.fields.BooleanField(server_default=sqlalchemy.text("true")) ...
- Generate the migrations and migrate
edgy makemigration edgy migrate
- Remove the server_default
class CustomModel(edgy.Model): active: bool = edgy.fields.BooleanField() ...
- Generate the migrations without the server_default and migrate
edgy makemigration edgy migrate
With implicit server_default (allow_auto_compute_server_defaults=True
(default))¶
This is the easiest way; it only works with fields which allow auto_compute_server_default
, which are the most.
Notable exceptions are Relationship fields and FileFields.
You just add a default... and that was it.
- Create the field with a default
class CustomModel(edgy.Model): active: bool = edgy.fields.BooleanField(default=True) ...
- Generate the migrations and migrate
edgy makemigration edgy migrate
In case of allow_auto_compute_server_defaults=False
you can enable the auto-compute of a server_default
by passing auto_compute_server_default=True
to the field. The first step would be here:
class CustomModel(edgy.Model):
active: bool = edgy.fields.BooleanField(default=True, auto_compute_server_default=True)
...
To disable the behaviour for one field you can either pass auto_compute_server_default=False
or server_default=None
to the field.
With null-field¶
Null-field is a feature to make fields nullable for one makemigration/revision. You can either specify
model:field_name
or just :field_name
for automatic detection of models.
Non-existing models are ignored, and only models in registry.models
are migrated.
In the migration file, you will find a construct monkay.instance.registry.apply_default_force_nullable_fields(...)
.
The model_defaults
argument can be used to provide one-time defaults that overwrite all other defaults.
You can also pass callables, which are executed in context of the extract_column_values
method and have all of the context variables available.
Let's see how to implement the last example with null-field and we add also ContentTypes. 1. Add the field with the default (and no server-default).
class CustomModel(edgy.Model):
active: bool = edgy.fields.BooleanField(default=True, server_default=None)
...
edgy makemigration --nf CustomModel:active --nf :content_type
edgy migrate
edgy makemigration
edgy migrate
Tip
In case you mess up the null-fields, you can also fix them manually in the script file. You can also specify custom defaults for fields.
Multi-Database Migrations¶
Edgy supports multi-database migrations. Continue using single-database migrations or update env.py
and existing migrations.
Migrate from Flask-Migrate¶
Flask-Migrate was the basis for the deprecated Migrate
object. Use edgy.Instance
and migration settings.
edgy.Instance
takes (registry, app=None)
as arguments, unlike Flask-Migrate's (app, database)
. Settings are now in the Edgy settings object.
Migrate env.py
¶
Replace env.py
with Edgy's default. Adjust migrations if necessary.
Migrate from Single-Database Migrations¶
Adapt old migrations for multi-database support:
- Add an
engine_name
parameter toupgrade
/downgrade
functions, defaulting to''
. - Prevent execution if
engine_name
is not empty.
For different default databases, add the database to extra and prevent execution for other names.
Example:
def downgrade():
...
Becomes:
def downgrade(engine_name: str = ""):
if engine_name != "": # or dbname you want
return
Multi-Schema Migrations¶
Enable multi-schema migrations by setting multi_schema
in Migration Settings. Filter schemas using schema parameters.
Migrations in Libraries and Middleware¶
Edgy has not only an interface for main applications but also for libraries. We can use Edgy even in an ASGI middleware when the main project is Django.
To integrate, there are two ways:
Extensions¶
Add an extension which, when included in Edgy settings extensions, injects the model into the current registry.
Pros:
- Reuses the registry and database.
- Migrations contain the injected models.
Cons:
- Requires Edgy as the main application.
- Only one registry is supported.
- Not completely independent. Affected by settings.
Automigrations¶
Provide an extra registry with the automigrate_config
parameter filled with an EdgySettings
object/string to the config.
Pros:
- Can use its own registry and database. Completely independent from the main application.
- Ideal for ASGI middleware.
- In the best case, no user interaction is required.
Cons:
- Requires DDL access on the database it is using. In the case of the offline mode of Alembic,
all libraries must be accessed manually via
edgy migrate -d librarypath/migrations
. - May need to be disabled via
allow_automigrations=False
in Edgy settings in case of missing DDL permissions.
What to Use¶
The optimal way is to provide the user the extension way and provide a fallback way with automigrations which reuses the extension way to inject into a registry.
import edgy
from monkay import ExtensionProtocol
from edgy import Registry, EdgySettings
class User(edgy.Model):
age: int = edgy.IntegerField(gte=18)
is_active: bool = edgy.BooleanField(default=True)
class AddUserExtension(ExtensionProtocol):
name = "add_user"
def apply(self, monkay_instance):
User.add_to_registry(monkay_instance.registry)
class LibraryConfig(EdgySettings):
extensions = [AddUserExtension()]
async def create_custom_registry():
return Registry("DB_URL", automigrate_config=LibraryConfig)
def get_application(): ...
app = create_custom_registry().asgi(get_application())
This way, the user is free to decide which way to use. If this is not enough, they can also directly attach the models to a registry and provide their own migration settings. But this is similar to automigrations just without extensions.
import edgy
from monkay import ExtensionProtocol
from edgy import Registry, EdgySettings
class User(edgy.Model):
age: int = edgy.IntegerField(gte=18)
is_active: bool = edgy.BooleanField(default=True)
class AddUserExtension(ExtensionProtocol):
name = "add_user"
def apply(self, monkay_instance):
User.add_to_registry(monkay_instance.registry)
class LibraryConfig(EdgySettings):
extensions = [AddUserExtension()]
class Config(EdgySettings):
extensions = [AddUserExtension()]
async def create_custom_registry():
return Registry("DB_URL", automigrate_config=LibraryConfig)
def get_application():
edgy.monkay.settings = Config
return Registry("DB_URL").asgi(...)
app = get_application()
In special environments without DDL change permissions, you need to disable the automigrations via configuration and extract the migrations with --sql
.
import edgy
from monkay import ExtensionProtocol
from edgy import Registry, EdgySettings
class User(edgy.Model):
age: int = edgy.IntegerField(gte=18)
is_active: bool = edgy.BooleanField(default=True)
class AddUserExtension(ExtensionProtocol):
name = "add_user"
def apply(self, monkay_instance):
User.add_to_registry(monkay_instance.registry)
class LibraryConfig(EdgySettings):
extensions = [AddUserExtension()]
class Config(EdgySettings):
allow_automigrations = False
async def create_custom_registry():
return Registry("DB_URL", automigrate_config=LibraryConfig)
def get_application():
edgy.monkay.settings = Config
return Registry("DB_URL").asgi(...)
app = create_custom_registry().asgi(get_application())
# get sql migrations with
# edgy -d library/migrations_folder migrate --sql
# this is also the way for downgrades as automigrations does only work for upgrades
Offline Mode¶
Sometimes, without DDL access, we need the offline mode. Offline means the database structure is only read, not modified, and the migrations are output as SQL scripts for the user to provide to the DBA.
Here we need the Environment Variables and add to edgy migrate
--sql
to get the SQL scripts for migrations one-by-one.
This can be quite time-intensive, especially if libraries need their own tables.
You may consider in this case to use the Extension way of integration or to use a different database like SQLite for the library registries which do not have the restrictions.