Merge pull request #3250 from simolus3/drift_migrator_tool

feat: all in one migration command

.github/workflows/main.yml

@@ -280,6 +280,7 @@ jobs:
  dart test
  test_flutter:
  runs-on: ubuntu-latest
+ needs: [setup]
  strategy:
  matrix:
  # We want to support the two latest stable Flutter versions
@@ -299,6 +300,22 @@ jobs:
  shell: bash
  - name: Install dependencies in drift/example/app
  run: melos bootstrap --scope app
+
+ - name: Download sqlite3
+ uses: actions/download-artifact@v4
+ with:
+ name: sqlite3
+ path: drift/.dart_tool/sqlite3/
+ - name: Use downloaded sqlite3
+ shell: bash
+ run: |
+ chmod a+x drift/.dart_tool/sqlite3/latest/sqlite3
+ echo $(realpath drift/.dart_tool/sqlite3/latest) >> $GITHUB_PATH
+ echo "LD_LIBRARY_PATH=$(realpath drift/.dart_tool/sqlite3/latest)" >> $GITHUB_ENV
+ - name: Check sqlite3 version
+ run: sqlite3 --version
+ shell: bash
+
  - name: Generate code
  run: dart run build_runner build --delete-conflicting-outputs
  working-directory: examples/app
@@ -327,15 +344,15 @@ jobs:
  working-directory: examples/migrations_example
  run: |
  dart run build_runner build --delete-conflicting-outputs
- dart run drift_dev schema generate drift_migrations/ test/generated/ --data-classes --companions
- dart run drift_dev schema generate drift_migrations/ lib/src/generated
+ dart run drift_dev schema generate drift_schemas/default test/generated/ --data-classes --companions
+ dart run drift_dev schema generate drift_schemas/ lib
  - name: Test
  working-directory: examples/migrations_example
  run: dart test
  - name: Test with older sqlite3
  working-directory: examples/migrations_example
  run: |
- LD_LIBRARY_PATH=$(realpath drift/.dart_tool/sqlite3/minimum) dart test
+ LD_LIBRARY_PATH=$(realpath ../../drift/.dart_tool/sqlite3/minimum) dart test
  - name: Check that extracting schema still works
  working-directory: examples/migrations_example
  run: dart run drift_dev schema dump lib/database.dart drift_migrations/

docs/build.yaml

@@ -41,6 +41,8 @@ targets:
  dialect: sqlite
  options:
  version: "3.39"
+ databases:
+ main_db: "lib/snippets/dart_api/manager.dart"
  generate_for:
  include: &modular
  - "lib/snippets/_shared/**"

docs/docs/Migrations/api.md

@@ -12,6 +12,21 @@ callback. However, the callbacks also give you an instance of `Migrator` as a
 parameter. This class knows about the target schema of the database and can be
 used to create, drop and alter most elements in your schema.
 
+## General tips 
+
+To ensure your schema stays consistent during a migration, you can wrap it in a `transaction` block.
+However, be aware that some pragmas (including `foreign_keys`) can't be changed inside transactions.
+Still, it can be useful to:
+
+- always re-enable foreign keys before using the database, by enabling them in [`beforeOpen`](#post-migration-callbacks).
+- disable foreign-keys before migrations
+- run migrations inside a transaction
+- make sure your migrations didn't introduce any inconsistencies with `PRAGMA foreign_key_check`.
+
+With all of this combined, a migration callback can look like this:
+
+{{ load_snippet('structured','lib/snippets/migrations/migrations.dart.excerpt.json') }}
+
 ## Migrating views, triggers and indices
 
 When changing the definition of a view, a trigger or an index, the easiest way

docs/docs/Migrations/exports.md

@@ -5,6 +5,12 @@ description: Store all schema versions of your app for validation.
 
 ---
 
+
+!!! warning "Important Note"
+
+ This command is specifically for exporting schemas. 
+ If you are using the `make-migrations` command, this is already done for you.
+
 By design, drift's code generator can only see the current state of your database
 schema. When you change it, it can be helpful to store a snapshot of the older
 schema in a file.
@@ -38,27 +44,16 @@ my_app
 Of course, you can also use another folder or a subfolder somewhere if that suits your workflow
 better.
 
-!!! note "Examples available"
-
-
- Exporting schemas and generating code for them can't be done with `build_runner` alone, which is
- why this setup described here is necessary.
-
- We hope it's worth it though! Verifying migrations can give you confidence that you won't run
- into issues after changing your database.
- If you get stuck along the way, don't hesitate to [open a discussion about it](https:/simolus3/drift/discussions).
-
- Also there are two examples in the drift repository which may be useful as a reference:
-
- - A [Flutter app](https:/simolus3/drift/tree/latest-release/examples/app)
- - An [example specific to migrations](https:/simolus3/drift/tree/latest-release/examples/migrations_example).
-
-
-
+Exporting schemas and generating code for them can't be done with `build_runner` alone, which is
+why this setup described here is necessary.
 
+We hope it's worth it though! Verifying migrations can give you confidence that you won't run
+into issues after changing your database.
+If you get stuck along the way, don't hesitate to [open a discussion about it](https:/simolus3/drift/discussions).
+
 ## Exporting the schema
 
-To begin, lets create the first schema representation:
+To begin, let's create the first schema representation:
 
 ```
 $ mkdir drift_schemas
@@ -93,9 +88,6 @@ $ dart run drift_dev schema dump lib/database/database.dart drift_schemas/drift_
  database file, you can do that as well! `drift_dev schema dump` recognizes a sqlite3 database file as its first
  argument and can extract the relevant schema from there.
 
-
-
-
 ## What now?
 
 Having exported your schema versions into files like this, drift tools are able

docs/docs/Migrations/index.md

@@ -5,33 +5,103 @@ description: Tooling and APIs to safely change the schema of your database.
 
 ---
 
-The strict schema of tables and columns is what enables type-safe queries to
-the database.
-But since the schema is stored in the database too, changing it needs to happen
-through migrations developed as part of your app. Drift provides APIs to make most
-migrations easy to write, as well as command-line and testing tools to ensure
-the migrations are correct.
+Drift ensures type-safe queries through a strict schema. To change this schema, you must write migrations. 
+Drift provides a range of APIs, command-line tools, and testing utilities to make writing and verifying database migrations easier and more reliable.
 
+## Guided Migrations
 
-## Drift tools
+Drift offers an all-in-one command for writing and testing migrations. 
+This tool helps you write your schema changes incrementally and generates tests to verify that your migrations are correct.
 
-Writing correct migrations is crucial to ensure that your users won't end up with a broken database
-in an inconsistent state.
-For this reason, drift offers tools that make writing and testing migrations safe and easy. By exporting
-each version of your schema to a json file, drift can reconstruct older versions of your database
-schema. This is helpful to:
+### Configuration
 
-1. Test migrations between any two versions of your app.
-2. Write migrations between older versions your app more easily, as the current schema generated by drift
- would also include subsequent changes from newer versions.
+To use the `make-migrations` command, you must add the location of your database(s) to your `build.yaml` file.
 
-For this reason, we recommend using drift's tools based on [exporting schemas](exports.md)
-for writing migrations.
-These also enable [unit tests](tests.md), giving you confidence that your schema migration
-is working correctly.
+```yaml title="build.yaml"
+targets:
+ $default:
+ builders:
+ drift_dev:
+ options:
+ databases:
+ # Required: A name for the database and it's path
+ my_database: lib/database.dart
 
+ # Optional: Add more databases
+ another_db: lib/database2.dart
+```
+
+You can also optionally specify the directory where the test files and schema (1) files are stored.
+{ .annotate }
+
+1. Drift will generate multiple schema files, one for each version of your database schema. These files are used to compare the current schema with the previous schema and generate the migration code.
+
+```yaml title="build.yaml"
+targets:
+ $default:
+ builders:
+ drift_dev:
+ options:
+ # The directory where the test files are stored: 
+ test_dir: test/drift/ # (default)
+
+ # The directory where the schema files are stored:
+ schema_dir: drift_schemas/ # (default)
+```
+
+### Usage
+
+Before you start making changes to your initial database schema, run this command to generate the initial schema file.
+
+```bash
+dart run drift_dev make-migrations
+```
+Once this initial schema file is saved, you can start making changes to your database schema.
+
+
+Once you're happy with the changes, bump the `schemaVersion` in your database class and run the command again.
+
+```bash
+dart run drift_dev make-migrations
+```
+
+This command will generate the following files:
+
+- A step-by-step migration file will be generated next to your database class. Use this function to write your migrations incrementally. See the [step-by-step migration guide](step_by_step.md) for more information.
+
+
+- Drift will also generate a test file for your migrations. After you've written your migration, run the tests to verify that your migrations are written correctly. This files will also contain a sample data integrity test for the first migration.
+
+
+If you get stuck along the way, don't hesitate to [open a discussion about it](https:/simolus3/drift/discussions).
+
+
+### Example
+
+See the [example](https:/simolus3/drift/tree/develop/examples/migrations_example) in the drift repository for a complete example of how to use the `make-migrations` command.
+
+### Switching to `make-migrations`
+
+If you've already been using the `schema` tools to write migrations, you can switch to `make-migrations` by following these steps:
+
+1. Run the `make-migrations` command to generate the initial schema file.
+2. Move all of your existing `schema` files into the schema directory for your database.
+3. Run the `make-migrations` command again to generate the step-by-step migration file and test files.
+
+## During development
+
+During development, you might be changing your schema very often and don't want to write migrations for that
+yet. You can just delete your apps' data and reinstall the app - the database will be deleted and all tables
+will be created again. Please note that uninstalling is not enough sometimes - Android might have backed up
+the database file and will re-create it when installing the app again.
+
+You can also delete and re-create all tables every time your app is opened, see [this comment](https:/simolus3/drift/issues/188#issuecomment-542682912)
+on how that can be achieved.
 
-## Manual setup 
+## Manual Migrations
+
+!!! warning "Manual migrations are error-prone"
+ Writing migrations manually is error-prone and can lead to data loss. We recommend using the `make-migrations` command to generate migrations and tests.
 
 Drift provides a migration API that can be used to gradually apply schema changes after bumping
 the `schemaVersion` getter inside the `Database` class. To use it, override the `migration`
@@ -54,28 +124,7 @@ However, be aware that drift expects the latest schema when creating SQL stateme
 For instance, when adding a new column to your database, you shouldn't run a `select` on that table before
 you've actually added the column. In general, try to avoid running queries in migration callbacks if possible.
 
-Writing migrations without any tooling support isn't easy. Since correct migrations are
-essential for app updates to work smoothly, we strongly recommend using the tools and testing
-framework provided by drift to ensure your migrations are correct.
-To do that, [export old versions](exports.md) to then use easy
-[step-by-step migrations](step_by_step.md) or [tests](tests.md).
-
-## General tips 
-
-To ensure your schema stays consistent during a migration, you can wrap it in a `transaction` block.
-However, be aware that some pragmas (including `foreign_keys`) can't be changed inside transactions.
-Still, it can be useful to:
-
-- always re-enable foreign keys before using the database, by enabling them in [`beforeOpen`](#post-migration-callbacks).
-- disable foreign-keys before migrations
-- run migrations inside a transaction
-- make sure your migrations didn't introduce any inconsistencies with `PRAGMA foreign_key_check`.
-
-With all of this combined, a migration callback can look like this:
-
-{{ load_snippet('structured','lib/snippets/migrations/migrations.dart.excerpt.json') }}
-
-## Post-migration callbacks
+## Post-Migration callbacks
 
 The `beforeOpen` parameter in `MigrationStrategy` can be used to populate data after the database has been created.
 It runs after migrations, but before any other query. Note that it will be called whenever the database is opened,
@@ -114,32 +163,3 @@ beforeOpen: (details) async {
 }
 ```
 
-## During development
-
-During development, you might be changing your schema very often and don't want to write migrations for that
-yet. You can just delete your apps' data and reinstall the app - the database will be deleted and all tables
-will be created again. Please note that uninstalling is not enough sometimes - Android might have backed up
-the database file and will re-create it when installing the app again.
-
-You can also delete and re-create all tables every time your app is opened, see [this comment](https:/simolus3/drift/issues/188#issuecomment-542682912)
-on how that can be achieved.
-
-## Verifying a database schema at runtime
-
-Instead (or in addition to) [writing tests](#verifying-a-database-schema-at-runtime) to ensure your migrations work as they should,
-you can use a new API from `drift_dev` 1.5.0 to verify the current schema without any additional setup.
-
-
-
-{{ load_snippet('(full)','lib/snippets/migrations/runtime_verification.dart.excerpt.json') }}
-
-When you use `validateDatabaseSchema`, drift will transparently:
-
-- collect information about your database by reading from `sqlite3_schema`.
-- create a fresh in-memory instance of your database and create a reference schema with `Migrator.createAll()`.
-- compare the two. Ideally, your actual schema at runtime should be identical to the fresh one even though it
- grew through different versions of your app.
-
-When a mismatch is found, an exception with a message explaining exactly where another value was expected will
-be thrown.
-This allows you to find issues with your schema migrations quickly.