Runtime: Refactor reconciliation

[begelundmuller]

Task breakdown:

• Runtime: Refactor drivers and connectors #2729
• Runtime: Implement DuckDB SQL utils #2732
• Runtime: Implement DB size estimation for DuckDB #2733
• Runtime: Implement file watching #2730
• Async reconcile: resource definitions and code file parser #2761
• Async reconcile controller APIs #2782
• File watcher refactors #2813
• Compiler: check refs and unique source/model names #2814
• Refactor connection cache for acquire-like semantics (so it doesn't close connections that are in use) (Driver abstractions improvements #2875)
• Leverage connector declarations in rill.yaml (add a default connector for stage.db) (Driver abstractions improvements #2875)
• Implement source reconciler (Async reconcilers for sources, models, metrics, migrations #2786)
• Implement model reconciler (Async reconcilers for sources, models, metrics, migrations #2786)
• Implement metrics view reconciler (Async reconcilers for sources, models, metrics, migrations #2786)
• Implement migration reconciler (Async reconcilers for sources, models, metrics, migrations #2786)
• Runtime: Misc. connector management changes #2904
• Implement controller event loop Async reconcile event loop #2946
• Implement new catalog format and caching
• Merge Support for setting local credentials in .env file #2889
• Replace FindInstance with cached lookups
• Simplify transporters to accept a uniform config object
• Controller: Before moving from pending to running, clear error
• Clone Get's from a reconciler if mutating the result
• Make calls to UpdateSpec(trigger=false) safe (in case projectparser updates at same time)
• Undelete if creating something deleted. Error if updating something deleted.
• Create probably has issue with State not being set. Should call New on reconciler.
• Lower name unique is enforced in parser. Also needed in catalog, controller, and in ProjectParser
• Rename package dag2 to dag
• Adds registryCache that caches instances in-memory and starts/stops a Controller for each instance
• Integrate into local CLI
• Removes old reconcile
• [P0] Reconcilers: Add refs checks for failed parents
• [P0] Limit concurrency of DuckDB source ingest/etc., so we don't block all connections
• [P0] Increase rill_pool_size on local app Set DuckDB pool size to 4 on local #3045
• [P0] Incorporate DuckDB transactions in new reconcile: Use DuckDB transaction for background source refresh #2897
• [P0] Fix TODOs in code
• Simplify connector source and sink props #3057
• [P0] Move connector-specific source property parsing and backwards compatibility rules into individual connectors/transporters (such as csv.delimiter, etc.) Move source properties backwards compatibility rules to drivers #3072
• [P0] Parsing ExtractPolicy
• [P0] Opening srcConnector with correct variables (connectorVariables) Make AcquireHandle work for any source connector #3076 Move source properties backwards compatibility rules to drivers #3072
• [P0] Rewrite SQL sources for s3/gcs (see mergeFromParsedQuery and Adding sql sources parsing to new reconcile #2915) DuckDB SQL sources for async reconcile #3082
• [P0] Enforce metrics policies in new resource APIs Add access policy checks for new resource API #3083
• [P0] Replace old reconcile with new reconcile Async reconcile integration #3092
• [P0] Use new catalog resource types in analytical APIs Async reconcile integration #3092
• [P0] Weird logs when opening on an uninitialized project (@begelundmuller)
• [P1] Fix renames not working for models (they become deletes)
• [P1] Deduplicate file watcher paths during batching
• [P1] Parser: Prevent distinct files overriding the same model (change upsertResource to insertResource?) (@begelundmuller)
• [P2] Don't use LongRunning for model views
• [P0] Implement stubbed tests in runtime/controller_test.go
• [P0] Integrate new reconcile with admin server (@begelundmuller)

[begelundmuller]

Closed by #3092