-
Notifications
You must be signed in to change notification settings - Fork 3
Parallelize history migration with configurable exponential backoff per entity type #397
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
@copilot, make the following hard coded values configurable: |
... Done! The values are now configurable via
Commit: dcc9e88 Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
dcc9e88 to
594b228
Compare
|
@copilot, fix failing tests. |
Fixed! The issues were:
These changes ensure proper exception handling and logging in parallel execution, and prevent NullPointerExceptions in test contexts where properties might not be fully initialized. Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
- Add null-safe getters for parallel migration properties - Make parallel migration parameters configurable via properties - Implement parallel migration with exponential backoff
1a78b22 to
8841318
Compare
Description
History migration now runs each entity type (process definitions, instances, flow nodes, tasks, variables, incidents, decisions) in parallel threads. Each thread polls for new work, backs off exponentially when idle, and exits after consecutive empty iterations. MIGRATE mode runs to completion before RETRY_SKIPPED begins.
Changes:
migrate()now callsrunParallelMigration()sequentially for MIGRATE then RETRY_SKIPPED modesrunParallelMigration()spawns 9 threads viaExecutorService, one per entity typecreateMigrationTask()wraps migration methods with work detection (timestamp comparison) and exponential backoff logicCountDownLatch, proper cleanup in finally blocksmax-suspensions,initial-backoff-ms, andbackoff-multipliertoMigratorPropertiesfor flexible tuningExceptionContext.HISTORYfor correct exception handling and loggingImplementation:
Work detection compares
findLatestCreateTimeByType()before/after each migration call. Changed timestamp = work done, reset counter. Unchanged = increment suspension counter, apply backoff.Thread safety verified: Spring-managed singletons (DbClient, C7Client), stateless MyBatis mappers, stateless converters. Each worker thread manages its own ThreadLocal exception context.
Configuration:
The following properties can be configured in
application.ymlorapplication.properties:Or in properties format:
Related issues
closes #
Original prompt
✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.