294718: Enhance Batch model and admin functionality#320
Open
294718: Enhance Batch model and admin functionality#320
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #320 +/- ##
===========================================
+ Coverage 96.46% 96.48% +0.02%
===========================================
Files 217 217
Lines 8221 8269 +48
Branches 833 839 +6
===========================================
+ Hits 7930 7978 +48
Misses 160 160
Partials 131 131 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
saxix
requested changes
Jan 30, 2026
Member
saxix
left a comment
saxix
left a comment
There was a problem hiding this comment.
It is not clear where the transaction management happen. Can we add some comment to details the process, this can be useful for future updates.
Linked to the transaction management, I saw test_import_data_reschedules_when_incomplete and test_import_asset_timeboxed_returns_incomplete_and_keeps_watermark, to test recovery I would add a test the check for errors but we also check previous data are persisted to the disk
arsen-vs
requested review from
datamik,
saxix,
sergey-misuk-valor and
vitali-yanushchyk-valor
saxix
approved these changes
Feb 4, 2026
Collaborator
|
@arsen-vs pls resolve conflicts |
added 8 commits
February 5, 2026 00:14
- Introduced `status` and `kobo_last_page` fields to the Batch model to track the import status and pagination. - Updated BatchAdmin to display the new fields and filter by batch status. - Modified import processing functions to set batch status during data import and handle loading states. - Added migration to create new fields and set existing batches to complete. - Enhanced tests to validate new functionality and ensure correct behavior during batch processing. These changes improve the tracking and management of batch imports, enhancing overall functionality and user experience.
- Added a test for `import_asset` to validate timeboxed updates and ensure correct handling of submissions, including the update of `kobo_last_page` and logging of last successful submission ID. - Introduced a new test for `import_data` to verify rescheduling behavior when the import is incomplete, ensuring proper job creation and validation of asset import calls. - Updated existing tests to include assertions for new functionality and improved coverage of the import process. These changes enhance the robustness of the Kobo sync functionality and ensure accurate tracking of import states.
- Removed `kobo_last_page` from `BatchAdmin` and `Batch` model to streamline the import process and focus on essential fields. - Updated `import_asset` and related functions to eliminate unnecessary parameters and improve clarity in submission handling. - Adjusted tests to reflect changes in the `submissions` method, ensuring accurate assertions without the `start_page` parameter. These modifications enhance the maintainability of the code and improve the overall efficiency of the import process.
- Adjusted assertions in `test_create_individuals` to check for keys from the processed mock data. - Removed references to `kobo_last_page` in `test_import_data` and `test_import_data_reschedules_when_incomplete` to align with recent refactoring of the Batch model. - These changes ensure that tests accurately reflect the current state of the codebase and improve clarity in the import process.
- Added a status assertion for the `test_import_from_rdi` function to reflect the loading state during import. - Updated the `test_household_filter_queryset_with_program` to include a filter for batch status, ensuring accurate filtering based on the new Batch model structure. These changes enhance the accuracy of the tests by aligning them with the current Batch model's functionality and improving the validation of import processes.
…lity - Introduced a new test class for `BatchAdmin.beneficiaries` to validate behavior when the batch status is LOADING, ensuring that households and individuals return empty querysets. - Added a test for `import_data` to confirm that it resumes an existing batch when `job.batch_id` is set, preventing the creation of a new batch and ensuring correct handling of import processes. These enhancements improve test coverage and ensure accurate validation of batch processing and import functionality.
…rtions - Replaced `RequestFactory` with `Client` for testing the `beneficiaries` view, improving the simulation of user interactions. - Updated assertions in `test_import_data_resumes_existing_batch` to ensure the call count for `select_for_update` is validated correctly. These changes enhance the reliability of the tests and ensure they accurately reflect the intended functionality of the BatchAdmin and import processes.
- Updated `import_asset` to ensure each submission is processed in its own transaction, allowing for partial success and rollback of only failed submissions. - Added detailed comments to clarify transaction handling and watermark persistence for recovery. - Introduced a new test to verify that earlier submissions remain intact when a later submission fails, ensuring data integrity during import processes. These changes improve the robustness of the import functionality and enhance error handling in the Kobo sync process.
arsen-vs
force-pushed
the
feature/294718-Kobo-import-refactor
branch
from
f29a687 to
38bbd79
Compare
- Updated the `test_base_import_form_includes_transformers_when_program_provided` to create transformers associated with a specific beneficiary group, enhancing the test setup. - Revised the docstring in `test_base_import_form_pops_individual_mapping_when_no_individual_checker` for better clarity regarding the behavior of household fields in people-only programs. These changes improve the readability and reliability of the tests related to import forms and transformers.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
statusandkobo_last_pagefields to the Batch model to track the import status and pagination.These changes improve the tracking and management of batch imports, enhancing overall functionality and user experience.
AB#294718