refacto: fix bunch of warnings - #84
Merged
Merged
Conversation
- Replace direct os module usage with importlib.resources for resource file handling - Use a try-except block to handle invalid configuration names and provide a clear error message - Ensure OmegaConf loads the configurations from the properly resolved path using resources.as_file
…djust worker settings for data modules - Introduce source_prefix and target_prefix fields in Seq2SeqTransformerDataModule configurations to allow for data customization - Set default number of workers to 0 in train_t5.yaml to avoid potential multiprocessing issues or resource contention
…otations - Replace pkg_resources with importlib.resources for accessing package resources to follow modern practices - Utilize pathlib for file path handling to improve readability and consistency - Update the return type annotation of label_dataset to specify integer list union - Change resource access to handle both absolute and packaged resource paths dynamically - Ensure datasets are loaded from the correct resource paths for both local and external configurations
…and config handling - Implemented caching for tokenizer instantiation in Seq2Seq data module to optimize performance - Refactored the dataset tokenization method to support batch processing and prefix addition for source and target texts - Enhanced configuration management by adding prefixes and number of workers to the Seq2SeqTransformerDataModule setup - Updated the dataset formatting to conform to the consistent string quotation style
…and importlib - Replaced `pkg_resources` with `pathlib` and `importlib` for loading teacher model checkpoints to improve path handling - Added support for loading models from both absolute and relative paths within the package resources
… external resources in ModelWrapper - Use the contextlib.ExitStack to manage resources files more reliably in the ModelWrapper - Remove usage of pkg_resources in favor of importlib.resources to adhere to modern standards - Ensure resources are properly released by adding a close method and handling in the destructor [tests] - refactor: remove deprecated pkg_resources usage across test modules - Replace pkg_resources with pathlib.Path to locate files in tests - Clear tokenizer cache in test_transformer_module_multisource to ensure fresh environment for each test run
…sing logic - Introduce methods to normalize dataset splits and ensure required splits exist - Support subsetting datasets by percentage with validation for percentage range - Adjust dataset configuration retrieval to handle missing prefix keys gracefully [bert_squeeze/distillation] - fix: add layer_norm.weight to list of no_decay parameters - Ensure consistency in parameter groups for optimization by including layer_norm weights [bert_squeeze/models] - refactor: update optimizer parameter handling for discriminative learning - Modify no_decay lists to include layer_norm weights across model modules - Introduce _IdentityParamList to ensure parameter identity comparison in optimizer groups
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Replaces deprecated
pkg_resourceswithimportlib.resourcesandpathlibacross the codebase to follow modern Python practices. The change addresses deprecation warnings and improves resource loading reliability by using context managers (ExitStack,as_file) to properly manage resource files.Risk
Low. All existing functionality is preserved with equivalent modern APIs.
Tests
Updated tests to remove
pkg_resourcesusage and added tokenizer cache clearing intest_transformer_module_multisourceto ensure clean test environments.