[utils] - feat: experiment tracker - #81
Merged
Merged
Conversation
…lize ExperimentLogger - Revised several modules to use the new ExperimentLogger for consistent logging - Improved condition checks by using isinstance() for checking multiple types simultaneously - Enhanced configuration handling in assistant constructors to better manage overrides [bert_squeeze] - feature: introduce optional Aim support for enhanced logging - Integrated Aim logger support as an optional feature that can be enabled during assistant setup - Added convenience method to easily add AimLogger to an assistant configuration [tests] - test: add tests for custom logger configuration in assistants - Created new tests to ensure custom logger configuration is handled correctly in TrainAssistant and DistilAssistant - Verified that the configured logger is properly set up and the save directory is respected [docs] - docs: document how to configure custom loggers in assistants - Updated documentation to describe the process of setting up custom loggers for assistants - Provided an example on using AimLogger with an assistant configuration [README.md] - docs: add instructions for installing Aim support - Updated the README to guide users on installing Aim support and configuring AimLogger in an assistant
- Replaced generic `Any` type hint with more specific types to ensure strict type checking - Updated type hints to `Optional` and `Union` where appropriate for better code clarity and safety - Added new `Typing` guidelines in AGENTS.md, prohibiting the use of type `Any` [AGENTS.md] - docs: add typing guideline to avoid `Any` type - Introduced a guideline to use strict types instead of type `Any` in the Typing section
…utility - Add `ExperimentLogger` class to handle logging of text and figures across different backends such as TensorBoard and Aim - Introduce helper functions to log text and figures to TensorBoard and Aim with the Resolve step and epoch functionality [tests] - test: add tests for new experiment logging utility - Test the logging of text and figures for TensorBoard through `_DummyTensorboardExperiment` mocks - Test the integration with Aim's `Run.track` feature using mocked Aim classes and methods for text and figure data
… functions flexibility - Allow the `lightning_logger` parameter to be optional across various logging functions - Improve compatibility with different versions of Aim by using a function to generate kwargs for Aim `track` dynamically
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds multi-backend experiment logging abstraction supporting TensorBoard and Aim. Introduces
ExperimentLoggerclass that wraps Lightning loggers and provides a unified API for logging text and figures, automatically detecting the underlying logger type. All modules migrated from directlogger.experimentaccess toExperimentLogger.from_module(). Also enforces stricter typing across the codebase (replacingAnywithOptional/Union) and adds typing guidelines.Risk
Medium - changes logging throughout the codebase. Backward-compatible with TensorBoard (default); Aim support is optional via
aimextra.Tests
Unit tests for
ExperimentLoggercovering TensorBoard and Aim backends, custom logger configuration tests for assistants, mocked Aim integration tests.