-
Notifications
You must be signed in to change notification settings - Fork 404
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New integration - CometLogger #910
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/910
Note: Links to docs will display an error until the docs builds have been completed. This comment was automatically generated by Dr. CI and updates every 15 minutes. |
Hi @Lothiraldan! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at [email protected]. Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Lothiraldan thanks so much for the PR! Really excited about new community integrations. Do you mind sharing more context on why this logger is needed and what problem it solves?
Highlighting ecosystem integrations is one of the core principles of torchtune and something we believe makes the ecosystem stronger.
We started with integrations for WandB and Tensorboard since these have come up as feature requests in the user conversations we had. Generally, we'd like to get more user signal into different integrations before we move them to the core repo.
Our general mode of operation is:
- Land integration in a fork
- Highlight the integration in the new community integration section we're adding to the README (PR should be up in a day or 2)
- As more folks look at this and we get requests to merge this into main, we can pull in this change
Let me know if this makes sense.
Hi, Jumping in since @Lothiraldan is on PTO! My name is Sid and I'm a ML Growth Engineer at Comet. We opened the PR since we had two enterprise customers asking for it. They’re currently using Transformers and looking to switch to torchtune. We’re happy to connect them directly to the torchtune team if that helps. There’s >100k researchers using Comet on cloud and given how disruptive TorchTune is we expect a lot of them will want to switch but will be blocked. The rest of the core training frameworks usually define a logging Abstract Base Class and we create/maintain a Comet Subclass for logging metrics and assets to Comet. It’s worked well in the Pytorch Lightning, Transformers, Composer libraries: all which have multiple logging options for users. We could create a fork and ask our Comet users to go there but i think it adds another layer of complexity for our mutual users and customers. Would really appreciate if you can consider a similar approach to other frameworks and happy to help on the implementation side. |
Hi @kartikayk, sorry I didn't introduce myself. I'm Boris Integration Product Manager at Comet. We started working on the Torchtune integrations after getting questions about it from our customers. I was able to catch up with my team on your suggestion. I am not a big fan of landing the integration inside a fork as it would require users to install transformers in a different way to use the integration. My team suggested that we package the Torchtune integration inside our Python SDK at first. Users would then be able to use the integration like this: metric_logger:
_component_: comet_ml.integration.torchtune.CometLogger
# the Comet project to log to
project: comet-examples-torchtune This would have two advantages when we want to move the integration to the Torchtune repository:
Please let me know what you think about this technical solution. Having a dedicated section for integrations in the Readme makes sense. Here is I think a good example of how another library highlights its integrations: https:/ultralytics/ultralytics?tab=readme-ov-file#integrations (the picture is big but helps to understand where each integration kicks in. It can probably be added later when you have more integrations). I'm not 100% sure that Torchtune users would understand the difference between integrations that live inside the Torchtune repository vs in an external repository. For example, we document all of our integrations the same no matter where the code lives: https://www.comet.com/docs/v2/integrations/overview/. |
Hi @Lothiraldan, looks like we're seeing more user interest in CometLogger here #1221. Wondering if you're still open to landing this? if not, we may incorporate some of your additions here in the other PR. Let me know your thoughts. |
Hi @RdoubleA, yes I'm still looking into landing this pull request. I took a quick look at the other PR and it seems roughly equivalent. Happy to help merging either one of the pull request. |
Let's converge on the other PR then. Do you mind reviewing it and working with @dzheng256 to get anything that is missing added (such as your excellent tutorial)? Really appreciate your work on this |
Thanks for this PR @Lothiraldan, again apologies for basically recreating it, I had written an internal CometLogger we were using with torchtune already so I turned it into a PR but should have double checked for open PRs first. In any case, hopefully in my PR we'll incorporate all the relevant changes so others can easily get started with Comet and torchtune. |
@dzheng256 No worries at all, it's great that we had the same idea around the same time. I will incorporate few changes in your PR, it already looks pretty good! |
Context
This PR introduces support for logging metrics to Comet https://www.comet.com/site/.
Changelog
torchtune.utils.metric_logging.CometLogger
Test plan
pre-commit install
)pytest tests
pytest tests -m integration_test