Skip to content

feat: save the model and stop training based on exit-duration-in-mins #67

Open

Description

@SaulLu

As discussed a long time ago in a meeting it would be really great if we had a feature to save the model and stop training after a certain time as the jobs on the JZ cluster are limited to 20 hours.

For example, in the architecture and scaling working group, they added the exit-duration-in-mins argument the library used to run trainings Megatron-DeepSpeed


related: #37 (#42)

Activity

  1. shanyas10 commented on Nov 23, 2021

    @shanyas10
    Collaborator

    does #42 serve half of the purpose (saving the model)?

  2. SaulLu commented on Nov 23, 2021

    @SaulLu
    CollaboratorAuthor

    Indeed your PR #42 is also really useful (it should be merged, I send you a private message about this)

    What I have in mind with this issue is more to launch the backup after a certain time as the jobs on JZ are limited to 20h. If I'm not mistaken it's something that is not included in your current PR #42 right?

  3. moved this from Open to In Review in Modeling Metadataon Apr 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions