Training checklist

[justheuristic]

Start some peers

• @razaidy starts initial CPU peer and shares her Peer ID (/ip4/something/tcp/something)
• @justheuristic starts another CPU peer for fault-tolerance and shares his Peer ID
• Start a few GPU workers

Technical features

• Scale sequence length dynamically during training [@justheuristic]
  • add 32 every 1000 global steps up to 512
• training with TPU

Volunteer starter kit

• draft notebook
• create an invite link for volunteers
• text explanation for running in colab
• upload notebook to CALM/notebooks?
• discord chatroom
• organization page
• instructions for training locally, with kaggle, with sagemaker
• email to volunteers (early ~wednesday )

After we start

• contributors dashboard [@SaulLu ]
• running evaluation every few hours
• set up monitoring & support shifts

Sanity checks

• make sure training data looks right (@razaidy @JAWHARAH123 @pr-Mais )
• make sure loss is in a reasonable range
• verify that volunteer starter kits work
• make sure training with TPU does not leak memory during host<->device transfer (@justheuristic )
• look at the data again just in case

Milestones

• pass loss 9 by ~1000 steps (without stagnating)
  
• do not blow up at peak learning rate (steps 2500-3500)
• downstream should be better than random after step 4000
• reach full sequence length (10000)

[pr-Mais]

I see sagemaker, could GCE w or w/o colab be used as well?

[justheuristic]

Anything can as long as it has a free tier :)
Sagemaker had a free studiolab: https://studiolab.sagemaker.aws/

[SaulLu]

Some news about the dashboard 😃

Here is what I did for the moment:

1. Create a streamlit space named Dashboard in the CALM organization
2. Copy the dashboard repo of training-transformers-together into the new github repo for the CALM dashboard
3. Change the target HF repository in the github workflow (here)
4. Create a machine-user for the organization and add it to the organization with a WRITE role (we need its token) - how should I share with you its password?

To finish setting up this dashboard, we should :
6. Add a secret named HF_TOKEN in the dashboard repository on github corresponding to a write access token of the machine user (I don't have the necessary rights on github)
7. Add two secrets WANDB_REPO_INDIVIDUAL_METRICS and WANDB_RUN_URL_MAIN_METRICS in the dashboard repository on the HUB corresponding to the links to the WANDBs storing the data.

[justheuristic]

Awesome work!

For step #6
@razaidy , you mentioned there is someone on your side who can admin the organization, right?

@SaulLu can they use this instruction for adding a secret?

For step #7, I took the liberty to add these two secrets, but I'm not entirely sure I got the format right. Since they are public knowledge, I'll copy them here as well:

• WANDB_REPO_INDIVIDUAL_METRICS=https://wandb.ai/calm/CALM-hivemind-trainers
• WANDB_RUN_URL_MAIN_METRICS=https://wandb.ai/calm/CALM

p.s. this seems way more elegant than what we did with Neuropark (i.e. hardcode access tokens), thanks!

[JAWHARAH123]

Thank you SaulLu for your effort and contribution. Can you please provide me with the write access token of the machine user? so I can add the secret to the repository.

…

On Wed, 29 Dec 2021 at 13:38, SaulLu ***@***.***> wrote: Some news about the dashboard 😃 Here is what I did for the moment: 1. Create a streamlit space named Dashboard <https://huggingface.co/spaces/CALM/Dashboard> in the CALM organization 2. Copy the dashboard repo of training-transformers-together <https:/training-transformers-together/dashboard> into the new github repo for the CALM dashboard <https:/NCAI-Research/dashboard> 3. Change the target HF repository in the github workflow (here <https:/NCAI-Research/dashboard/blob/main/.github/workflows/sync_to_hub.yaml#L20> ) 4. Create a machine-user <https://huggingface.co/ncai-calm-machine-user> for the organization and add it to the organization with a WRITE role (we need its token) - how should I share with you its password? To finish setting up this dashboard, we should : 6. Add a secret named HF_TOKEN in the dashboard repository on github <https:/NCAI-Research/dashboard> corresponding to a write access token of the machine user <https://huggingface.co/ncai-calm-machine-user> (I don't have the necessary rights on github) 7. Add two secrets WANDB_REPO_INDIVIDUAL_METRICS and WANDB_RUN_URL_MAIN_METRICS in the dashboard repository on the HUB <https://huggingface.co/spaces/CALM/Dashboard> corresponding to the links to the WANDBs storing the data. — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https:/notifications/unsubscribe-auth/AN7FTRYFZMWVBBHTK5IK66TUTLQLZANCNFSM5KQNMYGQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[SaulLu]

@JAWHARAH123 , with pleasure to share with you this secret information (I'll share the password to this machine user and the user access token)! I think that I can send you a private message on github, How do you want me to reach you? (email? discord?) 😄

The procedure to add the secret to the repository is indeed exactly the one you shared @justheuristic !

Thanks for the information regarding the step 7 @justheuristic 🤗

[JAWHARAH123]

Whatever you like ,you can find me in the CALM discord channel, or you can send me an email ***@***.***) في سبت، ١ يناير، ٢٠٢٢ في ٦:١١ م، كتب SaulLu ***@***.***>:

…

@JAWHARAH123 <https:/JAWHARAH123> , with pleasure to share with you this secret information (I'll share the password to this machine user and the user access token)! I think that I can send you a private message on github, How do you want me to reach you? (email? discord?) 😄 The procedure to add the secret to the repository is indeed exactly the one you shared @justheuristic <https:/justheuristic> ! Thanks for the information regarding the step 7 @justheuristic <https:/justheuristic> 🤗 — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https:/notifications/unsubscribe-auth/AN7FTR66QR7QSHFY7RN7NNDUT4KRXANCNFSM5KQNMYGQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[SaulLu]

Thank you very much for your answer, unfortunately I can't see your email address in your last message and there are more users with the same nickname as you on discord.

If it ever helps, on discord I am user SaulLu #0201.

[SaulLu]

Thank you all! The dashboard is live at this address: https://hf.co/spaces/CALM/Dashboard