Hugging Face's logo

google
/
gemma-3-4b-it

Image-Text-to-Text
Transformers
Safetensors
Model card Files
xet
 Community
89

Token Count Calculation in SFT Data Distribution Curation

#76
by tcy006 - opened
Discussion
tcy006

Regarding the curation of SFT data, including the curation of the data distribution, I would like to understand how you calculate the token count for each data entry when designing the distribution. Is the token count based only on the user tokens, or does it also include the assistant tokens? (The reason I ask is that I understand the SFT loss is calculated only based on the assistant tokens.)
BalakrishnaCh
 Google org

Hi @tcy006 ,

The token count isn't one number. Both counts are calculated and used. When considering training cost and efficiency, token count usually refers to the total tokens. When considering task balancing and model behavior, it refers to the assistant tokens.

  1. Assistant-Only Tokens: This metric measures the length of the assistant's response (len(assistant_response)). Its primary purpose is for learning & task balancing answering the question.
  2. Total Tokens: This metric measures the combined length of the user prompt and the assistant's response (len(user_prompt) + len(assistant_response)). Its primary purpose is for managing infrastructure & cost answering the question.

Thanks.
tcy006

Hi @tcy006 ,

The token count isn't one number. Both counts are calculated and used. When considering training cost and efficiency, token count usually refers to the total tokens. When considering task balancing and model behavior, it refers to the assistant tokens.

  1. Assistant-Only Tokens: This metric measures the length of the assistant's response (len(assistant_response)). Its primary purpose is for learning & task balancing answering the question.
  2. Total Tokens: This metric measures the combined length of the user prompt and the assistant's response (len(user_prompt) + len(assistant_response)). Its primary purpose is for managing infrastructure & cost answering the question.

Thanks.

Hi @BalakrishnaCh ,

Thank you for the clear explanation — it really helps clarify how both token counts play different roles in SFT data curation.
I appreciate you breaking down the distinction between assistant-only tokens and total tokens, and how each is used for task balancing versus training cost considerations. This makes a lot of sense.

Thanks again for the detailed and helpful response!

Sign up or log in to comment