Add 1pp-0.5b-asst-sft (1PP)

README.md

@@ -0,0 +1,61 @@
+---
+license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- 1pp
+- one-persona-pretraining
+- sft
+- asst
+---
+
+# 1pp-0.5b-asst-sft
+
+**One Persona Pretraining (1PP)** experiment model: 0.58B parameters, pretraining condition **rewritten conversations, assistant-turn loss**, followed by supervised fine-tuning (SFT).
+
+Part of a 3 × 3 study: three sizes (0.5B, 1B, 1.7B) × three pretraining conditions on the same 47.8M source documents in the same order (original documents; rewritten conversations with loss on assistant turns; rewritten conversations with loss on user and assistant turns). Every run saw the identical batch sequence, so the conditions differ only in the document text and the loss mask. Models are grouped in the [1pp collection](https://huggingface.co/collections/Raghav-Singhal/1pp-6a999df54bfcf9335355a649).
+
+## Architecture
+
+Llama-style decoder, 24 layers, hidden 1,152, FFN 4,608 (SwiGLU), attention heads / KV heads 9 / 3 (head dim 128), RMSNorm, RoPE base 10,000, untied embeddings, no biases, no QK-norm, sequence length 4,096. Tokenizer: SmolLM2 vocabulary (49,152) plus `<|pad|>`; `<|endoftext|>` is the end-of-document token.
+
+## Pretraining
+
+Data: the 1PP conversations rewritten from those documents; loss only on assistant turns (no loss on user turns or `<|endoftext|>`). One pass over 47.8M documents (66.2B tokens of original documents; 63.0B tokens as conversations), 31,777 steps at global batch 512 × 4,096 tokens, cross-document attention masking, best-fit packing with step-aligned document assignment. Optimizer: Muon (shape scaling, matrix LR 0.005) with Adam for embeddings and norms, warmup 2,000 steps, constant, linear decay over the last 10% to 1/100, weight decay 0.1, bf16.
+
+Validation loss (per token, 2,433 held-out documents, final checkpoint):
+
+| assistant text | user text | document text |
+|---|---|---|
+| 1.579 | 6.878 | 3.372 |
+
+## Supervised fine-tuning
+
+One epoch over a 400k-conversation mix: `jkminder/model-raising-pb-100k-3c-mt-sft` (98.5k multi-turn, constitution-cited track), `dlab-spp/sp-sft-normal-300k` minus prompts duplicated in the first set (271.6k), and a 30k sample of `dlab-spp/sp-sft-safety-180k`. Same stack as pretraining (Megatron, Muon, ChatML without a system turn, loss on assistant turns only). Matrix LR 0.002 selected per model from {0.0005, 0.001, 0.002, 0.005} by held-out loss; global batch 128 × 4,096, linear decay to 1/10 after 3% warmup.
+
+Held-out SFT loss (assistant tokens, 1,998 held-out conversations): 2.023
+
+## Chat format
+
+ChatML **without a system turn** (the models never saw one):
+
+```
+<|im_start|>user\n{message}<|im_end|>\n<|im_start|>assistant\n{reply}<|im_end|>\n
+```
+
+The bundled `chat_template` renders exactly this. Generation stops at `<|im_end|>`.
+
+## Verification
+
+The HF weights were checked against the Megatron checkpoint by recomputing validation losses with this model:
+
+| set | HF loss | Megatron reference | abs. diff |
+|---|---|---|---|
+| sft_val segments [3, 4] | 2.0234 | 2.0234 | 0.0000 |
+
+## Links
+
+- Training logs: wandb projects [1pp-training](https://wandb.ai/raghav_singhal/1pp-training) and [1pp-sft](https://wandb.ai/raghav_singhal/1pp-sft)
+- Research artifact from the 1PP project (EPFL DLAB); not a general-purpose assistant.

added_tokens.json

@@ -0,0 +1,3 @@
+{
+  "<|pad|>": 49152
+}

chat_template.jinja

@@ -0,0 +1,4 @@
+{% for message in messages %}{{ '<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+' }}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}

config.json

@@ -0,0 +1,33 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "dtype": "bfloat16",
+  "eos_token_id": [
+    2,
+    0
+  ],
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 1152,
+  "initializer_range": 0.02,
+  "intermediate_size": 4608,
+  "max_position_embeddings": 4096,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 9,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 3,
+  "pad_token_id": 49152,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000,
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.6",
+  "use_cache": true,
+  "vocab_size": 49153
+}

convert.log

@@ -0,0 +1,1020 @@
+
+WARNING: FI_CXI_RDZV_PROTO=alt_read is configured, but the Slurm network is
+not set to disable rendezvous GET. Performance/stability may be impacted.
+
+Expected Slurm launch:
+  srun --network=disable_rdzv_get ...
+or
+  SLURM_NETWORK=disable_rdzv_get srun ...
+
+Context:
+  SLURM_NETWORK=<unset>
+  FI_PROVIDER=cxi
+  FI_CXI_RDZV_PROTO=alt_read
+
+  Overriding a previously registered kernel for the same operator and the same dispatch key
+  operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
+    registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922
+  dispatch key: ADInplaceOrView
+  previous kernel: no debug info
+       new kernel: registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
+  self.m.impl(
+Loaded loader_core as the loader.
+Loaded saver_llama_hf as the saver.
+Starting saver...
+  Overriding a previously registered kernel for the same operator and the same dispatch key
+  operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
+    registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922
+  dispatch key: ADInplaceOrView
+  previous kernel: no debug info
+       new kernel: registered at /usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py:922 (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
+  self.m.impl(
+Starting loader...
+Setting num_layers to 24 from checkpoint
+Setting hidden_size to 1152 from checkpoint
+Setting ffn_hidden_size to 4608 from checkpoint
+Setting seq_length to 4096 from checkpoint
+Setting num_attention_heads to 9 from checkpoint
+Setting num_query_groups to 3 from checkpoint
+Setting group_query_attention to True from checkpoint
+Setting kv_channels to 128 from checkpoint
+Setting max_position_embeddings to 4096 from checkpoint
+Setting position_embedding_type to rope from checkpoint
+Setting add_position_embedding to True from checkpoint
+Setting use_rotary_position_embeddings to False from checkpoint
+Setting rotary_base to 10000 from checkpoint
+Setting rotary_percent to 1.0 from checkpoint
+Setting rotary_interleaved to False from checkpoint
+Setting add_bias_linear to False from checkpoint
+Setting add_qkv_bias to False from checkpoint
+Setting squared_relu to False from checkpoint
+Setting swiglu to True from checkpoint
+Setting ssglu to False from checkpoint
+Setting reglu to False from checkpoint
+Setting rlglu to False from checkpoint
+Setting sssglu to False from checkpoint
+Setting lglu to False from checkpoint
+Setting situ to False from checkpoint
+Setting pnglu to False from checkpoint
+Setting pnglu_fusion to True from checkpoint
+Setting untie_embeddings_and_output_weights to True from checkpoint
+Setting scale_embeddings_by_sqrt_hidden to False from checkpoint
+Checkpoint did not provide arguments apply_layernorm_1p
+Setting normalization to RMSNorm from checkpoint
+Setting residual_output_scaling to False from checkpoint
+Setting sandwich_norm to False from checkpoint
+Setting keel to False from checkpoint
+Checkpoint did not provide arguments keel_alpha
+Setting apply_query_key_layer_scaling to False from checkpoint
+Setting qk_layernorm to False from checkpoint
+Setting attention_dropout to 0.0 from checkpoint
+Setting hidden_dropout to 0.0 from checkpoint
+Checkpoint did not provide arguments window_size
+Checkpoint did not provide arguments window_attn_skip_freq
+Checkpoint did not provide arguments no_rope_freq
+Checkpoint did not provide arguments mtp_hybrid_override_pattern
+Checkpoint did not provide arguments mtp_num_layers
+Setting mtp_use_repeated_layer to False from checkpoint
+Checkpoint did not provide arguments spec
+Checkpoint did not provide arguments num_experts
+Checkpoint did not provide arguments mtp_num_layers
+Setting moe_layer_freq to 1 from checkpoint
+Setting moe_router_topk to 2 from checkpoint
+Setting moe_router_pre_softmax to False from checkpoint
+Setting moe_grouped_gemm to False from checkpoint
+Checkpoint did not provide arguments moe_shared_expert_intermediate_size
+Setting moe_router_score_function to softmax from checkpoint
+Setting moe_router_enable_expert_bias to False from checkpoint
+Checkpoint did not provide arguments moe_router_topk_scaling_factor
+Setting moe_router_load_balancing_type to aux_loss from checkpoint
+Setting moe_router_quantile_balancing_method to histogram from checkpoint
+Setting multi_latent_attention to False from checkpoint
+Checkpoint did not provide arguments q_lora_rank
+Setting kv_lora_rank to 32 from checkpoint
+Setting qk_head_dim to 128 from checkpoint
+Setting qk_pos_emb_head_dim to 64 from checkpoint
+Setting v_head_dim to 128 from checkpoint
+Setting rotary_scaling_factor to 1.0 from checkpoint
+Setting mamba_state_dim to 128 from checkpoint
+Setting mamba_head_dim to 64 from checkpoint
+Setting mamba_num_groups to 8 from checkpoint
+Checkpoint did not provide arguments mamba_num_heads
+Checkpoint did not provide arguments hybrid_layer_pattern
+Checkpoint did not provide arguments heterogeneous_layers_config_path
+Checkpoint did not provide arguments heterogeneous_layers_config_encoded_json
+Checkpoint did not provide arguments moe_latent_size
+Setting scale_embeddings_by_sqrt_hidden to False from checkpoint
+Setting residual_output_scaling to False from checkpoint
+Setting sandwich_norm to False from checkpoint
+Setting keel to False from checkpoint
+Checkpoint did not provide arguments keel_alpha
+Setting pnglu to False from checkpoint
+Setting pnglu_fusion to True from checkpoint
+Setting pn3glu to False from checkpoint
+Setting xpr to False from checkpoint
+Setting gxpr to False from checkpoint
+Setting gxpry to False from checkpoint
+Setting gxprv2 to False from checkpoint
+Setting xr2 to False from checkpoint
+Setting gxr2 to False from checkpoint
+Setting xr2glu to False from checkpoint
+Setting xssglu to False from checkpoint
+Setting polynorm to False from checkpoint
+Setting qk_layernorm to False from checkpoint
+Setting attention_output_gate to False from checkpoint
+Setting tokenizer_model to /capstor/store/cscs/swissai/infra01/users/rsinghal/1pp-training/tokenizers/smollm2_pretrain from checkpoint
+Setting tokenizer_type to HuggingFaceTokenizer from checkpoint
+Checkpoint did not provide arguments tiktoken_pattern
+Setting padded_vocab_size to 49280 from checkpoint
+Setting tensor_model_parallel_size to 1 from checkpoint
+Setting pipeline_model_parallel_size to 1 from checkpoint
+Checkpoint did not provide arguments virtual_pipeline_model_parallel_size
+Checkpoint did not provide arguments num_layers_per_virtual_pipeline_stage
+Setting expert_model_parallel_size to 1 from checkpoint
+using world size: 1, data-parallel size: 1, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1
+setting global batch size to 1
+Number of virtual stages per pipeline stage: None
+accumulate and all-reduce gradients in fp32 for bfloat16 data type.
+using torch.bfloat16 for parameters ...
+------------------------ arguments ------------------------
+  account_for_embedding_in_pipeline_split ......... False
+  account_for_loss_in_pipeline_split .............. False
+  accumulate_allreduce_grads_in_fp32 .............. True
+  activation_func_clamp_value ..................... None
+  activation_func_fp8_input_store ................. False
+  adam_beta1 ...................................... 0.9
+  adam_beta2 ...................................... 0.999
+  adam_eps ........................................ 1e-08
+  add_bias_linear ................................. False
+  add_position_embedding .......................... True
+  add_qkv_bias .................................... False
+  adlr_autoresume ................................. False
+  adlr_autoresume_interval ........................ 1000
+  align_grad_reduce ............................... True
+  align_param_gather .............................. False
+  allow_ambiguous_pad_tokens ...................... False
+  app_tag_run_name ................................ None
+  app_tag_run_version ............................. 0.0.0
+  apply_query_key_layer_scaling ................... False
+  apply_residual_connection_post_layernorm ........ False
+  apply_rope_fusion ............................... True
+  apply_wd_to_qk_layernorm ........................ False
+  async_ckpt_cpu_priority ......................... 10
+  async_ckpt_io_priority .......................... 3
+  async_save ...................................... False
+  async_strategy .................................. nvrx
+  attention_backend ............................... AttnBackend.auto
+  attention_dropout ............................... 0.0
+  attention_output_gate ........................... False
+  attention_softmax_in_fp32 ....................... False
+  auto_detect_ckpt_format ......................... False
+  barrier_with_L1_time ............................ True
+  batch_invariant_mode ............................ False
+  bert_binary_head ................................ True
+  bert_embedder_type .............................. megatron
+  bert_load ....................................... None
+  bf16 ............................................ True
+  bias_dropout_fusion ............................. False
+  bias_gelu_fusion ................................ False
+  bias_swiglu_fusion .............................. True
+  biencoder_projection_dim ........................ 0
+  biencoder_shared_query_context_model ............ False
+  block_data_path ................................. None
+  cache_mla_latents ............................... False
+  calc_ft_timeouts ................................ False
+  calculate_per_token_loss ........................ False
+  check_for_large_grads ........................... False
+  check_for_nan_in_loss_and_grad .................. True
+  check_for_spiky_loss ............................ False
+  check_grad_norm ................................. False
+  check_grad_norm_threshold ....................... 5.0
+  check_weight_hash_across_dp_replicas_interval ... None
+  ckpt_assume_constant_structure .................. False
+  ckpt_convert_format ............................. None
+  ckpt_convert_save ............................... None
+  ckpt_convert_update_legacy_dist_opt_format ...... False
+  ckpt_format ..................................... torch_dist
+  ckpt_fully_parallel_load ........................ False
+  ckpt_fully_parallel_load_exchange_algo .......... broadcast
+  ckpt_fully_parallel_load_process_group .......... dp
+  ckpt_fully_parallel_save ........................ True
+  ckpt_fully_parallel_save_deprecated ............. False
+  ckpt_fully_parallel_save_process_group .......... dp
+  ckpt_step ....................................... None
+  classes_fraction ................................ 1.0
+  clip_grad ....................................... 1.0
+  clone_scatter_output_in_embedding ............... True
+  config_logger_dir ............................... 
+  consumed_train_samples .......................... 0
+  consumed_valid_samples .......................... 0
+  context_parallel_size ........................... 1
+  cp_comm_type .................................... ['p2p']
+  cpu_offloading_num_layers ....................... 0
+  cpu_offloading_retain_pinned_cpu_buffers ........ False
+  create_all_gather_group ......................... False
+  create_attention_mask_in_dataloader ............. True
+  cross_entropy_fusion_impl ....................... native
+  cross_entropy_loss_fusion ....................... False
+  cuda_graph_impl ................................. none
+  cuda_graph_scope ................................ []
+  cuda_graph_warmup_steps ......................... 3
+  data_args_path .................................. None
+  data_cache_path ................................. None
+  data_parallel_random_init ....................... False
+  data_parallel_sharding_strategy ................. no_shard
+  data_parallel_size .............................. 1
+  data_path ....................................... None
+  data_per_class_fraction ......................... 1.0
+  data_sharding ................................... True
+  dataloader_defer_npy_index_mmap ................. False
+  dataloader_fast_cache_load ...................... False
+  dataloader_inter_document_masking ............... False
+  dataloader_type ................................. single
+  ddp_average_in_collective ....................... False
+  ddp_bucket_size ................................. None
+  ddp_num_buckets ................................. None
+  ddp_pad_buckets_for_high_nccl_busbw ............. False
+  ddp_param_name_patterns_for_fp32_local_accumulation  []
+  ddp_reduce_scatter_with_fp32_accumulation ....... False
+  decode_only_cuda_graphs ......................... False
+  decoder_first_pipeline_num_layers ............... None
+  decoder_last_pipeline_num_layers ................ None
+  decoder_num_layers .............................. None
+  decoder_seq_length .............................. None
+  decoupled_lr .................................... None
+  decoupled_min_lr ................................ None
+  decrease_batch_size_if_needed ................... False
+  defer_embedding_wgrad_compute ................... False
+  delay_wgrad_compute ............................. False
+  deprecated_use_mcore_models ..................... False
+  deterministic_mode .............................. False
+  dino_bottleneck_size ............................ 256
+  dino_freeze_last_layer .......................... 1
+  dino_head_hidden_size ........................... 2048
+  dino_local_crops_number ......................... 10
+  dino_local_img_size ............................. 96
+  dino_norm_last_layer ............................ False
+  dino_teacher_temp ............................... 0.07
+  dino_warmup_teacher_temp ........................ 0.04
+  dino_warmup_teacher_temp_epochs ................. 30
+  disable_bf16_reduced_precision_matmul ........... False
+  disable_jit_fuser ............................... False
+  disable_straggler_on_startup .................... False
+  disable_symmetric_registration .................. False
+  dist_ckpt_format_deprecated ..................... None
+  dist_ckpt_optim_fully_reshardable ............... False
+  dist_ckpt_save_pre_mcore_014 .................... False
+  dist_ckpt_strictness ............................ assume_ok_unexpected
+  dist_ckpt_workers ............................... 1
+  distrib_optim_fully_reshardable_mem_efficient ... False
+  distribute_saved_activations .................... False
+  distributed_backend ............................. nccl
+  distributed_timeout_minutes ..................... 10
+  distributed_timeout_seconds_after_init .......... None
+  dsa_indexer_head_dim ............................ None
+  dsa_indexer_loss_coeff .......................... None
+  dsa_indexer_n_heads ............................. None
+  dsa_indexer_topk ................................ None
+  dsa_indexer_use_sparse_loss ..................... False
+  dump_param_to_param_group_map ................... None
+  embedding_init_method_std ....................... None
+  embedding_lr_multiplier ......................... None
+  embedding_path .................................. None
+  empty_unused_memory_level ....................... 0
+  enable_chunked_prefill .......................... False
+  enable_cuda_graph ............................... False
+  enable_experimental ............................. False
+  enable_ft_package ............................... False
+  enable_full_sharding_in_hsdp .................... False
+  enable_msc ...................................... True
+  enable_one_logger ............................... False
+  encoder_num_layers .............................. 24
+  encoder_seq_length .............................. 4096
+  end_weight_decay ................................ 0.01
+  eod_mask_loss ................................... False
+  ep_overlap_early_attn_memory_release ............ False
+  error_injection_rate ............................ 0
+  error_injection_type ............................ transient_error
+  eval_interval ................................... None
+  eval_iters ...................................... 100
+  evidence_data_path .............................. None
+  exit_duration_in_mins ........................... None
+  exit_interval ................................... None
+  exit_on_missing_checkpoint ...................... True
+  exit_signal ..................................... 15
+  exit_signal_handler ............................. False
+  exit_signal_handler_for_dataloader .............. False
+  exit_signal_handler_for_training ................ False
+  exp_avg_dtype ................................... torch.float32
+  exp_avg_sq_dtype ................................ torch.float32
+  experimental_attention_variant .................. None
+  expert_model_parallel_size ...................... 1
+  expert_tensor_parallel_size ..................... 1
+  external_cuda_graph ............................. False
+  fake_process_group .............................. False
+  ffn_hidden_size ................................. 4608
+  fim_data ........................................ False
+  fim_eod_token ................................... <|endoftext|>
+  fim_fragment_rate ............................... None
+  fim_middle_token ................................ <fim_middle>
+  fim_no_prefix ................................... None
+  fim_pad_token ................................... <fim_pad>
+  fim_prefix_token ................................ <fim_prefix>
+  fim_rate ........................................ 0.5
+  fim_split_sample ................................ None
+  fim_spm_rate .................................... 0.5
+  fim_suffix_token ................................ <fim_suffix>
+  fine_grained_activation_offloading .............. False
+  finetune ........................................ False
+  first_last_layers_bf16 .......................... False
+  flash_decode .................................... False
+  flight_recorder_dump_on_timeout ................. True
+  flight_recorder_dump_path ....................... None
+  flight_recorder_extra_dump_on_exec .............. True
+  flight_recorder_include_only_active ............. True
+  flight_recorder_include_stack_trace ............. False
+  flight_recorder_trace_buffer_size ............... 2000
+  fp16 ............................................ False
+  fp16_lm_cross_entropy ........................... False
+  fp32_residual_connection ........................ False
+  fp4 ............................................. None
+  fp4_param ....................................... False
+  fp4_quantizer_factory ........................... None
+  fp4_recipe ...................................... nvfp4
+  fp8 ............................................. None
+  fp8_amax_compute_algo ........................... most_recent
+  fp8_amax_history_len ............................ 1
+  fp8_interval .................................... 1
+  fp8_margin ...................................... 0
+  fp8_param_gather ................................ False
+  fp8_quantizer_factory ........................... None
+  fp8_recipe ...................................... delayed
+  fp8_wgrad ....................................... True
+  fsdp_double_buffer .............................. False
+  fsdp_manual_registration ........................ False
+  ft_num_warmup_iters ............................. 5
+  full_validation ................................. False
+  fused_residual_rmsnorm .......................... False
+  gain_parametrization ............................ softplus
+  gains_lr ........................................ None
+  gains_no_clamp_min .............................. False
+  global_batch_size ............................... 1
+  glu_linear_offset ............................... 0.0
+  goldfish_h ...................................... 50
+  goldfish_k ...................................... 50
+  goldfish_loss ................................... False
+  grad_reduce_in_bf16 ............................. False
+  gradient_accumulation_fusion .................... True
+  gradient_reduce_div_fusion ...................... True
+  group_query_attention ........................... True
+  grpo_clamp_eps_lower ............................ 0.01
+  grpo_clamp_eps_upper ............................ 0.01
+  grpo_entropy_term_weight ........................ 0.0
+  grpo_filter_groups_with_same_reward ............. False
+  grpo_group_size ................................. 2
+  grpo_iterations ................................. 2
+  grpo_kl_beta .................................... 0.001
+  grpo_prompts_per_step ........................... 32
+  gxpr ............................................ False
+  gxpr_fusion ..................................... True
+  gxprv2 .......................................... False
+  gxpry ........................................... False
+  gxr2 ............................................ False
+  head_lr_mult .................................... 1.0
+  heterogeneous_layers_config_encoded_json ........ None
+  heterogeneous_layers_config_path ................ None
+  hidden_dropout .................................. 0.0
+  hidden_size ..................................... 1152
+  hierarchical_context_parallel_sizes ............. None
+  high_priority_stream_groups ..................... []
+  hybrid_context_parallel ......................... False
+  hybrid_layer_pattern ............................ None
+  hybrid_override_pattern ......................... None
+  hypersphere_embedding_mode ...................... row
+  hypersphere_gains_mode .......................... rowcol
+  hypersphere_gains_mode_embedding ................ none
+  hypersphere_gains_mode_output ................... inherit
+  hypersphere_gains_mode_router ................... rowcol
+  hypersphere_mode ................................ flat
+  hypersphere_preserve_init ....................... False
+  hypersphere_radius_mode ......................... shape_native
+  hypersphere_router_mode ......................... row
+  hypersphere_scale_out_proj_init ................. False
+  hypersphere_tangential_grad ..................... False
+  hysteresis ...................................... 2
+  ict_head_size ................................... None
+  ict_load ........................................ None
+  img_h ........................................... 224
+  img_w ........................................... 224
+  indexer_batch_size .............................. 128
+  indexer_log_interval ............................ 1000
+  inference_batch_times_seqlen_threshold .......... -1
+  inference_coordinator_port ...................... None
+  inference_disable_triton_nvls_kernels ........... False
+  inference_dynamic_batching ...................... False
+  inference_dynamic_batching_block_size ........... 256
+  inference_dynamic_batching_buffer_size_gb ....... 40.0
+  inference_dynamic_batching_cuda_graph_max_tokens  16384
+  inference_dynamic_batching_cuda_graph_mixed_prefill_count  16
+  inference_dynamic_batching_enable_prefix_caching  False
+  inference_dynamic_batching_mamba_memory_ratio ... None
+  inference_dynamic_batching_max_requests ......... None
+  inference_dynamic_batching_max_tokens ........... None
+  inference_dynamic_batching_num_cuda_graphs ...... 16
+  inference_dynamic_batching_paused_buffer_size_gb  None
+  inference_dynamic_batching_prefix_caching_coordinator_policy  first_prefix_block
+  inference_dynamic_batching_prefix_caching_eviction_policy  ref_zero
+  inference_dynamic_batching_prefix_caching_mamba_gb  None
+  inference_dynamic_batching_prefix_caching_routing_alpha  0.5
+  inference_dynamic_batching_track_generated_token_events  False
+  inference_dynamic_batching_track_paused_request_events  False
+  inference_dynamic_batching_unified_memory_level . 0
+  inference_fuse_tp_communication ................. False
+  inference_grouped_gemm_backend .................. auto
+  inference_logging_step_interval ................. 0
+  inference_max_requests .......................... 8
+  inference_max_seq_length ........................ 2560
+  inference_moe_disable_fused_quant_kernels ....... False
+  inference_rng_tracker ........................... False
+  inference_text_gen_server_logging ............... False
+  inference_use_synchronous_zmq_collectives ....... False
+  inference_wandb_logging ......................... False
+  init_method_std ................................. 0.02
+  init_method_xavier_uniform ...................... False
+  init_model_with_meta_device ..................... False
+  initial_loss_scale .............................. 4294967296
+  inprocess_active_world_size ..................... 1
+  inprocess_barrier_timeout ....................... 120
+  inprocess_completion_timeout .................... 120
+  inprocess_empty_cuda_cache ...................... False
+  inprocess_granularity ........................... node
+  inprocess_hard_timeout .......................... 90
+  inprocess_heartbeat_interval .................... 30
+  inprocess_heartbeat_timeout ..................... 60
+  inprocess_last_call_wait ........................ 1
+  inprocess_max_iterations ........................ None
+  inprocess_monitor_process_interval .............. 1.0
+  inprocess_monitor_thread_interval ............... 1.0
+  inprocess_progress_watchdog_interval ............ 1.0
+  inprocess_restart ............................... False
+  inprocess_soft_timeout .......................... 60
+  inprocess_termination_grace_time ................ 1
+  is_hybrid_model ................................. False
+  iter_per_epoch .................................. 1250
+  iteration ....................................... 582
+  iterations_to_skip .............................. []
+  keel ............................................ False
+  keel_alpha ...................................... None
+  keep_fp8_transpose_cache ........................ False
+  kitchen_attention_backend ....................... sdpa
+  kv_channels ..................................... 128
+  kv_lora_rank .................................... 32
+  langrl_env_config ............................... None
+  layernorm_epsilon ............................... 1e-05
+  layernorm_zero_centered_gamma ................... False
+  lazy_mpu_init ................................... False
+  lglu ............................................ False
+  linear_attention_allow_neg_eigval ............... False
+  linear_attention_beta_bias_init ................. 0.0
+  linear_attention_beta_scale ..................... 1.0
+  linear_attention_carried_state_max_frob ......... 0.0
+  linear_attention_carry_state .................... False
+  linear_attention_freq ........................... None
+  linear_attention_full_rank_output_gate .......... False
+  linear_attention_learnable_initial_state ........ False
+  linear_attention_n_erase ........................ 0
+  linear_attention_n_householder .................. 1
+  linear_attention_output_gate_form ............... per_channel
+  linear_attention_qk_norm ........................ l2norm
+  linear_attention_qk_norm_init_scale ............. 1.0
+  linear_attention_safe_output_gate ............... False
+  linear_attention_safe_output_gate_lower_bound ... -5.0
+  linear_attention_use_decay ...................... True
+  linear_attention_use_output_gate ................ True
+  linear_attention_v_norm ......................... none
+  linear_conv_kernel_dim .......................... 4
+  linear_key_head_dim ............................. 128
+  linear_num_key_heads ............................ 16
+  linear_num_value_heads .......................... 32
+  linear_value_head_dim ........................... 128
+  lion_beta1 ...................................... 0.95
+  lion_beta2 ...................................... 0.98
+  load ............................................ /iopsstor/scratch/cscs/rsinghal/1pp-hf-tmp/1pp-0.5b-asst-sft/torch
+  load_main_params_from_ckpt ...................... False
+  local_rank ...................................... 0
+  log_device_memory_used .......................... False
+  log_energy ...................................... False
+  log_interval .................................... 100
+  log_loss_scale_to_tensorboard ................... True
+  log_max_attention_logit ......................... False
+  log_memory_interval ............................. None
+  log_memory_to_tensorboard ....................... False
+  log_muon_gains .................................. False
+  log_muon_param_rms .............................. False
+  log_muon_per_layer .............................. False
+  log_muon_sparsity ............................... False
+  log_num_zeros_in_grad ........................... False
+  log_params_norm ................................. False
+  log_progress .................................... False
+  log_straggler ................................... False
+  log_throughput .................................. False
+  log_timers_to_tensorboard ....................... False
+  log_validation_ppl_to_tensorboard ............... False
+  log_world_size_to_tensorboard ................... False
+  logging_level ................................... None
+  loss_mask_segment_suffix ........................ None
+  loss_mask_train_segments ........................ None
+  loss_scale ...................................... None
+  loss_scale_window ............................... 1000
+  lr .............................................. None
+  lr_decay_iters .................................. None
+  lr_decay_samples ................................ None
+  lr_decay_style .................................. linear
+  lr_warmup_fraction .............................. None
+  lr_warmup_init .................................. 0.0
+  lr_warmup_iters ................................. 0
+  lr_warmup_samples ............................... 0
+  lr_wsd_decay_iters .............................. None
+  lr_wsd_decay_samples ............................ None
+  lr_wsd_decay_style .............................. exponential
+  main_grads_dtype ................................ torch.float32
+  main_params_dtype ............................... torch.float32
+  make_vocab_size_divisible_by .................... 128
+  mamba_head_dim .................................. 64
+  mamba_inference_conv_states_dtype ............... torch.bfloat16
+  mamba_inference_ssm_states_dtype ................ torch.bfloat16
+  mamba_num_groups ................................ 8
+  mamba_num_heads ................................. None
+  mamba_state_dim ................................. 128
+  manual_gc ....................................... False
+  manual_gc_eval .................................. True
+  manual_gc_interval .............................. 0
+  mask_factor ..................................... 1.0
+  mask_prob ....................................... 0.15
+  mask_type ....................................... random
+  masked_softmax_fusion ........................... False
+  matrix_lr ....................................... None
+  max_docs_per_bin ................................ 0
+  max_position_embeddings ......................... 4096
+  max_seqlen_per_dp_cp_rank ....................... None
+  max_tokens_to_oom ............................... 12000
+  md_normalize_update_to_weight_norm .............. False
+  md_router_use_orthogonal_updates ................ None
+  megatron_fsdp_grad_comm_dtype ................... None
+  megatron_fsdp_main_grads_dtype .................. None
+  megatron_fsdp_main_params_dtype ................. torch.float32
+  memory_snapshot_path ............................ snapshot.pickle
+  merge_file ...................................... None
+  micro_batch_size ................................ 1
+  microbatch_group_size_per_vp_stage .............. None
+  mid_level_dataset_surplus ....................... 0.005
+  min_loss_scale .................................. 1.0
+  min_lr .......................................... 0.0
+  min_lr_mode ..................................... relative
+  min_offloaded_tensor_size ....................... 1048576
+  mla_down_proj_fusion ............................ False
+  mlp_chunks_for_prefill .......................... 1
+  mmap_bin_files .................................. True
+  mock_data ....................................... True
+  moe_apply_probs_on_input ........................ False
+  moe_aux_loss_coeff .............................. 0.0
+  moe_deepep_num_sms .............................. 20
+  moe_enable_deepep ............................... False
+  moe_enable_routing_replay ....................... False
+  moe_expert_capacity_factor ...................... None
+  moe_ffn_hidden_size ............................. None
+  moe_flex_dispatcher_backend ..................... deepep
+  moe_grouped_gemm ................................ False
+  moe_hybridep_num_sms ............................ 16
+  moe_input_jitter_eps ............................ None
+  moe_latent_size ................................. None
+  moe_layer_freq .................................. 1
+  moe_layer_recompute ............................. False
+  moe_offload_activations ......................... None
+  moe_offload_main_grad ........................... False
+  moe_offloading_chunk_size ....................... -1
+  moe_offloading_experts_debug_mode ............... False
+  moe_offloading_experts_skip_post_backward_hook .. False
+  moe_offloading_experts_te_style_init ............ True
+  moe_offloading_mode ............................. fine-grained
+  moe_offloading_num_chunks ....................... 8
+  moe_offloading_num_stages ....................... 2
+  moe_pad_expert_input_to_capacity ................ False
+  moe_pad_experts_for_cuda_graph_inference ........ False
+  moe_per_layer_logging ........................... False
+  moe_permute_fusion .............................. False
+  moe_router_bias_metrics ......................... False
+  moe_router_bias_update_rate ..................... 0.001
+  moe_router_dtype ................................ None
+  moe_router_enable_expert_bias ................... False
+  moe_router_force_biased ......................... None
+  moe_router_force_load_balancing ................. False
+  moe_router_fusion ............................... False
+  moe_router_group_topk ........................... None
+  moe_router_inference_violation_metrics .......... []
+  moe_router_load_balancing_type .................. aux_loss
+  moe_router_num_groups ........................... None
+  moe_router_padding_for_fp8 ...................... False
+  moe_router_padding_for_quantization ............. False
+  moe_router_pre_softmax .......................... False
+  moe_router_quantile_balancing_ema ............... 0.0
+  moe_router_quantile_balancing_method ............ histogram
+  moe_router_quantile_balancing_num_bins .......... 1000
+  moe_router_score_function ....................... softmax
+  moe_router_topk ................................. 2
+  moe_router_topk_scaling_factor .................. None
+  moe_router_violation_metrics .................... ['mbs']
+  moe_shared_expert_gate .......................... False
+  moe_shared_expert_intermediate_size ............. None
+  moe_shared_expert_overlap ....................... False
+  moe_token_dispatcher_type ....................... allgather
+  moe_token_drop_policy ........................... probs
+  moe_upcycling_granularity ....................... 1
+  moe_use_extra_fp8_param_storage ................. False
+  moe_use_fp8_activation .......................... False
+  moe_use_fp8_dispatch ............................ False
+  moe_use_inplace_fp8_param ....................... False
+  moe_use_offloading_experts ...................... False
+  moe_use_upcycling ............................... False
+  moe_z_loss_coeff ................................ None
+  mrope_section ................................... None
+  mscale .......................................... 1.0
+  mscale_all_dim .................................. 0.0
+  mtp_hybrid_override_pattern ..................... None
+  mtp_loss_scaling_factor ......................... 0.1
+  mtp_num_layers .................................. None
+  mtp_standalone .................................. False
+  mtp_use_repeated_layer .......................... False
+  multi_latent_attention .......................... False
+  multiple_validation_sets ........................ False
+  muon_coefficient_type ........................... quintic
+  muon_extra_scale_factor ......................... 1.0
+  muon_fp32_matmul_prec ........................... medium
+  muon_log_interval ............................... None
+  muon_lr_factor .................................. 1.0
+  muon_momentum ................................... 0.95
+  muon_num_ns_steps ............................... 5
+  muon_router_scale_mode .......................... none
+  muon_scalar_optimizer ........................... adam
+  muon_scale_mode ................................. spectral
+  muon_sparsity_thresholds ........................ [1e-20, 1e-10, 1e-30]
+  muon_split_fc1 .................................. True
+  muon_split_mla_per_head ......................... False
+  muon_split_qkv .................................. True
+  muon_tp_mode .................................... duplicated
+  muon_use_nesterov ............................... False
+  mup_attn_scale_power ............................ 1.0
+  mup_base_head_dim ............................... None
+  mup_base_hidden_size ............................ None
+  mup_embedding_mult .............................. 1.0
+  mup_output_mult ................................. 1.0
+  mup_width_mult .................................. 1.0
+  nccl_all_reduce_for_prefill ..................... False
+  nccl_communicator_config_path ................... None
+  nccl_ub ......................................... False
+  no_load_optim ................................... True
+  no_load_rng ..................................... True
+  no_persist_layer_norm ........................... False
+  no_rope_freq .................................... None
+  no_save_optim ................................... True
+  no_save_rng ..................................... True
+  no_weight_decay_cond_type ....................... None
+  non_persistent_ckpt_type ........................ None
+  non_persistent_global_ckpt_dir .................. None
+  non_persistent_local_ckpt_algo .................. fully_parallel
+  non_persistent_local_ckpt_dir ................... None
+  non_persistent_save_interval .................... None
+  normalization ................................... RMSNorm
+  num_attention_heads ............................. 9
+  num_channels .................................... 3
+  num_classes ..................................... 1000
+  num_dataset_builder_threads ..................... 1
+  num_distributed_optimizer_instances ............. 1
+  num_experts ..................................... None
+  num_layers ...................................... 24
+  num_layers_at_end_in_bf16 ....................... 1
+  num_layers_at_start_in_bf16 ..................... 1
+  num_layers_per_virtual_pipeline_stage ........... None
+  num_query_groups ................................ 3
+  num_speculative_tokens .......................... 0
+  num_virtual_stages_per_pipeline_rank ............ None
+  num_workers ..................................... 2
+  object_storage_cache_path ....................... None
+  offload_modules ................................. []
+  one_logger_async ................................ False
+  one_logger_project .............................. megatron-lm
+  one_logger_run_name ............................. None
+  onnx_safe ....................................... None
+  openai_gelu ..................................... False
+  optimizer ....................................... adam
+  optimizer_cpu_offload ........................... False
+  optimizer_cuda_graph ............................ False
+  optimizer_offload_fraction ...................... 1.0
+  outer_dp_sharding_strategy ...................... no_shard
+  output_bert_embeddings .......................... False
+  output_lr ....................................... None
+  overlap_cpu_optimizer_d2h_h2d ................... False
+  overlap_grad_reduce ............................. False
+  overlap_moe_expert_parallel_comm ................ False
+  overlap_p2p_comm ................................ False
+  overlap_p2p_comm_warmup_flush ................... False
+  overlap_param_gather ............................ False
+  overlap_param_gather_with_optimizer_step ........ False
+  override_opt_param_scheduler .................... False
+  padded_vocab_size ............................... 49280
+  params_dtype .................................... torch.bfloat16
+  patch_dim ....................................... 16
+  per_dataset_sequences_path ...................... None
+  per_split_data_args_path ........................ None
+  perform_initialization .......................... False
+  perform_rl_step ................................. False
+  phase_transition_iterations ..................... None
+  pin_cpu_grads ................................... True
+  pin_cpu_params .................................. True
+  pipeline_model_parallel_comm_backend ............ None
+  pipeline_model_parallel_layout .................. None
+  pipeline_model_parallel_size .................... 1
+  pn3glu .......................................... False
+  pnglu ........................................... False
+  pnglu_fusion .................................... True
+  polynorm ........................................ False
+  position_embedding_type ......................... rope
+  post_attn_norm_zero_init ........................ False
+  prepacked_samples ............................... False
+  pretrained_checkpoint ........................... None
+  pretraining_packing_strategy .................... greedy
+  profile ......................................... False
+  profile_ranks ................................... []
+  profile_step_end ................................ 12
+  profile_step_start .............................. 10
+  pytorch_profiler_collect_callstack .............. False
+  pytorch_profiler_collect_chakra ................. False
+  pytorch_profiler_collect_shapes ................. False
+  q_lora_rank ..................................... None
+  qk_clip ......................................... False
+  qk_clip_alpha ................................... 0.5
+  qk_clip_threshold ............................... 100
+  qk_head_dim ..................................... 128
+  qk_l2_norm ...................................... False
+  qk_layernorm .................................... False
+  qk_pos_emb_head_dim ............................. 64
+  query_in_block_prob ............................. 0.1
+  quick_geglu ..................................... False
+  rampup_batch_size ............................... None
+  rank ............................................ 0
+  recompute_granularity ........................... None
+  recompute_method ................................ None
+  recompute_modules ............................... None
+  recompute_num_layers ............................ None
+  record_memory_history ........................... False
+  refit_method .................................... gloo
+  reglu ........................................... False
+  relative_attention_max_distance ................. 128
+  relative_attention_num_buckets .................. 32
+  replication ..................................... False
+  replication_factor .............................. 2
+  replication_jump ................................ None
+  rerun_mode ...................................... validate_results
+  rerun_strategy .................................. rerun_in_place
+  reset_attention_mask ............................ False
+  reset_position_ids .............................. False
+  residual_output_scaling ......................... False
+  result_rejected_tracker_filename ................ None
+  retriever_report_topk_accuracies ................ []
+  retriever_score_scaling ......................... False
+  reuse_grad_buf_for_mxfp8_param_ag ............... False
+  rl_default_temperature .......................... 1.0
+  rl_default_top_k ................................ -1
+  rl_default_top_p ................................ 0
+  rl_generation_batch_size ........................ None
+  rl_importance_sampling_truncation_coef .......... None
+  rl_inference_expert_model_parallel_size ......... None
+  rl_inference_expert_tensor_model_parallel_size .. None
+  rl_inference_logprobs_is_correction ............. False
+  rl_inference_model_unified_memory_level ......... 0
+  rl_inference_parsers ............................ []
+  rl_inference_pipeline_model_parallel_size ....... None
+  rl_inference_tensor_model_parallel_size ......... None
+  rl_kv_cache_management_mode ..................... persist
+  rl_num_parallel_generation_batches .............. None
+  rl_num_parallel_generations ..................... None
+  rl_offload_inference_model_weights_when_idle .... False
+  rl_offload_optimizer_during_inference ........... False
+  rl_parallel_generation_tasks .................... None
+  rl_partial_rollouts ............................. False
+  rl_persist_cuda_graphs .......................... False
+  rl_prompts_per_eval ............................. 32
+  rl_sequence_packing_algo ........................ fifo
+  rl_sequence_packing_max_sequences_per_bin ....... 50
+  rl_skip_bos_token ............................... False
+  rl_training_cuda_graphs ......................... False
+  rl_use_sequence_packing ......................... False
+  rl_verify_model_weights_swap .................... False
+  rlglu ........................................... False
+  rope_scaling_factor ............................. 1.0
+  rope_type ....................................... None
+  rotary_base ..................................... 10000
+  rotary_interleaved .............................. False
+  rotary_percent .................................. 1.0
+  rotary_scaling_factor ........................... 1.0
+  rotary_seq_len_interpolation_factor ............. None
+  run_workload_inspector_server ................... False
+  sample_rate ..................................... 1.0
+  sandwich_norm ................................... False
+  save ............................................ None
+  save_dgrads_interval ............................ None
+  save_interval ................................... None
+  save_iters ...................................... None
+  save_retain_interval ............................ None
+  save_wgrads_interval ............................ None
+  scale_embeddings_by_sqrt_hidden ................. False
+  scatter_gather_tensors_in_pipeline .............. True
+  seed ............................................ 1234
+  seq_length ...................................... 4096
+  sequence_parallel ............................... False
+  sft ............................................. False
+  sft_tokenizer_prompt_format ..................... nemotron-h-aligned
+  sgd_momentum .................................... 0.9
+  sharp_enabled_group ............................. None
+  short_seq_prob .................................. 0.1
+  situ ............................................ False
+  skip_train ...................................... False
+  skipped_train_samples ........................... 0
+  softmax_type .................................... vanilla
+  spec ............................................ None
+  split ........................................... None
+  squared_relu .................................... False
+  ssglu ........................................... False
+  sssglu .......................................... False
+  start_weight_decay .............................. 0.01
+  straggler_ctrlr_port ............................ 65535
+  straggler_minmax_count .......................... 1
+  strict_fsdp_dtensor_load ........................ True
+  suggested_communication_unit_size ............... None
+  swiglu .......................................... True
+  swin_backbone_type .............................. tiny
+  symmetric_ar_type ............................... None
+  te_precision_config_file ........................ None
+  te_rng_tracker .................................. False
+  tensor_model_parallel_size ...................... 1
+  tensorboard_dir ................................. None
+  tensorboard_log_interval ........................ 1
+  tensorboard_queue_size .......................... 1000
+  test_data_path .................................. None
+  test_mode ....................................... False
+  tiktoken_num_special_tokens ..................... 1000
+  tiktoken_pattern ................................ None
+  tiktoken_special_tokens ......................... None
+  timing_log_level ................................ 0
+  timing_log_option ............................... minmax
+  titles_data_path ................................ None
+  tokenizer_hf_include_special_tokens ............. True
+  tokenizer_hf_no_include_special_tokens .......... False
+  tokenizer_hf_no_use_fast ........................ False
+  tokenizer_hf_use_fast ........................... True
+  tokenizer_metadata .............................. None
+  tokenizer_model ................................. /capstor/store/cscs/swissai/infra01/users/rsinghal/1pp-training/tokenizers/smollm2_pretrain
+  tokenizer_sentencepiece_legacy .................. False
+  tokenizer_special_tokens ........................ None
+  tokenizer_type .................................. HuggingFaceTokenizer
+  torch_fsdp2_reshard_after_forward ............... True
+  tp_comm_bootstrap_backend ....................... nccl
+  tp_comm_bulk_dgrad .............................. True
+  tp_comm_bulk_wgrad .............................. True
+  tp_comm_overlap ................................. False
+  tp_comm_overlap_ag .............................. True
+  tp_comm_overlap_cfg ............................. None
+  tp_comm_overlap_rs .............................. True
+  tp_comm_overlap_rs_dgrad ........................ False
+  tp_comm_split_ag ................................ True
+  tp_comm_split_rs ................................ True
+  train_data_path ................................. None
+  train_iters ..................................... None
+  train_samples ................................... None
+  train_sync_interval ............................. None
+  transformer_impl ................................ transformer_engine
+  transformer_pipeline_model_parallel_size ........ 1
+  trust_remote_code ............................... False
+  untie_embeddings_and_output_weights ............. True
+  use_checkpoint_args ............................. False
+  use_checkpoint_opt_param_scheduler .............. False
+  use_cpu_initialization .......................... True
+  use_dist_ckpt ................................... True
+  use_dist_ckpt_deprecated ........................ False
+  use_distributed_optimizer ....................... False
+  use_flash_attn .................................. False
+  use_fused_weighted_squared_relu ................. False
+  use_gloo_process_groups ......................... True
+  use_kitchen_attention ........................... False
+  use_layer_wise_distributed_optimizer ............ False
+  use_legacy_models ............................... False
+  use_legacy_static_engine ........................ False
+  use_mamba_mem_eff_path .......................... True
+  use_megatron_fsdp ............................... False
+  use_mp_args_from_checkpoint_args ................ True
+  use_mup ......................................... False
+  use_one_sent_docs ............................... False
+  use_orthogonal_updates .......................... True
+  use_persistent_ckpt_worker ...................... False
+  use_precision_aware_optimizer ................... False
+  use_pytorch_profiler ............................ False
+  use_ring_exchange_p2p ........................... False
+  use_rope_scaling ................................ False
+  use_rotary_position_embeddings .................. False
+  use_sharp ....................................... False
+  use_te_activation_func .......................... False
+  use_tokenizer_model_from_checkpoint_args ........ True
+  use_torch_fsdp2 ................................. False
+  use_torch_optimizer_for_cpu_offload ............. False
+  use_tp_pp_dp_mapping ............................ False
+  v_head_dim ...................................... 128
+  valid_data_path ................................. None
+  variable_seq_lengths ............................ False
+  virtual_pipeline_model_parallel_size ............ None
+  vision_backbone_type ............................ vit
+  vision_pretraining .............................. False
+  vision_pretraining_type ......................... classify
+  vocab_extra_ids ................................. 0
+  vocab_file ...................................... None
+  vocab_size ...................................... None
+  wandb_entity .................................... None
+  wandb_exp_name .................................. None
+  wandb_project ................................... None
+  wandb_save_dir .................................. None
+  weight_decay .................................... 0.01
+  weight_decay_all_param .......................... False
+  weight_decay_incr_style ......................... constant
+  wgrad_deferral_limit ............................ 0
+  window_attn_skip_freq ........................... None
+  window_size ..................................... None
+  world_size ...................................... 1
+  xpr ............................................. False
+  xr2 ............................................. False
+  xr2glu .......................................... False
+  xssglu .......................................... False
+  yaml_cfg ........................................ None
+-------------------- end of arguments ---------------------
+INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 1
+building GPT model ...
+ loading checkpoint from /iopsstor/scratch/cscs/rsinghal/1pp-hf-tmp/1pp-0.5b-asst-sft/torch at iteration 582
+ checkpoint version 3.0
+  successfully loaded checkpoint from /iopsstor/scratch/cscs/rsinghal/1pp-hf-tmp/1pp-0.5b-asst-sft/torch [ t 1/1, p 1/1 ] at iteration 582
+`torch_dtype` is deprecated! Use `dtype` instead!
+received embeddings
+received transformer layer 0
+received transformer layer 1
+received transformer layer 2
+received transformer layer 3
+received transformer layer 4
+received transformer layer 5
+received transformer layer 6
+received transformer layer 7
+received transformer layer 8
+received transformer layer 9
+received transformer layer 10
+received transformer layer 11
+received transformer layer 12
+received transformer layer 13
+received transformer layer 14
+received transformer layer 15
+received transformer layer 16
+received transformer layer 17
+received transformer layer 18
+received transformer layer 19
+received transformer layer 20
+received transformer layer 21
+received transformer layer 22
+received transformer layer 23
+received final norm
+received output layer
+Building LlamaForCausalLM from converted weights …
+Saving model (safetensors) to /capstor/store/cscs/swissai/infra01/users/rsinghal/1pp-training/hf/models/1pp-0.5b-asst-sft
+> memory usage: 'loader', rank 0 / 1, mem 3.6/856.2 gb.
+sending embeddings
+sending transformer layer 0
+sending transformer layer 1
+sending transformer layer 2
+sending transformer layer 3
+sending transformer layer 4
+sending transformer layer 5
+sending transformer layer 6
+sending transformer layer 7
+sending transformer layer 8
+sending transformer layer 9
+sending transformer layer 10
+sending transformer layer 11
+sending transformer layer 12
+sending transformer layer 13
+sending transformer layer 14
+sending transformer layer 15
+sending transformer layer 16
+sending transformer layer 17
+sending transformer layer 18
+sending transformer layer 19
+sending transformer layer 20
+sending transformer layer 21
+sending transformer layer 22
+sending transformer layer 23
+sending final norm
+sending output layer
+Waiting for saver to complete...

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "bos_token_id": 1,
+  "do_sample": true,
+  "eos_token_id": [
+    2,
+    0
+  ],
+  "pad_token_id": 49152,
+  "temperature": 0.6,
+  "top_p": 0.9,
+  "transformers_version": "4.57.6"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ea95f8461efe78415f3283bbe394908de10a9a6d3c68ff73cec2d79045899680
+size 1160916120

special_tokens_map.json

@@ -0,0 +1,34 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "bos_token": {
+    "content": "<|im_start|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|pad|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,162 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<repo_name>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<reponame>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<file_sep>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "6": {
+      "content": "<filename>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "7": {
+      "content": "<gh_stars>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "8": {
+      "content": "<issue_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "9": {
+      "content": "<issue_comment>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "10": {
+      "content": "<issue_closed>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "11": {
+      "content": "<jupyter_start>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "12": {
+      "content": "<jupyter_text>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "13": {
+      "content": "<jupyter_code>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "14": {
+      "content": "<jupyter_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "15": {
+      "content": "<jupyter_script>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "16": {
+      "content": "<empty_output>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "49152": {
+      "content": "<|pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "bos_token": "<|im_start|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "extra_special_tokens": {},
+  "model_max_length": 8192,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}