2026-01-05 20:46:26,765 INFO    MainThread:763517 [wandb_setup.py:_flush():67] Current SDK version is 0.19.8
2026-01-05 20:46:26,765 INFO    MainThread:763517 [wandb_setup.py:_flush():67] Configure stats pid to 763517
2026-01-05 20:46:26,765 INFO    MainThread:763517 [wandb_setup.py:_flush():67] Loading settings from /home/agrivas/.config/wandb/settings
2026-01-05 20:46:26,765 INFO    MainThread:763517 [wandb_setup.py:_flush():67] Loading settings from /disk/scratch/agrivas/nanoGPT/logs/2026-01-05/20-41-36/wandb/settings
2026-01-05 20:46:26,765 INFO    MainThread:763517 [wandb_setup.py:_flush():67] Loading settings from environment variables
2026-01-05 20:46:26,765 INFO    MainThread:763517 [wandb_init.py:setup_run_log_directory():647] Logging user logs to /disk/scratch/agrivas/nanoGPT/logs/2026-01-05/20-41-36/wandb/run-20260105_204626-thouxsll/logs/debug.log
2026-01-05 20:46:26,765 INFO    MainThread:763517 [wandb_init.py:setup_run_log_directory():648] Logging internal logs to /disk/scratch/agrivas/nanoGPT/logs/2026-01-05/20-41-36/wandb/run-20260105_204626-thouxsll/logs/debug-internal.log
2026-01-05 20:46:26,765 INFO    MainThread:763517 [wandb_init.py:init():761] calling init triggers
2026-01-05 20:46:26,765 INFO    MainThread:763517 [wandb_init.py:init():766] wandb.init called with sweep_config: {}
config: {'compile': True, 'device': 'cuda', 'from_checkpoint': None, 'load_mtp_head_from_model': 'outputs/models/llama/no-lora/llama-lr-3e-4-no-lora-btree-n-8-r-32-s-1/model@900.pt', 'name': 'nanogpt', 'training': {'random_seed': 13, 'batch_size': 256, 'device_batch_size': 1, 'sequence_length': 8192, 'num_iterations': 900, 'learning_rate': 0.0003, 'use_scheduler': False, 'save_model': True, 'save_optimizer': True, 'save_model_every': 100, 'val_loss_every': 100, 'val_tokens': 4194304, 'expname': 'llama-lr-3e-4-lora-last-4-cont-btree-n-8-r-32-s-1'}, 'model': {'name': 'mtp', 'beta': 0.0, 'gamma': 1, 'kl_algorithm': 'full', 'kl_type': 'forward', 'model': {'_target_': 'mtp.models.mtp.MultiTokenLM', 'lm': '${lm.model}', 'circuit': '${circuit.model}', 'mt_head_kwargs': '${mt_head.hyperparameters}', 'init_from_lm_head': True, 'kl_type': '${model.kl_type}', 'kl_algorithm': '${model.kl_algorithm}', 'beta': 0, 'gamma': 0.9}}, 'circuit': {'name': 'btree', 'n_token': 8, 'n_component': 32, 'n_repetition': 1, 'model': {'_target_': 'mtp.models.circuits.CircuitModel', 'vocab_size': 268, 'n_token': 8, 'n_component': 32, 'n_repetition': 1, 'kind': 'btree'}}, 'mt_head': {'name': 'linear-evabyte', 'hyperparameters': {'type': 'evabyte', 'n_embd': 3072, 'transformer_n_head': 24, 'transformer_n_layer': 0, 'expander_type': 'linear', 'expander_n_layer': 1, 'freeze_vocab_unembedding': False, 'share_sum_weights': False, 'contextual_hmm_weights': True, 'init_hmm_identity': True}}, 'adaptor': {'name': 'lora-last-4', 'hyperparameters': {'r': 16, 'lora_alpha': 16, 'lora_dropout': 0.0, 'target_modules': ['gate_proj', 'down_proj', 'up_proj', 'q_proj', 'v_proj', 'k_proj', 'o_proj'], 'layers_to_transform': [24, 25, 26, 27]}}, 'lm': {'name': 'llama3-2-3b-byte', 'n_embd': 3072, 'n_head': 24, 'num_layers': 28, 'last_1_layers': [27], 'last_2_layers': [26, 27], 'last_4_layers': [24, 25, 26, 27], 'model': {'_target_': 'mtp.models.lm.LM', 'lm': None, 'encoder_only': True, 'from_checkpoint': None, 'from_huggingface': 'benjamin/Llama3-2-3B-IT-Byte', 'adaptor_kwargs': {'r': 16, 'lora_alpha': 16, 'lora_dropout': 0.0, 'target_modules': ['gate_proj', 'down_proj', 'up_proj', 'q_proj', 'v_proj', 'k_proj', 'o_proj'], 'layers_to_transform': [24, 25, 26, 27]}, 'ref_enc': 'model', 'ref_head': 'lm_head', 'freeze': True}}, 'data': {'name': 'tulu3-llama3', 'train_bin': 'agrv/tulu-v3-sft-llama3-packed-seq-len-8192', 'val_bin': None, 'vocab_size': 268}, 'generate': {'speculative': False}, '_wandb': {}}
2026-01-05 20:46:26,765 INFO    MainThread:763517 [wandb_init.py:init():784] starting backend
2026-01-05 20:46:26,766 INFO    MainThread:763517 [wandb_init.py:init():788] sending inform_init request
2026-01-05 20:46:26,789 INFO    MainThread:763517 [backend.py:_multiprocessing_setup():101] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2026-01-05 20:46:26,789 INFO    MainThread:763517 [wandb_init.py:init():798] backend started and connected
2026-01-05 20:46:26,794 INFO    MainThread:763517 [wandb_init.py:init():891] updated telemetry
2026-01-05 20:46:26,806 INFO    MainThread:763517 [wandb_init.py:init():915] communicating run to backend with 90.0 second timeout
2026-01-05 20:46:27,375 INFO    MainThread:763517 [wandb_init.py:init():990] starting run threads in backend
2026-01-05 20:46:28,018 INFO    MainThread:763517 [wandb_run.py:_console_start():2375] atexit reg
2026-01-05 20:46:28,019 INFO    MainThread:763517 [wandb_run.py:_redirect():2227] redirect: wrap_raw
2026-01-05 20:46:28,023 INFO    MainThread:763517 [wandb_run.py:_redirect():2292] Wrapping output streams.
2026-01-05 20:46:28,023 INFO    MainThread:763517 [wandb_run.py:_redirect():2315] Redirects installed.
2026-01-05 20:46:28,045 INFO    MainThread:763517 [wandb_init.py:init():1032] run started, returning control to user process
2026-01-08 12:16:49,986 INFO    MsgRouterThr:763517 [mailbox.py:close():129] Closing mailbox, abandoning 1 handles.