| > Training Environment: | |
| | > Backend: Torch | |
| | > Mixed precision: False | |
| | > Precision: float32 | |
| | > Current device: 0 | |
| | > Num. of GPUs: 1 | |
| | > Num. of CPUs: 256 | |
| | > Num. of Torch Threads: 1 | |
| | > Torch seed: 1 | |
| | > Torch CUDNN: True | |
| | > Torch CUDNN deterministic: False | |
| | > Torch CUDNN benchmark: False | |
| | > Torch TF32 MatMul: False | |
| > Start Tensorboard: tensorboard --logdir=/workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000 | |
| > Restoring from best_model_16244.pth ... | |
| > Restoring Model... | |
| > Restoring Optimizer... | |
| > Model restored from step 16244 | |
| > Model has 518442047 parameters | |
| [4m[1m > EPOCH: 0/6[0m | |
| --> /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000 | |
| [1m > EVALUATION [0m | |
| [1m--> EVAL PERFORMANCE[0m | |
| | > avg_loader_time: 0.07435984869261048 [0m(+0) | |
| | > avg_loss_text_ce: 0.030082402234846674 [0m(+0) | |
| | > avg_loss_mel_ce: 3.822696221841348 [0m(+0) | |
| | > avg_loss: 3.852778621622034 [0m(+0) | |
| > BEST MODEL : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/best_model_16245.pth | |
| [4m[1m > EPOCH: 1/6[0m | |
| --> /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000 | |
| [1m > TRAINING (2025-12-21 12:14:05) [0m | |
| [1m --> TIME: 2025-12-21 12:14:11 -- STEP: 5/2207 -- GLOBAL_STEP: 16250[0m | |
| | > loss_text_ce: 0.03219795227050781 (0.03218773603439331) | |
| | > loss_mel_ce: 4.446129322052002 (4.046912813186646) | |
| | > loss: 0.06997386366128922 (0.0637359470129013) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2695 (0.2764430522918701) | |
| | > loader_time: 0.0089 (0.010691452026367187) | |
| [1m --> TIME: 2025-12-21 12:14:30 -- STEP: 55/2207 -- GLOBAL_STEP: 16300[0m | |
| | > loss_text_ce: 0.03195399418473244 (0.030980682135982946) | |
| | > loss_mel_ce: 3.563579797744751 (3.869967087832364) | |
| | > loss: 0.05618021637201309 (0.0609523091126572) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2691 (0.27666049003601073) | |
| | > loader_time: 0.009 (0.01010052507573908) | |
| [1m --> TIME: 2025-12-21 12:14:49 -- STEP: 105/2207 -- GLOBAL_STEP: 16350[0m | |
| | > loss_text_ce: 0.03152564913034439 (0.03094851766668615) | |
| | > loss_mel_ce: 3.851048707962036 (3.864614899953206) | |
| | > loss: 0.06066522374749184 (0.060868178449925925) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2912 (0.27978511991954985) | |
| | > loader_time: 0.0092 (0.009624846776326497) | |
| [1m --> TIME: 2025-12-21 12:15:09 -- STEP: 155/2207 -- GLOBAL_STEP: 16400[0m | |
| | > loss_text_ce: 0.029248259961605072 (0.03089537698655359) | |
| | > loss_mel_ce: 3.628293514251709 (3.846851090461977) | |
| | > loss: 0.05714908987283707 (0.06058978856571259) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2925 (0.28088303073760007) | |
| | > loader_time: 0.0081 (0.009488651829381148) | |
| [1m --> TIME: 2025-12-21 12:15:28 -- STEP: 205/2207 -- GLOBAL_STEP: 16450[0m | |
| | > loss_text_ce: 0.032587237656116486 (0.03095879358489339) | |
| | > loss_mel_ce: 3.7422239780426025 (3.8365380601185124) | |
| | > loss: 0.05898142606019974 (0.060429638442469805) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2908 (0.2808621534487096) | |
| | > loader_time: 0.0082 (0.009342936771671948) | |
| [1m --> TIME: 2025-12-21 12:15:47 -- STEP: 255/2207 -- GLOBAL_STEP: 16500[0m | |
| | > loss_text_ce: 0.03305845335125923 (0.030955966971084183) | |
| | > loss_mel_ce: 3.9918406009674072 (3.8380968393064014) | |
| | > loss: 0.06288904696702957 (0.06045395013164071) | |
| | > grad_norm: 0 (0.0) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2862 (0.2800759567933923) | |
| | > loader_time: 0.0086 (0.009221519208421897) | |
| [1m --> TIME: 2025-12-21 12:16:06 -- STEP: 305/2207 -- GLOBAL_STEP: 16550[0m | |
| | > loss_text_ce: 0.031405024230480194 (0.030954475768032622) | |
| | > loss_mel_ce: 3.559880495071411 (3.8379844337213234) | |
| | > loss: 0.0561138354241848 (0.06045217049903557) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2679 (0.2793264326502066) | |
| | > loader_time: 0.0081 (0.00907517104852395) | |
| [1m --> TIME: 2025-12-21 12:16:25 -- STEP: 355/2207 -- GLOBAL_STEP: 16600[0m | |
| | > loss_text_ce: 0.03148230165243149 (0.03097983851084407) | |
| | > loss_mel_ce: 3.8917019367218018 (3.8419334841446138) | |
| | > loss: 0.061299752444028854 (0.06051427065066888) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2688 (0.27979512684781777) | |
| | > loader_time: 0.0079 (0.008994941980066435) | |
| [1m --> TIME: 2025-12-21 12:16:45 -- STEP: 405/2207 -- GLOBAL_STEP: 16650[0m | |
| | > loss_text_ce: 0.030755072832107544 (0.03096303568210131) | |
| | > loss_mel_ce: 3.5810444355010986 (3.837673578733279) | |
| | > loss: 0.05643436685204506 (0.060447447075520036) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2877 (0.27977862004880527) | |
| | > loader_time: 0.0082 (0.008914067421430426) | |
| [1m --> TIME: 2025-12-21 12:17:04 -- STEP: 455/2207 -- GLOBAL_STEP: 16700[0m | |
| | > loss_text_ce: 0.032188721001148224 (0.030962336661068946) | |
| | > loss_mel_ce: 3.918936252593994 (3.8396105085100447) | |
| | > loss: 0.0617363266646862 (0.06047770069031925) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2866 (0.27930216998844304) | |
| | > loader_time: 0.0084 (0.008828892026628768) | |
| [1m --> TIME: 2025-12-21 12:17:23 -- STEP: 505/2207 -- GLOBAL_STEP: 16750[0m | |
| | > loss_text_ce: 0.0326453261077404 (0.03099347592359132) | |
| | > loss_mel_ce: 3.800915241241455 (3.8388911927100455) | |
| | > loss: 0.059899382293224335 (0.06046694796716813) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2867 (0.27900417252342247) | |
| | > loader_time: 0.0081 (0.008783628916976477) | |
| [1m --> TIME: 2025-12-21 12:17:42 -- STEP: 555/2207 -- GLOBAL_STEP: 16800[0m | |
| | > loss_text_ce: 0.030252568423748016 (0.030994508647032685) | |
| | > loss_mel_ce: 3.620875597000122 (3.8386023968189686) | |
| | > loss: 0.05704887583851814 (0.06046245165907585) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2862 (0.2790829800270695) | |
| | > loader_time: 0.0077 (0.008722289832862648) | |
| [1m --> TIME: 2025-12-21 12:18:01 -- STEP: 605/2207 -- GLOBAL_STEP: 16850[0m | |
| | > loss_text_ce: 0.03053409792482853 (0.030986474165000208) | |
| | > loss_mel_ce: 4.023441791534424 (3.8384223346867836) | |
| | > loss: 0.06334337592124939 (0.060459512675350366) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2666 (0.2788679410603417) | |
| | > loader_time: 0.0078 (0.008667540353191784) | |
| [1m --> TIME: 2025-12-21 12:18:20 -- STEP: 655/2207 -- GLOBAL_STEP: 16900[0m | |
| | > loss_text_ce: 0.030561409890651703 (0.03099969866337212) | |
| | > loss_mel_ce: 3.8719499111175537 (3.8380026737242257) | |
| | > loss: 0.06097673997282982 (0.060453162103436375) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2663 (0.27882370475594254) | |
| | > loader_time: 0.0089 (0.008689285962636233) | |
| [1m --> TIME: 2025-12-21 12:18:39 -- STEP: 705/2207 -- GLOBAL_STEP: 16950[0m | |
| | > loss_text_ce: 0.031610529869794846 (0.03098056488637383) | |
| | > loss_mel_ce: 3.83477520942688 (3.838375516647988) | |
| | > loss: 0.06041227653622627 (0.06045868881839387) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.291 (0.27892449933586333) | |
| | > loader_time: 0.0082 (0.008669162642025784) | |
| [1m --> TIME: 2025-12-21 12:18:58 -- STEP: 755/2207 -- GLOBAL_STEP: 17000[0m | |
| | > loss_text_ce: 0.03250373899936676 (0.030986027972192954) | |
| | > loss_mel_ce: 3.5191447734832764 (3.835920715332031) | |
| | > loss: 0.055494509637355804 (0.060420417894196034) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2686 (0.27887622504834264) | |
| | > loader_time: 0.0083 (0.00863919352853536) | |
| > CHECKPOINT : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/checkpoint_17000.pth | |
| [1m --> TIME: 2025-12-21 12:19:41 -- STEP: 805/2207 -- GLOBAL_STEP: 17050[0m | |
| | > loss_text_ce: 0.02891026996076107 (0.030961797997262907) | |
| | > loss_mel_ce: 3.8255555629730225 (3.8371545169664465) | |
| | > loss: 0.06022602692246437 (0.060439317442060256) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2688 (0.2789103673852005) | |
| | > loader_time: 0.0077 (0.008619996183407246) | |
| [1m --> TIME: 2025-12-21 12:20:00 -- STEP: 855/2207 -- GLOBAL_STEP: 17100[0m | |
| | > loss_text_ce: 0.030247919261455536 (0.03095182662949576) | |
| | > loss_mel_ce: 3.7221128940582275 (3.8346250587039523) | |
| | > loss: 0.05863063782453537 (0.06039963886577483) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2612 (0.2788665866294099) | |
| | > loader_time: 0.0085 (0.008600194272939227) | |
| [1m --> TIME: 2025-12-21 12:20:20 -- STEP: 905/2207 -- GLOBAL_STEP: 17150[0m | |
| | > loss_text_ce: 0.03263920545578003 (0.030970429752577733) | |
| | > loss_mel_ce: 3.8033525943756104 (3.8346228056849694) | |
| | > loss: 0.05993737280368805 (0.06039989432988904) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2658 (0.2798094433315545) | |
| | > loader_time: 0.0085 (0.008578953031676908) | |
| [1m --> TIME: 2025-12-21 12:20:39 -- STEP: 955/2207 -- GLOBAL_STEP: 17200[0m | |
| | > loss_text_ce: 0.03272069990634918 (0.030973391158808587) | |
| | > loss_mel_ce: 3.866997718811035 (3.8342152485672716) | |
| | > loss: 0.060933101922273636 (0.06039357252848086) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2957 (0.2798374602931952) | |
| | > loader_time: 0.0083 (0.008563283850385253) | |
| [1m --> TIME: 2025-12-21 12:20:58 -- STEP: 1005/2207 -- GLOBAL_STEP: 17250[0m | |
| | > loss_text_ce: 0.03107460029423237 (0.03097570813191471) | |
| | > loss_mel_ce: 3.7447755336761475 (3.8325845606884554) | |
| | > loss: 0.058997657150030136 (0.06036812922551264) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2864 (0.279714151164192) | |
| | > loader_time: 0.0085 (0.008561972954973057) | |
| [1m --> TIME: 2025-12-21 12:21:17 -- STEP: 1055/2207 -- GLOBAL_STEP: 17300[0m | |
| | > loss_text_ce: 0.030682062730193138 (0.030976020410586308) | |
| | > loss_mel_ce: 3.643967866897583 (3.8322222506265504) | |
| | > loss: 0.0574164055287838 (0.06036247299957615) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2883 (0.27947213502856744) | |
| | > loader_time: 0.0083 (0.008548689340528149) | |
| [1m --> TIME: 2025-12-21 12:21:36 -- STEP: 1105/2207 -- GLOBAL_STEP: 17350[0m | |
| | > loss_text_ce: 0.032925937324762344 (0.03097810615106945) | |
| | > loss_mel_ce: 3.7915194034576416 (3.8325317693511827) | |
| | > loss: 0.05975695699453354 (0.060367341838541075) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2868 (0.27930227965790716) | |
| | > loader_time: 0.0077 (0.008524640760810121) | |
| [1m --> TIME: 2025-12-21 12:21:55 -- STEP: 1155/2207 -- GLOBAL_STEP: 17400[0m | |
| | > loss_text_ce: 0.03101305104792118 (0.03097473747576728) | |
| | > loss_mel_ce: 3.926995038986206 (3.830362285473646) | |
| | > loss: 0.0618438757956028 (0.060333391017728034) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2953 (0.27925692182598655) | |
| | > loader_time: 0.0081 (0.00850323412841532) | |
| [1m --> TIME: 2025-12-21 12:22:15 -- STEP: 1205/2207 -- GLOBAL_STEP: 17450[0m | |
| | > loss_text_ce: 0.031288035213947296 (0.030977744015544283) | |
| | > loss_mel_ce: 4.4484028816223145 (3.831367015838623) | |
| | > loss: 0.06999517232179642 (0.060349136900110366) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2619 (0.27930984556427596) | |
| | > loader_time: 0.0081 (0.008497552159416222) | |
| [1m --> TIME: 2025-12-21 12:22:34 -- STEP: 1255/2207 -- GLOBAL_STEP: 17500[0m | |
| | > loss_text_ce: 0.029901018366217613 (0.030980763323159806) | |
| | > loss_mel_ce: 3.783792734146118 (3.83137526474151) | |
| | > loss: 0.0595889650285244 (0.06034931295599596) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.302 (0.2792716653223529) | |
| | > loader_time: 0.0089 (0.008480142122245867) | |
| [1m --> TIME: 2025-12-21 12:22:53 -- STEP: 1305/2207 -- GLOBAL_STEP: 17550[0m | |
| | > loss_text_ce: 0.030708273872733116 (0.030976130990347187) | |
| | > loss_mel_ce: 3.928159236907959 (3.831721293606521) | |
| | > loss: 0.061857305467128754 (0.06035464728015593) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2912 (0.2792421129014756) | |
| | > loader_time: 0.008 (0.008464955735480633) | |
| [1m --> TIME: 2025-12-21 12:23:12 -- STEP: 1355/2207 -- GLOBAL_STEP: 17600[0m | |
| | > loss_text_ce: 0.030456705018877983 (0.030980337833181517) | |
| | > loss_mel_ce: 3.610086441040039 (3.8301265816846897) | |
| | > loss: 0.05688348785042763 (0.06032979563428467) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.289 (0.27909282828609) | |
| | > loader_time: 0.0084 (0.008450167117523512) | |
| [1m --> TIME: 2025-12-21 12:23:31 -- STEP: 1405/2207 -- GLOBAL_STEP: 17650[0m | |
| | > loss_text_ce: 0.0315256342291832 (0.03096651314787593) | |
| | > loss_mel_ce: 3.5720529556274414 (3.8309017710838456) | |
| | > loss: 0.056305915117263794 (0.06034169195756794) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2542 (0.27900979221927735) | |
| | > loader_time: 0.0081 (0.00843814344168556) | |
| [1m --> TIME: 2025-12-21 12:23:50 -- STEP: 1455/2207 -- GLOBAL_STEP: 17700[0m | |
| | > loss_text_ce: 0.029983242973685265 (0.030972939362282195) | |
| | > loss_mel_ce: 3.7503371238708496 (3.830120104366971) | |
| | > loss: 0.0590675063431263 (0.0603295788296924) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2862 (0.279018137299318) | |
| | > loader_time: 0.0086 (0.008426600551277487) | |
| [1m --> TIME: 2025-12-21 12:24:09 -- STEP: 1505/2207 -- GLOBAL_STEP: 17750[0m | |
| | > loss_text_ce: 0.03367031365633011 (0.0309653219120249) | |
| | > loss_mel_ce: 3.723371744155884 (3.830078932613233) | |
| | > loss: 0.05870378389954567 (0.06032881649625658) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2685 (0.27902523592064943) | |
| | > loader_time: 0.0079 (0.00841875092135711) | |
| [1m --> TIME: 2025-12-21 12:24:28 -- STEP: 1555/2207 -- GLOBAL_STEP: 17800[0m | |
| | > loss_text_ce: 0.02944960817694664 (0.03096569205403711) | |
| | > loss_mel_ce: 4.062541961669922 (3.8309391785281264) | |
| | > loss: 0.06393736600875854 (0.060342263636385896) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2902 (0.2789708040918187) | |
| | > loader_time: 0.0081 (0.008411988568075979) | |
| [1m --> TIME: 2025-12-21 12:24:47 -- STEP: 1605/2207 -- GLOBAL_STEP: 17850[0m | |
| | > loss_text_ce: 0.029368538409471512 (0.03096987812481008) | |
| | > loss_mel_ce: 3.616783857345581 (3.8293144246870856) | |
| | > loss: 0.05697113275527954 (0.060316942265471934) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2896 (0.27894843597649793) | |
| | > loader_time: 0.0077 (0.008404218667764156) | |
| [1m --> TIME: 2025-12-21 12:25:06 -- STEP: 1655/2207 -- GLOBAL_STEP: 17900[0m | |
| | > loss_text_ce: 0.02917139232158661 (0.030966688455096543) | |
| | > loss_mel_ce: 3.955526351928711 (3.8295799357653384) | |
| | > loss: 0.06226090341806412 (0.06032104103676862) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2508 (0.2788965385125842) | |
| | > loader_time: 0.008 (0.008392788348241155) | |
| [1m --> TIME: 2025-12-21 12:25:25 -- STEP: 1705/2207 -- GLOBAL_STEP: 17950[0m | |
| | > loss_text_ce: 0.036614060401916504 (0.030967657537885076) | |
| | > loss_mel_ce: 3.92380690574646 (3.8291027352956735) | |
| | > loss: 0.06188157945871353 (0.06031359991938138) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.4298 (0.2789531161009043) | |
| | > loader_time: 0.0081 (0.008383412235293566) | |
| [1m --> TIME: 2025-12-21 12:25:44 -- STEP: 1755/2207 -- GLOBAL_STEP: 18000[0m | |
| | > loss_text_ce: 0.031244533136487007 (0.030965659205942412) | |
| | > loss_mel_ce: 3.8533778190612793 (3.8291114841091667) | |
| | > loss: 0.060697223991155624 (0.06031370539525975) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2861 (0.2788446566318174) | |
| | > loader_time: 0.0081 (0.008380908911724024) | |
| > CHECKPOINT : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/checkpoint_18000.pth | |
| [1m --> TIME: 2025-12-21 12:26:27 -- STEP: 1805/2207 -- GLOBAL_STEP: 18050[0m | |
| | > loss_text_ce: 0.02912035584449768 (0.03096764023696947) | |
| | > loss_mel_ce: 3.675009250640869 (3.8278143662164745) | |
| | > loss: 0.05787702649831772 (0.060293468884566485) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2666 (0.2788151725177289) | |
| | > loader_time: 0.0083 (0.008385370840986658) | |
| [1m --> TIME: 2025-12-21 12:26:46 -- STEP: 1855/2207 -- GLOBAL_STEP: 18100[0m | |
| | > loss_text_ce: 0.030438384041190147 (0.030971936526083238) | |
| | > loss_mel_ce: 3.6038551330566406 (3.827252550587821) | |
| | > loss: 0.056785836815834045 (0.06028475765028411) | |
| | > grad_norm: 0 (0.0) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.3188 (0.2787536723915779) | |
| | > loader_time: 0.0083 (0.00838110569031089) | |
| [1m --> TIME: 2025-12-21 12:27:05 -- STEP: 1905/2207 -- GLOBAL_STEP: 18150[0m | |
| | > loss_text_ce: 0.029724599793553352 (0.03097888275668064) | |
| | > loss_mel_ce: 3.7439305782318115 (3.827816226914173) | |
| | > loss: 0.058963362127542496 (0.0602936736300705) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2651 (0.27881914574330247) | |
| | > loader_time: 0.0086 (0.008379007574767254) | |
| [1m --> TIME: 2025-12-21 12:27:24 -- STEP: 1955/2207 -- GLOBAL_STEP: 18200[0m | |
| | > loss_text_ce: 0.030410582199692726 (0.03097874500200419) | |
| | > loss_mel_ce: 3.681417226791382 (3.826422531708427) | |
| | > loss: 0.05799730867147446 (0.06027189498133671) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2678 (0.2787622943253773) | |
| | > loader_time: 0.0085 (0.008377984844510192) | |
| [1m --> TIME: 2025-12-21 12:27:43 -- STEP: 2005/2207 -- GLOBAL_STEP: 18250[0m | |
| | > loss_text_ce: 0.03071397729218006 (0.030976552677868013) | |
| | > loss_mel_ce: 3.844003438949585 (3.825914057829136) | |
| | > loss: 0.06054246053099632 (0.06026391582447394) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2667 (0.2787343830241824) | |
| | > loader_time: 0.0081 (0.008377488533457602) | |
| [1m --> TIME: 2025-12-21 12:28:02 -- STEP: 2055/2207 -- GLOBAL_STEP: 18300[0m | |
| | > loss_text_ce: 0.03158266097307205 (0.030973146271205298) | |
| | > loss_mel_ce: 3.630298376083374 (3.8251054886773845) | |
| | > loss: 0.057216890156269073 (0.06025122870348956) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2654 (0.27866049694608885) | |
| | > loader_time: 0.009 (0.00838099222113615) | |
| [1m --> TIME: 2025-12-21 12:28:22 -- STEP: 2105/2207 -- GLOBAL_STEP: 18350[0m | |
| | > loss_text_ce: 0.029528765007853508 (0.030974545680216542) | |
| | > loss_mel_ce: 3.808253765106201 (3.8251185174792504) | |
| | > loss: 0.0599653534591198 (0.06025145414535336) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2895 (0.2786942468403071) | |
| | > loader_time: 0.0084 (0.008381317573601716) | |
| [1m --> TIME: 2025-12-21 12:28:41 -- STEP: 2155/2207 -- GLOBAL_STEP: 18400[0m | |
| | > loss_text_ce: 0.03116406314074993 (0.03098027495334569) | |
| | > loss_mel_ce: 3.9434666633605957 (3.8245020461469683) | |
| | > loss: 0.06210360676050186 (0.060241911300996064) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2905 (0.27876709453467685) | |
| | > loader_time: 0.0083 (0.008375044377939872) | |
| [1m --> TIME: 2025-12-21 12:29:00 -- STEP: 2205/2207 -- GLOBAL_STEP: 18450[0m | |
| | > loss_text_ce: 0.03149457648396492 (0.030984393438818505) | |
| | > loss_mel_ce: 3.63926362991333 (3.8233453620858744) | |
| | > loss: 0.057355597615242004 (0.06022390246289928) | |
| | > current_lr: 1e-05 | |
| | > step_time: 0.2661 (0.27879359316663666) | |
| | > loader_time: 0.0079 (0.008370566530292528) | |
| [1m > EVALUATION [0m | |
| [1m--> EVAL PERFORMANCE[0m | |
| | > avg_loader_time:[92m 0.07265385904827633 [0m(-0.0017059896443341466) | |
| | > avg_loss_text_ce:[92m 0.02991473435650806 [0m(-0.00016766787833861346) | |
| | > avg_loss_mel_ce:[92m 3.808113165803858 [0m(-0.014583056037490216) | |
| | > avg_loss:[92m 3.8380279089953446 [0m(-0.014750712626689477) | |
| > BEST MODEL : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/best_model_18452.pth | |
| [4m[1m > EPOCH: 2/6[0m | |
| --> /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000 | |
| [1m > TRAINING (2025-12-21 12:33:37) [0m | |
| [1m --> TIME: 2025-12-21 12:33:58 -- STEP: 48/2207 -- GLOBAL_STEP: 18500[0m | |
| | > loss_text_ce: 0.030328935012221336 (0.03124918350173782) | |
| | > loss_mel_ce: 3.6012001037597656 (3.80705534418424) | |
| | > loss: 0.05674264207482338 (0.05997350811958313) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2685 (0.2786668340365091) | |
| | > loader_time: 0.0078 (0.008490150173505148) | |
| [1m --> TIME: 2025-12-21 12:34:17 -- STEP: 98/2207 -- GLOBAL_STEP: 18550[0m | |
| | > loss_text_ce: 0.029475053772330284 (0.031117105039254748) | |
| | > loss_mel_ce: 3.6500818729400635 (3.802043304151418) | |
| | > loss: 0.05749307572841644 (0.059893131370143016) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.266 (0.27715023439757674) | |
| | > loader_time: 0.0085 (0.008422819935545629) | |
| [1m --> TIME: 2025-12-21 12:34:36 -- STEP: 148/2207 -- GLOBAL_STEP: 18600[0m | |
| | > loss_text_ce: 0.029837869107723236 (0.03100120587073065) | |
| | > loss_mel_ce: 4.100630760192871 (3.7966143134477974) | |
| | > loss: 0.06453857570886612 (0.05980649250685363) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.3043 (0.2776310814393531) | |
| | > loader_time: 0.0081 (0.00839655947040867) | |
| [1m --> TIME: 2025-12-21 12:34:55 -- STEP: 198/2207 -- GLOBAL_STEP: 18650[0m | |
| | > loss_text_ce: 0.031300414353609085 (0.03097762127943111) | |
| | > loss_mel_ce: 4.068465709686279 (3.808431427888196) | |
| | > loss: 0.06405884772539139 (0.05999076642999143) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2677 (0.27762243723628477) | |
| | > loader_time: 0.0077 (0.008417524472631586) | |
| [1m --> TIME: 2025-12-21 12:35:14 -- STEP: 248/2207 -- GLOBAL_STEP: 18700[0m | |
| | > loss_text_ce: 0.030506588518619537 (0.030958888443907904) | |
| | > loss_mel_ce: 3.6368277072906494 (3.8152807095358447) | |
| | > loss: 0.05730209872126579 (0.060097493787085815) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2867 (0.2775823743112623) | |
| | > loader_time: 0.008 (0.00839974322626667) | |
| [1m --> TIME: 2025-12-21 12:35:33 -- STEP: 298/2207 -- GLOBAL_STEP: 18750[0m | |
| | > loss_text_ce: 0.03192504867911339 (0.030931028593706604) | |
| | > loss_mel_ce: 3.560392141342163 (3.812415851842637) | |
| | > loss: 0.056129954755306244 (0.06005229498415986) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2872 (0.2775322710907697) | |
| | > loader_time: 0.0082 (0.008380010624059893) | |
| [1m --> TIME: 2025-12-21 12:35:53 -- STEP: 348/2207 -- GLOBAL_STEP: 18800[0m | |
| | > loss_text_ce: 0.030473029240965843 (0.030946150266875822) | |
| | > loss_mel_ce: 4.083008766174316 (3.807196080684662) | |
| | > loss: 0.0642731562256813 (0.059970972342309595) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2903 (0.27775018790672557) | |
| | > loader_time: 0.0083 (0.008361554008790817) | |
| [1m --> TIME: 2025-12-21 12:36:12 -- STEP: 398/2207 -- GLOBAL_STEP: 18850[0m | |
| | > loss_text_ce: 0.032361965626478195 (0.03095491271203936) | |
| | > loss_mel_ce: 3.980205774307251 (3.8102012603127178) | |
| | > loss: 0.06269636750221252 (0.060018065156199826) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.3053 (0.2776316464246817) | |
| | > loader_time: 0.0083 (0.008362287852033301) | |
| [1m --> TIME: 2025-12-21 12:36:31 -- STEP: 448/2207 -- GLOBAL_STEP: 18900[0m | |
| | > loss_text_ce: 0.03188605234026909 (0.030948849365813658) | |
| | > loss_mel_ce: 3.859243869781494 (3.8156256579927037) | |
| | > loss: 0.06079890578985214 (0.06010272662388161) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2417 (0.27759859870587084) | |
| | > loader_time: 0.0083 (0.008384346429790766) | |
| [1m --> TIME: 2025-12-21 12:36:50 -- STEP: 498/2207 -- GLOBAL_STEP: 18950[0m | |
| | > loss_text_ce: 0.03379356116056442 (0.03096674965043743) | |
| | > loss_mel_ce: 3.9159152507781982 (3.8173977414288194) | |
| | > loss: 0.06171419844031334 (0.06013069510549666) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2908 (0.2774093696870001) | |
| | > loader_time: 0.0084 (0.008369181529585133) | |
| [1m --> TIME: 2025-12-21 12:37:09 -- STEP: 548/2207 -- GLOBAL_STEP: 19000[0m | |
| | > loss_text_ce: 0.031926073133945465 (0.030988967725939124) | |
| | > loss_mel_ce: 3.838353395462036 (3.8168183700011586) | |
| | > loss: 0.06047311797738075 (0.0601219896050374) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2895 (0.27765954236914653) | |
| | > loader_time: 0.0084 (0.008360996733616734) | |
| > CHECKPOINT : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/checkpoint_19000.pth | |
| [1m --> TIME: 2025-12-21 12:37:53 -- STEP: 598/2207 -- GLOBAL_STEP: 19050[0m | |
| | > loss_text_ce: 0.031638141721487045 (0.030977829051107467) | |
| | > loss_mel_ce: 3.594619035720825 (3.8176495937200685) | |
| | > loss: 0.05666026845574379 (0.06013480343398043) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2616 (0.2777918074840685) | |
| | > loader_time: 0.0082 (0.008385346884711521) | |
| [1m --> TIME: 2025-12-21 12:38:12 -- STEP: 648/2207 -- GLOBAL_STEP: 19100[0m | |
| | > loss_text_ce: 0.029235929250717163 (0.03097168711749952) | |
| | > loss_mel_ce: 3.780961513519287 (3.8170210332782175) | |
| | > loss: 0.059534333646297455 (0.060124886210685895) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2874 (0.2776544944003776) | |
| | > loader_time: 0.009 (0.008395500757076114) | |
| [1m --> TIME: 2025-12-21 12:38:31 -- STEP: 698/2207 -- GLOBAL_STEP: 19150[0m | |
| | > loss_text_ce: 0.028909575194120407 (0.03093947344232767) | |
| | > loss_mel_ce: 3.942351818084717 (3.8171628662371697) | |
| | > loss: 0.062050960958004 (0.0601265990097919) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2901 (0.2778345213236986) | |
| | > loader_time: 0.0088 (0.008406749085915455) | |
| [1m --> TIME: 2025-12-21 12:38:50 -- STEP: 748/2207 -- GLOBAL_STEP: 19200[0m | |
| | > loss_text_ce: 0.031155679374933243 (0.030945500974010498) | |
| | > loss_mel_ce: 3.9143738746643066 (3.816780863917447) | |
| | > loss: 0.0616488978266716 (0.060120724395594814) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2689 (0.27826202616972096) | |
| | > loader_time: 0.0081 (0.008410301437989907) | |
| [1m --> TIME: 2025-12-21 12:39:09 -- STEP: 798/2207 -- GLOBAL_STEP: 19250[0m | |
| | > loss_text_ce: 0.029925458133220673 (0.030928961321114) | |
| | > loss_mel_ce: 3.9727892875671387 (3.8165484224644515) | |
| | > loss: 0.06254241615533829 (0.06011683405622056) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2659 (0.2782398151574578) | |
| | > loader_time: 0.0083 (0.00841741394578365) | |
| [1m --> TIME: 2025-12-21 12:39:29 -- STEP: 848/2207 -- GLOBAL_STEP: 19300[0m | |
| | > loss_text_ce: 0.0316762737929821 (0.030928653658297407) | |
| | > loss_mel_ce: 3.722557783126831 (3.8153149249418723) | |
| | > loss: 0.058659907430410385 (0.06009755585395362) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2698 (0.2783565099509257) | |
| | > loader_time: 0.008 (0.008425706118907571) | |
| [1m --> TIME: 2025-12-21 12:39:48 -- STEP: 898/2207 -- GLOBAL_STEP: 19350[0m | |
| | > loss_text_ce: 0.03213763236999512 (0.03091699258322705) | |
| | > loss_mel_ce: 3.923859119415283 (3.8141845504531346) | |
| | > loss: 0.061812449246644974 (0.06007971154538321) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2666 (0.2783797982009853) | |
| | > loader_time: 0.0083 (0.008426021626903646) | |
| [1m --> TIME: 2025-12-21 12:40:07 -- STEP: 948/2207 -- GLOBAL_STEP: 19400[0m | |
| | > loss_text_ce: 0.032980721443891525 (0.030903644741901867) | |
| | > loss_mel_ce: 3.861752986907959 (3.8123343091473796) | |
| | > loss: 0.060855213552713394 (0.060050592962102294) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2669 (0.2783489184540535) | |
| | > loader_time: 0.0079 (0.008424995317740783) | |
| [1m --> TIME: 2025-12-21 12:40:26 -- STEP: 998/2207 -- GLOBAL_STEP: 19450[0m | |
| | > loss_text_ce: 0.03054073452949524 (0.03090524785071313) | |
| | > loss_mel_ce: 4.015505313873291 (3.8130178864828803) | |
| | > loss: 0.06321947276592255 (0.06006129889679934) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2901 (0.2784373222706552) | |
| | > loader_time: 0.0091 (0.008492851305103494) | |
| [1m --> TIME: 2025-12-21 12:40:46 -- STEP: 1048/2207 -- GLOBAL_STEP: 19500[0m | |
| | > loss_text_ce: 0.030669590458273888 (0.030904853401904683) | |
| | > loss_mel_ce: 3.722041606903076 (3.8116825097844798) | |
| | > loss: 0.05863611400127411 (0.060040427482054216) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.315 (0.27876375604221837) | |
| | > loader_time: 0.0088 (0.008539806567985593) | |
| [1m --> TIME: 2025-12-21 12:41:05 -- STEP: 1098/2207 -- GLOBAL_STEP: 19550[0m | |
| | > loss_text_ce: 0.03170514851808548 (0.0308953984343669) | |
| | > loss_mel_ce: 3.862687349319458 (3.8120278489177126) | |
| | > loss: 0.060849882662296295 (0.06004567566465159) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2922 (0.27895870430221975) | |
| | > loader_time: 0.0111 (0.00858209975647362) | |
| [1m --> TIME: 2025-12-21 12:41:25 -- STEP: 1148/2207 -- GLOBAL_STEP: 19600[0m | |
| | > loss_text_ce: 0.031861841678619385 (0.030892794122692288) | |
| | > loss_mel_ce: 3.8248915672302246 (3.811262029596322) | |
| | > loss: 0.06026177108287811 (0.06003366905274291) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2674 (0.27946521783124284) | |
| | > loader_time: 0.0098 (0.00863001051680136) | |
| [1m --> TIME: 2025-12-21 12:41:44 -- STEP: 1198/2207 -- GLOBAL_STEP: 19650[0m | |
| | > loss_text_ce: 0.031165527179837227 (0.030906598595202665) | |
| | > loss_mel_ce: 3.6397016048431396 (3.811939091037629) | |
| | > loss: 0.0573573000729084 (0.06004446385139117) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2662 (0.27954068605808274) | |
| | > loader_time: 0.0096 (0.00866090156797176) | |
| [1m --> TIME: 2025-12-21 12:42:04 -- STEP: 1248/2207 -- GLOBAL_STEP: 19700[0m | |
| | > loss_text_ce: 0.030565952882170677 (0.030907038918648582) | |
| | > loss_mel_ce: 4.019072532653809 (3.812627971554414) | |
| | > loss: 0.06327559798955917 (0.0600552344843984) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2681 (0.27955424575469423) | |
| | > loader_time: 0.0098 (0.008690576713818758) | |
| [1m --> TIME: 2025-12-21 12:42:23 -- STEP: 1298/2207 -- GLOBAL_STEP: 19750[0m | |
| | > loss_text_ce: 0.033700186759233475 (0.030912928441065853) | |
| | > loss_mel_ce: 3.9421839714050293 (3.8117501079944325) | |
| | > loss: 0.062123190611600876 (0.060041609896097234) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2703 (0.2795349096847794) | |
| | > loader_time: 0.0094 (0.008714687842617403) | |
| [1m --> TIME: 2025-12-21 12:42:42 -- STEP: 1348/2207 -- GLOBAL_STEP: 19800[0m | |
| | > loss_text_ce: 0.031137095764279366 (0.030893902244593516) | |
| | > loss_mel_ce: 4.025448799133301 (3.8119540738068984) | |
| | > loss: 0.06338415294885635 (0.06004449958948955) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2629 (0.27941010755318224) | |
| | > loader_time: 0.0101 (0.008745452770493491) | |
| [1m --> TIME: 2025-12-21 12:43:01 -- STEP: 1398/2207 -- GLOBAL_STEP: 19850[0m | |
| | > loss_text_ce: 0.031674209982156754 (0.030885134482535083) | |
| | > loss_mel_ce: 3.708361864089966 (3.8112786298828234) | |
| | > loss: 0.05843806266784668 (0.06003380877009789) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2671 (0.2793427293051639) | |
| | > loader_time: 0.0114 (0.008763332223687543) | |
| [1m --> TIME: 2025-12-21 12:43:20 -- STEP: 1448/2207 -- GLOBAL_STEP: 19900[0m | |
| | > loss_text_ce: 0.030328992754220963 (0.03087736206329865) | |
| | > loss_mel_ce: 3.821791172027588 (3.8124404337195403) | |
| | > loss: 0.0601893775165081 (0.06005184050875803) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.269 (0.27929688224476384) | |
| | > loader_time: 0.0094 (0.008786394615858279) | |
| [1m --> TIME: 2025-12-21 12:43:39 -- STEP: 1498/2207 -- GLOBAL_STEP: 19950[0m | |
| | > loss_text_ce: 0.03165443241596222 (0.03087984636619946) | |
| | > loss_mel_ce: 3.698460578918457 (3.8117194285539187) | |
| | > loss: 0.05828304588794708 (0.06004061362394026) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.266 (0.2792639877194562) | |
| | > loader_time: 0.0098 (0.008800205305835413) | |
| [1m --> TIME: 2025-12-21 12:43:59 -- STEP: 1548/2207 -- GLOBAL_STEP: 20000[0m | |
| | > loss_text_ce: 0.033760420978069305 (0.030875577592562875) | |
| | > loss_mel_ce: 3.544578790664673 (3.8111829480459525) | |
| | > loss: 0.05591154843568802 (0.06003216441085403) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2706 (0.27927306163526666) | |
| | > loader_time: 0.0093 (0.008809470237071489) | |
| > CHECKPOINT : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/checkpoint_20000.pth | |
| [1m --> TIME: 2025-12-21 12:44:43 -- STEP: 1598/2207 -- GLOBAL_STEP: 20050[0m | |
| | > loss_text_ce: 0.031220965087413788 (0.03087228273644167) | |
| | > loss_mel_ce: 4.057143688201904 (3.811688291116411) | |
| | > loss: 0.06388069689273834 (0.06004000892011782) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.298 (0.27934688292397175) | |
| | > loader_time: 0.0089 (0.008850628055529532) | |
| [1m --> TIME: 2025-12-21 12:45:02 -- STEP: 1648/2207 -- GLOBAL_STEP: 20100[0m | |
| | > loss_text_ce: 0.03015831671655178 (0.030872867847295496) | |
| | > loss_mel_ce: 3.5962467193603516 (3.811391503342147) | |
| | > loss: 0.056662578135728836 (0.060035380760876875) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2671 (0.27928056546206664) | |
| | > loader_time: 0.0087 (0.008861764252764507) | |
| [1m --> TIME: 2025-12-21 12:45:21 -- STEP: 1698/2207 -- GLOBAL_STEP: 20150[0m | |
| | > loss_text_ce: 0.029446423053741455 (0.030872317737470806) | |
| | > loss_mel_ce: 3.346419334411621 (3.8103991141729274) | |
| | > loss: 0.052747901529073715 (0.06001986607742464) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2872 (0.27919970594952087) | |
| | > loader_time: 0.0084 (0.008857855808046877) | |
| [1m --> TIME: 2025-12-21 12:45:40 -- STEP: 1748/2207 -- GLOBAL_STEP: 20200[0m | |
| | > loss_text_ce: 0.031714435666799545 (0.030864485928435006) | |
| | > loss_mel_ce: 3.784182071685791 (3.809945785480997) | |
| | > loss: 0.05962338298559189 (0.060012660449630745) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2625 (0.2791893772729752) | |
| | > loader_time: 0.0087 (0.008868558728722192) | |
| [1m --> TIME: 2025-12-21 12:46:00 -- STEP: 1798/2207 -- GLOBAL_STEP: 20250[0m | |
| | > loss_text_ce: 0.030244555324316025 (0.030862935044874206) | |
| | > loss_mel_ce: 3.833282232284546 (3.8102427565879102) | |
| | > loss: 0.060367606580257416 (0.06001727638954713) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2691 (0.27920447560120526) | |
| | > loader_time: 0.0092 (0.008864384604508141) | |
| [1m --> TIME: 2025-12-21 12:46:19 -- STEP: 1848/2207 -- GLOBAL_STEP: 20300[0m | |
| | > loss_text_ce: 0.030655691400170326 (0.03086753052788347) | |
| | > loss_mel_ce: 3.7304117679595947 (3.8096691434517567) | |
| | > loss: 0.0587666779756546 (0.06000838548477207) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2672 (0.2791787206610565) | |
| | > loader_time: 0.0095 (0.008876425621313447) | |
| [1m --> TIME: 2025-12-21 12:46:38 -- STEP: 1898/2207 -- GLOBAL_STEP: 20350[0m | |
| | > loss_text_ce: 0.029659613966941833 (0.030867560519331438) | |
| | > loss_mel_ce: 3.8518874645233154 (3.80958583947856) | |
| | > loss: 0.060649171471595764 (0.060007084327876506) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2905 (0.2792153088385744) | |
| | > loader_time: 0.0089 (0.008891735865770824) | |
| [1m --> TIME: 2025-12-21 12:46:57 -- STEP: 1948/2207 -- GLOBAL_STEP: 20400[0m | |
| | > loss_text_ce: 0.029927784577012062 (0.03085980789154554) | |
| | > loss_mel_ce: 3.7252941131591797 (3.809625461116219) | |
| | > loss: 0.05867534130811691 (0.06000758227710545) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2695 (0.2792228915608154) | |
| | > loader_time: 0.0095 (0.008891950152982673) | |
| [1m --> TIME: 2025-12-21 12:47:17 -- STEP: 1998/2207 -- GLOBAL_STEP: 20450[0m | |
| | > loss_text_ce: 0.03319452330470085 (0.030859624606196408) | |
| | > loss_mel_ce: 3.891561508178711 (3.809653092194367) | |
| | > loss: 0.061324313282966614 (0.06000801114676831) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.266 (0.2792890162558651) | |
| | > loader_time: 0.0083 (0.00888882241807542) | |
| [1m --> TIME: 2025-12-21 12:47:36 -- STEP: 2048/2207 -- GLOBAL_STEP: 20500[0m | |
| | > loss_text_ce: 0.029439985752105713 (0.03084829578165227) | |
| | > loss_mel_ce: 3.90108585357666 (3.8098780608270317) | |
| | > loss: 0.06141446530818939 (0.060011349260093994) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.24 (0.2793646088102837) | |
| | > loader_time: 0.0088 (0.00888665369711816) | |
| [1m --> TIME: 2025-12-21 12:47:55 -- STEP: 2098/2207 -- GLOBAL_STEP: 20550[0m | |
| | > loss_text_ce: 0.028744757175445557 (0.03084498683738214) | |
| | > loss_mel_ce: 3.8752195835113525 (3.808677380828203) | |
| | > loss: 0.060999441891908646 (0.059992536937940454) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.2661 (0.2794211697192055) | |
| | > loader_time: 0.0095 (0.008883777405672462) | |
| [1m --> TIME: 2025-12-21 12:48:15 -- STEP: 2148/2207 -- GLOBAL_STEP: 20600[0m | |
| | > loss_text_ce: 0.032124441117048264 (0.030849136339302835) | |
| | > loss_mel_ce: 3.6190860271453857 (3.8092889245217965) | |
| | > loss: 0.05705016478896141 (0.06000215714897965) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.291 (0.2793659517884924) | |
| | > loader_time: 0.0096 (0.008893648450379042) | |
| [1m --> TIME: 2025-12-21 12:48:35 -- STEP: 2198/2207 -- GLOBAL_STEP: 20650[0m | |
| | > loss_text_ce: 0.03118201717734337 (0.030848595168817793) | |
| | > loss_mel_ce: 3.7164382934570312 (3.807630991914036) | |
| | > loss: 0.05855656787753105 (0.059976243487710926) | |
| | > current_lr: 9.5e-06 | |
| | > step_time: 0.3316 (0.2800430354473267) | |
| | > loader_time: 0.0084 (0.009042994340405454) | |
| [1m > EVALUATION [0m | |
| [1m--> EVAL PERFORMANCE[0m | |
| | > avg_loader_time:[91m 0.12325521095378979 [0m(+0.05060135190551346) | |
| | > avg_loss_text_ce:[92m 0.02977932435837952 [0m(-0.00013540999812854115) | |
| | > avg_loss_mel_ce:[92m 3.7917954229019784 [0m(-0.016317742901879484) | |
| | > avg_loss:[92m 3.821574736285854 [0m(-0.016453172709490538) | |
| > BEST MODEL : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/best_model_20659.pth | |