> Training Environment: | > Backend: Torch | > Mixed precision: False | > Precision: float32 | > Current device: 0 | > Num. of GPUs: 1 | > Num. of CPUs: 256 | > Num. of Torch Threads: 1 | > Torch seed: 1 | > Torch CUDNN: True | > Torch CUDNN deterministic: False | > Torch CUDNN benchmark: False | > Torch TF32 MatMul: False > Start Tensorboard: tensorboard --logdir=/workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-19-2025_11+20PM-0000000 > Restoring from best_model_5208.pth ... > Restoring Model... > Restoring Optimizer... > Model restored from step 5208 > Model has 514771807 parameters  > EPOCH: 0/6 --> /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-19-2025_11+20PM-0000000  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.07972378827430109 (+0) | > avg_loss_text_ce: 0.03184887157702769 (+0) | > avg_loss_mel_ce: 4.041015415578272 (+0) | > avg_loss: 4.072864298884932 (+0) > BEST MODEL : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-19-2025_11+20PM-0000000/best_model_5209.pth  > EPOCH: 1/6 --> /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-19-2025_11+20PM-0000000  > TRAINING (2025-12-19 23:23:38)   --> TIME: 2025-12-19 23:23:58 -- STEP: 41/2207 -- GLOBAL_STEP: 5250 | > loss_text_ce: 0.033330123871564865 (0.03260848189635974) | > loss_mel_ce: 4.138209342956543 (4.13933496358918) | > loss: 0.06518030166625977 (0.06518661657848009) | > current_lr: 1e-05 | > step_time: 0.2912 (0.2791317788566031) | > loader_time: 0.0083 (0.010119118341585485)  --> TIME: 2025-12-19 23:24:17 -- STEP: 91/2207 -- GLOBAL_STEP: 5300 | > loss_text_ce: 0.032737262547016144 (0.032768114295962075) | > loss_mel_ce: 3.9631412029266357 (4.132967893893904) | > loss: 0.06243560090661049 (0.065089625030101) | > current_lr: 1e-05 | > step_time: 0.2703 (0.28123985542045854) | > loader_time: 0.008 (0.009950027361020936)  --> TIME: 2025-12-19 23:24:37 -- STEP: 141/2207 -- GLOBAL_STEP: 5350 | > loss_text_ce: 0.03263312578201294 (0.03275274914682754) | > loss_mel_ce: 4.11068868637085 (4.126875228070198) | > loss: 0.0647394061088562 (0.06499418716692758) | > current_lr: 1e-05 | > step_time: 0.2669 (0.2816294220322413) | > loader_time: 0.0088 (0.009555863995924065)  --> TIME: 2025-12-19 23:24:56 -- STEP: 191/2207 -- GLOBAL_STEP: 5400 | > loss_text_ce: 0.033539168536663055 (0.032767798459030544) | > loss_mel_ce: 3.9542629718780518 (4.11662355642668) | > loss: 0.06230940669775009 (0.06483424000481042) | > grad_norm: 0 (0.0) | > current_lr: 1e-05 | > step_time: 0.31 (0.2820051752459942) | > loader_time: 0.008 (0.009320748414044603)  --> TIME: 2025-12-19 23:25:15 -- STEP: 241/2207 -- GLOBAL_STEP: 5450 | > loss_text_ce: 0.03252074122428894 (0.03277294945234578) | > loss_mel_ce: 4.288866996765137 (4.110062500253256) | > loss: 0.06752168387174606 (0.06473180396613738) | > current_lr: 1e-05 | > step_time: 0.2924 (0.2812330376557791) | > loader_time: 0.008 (0.009211289932124347)  --> TIME: 2025-12-19 23:25:34 -- STEP: 291/2207 -- GLOBAL_STEP: 5500 | > loss_text_ce: 0.0315721370279789 (0.03275071695139728) | > loss_mel_ce: 4.302829742431641 (4.108650724502771) | > loss: 0.0677250325679779 (0.06470939760253193) | > current_lr: 1e-05 | > step_time: 0.2699 (0.28055939395812796) | > loader_time: 0.0078 (0.009046153104592025)  --> TIME: 2025-12-19 23:25:54 -- STEP: 341/2207 -- GLOBAL_STEP: 5550 | > loss_text_ce: 0.03177058324217796 (0.03277641668631708) | > loss_mel_ce: 3.9036824703216553 (4.110928934690188) | > loss: 0.061491452157497406 (0.06474539622271169) | > current_lr: 1e-05 | > step_time: 0.2694 (0.2810395141500883) | > loader_time: 0.008 (0.008892246704996507)  --> TIME: 2025-12-19 23:26:13 -- STEP: 391/2207 -- GLOBAL_STEP: 5600 | > loss_text_ce: 0.031645581126213074 (0.0327805225901744) | > loss_mel_ce: 4.2700514793396 (4.109736498054638) | > loss: 0.06721401959657669 (0.06472682855699381) | > current_lr: 1e-05 | > step_time: 0.289 (0.2811980326767166) | > loader_time: 0.008 (0.00883669011733111)  --> TIME: 2025-12-19 23:26:32 -- STEP: 441/2207 -- GLOBAL_STEP: 5650 | > loss_text_ce: 0.031354326754808426 (0.03274994752371934) | > loss_mel_ce: 4.180124282836914 (4.105678926519797) | > loss: 0.06580435484647751 (0.06466295119890289) | > current_lr: 1e-05 | > step_time: 0.2701 (0.28069895249105103) | > loader_time: 0.0077 (0.008764371309691276)  --> TIME: 2025-12-19 23:26:51 -- STEP: 491/2207 -- GLOBAL_STEP: 5700 | > loss_text_ce: 0.031878069043159485 (0.03277220907801282) | > loss_mel_ce: 3.976787805557251 (4.106889547010062) | > loss: 0.0626354068517685 (0.06468221494385533) | > current_lr: 1e-05 | > step_time: 0.2708 (0.28033951343684005) | > loader_time: 0.008 (0.008695911244316644)  --> TIME: 2025-12-19 23:27:10 -- STEP: 541/2207 -- GLOBAL_STEP: 5750 | > loss_text_ce: 0.0341142900288105 (0.03278052015316135) | > loss_mel_ce: 4.066941738128662 (4.106222606630732) | > loss: 0.06407900154590607 (0.06467192390048801) | > current_lr: 1e-05 | > step_time: 0.2888 (0.2802339895816036) | > loader_time: 0.008 (0.008648681552485046)  --> TIME: 2025-12-19 23:27:29 -- STEP: 591/2207 -- GLOBAL_STEP: 5800 | > loss_text_ce: 0.03217954561114311 (0.032738844865646266) | > loss_mel_ce: 3.9995813369750977 (4.106167038886884) | > loss: 0.0629962608218193 (0.06467040447288165) | > current_lr: 1e-05 | > step_time: 0.2909 (0.2802896427018994) | > loader_time: 0.0081 (0.008587077991208046)  --> TIME: 2025-12-19 23:27:48 -- STEP: 641/2207 -- GLOBAL_STEP: 5850 | > loss_text_ce: 0.03290794417262077 (0.03274818261367576) | > loss_mel_ce: 4.050463676452637 (4.102661906464055) | > loss: 0.06380268186330795 (0.06461578270481462) | > current_lr: 1e-05 | > step_time: 0.2891 (0.2799590459665905) | > loader_time: 0.0076 (0.008560780244014934)  --> TIME: 2025-12-19 23:28:08 -- STEP: 691/2207 -- GLOBAL_STEP: 5900 | > loss_text_ce: 0.033096350729465485 (0.03273934032094115) | > loss_mel_ce: 4.230457305908203 (4.103105344234086) | > loss: 0.06661802530288696 (0.06462257329034916) | > current_lr: 1e-05 | > step_time: 0.2888 (0.28009270345077864) | > loader_time: 0.0079 (0.008544543372566901)  --> TIME: 2025-12-19 23:28:27 -- STEP: 741/2207 -- GLOBAL_STEP: 5950 | > loss_text_ce: 0.03447591885924339 (0.03272524028362937) | > loss_mel_ce: 3.9467613697052 (4.099739815059457) | > loss: 0.06220683082938194 (0.06456976656911358) | > current_lr: 1e-05 | > step_time: 0.2708 (0.28011725209502575) | > loader_time: 0.0085 (0.008504998828718061)  --> TIME: 2025-12-19 23:28:46 -- STEP: 791/2207 -- GLOBAL_STEP: 6000 | > loss_text_ce: 0.031701214611530304 (0.03270154030977099) | > loss_mel_ce: 3.9861724376678467 (4.099287290488723) | > loss: 0.06277927756309509 (0.06456232558366029) | > current_lr: 1e-05 | > step_time: 0.2628 (0.2801193307835597) | > loader_time: 0.0081 (0.008462230414717302) > CHECKPOINT : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-19-2025_11+20PM-0000000/checkpoint_6000.pth  --> TIME: 2025-12-19 23:29:29 -- STEP: 841/2207 -- GLOBAL_STEP: 6050 | > loss_text_ce: 0.03385571017861366 (0.03268137809407413) | > loss_mel_ce: 4.031688690185547 (4.097443981204673) | > loss: 0.06352413445711136 (0.06453320885898818) | > current_lr: 1e-05 | > step_time: 0.2884 (0.28015128298973807) | > loader_time: 0.0077 (0.008446370237080577)  --> TIME: 2025-12-19 23:29:48 -- STEP: 891/2207 -- GLOBAL_STEP: 6100 | > loss_text_ce: 0.032470740377902985 (0.032681752594164844) | > loss_mel_ce: 3.865196466445923 (4.095037801230928) | > loss: 0.06090104952454567 (0.06449561814467121) | > current_lr: 1e-05 | > step_time: 0.2699 (0.27991642561290686) | > loader_time: 0.0079 (0.008440049007685494)  --> TIME: 2025-12-19 23:30:08 -- STEP: 941/2207 -- GLOBAL_STEP: 6150 | > loss_text_ce: 0.03259256109595299 (0.032686683451252044) | > loss_mel_ce: 4.243472099304199 (4.092387335966801) | > loss: 0.06681351363658905 (0.06445428167690513) | > current_lr: 1e-05 | > step_time: 0.2985 (0.2799384107498512) | > loader_time: 0.0083 (0.008427470700773756)  --> TIME: 2025-12-19 23:30:27 -- STEP: 991/2207 -- GLOBAL_STEP: 6200 | > loss_text_ce: 0.033036742359399796 (0.03268012933350354) | > loss_mel_ce: 4.026215076446533 (4.090084233028976) | > loss: 0.06342580914497375 (0.06441819329119239) | > current_lr: 1e-05 | > step_time: 0.2664 (0.27988023449987287) | > loader_time: 0.0089 (0.00843105017838156)  --> TIME: 2025-12-19 23:30:46 -- STEP: 1041/2207 -- GLOBAL_STEP: 6250 | > loss_text_ce: 0.033400777727365494 (0.0326768395926635) | > loss_mel_ce: 4.0089898109436035 (4.089549863716366) | > loss: 0.06316235661506653 (0.06440979238030552) | > current_lr: 1e-05 | > step_time: 0.2701 (0.27983240641503265) | > loader_time: 0.0085 (0.008462739838188828)  --> TIME: 2025-12-19 23:31:05 -- STEP: 1091/2207 -- GLOBAL_STEP: 6300 | > loss_text_ce: 0.031533099710941315 (0.03267257152027923) | > loss_mel_ce: 4.511966705322266 (4.088653826691708) | > loss: 0.07099218666553497 (0.06439572509197226) | > current_lr: 1e-05 | > step_time: 0.2671 (0.27962432510802576) | > loader_time: 0.0077 (0.008466678185816975)  --> TIME: 2025-12-19 23:31:24 -- STEP: 1141/2207 -- GLOBAL_STEP: 6350 | > loss_text_ce: 0.03148523345589638 (0.032665948479009436) | > loss_mel_ce: 3.9132697582244873 (4.085592917869427) | > loss: 0.061636798083782196 (0.0643477948981613) | > current_lr: 1e-05 | > step_time: 0.29 (0.2796653886305254) | > loader_time: 0.0079 (0.008451628329772682)  --> TIME: 2025-12-19 23:31:43 -- STEP: 1191/2207 -- GLOBAL_STEP: 6400 | > loss_text_ce: 0.03140372782945633 (0.03266367260100808) | > loss_mel_ce: 4.076885223388672 (4.08433563062666) | > loss: 0.06419201195240021 (0.06432811422922557) | > current_lr: 1e-05 | > step_time: 0.2867 (0.27976663486983605) | > loader_time: 0.0086 (0.008452464911639444)  --> TIME: 2025-12-19 23:32:02 -- STEP: 1241/2207 -- GLOBAL_STEP: 6450 | > loss_text_ce: 0.03408047929406166 (0.03266447187100415) | > loss_mel_ce: 4.360547065734863 (4.084415867864647) | > loss: 0.06866605579853058 (0.0643293804111595) | > current_lr: 1e-05 | > step_time: 0.2875 (0.279699163413836) | > loader_time: 0.0079 (0.0084403975177061)  --> TIME: 2025-12-19 23:32:22 -- STEP: 1291/2207 -- GLOBAL_STEP: 6500 | > loss_text_ce: 0.031175393611192703 (0.03264621023224311) | > loss_mel_ce: 4.075261116027832 (4.083227412632316) | > loss: 0.06416307389736176 (0.06431052546679841) | > current_lr: 1e-05 | > step_time: 0.2701 (0.2796680572695926) | > loader_time: 0.008 (0.008421044124363926)  --> TIME: 2025-12-19 23:32:41 -- STEP: 1341/2207 -- GLOBAL_STEP: 6550 | > loss_text_ce: 0.033355582505464554 (0.0326451789712745) | > loss_mel_ce: 4.339946746826172 (4.0817988941155585) | > loss: 0.06833285093307495 (0.06428818874988965) | > current_lr: 1e-05 | > step_time: 0.3145 (0.2795609117532119) | > loader_time: 0.0077 (0.008406509666812676)  --> TIME: 2025-12-19 23:33:00 -- STEP: 1391/2207 -- GLOBAL_STEP: 6600 | > loss_text_ce: 0.03157105669379234 (0.03262844312558443) | > loss_mel_ce: 3.9514052867889404 (4.080753967455179) | > loss: 0.06223400682210922 (0.06427160027556764) | > current_lr: 1e-05 | > step_time: 0.2671 (0.2795383186326623) | > loader_time: 0.0087 (0.008389373891735833)  --> TIME: 2025-12-19 23:33:19 -- STEP: 1441/2207 -- GLOBAL_STEP: 6650 | > loss_text_ce: 0.03227873519062996 (0.032622452478889404) | > loss_mel_ce: 3.937110185623169 (4.079359220349892) | > loss: 0.06202170252799988 (0.06424971374164105) | > current_lr: 1e-05 | > step_time: 0.2667 (0.2795068340778024) | > loader_time: 0.0081 (0.0083797916118508)  --> TIME: 2025-12-19 23:33:39 -- STEP: 1491/2207 -- GLOBAL_STEP: 6700 | > loss_text_ce: 0.03508618474006653 (0.032608193004944774) | > loss_mel_ce: 4.201953887939453 (4.078866420537328) | > loss: 0.06620375066995621 (0.06424179094038436) | > current_lr: 1e-05 | > step_time: 0.2621 (0.2797854189741622) | > loader_time: 0.008 (0.008368091723968483)  --> TIME: 2025-12-19 23:33:58 -- STEP: 1541/2207 -- GLOBAL_STEP: 6750 | > loss_text_ce: 0.03175294026732445 (0.03260520016828921) | > loss_mel_ce: 3.958707094192505 (4.078298873053517) | > loss: 0.062350936233997345 (0.06423287624511387) | > current_lr: 1e-05 | > step_time: 0.2906 (0.2797788922311735) | > loader_time: 0.0079 (0.008361948379675025)  --> TIME: 2025-12-19 23:34:17 -- STEP: 1591/2207 -- GLOBAL_STEP: 6800 | > loss_text_ce: 0.03180113434791565 (0.03259606553811753) | > loss_mel_ce: 3.947313070297241 (4.076905974646468) | > loss: 0.06217366084456444 (0.06421096947940934) | > current_lr: 1e-05 | > step_time: 0.2888 (0.2797808211079042) | > loader_time: 0.01 (0.008361990537229825)  --> TIME: 2025-12-19 23:34:36 -- STEP: 1641/2207 -- GLOBAL_STEP: 6850 | > loss_text_ce: 0.031015565618872643 (0.032597589773613364) | > loss_mel_ce: 4.251021862030029 (4.075699189172724) | > loss: 0.06690683215856552 (0.06419213727970165) | > current_lr: 1e-05 | > step_time: 0.2867 (0.27970779399767254) | > loader_time: 0.0078 (0.00836158728614076)  --> TIME: 2025-12-19 23:34:55 -- STEP: 1691/2207 -- GLOBAL_STEP: 6900 | > loss_text_ce: 0.032971665263175964 (0.032580885582337404) | > loss_mel_ce: 4.157569408416748 (4.074683435142281) | > loss: 0.06547720730304718 (0.06417600510698117) | > current_lr: 1e-05 | > step_time: 0.2659 (0.2796958701135421) | > loader_time: 0.0079 (0.008353771788350643)  --> TIME: 2025-12-19 23:35:15 -- STEP: 1741/2207 -- GLOBAL_STEP: 6950 | > loss_text_ce: 0.03412429243326187 (0.03258349168735418) | > loss_mel_ce: 3.9358301162719727 (4.073978549649421) | > loss: 0.06203053891658783 (0.06416503198568956) | > current_lr: 1e-05 | > step_time: 0.2867 (0.2796876093630543) | > loader_time: 0.0079 (0.008347673843401302)  --> TIME: 2025-12-19 23:35:34 -- STEP: 1791/2207 -- GLOBAL_STEP: 7000 | > loss_text_ce: 0.03215445205569267 (0.03257655235873722) | > loss_mel_ce: 4.294580936431885 (4.072014079554524) | > loss: 0.06760524213314056 (0.06413422870626008) | > grad_norm: 0 (0.0) | > current_lr: 1e-05 | > step_time: 0.3286 (0.27962819615400836) | > loader_time: 0.0094 (0.008355755036533493) > CHECKPOINT : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-19-2025_11+20PM-0000000/checkpoint_7000.pth  --> TIME: 2025-12-19 23:36:17 -- STEP: 1841/2207 -- GLOBAL_STEP: 7050 | > loss_text_ce: 0.03530712425708771 (0.03257391380934299) | > loss_mel_ce: 4.284851551055908 (4.070986293956415) | > loss: 0.0675024762749672 (0.06411812833466335) | > current_lr: 1e-05 | > step_time: 0.2648 (0.2795902267219839) | > loader_time: 0.0078 (0.008350239701403134)  --> TIME: 2025-12-19 23:36:36 -- STEP: 1891/2207 -- GLOBAL_STEP: 7100 | > loss_text_ce: 0.03220251947641373 (0.03257080443431966) | > loss_mel_ce: 4.281186103820801 (4.070231137737148) | > loss: 0.06739670038223267 (0.0641062804382041) | > current_lr: 1e-05 | > step_time: 0.2962 (0.2795604967301893) | > loader_time: 0.0077 (0.008338792811998402)  --> TIME: 2025-12-19 23:36:55 -- STEP: 1941/2207 -- GLOBAL_STEP: 7150 | > loss_text_ce: 0.030263639986515045 (0.03256799163376465) | > loss_mel_ce: 3.867950201034546 (4.068827845889346) | > loss: 0.06090959161520004 (0.06408431003659984) | > current_lr: 1e-05 | > step_time: 0.2866 (0.27960251705478245) | > loader_time: 0.0078 (0.008333902024903394)  --> TIME: 2025-12-19 23:37:14 -- STEP: 1991/2207 -- GLOBAL_STEP: 7200 | > loss_text_ce: 0.033825602382421494 (0.03256503755478388) | > loss_mel_ce: 3.9784209728240967 (4.067005923739147) | > loss: 0.0626913532614708 (0.06405579635152145) | > current_lr: 1e-05 | > step_time: 0.2888 (0.27955440909703266) | > loader_time: 0.0082 (0.008326804081079992)  --> TIME: 2025-12-19 23:37:33 -- STEP: 2041/2207 -- GLOBAL_STEP: 7250 | > loss_text_ce: 0.0330999493598938 (0.032551859421970364) | > loss_mel_ce: 4.061833381652832 (4.065660371084173) | > loss: 0.06398333609104156 (0.06403456617918804) | > current_lr: 1e-05 | > step_time: 0.2625 (0.2794767479754033) | > loader_time: 0.0078 (0.008326117750594903)  --> TIME: 2025-12-19 23:37:52 -- STEP: 2091/2207 -- GLOBAL_STEP: 7300 | > loss_text_ce: 0.03444935381412506 (0.03255168464977251) | > loss_mel_ce: 3.84794282913208 (4.064449771404943) | > loss: 0.06066237762570381 (0.06401564783026077) | > current_lr: 1e-05 | > step_time: 0.2682 (0.2794808959459245) | > loader_time: 0.0078 (0.008326130241711461)  --> TIME: 2025-12-19 23:38:12 -- STEP: 2141/2207 -- GLOBAL_STEP: 7350 | > loss_text_ce: 0.03163890913128853 (0.03255556912684707) | > loss_mel_ce: 3.6894760131835938 (4.063137030055838) | > loss: 0.058142419904470444 (0.06399519694555815) | > current_lr: 1e-05 | > step_time: 0.2652 (0.2795219843435491) | > loader_time: 0.0082 (0.008321784133946753)  --> TIME: 2025-12-19 23:38:31 -- STEP: 2191/2207 -- GLOBAL_STEP: 7400 | > loss_text_ce: 0.03182921186089516 (0.03254674895764727) | > loss_mel_ce: 3.9978630542755127 (4.062054034728858) | > loss: 0.06296394020318985 (0.06397813732686077) | > current_lr: 1e-05 | > step_time: 0.2911 (0.2795370181147665) | > loader_time: 0.0082 (0.008315004207727725)  > EVALUATION  --> EVAL PERFORMANCE | > avg_loader_time: 0.16947253330333814 (+0.08974874502903706) | > avg_loss_text_ce: 0.03120868057767682 (-0.0006401909993508717) | > avg_loss_mel_ce: 3.949908100269936 (-0.09110731530833593) | > avg_loss: 3.9811167942511068 (-0.0917475046338252) > BEST MODEL : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-19-2025_11+20PM-0000000/best_model_7416.pth