File size: 38,807 Bytes
4fb2cc1 2a99c63 4fb2cc1 2a99c63 4fb2cc1 2a99c63 4fb2cc1 2a99c63 4fb2cc1 2a99c63 4fb2cc1 2a99c63 8d31cf0 fc628c6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 | > Training Environment:
| > Backend: Torch
| > Mixed precision: False
| > Precision: float32
| > Current device: 0
| > Num. of GPUs: 1
| > Num. of CPUs: 256
| > Num. of Torch Threads: 1
| > Torch seed: 1
| > Torch CUDNN: True
| > Torch CUDNN deterministic: False
| > Torch CUDNN benchmark: False
| > Torch TF32 MatMul: False
> Start Tensorboard: tensorboard --logdir=/workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000
> Restoring from best_model_16244.pth ...
> Restoring Model...
> Restoring Optimizer...
> Model restored from step 16244
> Model has 518442047 parameters
[4m[1m > EPOCH: 0/6[0m
--> /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000
[1m > EVALUATION [0m
[1m--> EVAL PERFORMANCE[0m
| > avg_loader_time: 0.07435984869261048 [0m(+0)
| > avg_loss_text_ce: 0.030082402234846674 [0m(+0)
| > avg_loss_mel_ce: 3.822696221841348 [0m(+0)
| > avg_loss: 3.852778621622034 [0m(+0)
> BEST MODEL : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/best_model_16245.pth
[4m[1m > EPOCH: 1/6[0m
--> /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000
[1m > TRAINING (2025-12-21 12:14:05) [0m
[1m --> TIME: 2025-12-21 12:14:11 -- STEP: 5/2207 -- GLOBAL_STEP: 16250[0m
| > loss_text_ce: 0.03219795227050781 (0.03218773603439331)
| > loss_mel_ce: 4.446129322052002 (4.046912813186646)
| > loss: 0.06997386366128922 (0.0637359470129013)
| > current_lr: 1e-05
| > step_time: 0.2695 (0.2764430522918701)
| > loader_time: 0.0089 (0.010691452026367187)
[1m --> TIME: 2025-12-21 12:14:30 -- STEP: 55/2207 -- GLOBAL_STEP: 16300[0m
| > loss_text_ce: 0.03195399418473244 (0.030980682135982946)
| > loss_mel_ce: 3.563579797744751 (3.869967087832364)
| > loss: 0.05618021637201309 (0.0609523091126572)
| > current_lr: 1e-05
| > step_time: 0.2691 (0.27666049003601073)
| > loader_time: 0.009 (0.01010052507573908)
[1m --> TIME: 2025-12-21 12:14:49 -- STEP: 105/2207 -- GLOBAL_STEP: 16350[0m
| > loss_text_ce: 0.03152564913034439 (0.03094851766668615)
| > loss_mel_ce: 3.851048707962036 (3.864614899953206)
| > loss: 0.06066522374749184 (0.060868178449925925)
| > current_lr: 1e-05
| > step_time: 0.2912 (0.27978511991954985)
| > loader_time: 0.0092 (0.009624846776326497)
[1m --> TIME: 2025-12-21 12:15:09 -- STEP: 155/2207 -- GLOBAL_STEP: 16400[0m
| > loss_text_ce: 0.029248259961605072 (0.03089537698655359)
| > loss_mel_ce: 3.628293514251709 (3.846851090461977)
| > loss: 0.05714908987283707 (0.06058978856571259)
| > current_lr: 1e-05
| > step_time: 0.2925 (0.28088303073760007)
| > loader_time: 0.0081 (0.009488651829381148)
[1m --> TIME: 2025-12-21 12:15:28 -- STEP: 205/2207 -- GLOBAL_STEP: 16450[0m
| > loss_text_ce: 0.032587237656116486 (0.03095879358489339)
| > loss_mel_ce: 3.7422239780426025 (3.8365380601185124)
| > loss: 0.05898142606019974 (0.060429638442469805)
| > current_lr: 1e-05
| > step_time: 0.2908 (0.2808621534487096)
| > loader_time: 0.0082 (0.009342936771671948)
[1m --> TIME: 2025-12-21 12:15:47 -- STEP: 255/2207 -- GLOBAL_STEP: 16500[0m
| > loss_text_ce: 0.03305845335125923 (0.030955966971084183)
| > loss_mel_ce: 3.9918406009674072 (3.8380968393064014)
| > loss: 0.06288904696702957 (0.06045395013164071)
| > grad_norm: 0 (0.0)
| > current_lr: 1e-05
| > step_time: 0.2862 (0.2800759567933923)
| > loader_time: 0.0086 (0.009221519208421897)
[1m --> TIME: 2025-12-21 12:16:06 -- STEP: 305/2207 -- GLOBAL_STEP: 16550[0m
| > loss_text_ce: 0.031405024230480194 (0.030954475768032622)
| > loss_mel_ce: 3.559880495071411 (3.8379844337213234)
| > loss: 0.0561138354241848 (0.06045217049903557)
| > current_lr: 1e-05
| > step_time: 0.2679 (0.2793264326502066)
| > loader_time: 0.0081 (0.00907517104852395)
[1m --> TIME: 2025-12-21 12:16:25 -- STEP: 355/2207 -- GLOBAL_STEP: 16600[0m
| > loss_text_ce: 0.03148230165243149 (0.03097983851084407)
| > loss_mel_ce: 3.8917019367218018 (3.8419334841446138)
| > loss: 0.061299752444028854 (0.06051427065066888)
| > current_lr: 1e-05
| > step_time: 0.2688 (0.27979512684781777)
| > loader_time: 0.0079 (0.008994941980066435)
[1m --> TIME: 2025-12-21 12:16:45 -- STEP: 405/2207 -- GLOBAL_STEP: 16650[0m
| > loss_text_ce: 0.030755072832107544 (0.03096303568210131)
| > loss_mel_ce: 3.5810444355010986 (3.837673578733279)
| > loss: 0.05643436685204506 (0.060447447075520036)
| > current_lr: 1e-05
| > step_time: 0.2877 (0.27977862004880527)
| > loader_time: 0.0082 (0.008914067421430426)
[1m --> TIME: 2025-12-21 12:17:04 -- STEP: 455/2207 -- GLOBAL_STEP: 16700[0m
| > loss_text_ce: 0.032188721001148224 (0.030962336661068946)
| > loss_mel_ce: 3.918936252593994 (3.8396105085100447)
| > loss: 0.0617363266646862 (0.06047770069031925)
| > current_lr: 1e-05
| > step_time: 0.2866 (0.27930216998844304)
| > loader_time: 0.0084 (0.008828892026628768)
[1m --> TIME: 2025-12-21 12:17:23 -- STEP: 505/2207 -- GLOBAL_STEP: 16750[0m
| > loss_text_ce: 0.0326453261077404 (0.03099347592359132)
| > loss_mel_ce: 3.800915241241455 (3.8388911927100455)
| > loss: 0.059899382293224335 (0.06046694796716813)
| > current_lr: 1e-05
| > step_time: 0.2867 (0.27900417252342247)
| > loader_time: 0.0081 (0.008783628916976477)
[1m --> TIME: 2025-12-21 12:17:42 -- STEP: 555/2207 -- GLOBAL_STEP: 16800[0m
| > loss_text_ce: 0.030252568423748016 (0.030994508647032685)
| > loss_mel_ce: 3.620875597000122 (3.8386023968189686)
| > loss: 0.05704887583851814 (0.06046245165907585)
| > current_lr: 1e-05
| > step_time: 0.2862 (0.2790829800270695)
| > loader_time: 0.0077 (0.008722289832862648)
[1m --> TIME: 2025-12-21 12:18:01 -- STEP: 605/2207 -- GLOBAL_STEP: 16850[0m
| > loss_text_ce: 0.03053409792482853 (0.030986474165000208)
| > loss_mel_ce: 4.023441791534424 (3.8384223346867836)
| > loss: 0.06334337592124939 (0.060459512675350366)
| > current_lr: 1e-05
| > step_time: 0.2666 (0.2788679410603417)
| > loader_time: 0.0078 (0.008667540353191784)
[1m --> TIME: 2025-12-21 12:18:20 -- STEP: 655/2207 -- GLOBAL_STEP: 16900[0m
| > loss_text_ce: 0.030561409890651703 (0.03099969866337212)
| > loss_mel_ce: 3.8719499111175537 (3.8380026737242257)
| > loss: 0.06097673997282982 (0.060453162103436375)
| > current_lr: 1e-05
| > step_time: 0.2663 (0.27882370475594254)
| > loader_time: 0.0089 (0.008689285962636233)
[1m --> TIME: 2025-12-21 12:18:39 -- STEP: 705/2207 -- GLOBAL_STEP: 16950[0m
| > loss_text_ce: 0.031610529869794846 (0.03098056488637383)
| > loss_mel_ce: 3.83477520942688 (3.838375516647988)
| > loss: 0.06041227653622627 (0.06045868881839387)
| > current_lr: 1e-05
| > step_time: 0.291 (0.27892449933586333)
| > loader_time: 0.0082 (0.008669162642025784)
[1m --> TIME: 2025-12-21 12:18:58 -- STEP: 755/2207 -- GLOBAL_STEP: 17000[0m
| > loss_text_ce: 0.03250373899936676 (0.030986027972192954)
| > loss_mel_ce: 3.5191447734832764 (3.835920715332031)
| > loss: 0.055494509637355804 (0.060420417894196034)
| > current_lr: 1e-05
| > step_time: 0.2686 (0.27887622504834264)
| > loader_time: 0.0083 (0.00863919352853536)
> CHECKPOINT : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/checkpoint_17000.pth
[1m --> TIME: 2025-12-21 12:19:41 -- STEP: 805/2207 -- GLOBAL_STEP: 17050[0m
| > loss_text_ce: 0.02891026996076107 (0.030961797997262907)
| > loss_mel_ce: 3.8255555629730225 (3.8371545169664465)
| > loss: 0.06022602692246437 (0.060439317442060256)
| > current_lr: 1e-05
| > step_time: 0.2688 (0.2789103673852005)
| > loader_time: 0.0077 (0.008619996183407246)
[1m --> TIME: 2025-12-21 12:20:00 -- STEP: 855/2207 -- GLOBAL_STEP: 17100[0m
| > loss_text_ce: 0.030247919261455536 (0.03095182662949576)
| > loss_mel_ce: 3.7221128940582275 (3.8346250587039523)
| > loss: 0.05863063782453537 (0.06039963886577483)
| > current_lr: 1e-05
| > step_time: 0.2612 (0.2788665866294099)
| > loader_time: 0.0085 (0.008600194272939227)
[1m --> TIME: 2025-12-21 12:20:20 -- STEP: 905/2207 -- GLOBAL_STEP: 17150[0m
| > loss_text_ce: 0.03263920545578003 (0.030970429752577733)
| > loss_mel_ce: 3.8033525943756104 (3.8346228056849694)
| > loss: 0.05993737280368805 (0.06039989432988904)
| > current_lr: 1e-05
| > step_time: 0.2658 (0.2798094433315545)
| > loader_time: 0.0085 (0.008578953031676908)
[1m --> TIME: 2025-12-21 12:20:39 -- STEP: 955/2207 -- GLOBAL_STEP: 17200[0m
| > loss_text_ce: 0.03272069990634918 (0.030973391158808587)
| > loss_mel_ce: 3.866997718811035 (3.8342152485672716)
| > loss: 0.060933101922273636 (0.06039357252848086)
| > current_lr: 1e-05
| > step_time: 0.2957 (0.2798374602931952)
| > loader_time: 0.0083 (0.008563283850385253)
[1m --> TIME: 2025-12-21 12:20:58 -- STEP: 1005/2207 -- GLOBAL_STEP: 17250[0m
| > loss_text_ce: 0.03107460029423237 (0.03097570813191471)
| > loss_mel_ce: 3.7447755336761475 (3.8325845606884554)
| > loss: 0.058997657150030136 (0.06036812922551264)
| > current_lr: 1e-05
| > step_time: 0.2864 (0.279714151164192)
| > loader_time: 0.0085 (0.008561972954973057)
[1m --> TIME: 2025-12-21 12:21:17 -- STEP: 1055/2207 -- GLOBAL_STEP: 17300[0m
| > loss_text_ce: 0.030682062730193138 (0.030976020410586308)
| > loss_mel_ce: 3.643967866897583 (3.8322222506265504)
| > loss: 0.0574164055287838 (0.06036247299957615)
| > current_lr: 1e-05
| > step_time: 0.2883 (0.27947213502856744)
| > loader_time: 0.0083 (0.008548689340528149)
[1m --> TIME: 2025-12-21 12:21:36 -- STEP: 1105/2207 -- GLOBAL_STEP: 17350[0m
| > loss_text_ce: 0.032925937324762344 (0.03097810615106945)
| > loss_mel_ce: 3.7915194034576416 (3.8325317693511827)
| > loss: 0.05975695699453354 (0.060367341838541075)
| > current_lr: 1e-05
| > step_time: 0.2868 (0.27930227965790716)
| > loader_time: 0.0077 (0.008524640760810121)
[1m --> TIME: 2025-12-21 12:21:55 -- STEP: 1155/2207 -- GLOBAL_STEP: 17400[0m
| > loss_text_ce: 0.03101305104792118 (0.03097473747576728)
| > loss_mel_ce: 3.926995038986206 (3.830362285473646)
| > loss: 0.0618438757956028 (0.060333391017728034)
| > current_lr: 1e-05
| > step_time: 0.2953 (0.27925692182598655)
| > loader_time: 0.0081 (0.00850323412841532)
[1m --> TIME: 2025-12-21 12:22:15 -- STEP: 1205/2207 -- GLOBAL_STEP: 17450[0m
| > loss_text_ce: 0.031288035213947296 (0.030977744015544283)
| > loss_mel_ce: 4.4484028816223145 (3.831367015838623)
| > loss: 0.06999517232179642 (0.060349136900110366)
| > current_lr: 1e-05
| > step_time: 0.2619 (0.27930984556427596)
| > loader_time: 0.0081 (0.008497552159416222)
[1m --> TIME: 2025-12-21 12:22:34 -- STEP: 1255/2207 -- GLOBAL_STEP: 17500[0m
| > loss_text_ce: 0.029901018366217613 (0.030980763323159806)
| > loss_mel_ce: 3.783792734146118 (3.83137526474151)
| > loss: 0.0595889650285244 (0.06034931295599596)
| > current_lr: 1e-05
| > step_time: 0.302 (0.2792716653223529)
| > loader_time: 0.0089 (0.008480142122245867)
[1m --> TIME: 2025-12-21 12:22:53 -- STEP: 1305/2207 -- GLOBAL_STEP: 17550[0m
| > loss_text_ce: 0.030708273872733116 (0.030976130990347187)
| > loss_mel_ce: 3.928159236907959 (3.831721293606521)
| > loss: 0.061857305467128754 (0.06035464728015593)
| > current_lr: 1e-05
| > step_time: 0.2912 (0.2792421129014756)
| > loader_time: 0.008 (0.008464955735480633)
[1m --> TIME: 2025-12-21 12:23:12 -- STEP: 1355/2207 -- GLOBAL_STEP: 17600[0m
| > loss_text_ce: 0.030456705018877983 (0.030980337833181517)
| > loss_mel_ce: 3.610086441040039 (3.8301265816846897)
| > loss: 0.05688348785042763 (0.06032979563428467)
| > current_lr: 1e-05
| > step_time: 0.289 (0.27909282828609)
| > loader_time: 0.0084 (0.008450167117523512)
[1m --> TIME: 2025-12-21 12:23:31 -- STEP: 1405/2207 -- GLOBAL_STEP: 17650[0m
| > loss_text_ce: 0.0315256342291832 (0.03096651314787593)
| > loss_mel_ce: 3.5720529556274414 (3.8309017710838456)
| > loss: 0.056305915117263794 (0.06034169195756794)
| > current_lr: 1e-05
| > step_time: 0.2542 (0.27900979221927735)
| > loader_time: 0.0081 (0.00843814344168556)
[1m --> TIME: 2025-12-21 12:23:50 -- STEP: 1455/2207 -- GLOBAL_STEP: 17700[0m
| > loss_text_ce: 0.029983242973685265 (0.030972939362282195)
| > loss_mel_ce: 3.7503371238708496 (3.830120104366971)
| > loss: 0.0590675063431263 (0.0603295788296924)
| > current_lr: 1e-05
| > step_time: 0.2862 (0.279018137299318)
| > loader_time: 0.0086 (0.008426600551277487)
[1m --> TIME: 2025-12-21 12:24:09 -- STEP: 1505/2207 -- GLOBAL_STEP: 17750[0m
| > loss_text_ce: 0.03367031365633011 (0.0309653219120249)
| > loss_mel_ce: 3.723371744155884 (3.830078932613233)
| > loss: 0.05870378389954567 (0.06032881649625658)
| > current_lr: 1e-05
| > step_time: 0.2685 (0.27902523592064943)
| > loader_time: 0.0079 (0.00841875092135711)
[1m --> TIME: 2025-12-21 12:24:28 -- STEP: 1555/2207 -- GLOBAL_STEP: 17800[0m
| > loss_text_ce: 0.02944960817694664 (0.03096569205403711)
| > loss_mel_ce: 4.062541961669922 (3.8309391785281264)
| > loss: 0.06393736600875854 (0.060342263636385896)
| > current_lr: 1e-05
| > step_time: 0.2902 (0.2789708040918187)
| > loader_time: 0.0081 (0.008411988568075979)
[1m --> TIME: 2025-12-21 12:24:47 -- STEP: 1605/2207 -- GLOBAL_STEP: 17850[0m
| > loss_text_ce: 0.029368538409471512 (0.03096987812481008)
| > loss_mel_ce: 3.616783857345581 (3.8293144246870856)
| > loss: 0.05697113275527954 (0.060316942265471934)
| > current_lr: 1e-05
| > step_time: 0.2896 (0.27894843597649793)
| > loader_time: 0.0077 (0.008404218667764156)
[1m --> TIME: 2025-12-21 12:25:06 -- STEP: 1655/2207 -- GLOBAL_STEP: 17900[0m
| > loss_text_ce: 0.02917139232158661 (0.030966688455096543)
| > loss_mel_ce: 3.955526351928711 (3.8295799357653384)
| > loss: 0.06226090341806412 (0.06032104103676862)
| > current_lr: 1e-05
| > step_time: 0.2508 (0.2788965385125842)
| > loader_time: 0.008 (0.008392788348241155)
[1m --> TIME: 2025-12-21 12:25:25 -- STEP: 1705/2207 -- GLOBAL_STEP: 17950[0m
| > loss_text_ce: 0.036614060401916504 (0.030967657537885076)
| > loss_mel_ce: 3.92380690574646 (3.8291027352956735)
| > loss: 0.06188157945871353 (0.06031359991938138)
| > current_lr: 1e-05
| > step_time: 0.4298 (0.2789531161009043)
| > loader_time: 0.0081 (0.008383412235293566)
[1m --> TIME: 2025-12-21 12:25:44 -- STEP: 1755/2207 -- GLOBAL_STEP: 18000[0m
| > loss_text_ce: 0.031244533136487007 (0.030965659205942412)
| > loss_mel_ce: 3.8533778190612793 (3.8291114841091667)
| > loss: 0.060697223991155624 (0.06031370539525975)
| > current_lr: 1e-05
| > step_time: 0.2861 (0.2788446566318174)
| > loader_time: 0.0081 (0.008380908911724024)
> CHECKPOINT : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/checkpoint_18000.pth
[1m --> TIME: 2025-12-21 12:26:27 -- STEP: 1805/2207 -- GLOBAL_STEP: 18050[0m
| > loss_text_ce: 0.02912035584449768 (0.03096764023696947)
| > loss_mel_ce: 3.675009250640869 (3.8278143662164745)
| > loss: 0.05787702649831772 (0.060293468884566485)
| > current_lr: 1e-05
| > step_time: 0.2666 (0.2788151725177289)
| > loader_time: 0.0083 (0.008385370840986658)
[1m --> TIME: 2025-12-21 12:26:46 -- STEP: 1855/2207 -- GLOBAL_STEP: 18100[0m
| > loss_text_ce: 0.030438384041190147 (0.030971936526083238)
| > loss_mel_ce: 3.6038551330566406 (3.827252550587821)
| > loss: 0.056785836815834045 (0.06028475765028411)
| > grad_norm: 0 (0.0)
| > current_lr: 1e-05
| > step_time: 0.3188 (0.2787536723915779)
| > loader_time: 0.0083 (0.00838110569031089)
[1m --> TIME: 2025-12-21 12:27:05 -- STEP: 1905/2207 -- GLOBAL_STEP: 18150[0m
| > loss_text_ce: 0.029724599793553352 (0.03097888275668064)
| > loss_mel_ce: 3.7439305782318115 (3.827816226914173)
| > loss: 0.058963362127542496 (0.0602936736300705)
| > current_lr: 1e-05
| > step_time: 0.2651 (0.27881914574330247)
| > loader_time: 0.0086 (0.008379007574767254)
[1m --> TIME: 2025-12-21 12:27:24 -- STEP: 1955/2207 -- GLOBAL_STEP: 18200[0m
| > loss_text_ce: 0.030410582199692726 (0.03097874500200419)
| > loss_mel_ce: 3.681417226791382 (3.826422531708427)
| > loss: 0.05799730867147446 (0.06027189498133671)
| > current_lr: 1e-05
| > step_time: 0.2678 (0.2787622943253773)
| > loader_time: 0.0085 (0.008377984844510192)
[1m --> TIME: 2025-12-21 12:27:43 -- STEP: 2005/2207 -- GLOBAL_STEP: 18250[0m
| > loss_text_ce: 0.03071397729218006 (0.030976552677868013)
| > loss_mel_ce: 3.844003438949585 (3.825914057829136)
| > loss: 0.06054246053099632 (0.06026391582447394)
| > current_lr: 1e-05
| > step_time: 0.2667 (0.2787343830241824)
| > loader_time: 0.0081 (0.008377488533457602)
[1m --> TIME: 2025-12-21 12:28:02 -- STEP: 2055/2207 -- GLOBAL_STEP: 18300[0m
| > loss_text_ce: 0.03158266097307205 (0.030973146271205298)
| > loss_mel_ce: 3.630298376083374 (3.8251054886773845)
| > loss: 0.057216890156269073 (0.06025122870348956)
| > current_lr: 1e-05
| > step_time: 0.2654 (0.27866049694608885)
| > loader_time: 0.009 (0.00838099222113615)
[1m --> TIME: 2025-12-21 12:28:22 -- STEP: 2105/2207 -- GLOBAL_STEP: 18350[0m
| > loss_text_ce: 0.029528765007853508 (0.030974545680216542)
| > loss_mel_ce: 3.808253765106201 (3.8251185174792504)
| > loss: 0.0599653534591198 (0.06025145414535336)
| > current_lr: 1e-05
| > step_time: 0.2895 (0.2786942468403071)
| > loader_time: 0.0084 (0.008381317573601716)
[1m --> TIME: 2025-12-21 12:28:41 -- STEP: 2155/2207 -- GLOBAL_STEP: 18400[0m
| > loss_text_ce: 0.03116406314074993 (0.03098027495334569)
| > loss_mel_ce: 3.9434666633605957 (3.8245020461469683)
| > loss: 0.06210360676050186 (0.060241911300996064)
| > current_lr: 1e-05
| > step_time: 0.2905 (0.27876709453467685)
| > loader_time: 0.0083 (0.008375044377939872)
[1m --> TIME: 2025-12-21 12:29:00 -- STEP: 2205/2207 -- GLOBAL_STEP: 18450[0m
| > loss_text_ce: 0.03149457648396492 (0.030984393438818505)
| > loss_mel_ce: 3.63926362991333 (3.8233453620858744)
| > loss: 0.057355597615242004 (0.06022390246289928)
| > current_lr: 1e-05
| > step_time: 0.2661 (0.27879359316663666)
| > loader_time: 0.0079 (0.008370566530292528)
[1m > EVALUATION [0m
[1m--> EVAL PERFORMANCE[0m
| > avg_loader_time:[92m 0.07265385904827633 [0m(-0.0017059896443341466)
| > avg_loss_text_ce:[92m 0.02991473435650806 [0m(-0.00016766787833861346)
| > avg_loss_mel_ce:[92m 3.808113165803858 [0m(-0.014583056037490216)
| > avg_loss:[92m 3.8380279089953446 [0m(-0.014750712626689477)
> BEST MODEL : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/best_model_18452.pth
[4m[1m > EPOCH: 2/6[0m
--> /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000
[1m > TRAINING (2025-12-21 12:33:37) [0m
[1m --> TIME: 2025-12-21 12:33:58 -- STEP: 48/2207 -- GLOBAL_STEP: 18500[0m
| > loss_text_ce: 0.030328935012221336 (0.03124918350173782)
| > loss_mel_ce: 3.6012001037597656 (3.80705534418424)
| > loss: 0.05674264207482338 (0.05997350811958313)
| > current_lr: 9.5e-06
| > step_time: 0.2685 (0.2786668340365091)
| > loader_time: 0.0078 (0.008490150173505148)
[1m --> TIME: 2025-12-21 12:34:17 -- STEP: 98/2207 -- GLOBAL_STEP: 18550[0m
| > loss_text_ce: 0.029475053772330284 (0.031117105039254748)
| > loss_mel_ce: 3.6500818729400635 (3.802043304151418)
| > loss: 0.05749307572841644 (0.059893131370143016)
| > current_lr: 9.5e-06
| > step_time: 0.266 (0.27715023439757674)
| > loader_time: 0.0085 (0.008422819935545629)
[1m --> TIME: 2025-12-21 12:34:36 -- STEP: 148/2207 -- GLOBAL_STEP: 18600[0m
| > loss_text_ce: 0.029837869107723236 (0.03100120587073065)
| > loss_mel_ce: 4.100630760192871 (3.7966143134477974)
| > loss: 0.06453857570886612 (0.05980649250685363)
| > current_lr: 9.5e-06
| > step_time: 0.3043 (0.2776310814393531)
| > loader_time: 0.0081 (0.00839655947040867)
[1m --> TIME: 2025-12-21 12:34:55 -- STEP: 198/2207 -- GLOBAL_STEP: 18650[0m
| > loss_text_ce: 0.031300414353609085 (0.03097762127943111)
| > loss_mel_ce: 4.068465709686279 (3.808431427888196)
| > loss: 0.06405884772539139 (0.05999076642999143)
| > current_lr: 9.5e-06
| > step_time: 0.2677 (0.27762243723628477)
| > loader_time: 0.0077 (0.008417524472631586)
[1m --> TIME: 2025-12-21 12:35:14 -- STEP: 248/2207 -- GLOBAL_STEP: 18700[0m
| > loss_text_ce: 0.030506588518619537 (0.030958888443907904)
| > loss_mel_ce: 3.6368277072906494 (3.8152807095358447)
| > loss: 0.05730209872126579 (0.060097493787085815)
| > current_lr: 9.5e-06
| > step_time: 0.2867 (0.2775823743112623)
| > loader_time: 0.008 (0.00839974322626667)
[1m --> TIME: 2025-12-21 12:35:33 -- STEP: 298/2207 -- GLOBAL_STEP: 18750[0m
| > loss_text_ce: 0.03192504867911339 (0.030931028593706604)
| > loss_mel_ce: 3.560392141342163 (3.812415851842637)
| > loss: 0.056129954755306244 (0.06005229498415986)
| > current_lr: 9.5e-06
| > step_time: 0.2872 (0.2775322710907697)
| > loader_time: 0.0082 (0.008380010624059893)
[1m --> TIME: 2025-12-21 12:35:53 -- STEP: 348/2207 -- GLOBAL_STEP: 18800[0m
| > loss_text_ce: 0.030473029240965843 (0.030946150266875822)
| > loss_mel_ce: 4.083008766174316 (3.807196080684662)
| > loss: 0.0642731562256813 (0.059970972342309595)
| > current_lr: 9.5e-06
| > step_time: 0.2903 (0.27775018790672557)
| > loader_time: 0.0083 (0.008361554008790817)
[1m --> TIME: 2025-12-21 12:36:12 -- STEP: 398/2207 -- GLOBAL_STEP: 18850[0m
| > loss_text_ce: 0.032361965626478195 (0.03095491271203936)
| > loss_mel_ce: 3.980205774307251 (3.8102012603127178)
| > loss: 0.06269636750221252 (0.060018065156199826)
| > current_lr: 9.5e-06
| > step_time: 0.3053 (0.2776316464246817)
| > loader_time: 0.0083 (0.008362287852033301)
[1m --> TIME: 2025-12-21 12:36:31 -- STEP: 448/2207 -- GLOBAL_STEP: 18900[0m
| > loss_text_ce: 0.03188605234026909 (0.030948849365813658)
| > loss_mel_ce: 3.859243869781494 (3.8156256579927037)
| > loss: 0.06079890578985214 (0.06010272662388161)
| > current_lr: 9.5e-06
| > step_time: 0.2417 (0.27759859870587084)
| > loader_time: 0.0083 (0.008384346429790766)
[1m --> TIME: 2025-12-21 12:36:50 -- STEP: 498/2207 -- GLOBAL_STEP: 18950[0m
| > loss_text_ce: 0.03379356116056442 (0.03096674965043743)
| > loss_mel_ce: 3.9159152507781982 (3.8173977414288194)
| > loss: 0.06171419844031334 (0.06013069510549666)
| > current_lr: 9.5e-06
| > step_time: 0.2908 (0.2774093696870001)
| > loader_time: 0.0084 (0.008369181529585133)
[1m --> TIME: 2025-12-21 12:37:09 -- STEP: 548/2207 -- GLOBAL_STEP: 19000[0m
| > loss_text_ce: 0.031926073133945465 (0.030988967725939124)
| > loss_mel_ce: 3.838353395462036 (3.8168183700011586)
| > loss: 0.06047311797738075 (0.0601219896050374)
| > current_lr: 9.5e-06
| > step_time: 0.2895 (0.27765954236914653)
| > loader_time: 0.0084 (0.008360996733616734)
> CHECKPOINT : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/checkpoint_19000.pth
[1m --> TIME: 2025-12-21 12:37:53 -- STEP: 598/2207 -- GLOBAL_STEP: 19050[0m
| > loss_text_ce: 0.031638141721487045 (0.030977829051107467)
| > loss_mel_ce: 3.594619035720825 (3.8176495937200685)
| > loss: 0.05666026845574379 (0.06013480343398043)
| > current_lr: 9.5e-06
| > step_time: 0.2616 (0.2777918074840685)
| > loader_time: 0.0082 (0.008385346884711521)
[1m --> TIME: 2025-12-21 12:38:12 -- STEP: 648/2207 -- GLOBAL_STEP: 19100[0m
| > loss_text_ce: 0.029235929250717163 (0.03097168711749952)
| > loss_mel_ce: 3.780961513519287 (3.8170210332782175)
| > loss: 0.059534333646297455 (0.060124886210685895)
| > current_lr: 9.5e-06
| > step_time: 0.2874 (0.2776544944003776)
| > loader_time: 0.009 (0.008395500757076114)
[1m --> TIME: 2025-12-21 12:38:31 -- STEP: 698/2207 -- GLOBAL_STEP: 19150[0m
| > loss_text_ce: 0.028909575194120407 (0.03093947344232767)
| > loss_mel_ce: 3.942351818084717 (3.8171628662371697)
| > loss: 0.062050960958004 (0.0601265990097919)
| > current_lr: 9.5e-06
| > step_time: 0.2901 (0.2778345213236986)
| > loader_time: 0.0088 (0.008406749085915455)
[1m --> TIME: 2025-12-21 12:38:50 -- STEP: 748/2207 -- GLOBAL_STEP: 19200[0m
| > loss_text_ce: 0.031155679374933243 (0.030945500974010498)
| > loss_mel_ce: 3.9143738746643066 (3.816780863917447)
| > loss: 0.0616488978266716 (0.060120724395594814)
| > current_lr: 9.5e-06
| > step_time: 0.2689 (0.27826202616972096)
| > loader_time: 0.0081 (0.008410301437989907)
[1m --> TIME: 2025-12-21 12:39:09 -- STEP: 798/2207 -- GLOBAL_STEP: 19250[0m
| > loss_text_ce: 0.029925458133220673 (0.030928961321114)
| > loss_mel_ce: 3.9727892875671387 (3.8165484224644515)
| > loss: 0.06254241615533829 (0.06011683405622056)
| > current_lr: 9.5e-06
| > step_time: 0.2659 (0.2782398151574578)
| > loader_time: 0.0083 (0.00841741394578365)
[1m --> TIME: 2025-12-21 12:39:29 -- STEP: 848/2207 -- GLOBAL_STEP: 19300[0m
| > loss_text_ce: 0.0316762737929821 (0.030928653658297407)
| > loss_mel_ce: 3.722557783126831 (3.8153149249418723)
| > loss: 0.058659907430410385 (0.06009755585395362)
| > current_lr: 9.5e-06
| > step_time: 0.2698 (0.2783565099509257)
| > loader_time: 0.008 (0.008425706118907571)
[1m --> TIME: 2025-12-21 12:39:48 -- STEP: 898/2207 -- GLOBAL_STEP: 19350[0m
| > loss_text_ce: 0.03213763236999512 (0.03091699258322705)
| > loss_mel_ce: 3.923859119415283 (3.8141845504531346)
| > loss: 0.061812449246644974 (0.06007971154538321)
| > current_lr: 9.5e-06
| > step_time: 0.2666 (0.2783797982009853)
| > loader_time: 0.0083 (0.008426021626903646)
[1m --> TIME: 2025-12-21 12:40:07 -- STEP: 948/2207 -- GLOBAL_STEP: 19400[0m
| > loss_text_ce: 0.032980721443891525 (0.030903644741901867)
| > loss_mel_ce: 3.861752986907959 (3.8123343091473796)
| > loss: 0.060855213552713394 (0.060050592962102294)
| > current_lr: 9.5e-06
| > step_time: 0.2669 (0.2783489184540535)
| > loader_time: 0.0079 (0.008424995317740783)
[1m --> TIME: 2025-12-21 12:40:26 -- STEP: 998/2207 -- GLOBAL_STEP: 19450[0m
| > loss_text_ce: 0.03054073452949524 (0.03090524785071313)
| > loss_mel_ce: 4.015505313873291 (3.8130178864828803)
| > loss: 0.06321947276592255 (0.06006129889679934)
| > current_lr: 9.5e-06
| > step_time: 0.2901 (0.2784373222706552)
| > loader_time: 0.0091 (0.008492851305103494)
[1m --> TIME: 2025-12-21 12:40:46 -- STEP: 1048/2207 -- GLOBAL_STEP: 19500[0m
| > loss_text_ce: 0.030669590458273888 (0.030904853401904683)
| > loss_mel_ce: 3.722041606903076 (3.8116825097844798)
| > loss: 0.05863611400127411 (0.060040427482054216)
| > current_lr: 9.5e-06
| > step_time: 0.315 (0.27876375604221837)
| > loader_time: 0.0088 (0.008539806567985593)
[1m --> TIME: 2025-12-21 12:41:05 -- STEP: 1098/2207 -- GLOBAL_STEP: 19550[0m
| > loss_text_ce: 0.03170514851808548 (0.0308953984343669)
| > loss_mel_ce: 3.862687349319458 (3.8120278489177126)
| > loss: 0.060849882662296295 (0.06004567566465159)
| > current_lr: 9.5e-06
| > step_time: 0.2922 (0.27895870430221975)
| > loader_time: 0.0111 (0.00858209975647362)
[1m --> TIME: 2025-12-21 12:41:25 -- STEP: 1148/2207 -- GLOBAL_STEP: 19600[0m
| > loss_text_ce: 0.031861841678619385 (0.030892794122692288)
| > loss_mel_ce: 3.8248915672302246 (3.811262029596322)
| > loss: 0.06026177108287811 (0.06003366905274291)
| > current_lr: 9.5e-06
| > step_time: 0.2674 (0.27946521783124284)
| > loader_time: 0.0098 (0.00863001051680136)
[1m --> TIME: 2025-12-21 12:41:44 -- STEP: 1198/2207 -- GLOBAL_STEP: 19650[0m
| > loss_text_ce: 0.031165527179837227 (0.030906598595202665)
| > loss_mel_ce: 3.6397016048431396 (3.811939091037629)
| > loss: 0.0573573000729084 (0.06004446385139117)
| > current_lr: 9.5e-06
| > step_time: 0.2662 (0.27954068605808274)
| > loader_time: 0.0096 (0.00866090156797176)
[1m --> TIME: 2025-12-21 12:42:04 -- STEP: 1248/2207 -- GLOBAL_STEP: 19700[0m
| > loss_text_ce: 0.030565952882170677 (0.030907038918648582)
| > loss_mel_ce: 4.019072532653809 (3.812627971554414)
| > loss: 0.06327559798955917 (0.0600552344843984)
| > current_lr: 9.5e-06
| > step_time: 0.2681 (0.27955424575469423)
| > loader_time: 0.0098 (0.008690576713818758)
[1m --> TIME: 2025-12-21 12:42:23 -- STEP: 1298/2207 -- GLOBAL_STEP: 19750[0m
| > loss_text_ce: 0.033700186759233475 (0.030912928441065853)
| > loss_mel_ce: 3.9421839714050293 (3.8117501079944325)
| > loss: 0.062123190611600876 (0.060041609896097234)
| > current_lr: 9.5e-06
| > step_time: 0.2703 (0.2795349096847794)
| > loader_time: 0.0094 (0.008714687842617403)
[1m --> TIME: 2025-12-21 12:42:42 -- STEP: 1348/2207 -- GLOBAL_STEP: 19800[0m
| > loss_text_ce: 0.031137095764279366 (0.030893902244593516)
| > loss_mel_ce: 4.025448799133301 (3.8119540738068984)
| > loss: 0.06338415294885635 (0.06004449958948955)
| > current_lr: 9.5e-06
| > step_time: 0.2629 (0.27941010755318224)
| > loader_time: 0.0101 (0.008745452770493491)
[1m --> TIME: 2025-12-21 12:43:01 -- STEP: 1398/2207 -- GLOBAL_STEP: 19850[0m
| > loss_text_ce: 0.031674209982156754 (0.030885134482535083)
| > loss_mel_ce: 3.708361864089966 (3.8112786298828234)
| > loss: 0.05843806266784668 (0.06003380877009789)
| > current_lr: 9.5e-06
| > step_time: 0.2671 (0.2793427293051639)
| > loader_time: 0.0114 (0.008763332223687543)
[1m --> TIME: 2025-12-21 12:43:20 -- STEP: 1448/2207 -- GLOBAL_STEP: 19900[0m
| > loss_text_ce: 0.030328992754220963 (0.03087736206329865)
| > loss_mel_ce: 3.821791172027588 (3.8124404337195403)
| > loss: 0.0601893775165081 (0.06005184050875803)
| > current_lr: 9.5e-06
| > step_time: 0.269 (0.27929688224476384)
| > loader_time: 0.0094 (0.008786394615858279)
[1m --> TIME: 2025-12-21 12:43:39 -- STEP: 1498/2207 -- GLOBAL_STEP: 19950[0m
| > loss_text_ce: 0.03165443241596222 (0.03087984636619946)
| > loss_mel_ce: 3.698460578918457 (3.8117194285539187)
| > loss: 0.05828304588794708 (0.06004061362394026)
| > current_lr: 9.5e-06
| > step_time: 0.266 (0.2792639877194562)
| > loader_time: 0.0098 (0.008800205305835413)
[1m --> TIME: 2025-12-21 12:43:59 -- STEP: 1548/2207 -- GLOBAL_STEP: 20000[0m
| > loss_text_ce: 0.033760420978069305 (0.030875577592562875)
| > loss_mel_ce: 3.544578790664673 (3.8111829480459525)
| > loss: 0.05591154843568802 (0.06003216441085403)
| > current_lr: 9.5e-06
| > step_time: 0.2706 (0.27927306163526666)
| > loader_time: 0.0093 (0.008809470237071489)
> CHECKPOINT : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/checkpoint_20000.pth
[1m --> TIME: 2025-12-21 12:44:43 -- STEP: 1598/2207 -- GLOBAL_STEP: 20050[0m
| > loss_text_ce: 0.031220965087413788 (0.03087228273644167)
| > loss_mel_ce: 4.057143688201904 (3.811688291116411)
| > loss: 0.06388069689273834 (0.06004000892011782)
| > current_lr: 9.5e-06
| > step_time: 0.298 (0.27934688292397175)
| > loader_time: 0.0089 (0.008850628055529532)
[1m --> TIME: 2025-12-21 12:45:02 -- STEP: 1648/2207 -- GLOBAL_STEP: 20100[0m
| > loss_text_ce: 0.03015831671655178 (0.030872867847295496)
| > loss_mel_ce: 3.5962467193603516 (3.811391503342147)
| > loss: 0.056662578135728836 (0.060035380760876875)
| > current_lr: 9.5e-06
| > step_time: 0.2671 (0.27928056546206664)
| > loader_time: 0.0087 (0.008861764252764507)
[1m --> TIME: 2025-12-21 12:45:21 -- STEP: 1698/2207 -- GLOBAL_STEP: 20150[0m
| > loss_text_ce: 0.029446423053741455 (0.030872317737470806)
| > loss_mel_ce: 3.346419334411621 (3.8103991141729274)
| > loss: 0.052747901529073715 (0.06001986607742464)
| > current_lr: 9.5e-06
| > step_time: 0.2872 (0.27919970594952087)
| > loader_time: 0.0084 (0.008857855808046877)
[1m --> TIME: 2025-12-21 12:45:40 -- STEP: 1748/2207 -- GLOBAL_STEP: 20200[0m
| > loss_text_ce: 0.031714435666799545 (0.030864485928435006)
| > loss_mel_ce: 3.784182071685791 (3.809945785480997)
| > loss: 0.05962338298559189 (0.060012660449630745)
| > current_lr: 9.5e-06
| > step_time: 0.2625 (0.2791893772729752)
| > loader_time: 0.0087 (0.008868558728722192)
[1m --> TIME: 2025-12-21 12:46:00 -- STEP: 1798/2207 -- GLOBAL_STEP: 20250[0m
| > loss_text_ce: 0.030244555324316025 (0.030862935044874206)
| > loss_mel_ce: 3.833282232284546 (3.8102427565879102)
| > loss: 0.060367606580257416 (0.06001727638954713)
| > current_lr: 9.5e-06
| > step_time: 0.2691 (0.27920447560120526)
| > loader_time: 0.0092 (0.008864384604508141)
[1m --> TIME: 2025-12-21 12:46:19 -- STEP: 1848/2207 -- GLOBAL_STEP: 20300[0m
| > loss_text_ce: 0.030655691400170326 (0.03086753052788347)
| > loss_mel_ce: 3.7304117679595947 (3.8096691434517567)
| > loss: 0.0587666779756546 (0.06000838548477207)
| > current_lr: 9.5e-06
| > step_time: 0.2672 (0.2791787206610565)
| > loader_time: 0.0095 (0.008876425621313447)
[1m --> TIME: 2025-12-21 12:46:38 -- STEP: 1898/2207 -- GLOBAL_STEP: 20350[0m
| > loss_text_ce: 0.029659613966941833 (0.030867560519331438)
| > loss_mel_ce: 3.8518874645233154 (3.80958583947856)
| > loss: 0.060649171471595764 (0.060007084327876506)
| > current_lr: 9.5e-06
| > step_time: 0.2905 (0.2792153088385744)
| > loader_time: 0.0089 (0.008891735865770824)
[1m --> TIME: 2025-12-21 12:46:57 -- STEP: 1948/2207 -- GLOBAL_STEP: 20400[0m
| > loss_text_ce: 0.029927784577012062 (0.03085980789154554)
| > loss_mel_ce: 3.7252941131591797 (3.809625461116219)
| > loss: 0.05867534130811691 (0.06000758227710545)
| > current_lr: 9.5e-06
| > step_time: 0.2695 (0.2792228915608154)
| > loader_time: 0.0095 (0.008891950152982673)
[1m --> TIME: 2025-12-21 12:47:17 -- STEP: 1998/2207 -- GLOBAL_STEP: 20450[0m
| > loss_text_ce: 0.03319452330470085 (0.030859624606196408)
| > loss_mel_ce: 3.891561508178711 (3.809653092194367)
| > loss: 0.061324313282966614 (0.06000801114676831)
| > current_lr: 9.5e-06
| > step_time: 0.266 (0.2792890162558651)
| > loader_time: 0.0083 (0.00888882241807542)
[1m --> TIME: 2025-12-21 12:47:36 -- STEP: 2048/2207 -- GLOBAL_STEP: 20500[0m
| > loss_text_ce: 0.029439985752105713 (0.03084829578165227)
| > loss_mel_ce: 3.90108585357666 (3.8098780608270317)
| > loss: 0.06141446530818939 (0.060011349260093994)
| > current_lr: 9.5e-06
| > step_time: 0.24 (0.2793646088102837)
| > loader_time: 0.0088 (0.00888665369711816)
[1m --> TIME: 2025-12-21 12:47:55 -- STEP: 2098/2207 -- GLOBAL_STEP: 20550[0m
| > loss_text_ce: 0.028744757175445557 (0.03084498683738214)
| > loss_mel_ce: 3.8752195835113525 (3.808677380828203)
| > loss: 0.060999441891908646 (0.059992536937940454)
| > current_lr: 9.5e-06
| > step_time: 0.2661 (0.2794211697192055)
| > loader_time: 0.0095 (0.008883777405672462)
[1m --> TIME: 2025-12-21 12:48:15 -- STEP: 2148/2207 -- GLOBAL_STEP: 20600[0m
| > loss_text_ce: 0.032124441117048264 (0.030849136339302835)
| > loss_mel_ce: 3.6190860271453857 (3.8092889245217965)
| > loss: 0.05705016478896141 (0.06000215714897965)
| > current_lr: 9.5e-06
| > step_time: 0.291 (0.2793659517884924)
| > loader_time: 0.0096 (0.008893648450379042)
[1m --> TIME: 2025-12-21 12:48:35 -- STEP: 2198/2207 -- GLOBAL_STEP: 20650[0m
| > loss_text_ce: 0.03118201717734337 (0.030848595168817793)
| > loss_mel_ce: 3.7164382934570312 (3.807630991914036)
| > loss: 0.05855656787753105 (0.059976243487710926)
| > current_lr: 9.5e-06
| > step_time: 0.3316 (0.2800430354473267)
| > loader_time: 0.0084 (0.009042994340405454)
[1m > EVALUATION [0m
[1m--> EVAL PERFORMANCE[0m
| > avg_loader_time:[91m 0.12325521095378979 [0m(+0.05060135190551346)
| > avg_loss_text_ce:[92m 0.02977932435837952 [0m(-0.00013540999812854115)
| > avg_loss_mel_ce:[92m 3.7917954229019784 [0m(-0.016317742901879484)
| > avg_loss:[92m 3.821574736285854 [0m(-0.016453172709490538)
> BEST MODEL : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/best_model_20659.pth
|