YomnaGharib commited on
Commit
8d31cf0
·
verified ·
1 Parent(s): 2a99c63

Checkpoint at step 18452

Browse files
best_model.pth CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:78a7ac87b2d769eabc272cd96fb06c15d645132b3242337aa9d2a3c1a548fc13
3
  size 5607926677
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9990a628a13fd1857917613f487a7c0bace7970ba47398c0e538d5fc94569ce8
3
  size 5607926677
best_model_18452.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9990a628a13fd1857917613f487a7c0bace7970ba47398c0e538d5fc94569ce8
3
+ size 5607926677
checkpoint_17000.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a77ae6f50f35f1eca2353118df615d9b61c83dd17bb10a8421866b240b4e5be1
3
+ size 5607926677
checkpoint_18000.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ba2b7ececdf07407775447e43aa4cfc644e3a1db19b482b94cdc83ff4e48ce66
3
+ size 5607926677
events.out.tfevents.1766319018.107952f8efb6.1181.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:c9011e10a3bbf5c7124614cee172edacdde5e0720a5714074060739d30686db5
3
- size 6305
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f28e76b087d0f6f174f1b8a955454adcbf97777bac3f932975dfe81d45f4164f
3
+ size 15677
trainer_0_log.txt CHANGED
@@ -32,3 +32,430 @@
32
  | > avg_loss: 3.852778621622034 (+0)
33
 
34
  > BEST MODEL : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/best_model_16245.pth
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
32
  | > avg_loss: 3.852778621622034 (+0)
33
 
34
  > BEST MODEL : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/best_model_16245.pth
35
+
36
+  > EPOCH: 1/6
37
+ --> /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000
38
+
39
+  > TRAINING (2025-12-21 12:14:05) 
40
+
41
+  --> TIME: 2025-12-21 12:14:11 -- STEP: 5/2207 -- GLOBAL_STEP: 16250
42
+ | > loss_text_ce: 0.03219795227050781 (0.03218773603439331)
43
+ | > loss_mel_ce: 4.446129322052002 (4.046912813186646)
44
+ | > loss: 0.06997386366128922 (0.0637359470129013)
45
+ | > current_lr: 1e-05
46
+ | > step_time: 0.2695 (0.2764430522918701)
47
+ | > loader_time: 0.0089 (0.010691452026367187)
48
+
49
+
50
+  --> TIME: 2025-12-21 12:14:30 -- STEP: 55/2207 -- GLOBAL_STEP: 16300
51
+ | > loss_text_ce: 0.03195399418473244 (0.030980682135982946)
52
+ | > loss_mel_ce: 3.563579797744751 (3.869967087832364)
53
+ | > loss: 0.05618021637201309 (0.0609523091126572)
54
+ | > current_lr: 1e-05
55
+ | > step_time: 0.2691 (0.27666049003601073)
56
+ | > loader_time: 0.009 (0.01010052507573908)
57
+
58
+
59
+  --> TIME: 2025-12-21 12:14:49 -- STEP: 105/2207 -- GLOBAL_STEP: 16350
60
+ | > loss_text_ce: 0.03152564913034439 (0.03094851766668615)
61
+ | > loss_mel_ce: 3.851048707962036 (3.864614899953206)
62
+ | > loss: 0.06066522374749184 (0.060868178449925925)
63
+ | > current_lr: 1e-05
64
+ | > step_time: 0.2912 (0.27978511991954985)
65
+ | > loader_time: 0.0092 (0.009624846776326497)
66
+
67
+
68
+  --> TIME: 2025-12-21 12:15:09 -- STEP: 155/2207 -- GLOBAL_STEP: 16400
69
+ | > loss_text_ce: 0.029248259961605072 (0.03089537698655359)
70
+ | > loss_mel_ce: 3.628293514251709 (3.846851090461977)
71
+ | > loss: 0.05714908987283707 (0.06058978856571259)
72
+ | > current_lr: 1e-05
73
+ | > step_time: 0.2925 (0.28088303073760007)
74
+ | > loader_time: 0.0081 (0.009488651829381148)
75
+
76
+
77
+  --> TIME: 2025-12-21 12:15:28 -- STEP: 205/2207 -- GLOBAL_STEP: 16450
78
+ | > loss_text_ce: 0.032587237656116486 (0.03095879358489339)
79
+ | > loss_mel_ce: 3.7422239780426025 (3.8365380601185124)
80
+ | > loss: 0.05898142606019974 (0.060429638442469805)
81
+ | > current_lr: 1e-05
82
+ | > step_time: 0.2908 (0.2808621534487096)
83
+ | > loader_time: 0.0082 (0.009342936771671948)
84
+
85
+
86
+  --> TIME: 2025-12-21 12:15:47 -- STEP: 255/2207 -- GLOBAL_STEP: 16500
87
+ | > loss_text_ce: 0.03305845335125923 (0.030955966971084183)
88
+ | > loss_mel_ce: 3.9918406009674072 (3.8380968393064014)
89
+ | > loss: 0.06288904696702957 (0.06045395013164071)
90
+ | > grad_norm: 0 (0.0)
91
+ | > current_lr: 1e-05
92
+ | > step_time: 0.2862 (0.2800759567933923)
93
+ | > loader_time: 0.0086 (0.009221519208421897)
94
+
95
+
96
+  --> TIME: 2025-12-21 12:16:06 -- STEP: 305/2207 -- GLOBAL_STEP: 16550
97
+ | > loss_text_ce: 0.031405024230480194 (0.030954475768032622)
98
+ | > loss_mel_ce: 3.559880495071411 (3.8379844337213234)
99
+ | > loss: 0.0561138354241848 (0.06045217049903557)
100
+ | > current_lr: 1e-05
101
+ | > step_time: 0.2679 (0.2793264326502066)
102
+ | > loader_time: 0.0081 (0.00907517104852395)
103
+
104
+
105
+  --> TIME: 2025-12-21 12:16:25 -- STEP: 355/2207 -- GLOBAL_STEP: 16600
106
+ | > loss_text_ce: 0.03148230165243149 (0.03097983851084407)
107
+ | > loss_mel_ce: 3.8917019367218018 (3.8419334841446138)
108
+ | > loss: 0.061299752444028854 (0.06051427065066888)
109
+ | > current_lr: 1e-05
110
+ | > step_time: 0.2688 (0.27979512684781777)
111
+ | > loader_time: 0.0079 (0.008994941980066435)
112
+
113
+
114
+  --> TIME: 2025-12-21 12:16:45 -- STEP: 405/2207 -- GLOBAL_STEP: 16650
115
+ | > loss_text_ce: 0.030755072832107544 (0.03096303568210131)
116
+ | > loss_mel_ce: 3.5810444355010986 (3.837673578733279)
117
+ | > loss: 0.05643436685204506 (0.060447447075520036)
118
+ | > current_lr: 1e-05
119
+ | > step_time: 0.2877 (0.27977862004880527)
120
+ | > loader_time: 0.0082 (0.008914067421430426)
121
+
122
+
123
+  --> TIME: 2025-12-21 12:17:04 -- STEP: 455/2207 -- GLOBAL_STEP: 16700
124
+ | > loss_text_ce: 0.032188721001148224 (0.030962336661068946)
125
+ | > loss_mel_ce: 3.918936252593994 (3.8396105085100447)
126
+ | > loss: 0.0617363266646862 (0.06047770069031925)
127
+ | > current_lr: 1e-05
128
+ | > step_time: 0.2866 (0.27930216998844304)
129
+ | > loader_time: 0.0084 (0.008828892026628768)
130
+
131
+
132
+  --> TIME: 2025-12-21 12:17:23 -- STEP: 505/2207 -- GLOBAL_STEP: 16750
133
+ | > loss_text_ce: 0.0326453261077404 (0.03099347592359132)
134
+ | > loss_mel_ce: 3.800915241241455 (3.8388911927100455)
135
+ | > loss: 0.059899382293224335 (0.06046694796716813)
136
+ | > current_lr: 1e-05
137
+ | > step_time: 0.2867 (0.27900417252342247)
138
+ | > loader_time: 0.0081 (0.008783628916976477)
139
+
140
+
141
+  --> TIME: 2025-12-21 12:17:42 -- STEP: 555/2207 -- GLOBAL_STEP: 16800
142
+ | > loss_text_ce: 0.030252568423748016 (0.030994508647032685)
143
+ | > loss_mel_ce: 3.620875597000122 (3.8386023968189686)
144
+ | > loss: 0.05704887583851814 (0.06046245165907585)
145
+ | > current_lr: 1e-05
146
+ | > step_time: 0.2862 (0.2790829800270695)
147
+ | > loader_time: 0.0077 (0.008722289832862648)
148
+
149
+
150
+  --> TIME: 2025-12-21 12:18:01 -- STEP: 605/2207 -- GLOBAL_STEP: 16850
151
+ | > loss_text_ce: 0.03053409792482853 (0.030986474165000208)
152
+ | > loss_mel_ce: 4.023441791534424 (3.8384223346867836)
153
+ | > loss: 0.06334337592124939 (0.060459512675350366)
154
+ | > current_lr: 1e-05
155
+ | > step_time: 0.2666 (0.2788679410603417)
156
+ | > loader_time: 0.0078 (0.008667540353191784)
157
+
158
+
159
+  --> TIME: 2025-12-21 12:18:20 -- STEP: 655/2207 -- GLOBAL_STEP: 16900
160
+ | > loss_text_ce: 0.030561409890651703 (0.03099969866337212)
161
+ | > loss_mel_ce: 3.8719499111175537 (3.8380026737242257)
162
+ | > loss: 0.06097673997282982 (0.060453162103436375)
163
+ | > current_lr: 1e-05
164
+ | > step_time: 0.2663 (0.27882370475594254)
165
+ | > loader_time: 0.0089 (0.008689285962636233)
166
+
167
+
168
+  --> TIME: 2025-12-21 12:18:39 -- STEP: 705/2207 -- GLOBAL_STEP: 16950
169
+ | > loss_text_ce: 0.031610529869794846 (0.03098056488637383)
170
+ | > loss_mel_ce: 3.83477520942688 (3.838375516647988)
171
+ | > loss: 0.06041227653622627 (0.06045868881839387)
172
+ | > current_lr: 1e-05
173
+ | > step_time: 0.291 (0.27892449933586333)
174
+ | > loader_time: 0.0082 (0.008669162642025784)
175
+
176
+
177
+  --> TIME: 2025-12-21 12:18:58 -- STEP: 755/2207 -- GLOBAL_STEP: 17000
178
+ | > loss_text_ce: 0.03250373899936676 (0.030986027972192954)
179
+ | > loss_mel_ce: 3.5191447734832764 (3.835920715332031)
180
+ | > loss: 0.055494509637355804 (0.060420417894196034)
181
+ | > current_lr: 1e-05
182
+ | > step_time: 0.2686 (0.27887622504834264)
183
+ | > loader_time: 0.0083 (0.00863919352853536)
184
+
185
+
186
+ > CHECKPOINT : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/checkpoint_17000.pth
187
+
188
+  --> TIME: 2025-12-21 12:19:41 -- STEP: 805/2207 -- GLOBAL_STEP: 17050
189
+ | > loss_text_ce: 0.02891026996076107 (0.030961797997262907)
190
+ | > loss_mel_ce: 3.8255555629730225 (3.8371545169664465)
191
+ | > loss: 0.06022602692246437 (0.060439317442060256)
192
+ | > current_lr: 1e-05
193
+ | > step_time: 0.2688 (0.2789103673852005)
194
+ | > loader_time: 0.0077 (0.008619996183407246)
195
+
196
+
197
+  --> TIME: 2025-12-21 12:20:00 -- STEP: 855/2207 -- GLOBAL_STEP: 17100
198
+ | > loss_text_ce: 0.030247919261455536 (0.03095182662949576)
199
+ | > loss_mel_ce: 3.7221128940582275 (3.8346250587039523)
200
+ | > loss: 0.05863063782453537 (0.06039963886577483)
201
+ | > current_lr: 1e-05
202
+ | > step_time: 0.2612 (0.2788665866294099)
203
+ | > loader_time: 0.0085 (0.008600194272939227)
204
+
205
+
206
+  --> TIME: 2025-12-21 12:20:20 -- STEP: 905/2207 -- GLOBAL_STEP: 17150
207
+ | > loss_text_ce: 0.03263920545578003 (0.030970429752577733)
208
+ | > loss_mel_ce: 3.8033525943756104 (3.8346228056849694)
209
+ | > loss: 0.05993737280368805 (0.06039989432988904)
210
+ | > current_lr: 1e-05
211
+ | > step_time: 0.2658 (0.2798094433315545)
212
+ | > loader_time: 0.0085 (0.008578953031676908)
213
+
214
+
215
+  --> TIME: 2025-12-21 12:20:39 -- STEP: 955/2207 -- GLOBAL_STEP: 17200
216
+ | > loss_text_ce: 0.03272069990634918 (0.030973391158808587)
217
+ | > loss_mel_ce: 3.866997718811035 (3.8342152485672716)
218
+ | > loss: 0.060933101922273636 (0.06039357252848086)
219
+ | > current_lr: 1e-05
220
+ | > step_time: 0.2957 (0.2798374602931952)
221
+ | > loader_time: 0.0083 (0.008563283850385253)
222
+
223
+
224
+  --> TIME: 2025-12-21 12:20:58 -- STEP: 1005/2207 -- GLOBAL_STEP: 17250
225
+ | > loss_text_ce: 0.03107460029423237 (0.03097570813191471)
226
+ | > loss_mel_ce: 3.7447755336761475 (3.8325845606884554)
227
+ | > loss: 0.058997657150030136 (0.06036812922551264)
228
+ | > current_lr: 1e-05
229
+ | > step_time: 0.2864 (0.279714151164192)
230
+ | > loader_time: 0.0085 (0.008561972954973057)
231
+
232
+
233
+  --> TIME: 2025-12-21 12:21:17 -- STEP: 1055/2207 -- GLOBAL_STEP: 17300
234
+ | > loss_text_ce: 0.030682062730193138 (0.030976020410586308)
235
+ | > loss_mel_ce: 3.643967866897583 (3.8322222506265504)
236
+ | > loss: 0.0574164055287838 (0.06036247299957615)
237
+ | > current_lr: 1e-05
238
+ | > step_time: 0.2883 (0.27947213502856744)
239
+ | > loader_time: 0.0083 (0.008548689340528149)
240
+
241
+
242
+  --> TIME: 2025-12-21 12:21:36 -- STEP: 1105/2207 -- GLOBAL_STEP: 17350
243
+ | > loss_text_ce: 0.032925937324762344 (0.03097810615106945)
244
+ | > loss_mel_ce: 3.7915194034576416 (3.8325317693511827)
245
+ | > loss: 0.05975695699453354 (0.060367341838541075)
246
+ | > current_lr: 1e-05
247
+ | > step_time: 0.2868 (0.27930227965790716)
248
+ | > loader_time: 0.0077 (0.008524640760810121)
249
+
250
+
251
+  --> TIME: 2025-12-21 12:21:55 -- STEP: 1155/2207 -- GLOBAL_STEP: 17400
252
+ | > loss_text_ce: 0.03101305104792118 (0.03097473747576728)
253
+ | > loss_mel_ce: 3.926995038986206 (3.830362285473646)
254
+ | > loss: 0.0618438757956028 (0.060333391017728034)
255
+ | > current_lr: 1e-05
256
+ | > step_time: 0.2953 (0.27925692182598655)
257
+ | > loader_time: 0.0081 (0.00850323412841532)
258
+
259
+
260
+  --> TIME: 2025-12-21 12:22:15 -- STEP: 1205/2207 -- GLOBAL_STEP: 17450
261
+ | > loss_text_ce: 0.031288035213947296 (0.030977744015544283)
262
+ | > loss_mel_ce: 4.4484028816223145 (3.831367015838623)
263
+ | > loss: 0.06999517232179642 (0.060349136900110366)
264
+ | > current_lr: 1e-05
265
+ | > step_time: 0.2619 (0.27930984556427596)
266
+ | > loader_time: 0.0081 (0.008497552159416222)
267
+
268
+
269
+  --> TIME: 2025-12-21 12:22:34 -- STEP: 1255/2207 -- GLOBAL_STEP: 17500
270
+ | > loss_text_ce: 0.029901018366217613 (0.030980763323159806)
271
+ | > loss_mel_ce: 3.783792734146118 (3.83137526474151)
272
+ | > loss: 0.0595889650285244 (0.06034931295599596)
273
+ | > current_lr: 1e-05
274
+ | > step_time: 0.302 (0.2792716653223529)
275
+ | > loader_time: 0.0089 (0.008480142122245867)
276
+
277
+
278
+  --> TIME: 2025-12-21 12:22:53 -- STEP: 1305/2207 -- GLOBAL_STEP: 17550
279
+ | > loss_text_ce: 0.030708273872733116 (0.030976130990347187)
280
+ | > loss_mel_ce: 3.928159236907959 (3.831721293606521)
281
+ | > loss: 0.061857305467128754 (0.06035464728015593)
282
+ | > current_lr: 1e-05
283
+ | > step_time: 0.2912 (0.2792421129014756)
284
+ | > loader_time: 0.008 (0.008464955735480633)
285
+
286
+
287
+  --> TIME: 2025-12-21 12:23:12 -- STEP: 1355/2207 -- GLOBAL_STEP: 17600
288
+ | > loss_text_ce: 0.030456705018877983 (0.030980337833181517)
289
+ | > loss_mel_ce: 3.610086441040039 (3.8301265816846897)
290
+ | > loss: 0.05688348785042763 (0.06032979563428467)
291
+ | > current_lr: 1e-05
292
+ | > step_time: 0.289 (0.27909282828609)
293
+ | > loader_time: 0.0084 (0.008450167117523512)
294
+
295
+
296
+  --> TIME: 2025-12-21 12:23:31 -- STEP: 1405/2207 -- GLOBAL_STEP: 17650
297
+ | > loss_text_ce: 0.0315256342291832 (0.03096651314787593)
298
+ | > loss_mel_ce: 3.5720529556274414 (3.8309017710838456)
299
+ | > loss: 0.056305915117263794 (0.06034169195756794)
300
+ | > current_lr: 1e-05
301
+ | > step_time: 0.2542 (0.27900979221927735)
302
+ | > loader_time: 0.0081 (0.00843814344168556)
303
+
304
+
305
+  --> TIME: 2025-12-21 12:23:50 -- STEP: 1455/2207 -- GLOBAL_STEP: 17700
306
+ | > loss_text_ce: 0.029983242973685265 (0.030972939362282195)
307
+ | > loss_mel_ce: 3.7503371238708496 (3.830120104366971)
308
+ | > loss: 0.0590675063431263 (0.0603295788296924)
309
+ | > current_lr: 1e-05
310
+ | > step_time: 0.2862 (0.279018137299318)
311
+ | > loader_time: 0.0086 (0.008426600551277487)
312
+
313
+
314
+  --> TIME: 2025-12-21 12:24:09 -- STEP: 1505/2207 -- GLOBAL_STEP: 17750
315
+ | > loss_text_ce: 0.03367031365633011 (0.0309653219120249)
316
+ | > loss_mel_ce: 3.723371744155884 (3.830078932613233)
317
+ | > loss: 0.05870378389954567 (0.06032881649625658)
318
+ | > current_lr: 1e-05
319
+ | > step_time: 0.2685 (0.27902523592064943)
320
+ | > loader_time: 0.0079 (0.00841875092135711)
321
+
322
+
323
+  --> TIME: 2025-12-21 12:24:28 -- STEP: 1555/2207 -- GLOBAL_STEP: 17800
324
+ | > loss_text_ce: 0.02944960817694664 (0.03096569205403711)
325
+ | > loss_mel_ce: 4.062541961669922 (3.8309391785281264)
326
+ | > loss: 0.06393736600875854 (0.060342263636385896)
327
+ | > current_lr: 1e-05
328
+ | > step_time: 0.2902 (0.2789708040918187)
329
+ | > loader_time: 0.0081 (0.008411988568075979)
330
+
331
+
332
+  --> TIME: 2025-12-21 12:24:47 -- STEP: 1605/2207 -- GLOBAL_STEP: 17850
333
+ | > loss_text_ce: 0.029368538409471512 (0.03096987812481008)
334
+ | > loss_mel_ce: 3.616783857345581 (3.8293144246870856)
335
+ | > loss: 0.05697113275527954 (0.060316942265471934)
336
+ | > current_lr: 1e-05
337
+ | > step_time: 0.2896 (0.27894843597649793)
338
+ | > loader_time: 0.0077 (0.008404218667764156)
339
+
340
+
341
+  --> TIME: 2025-12-21 12:25:06 -- STEP: 1655/2207 -- GLOBAL_STEP: 17900
342
+ | > loss_text_ce: 0.02917139232158661 (0.030966688455096543)
343
+ | > loss_mel_ce: 3.955526351928711 (3.8295799357653384)
344
+ | > loss: 0.06226090341806412 (0.06032104103676862)
345
+ | > current_lr: 1e-05
346
+ | > step_time: 0.2508 (0.2788965385125842)
347
+ | > loader_time: 0.008 (0.008392788348241155)
348
+
349
+
350
+  --> TIME: 2025-12-21 12:25:25 -- STEP: 1705/2207 -- GLOBAL_STEP: 17950
351
+ | > loss_text_ce: 0.036614060401916504 (0.030967657537885076)
352
+ | > loss_mel_ce: 3.92380690574646 (3.8291027352956735)
353
+ | > loss: 0.06188157945871353 (0.06031359991938138)
354
+ | > current_lr: 1e-05
355
+ | > step_time: 0.4298 (0.2789531161009043)
356
+ | > loader_time: 0.0081 (0.008383412235293566)
357
+
358
+
359
+  --> TIME: 2025-12-21 12:25:44 -- STEP: 1755/2207 -- GLOBAL_STEP: 18000
360
+ | > loss_text_ce: 0.031244533136487007 (0.030965659205942412)
361
+ | > loss_mel_ce: 3.8533778190612793 (3.8291114841091667)
362
+ | > loss: 0.060697223991155624 (0.06031370539525975)
363
+ | > current_lr: 1e-05
364
+ | > step_time: 0.2861 (0.2788446566318174)
365
+ | > loader_time: 0.0081 (0.008380908911724024)
366
+
367
+
368
+ > CHECKPOINT : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/checkpoint_18000.pth
369
+
370
+  --> TIME: 2025-12-21 12:26:27 -- STEP: 1805/2207 -- GLOBAL_STEP: 18050
371
+ | > loss_text_ce: 0.02912035584449768 (0.03096764023696947)
372
+ | > loss_mel_ce: 3.675009250640869 (3.8278143662164745)
373
+ | > loss: 0.05787702649831772 (0.060293468884566485)
374
+ | > current_lr: 1e-05
375
+ | > step_time: 0.2666 (0.2788151725177289)
376
+ | > loader_time: 0.0083 (0.008385370840986658)
377
+
378
+
379
+  --> TIME: 2025-12-21 12:26:46 -- STEP: 1855/2207 -- GLOBAL_STEP: 18100
380
+ | > loss_text_ce: 0.030438384041190147 (0.030971936526083238)
381
+ | > loss_mel_ce: 3.6038551330566406 (3.827252550587821)
382
+ | > loss: 0.056785836815834045 (0.06028475765028411)
383
+ | > grad_norm: 0 (0.0)
384
+ | > current_lr: 1e-05
385
+ | > step_time: 0.3188 (0.2787536723915779)
386
+ | > loader_time: 0.0083 (0.00838110569031089)
387
+
388
+
389
+  --> TIME: 2025-12-21 12:27:05 -- STEP: 1905/2207 -- GLOBAL_STEP: 18150
390
+ | > loss_text_ce: 0.029724599793553352 (0.03097888275668064)
391
+ | > loss_mel_ce: 3.7439305782318115 (3.827816226914173)
392
+ | > loss: 0.058963362127542496 (0.0602936736300705)
393
+ | > current_lr: 1e-05
394
+ | > step_time: 0.2651 (0.27881914574330247)
395
+ | > loader_time: 0.0086 (0.008379007574767254)
396
+
397
+
398
+  --> TIME: 2025-12-21 12:27:24 -- STEP: 1955/2207 -- GLOBAL_STEP: 18200
399
+ | > loss_text_ce: 0.030410582199692726 (0.03097874500200419)
400
+ | > loss_mel_ce: 3.681417226791382 (3.826422531708427)
401
+ | > loss: 0.05799730867147446 (0.06027189498133671)
402
+ | > current_lr: 1e-05
403
+ | > step_time: 0.2678 (0.2787622943253773)
404
+ | > loader_time: 0.0085 (0.008377984844510192)
405
+
406
+
407
+  --> TIME: 2025-12-21 12:27:43 -- STEP: 2005/2207 -- GLOBAL_STEP: 18250
408
+ | > loss_text_ce: 0.03071397729218006 (0.030976552677868013)
409
+ | > loss_mel_ce: 3.844003438949585 (3.825914057829136)
410
+ | > loss: 0.06054246053099632 (0.06026391582447394)
411
+ | > current_lr: 1e-05
412
+ | > step_time: 0.2667 (0.2787343830241824)
413
+ | > loader_time: 0.0081 (0.008377488533457602)
414
+
415
+
416
+  --> TIME: 2025-12-21 12:28:02 -- STEP: 2055/2207 -- GLOBAL_STEP: 18300
417
+ | > loss_text_ce: 0.03158266097307205 (0.030973146271205298)
418
+ | > loss_mel_ce: 3.630298376083374 (3.8251054886773845)
419
+ | > loss: 0.057216890156269073 (0.06025122870348956)
420
+ | > current_lr: 1e-05
421
+ | > step_time: 0.2654 (0.27866049694608885)
422
+ | > loader_time: 0.009 (0.00838099222113615)
423
+
424
+
425
+  --> TIME: 2025-12-21 12:28:22 -- STEP: 2105/2207 -- GLOBAL_STEP: 18350
426
+ | > loss_text_ce: 0.029528765007853508 (0.030974545680216542)
427
+ | > loss_mel_ce: 3.808253765106201 (3.8251185174792504)
428
+ | > loss: 0.0599653534591198 (0.06025145414535336)
429
+ | > current_lr: 1e-05
430
+ | > step_time: 0.2895 (0.2786942468403071)
431
+ | > loader_time: 0.0084 (0.008381317573601716)
432
+
433
+
434
+  --> TIME: 2025-12-21 12:28:41 -- STEP: 2155/2207 -- GLOBAL_STEP: 18400
435
+ | > loss_text_ce: 0.03116406314074993 (0.03098027495334569)
436
+ | > loss_mel_ce: 3.9434666633605957 (3.8245020461469683)
437
+ | > loss: 0.06210360676050186 (0.060241911300996064)
438
+ | > current_lr: 1e-05
439
+ | > step_time: 0.2905 (0.27876709453467685)
440
+ | > loader_time: 0.0083 (0.008375044377939872)
441
+
442
+
443
+  --> TIME: 2025-12-21 12:29:00 -- STEP: 2205/2207 -- GLOBAL_STEP: 18450
444
+ | > loss_text_ce: 0.03149457648396492 (0.030984393438818505)
445
+ | > loss_mel_ce: 3.63926362991333 (3.8233453620858744)
446
+ | > loss: 0.057355597615242004 (0.06022390246289928)
447
+ | > current_lr: 1e-05
448
+ | > step_time: 0.2661 (0.27879359316663666)
449
+ | > loader_time: 0.0079 (0.008370566530292528)
450
+
451
+
452
+  > EVALUATION 
453
+
454
+
455
+ --> EVAL PERFORMANCE
456
+ | > avg_loader_time: 0.07265385904827633 (-0.0017059896443341466)
457
+ | > avg_loss_text_ce: 0.02991473435650806 (-0.00016766787833861346)
458
+ | > avg_loss_mel_ce: 3.808113165803858 (-0.014583056037490216)
459
+ | > avg_loss: 3.8380279089953446 (-0.014750712626689477)
460
+
461
+ > BEST MODEL : /workspace/training_output/XTTS_Dahih_OmarSamir_Finetune-December-21-2025_12+10PM-0000000/best_model_18452.pth