naksyu/LimeCore-all-4.2B-lora-balanced-fft-bake-BF16-GGUF

LimeCore all-LoRA balanced FFT bake BF16 GGUF

English

This repository contains a BF16 GGUF export of an experimental performance-first dense merge.

The model was built from a Qwen3.5-4B core, multiple specialist LoRA adapters, an all-LoRA balanced merge, and a short full-parameter FFT bake. It is intended as a research artifact for local inference and downstream quantization such as Q4_K_M, Q6_K, or Q8_0.

Current file:

File	Type	Purpose
LimeCore-all-4.2B-lora-balanced-fft-bake-BF16.gguf	BF16 GGUF	Source GGUF for local quantization and testing

Data Summary

No personal identifiers, access tokens, private URLs, or user account details are included in this model card.

The training pipeline used expert-specific cleaned subsets and synthetic expert-control data. The final FFT bake used a smaller distilled text-only mixture from these expert pools.

Text expert preparation:

Item	Count
Source rows scanned	113,562
Text expert rows selected/written	57,371
Text selection ratio	50.52%

Source categories used during expert preparation:

Source category	Usage
General SFT mixture	Repair, Python, math, code, experiment, persona, safety routing
Sanitized Lime SFT/alignment data	General repair and assistant behavior cleanup
Synthetic reasoning trajectory data	Internal planning, verification, routing, and no-CoT control
Synthetic time-series text data	Trend, anomaly, metric-log, and experiment-series reasoning
Synthetic dense anchor data	Prompt-conditioned anchor behavior for tags such as [PYTHON_TOOL] and [MATH_REASONING]
Vision reasoning data	Used to train the vision adapter; image tensors are not packaged in this text GGUF unless a separate multimodal projection is provided

Prepared expert training pools:

Expert pool	Train rows
general_repair	5,478
python_tool	9,507
math_reasoning	9,513
code_debug	9,499
experiment_loop	9,519
persona_lime	9,489
safety_control	1,555
time_series_text_prediction	5,000
reasoning_trajectory_prediction	20,000
dense_anchor_alignment	1,600
vision_reasoning	9,500

Final FFT bake mixture:

Expert source	Pool rows	Used rows	Pool retention	Bake mix ratio
general_repair	5,478	1,000	18.25%	12.05%
python_tool	9,507	1,000	10.52%	12.05%
math_reasoning	9,513	1,000	10.51%	12.05%
code_debug	9,499	1,000	10.53%	12.05%
experiment_loop	9,519	800	8.40%	9.64%
persona_lime	9,489	400	4.22%	4.82%
safety_control	1,555	500	32.15%	6.02%
time_series_text_prediction	5,000	800	16.00%	9.64%
reasoning_trajectory_prediction	20,000	1,000	5.00%	12.05%
dense_anchor_alignment	1,600	800	50.00%	9.64%
Total	-	8,300	-	100.00%

Experimental Techniques

• Sparse expert LoRA training: specialist adapters were trained for Python/tool use, math reasoning, code debugging, experiment loops, safety control, persona behavior, time-series text prediction, reasoning trajectory control, and vision reasoning.
• Performance-first all-LoRA dense merge: all available adapters were merged into a single dense checkpoint with balanced merge scales. This favors broad capability over preserving clean expert separation or a fixed style.
• Qwen3.5-specific anchor alignment: the dense anchor adapter used tags such as [LIME], [PYTHON_TOOL], [MATH_REASONING], [CODE_DEBUG], and [SAFETY]. For Qwen3.5 hybrid attention, the target set included standard attention/MLP projections plus linear-attention projections such as in_proj_qkv, in_proj_z, and out_proj.
• Short full-parameter FFT bake: after merging LoRAs, the dense checkpoint was lightly baked for 100 steps with BF16 weights, low learning rate (5e-6), sequence length 2048, and effective batch size 8. The goal was to stabilize the merged dense model before GGUF export.
• GGUF BF16 export: the baked dense checkpoint was exported to BF16 GGUF as a source artifact for local quantization.

Limitations

• This is an experimental research merge, not a production safety release.
• The model may inherit verbosity, formatting habits, or reasoning-style traces from the expert datasets.
• The BF16 GGUF is intended as a source file for local quantization; Q4/Q6/Q8 behavior should be evaluated separately after quantization.
• Vision adapter training was part of the experiment, but this GGUF file should be treated as a text GGUF unless an accompanying multimodal projection is provided.

한국어

이 저장소는 실험용 성능 우선 dense merge 모델의 BF16 GGUF 내보내기 파일을 담고 있습니다.

이 모델은 Qwen3.5-4B 코어 위에 여러 specialist LoRA를 학습하고, balanced all-LoRA merge를 수행한 뒤, 짧은 full-parameter FFT bake를 거쳐 만든 연구용 산출물입니다. 로컬 추론과 Q4_K_M, Q6_K, Q8_0 같은 후속 양자화를 위한 원본 GGUF로 사용하는 것을 목표로 합니다.

현재 파일:

파일	형식	용도
LimeCore-all-4.2B-lora-balanced-fft-bake-BF16.gguf	BF16 GGUF	로컬 양자화와 테스트를 위한 원본 GGUF

데이터 요약

이 모델 카드에는 개인정보, 액세스 토큰, 비공개 URL, 사용자 계정 정보가 포함되어 있지 않습니다.

학습 파이프라인은 expert별 정제 데이터와 synthetic expert-control 데이터를 사용했습니다. 최종 FFT bake에는 이 expert pool에서 뽑은 더 작은 text-only distilled mix가 사용되었습니다.

텍스트 expert 정제:

항목	수량
스캔한 source row	113,562
선택/저장된 text expert row	57,371
텍스트 선택 비율	50.52%

사용 데이터 범주:

데이터 범주	용도
General SFT mixture	일반 수리, Python, 수학, 코드, 실험, persona, safety 라우팅
Sanitized Lime SFT/alignment data	일반 응답 수리와 assistant behavior 정리
Synthetic reasoning trajectory data	내부 계획, 검증, 라우팅, no-CoT 제어
Synthetic time-series text data	추세, 이상치, metric log, 실험 시계열 reasoning
Synthetic dense anchor data	[PYTHON_TOOL], [MATH_REASONING] 같은 태그 기반 anchor 동작
Vision reasoning data	vision adapter 학습에 사용; 별도 multimodal projection이 없으면 이 GGUF는 text GGUF로 취급

준비된 expert training pool:

Expert pool	Train rows
general_repair	5,478
python_tool	9,507
math_reasoning	9,513
code_debug	9,499
experiment_loop	9,519
persona_lime	9,489
safety_control	1,555
time_series_text_prediction	5,000
reasoning_trajectory_prediction	20,000
dense_anchor_alignment	1,600
vision_reasoning	9,500

최종 FFT bake mix:

Expert source	Pool rows	Used rows	Pool retention	Bake mix ratio
general_repair	5,478	1,000	18.25%	12.05%
python_tool	9,507	1,000	10.52%	12.05%
math_reasoning	9,513	1,000	10.51%	12.05%
code_debug	9,499	1,000	10.53%	12.05%
experiment_loop	9,519	800	8.40%	9.64%
persona_lime	9,489	400	4.22%	4.82%
safety_control	1,555	500	32.15%	6.02%
time_series_text_prediction	5,000	800	16.00%	9.64%
reasoning_trajectory_prediction	20,000	1,000	5.00%	12.05%
dense_anchor_alignment	1,600	800	50.00%	9.64%
Total	-	8,300	-	100.00%

사용된 실험 기술

• Sparse expert LoRA 학습: Python/tool, math reasoning, code debugging, experiment loop, safety control, persona behavior, time-series text prediction, reasoning trajectory control, vision reasoning용 specialist adapter를 학습했습니다.
• 성능 우선 all-LoRA dense merge: 모든 사용 가능한 adapter를 balanced scale로 하나의 dense checkpoint에 병합했습니다. 이는 expert 분리나 고정된 말투 보존보다 전체 성능을 우선하는 실험입니다.
• Qwen3.5 전용 anchor alignment: dense anchor adapter는 [LIME], [PYTHON_TOOL], [MATH_REASONING], [CODE_DEBUG], [SAFETY] 같은 태그를 사용했습니다. Qwen3.5의 hybrid attention 구조를 고려해 일반 attention/MLP projection 외에도 in_proj_qkv, in_proj_z, out_proj 같은 linear-attention projection을 target에 포함했습니다.
• 짧은 full-parameter FFT bake: LoRA 병합 후 BF16 weight, 낮은 learning rate(5e-6), sequence length 2048, effective batch size 8 조건으로 100 step bake를 수행했습니다. 목적은 GGUF export 전에 merge된 dense 모델을 안정화하는 것입니다.
• GGUF BF16 export: bake된 dense checkpoint를 로컬 양자화용 source artifact로 BF16 GGUF 변환했습니다.

한계

• 이 모델은 production safety release가 아니라 실험용 research merge입니다.
• expert 데이터의 영향으로 장황함, 특정 포맷 습관, reasoning-style 흔적이 나타날 수 있습니다.
• BF16 GGUF는 로컬 양자화를 위한 원본 파일입니다. Q4/Q6/Q8 결과는 양자화 후 별도로 평가해야 합니다.
• vision adapter 학습은 실험에 포함되었지만, 별도 multimodal projection이 제공되지 않는 한 이 GGUF는 text GGUF로 취급해야 합니다.