Instructions to use tamdiep106/alpaca_lora_ja_en_emb-7b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tamdiep106/alpaca_lora_ja_en_emb-7b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tamdiep106/alpaca_lora_ja_en_emb-7b")

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("tamdiep106/alpaca_lora_ja_en_emb-7b")
model = AutoModelForMultimodalLM.from_pretrained("tamdiep106/alpaca_lora_ja_en_emb-7b")

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tamdiep106/alpaca_lora_ja_en_emb-7b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tamdiep106/alpaca_lora_ja_en_emb-7b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tamdiep106/alpaca_lora_ja_en_emb-7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/tamdiep106/alpaca_lora_ja_en_emb-7b

SGLang

How to use tamdiep106/alpaca_lora_ja_en_emb-7b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tamdiep106/alpaca_lora_ja_en_emb-7b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tamdiep106/alpaca_lora_ja_en_emb-7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tamdiep106/alpaca_lora_ja_en_emb-7b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tamdiep106/alpaca_lora_ja_en_emb-7b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use tamdiep106/alpaca_lora_ja_en_emb-7b with Docker Model Runner:
```
docker model run hf.co/tamdiep106/alpaca_lora_ja_en_emb-7b
```

YAML Metadata Warning:The pipeline tag "text2text-generation" is not in the official list: text-classification, token-classification, table-question-answering, question-answering, zero-shot-classification, translation, summarization, feature-extraction, text-generation, fill-mask, sentence-similarity, text-to-speech, text-to-audio, automatic-speech-recognition, audio-to-audio, audio-classification, audio-text-to-text, voice-activity-detection, depth-estimation, image-classification, object-detection, image-segmentation, text-to-image, image-to-text, image-to-image, image-to-video, unconditional-image-generation, video-classification, reinforcement-learning, robotics, tabular-classification, tabular-regression, tabular-to-text, table-to-text, multiple-choice, text-ranking, text-retrieval, time-series-forecasting, text-to-video, image-text-to-text, image-text-to-image, image-text-to-video, visual-question-answering, document-question-answering, zero-shot-image-classification, graph-ml, mask-generation, zero-shot-object-detection, text-to-3d, image-to-3d, image-feature-extraction, video-text-to-text, keypoint-detection, visual-document-retrieval, any-to-any, video-to-video, other

This model is finetune on Japanese and English prompt

Usage:

Init model:

To use in code:

import torch
import peft
from transformers import LlamaTokenizer, LlamaForCausalLM, GenerationConfig

tokenizer = LlamaTokenizer.from_pretrained(
    "decapoda-research/llama-7b-hf"
    )

model = LlamaForCausalLM.from_pretrained(
    "tamdiep106/alpaca_lora_ja_en_emb-7b",
    load_in_8bit=False,
    device_map="auto",
    torch_dtype=torch.float16
    )

tokenizer.pad_token_id = 0  # unk. we want this to be different from the eos token
tokenizer.bos_token_id = 1
tokenizer.eos_token_id = 2

Try this model

To try out this model, use this colab space GOOGLE COLAB LINK

Recommend Generation parameters:

temperature: 0.5~0.7
top p: 0.65~1.0
top k: 30~50
repeat penalty: 1.03~1.17

Japanese prompt:

instruction_input_JP = 'あなたはアシスタントです。以下に、タスクを説明する指示と、さらなるコンテキストを提供する入力を組み合わせます。 リクエストを適切に完了するレスポンスを作成します。'
instruction_no_input_JP = 'あなたはアシスタントです。以下はタスクを説明する指示です。 リクエストを適切に完了するレスポンスを作成します。'

prompt = """{}
### Instruction:
{}

### Response:"""

if input=='':
    prompt = prompt.format(
        instruction_no_input_JP, instruction
        )
else:
    prompt = prompt.format("{}\n\n### input:\n{}""").format(
        instruction_input_JP, instruction, input
        )

result:

English prompt:

instruction_input_EN = 'You are an Assistant, below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.'
instruction_no_input_EN = 'You are an Assistant, below is an instruction that describes a task. Write a response that appropriately completes the request.'

prompt = """{}
### Instruction:
{}

### Response:"""

instruction = "write an email for my boss letting him know that i will resign from the position" #@param {type:"string"}
input = "" #@param {type:"string"}

if input=='':
    prompt = prompt.format(
        instruction_no_input_EN, instruction
        )
else:
    prompt = prompt.format("{}\n\n### input:\n{}""").format(
        instruction_input_EN, instruction, input
        )

result:

Use this code to decode output of model

for s in generation_output.sequences:
    result = tokenizer.decode(s).strip()
    result = result.replace(prompt, '')
    result = result.replace("<s>", "")
    result = result.replace("</s>", "")
    if result=='':
        print('No output')
        print(prompt)
        print(result)
        continue
    print('\nResponse: ')

    print(result)

Training:

Dataset:

Jumtra/oasst1_ja
Jumtra/jglue_jsquads_with_input
Jumtra/dolly_oast_jglue_ja
Aruno/guanaco_jp
yahma/alpaca-cleaned
databricks/databricks-dolly-15k

with about 750k entries, 2k entries used for evaluate process

Training setup

I trained this model on an instance from vast.ai

1 NVIDIA RTX 4090
90 GB Storage
Time spend about 3 and a half days
use python export.py to merge weight
Training script: https://github.com/Tamminhdiep97/alpaca-lora_finetune/tree/master

Result

Training loss

Eval loss chart

Acknowledgement

Special thank to KBlueLeaf and the repo https://huggingface.co/KBlueLeaf/guanaco-7b-leh-v2 that helped and inspired me to train this model, without this help, i wouldn't never thought that i could finetune a llm myself

Downloads last month: 47

tamdiep106
/

alpaca_lora_ja_en_emb-7b