Update README.md
Browse files
README.md
CHANGED
|
@@ -15,9 +15,9 @@ tags:
|
|
| 15 |
|
| 16 |
# Nugget H3 EasyR2V β Prompt Enhancer & Full Workflow
|
| 17 |
|
| 18 |
-
Why spend 30 minutes doing something when you can spend 2 weeks making a tool to automate it. That's basically the whole reason this exists.
|
| 19 |
|
| 20 |
-
I wanted an easy to use workflow for us simple folk who want to do **Ref2V** on MiniMax H3 but keep running into
|
| 21 |
|
| 22 |
Think of it as **"Ref2Video for dummies"**. But it's not just limited to that.
|
| 23 |
|
|
@@ -26,14 +26,20 @@ Think of it as **"Ref2Video for dummies"**. But it's not just limited to that.
|
|
| 26 |
## What it does
|
| 27 |
|
| 28 |
- **Scans your video** (if you're using one) to caption it and transcribe the audio
|
| 29 |
-
- **Captions all your images**
|
| 30 |
-
- **Loads a small LLM of your choice** and writes your H3 R2V prompt in the correct format
|
| 31 |
- **If you're on the Full workflow**, it generates the video too
|
| 32 |
|
| 33 |
-
##
|
| 34 |
|
| 35 |
-
-
|
| 36 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
|
| 38 |
---
|
| 39 |
|
|
@@ -96,20 +102,17 @@ If the file won't open in ComfyUI, a node pack is missing. Install it, restart,
|
|
| 96 |
|
| 97 |
## Tips for good character replacement
|
| 98 |
|
| 99 |
-
- **
|
| 100 |
-
- **
|
|
|
|
|
|
|
| 101 |
- **Give the LLM some context.** A one-liner in the user prompt like `<video 1> is a video of two girls eating a cup of chocolate ice cream` really helps the LLM understand what it's looking at. Especially useful with multiple scenes
|
| 102 |
-
- **H3 is a tool, you're the one using it.** If you don't specify
|
| 103 |
-
- **Don't expect miracles.** Sometimes less is more, sometimes more is more. You'll get a feel for it
|
| 104 |
- **Limit the resolution and length.** There seems to be an arbitrary token limit that's affected by higher res/longer time. Keep it smaller and your success rate goes up
|
|
|
|
|
|
|
|
|
|
| 105 |
|
| 106 |
-
---
|
| 107 |
-
|
| 108 |
-
## Caveats (being honest here)
|
| 109 |
-
|
| 110 |
-
- This is still working within the limits of H3, so sometimes it'll go funny. Not much I can do about that from a workflow
|
| 111 |
-
- Clips run best around **8β20 seconds**. Longer than that and the transcriber gets memory-hungry β a minute of 1080p is more RAM than most machines want to spend. Trim first
|
| 112 |
-
- I'm still testing more challenging v2v scenarios. If you hit something weird, let me know
|
| 113 |
|
| 114 |
---
|
| 115 |
|
|
@@ -123,22 +126,6 @@ To skip the video entirely, bypass **Load source video** (Ctrl+B). Prompt is wri
|
|
| 123 |
|
| 124 |
---
|
| 125 |
|
| 126 |
-
## Credit
|
| 127 |
-
|
| 128 |
-
Workflow build, tuning and prompt-enhancer chain by **C_Nugget**.
|
| 129 |
-
|
| 130 |
-
Original single-image workflow by **mackyb** (H3 Basic prompt enhancer v2). This is a modified variant, not their release.
|
| 131 |
-
|
| 132 |
-
---
|
| 133 |
-
|
| 134 |
-
## Licence stuff
|
| 135 |
-
|
| 136 |
-
The workflow files are mine, do what you like with them.
|
| 137 |
-
|
| 138 |
-
**The model isn't.** MiniMax H3 has a community licence that excludes the EU, UK, South Korea and the USA, and it covers **outputs** as well as the weights. If you're planning anything commercial, go read [the actual licence](https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/LICENSE) rather than taking my word for it. I'm not a lawyer.
|
| 139 |
-
|
| 140 |
-
---
|
| 141 |
-
|
| 142 |
**[If this has helped you, consider chucking a few dollars my way. Any tips help with the power bills. Thank you.](https://ko-fi.com/c_nugget)**
|
| 143 |
|
| 144 |
By C_Nugget
|
|
|
|
| 15 |
|
| 16 |
# Nugget H3 EasyR2V β Prompt Enhancer & Full Workflow
|
| 17 |
|
| 18 |
+
**Why spend 30 minutes doing something when you can spend 2.5 weeks making a tool to automate it.** That's basically the whole reason this exists.
|
| 19 |
|
| 20 |
+
I wanted an easy to use workflow for us simple folk who want to do **Ref2V** on MiniMax H3 but keep running into troubles with the model understanding me. A good prompt does 80β90% of the work with H3 R2V β so instead of writing them by hand every time, this workflow writes them for you.
|
| 21 |
|
| 22 |
Think of it as **"Ref2Video for dummies"**. But it's not just limited to that.
|
| 23 |
|
|
|
|
| 26 |
## What it does
|
| 27 |
|
| 28 |
- **Scans your video** (if you're using one) to caption it and transcribe the audio
|
| 29 |
+
- **Captions all your images** - so it also works as a pure image-to-video workflow
|
| 30 |
+
- **Loads a small LLM of your choice** and writes your H3 R2V prompt in the correct format with your stated intent (user prompt)
|
| 31 |
- **If you're on the Full workflow**, it generates the video too
|
| 32 |
|
| 33 |
+
## What it does NOT do
|
| 34 |
|
| 35 |
+
- **Be creative for you** - The current WF is only setup to do the prompt formatting, it is not able to generate new ideas for you
|
| 36 |
+
- **It cannot perform magic** - You are still limited to what the H3 model can and cannot do. Complex scenes are still very difficult
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
### Saving Time
|
| 40 |
+
|
| 41 |
+
- If your video doesn't change, a second run doesn't trigger a new video transcription (If you have "FIXED SEED")
|
| 42 |
+
- If your images and user prompt don't change, it doesn't write a new prompt β so you can re-run on a new seed to test without paying the LLM cost every time (If you have "FIXED SEED")
|
| 43 |
|
| 44 |
---
|
| 45 |
|
|
|
|
| 102 |
|
| 103 |
## Tips for good character replacement
|
| 104 |
|
| 105 |
+
- **Don't expect miracles.** - It is still H3 model and sometimes tempermental. Check your enhanced prompt and consider rolling again if it is not right.
|
| 106 |
+
- **If the video is easy, replacement should be easy too.** H3 has a quirk though β if the original person and the new person look too similar, it sometimes converges back to the original. A prompt won't always fix that. If you hit it, consider changing the person to a intermediate step (faceless green person). The new body/face will transfer over better. Alternatively you can look into Sam3 character replacement method.
|
| 107 |
+
- **More than one person in the scene?** Describe the scene properly. `replace the man wearing white shorts with the man in <picture 1>` beats `replace the man with <picture 1>` every time.
|
| 108 |
+
- **Complex scenes?** It will be very difficult (I've tried), scenes with too many people, too many cuts, characters obstructed are very difficult for the model to properly identify and swap.
|
| 109 |
- **Give the LLM some context.** A one-liner in the user prompt like `<video 1> is a video of two girls eating a cup of chocolate ice cream` really helps the LLM understand what it's looking at. Especially useful with multiple scenes
|
| 110 |
+
- **H3 is a tool, you're the one using it.** If you don't specify emotions, expect expressionless results. This current setup will only do what you intend for it to do. **Slop prompt in, slop video out**
|
|
|
|
| 111 |
- **Limit the resolution and length.** There seems to be an arbitrary token limit that's affected by higher res/longer time. Keep it smaller and your success rate goes up
|
| 112 |
+
- **Test! Test! Test!.** Sometimes less is more, sometimes more is more when it comes to the user prompt. You'll get a feel for it, tweak it until you get your desired results.
|
| 113 |
+
- **Use 24 FPS videos** you can trim, crop and edit the FPS before you send it to the WF. [I have a (unfortunately a bit buggy at times) tool for that](https://huggingface.co/PoopMan333/Video_Tools/tree/main). Alternatively you can use the VHS Load Video nodes
|
| 114 |
+
- **Use Character Sheets** through my testing, I have found that replacing a character is much easier when a character sheet is supplied. [Consider checking out my character generator to make a 360 orbit and generate a full sheet or individual shots](https://huggingface.co/PoopMan333/H3_Character_Sheet_Generator)
|
| 115 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 116 |
|
| 117 |
---
|
| 118 |
|
|
|
|
| 126 |
|
| 127 |
---
|
| 128 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 129 |
**[If this has helped you, consider chucking a few dollars my way. Any tips help with the power bills. Thank you.](https://ko-fi.com/c_nugget)**
|
| 130 |
|
| 131 |
By C_Nugget
|