DarkKitsune/gemma-4-E2B-it-heretic-ara-custom-UQFF

This is a UQFF quantized version of MoonRide/gemma-4-E2B-it-heretic-ara-custom, a version of gemma-4-E2B-it that has been abliterated with ARA (arbitrary-rank abliation). This quant is meant to be run with mistral.rs.

Rust

#[tokio::main]
async fn main() -> () {
    // Build the model
    let model = UqffMultimodalModelBuilder::new(
        "DarkKitsune/gemma-4-E2B-it-heretic-ara-custom-UQFF",
        vec!["q4k-0.uqff".into()],
    )
        .into_inner()
        .with_logging()
        .build()
        .await
        .unwrap();

    println!("Model initialized successfully.");

    /// Build the chat
    let messages = TextMessages::new()
        .add_message(
            TextMessageRole::System,
            "You are a helpful assistant.",
        )
        .add_message(
            TextMessageRole::User,
            "Hello, is anyone there?",
        );

    println!("Getting response...");

    // Get the response
    let response = model.send_chat_request(messages).await.unwrap();

    // Handle the response
    println!("Model Response:\n{}", response.choices.first().unwrap().message.content.as_ref().unwrap());
    
    // Print timing
    println!("tok/s: {}", response.usage.avg_compl_tok_per_sec);
    println!("prefill tok/s: {}", response.usage.avg_prompt_tok_per_sec);
}