Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+output/flappy_bird_evaluation.gif filter=lfs diff=lfs merge=lfs -text
+output/flappy_bird_evaluation.mp4 filter=lfs diff=lfs merge=lfs -text
+output/flappy_bird_training.mp4 filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,182 @@
+# Flappy Bird Reinforcement Learning Agent
+
+This project implements a reinforcement learning agent for the Flappy Bird game using the REINFORCE (Policy Gradient) algorithm. The agent learns to play Flappy Bird by maximizing the cumulative reward through trial and error.
+
+## Features
+
+- Policy Gradient (REINFORCE) algorithm implementation
+- Neural network policy with multiple hidden layers
+- Checkpoint saving and resuming for training
+- Evaluation with statistical analysis
+- Video recording of gameplay
+- Dynamic plotting of training progress
+
+## Dependencies
+
+- Python 3.13
+- PyTorch
+- NumPy
+- Matplotlib
+- Gymnasium
+- Flappy Bird Gymnasium
+- ImageIO
+- tqdm
+- PIL (Pillow)
+- Hugging Face Hub (for model uploading)
+
+## Installation
+
+1. Create a virtual environment:
+   ```bash
+   python -m venv env
+   source env/bin/activate  # On Windows: env\Scripts\activate
+   ```
+
+2. Install dependencies:
+   ```bash
+   pip install -r requirements.txt
+   ```
+   
+   Or manually:
+   ```bash
+   pip install torch numpy matplotlib gymnasium flappy-bird-gymnasium imageio tqdm pillow huggingface-hub
+   ```
+
+## Usage
+
+### Training
+
+Run the training script with default settings (trains and evaluates):
+```bash
+python agent.py
+```
+
+The agent will train for 100,000 episodes by default. Checkpoints are saved every 100 episodes to `checkpoints/checkpoint.pth`, and the final model is saved to `checkpoints/final_model.pth`.
+
+### Configuration
+
+Modify the `hyperparams` dictionary in `agent.py` to adjust:
+
+- `no_of_episodes`: Number of training episodes
+- `gamma`: Discount factor
+- `lr`: Learning rate
+- `fps`: Frames per second for video output
+- `save_videos`: Whether to save training/evaluation videos
+- `train`: Whether to run training (default: True)
+- `evaluate`: Whether to run evaluation (default: True)
+- `push_hf`: Whether to push model to Hugging Face Hub (default: False)
+- `hf_repo`: Hugging Face repository name (required if push_hf is True)
+- `eval_episodes`: Number of evaluation episodes (default: 5)
+
+### Examples
+
+Train only (modify hyperparams):
+```python
+hyperparams["evaluate"] = False
+```
+
+Evaluate only (modify hyperparams):
+```python
+hyperparams["train"] = False
+hyperparams["eval_episodes"] = 10
+```
+
+Train and push to HF (modify hyperparams):
+```python
+hyperparams["push_hf"] = True
+hyperparams["hf_repo"] = "your-username/flappy-bird-rl-agent"
+```
+
+### Resuming Training
+
+If checkpoints exist, the training will automatically resume from the last saved checkpoint.
+
+### Evaluation
+
+The script automatically evaluates the trained model after training. To evaluate a saved model separately, modify the `evaluate_policy` call in `agent.py`.
+
+### Uploading to Hugging Face
+
+To upload your trained model and project to Hugging Face Hub:
+
+1. **Get your Hugging Face token**:
+   - Go to [Hugging Face Settings](https://huggingface.co/settings/tokens)
+   - Create a new token with "Write" permissions
+   - Copy the token
+
+2. **Set the token as an environment variable**:
+   ```bash
+   export HF_TOKEN=your_token_here
+   ```
+
+3. **Modify hyperparams in `agent.py`**:
+   ```python
+   hyperparams["push_hf"] = True
+   hyperparams["hf_repo"] = "your-username/flappy-bird-rl-agent"
+   ```
+
+The entire project (excluding unnecessary files) will be uploaded to your repository.
+
+## Configuration
+
+Modify the `hyperparams` dictionary in `agent.py` to adjust:
+
+- `no_of_episodes`: Number of training episodes
+- `gamma`: Discount factor
+- `lr`: Learning rate
+- `fps`: Frames per second for video output
+- `save_videos`: Whether to save training/evaluation videos
+
+## Output
+
+- Training plots: `output/rewards_vs_episodes.png`, `output/loss_vs_episodes.png`
+- Training video: `output/flappy_bird_training.mp4` (if `save_videos=True`)
+- Evaluation video: `output/flappy_bird_evaluation.mp4`
+- Checkpoints: `checkpoints/`
+
+## Results
+
+### Training Progress
+
+![Training Rewards](output/rewards_vs_episodes.png)
+*Figure 1: Total reward per episode during training*
+
+![Training Loss](output/loss_vs_episodes.png)
+*Figure 2: Loss per episode during training*
+
+### Evaluation
+
+The evaluation video shows the trained agent playing Flappy Bird. To create a 3-second GIF from the last 3 seconds of the video:
+
+```bash
+ffmpeg -sseof -3 -i output/flappy_bird_evaluation.mp4 -t 3 -vf "fps=10,scale=320:-1:flags=lanczos" output/flappy_bird_evaluation.gif
+```
+
+![Evaluation GIF](output/flappy_bird_evaluation.gif)
+*Figure 3: GIF of the evaluation gameplay*
+
+### Final Evaluation Statistics
+
+Episode 1: Total Reward: 597.100000000053
+Episode 2: Total Reward: 2084.89999999937
+Episode 3: Total Reward: 428.4000000000229
+Episode 4: Total Reward: 22.700000000000063
+Episode 5: Total Reward: 152.89999999999645
+
+Evaluation Statistics:
+Mean Reward: 657.20
+Standard Deviation: 741.78
+Max Reward: 2084.90
+Mean Score: 446.00
+Max Score: 446.00
+
+## Architecture
+
+The policy network consists of:
+- Input layer: Matches observation space dimension
+- Hidden layers: 256, 128, 64 neurons with ReLU activation
+- Output layer: Softmax over action space
+
+## Algorithm
+
+REINFORCE uses Monte Carlo policy gradients to update the policy parameters by maximizing the expected cumulative reward. The agent samples actions from the current policy, collects trajectories, and updates the policy using the discounted returns.

agent.py

@@ -0,0 +1,335 @@
+import flappy_bird_gymnasium
+import gymnasium
+from policy import PolicyNetwork
+import torch
+import matplotlib.pyplot as plt
+import numpy as np
+import imageio
+import tqdm
+import os
+from PIL import Image, ImageDraw, ImageFont
+import matplotlib
+matplotlib.use('TkAgg')  # Use interactive backend
+plt.ion()  # Turn on interactive mode
+from huggingface_hub import HfApi, upload_file, login, upload_folder
+
+hyperparams = {
+    "no_of_episodes": 52300,
+    "gamma": 0.99,
+    "lr": 3e-4,  # Lower learning rate for stability
+    "fps": 30,
+    "out_directory": "output/flappy_bird_evaluation.mp4",
+    "save_videos": True,
+    "train": False,
+    "evaluate": False,
+    "push_hf": True,
+    "hf_repo": "Heshwa/flappy-bird-reinforce-v1",
+    "eval_episodes": 3
+}
+
+def discounted_returns(rewards, gamma):
+    G = []
+    Gt = 0.0
+    for r in reversed(rewards):
+        Gt = r + gamma * Gt
+        G.insert(0, Gt)
+
+    eps = np.finfo(np.float32).eps.item()
+    G = torch.tensor(G, dtype=torch.float32)
+    G = (G - G.mean()) / (G.std() + eps)
+    return G
+
+
+def annotate_frame(frame, episode, score):
+    """Draw episode number and score onto an RGB ndarray frame and return ndarray."""
+    try:
+        img = Image.fromarray(frame)
+    except Exception:
+        # If frame is already a PIL Image
+        img = frame
+    draw = ImageDraw.Draw(img)
+    font = ImageFont.load_default()
+    text = f"Ep:{episode}  Score:{score}"
+    x, y = 8, 8
+    # draw black outline for readability
+    draw.text((x - 1, y - 1), text, font=font, fill=(0, 0, 0))
+    draw.text((x + 1, y - 1), text, font=font, fill=(0, 0, 0))
+    draw.text((x - 1, y + 1), text, font=font, fill=(0, 0, 0))
+    draw.text((x + 1, y + 1), text, font=font, fill=(0, 0, 0))
+    # draw main text
+    draw.text((x, y), text, font=font, fill=(255, 255, 255))
+    return np.array(img)
+
+
+
+
+
+
+env = gymnasium.make("FlappyBird-v0",render_mode="rgb_array",use_lidar=False)
+policy_net = PolicyNetwork(input_dim=env.observation_space.shape[0], output_dim=env.action_space.n)
+optimizer = torch.optim.Adam(policy_net.parameters(), lr=hyperparams["lr"])
+
+ensure_dir = lambda p: os.makedirs(p, exist_ok=True)
+
+
+def load_checkpoint(policy_net, optimizer, checkpoint_path):
+    if os.path.exists(checkpoint_path):
+        checkpoint = torch.load(checkpoint_path)
+        policy_net.load_state_dict(checkpoint['model_state_dict'])
+        optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
+        start_episode = checkpoint['episode']
+        rewards = checkpoint.get('rewards', [])
+        losses = checkpoint.get('losses', [])
+        print(f"Resumed from episode {start_episode}")
+        return start_episode, rewards, losses
+    else:
+        print("No checkpoint found, starting from scratch")
+        return 0, [], []
+
+
+def get_latest_checkpoint(checkpoint_dir):
+    checkpoint_path = os.path.join(checkpoint_dir, "checkpoint.pth")
+    if os.path.exists(checkpoint_path):
+        return checkpoint_path
+    final_path = os.path.join(checkpoint_dir, "final_model.pth")
+    if os.path.exists(final_path):
+        return final_path
+    return None
+
+
+def reinforce(policy_net, optimizer, env, hyperparams, start_episode=0, rewards=[], losses=[]): 
+    
+    
+    
+    # Initialize dynamic plots
+    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))
+    ax1.set_xlabel("Episode")
+    ax1.set_ylabel("Total Reward")
+    ax1.set_title("Total Reward vs Episode")
+    line1, = ax1.plot([], [], 'b-')
+    
+    ax2.set_xlabel("Episode")
+    ax2.set_ylabel("Loss")
+    ax2.set_title("Loss vs Episode")
+    line2, = ax2.plot([], [], 'r-')
+    
+    plt.tight_layout()
+    
+    checkpoint_dir = "checkpoints"
+    ensure_dir(checkpoint_dir)
+    
+    for i in tqdm.tqdm(range(start_episode, hyperparams["no_of_episodes"])):
+        returns = []
+        log_probs = []
+        obs, _ = env.reset()
+        terminated = False
+        truncated = False
+        while not (terminated or truncated):
+            # action = env.action_space.sample()
+            action, log_prob = policy_net.act(obs)
+            obs, reward, terminated, truncated, info = env.step(action)
+            returns.append(reward)
+            log_probs.append(log_prob)
+            # print(f"Action: {action}, Reward: {reward}, Terminated: {terminated}")
+
+        loss = 0.0
+        # print("Episode:", i+1)
+        for log_prob, Gt in zip(log_probs, discounted_returns(returns, hyperparams["gamma"])):
+            loss += -log_prob * Gt
+        loss = loss/len(returns)
+        # print("Loss:", loss)
+        # print("total reward:", sum(returns))
+        optimizer.zero_grad()
+        loss.backward()
+        torch.nn.utils.clip_grad_norm_(policy_net.parameters(), max_norm=1.0)  # Add gradient clipping
+        optimizer.step()
+        losses.append(loss.item())
+        rewards.append(sum(returns))
+        
+        # Save checkpoint every 100 episodes
+        if (i + 1) % 100 == 0: 
+            checkpoint_path = os.path.join(checkpoint_dir, "checkpoint.pth")
+            torch.save({
+                'model_state_dict': policy_net.state_dict(),
+                'optimizer_state_dict': optimizer.state_dict(),
+                'episode': i + 1,
+                'rewards': rewards,
+                'losses': losses
+            }, checkpoint_path)
+            print(f"Checkpoint saved at episode {i+1} to {checkpoint_path}")        
+        
+
+        
+        # Update plots every 100 episodes
+        if (i + 1) % 100 == 0:
+            line1.set_xdata(range(1, len(rewards) + 1))
+            line1.set_ydata(rewards)
+            ax1.relim()
+            ax1.autoscale_view()
+            
+            line2.set_xdata(range(1, len(losses) + 1))
+            line2.set_ydata(losses)
+            ax2.relim()
+            ax2.autoscale_view()
+            
+            plt.draw()
+            plt.pause(0.01)  # Small pause to update the plot
+
+
+    ensure_dir("output")
+    plt.close(fig)  # Close the dynamic plot
+    plt.close('all')  # Close all plots
+    plt.figure()
+    plt.plot(rewards)
+    plt.xlabel("Episode")
+    plt.ylabel("Total Reward")
+    plt.title("Total Reward vs Episode")
+    plt.savefig("output/rewards_vs_episodes.png")
+    plt.close()
+    plt.figure()
+    plt.plot(losses)
+    plt.xlabel("Episode")
+    plt.ylabel("Loss")
+    plt.title("Loss vs Episode")
+    plt.savefig("output/loss_vs_episodes.png")
+    plt.close()
+
+    return rewards, losses
+
+
+
+def evaluate_policy(policy_net, env, episodes=5,save_videos=False, checkpoint_path=None,out_directory="output/flappy_bird_evaluation.mp4"):
+    if checkpoint_path is None:
+        checkpoint_path = get_latest_checkpoint("checkpoints") or "checkpoints/final_model.pth"
+    
+    if checkpoint_path and os.path.exists(checkpoint_path):
+        checkpoint = torch.load(checkpoint_path)
+        policy_net.load_state_dict(checkpoint['model_state_dict'])
+        print(f"Loaded model from {checkpoint_path}")
+    else:
+        print("No checkpoint found, using current model")
+
+    if save_videos:
+        images = []
+    
+    episode_rewards = []
+    episode_scores = []
+    for i in range(episodes):
+        obs, _ = env.reset()
+        if save_videos:
+            frame = env.render()
+            if frame is not None:
+                images.append(annotate_frame(frame, i + 1, 0))
+        terminated = False
+        truncated = False
+        total_reward = 0
+        while not (terminated or truncated):
+            action, _ = policy_net.act(obs)
+            obs, reward, terminated, truncated, info = env.step(action)
+            if save_videos:
+                frame = env.render()
+                if frame is not None:
+                    images.append(annotate_frame(frame, i + 1, int(info.get("score"))))
+            total_reward += reward
+            env.render()
+        episode_rewards.append(total_reward)
+        episode_scores.append(info.get("score"))
+        print(f"Episode {i+1}: Total Reward: {total_reward}")
+    
+    # Compute statistics
+    mean_reward = np.mean(episode_rewards)
+    std_reward = np.std(episode_rewards)
+    max_reward = np.max(episode_rewards)
+    mean_score = np.max(episode_scores)
+    max_score = np.max(episode_scores)
+    print(f"\nEvaluation Statistics:")
+    print(f"Mean Reward: {mean_reward:.2f}")
+    print(f"Standard Deviation: {std_reward:.2f}")
+    print(f"Max Reward: {max_reward:.2f}")
+    print(f"Mean Score: {mean_score:.2f}")
+    print(f"Max Score: {max_score:.2f}")
+
+    
+    if save_videos:
+        ensure_dir("output")
+        imageio.mimwrite(out_directory, images, fps=hyperparams["fps"])
+
+
+
+def push_to_hf(repo_name, token=None):
+    """
+    Push the project to Hugging Face Hub.
+    
+    Args:
+        repo_name (str): Name of the HF repository (e.g., 'username/model-name')
+        token (str, optional): HF token. If None, uses environment variable HF_TOKEN
+    """
+    if token:
+        login(token)
+    else:
+        login()
+    
+    # Upload the project folder, excluding unnecessary files
+    ignore_patterns = [
+        "env/",
+        "__pycache__/",
+        "*.pyc",
+        ".git/",
+        "*.log",
+        "text.ipynb",
+        # "*.mp4",  # Optional: exclude videos if too large
+        # "*.gif"   # Optional: exclude GIFs
+    ]
+    
+    try:
+        upload_folder(
+            folder_path=".",
+            repo_id=repo_name,
+            repo_type="model",
+            ignore_patterns=ignore_patterns
+        )
+        print(f"Project uploaded to https://huggingface.co/{repo_name}")
+    except Exception as e:
+        print(f"Upload failed: {e}")
+
+
+
+if __name__ == "__main__":
+    env = gymnasium.make("FlappyBird-v0",render_mode="rgb_array",use_lidar=False)
+    policy_net = PolicyNetwork(input_dim=env.observation_space.shape[0], output_dim=env.action_space.n)
+    optimizer = torch.optim.Adam(policy_net.parameters(), lr=hyperparams["lr"])
+
+    ensure_dir = lambda p: os.makedirs(p, exist_ok=True)
+
+    # Load checkpoint if exists (prefer checkpoint.pth for resuming, else final_model.pth)
+    checkpoint_path = get_latest_checkpoint("checkpoints")
+    if checkpoint_path:
+        start_episode, rewards, losses = load_checkpoint(policy_net, optimizer, checkpoint_path)
+    else:
+        start_episode, rewards, losses = 0, [], []
+
+    if hyperparams["train"]:
+        rewards, losses = reinforce(policy_net, optimizer, env, hyperparams,start_episode=start_episode, rewards=rewards, losses=losses)
+
+        # Save final model
+        final_checkpoint_path = "checkpoints/final_model.pth"
+        ensure_dir("checkpoints")
+        torch.save({
+            'model_state_dict': policy_net.state_dict(),
+            'optimizer_state_dict': optimizer.state_dict(),
+            'episode': hyperparams["no_of_episodes"],
+            'rewards': rewards,
+            'losses': losses
+        }, final_checkpoint_path)
+        print(f"Final model saved to {final_checkpoint_path}")
+
+    if hyperparams["evaluate"]:
+        evaluate_policy(policy_net, env, episodes=hyperparams["eval_episodes"], save_videos=hyperparams["save_videos"], checkpoint_path=None, out_directory=hyperparams["out_directory"])
+
+    if hyperparams["push_hf"]:
+        if hyperparams["hf_repo"] is None:
+            print("Error: hf_repo is required when push_hf is True")
+        else:
+            push_to_hf(hyperparams["hf_repo"])
+
+    env.close()

checkpoints/checkpoint.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07e044c7c4e27d05f6b4d36714994120b51996a39703a964b95cef214f3ce238
+size 1490549

checkpoints/final_model.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ea3fa287930789d32a4dac03f8ef60f5d7000b9020daf05758a4400c7727515
+size 1490587

output/flappy_bird_evaluation.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:859c760509b5059fcd5798b260bcff082ce107a37a8af1f9d58c002f5b38e063
+size 15957601

output/flappy_bird_training.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:69aa820437344b05e9e3d009b96a36c3859df65914e1c305d87ef91ec7f696a1
+size 2672703

policy.py

@@ -0,0 +1,31 @@
+import torch.nn as nn
+import torch
+from torch.distributions import Categorical
+import numpy as np
+
+
+class PolicyNetwork(nn.Module):
+    def __init__(self, input_dim, output_dim):
+        super(PolicyNetwork, self).__init__()
+        self.fc1 = nn.Linear(input_dim, 256)  # Increased from 128
+        self.fc2 = nn.Linear(256, 128)        # Increased from 64
+        self.fc3 = nn.Linear(128, 64)         # New layer
+        self.fc4 = nn.Linear(64, output_dim)
+        self.relu = nn.ReLU()
+        self.softmax = nn.Softmax(dim=-1)
+
+    def forward(self, x):
+        x = self.relu(self.fc1(x))
+        x = self.relu(self.fc2(x))
+        x = self.relu(self.fc3(x))
+        x = self.softmax(self.fc4(x))
+        return x
+    
+    def act(self, state):
+        state = torch.from_numpy(state).float().unsqueeze(0)
+        probs = self.forward(state)
+        m = Categorical(probs)
+        # action = np.argmax(m)
+        action = m.sample()
+        return action.item(), m.log_prob(action)
+    

requirements.txt

@@ -0,0 +1,9 @@
+torch>=2.0.0
+numpy>=1.21.0
+matplotlib>=3.5.0
+gymnasium>=0.29.0
+flappy-bird-gymnasium>=0.4.0
+imageio>=2.31.0
+tqdm>=4.64.0
+Pillow>=9.0.0
+huggingface-hub>=0.17.0