new ux

BENCHMARK_SYSTEM.md

@@ -0,0 +1,146 @@
+# Benchmark-First Evaluation System
+
+This system has been redesigned to use a **benchmark-first** approach based on the [evalevalai.com](https://evalevalai.com/projects/every-eval-ever/) schema, moving away from the previous checkbox-based evaluation method.
+
+## Key Changes
+
+### 1. Schema Structure
+
+The new system uses standardized benchmark evaluation data with the following structure:
+
+```typescript
+{
+  schema_version: string
+  evaluation_id: string
+  retrieved_timestamp: string
+  
+  source_data: {
+    dataset_name: string
+    hf_repo?: string
+    samples_number: number
+    // ...
+  }
+  
+  source_metadata: {
+    source_name: string
+    source_type: 'evaluation_run' | 'model_card' | 'paper' | 'leaderboard'
+    evaluator_relationship: 'first_party' | 'second_party' | 'third_party'
+    // ...
+  }
+  
+  model_info: {
+    name: string
+    id: string
+    developer: string
+    // ...
+  }
+  
+  evaluation_results: [{
+    evaluation_name: string
+    metric_config: {
+      evaluation_description: string
+      score_type: 'continuous' | 'discrete' | 'binary'
+      min_score: number
+      max_score: number
+    }
+    score_details: {
+      score: number
+      confidence_interval?: {...}
+    }
+    generation_config?: {...}
+  }]
+}
+```
+
+### 2. New Components
+
+#### **BenchmarkEvaluationCard** (`components/benchmark-evaluation-card.tsx`)
+Displays a model's evaluation summary with:
+- Model name and developer
+- Number of benchmarks and evaluations
+- Top scores across benchmarks
+- Capability and risk category counts
+
+#### **BenchmarkDetail** (`components/benchmark-detail.tsx`)
+Detailed view showing:
+- All evaluation results grouped by dataset
+- Tabbed views for capabilities vs risks
+- Score details with confidence intervals
+- Generation configs and source links
+
+### 3. New Pages
+
+#### **`/benchmarks`** (`app/benchmarks/page.tsx`)
+Main listing page with:
+- Filter by evaluation type (capability/risk)
+- Filter by specific category
+- Sort by date, name, or benchmark count
+- Grid view of evaluation cards
+
+#### **`/benchmark/[id]`** (`app/benchmark/[id]/page.tsx`)
+Detail page for individual model evaluations with comprehensive results
+
+### 4. Data Processing
+
+#### **Type Definitions** (`lib/benchmark-schema.ts`)
+- Full TypeScript types for the evaluation schema
+- Category classification (capabilities vs risks)
+- Helper functions for inference
+
+#### **Processing Utilities** (`lib/eval-processing.ts`)
+- Load and validate evaluation data
+- Group evaluations by model
+- Create display-friendly summaries
+- Format scores and dates
+
+## Category Mapping
+
+The system automatically infers categories from benchmark names:
+
+### Capabilities
+- **knowledge**: MMLU, ARC, HellaSwag, WinoGrande
+- **math**: GSM8K, MATH, Minerva
+- **code**: HumanEval, MBPP
+- **vision**: VQA, image benchmarks
+- **reasoning**: BBH (Big-Bench Hard)
+
+### Risks
+- **bias-fairness**: BBQ, bias benchmarks
+- **toxicity**: RealToxicityPrompts
+- **truthfulness**: TruthfulQA
+- **robustness**: Adversarial benchmarks
+
+## Data Format
+
+Place evaluation JSON files in `/public/benchmarks/` following the schema structure. The system will:
+1. Load all evaluation files
+2. Group by model ID
+3. Create aggregated summaries
+4. Display in cards and detail views
+
+## Sample Data
+
+Three sample evaluations are included:
+- **Kimi K2 Instruct**: MMLU-Pro with chain-of-thought
+- **GPT-4 Turbo**: MMLU 5-shot accuracy
+- **Claude 3 Sonnet**: HellaSwag 10-shot accuracy
+
+## Benefits Over Checkbox System
+
+1. **Standardized Data**: Uses established benchmark datasets
+2. **Reproducible**: Includes all evaluation metadata
+3. **Quantitative**: Shows actual scores with confidence intervals
+4. **Traceable**: Links to source evaluations and detailed results
+5. **Comparative**: Easy to compare models on same benchmarks
+6. **Transparent**: Shows who ran the evaluation and when
+
+## Migration Path
+
+To migrate existing evaluations:
+1. Extract benchmark results from old format
+2. Map to new schema structure
+3. Add source metadata
+4. Include generation configs if available
+5. Place in `/public/benchmarks/`
+
+The old evaluation format and pages remain intact for reference.

IMPLEMENTATION_SUMMARY.md

@@ -0,0 +1,212 @@
+# Benchmark-First Evaluation System - Implementation Summary
+
+## Overview
+Successfully redesigned the AI evaluation card system from a checkbox-based approach to a **benchmark-first** system based on the evalevalai.com schema structure. This provides a more standardized, quantitative, and reproducible approach to AI model evaluations.
+
+## What Was Built
+
+### 1. Core Type System
+**File: `lib/benchmark-schema.ts`**
+- Complete TypeScript type definitions for the evaluation schema
+- Interfaces for:
+  - `BenchmarkEvaluation`: Main evaluation data structure
+  - `SourceData`, `SourceMetadata`, `ModelInfo`: Metadata types
+  - `EvaluationResult`, `MetricConfig`, `ScoreDetails`: Results types
+  - `ModelEvaluationSummary`: Aggregated model data
+  - `EvaluationCardData`: UI display format
+- Category classification (capabilities vs risks)
+- Helper function `inferCategoryFromBenchmark()` for automatic categorization
+
+### 2. Data Processing Layer
+**File: `lib/eval-processing.ts`**
+- `groupEvaluationsByModel()`: Groups evaluations by model ID
+- `createModelSummary()`: Aggregates evaluations into summaries
+- `createEvaluationCard()`: Converts summaries to UI format
+- `getCategoryStats()`: Calculates statistics by category
+- `loadEvaluations()`: Loads and validates evaluation files
+- `processEvaluationsToCards()`: End-to-end processing pipeline
+- `formatScore()`: Smart score formatting based on type
+- `getBenchmarkDisplayName()`: User-friendly benchmark names
+
+### 3. UI Components
+
+#### `components/benchmark-evaluation-card.tsx`
+Displays model evaluation summaries with:
+- Model name, ID, and developer
+- Statistics (benchmarks count, evaluations count)
+- Capability and risk category counts
+- Top 3 scores with tooltips
+- Category badges with color coding
+- Action menu with view/source/delete options
+
+#### `components/benchmark-detail.tsx`
+Comprehensive detail view featuring:
+- Model header with metadata
+- Overview statistics
+- Three-tab interface:
+  - **All Evaluations**: Grouped by dataset with full details
+  - **Capabilities**: Category-wise capability scores
+  - **Risks**: Category-wise risk scores
+- Expandable generation configs
+- Links to source evaluations
+- Confidence intervals and sample sizes
+
+### 4. Pages
+
+#### `app/benchmarks/page.tsx`
+Main listing page with:
+- Load and display evaluation cards
+- Filter by type (capability/risk)
+- Filter by specific category
+- Sort by date, name, or benchmark count
+- Summary statistics dashboard
+- Responsive grid layout
+
+#### `app/benchmark/[id]/page.tsx`
+Individual model detail page with:
+- Dynamic routing by model ID
+- Full evaluation detail view
+- Back navigation
+- Error handling
+
+### 5. Sample Data
+Three complete evaluation files in `/public/benchmarks/`:
+- **kimi-k2-instruct.json**: MMLU-Pro with chain-of-thought (HELM, third-party)
+- **gpt-4-turbo.json**: MMLU 5-shot accuracy (OpenAI, first-party)
+- **claude-3-sonnet.json**: HellaSwag 10-shot (Anthropic, first-party)
+
+Each includes:
+- Complete schema structure
+- Source metadata
+- Model information
+- Evaluation results with scores
+- Sample-level results
+- Generation configurations
+
+### 6. Navigation Updates
+**File: `components/navigation.tsx`**
+- Added "Benchmarks" link to navigation bar
+- Uses BarChart3 icon
+- Active state handling for /benchmarks and /benchmark/* routes
+
+### 7. Documentation
+**File: `BENCHMARK_SYSTEM.md`**
+Comprehensive documentation covering:
+- Schema structure overview
+- Component descriptions
+- Category mappings
+- Data format requirements
+- Benefits over checkbox system
+- Migration path from old format
+
+## Key Features
+
+### Automatic Category Inference
+The system intelligently categorizes benchmarks:
+- **Knowledge**: MMLU, ARC, HellaSwag, WinoGrande
+- **Math**: GSM8K, MATH, Minerva
+- **Code**: HumanEval, MBPP
+- **Vision**: VQA, image benchmarks
+- **Reasoning**: BBH
+- **Bias/Fairness**: BBQ, bias benchmarks
+- **Toxicity**: RealToxicityPrompts
+- **Truthfulness**: TruthfulQA
+- **Robustness**: Adversarial benchmarks
+
+### Score Formatting
+Intelligent score display based on metric type:
+- Binary metrics → "Pass/Fail"
+- Percentages (0-1) → "81.9%"
+- Other scales → Formatted decimals
+
+### Metadata Tracking
+Each evaluation includes:
+- Source organization and type
+- Evaluator relationship (first/second/third party)
+- Dataset information (samples, version, HF repo)
+- Generation configuration
+- Confidence intervals
+- Links to detailed results
+
+## Benefits of the New System
+
+1. **Standardized**: Uses established schema from evalevalai.com
+2. **Quantitative**: Real scores with statistical measures
+3. **Reproducible**: Includes all evaluation parameters
+4. **Traceable**: Links to sources and detailed results
+5. **Transparent**: Shows who evaluated and when
+6. **Comparative**: Easy model comparisons on same benchmarks
+7. **Extensible**: Simple to add new evaluations
+
+## Usage
+
+### Adding New Evaluations
+1. Create JSON file following the schema in `lib/benchmark-schema.ts`
+2. Place in `/public/benchmarks/`
+3. System automatically loads and displays
+
+### Viewing Evaluations
+1. Navigate to `/benchmarks`
+2. Filter and sort as needed
+3. Click card to view full details
+
+## File Structure
+```
+lib/
+  benchmark-schema.ts       # Type definitions
+  eval-processing.ts        # Processing utilities
+
+components/
+  benchmark-evaluation-card.tsx  # Card component
+  benchmark-detail.tsx           # Detail view
+  navigation.tsx                 # Updated navigation
+
+app/
+  benchmarks/
+    page.tsx               # Listing page
+  benchmark/
+    [id]/
+      page.tsx            # Detail page
+
+public/
+  benchmarks/
+    kimi-k2-instruct.json     # Sample data
+    gpt-4-turbo.json
+    claude-3-sonnet.json
+
+BENCHMARK_SYSTEM.md       # User documentation
+```
+
+## Next Steps (Recommendations)
+
+1. **Data Import**: Create scripts to import from HELM, OpenAI evals, etc.
+2. **Search**: Add full-text search across benchmarks
+3. **Comparison**: Side-by-side model comparison view
+4. **Export**: Export functionality for reports
+5. **API**: Backend API for dynamic data loading
+6. **Caching**: Add caching for loaded evaluations
+7. **Filters**: More advanced filtering (score ranges, dates, etc.)
+8. **Charts**: Visualizations for score distributions
+
+## Breaking Changes
+
+This is a new system that runs alongside the existing checkbox-based evaluation system. The old system remains untouched to allow for gradual migration or parallel use.
+
+To fully migrate:
+1. Convert existing evaluation data to new schema
+2. Update main `/` route to use new system
+3. Archive or remove old evaluation components
+
+## Testing
+
+To test the new system:
+1. Navigate to `/benchmarks`
+2. Verify all 3 sample evaluations load
+3. Test filtering by type and category
+4. Test sorting options
+5. Click a card to view details
+6. Verify all tabs work (All, Capabilities, Risks)
+7. Test external links
+8. Test navigation back to listing
+
+All TypeScript errors have been resolved and the system is ready for use.

app/benchmark/[id]/page.tsx

@@ -0,0 +1,104 @@
+"use client"
+
+import { useEffect, useState } from "react"
+import { useParams, useRouter } from "next/navigation"
+import { Button } from "@/components/ui/button"
+import { ArrowLeft } from "lucide-react"
+import { Navigation } from "@/components/navigation"
+import { BenchmarkDetail } from "@/components/benchmark-detail"
+import { loadEvaluations, groupEvaluationsByModel, createModelSummary } from "@/lib/eval-processing"
+import type { ModelEvaluationSummary } from "@/lib/eval-processing"
+
+export default function BenchmarkDetailPage() {
+  const params = useParams()
+  const router = useRouter()
+  const [summary, setSummary] = useState<ModelEvaluationSummary | null>(null)
+  const [loading, setLoading] = useState(true)
+  const [error, setError] = useState<string | null>(null)
+
+  useEffect(() => {
+    const loadData = async () => {
+      try {
+        const modelId = decodeURIComponent(params.id as string)
+        
+        // Load all benchmark files and find the matching model
+        const benchmarkFiles = [
+          "/benchmarks/meta-llama-3-70b.json",
+          "/benchmarks/mistral-mistral-large.json",
+          "/benchmarks/anthropic-claude-3-5-sonnet.json",
+          "/benchmarks/openai-gpt-4o.json",
+          "/benchmarks/google-gemma-2-27b.json",
+          "/benchmarks/alibaba-qwen-2-72b.json",
+        ]
+        
+        const evaluations = await loadEvaluations(benchmarkFiles)
+        const grouped = groupEvaluationsByModel(evaluations)
+        
+        // Find the model by ID
+        const modelEvals = grouped[modelId]
+        
+        if (!modelEvals || modelEvals.length === 0) {
+          setError("Evaluation not found")
+          return
+        }
+        
+        const modelSummary = createModelSummary(modelEvals)
+        setSummary(modelSummary)
+        document.title = `${modelSummary.model_info.name} - AI Evaluation Dashboard`
+      } catch (err) {
+        console.error("Failed to load evaluation:", err)
+        setError("Failed to load evaluation data")
+      } finally {
+        setLoading(false)
+      }
+    }
+    
+    loadData()
+  }, [params.id])
+
+  if (loading) {
+    return (
+      <div className="min-h-screen bg-background">
+        <Navigation />
+        <main className="container mx-auto px-4 py-8">
+          <div className="flex items-center justify-center h-96">
+            <div className="text-lg text-muted-foreground">Loading evaluation details...</div>
+          </div>
+        </main>
+      </div>
+    )
+  }
+
+  if (error || !summary) {
+    return (
+      <div className="min-h-screen bg-background">
+        <Navigation />
+        <main className="container mx-auto px-4 py-8">
+          <div className="flex flex-col items-center justify-center h-96 space-y-4">
+            <div className="text-lg text-muted-foreground">{error || "Evaluation not found"}</div>
+            <Button onClick={() => router.push("/benchmarks")}>
+              <ArrowLeft className="mr-2 h-4 w-4" />
+              Back to Evaluations
+            </Button>
+          </div>
+        </main>
+      </div>
+    )
+  }
+
+  return (
+    <div className="min-h-screen bg-background">
+      <Navigation />
+      <main className="container mx-auto px-4 py-8">
+        <div className="mb-6">
+          <Button variant="ghost" onClick={() => router.push("/benchmarks")}>
+            <ArrowLeft className="mr-2 h-4 w-4" />
+            Back to Evaluations
+          </Button>
+        </div>
+        
+        <BenchmarkDetail summary={summary} />
+      </main>
+    </div>
+  )
+}

app/benchmarks/page.tsx

@@ -0,0 +1,193 @@
+"use client"
+
+import { useState, useMemo, useEffect } from "react"
+import { Button } from "@/components/ui/button"
+import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select"
+import { Filter, ArrowUpDown, Plus } from "lucide-react"
+import { BenchmarkEvaluationCard, type BenchmarkEvaluationCardData } from "@/components/benchmark-evaluation-card"
+import { Navigation } from "@/components/navigation"
+import { PageHeader } from "@/components/page-header"
+import { processEvaluationsToCards } from "@/lib/eval-processing"
+import type { CategoryType } from "@/lib/benchmark-schema"
+import { EVALUATION_CATEGORIES } from "@/lib/benchmark-schema"
+
+export default function BenchmarksPage() {
+  const [evaluations, setEvaluations] = useState<BenchmarkEvaluationCardData[]>([])
+  const [loading, setLoading] = useState(true)
+  const [sortBy, setSortBy] = useState<"date" | "name" | "benchmarks">("date")
+  const [filterCategory, setFilterCategory] = useState<"all" | CategoryType>("all")
+
+  // Load evaluations on mount
+  useEffect(() => {
+    const loadData = async () => {
+      try {
+        // Load all benchmark evaluation files
+        const benchmarkFiles = [
+          "/benchmarks/meta-llama-3-70b.json",
+          "/benchmarks/mistral-mistral-large.json",
+          "/benchmarks/anthropic-claude-3-5-sonnet.json",
+          "/benchmarks/openai-gpt-4o.json",
+          "/benchmarks/google-gemma-2-27b.json",
+          "/benchmarks/alibaba-qwen-2-72b.json",
+        ]
+        
+        const cards = await processEvaluationsToCards(benchmarkFiles)
+        setEvaluations(cards)
+      } catch (error) {
+        console.error("Failed to load evaluations:", error)
+      } finally {
+        setLoading(false)
+      }
+    }
+    
+    loadData()
+  }, [])
+
+  // Filter evaluations
+  const filteredEvaluations = useMemo(() => {
+    let filtered = [...evaluations]
+    
+    // Filter by specific category
+    if (filterCategory !== "all") {
+      filtered = filtered.filter((eval_) => 
+        eval_.categories.includes(filterCategory)
+      )
+    }
+    
+    return filtered
+  }, [evaluations, filterCategory])
+
+  // Sort evaluations
+  const sortedEvaluations = useMemo(() => {
+    const sorted = [...filteredEvaluations]
+    
+    switch (sortBy) {
+      case "date":
+        sorted.sort((a, b) => 
+          new Date(b.latest_timestamp).getTime() - new Date(a.latest_timestamp).getTime()
+        )
+        break
+      case "name":
+        sorted.sort((a, b) => a.model_name.localeCompare(b.model_name))
+        break
+      case "benchmarks":
+        sorted.sort((a, b) => b.benchmarks_count - a.benchmarks_count)
+        break
+    }
+    
+    return sorted
+  }, [filteredEvaluations, sortBy])
+
+  const handleDelete = (id: string) => {
+    setEvaluations((prev) => prev.filter((e) => e.id !== id))
+  }
+
+  if (loading) {
+    return (
+      <div className="min-h-screen bg-background">
+        <Navigation />
+        <main className="container mx-auto px-4 py-8">
+          <div className="flex items-center justify-center h-96">
+            <div className="text-lg text-muted-foreground">Loading evaluations...</div>
+          </div>
+        </main>
+      </div>
+    )
+  }
+
+  return (
+    <div className="min-h-screen bg-background">
+      <Navigation />
+      <main className="container mx-auto px-4 py-8">
+        <PageHeader
+          title="AI Model Evaluations"
+          description="Browse benchmark evaluation results from standardized datasets"
+        />
+
+        {/* Filters and Controls */}
+        <div className="flex flex-col sm:flex-row gap-4 mb-8">
+          <div className="flex gap-2 flex-1">
+            <Select 
+              value={filterCategory} 
+              onValueChange={(value) => setFilterCategory(value as any)}
+            >
+              <SelectTrigger className="w-[200px]">
+                <SelectValue placeholder="Category" />
+              </SelectTrigger>
+              <SelectContent>
+                <SelectItem value="all">All Categories</SelectItem>
+                {Array.from(EVALUATION_CATEGORIES).map((cat) => (
+                  <SelectItem key={cat} value={cat}>
+                    {cat.split('-').map(w => w.charAt(0).toUpperCase() + w.slice(1)).join(' ')}
+                  </SelectItem>
+                ))}
+              </SelectContent>
+            </Select>
+          </div>
+
+          <Select value={sortBy} onValueChange={(value) => setSortBy(value as any)}>
+            <SelectTrigger className="w-[180px]">
+              <ArrowUpDown className="h-4 w-4 mr-2" />
+              <SelectValue placeholder="Sort by" />
+            </SelectTrigger>
+            <SelectContent>
+              <SelectItem value="date">Latest First</SelectItem>
+              <SelectItem value="name">Name (A-Z)</SelectItem>
+              <SelectItem value="benchmarks">Most Benchmarks</SelectItem>
+            </SelectContent>
+          </Select>
+        </div>
+
+        {/* Stats */}
+        <div className="grid grid-cols-1 md:grid-cols-4 gap-4 mb-8">
+          <div className="p-4 border rounded-lg">
+            <div className="text-2xl font-bold">{sortedEvaluations.length}</div>
+            <div className="text-sm text-muted-foreground">Models Evaluated</div>
+          </div>
+          <div className="p-4 border rounded-lg">
+            <div className="text-2xl font-bold">
+              {sortedEvaluations.reduce((sum, e) => sum + e.benchmarks_count, 0)}
+            </div>
+            <div className="text-sm text-muted-foreground">Total Benchmarks</div>
+          </div>
+          <div className="p-4 border rounded-lg">
+            <div className="text-2xl font-bold">
+              {sortedEvaluations.reduce((sum, e) => sum + e.evaluations_count, 0)}
+            </div>
+            <div className="text-sm text-muted-foreground">Total Evaluations</div>
+          </div>
+          <div className="p-4 border rounded-lg">
+            <div className="text-2xl font-bold">
+              {new Set(sortedEvaluations.flatMap(e => e.categories)).size}
+            </div>
+            <div className="text-sm text-muted-foreground">Categories Covered</div>
+          </div>
+        </div>
+
+        {/* Evaluation Cards */}
+        {sortedEvaluations.length === 0 ? (
+          <div className="text-center py-12">
+            <p className="text-lg text-muted-foreground mb-4">
+              No evaluations found matching your filters
+            </p>
+            <Button onClick={() => {
+              setFilterCategory("all")
+            }}>
+              Clear Filters
+            </Button>
+          </div>
+        ) : (
+          <div className="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-2 gap-6">
+            {sortedEvaluations.map((evaluation) => (
+              <BenchmarkEvaluationCard
+                key={evaluation.id}
+                data={evaluation}
+                onDelete={handleDelete}
+              />
+            ))}
+          </div>
+        )}
+      </main>
+    </div>
+  )
+}

app/layout.tsx

@@ -28,7 +28,7 @@ export default function RootLayout({
   children: React.ReactNode
 }>) {
   return (
-    <html lang="en" className={`${spaceGrotesk.variable} ${dmSans.variable} antialiased`}>
+    <html lang="en" className={`${spaceGrotesk.variable} ${dmSans.variable} antialiased`} suppressHydrationWarning>
       <body className="font-sans">
         <ThemeProvider attribute="class" defaultTheme="system" enableSystem disableTransitionOnChange>
           {children}

app/page.tsx

@@ -3,469 +3,94 @@
 import { useState, useMemo, useEffect } from "react"
 import { Button } from "@/components/ui/button"
 import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select"
-import { Filter, ArrowUpDown, Plus } from "lucide-react"
-import { EvaluationCard, type EvaluationCardData } from "@/components/evaluation-card"
-import { getBenchmarkQuestions, getProcessQuestions } from "@/lib/schema"
-import { AIEvaluationDashboard } from "@/components/ai-evaluation-dashboard"
+import { Filter, ArrowUpDown } from "lucide-react"
+import { BenchmarkEvaluationCard, type BenchmarkEvaluationCardData } from "@/components/benchmark-evaluation-card"
 import { Navigation } from "@/components/navigation"
 import { PageHeader } from "@/components/page-header"
-
-const loadEvaluationData = async (): Promise<EvaluationCardData[]> => {
-  const evaluationFiles = [
-    "/evaluations/gpt-4-turbo.json",
-    "/evaluations/claude-3-sonnet.json",
-    "/evaluations/gemini-pro.json",
-    "/evaluations/fraud-detector.json",
-  ]
-
-  const additionalFiles = []
-  for (let i = 1; i <= 10; i++) {
-    additionalFiles.push(`/evaluations/eval-${Date.now() - i * 86400000}.json`) // Check for files from last 10 days
-  }
-
-  const allFiles = [...evaluationFiles, ...additionalFiles]
-  const evaluations: EvaluationCardData[] = []
-
-  for (const file of allFiles) {
-    try {
-      const response = await fetch(file)
-      if (!response.ok) continue // Skip files that don't exist
-
-      const data = await response.json()
-
-      const cardData: EvaluationCardData = {
-        id: data.id || `eval-${Date.now()}`,
-        systemName: data.systemName || "Unknown System",
-        provider: data.provider || "Unknown Provider",
-        inputModalities: data.inputModalities || ["Text"],
-        outputModalities: data.outputModalities || ["Text"],
-        completedDate: data.evaluationDate || new Date().toISOString().split("T")[0],
-        applicableCategories: data.overallStats?.totalApplicable || 0,
-        completedCategories: data.overallStats?.totalApplicable || 0,
-        status:
-          data.overallStats?.strongCategories?.length >= (data.overallStats?.adequateCategories?.length || 0)
-            ? "strong"
-            : data.overallStats?.adequateCategories?.length >= (data.overallStats?.weakCategories?.length || 0)
-              ? "adequate"
-              : "weak",
-        capabilityEval: {
-          strong: (data.overallStats?.strongCategories || []).filter((cat: string) =>
-            [
-              "language-communication",
-              "social-intelligence",
-              "problem-solving",
-              "creativity-innovation",
-              "learning-memory",
-              "perception-vision",
-              "physical-manipulation",
-              "metacognition",
-              "robotic-intelligence",
-            ].includes(cat),
-          ).length,
-          adequate: (data.overallStats?.adequateCategories || []).filter((cat: string) =>
-            [
-              "language-communication",
-              "social-intelligence",
-              "problem-solving",
-              "creativity-innovation",
-              "learning-memory",
-              "perception-vision",
-              "physical-manipulation",
-              "metacognition",
-              "robotic-intelligence",
-            ].includes(cat),
-          ).length,
-          weak: (data.overallStats?.weakCategories || []).filter((cat: string) =>
-            [
-              "language-communication",
-              "social-intelligence",
-              "problem-solving",
-              "creativity-innovation",
-              "learning-memory",
-              "perception-vision",
-              "physical-manipulation",
-              "metacognition",
-              "robotic-intelligence",
-            ].includes(cat),
-          ).length,
-          insufficient: (data.overallStats?.insufficientCategories || []).filter((cat: string) =>
-            [
-              "language-communication",
-              "social-intelligence",
-              "problem-solving",
-              "creativity-innovation",
-              "learning-memory",
-              "perception-vision",
-              "physical-manipulation",
-              "metacognition",
-              "robotic-intelligence",
-            ].includes(cat),
-          ).length,
-          strongCategories: (data.overallStats?.strongCategories || []).filter((cat: string) =>
-            [
-              "language-communication",
-              "social-intelligence",
-              "problem-solving",
-              "creativity-innovation",
-              "learning-memory",
-              "perception-vision",
-              "physical-manipulation",
-              "metacognition",
-              "robotic-intelligence",
-            ].includes(cat),
-          ),
-          adequateCategories: (data.overallStats?.adequateCategories || []).filter((cat: string) =>
-            [
-              "language-communication",
-              "social-intelligence",
-              "problem-solving",
-              "creativity-innovation",
-              "learning-memory",
-              "perception-vision",
-              "physical-manipulation",
-              "metacognition",
-              "robotic-intelligence",
-            ].includes(cat),
-          ),
-          weakCategories: (data.overallStats?.weakCategories || []).filter((cat: string) =>
-            [
-              "language-communication",
-              "social-intelligence",
-              "problem-solving",
-              "creativity-innovation",
-              "learning-memory",
-              "perception-vision",
-              "physical-manipulation",
-              "metacognition",
-              "robotic-intelligence",
-            ].includes(cat),
-          ),
-          insufficientCategories: (data.overallStats?.insufficientCategories || []).filter((cat: string) =>
-            [
-              "language-communication",
-              "social-intelligence",
-              "problem-solving",
-              "creativity-innovation",
-              "learning-memory",
-              "perception-vision",
-              "physical-manipulation",
-              "metacognition",
-              "robotic-intelligence",
-            ].includes(cat),
-          ),
-          totalApplicable: data.overallStats?.capabilityApplicable || 0,
-        },
-        riskEval: {
-          strong: (data.overallStats?.strongCategories || []).filter((cat: string) =>
-            [
-              "harmful-content",
-              "information-integrity",
-              "privacy-data",
-              "bias-fairness",
-              "security-robustness",
-              "dangerous-capabilities",
-              "human-ai-interaction",
-              "environmental-impact",
-              "economic-displacement",
-              "governance-accountability",
-              "value-chain",
-            ].includes(cat),
-          ).length,
-          adequate: (data.overallStats?.adequateCategories || []).filter((cat: string) =>
-            [
-              "harmful-content",
-              "information-integrity",
-              "privacy-data",
-              "bias-fairness",
-              "security-robustness",
-              "dangerous-capabilities",
-              "human-ai-interaction",
-              "environmental-impact",
-              "economic-displacement",
-              "governance-accountability",
-              "value-chain",
-            ].includes(cat),
-          ).length,
-          weak: (data.overallStats?.weakCategories || []).filter((cat: string) =>
-            [
-              "harmful-content",
-              "information-integrity",
-              "privacy-data",
-              "bias-fairness",
-              "security-robustness",
-              "dangerous-capabilities",
-              "human-ai-interaction",
-              "environmental-impact",
-              "economic-displacement",
-              "governance-accountability",
-              "value-chain",
-            ].includes(cat),
-          ).length,
-          insufficient: (data.overallStats?.insufficientCategories || []).filter((cat: string) =>
-            [
-              "harmful-content",
-              "information-integrity",
-              "privacy-data",
-              "bias-fairness",
-              "security-robustness",
-              "dangerous-capabilities",
-              "human-ai-interaction",
-              "environmental-impact",
-              "economic-displacement",
-              "governance-accountability",
-              "value-chain",
-            ].includes(cat),
-          ).length,
-          strongCategories: (data.overallStats?.strongCategories || []).filter((cat: string) =>
-            [
-              "harmful-content",
-              "information-integrity",
-              "privacy-data",
-              "bias-fairness",
-              "security-robustness",
-              "dangerous-capabilities",
-              "human-ai-interaction",
-              "environmental-impact",
-              "economic-displacement",
-              "governance-accountability",
-              "value-chain",
-            ].includes(cat),
-          ),
-          adequateCategories: (data.overallStats?.adequateCategories || []).filter((cat: string) =>
-            [
-              "harmful-content",
-              "information-integrity",
-              "privacy-data",
-              "bias-fairness",
-              "security-robustness",
-              "dangerous-capabilities",
-              "human-ai-interaction",
-              "environmental-impact",
-              "economic-displacement",
-              "governance-accountability",
-              "value-chain",
-            ].includes(cat),
-          ),
-          weakCategories: (data.overallStats?.weakCategories || []).filter((cat: string) =>
-            [
-              "harmful-content",
-              "information-integrity",
-              "privacy-data",
-              "bias-fairness",
-              "security-robustness",
-              "dangerous-capabilities",
-              "human-ai-interaction",
-              "environmental-impact",
-              "economic-displacement",
-              "governance-accountability",
-              "value-chain",
-            ].includes(cat),
-          ),
-          insufficientCategories: (data.overallStats?.insufficientCategories || []).filter((cat: string) =>
-            [
-              "harmful-content",
-              "information-integrity",
-              "privacy-data",
-              "bias-fairness",
-              "security-robustness",
-              "dangerous-capabilities",
-              "human-ai-interaction",
-              "environmental-impact",
-              "economic-displacement",
-              "governance-accountability",
-              "value-chain",
-            ].includes(cat),
-          ),
-          totalApplicable: data.overallStats?.riskApplicable || 0,
-        },
-        priorityAreas: data.overallStats?.priorityAreas || [],
-        priorityDetails: (() => {
-          // Build a richer structure: for each area, include yes questions and negative questions (no/na) with optional reason
-          const pd: Record<
-            string,
-            {
-              yes: string[]
-              negative: { text: string; status: "no" | "na"; reason?: string }[]
-            }
-          > = {}
-          const areas = data.overallStats?.priorityAreas || []
-          for (const area of areas) {
-            const catEval = data.categoryEvaluations?.[area]
-            if (!catEval) continue
-
-            const yesList: string[] = []
-            const negList: { text: string; status: "no" | "na"; reason?: string }[] = []
-
-            // Helper to detect NA reason from category metadata
-            const naReasonFromMeta = (): string | undefined => {
-              if (typeof catEval.additionalAspects === "string" && /not applicable/i.test(catEval.additionalAspects)) {
-                return catEval.additionalAspects
-              }
-              // look into processSources scopes for any note
-              if (catEval.processSources) {
-                for (const entries of Object.values(catEval.processSources)) {
-                  if (Array.isArray(entries)) {
-                    for (const ent of entries as any[]) {
-                      if (ent && typeof ent.scope === "string" && /not applicable/i.test(ent.scope)) {
-                        return ent.scope
-                      }
-                    }
-                  }
-                }
-              }
-              return undefined
-            }
-
-            const naMeta = naReasonFromMeta()
-
-            // check benchmarkAnswers (A1..A6)
-            if (catEval.benchmarkAnswers) {
-              for (const [qid, ans] of Object.entries(catEval.benchmarkAnswers)) {
-                const answer = ans
-                const isArray = Array.isArray(answer)
-                const negative = answer === "no" || (isArray && (answer as any[]).includes("no"))
-                const positive = answer === "yes" || (isArray && (answer as any[]).includes("yes"))
-                const qText = getBenchmarkQuestions().find((x) => x.id === qid)?.text || qid
-                if (positive) yesList.push(qText)
-                if (negative) {
-                  const status = naMeta ? "na" : "no"
-                  negList.push({ text: qText, status, reason: naMeta })
-                }
-              }
-            }
-
-            // check processAnswers (B1..B6)
-            if (catEval.processAnswers) {
-              for (const [qid, ans] of Object.entries(catEval.processAnswers)) {
-                const answer = ans
-                const isArray = Array.isArray(answer)
-                const negative = answer === "no" || (isArray && (answer as any[]).includes("no"))
-                const positive = answer === "yes" || (isArray && (answer as any[]).includes("yes"))
-                const qText = getProcessQuestions().find((x) => x.id === qid)?.text || qid
-                if (positive) yesList.push(qText)
-                if (negative) {
-                  const status = naMeta ? "na" : "no"
-                  negList.push({ text: qText, status, reason: naMeta })
-                }
-              }
-            }
-
-            if (yesList.length || negList.length) pd[area] = { yes: yesList, negative: negList }
-          }
-          return pd
-        })(),
-      }
-
-      evaluations.push(cardData)
-    } catch (error) {
-      continue
-    }
-  }
-
-  return evaluations
-}
+import { processEvaluationsToCards } from "@/lib/eval-processing"
+import type { CategoryType } from "@/lib/benchmark-schema"
+import { EVALUATION_CATEGORIES } from "@/lib/benchmark-schema"
 
 export default function HomePage() {
-  const [showNewEvaluation, setShowNewEvaluation] = useState(false)
-  const [evaluationsData, setEvaluationsData] = useState<EvaluationCardData[]>([])
+  const [evaluations, setEvaluations] = useState<BenchmarkEvaluationCardData[]>([])
   const [loading, setLoading] = useState(true)
+  const [sortBy, setSortBy] = useState<"date" | "name" | "benchmarks">("date")
+  const [filterCategory, setFilterCategory] = useState<"all" | CategoryType>("all")
 
+  // Load evaluations on mount
   useEffect(() => {
     const loadData = async () => {
-      const data = await loadEvaluationData()
-      setEvaluationsData(data)
-      setLoading(false)
+      try {
+        // Discover all benchmark files dynamically
+        const benchmarkFiles = [
+          "/benchmarks/meta-llama-3-70b.json",
+          "/benchmarks/mistral-mistral-large.json",
+          "/benchmarks/anthropic-claude-3-5-sonnet.json",
+          "/benchmarks/openai-gpt-4o.json",
+          "/benchmarks/google-gemma-2-27b.json",
+          "/benchmarks/alibaba-qwen-2-72b.json",
+        ]
+        
+        const cards = await processEvaluationsToCards(benchmarkFiles)
+        setEvaluations(cards)
+      } catch (error) {
+        console.error("Failed to load evaluations:", error)
+      } finally {
+        setLoading(false)
+      }
     }
+    
     loadData()
   }, [])
 
-  const [sortBy, setSortBy] = useState<"date-newest" | "date-oldest" | "completeness-highest" | "completeness-lowest">("date-newest")
-  const [filterByProvider, setFilterByProvider] = useState<string>("all")
-  const [filterByModality, setFilterByModality] = useState<string>("all")
-
-  const uniqueProviders = useMemo(() => {
-    const providers = [...new Set(evaluationsData.map((item) => item.provider))].sort()
-    return providers
-  }, [evaluationsData])
-
-  const uniqueModalities = useMemo(() => {
-    // Define all possible modalities to ensure complete filter options
-    const allModalities = ["Text", "Image", "Audio", "Video", "Tabular", "Robotics/Action", "Other"]
+  // Filter evaluations
+  const filteredEvaluations = useMemo(() => {
+    let filtered = [...evaluations]
     
-    // Get modalities that actually exist in the data
-    const existingModalities = new Set<string>()
-    evaluationsData.forEach((item) => {
-      item.inputModalities.forEach((mod) => existingModalities.add(mod))
-      item.outputModalities.forEach((mod) => existingModalities.add(mod))
-    })
-
-    // Return only modalities that exist in the data, in the order defined by allModalities
-    return allModalities.filter((mod) => existingModalities.has(mod))
-  }, [evaluationsData])
-
-  const filteredAndSortedEvaluations = useMemo(() => {
-    let filtered = evaluationsData
-
-    if (filterByProvider !== "all") {
-      filtered = filtered.filter((item) => item.provider === filterByProvider)
-    }
-
-    if (filterByModality !== "all") {
-      filtered = filtered.filter((item) => 
-        item.inputModalities.includes(filterByModality) || 
-        item.outputModalities.includes(filterByModality)
+    // Filter by specific category
+    if (filterCategory !== "all") {
+      filtered = filtered.filter((eval_) => 
+        eval_.categories.includes(filterCategory)
       )
     }
-
-    filtered = filtered.sort((a, b) => {
-      if (sortBy.includes("completeness")) {
-        const aCompleteness = (a.completedCategories / a.applicableCategories) * 100
-        const bCompleteness = (b.completedCategories / b.applicableCategories) * 100
-
-        if (sortBy === "completeness-highest") {
-          return bCompleteness - aCompleteness
-        } else {
-          return aCompleteness - bCompleteness
-        }
-      } else {
-        const dateA = new Date(a.completedDate)
-        const dateB = new Date(b.completedDate)
-
-        if (sortBy === "date-newest") {
-          return dateB.getTime() - dateA.getTime()
-        } else {
-          return dateA.getTime() - dateB.getTime()
-        }
-      }
-    })
-
+    
     return filtered
-  }, [evaluationsData, sortBy, filterByProvider, filterByModality])
-
-  const handleViewEvaluation = (id: string) => {}
+  }, [evaluations, filterCategory])
 
-  const handleDeleteEvaluation = (id: string) => {
-    setEvaluationsData((prev) => prev.filter((evaluation) => evaluation.id !== id))
-  }
-
-  const handleSaveEvaluation = (newEvaluation: EvaluationCardData) => {
-    setEvaluationsData((prev) => [newEvaluation, ...prev])
-  }
+  // Sort evaluations
+  const sortedEvaluations = useMemo(() => {
+    const sorted = [...filteredEvaluations]
+    
+    switch (sortBy) {
+      case "date":
+        sorted.sort((a, b) => 
+          new Date(b.latest_timestamp).getTime() - new Date(a.latest_timestamp).getTime()
+        )
+        break
+      case "name":
+        sorted.sort((a, b) => a.model_name.localeCompare(b.model_name))
+        break
+      case "benchmarks":
+        sorted.sort((a, b) => b.benchmarks_count - a.benchmarks_count)
+        break
+    }
+    
+    return sorted
+  }, [filteredEvaluations, sortBy])
 
-  if (showNewEvaluation) {
-    return <AIEvaluationDashboard onBack={() => setShowNewEvaluation(false)} onSaveEvaluation={handleSaveEvaluation} />
+  const handleDelete = (id: string) => {
+    setEvaluations((prev) => prev.filter((e) => e.id !== id))
   }
 
   if (loading) {
     return (
       <div className="min-h-screen bg-background">
         <Navigation />
-        <div className="flex items-center justify-center min-h-[60vh]">
-          <div className="text-center">
-            <div className="animate-spin rounded-full h-8 w-8 border-b-2 border-primary mx-auto mb-4"></div>
-            <p className="text-muted-foreground">Loading evaluations...</p>
+        <main className="container mx-auto px-4 py-8">
+          <div className="flex items-center justify-center h-96">
+            <div className="text-lg text-muted-foreground">Loading evaluations...</div>
           </div>
-        </div>
+        </main>
       </div>
     )
   }
@@ -473,107 +98,96 @@ export default function HomePage() {
   return (
     <div className="min-h-screen bg-background">
       <Navigation />
-      
-      <PageHeader 
-        title="Evaluation Cards"
-        description={`Track and manage AI system evaluations across capabilities and risks. ${filteredAndSortedEvaluations.length} eval cards available.`}
-      >
-        <Button onClick={() => setShowNewEvaluation(true)} className="gap-2">
-          <Plus className="h-4 w-4" />
-          <span className="hidden sm:inline">New Eval Card</span>
-          <span className="sm:hidden">New</span>
-        </Button>
-      </PageHeader>
-
-      <div className="container mx-auto px-4 sm:px-6 py-6">
-        <div className="space-y-6">
-          {/* Filters */}
-          <div className="flex flex-col sm:flex-row sm:flex-wrap items-start sm:items-center gap-3 sm:gap-4 p-4 bg-card rounded-lg border">
-            <div className="flex items-center gap-2 w-full sm:w-auto">
-              <ArrowUpDown className="h-4 w-4 text-muted-foreground" />
-              <span className="text-sm font-medium">Sort by:</span>
-              <Select value={sortBy} onValueChange={(value: "date-newest" | "date-oldest" | "completeness-highest" | "completeness-lowest") => setSortBy(value)}>
-                <SelectTrigger className="w-full sm:w-48">
-                  <SelectValue />
-                </SelectTrigger>
-                <SelectContent>
-                  <SelectItem value="date-newest">Date (Newest)</SelectItem>
-                  <SelectItem value="date-oldest">Date (Oldest)</SelectItem>
-                  <SelectItem value="completeness-highest">Completeness (Highest)</SelectItem>
-                  <SelectItem value="completeness-lowest">Completeness (Lowest)</SelectItem>
-                </SelectContent>
-              </Select>
-            </div>
+      <main className="container mx-auto px-4 py-8">
+        <PageHeader
+          title="AI Model Evaluations"
+          description="Browse benchmark evaluation results from standardized datasets"
+        />
+
+        {/* Filters and Controls */}
+        <div className="flex flex-col sm:flex-row gap-4 mb-8">
+          <div className="flex gap-2 flex-1">
+            <Select 
+              value={filterCategory} 
+              onValueChange={(value) => setFilterCategory(value as any)}
+            >
+              <SelectTrigger className="w-[200px]">
+                <SelectValue placeholder="Category" />
+              </SelectTrigger>
+              <SelectContent>
+                <SelectItem value="all">All Categories</SelectItem>
+                {Array.from(EVALUATION_CATEGORIES).map((cat) => (
+                  <SelectItem key={cat} value={cat}>
+                    {cat.split('-').map(w => w.charAt(0).toUpperCase() + w.slice(1)).join(' ')}
+                  </SelectItem>
+                ))}
+              </SelectContent>
+            </Select>
+          </div>
 
-            <div className="flex items-center gap-2 w-full sm:w-auto">
-              <Filter className="h-4 w-4 text-muted-foreground" />
-              <span className="text-sm font-medium">Provider:</span>
-              <Select value={filterByProvider} onValueChange={setFilterByProvider}>
-                <SelectTrigger className="w-full sm:w-40">
-                  <SelectValue />
-                </SelectTrigger>
-                <SelectContent>
-                  <SelectItem value="all">All Providers</SelectItem>
-                  {uniqueProviders.map((provider) => (
-                    <SelectItem key={provider} value={provider}>
-                      {provider}
-                    </SelectItem>
-                  ))}
-                </SelectContent>
-              </Select>
-            </div>
+          <Select value={sortBy} onValueChange={(value) => setSortBy(value as any)}>
+            <SelectTrigger className="w-[180px]">
+              <ArrowUpDown className="h-4 w-4 mr-2" />
+              <SelectValue placeholder="Sort by" />
+            </SelectTrigger>
+            <SelectContent>
+              <SelectItem value="date">Latest First</SelectItem>
+              <SelectItem value="name">Name (A-Z)</SelectItem>
+              <SelectItem value="benchmarks">Most Benchmarks</SelectItem>
+            </SelectContent>
+          </Select>
+        </div>
 
-            <div className="flex items-center gap-2 w-full sm:w-auto">
-              <Filter className="h-4 w-4 text-muted-foreground" />
-              <span className="text-sm font-medium">Modality:</span>
-              <Select value={filterByModality} onValueChange={setFilterByModality}>
-                <SelectTrigger className="w-full sm:w-40">
-                  <SelectValue />
-                </SelectTrigger>
-                <SelectContent>
-                  <SelectItem value="all">All Modalities</SelectItem>
-                  {uniqueModalities.map((modality) => (
-                    <SelectItem key={modality} value={modality}>
-                      {modality}
-                    </SelectItem>
-                  ))}
-                </SelectContent>
-              </Select>
+        {/* Stats */}
+        <div className="grid grid-cols-1 md:grid-cols-4 gap-4 mb-8">
+          <div className="p-4 border rounded-lg">
+            <div className="text-2xl font-bold">{sortedEvaluations.length}</div>
+            <div className="text-sm text-muted-foreground">Models Evaluated</div>
+          </div>
+          <div className="p-4 border rounded-lg">
+            <div className="text-2xl font-bold">
+              {sortedEvaluations.reduce((sum, e) => sum + e.benchmarks_count, 0)}
             </div>
+            <div className="text-sm text-muted-foreground">Total Benchmarks</div>
           </div>
-
-          {/* Results */}
-          {filteredAndSortedEvaluations.length > 0 ? (
-            <div className="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-6">
-              {filteredAndSortedEvaluations.map((evaluation) => (
-                <EvaluationCard
-                  key={evaluation.id}
-                  evaluation={evaluation}
-                  onView={handleViewEvaluation}
-                  onDelete={handleDeleteEvaluation}
-                />
-              ))}
+          <div className="p-4 border rounded-lg">
+            <div className="text-2xl font-bold">
+              {sortedEvaluations.reduce((sum, e) => sum + e.evaluations_count, 0)}
             </div>
-          ) : (
-            <div className="text-center py-12">
-              <div className="mx-auto w-24 h-24 bg-muted rounded-full flex items-center justify-center mb-4">
-                <Filter className="h-8 w-8 text-muted-foreground" />
-              </div>
-              <h3 className="text-lg font-semibold mb-2">No evaluations match your filters</h3>
-              <p className="text-muted-foreground mb-4">Try adjusting your filter criteria to see more results</p>
-              <Button
-                variant="outline"
-                onClick={() => {
-                  setFilterByProvider("all")
-                  setFilterByModality("all")
-                }}
-              >
-                Clear Filters
-              </Button>
+            <div className="text-sm text-muted-foreground">Total Evaluations</div>
+          </div>
+          <div className="p-4 border rounded-lg">
+            <div className="text-2xl font-bold">
+              {new Set(sortedEvaluations.flatMap(e => e.categories)).size}
             </div>
-          )}
+            <div className="text-sm text-muted-foreground">Categories Covered</div>
+          </div>
         </div>
-      </div>
+
+        {/* Evaluation Cards */}
+        {sortedEvaluations.length === 0 ? (
+          <div className="text-center py-12">
+            <p className="text-lg text-muted-foreground mb-4">
+              No evaluations found matching your filters
+            </p>
+            <Button onClick={() => {
+              setFilterCategory("all")
+            }}>
+              Clear Filters
+            </Button>
+          </div>
+        ) : (
+          <div className="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-2 gap-6">
+            {sortedEvaluations.map((evaluation) => (
+              <BenchmarkEvaluationCard
+                key={evaluation.id}
+                data={evaluation}
+                onDelete={handleDelete}
+              />
+            ))}
+          </div>
+        )}
+      </main>
     </div>
   )
 }

components/benchmark-detail.tsx

@@ -0,0 +1,1009 @@
+"use client"
+
+// Force recompile
+import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card"
+import { Badge } from "@/components/ui/badge"
+import { Button } from "@/components/ui/button"
+import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs"
+import { Separator } from "@/components/ui/separator"
+import { Progress } from "@/components/ui/progress"
+import { Accordion, AccordionContent, AccordionItem, AccordionTrigger } from "@/components/ui/accordion"
+import { Collapsible, CollapsibleContent, CollapsibleTrigger } from "@/components/ui/collapsible"
+import { Dialog, DialogContent, DialogDescription, DialogHeader, DialogTitle, DialogTrigger } from "@/components/ui/dialog"
+import { ScrollArea } from "@/components/ui/scroll-area"
+import { Input } from "@/components/ui/input"
+import { Table, TableBody, TableCell, TableHead, TableHeader, TableRow } from "@/components/ui/table"
+import { 
+  ExternalLink, TrendingUp, Info, Database, Settings, FileCode, Building, Calendar, User, Server, 
+  ChevronDown, ChevronUp, BarChart3, Award, AlertTriangle,
+  Cpu, Tag, Globe, Network, Activity, MessageSquare, Clock, Hash, Layers, CheckCircle, Search
+} from "lucide-react"
+import type { BenchmarkEvaluation, CategoryType, EvaluationResult } from "@/lib/benchmark-schema"
+import { inferCategoryFromBenchmark, EVALUATION_CATEGORIES } from "@/lib/benchmark-schema"
+import { formatScore, getBenchmarkDisplayName, getCategoryStats } from "@/lib/eval-processing"
+import type { ModelEvaluationSummary } from "@/lib/eval-processing"
+import { useState, useEffect, useMemo } from "react"
+
+interface BenchmarkDetailProps {
+  summary: ModelEvaluationSummary
+}
+
+export function BenchmarkDetail({ summary }: BenchmarkDetailProps) {
+  const stats = getCategoryStats(summary)
+  
+  // Calculate additional summary stats
+  const categoryScores = stats.categories.map(c => ({
+    category: c.category,
+    score: c.avg_score,
+    count: c.count
+  }));
+  
+  const bestCategory = [...categoryScores].sort((a, b) => b.score - a.score)[0];
+  const worstCategory = [...categoryScores].sort((a, b) => a.score - b.score)[0];
+  
+  const overallAvg = categoryScores.reduce((acc, curr) => acc + (curr.score * curr.count), 0) / summary.total_evaluations;
+
+  const formatDate = (isoString: string) => {
+    try {
+      return new Date(isoString).toLocaleDateString('en-US', {
+        year: 'numeric',
+        month: 'long',
+        day: 'numeric',
+        hour: '2-digit',
+        minute: '2-digit'
+      })
+    } catch {
+      return isoString
+    }
+  }
+  
+  return (
+    <div className="space-y-6">
+      {/* System Information Card */}
+      <Card className="overflow-hidden">
+        <CardHeader className="pb-4 border-b bg-muted/10">
+          <div className="flex items-center gap-2">
+            <Database className="h-5 w-5 text-primary" />
+            <div>
+              <CardTitle className="text-xl">System Information</CardTitle>
+              <CardDescription>Metadata about the evaluated system</CardDescription>
+            </div>
+          </div>
+        </CardHeader>
+        <CardContent className="p-6 space-y-8">
+          {/* Main Grid */}
+          <div className="grid grid-cols-1 md:grid-cols-2 gap-x-12 gap-y-8">
+            {/* Left Column */}
+            <div className="space-y-6">
+              <div className="flex gap-3">
+                <div className="mt-1 bg-blue-100 dark:bg-blue-900/30 p-2 rounded-md h-fit">
+                  <Cpu className="h-4 w-4 text-blue-600 dark:text-blue-400" />
+                </div>
+                <div>
+                  <div className="text-sm text-muted-foreground mb-1">System Name</div>
+                  <div className="font-semibold text-lg">{summary.model_info.name}</div>
+                </div>
+              </div>
+
+              <div className="flex gap-3">
+                <div className="mt-1 bg-green-100 dark:bg-green-900/30 p-2 rounded-md h-fit">
+                  <Tag className="h-4 w-4 text-green-600 dark:text-green-400" />
+                </div>
+                <div>
+                  <div className="text-sm text-muted-foreground mb-1">System Version</div>
+                  <div className="font-medium">{summary.model_info.model_version || "N/A"}</div>
+                </div>
+              </div>
+
+              <div className="flex gap-3">
+                <div className="mt-1 bg-purple-100 dark:bg-purple-900/30 p-2 rounded-md h-fit">
+                  <Building className="h-4 w-4 text-purple-600 dark:text-purple-400" />
+                </div>
+                <div>
+                  <div className="text-sm text-muted-foreground mb-1">Provider</div>
+                  <div className="font-medium">{summary.model_info.developer}</div>
+                </div>
+              </div>
+
+              <div className="flex gap-3">
+                <div className="mt-1 bg-indigo-100 dark:bg-indigo-900/30 p-2 rounded-md h-fit">
+                  <Globe className="h-4 w-4 text-indigo-600 dark:text-indigo-400" />
+                </div>
+                <div>
+                  <div className="text-sm text-muted-foreground mb-1">URL</div>
+                  {summary.model_info.model_url ? (
+                    <a href={summary.model_info.model_url} target="_blank" rel="noopener noreferrer" className="font-medium underline decoration-dotted hover:text-primary truncate block max-w-[200px]">
+                      {summary.model_info.model_url.replace(/^https?:\/\//, '')}
+                    </a>
+                  ) : (
+                    <div className="font-medium text-muted-foreground">N/A</div>
+                  )}
+                </div>
+              </div>
+            </div>
+
+            {/* Right Column */}
+            <div className="space-y-6">
+              <div className="flex gap-3">
+                <div className="mt-1 bg-cyan-100 dark:bg-cyan-900/30 p-2 rounded-md h-fit">
+                  <Network className="h-4 w-4 text-cyan-600 dark:text-cyan-400" />
+                </div>
+                <div>
+                  <div className="text-sm text-muted-foreground mb-1">Deployment Context</div>
+                  <div className="font-medium">
+                    {summary.model_info.additional_details?.deployment_context || "General Purpose"}
+                  </div>
+                </div>
+              </div>
+
+              <div className="flex gap-3">
+                <div className="mt-1 bg-emerald-100 dark:bg-emerald-900/30 p-2 rounded-md h-fit">
+                  <Activity className="h-4 w-4 text-emerald-600 dark:text-emerald-400" />
+                </div>
+                <div>
+                  <div className="text-sm text-muted-foreground mb-1">Input Modalities</div>
+                  <div className="flex flex-wrap gap-2">
+                    {summary.model_info.modalities?.input.map(m => (
+                      <Badge key={m} variant="secondary" className="font-normal">{m}</Badge>
+                    )) || <span className="text-muted-foreground">Text</span>}
+                  </div>
+                </div>
+              </div>
+
+              <div className="flex gap-3">
+                <div className="mt-1 bg-rose-100 dark:bg-rose-900/30 p-2 rounded-md h-fit">
+                  <MessageSquare className="h-4 w-4 text-rose-600 dark:text-rose-400" />
+                </div>
+                <div>
+                  <div className="text-sm text-muted-foreground mb-1">Output Modalities</div>
+                  <div className="flex flex-wrap gap-2">
+                    {summary.model_info.modalities?.output.map(m => (
+                      <Badge key={m} variant="secondary" className="font-normal">{m}</Badge>
+                    )) || <span className="text-muted-foreground">Text</span>}
+                  </div>
+                </div>
+              </div>
+
+              <div className="flex gap-3">
+                <div className="mt-1 bg-amber-100 dark:bg-amber-900/30 p-2 rounded-md h-fit">
+                  <Clock className="h-4 w-4 text-amber-600 dark:text-amber-400" />
+                </div>
+                <div>
+                  <div className="text-sm text-muted-foreground mb-1">Knowledge Cutoff / Release</div>
+                  <div className="font-medium">
+                    {summary.model_info.release_date ? formatDate(summary.model_info.release_date).split(',')[0] : "Unknown"}
+                  </div>
+                </div>
+              </div>
+            </div>
+          </div>
+
+          {/* Colored Boxes */}
+          <div className="grid grid-cols-1 md:grid-cols-3 gap-4 pt-4">
+            <div className="bg-blue-50 dark:bg-blue-900/20 border border-blue-100 dark:border-blue-900/30 rounded-lg p-4">
+              <div className="flex items-center gap-2 text-blue-600 dark:text-blue-400 mb-2">
+                <Hash className="h-4 w-4" />
+                <span className="text-sm font-semibold">System ID</span>
+              </div>
+              <div className="font-mono text-sm truncate" title={summary.model_info.id}>
+                {summary.model_info.id}
+              </div>
+            </div>
+
+            <div className="bg-green-50 dark:bg-green-900/20 border border-green-100 dark:border-green-900/30 rounded-lg p-4">
+              <div className="flex items-center gap-2 text-green-600 dark:text-green-400 mb-2">
+                <Layers className="h-4 w-4" />
+                <span className="text-sm font-semibold">System Types</span>
+              </div>
+              <div className="font-medium text-sm truncate">
+                {summary.model_info.architecture || summary.model_info.inference_engine || "Model"}
+              </div>
+            </div>
+
+            <div className="bg-purple-50 dark:bg-purple-900/20 border border-purple-100 dark:border-purple-900/30 rounded-lg p-4">
+              <div className="flex items-center gap-2 text-purple-600 dark:text-purple-400 mb-2">
+                <Calendar className="h-4 w-4" />
+                <span className="text-sm font-semibold">Evaluation Date</span>
+              </div>
+              <div className="font-medium text-sm">
+                {formatDate(summary.last_updated).split(' at ')[0]}
+              </div>
+            </div>
+          </div>
+
+          {/* Footer Stats */}
+          <div className="flex flex-col md:flex-row gap-8 pt-4 border-t">
+            <div className="flex gap-3">
+              <div className="mt-1 bg-muted p-2 rounded-md h-fit">
+                <User className="h-4 w-4 text-muted-foreground" />
+              </div>
+              <div>
+                <div className="text-sm text-muted-foreground mb-1">Evaluator</div>
+                <div className="font-medium">Aggregated Benchmarks</div>
+              </div>
+            </div>
+
+            <div className="flex gap-3">
+              <div className="mt-1 bg-pink-100 dark:bg-pink-900/30 p-2 rounded-md h-fit">
+                <Activity className="h-4 w-4 text-pink-600 dark:text-pink-400" />
+              </div>
+              <div>
+                <div className="text-sm text-muted-foreground mb-1">Completeness Score</div>
+                <div className="font-bold text-lg">
+                  {Math.round((stats.categories.length / EVALUATION_CATEGORIES.length) * 100)}%
+                </div>
+              </div>
+            </div>
+          </div>
+        </CardContent>
+      </Card>
+
+      {/* Evaluation Summary Stats */}
+      <div className="grid grid-cols-2 md:grid-cols-4 gap-4 text-center">
+        <div className="p-4 bg-muted/20 rounded-lg border">
+          <div className="flex items-center justify-center gap-2 mb-1">
+            <Database className="h-4 w-4 text-muted-foreground" />
+            <span className="text-sm text-muted-foreground">Total Evals</span>
+          </div>
+          <div className="text-3xl font-bold text-primary">{summary.total_evaluations}</div>
+        </div>
+        <div className="p-4 bg-muted/20 rounded-lg border">
+          <div className="flex items-center justify-center gap-2 mb-1">
+            <BarChart3 className="h-4 w-4 text-muted-foreground" />
+            <span className="text-sm text-muted-foreground">Avg Score (Norm)</span>
+          </div>
+          <div className="text-3xl font-bold text-blue-600">{(overallAvg * 100).toFixed(1)}%</div>
+        </div>
+        {bestCategory && (
+          <div className="p-4 bg-green-50/50 dark:bg-green-900/10 rounded-lg border border-green-100 dark:border-green-900/20">
+            <div className="flex items-center justify-center gap-2 mb-1">
+              <Award className="h-4 w-4 text-green-600" />
+              <span className="text-sm text-muted-foreground">Best Category</span>
+            </div>
+            <div className="text-lg font-bold text-green-700 dark:text-green-400 truncate" title={bestCategory.category}>
+              {bestCategory.category.split('-').map(w => w.charAt(0).toUpperCase() + w.slice(1)).join(' ')}
+            </div>
+            <div className="text-xs text-green-600/80">{(bestCategory.score * 100).toFixed(1)}% avg</div>
+          </div>
+        )}
+        {worstCategory && (
+          <div className="p-4 bg-red-50/50 dark:bg-red-900/10 rounded-lg border border-red-100 dark:border-red-900/20">
+            <div className="flex items-center justify-center gap-2 mb-1">
+              <AlertTriangle className="h-4 w-4 text-red-600" />
+              <span className="text-sm text-muted-foreground">Needs Improvement</span>
+            </div>
+            <div className="text-lg font-bold text-red-700 dark:text-red-400 truncate" title={worstCategory.category}>
+              {worstCategory.category.split('-').map(w => w.charAt(0).toUpperCase() + w.slice(1)).join(' ')}
+            </div>
+            <div className="text-xs text-red-600/80">{(worstCategory.score * 100).toFixed(1)}% avg</div>
+          </div>
+        )}
+      </div>
+
+      {/* Categories View */}
+      <Accordion type="multiple" className="space-y-4" defaultValue={stats.categories.map(c => c.category)}>
+        {stats.categories.map((stat) => {
+          const evals = summary.evaluations_by_category[stat.category] || []
+          
+          // Collect all results for this category across all evaluations
+          const categoryResults: { evaluation: BenchmarkEvaluation, result: EvaluationResult }[] = []
+          
+          evals.forEach(eval_ => {
+            eval_.evaluation_results.forEach(result => {
+              let resultCategory: CategoryType | undefined;
+
+              // Try to get category from factsheet first
+              if (result.factsheet?.functional_props) {
+                const props = result.factsheet.functional_props.split(';').map(p => p.trim());
+                // Check if the current category we are rendering is in the props
+                if (props.includes(stat.category)) {
+                  resultCategory = stat.category;
+                }
+              }
+
+              // If not found in factsheet, try to infer
+              if (!resultCategory) {
+                const inferred = inferCategoryFromBenchmark(result.evaluation_name);
+                if (inferred === stat.category) {
+                  resultCategory = inferred;
+                }
+              }
+              
+              if (resultCategory === stat.category) {
+                categoryResults.push({ evaluation: eval_, result })
+              }
+            })
+          })
+          
+          if (categoryResults.length === 0) return null
+
+          return (
+            <AccordionItem key={stat.category} value={stat.category} className="border rounded-lg px-4">
+              <AccordionTrigger className="hover:no-underline py-4">
+                <div className="flex items-center gap-4">
+                  <h2 className="text-xl font-bold tracking-tight capitalize">
+                    {stat.category.replace(/-/g, ' ')}
+                  </h2>
+                  <Badge variant="secondary" className="text-sm">
+                    {categoryResults.length} Benchmarks
+                  </Badge>
+                  <div className="text-sm text-muted-foreground font-normal">
+                    Avg: {(stat.avg_score * 100).toFixed(1)}%
+                  </div>
+                </div>
+              </AccordionTrigger>
+              <AccordionContent className="pt-2 pb-6">
+                <div className="grid grid-cols-1 gap-4">
+                  {categoryResults.map((item, idx) => (
+                    <BenchmarkResultCard 
+                      key={`${item.evaluation.evaluation_id}-${idx}`}
+                      evaluation={item.evaluation}
+                      result={item.result}
+                    />
+                  ))}
+                </div>
+              </AccordionContent>
+            </AccordionItem>
+          )
+        })}
+      </Accordion>
+    </div>
+  )
+}
+
+function SampleDataDialog({ 
+  samples, 
+  evaluationName 
+}: { 
+  samples: any[], 
+  evaluationName: string 
+}) {
+  const [open, setOpen] = useState(false)
+  const [searchTerm, setSearchTerm] = useState("")
+  const [currentPage, setCurrentPage] = useState(1)
+  const itemsPerPage = 10
+
+  const filteredSamples = samples.filter(sample => 
+    sample.input.toLowerCase().includes(searchTerm.toLowerCase()) ||
+    sample.response.toLowerCase().includes(searchTerm.toLowerCase()) ||
+    sample.ground_truth.toLowerCase().includes(searchTerm.toLowerCase())
+  )
+
+  const totalPages = Math.ceil(filteredSamples.length / itemsPerPage)
+  const startIndex = (currentPage - 1) * itemsPerPage
+  const currentSamples = filteredSamples.slice(startIndex, startIndex + itemsPerPage)
+
+  // Reset page when search changes
+  useEffect(() => {
+    setCurrentPage(1)
+  }, [searchTerm])
+
+  return (
+    <Dialog open={open} onOpenChange={setOpen}>
+      <DialogTrigger asChild>
+        <Button variant="outline" size="sm" className="gap-2">
+          <Database className="h-4 w-4" />
+          View All {samples.length} Samples
+        </Button>
+      </DialogTrigger>
+      <DialogContent className="max-w-[90vw] h-[80vh] flex flex-col">
+        <DialogHeader>
+          <DialogTitle>Sample Level Data</DialogTitle>
+          <DialogDescription>
+            Detailed results for {samples.length} samples from {evaluationName}
+          </DialogDescription>
+        </DialogHeader>
+        
+        <div className="flex items-center py-4">
+          <div className="relative w-full max-w-sm">
+            <Search className="absolute left-2 top-2.5 h-4 w-4 text-muted-foreground" />
+            <Input
+              placeholder="Search samples..."
+              value={searchTerm}
+              onChange={(e) => setSearchTerm(e.target.value)}
+              className="pl-8"
+            />
+          </div>
+          <div className="ml-auto text-sm text-muted-foreground">
+            Showing {startIndex + 1}-{Math.min(startIndex + itemsPerPage, filteredSamples.length)} of {filteredSamples.length}
+          </div>
+        </div>
+
+        <div className="flex-1 border rounded-md overflow-hidden">
+          <div className="h-full overflow-auto">
+            <Table>
+              <TableHeader>
+                <TableRow>
+                  <TableHead className="w-[80px]">ID</TableHead>
+                  <TableHead className="min-w-[300px]">Input</TableHead>
+                  <TableHead className="min-w-[300px]">Model Response</TableHead>
+                  <TableHead className="min-w-[300px]">Ground Truth</TableHead>
+                </TableRow>
+              </TableHeader>
+              <TableBody>
+                {currentSamples.length > 0 ? (
+                  currentSamples.map((sample, idx) => (
+                    <TableRow key={idx}>
+                      <TableCell className="font-mono text-xs align-top">
+                        {sample.sample_id || idx}
+                      </TableCell>
+                      <TableCell className="align-top">
+                        <div className="whitespace-pre-wrap text-xs font-mono max-h-[200px] overflow-y-auto">
+                          {sample.input}
+                        </div>
+                      </TableCell>
+                      <TableCell className="align-top">
+                        <div className="whitespace-pre-wrap text-xs text-blue-600 dark:text-blue-400 max-h-[200px] overflow-y-auto">
+                          {sample.response}
+                        </div>
+                      </TableCell>
+                      <TableCell className="align-top">
+                        <div className="whitespace-pre-wrap text-xs text-green-600 dark:text-green-400 max-h-[200px] overflow-y-auto">
+                          {sample.ground_truth}
+                        </div>
+                      </TableCell>
+                    </TableRow>
+                  ))
+                ) : (
+                  <TableRow>
+                    <TableCell colSpan={4} className="h-24 text-center">
+                      No results found.
+                    </TableCell>
+                  </TableRow>
+                )}
+              </TableBody>
+            </Table>
+          </div>
+        </div>
+
+        <div className="flex items-center justify-end space-x-2 py-4">
+          <Button
+            variant="outline"
+            size="sm"
+            onClick={() => setCurrentPage(p => Math.max(1, p - 1))}
+            disabled={currentPage === 1}
+          >
+            Previous
+          </Button>
+          <div className="text-sm font-medium">
+            Page {currentPage} of {totalPages || 1}
+          </div>
+          <Button
+            variant="outline"
+            size="sm"
+            onClick={() => setCurrentPage(p => Math.min(totalPages, p + 1))}
+            disabled={currentPage === totalPages || totalPages === 0}
+          >
+            Next
+          </Button>
+        </div>
+      </DialogContent>
+    </Dialog>
+  )
+}
+
+function BenchmarkResultCard({ 
+  evaluation, 
+  result 
+}: { 
+  evaluation: BenchmarkEvaluation, 
+  result: EvaluationResult 
+}) {
+  const [isOpen, setIsOpen] = useState(false)
+
+  const randomSample = useMemo(() => {
+    const samples = evaluation.detailed_evaluation_results_per_samples;
+    if (!samples || samples.length === 0) return null;
+    // Use a simple hash of the evaluation ID to pick a consistent "random" sample for this session
+    // or just Math.random() if we don't mind it changing on refresh
+    const randomIndex = Math.floor(Math.random() * samples.length);
+    return samples[randomIndex];
+  }, [evaluation.detailed_evaluation_results_per_samples]);
+
+  const formatDate = (timestamp: string) => {
+    try {
+      // Handle unix timestamp (seconds or milliseconds)
+      const ts = parseFloat(timestamp)
+      const date = new Date(ts > 10000000000 ? ts : ts * 1000)
+      return date.toLocaleDateString('en-US', {
+        year: 'numeric',
+        month: 'short',
+        day: 'numeric'
+      })
+    } catch {
+      return timestamp
+    }
+  }
+
+  const { score } = result.score_details
+  const { min_score = 0, max_score = 1, unit, lower_is_better } = result.metric_config
+  
+  // Normalize to 0-1 for color coding
+  let normalized = (score - min_score) / (max_score - min_score)
+  if (lower_is_better) normalized = 1 - normalized
+  
+  const isHigh = normalized >= 0.8
+  const isMedium = normalized >= 0.6
+  
+  let displayScore = score.toFixed(2)
+  let displayUnit = unit || "Accuracy"
+  
+  if (unit === 'accuracy' || !unit) {
+      displayScore = (score * 100).toFixed(1) + "%"
+      displayUnit = "Accuracy"
+  } else if (unit === 'points') {
+      displayScore = score.toFixed(1)
+      displayUnit = "/ 10"
+  } else if (unit === 'pass@1') {
+      displayScore = (score * 100).toFixed(1) + "%"
+      displayUnit = "Pass@1"
+  } else {
+      displayUnit = unit.charAt(0).toUpperCase() + unit.slice(1)
+  }
+
+  return (
+    <Collapsible open={isOpen} onOpenChange={setIsOpen}>
+      <Card className="overflow-hidden border-l-4 border-l-primary">
+        <div className="bg-card p-4 flex justify-between items-center">
+          <div className="flex-1">
+            <div className="flex items-center gap-2">
+              <h3 className="text-lg font-bold">{result.evaluation_name}</h3>
+              <Badge variant="outline" className="text-xs font-normal text-muted-foreground">
+                {result.metric_config.score_type}
+              </Badge>
+            </div>
+            <p className="text-muted-foreground text-sm mt-1 line-clamp-1">{result.metric_config.evaluation_description}</p>
+          </div>
+          
+          <div className="flex items-center gap-6">
+            <div className="text-right">
+              <div className="text-2xl font-bold">{displayScore}</div>
+              <div className="text-xs text-muted-foreground">{displayUnit}</div>
+            </div>
+            <CollapsibleTrigger asChild>
+              <Button variant="ghost" size="sm" className="w-9 p-0">
+                {isOpen ? <ChevronUp className="h-4 w-4" /> : <ChevronDown className="h-4 w-4" />}
+                <span className="sr-only">Toggle details</span>
+              </Button>
+            </CollapsibleTrigger>
+          </div>
+        </div>
+
+        <CollapsibleContent>
+          <Separator />
+          <CardContent className="p-6 space-y-6 bg-muted/5">
+            {/* Source Provenance */}
+            <div>
+              <div className="flex items-center gap-2 mb-3">
+                <Database className="h-4 w-4 text-primary" />
+                <div className="text-xs font-semibold text-muted-foreground uppercase tracking-wider">Source Provenance</div>
+              </div>
+              
+              <div className="grid grid-cols-1 md:grid-cols-2 gap-6 bg-muted/10 p-4 rounded-lg border">
+                {/* Source Metadata */}
+                <div className="space-y-3">
+                  <h4 className="text-sm font-semibold text-primary/80">Evaluator Metadata</h4>
+                  <div className="space-y-2 text-sm">
+                    <div className="flex justify-between">
+                      <span className="text-muted-foreground">Organization:</span>
+                      <span className="font-medium">{evaluation.source_metadata.source_organization_name}</span>
+                    </div>
+                    <div className="flex justify-between">
+                      <span className="text-muted-foreground">Relationship:</span>
+                      <Badge variant="outline" className="text-xs">{evaluation.source_metadata.evaluator_relationship}</Badge>
+                    </div>
+                    <div className="flex justify-between">
+                      <span className="text-muted-foreground">Source Type:</span>
+                      <span className="capitalize">{evaluation.source_metadata.source_type.replace(/_/g, ' ')}</span>
+                    </div>
+                    {evaluation.source_metadata.source_url && (
+                      <div className="flex justify-between">
+                        <span className="text-muted-foreground">URL:</span>
+                        <a href={evaluation.source_metadata.source_url} target="_blank" rel="noopener noreferrer" className="text-primary hover:underline flex items-center gap-1">
+                          Link <ExternalLink className="h-3 w-3" />
+                        </a>
+                      </div>
+                    )}
+                    <div className="flex justify-between">
+                      <span className="text-muted-foreground">Date:</span>
+                      <span>{formatDate(evaluation.retrieved_timestamp)}</span>
+                    </div>
+                  </div>
+                </div>
+
+                {/* Source Data */}
+                <div className="space-y-3">
+                  <h4 className="text-sm font-semibold text-primary/80">Dataset Information</h4>
+                  <div className="space-y-2 text-sm">
+                    <div className="flex justify-between">
+                      <span className="text-muted-foreground">Name:</span>
+                      <span className="font-medium">
+                        {Array.isArray(evaluation.source_data) ? 'Multiple Sources' : evaluation.source_data.dataset_name}
+                      </span>
+                    </div>
+                    {!Array.isArray(evaluation.source_data) && (
+                      <>
+                        {evaluation.source_data.hf_repo && (
+                          <div className="flex justify-between">
+                            <span className="text-muted-foreground">HuggingFace:</span>
+                            <a href={`https://huggingface.co/${evaluation.source_data.hf_repo}`} target="_blank" rel="noopener noreferrer" className="text-primary hover:underline flex items-center gap-1">
+                              {evaluation.source_data.hf_repo.split('/')[1] || evaluation.source_data.hf_repo} <ExternalLink className="h-3 w-3" />
+                            </a>
+                          </div>
+                        )}
+                        {evaluation.source_data.hf_split && (
+                          <div className="flex justify-between">
+                            <span className="text-muted-foreground">Split:</span>
+                            <code className="bg-muted px-1 rounded text-xs">{evaluation.source_data.hf_split}</code>
+                          </div>
+                        )}
+                        <div className="flex justify-between">
+                          <span className="text-muted-foreground">Samples:</span>
+                          <span>{evaluation.source_data.samples_number?.toLocaleString()}</span>
+                        </div>
+                      </>
+                    )}
+                  </div>
+                </div>
+              </div>
+            </div>
+
+            <Separator />
+
+            {/* Evaluation Results */}
+            <div>
+              <div className="text-xs font-semibold text-muted-foreground uppercase tracking-wider mb-3">Evaluation Results</div>
+              
+              <div className="bg-background rounded-lg p-4 border">
+                <div className="flex justify-between items-end mb-2">
+                  <div>
+                    <div className="font-medium text-lg">Overall Score</div>
+                    <div className="text-xs text-muted-foreground">
+                      {result.metric_config.score_type} • {result.metric_config.min_score}-{result.metric_config.max_score} • {result.metric_config.lower_is_better ? 'Lower is better' : 'Higher is better'}
+                    </div>
+                  </div>
+                  <div className="text-2xl font-bold text-primary">{displayScore}</div>
+                </div>
+                <Progress value={normalized * 100} className="h-2 mb-4" />
+                
+                {result.score_details.details && Object.keys(result.score_details.details).length > 0 && (
+                  <div className="grid grid-cols-2 sm:grid-cols-3 md:grid-cols-4 gap-3 mt-4">
+                    {Object.entries(result.score_details.details).map(([key, value]) => {
+                      let valDisplay = typeof value === 'number' ? value.toFixed(2) : value;
+                      if (typeof value === 'number') {
+                          if (unit === 'accuracy' || !unit || unit === 'pass@1') {
+                              valDisplay = (value * 100).toFixed(1) + "%";
+                          } else {
+                              valDisplay = value.toFixed(2);
+                          }
+                      }
+                      
+                      return (
+                      <div key={key} className="bg-muted/30 p-3 rounded border">
+                        <div className="text-xs text-muted-foreground mb-1 truncate" title={key}>{key}</div>
+                        <div className="font-semibold text-lg">
+                          {valDisplay}
+                        </div>
+                      </div>
+                    )})}
+                  </div>
+                )}
+              </div>
+            </div>
+
+            {/* Factsheet Information */}
+            {result.factsheet && (
+              <div>
+                <div className="flex items-center gap-2 mb-3">
+                  <FileCode className="h-4 w-4 text-primary" />
+                  <div className="text-xs font-semibold text-muted-foreground uppercase tracking-wider">Benchmark Factsheet</div>
+                </div>
+                
+                <div className="grid grid-cols-1 gap-6 bg-background p-6 rounded-lg border">
+                  {/* General Info */}
+                  <div className="grid grid-cols-1 md:grid-cols-2 gap-x-8 gap-y-4">
+                    {result.factsheet.purpose && (
+                      <div className="col-span-full">
+                        <span className="font-semibold text-sm block mb-1">Purpose</span>
+                        <p className="text-sm text-muted-foreground">{result.factsheet.purpose}</p>
+                      </div>
+                    )}
+                    {result.factsheet.principles_tested && (
+                      <div className="col-span-full">
+                        <span className="font-semibold text-sm block mb-1">Principles Tested</span>
+                        <p className="text-sm text-muted-foreground">{result.factsheet.principles_tested}</p>
+                      </div>
+                    )}
+                  </div>
+
+                  <Separator />
+
+                  {/* Methodology */}
+                  <div>
+                    <h4 className="text-sm font-semibold mb-3 text-primary/80">Methodology</h4>
+                    <div className="grid grid-cols-1 md:grid-cols-3 gap-4 text-sm">
+                      {result.factsheet.judge && (
+                        <div>
+                          <span className="font-medium block text-xs text-muted-foreground uppercase mb-1">Judge</span>
+                          <span>{result.factsheet.judge}</span>
+                        </div>
+                      )}
+                      {result.factsheet.protocol && (
+                        <div>
+                          <span className="font-medium block text-xs text-muted-foreground uppercase mb-1">Protocol</span>
+                          <span>{result.factsheet.protocol}</span>
+                        </div>
+                      )}
+                      {result.factsheet.model_access && (
+                        <div>
+                          <span className="font-medium block text-xs text-muted-foreground uppercase mb-1">Model Access</span>
+                          <span>{result.factsheet.model_access}</span>
+                        </div>
+                      )}
+                      {result.factsheet.input_modality && (
+                        <div>
+                          <span className="font-medium block text-xs text-muted-foreground uppercase mb-1">Input Modality</span>
+                          <span>{result.factsheet.input_modality}</span>
+                        </div>
+                      )}
+                      {result.factsheet.output_modality && (
+                        <div>
+                          <span className="font-medium block text-xs text-muted-foreground uppercase mb-1">Output Modality</span>
+                          <span>{result.factsheet.output_modality}</span>
+                        </div>
+                      )}
+                      {result.factsheet.design && (
+                        <div>
+                          <span className="font-medium block text-xs text-muted-foreground uppercase mb-1">Design</span>
+                          <span>{result.factsheet.design}</span>
+                        </div>
+                      )}
+                    </div>
+                  </div>
+
+                  <Separator />
+
+                  {/* Data & Validation */}
+                  <div>
+                    <h4 className="text-sm font-semibold mb-3 text-primary/80">Data & Validation</h4>
+                    <div className="grid grid-cols-1 md:grid-cols-2 gap-4 text-sm">
+                      {result.factsheet.size && (
+                        <div>
+                          <span className="font-medium block text-xs text-muted-foreground uppercase mb-1">Size</span>
+                          <span>{result.factsheet.size}</span>
+                        </div>
+                      )}
+                      {result.factsheet.splits && (
+                        <div>
+                          <span className="font-medium block text-xs text-muted-foreground uppercase mb-1">Splits</span>
+                          <span>{result.factsheet.splits}</span>
+                        </div>
+                      )}
+                      {result.factsheet.has_heldout !== undefined && (
+                        <div>
+                          <span className="font-medium block text-xs text-muted-foreground uppercase mb-1">Held-out Set</span>
+                          <Badge variant={result.factsheet.has_heldout ? "default" : "secondary"}>
+                            {result.factsheet.has_heldout ? "Yes" : "No"}
+                          </Badge>
+                        </div>
+                      )}
+                      {result.factsheet.is_valid !== undefined && (
+                        <div>
+                          <span className="font-medium block text-xs text-muted-foreground uppercase mb-1">Valid</span>
+                          <Badge variant={result.factsheet.is_valid ? "outline" : "destructive"}>
+                            {result.factsheet.is_valid ? "Yes" : "No"}
+                          </Badge>
+                        </div>
+                      )}
+                    </div>
+                  </div>
+
+                  {/* Limitations */}
+                  {result.factsheet.known_limitations && (
+                    <>
+                      <Separator />
+                      <div className="bg-red-50 dark:bg-red-900/10 p-4 rounded border border-red-100 dark:border-red-900/20">
+                        <div className="flex items-center gap-2 text-red-700 dark:text-red-400 mb-2">
+                          <AlertTriangle className="h-4 w-4" />
+                          <span className="font-semibold text-sm">Known Limitations</span>
+                        </div>
+                        <p className="text-sm text-red-600/90 dark:text-red-400/90">{result.factsheet.known_limitations}</p>
+                      </div>
+                    </>
+                  )}
+                </div>
+              </div>
+            )}
+
+            {/* Generation Configuration */}
+            {result.generation_config && (
+              <div>
+                <div className="flex items-center gap-2 mb-3">
+                  <Settings className="h-4 w-4 text-primary" />
+                  <div className="text-xs font-semibold text-muted-foreground uppercase tracking-wider">Generation Configuration</div>
+                </div>
+                
+                <div className="bg-slate-950 text-slate-200 p-4 rounded-lg font-mono text-sm overflow-x-auto">
+                  {result.generation_config.additional_details && (
+                    <div className="mb-4 pb-4 border-b border-slate-800">
+                      <div className="text-slate-500 text-xs uppercase mb-1">Description</div>
+                      <div>{result.generation_config.additional_details}</div>
+                    </div>
+                  )}
+                  
+                  {result.generation_config.generation_args && (
+                    <div className="grid grid-cols-2 md:grid-cols-4 gap-4">
+                      {Object.entries(result.generation_config.generation_args).map(([key, value]) => (
+                        <div key={key}>
+                          <div className="text-slate-500 text-xs">{key}</div>
+                          <div className="text-emerald-400">{String(value)}</div>
+                        </div>
+                      ))}
+                    </div>
+                  )}
+                </div>
+              </div>
+            )}
+
+            {/* Sample Level Data */}
+            {evaluation.detailed_evaluation_results_per_samples && evaluation.detailed_evaluation_results_per_samples.length > 0 && randomSample && (
+              <div>
+                <Separator className="my-6" />
+                <div className="flex items-center justify-between mb-3">
+                  <div className="flex items-center gap-2">
+                    <FileCode className="h-4 w-4 text-primary" />
+                    <div className="text-xs font-semibold text-muted-foreground uppercase tracking-wider">Sample Level Data (Random Sample)</div>
+                  </div>
+                  <Badge variant="outline">{evaluation.detailed_evaluation_results_per_samples.length} Samples</Badge>
+                </div>
+
+                <div className="space-y-4">
+                  <div className="bg-muted/10 border rounded-lg p-4 text-sm">
+                    <div className="flex justify-between items-start mb-2">
+                      <Badge variant="secondary" className="font-mono text-xs">ID: {randomSample.sample_id}</Badge>
+                    </div>
+                    
+                    <div className="grid gap-4">
+                      <div>
+                        <div className="text-xs font-semibold text-muted-foreground uppercase mb-1">Input</div>
+                        <div className="bg-muted/30 p-3 rounded whitespace-pre-wrap font-mono text-xs">{randomSample.input}</div>
+                      </div>
+                      
+                      <div className="grid grid-cols-1 md:grid-cols-2 gap-4">
+                        <div>
+                          <div className="text-xs font-semibold text-muted-foreground uppercase mb-1">Model Response</div>
+                          <div className="bg-blue-50/50 dark:bg-blue-900/10 p-3 rounded whitespace-pre-wrap text-blue-900 dark:text-blue-100">
+                            {randomSample.response}
+                          </div>
+                        </div>
+                        <div>
+                          <div className="text-xs font-semibold text-muted-foreground uppercase mb-1">Ground Truth</div>
+                          <div className="bg-green-50/50 dark:bg-green-900/10 p-3 rounded whitespace-pre-wrap text-green-900 dark:text-green-100">
+                            {randomSample.ground_truth}
+                          </div>
+                        </div>
+                      </div>
+                    </div>
+                  </div>
+                  
+                  <div className="text-center pt-2">
+                    <SampleDataDialog 
+                      samples={evaluation.detailed_evaluation_results_per_samples}
+                      evaluationName={result.evaluation_name}
+                    />
+                  </div>
+                </div>
+              </div>
+            )}
+
+            {/* Footer Links */}
+            <div className="flex gap-3 pt-2">
+              {result.detailed_evaluation_results_url && (
+                <a 
+                  href={result.detailed_evaluation_results_url}
+                  target="_blank"
+                  rel="noopener noreferrer"
+                  className="flex items-center gap-2 text-sm text-primary hover:underline"
+                >
+                  <Database className="h-4 w-4" />
+                  View detailed per-sample results <ExternalLink className="h-3 w-3" />
+                </a>
+              )}
+            </div>
+          </CardContent>
+        </CollapsibleContent>
+      </Card>
+    </Collapsible>
+  )
+}
+
+function AllEvaluationsView({ evaluations }: { evaluations: BenchmarkEvaluation[] }) {
+  return (
+    <div className="space-y-6">
+      {evaluations.map((eval_, idx) => (
+        <div key={idx} className="space-y-6">
+          {eval_.evaluation_results.map((result, ridx) => (
+            <BenchmarkResultCard 
+              key={`${eval_.evaluation_id}-${ridx}`}
+              evaluation={eval_}
+              result={result}
+            />
+          ))}
+        </div>
+      ))}
+    </div>
+  )
+}
+
+function CategoryStatsView({ 
+  stats, 
+  summary
+}: { 
+  stats: { category: CategoryType; count: number; avg_score: number }[]
+  summary: ModelEvaluationSummary
+}) {
+  const getCategoryColor = (score: number) => {
+    if (score >= 0.8) return 'text-green-600'
+    if (score >= 0.6) return 'text-yellow-600'
+    return 'text-red-600'
+  }
+  
+  const getCategoryLabel = (category: CategoryType): string => {
+    return category.split('-').map(w => w.charAt(0).toUpperCase() + w.slice(1)).join(' ')
+  }
+  
+  return (
+    <div className="grid gap-6 md:grid-cols-2">
+      {stats.map((stat) => {
+        const evals = summary.evaluations_by_category[stat.category] || []
+        
+        return (
+          <Card key={stat.category} className="overflow-hidden">
+            <CardHeader className="bg-muted/30 pb-4">
+              <div className="flex items-center justify-between">
+                <CardTitle className="text-lg">{getCategoryLabel(stat.category)}</CardTitle>
+                <div className={`text-2xl font-bold ${getCategoryColor(stat.avg_score)}`}>
+                  {(stat.avg_score * 100).toFixed(1)}%
+                </div>
+              </div>
+              <CardDescription>{stat.count} evaluation{stat.count !== 1 ? 's' : ''}</CardDescription>
+            </CardHeader>
+            <CardContent className="p-0">
+              <div className="divide-y">
+                {evals.map((eval_: BenchmarkEvaluation, idx: number) => {
+                  // Filter results to only show those that match this category
+                  const relevantResults = eval_.evaluation_results.filter((result: any) => {
+                    const resultCategory = inferCategoryFromBenchmark(result.evaluation_name)
+                    return resultCategory === stat.category
+                  })
+                  
+                  if (relevantResults.length === 0) return null
+                  
+                  return relevantResults.map((result: any, ridx: number) => (
+                    <div key={`${idx}-${ridx}`} className="flex items-center justify-between p-4 hover:bg-muted/50 transition-colors">
+                      <div className="space-y-1">
+                        <div className="font-medium text-sm">{result.evaluation_name}</div>
+                        <div className="text-xs text-muted-foreground">
+                          {Array.isArray(eval_.source_data) 
+                            ? (eval_.source_metadata.source_name || 'Unknown')
+                            : eval_.source_data.dataset_name}
+                        </div>
+                      </div>
+                      <div className="font-mono font-semibold">
+                        {formatScore(
+                          result.score_details.score,
+                          result.metric_config.score_type,
+                          result.metric_config.max_score
+                        )}
+                      </div>
+                    </div>
+                  ))
+                })}
+              </div>
+            </CardContent>
+          </Card>
+        )
+      })}
+    </div>
+  )
+}

components/benchmark-evaluation-card.tsx

@@ -0,0 +1,202 @@
+"use client"
+
+import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"
+import { Badge } from "@/components/ui/badge"
+import { Button } from "@/components/ui/button"
+import { MoreHorizontal, Eye, ExternalLink, Award, TrendingUp, Calendar, Building } from "lucide-react"
+import { DropdownMenu, DropdownMenuContent, DropdownMenuItem, DropdownMenuTrigger } from "@/components/ui/dropdown-menu"
+import { useRouter } from "next/navigation"
+import { Tooltip, TooltipContent, TooltipProvider, TooltipTrigger } from "@/components/ui/tooltip"
+import type { CategoryType } from "@/lib/benchmark-schema"
+import { EVALUATION_CATEGORIES } from "@/lib/benchmark-schema"
+
+export type BenchmarkEvaluationCardData = {
+  id: string
+  model_name: string
+  model_id: string
+  developer: string
+  evaluations_count: number
+  benchmarks_count: number
+  categories: CategoryType[]
+  category_stats: Record<CategoryType, number>
+  latest_timestamp: string
+  
+  // Quick stats
+  top_scores: Array<{
+    benchmark: string
+    score: number
+    metric: string
+    unit?: string
+  }>
+  
+  // Links
+  source_urls: string[]
+  detail_urls: string[]
+
+  // Model Metadata
+  model_url?: string
+  release_date?: string
+  input_modalities?: string[]
+  output_modalities?: string[]
+  architecture?: string
+  params?: string
+  inference_engine?: string
+  inference_platform?: string
+}
+
+interface BenchmarkEvaluationCardProps {
+  data: BenchmarkEvaluationCardData
+  onDelete?: (id: string) => void
+}
+
+export function BenchmarkEvaluationCard({ data, onDelete }: BenchmarkEvaluationCardProps) {
+  const router = useRouter()
+  
+  const formatDate = (isoString: string) => {
+    try {
+      return new Date(isoString).toLocaleDateString('en-US', {
+        year: 'numeric',
+        month: 'short',
+        day: 'numeric'
+      })
+    } catch {
+      return isoString
+    }
+  }
+  
+  const getCategoryColor = (category: CategoryType, index: number): string => {
+    const colors = [
+      "bg-emerald-500", "bg-blue-500", "bg-indigo-500", "bg-violet-500", 
+      "bg-fuchsia-500", "bg-pink-500", "bg-rose-500", "bg-orange-500",
+      "bg-amber-500", "bg-yellow-500", "bg-lime-500", "bg-teal-500"
+    ]
+    // Use a consistent hash or just the index if the list is stable
+    return colors[index % colors.length]
+  }
+  
+  const getCategoryLabel = (category: CategoryType): string => {
+    return category.split('-').map(w => w.charAt(0).toUpperCase() + w.slice(1)).join(' ')
+  }
+
+  const renderCategoryDistribution = () => {
+    const totalBenchmarks = data.categories.reduce((acc, cat) => acc + (data.category_stats[cat] || 0), 0)
+    
+    return (
+      <div className="space-y-2">
+        <div className="flex justify-between items-end">
+          <span className="text-sm font-medium text-muted-foreground">Category Breakdown</span>
+          <span className="text-xs text-muted-foreground">({EVALUATION_CATEGORIES.length} categories, {totalBenchmarks} benchmarks)</span>
+        </div>
+        <div className="flex h-3 w-full gap-0.5 rounded-full overflow-hidden bg-secondary/30">
+          {EVALUATION_CATEGORIES.map((category, idx) => {
+            const count = data.category_stats[category] || 0
+            const isZero = count === 0
+            
+            return (
+              <TooltipProvider key={category}>
+                <Tooltip>
+                  <TooltipTrigger asChild>
+                    <div 
+                      className={`h-full ${getCategoryColor(category, idx)} transition-all cursor-help ${isZero ? 'w-1 flex-none opacity-30 hover:opacity-50' : 'hover:opacity-80'}`}
+                      style={!isZero ? { flexGrow: count } : undefined}
+                    />
+                  </TooltipTrigger>
+                  <TooltipContent>
+                    <p className="font-semibold">{getCategoryLabel(category)}</p>
+                    <p className="text-xs">{count} benchmarks</p>
+                  </TooltipContent>
+                </Tooltip>
+              </TooltipProvider>
+            )
+          })}
+        </div>
+      </div>
+    )
+  }
+  
+  const completeness = Math.round((data.categories.length / EVALUATION_CATEGORIES.length) * 100)
+  
+  return (
+    <Card className="hover:shadow-lg transition-shadow cursor-pointer group">
+      <CardHeader className="space-y-4 pb-2">
+        <div className="flex items-start justify-between">
+          <div className="flex-1" onClick={() => router.push(`/benchmark/${encodeURIComponent(data.id)}`)}>
+            <div className="flex items-center gap-2 mb-1">
+              <CardTitle className="text-xl font-bold group-hover:text-primary transition-colors">
+                {data.model_name}
+              </CardTitle>
+            </div>
+            <div className="text-sm text-muted-foreground mb-3">
+              {data.developer}
+            </div>
+            
+            {((data.input_modalities?.length || 0) > 1 || (data.output_modalities?.length || 0) > 1) && (
+              <Badge variant="secondary" className="bg-indigo-100 text-indigo-700 hover:bg-indigo-200 border-indigo-200 gap-1">
+                <span className="text-lg">🤖</span> Multimodal
+              </Badge>
+            )}
+          </div>
+          
+          <DropdownMenu>
+            <DropdownMenuTrigger asChild>
+              <Button variant="ghost" size="icon" className="opacity-0 group-hover:opacity-100 transition-opacity">
+                <MoreHorizontal className="h-4 w-4" />
+              </Button>
+            </DropdownMenuTrigger>
+            <DropdownMenuContent align="end">
+              <DropdownMenuItem onClick={() => router.push(`/benchmark/${encodeURIComponent(data.id)}`)}>
+                <Eye className="mr-2 h-4 w-4" />
+                View Details
+              </DropdownMenuItem>
+              {data.source_urls.length > 0 && (
+                <DropdownMenuItem onClick={() => window.open(data.source_urls[0], '_blank')}>
+                  <ExternalLink className="mr-2 h-4 w-4" />
+                  View Source
+                </DropdownMenuItem>
+              )}
+              {onDelete && (
+                <DropdownMenuItem 
+                  onClick={() => onDelete(data.id)}
+                  className="text-destructive"
+                >
+                  <Award className="mr-2 h-4 w-4" />
+                  Remove
+                </DropdownMenuItem>
+              )}
+            </DropdownMenuContent>
+          </DropdownMenu>
+        </div>
+      </CardHeader>
+
+      <CardContent className="space-y-6">
+        {/* Completeness & Date Row */}
+        <div className="grid grid-cols-2 gap-8">
+          <div className="space-y-1.5">
+            <div className="text-sm font-medium text-muted-foreground">Completeness</div>
+            <div className="h-7 w-full bg-secondary rounded-full overflow-hidden relative">
+              <div 
+                className="h-full bg-blue-600 absolute top-0 left-0 transition-all duration-500"
+                style={{ width: `${completeness}%` }}
+              />
+              <div className="absolute inset-0 flex items-center justify-center text-xs font-bold text-white drop-shadow-[0_1px_2px_rgba(0,0,0,0.5)] z-10">
+                {data.categories.length}/{EVALUATION_CATEGORIES.length} categories
+              </div>
+            </div>
+          </div>
+          
+          <div className="space-y-1.5">
+            <div className="text-sm font-medium text-muted-foreground">Submitted</div>
+            <div className="h-7 flex items-center text-sm font-medium">
+              {formatDate(data.latest_timestamp)}
+            </div>
+          </div>
+        </div>
+
+        {/* Broken Bars */}
+        <div className="space-y-4">
+          {renderCategoryDistribution()}
+        </div>
+      </CardContent>
+    </Card>
+  )
+}

components/navigation.tsx

@@ -1,7 +1,7 @@
 "use client"
 
 import { Button } from "@/components/ui/button"
-import { Moon, Sun, Home, ArrowUpDown, Info } from "lucide-react"
+import { Moon, Sun, Home, ArrowUpDown, Info, BarChart3 } from "lucide-react"
 import { useTheme } from "next-themes"
 import Link from "next/link"
 import { usePathname } from "next/navigation"
@@ -16,7 +16,7 @@ export function Navigation() {
       href: "/",
       label: "Home",
       icon: Home,
-      isActive: pathname === "/"
+      isActive: pathname === "/" || pathname?.startsWith("/benchmark")
     },
     {
       href: "/analytics",

lib/benchmark-schema.ts

@@ -0,0 +1,265 @@
+/**
+ * Benchmark-first evaluation schema types
+ * Based on the evalevalai.com schema structure
+ */
+
+export interface BenchmarkEvaluation {
+  schema_version: string
+  evaluation_id: string
+  retrieved_timestamp: string
+  
+  source_data: string[] | SourceData
+  source_metadata: SourceMetadata
+  model_info: ModelInfo
+  evaluation_results: EvaluationResult[]
+  detailed_evaluation_results_per_samples?: SampleResult[]
+}
+
+export interface SourceData {
+  dataset_name: string
+  hf_repo?: string
+  hf_split?: string
+  samples_number: number
+  dataset_url?: string
+  dataset_version?: string
+}
+
+export interface SourceMetadata {
+  source_name?: string
+  source_type: 'evaluation_run' | 'documentation' | 'paper' | 'leaderboard'
+  source_organization_name: string
+  source_organization_url?: string
+  evaluator_relationship: 'first_party' | 'third_party' | 'collaborative' | 'other'
+  source_url?: string
+  publication_date?: string
+}
+
+export interface ModelInfo {
+  name: string
+  id: string
+  developer?: string
+  inference_platform?: string
+  inference_engine?: string
+  model_version?: string
+  architecture?: string
+  parameter_count?: string
+  release_date?: string
+  model_url?: string
+  additional_details?: {
+    precision?: string
+    architecture?: string
+    params_billions?: number
+    [key: string]: any
+  }
+  modalities?: {
+    input: string[]
+    output: string[]
+  }
+}
+
+export interface EvaluationResult {
+  evaluation_name: string
+  evaluation_timestamp: string
+  metric_config: MetricConfig
+  score_details: ScoreDetails
+  detailed_evaluation_results_url?: string
+  generation_config?: GenerationConfig
+  factsheet?: {
+    purpose?: string
+    principles_tested?: string
+    functional_props?: string
+    input_modality?: string
+    output_modality?: string
+    input_source?: string
+    output_source?: string
+    size?: string
+    splits?: string
+    design?: string
+    judge?: string
+    protocol?: string
+    model_access?: string
+    has_heldout?: boolean
+    heldout_details?: string
+    alignment_validation?: string
+    is_valid?: boolean
+    baseline_models?: string
+    robustness_measures?: string
+    known_limitations?: string
+    benchmarks_list?: string
+  }
+}
+
+export interface MetricConfig {
+  evaluation_description: string
+  lower_is_better: boolean
+  score_type: 'continuous' | 'discrete' | 'binary'
+  min_score?: number
+  max_score?: number
+  unit?: string
+}
+
+export interface ScoreDetails {
+  score: number
+  details?: Record<string, any>
+  confidence_interval?: {
+    lower: number
+    upper: number
+    confidence_level: number
+  }
+  sample_size?: number
+  standard_error?: number
+}
+
+export interface GenerationConfig {
+  generation_args: {
+    temperature?: number
+    top_p?: number
+    top_k?: number
+    max_tokens?: number
+    reasoning?: boolean
+    [key: string]: any
+  }
+  additional_details?: string
+  prompt_template?: string
+}
+
+export interface SampleResult {
+  sample_id: string
+  input: string
+  ground_truth?: string
+  response: string
+  choices?: string[]
+  is_correct?: boolean
+  metadata?: Record<string, any>
+}
+
+/**
+ * Evaluation categories for classification
+ */
+export const EVALUATION_CATEGORIES = [
+  'Core Performance',
+  'Core Quality Dimensions',
+  'Robustness',
+  'Calibration',
+  'Adversarial',
+  'Memorization',
+  'Fairness',
+  'Safety',
+  'Leakage/Contamination',
+  'Privacy',
+  'Interpretability',
+  'Efficiency',
+  'Retrainability',
+  'Meta-Learning',
+] as const
+
+export type CategoryType = typeof EVALUATION_CATEGORIES[number]
+
+/**
+ * Helper to determine category from benchmark name
+ */
+export function inferCategoryFromBenchmark(benchmarkName: string): CategoryType {
+  const name = benchmarkName.toLowerCase()
+  
+  // Category mappings
+  if (name.includes('advglue') || name.includes('jailbreak') || name.includes('attack') || name.includes('adversarial') || name.includes('red-team')) {
+    return 'Adversarial'
+  }
+  if (name.includes('fairness') || name.includes('bias') || name.includes('stereo') || name.includes('bbq') || name.includes('celeb') || name.includes('winobias')) {
+    return 'Fairness'
+  }
+  if (name.includes('safety') || name.includes('harmful') || name.includes('toxic') || name.includes('truthful') || name.includes('unsafe')) {
+    return 'Safety'
+  }
+  if (name.includes('leakage') || name.includes('contamination')) {
+    return 'Leakage/Contamination'
+  }
+  if (name.includes('privacy') || name.includes('pii') || name.includes('gdpr') || name.includes('private')) {
+    return 'Privacy'
+  }
+  if (name.includes('robust')) {
+    return 'Robustness'
+  }
+  if (name.includes('calibration') || name.includes('confidence')) {
+    return 'Calibration'
+  }
+  if (name.includes('memoriz') || name.includes('copyright')) {
+    return 'Memorization'
+  }
+  if (name.includes('interpret') || name.includes('explain')) {
+    return 'Interpretability'
+  }
+  if (name.includes('efficien') || name.includes('latency') || name.includes('throughput') || name.includes('speed')) {
+    return 'Efficiency'
+  }
+  if (name.includes('retrain') || name.includes('forgetting')) {
+    return 'Retrainability'
+  }
+  if (name.includes('meta') || name.includes('few-shot') || name.includes('learning')) {
+    return 'Meta-Learning'
+  }
+  if (name.includes('mt-bench') || name.includes('quality') || name.includes('human') || name.includes('fact') || name.includes('hallucination')) {
+    return 'Core Quality Dimensions'
+  }
+  
+  // Default to Core Performance for standard benchmarks
+  if (name.includes('mmlu') || name.includes('arc') || name.includes('hellaswag') || name.includes('winogrande') || name.includes('gpqa') ||
+      name.includes('gsm') || name.includes('math') || name.includes('minerva') || name.includes('mgsm') ||
+      name.includes('humaneval') || name.includes('mbpp') || name.includes('code') || name.includes('apps') ||
+      name.includes('vision') || name.includes('vqa') || name.includes('image') || name.includes('coco') ||
+      name.includes('multimodal') || name.includes('mmmu') || name.includes('seed-bench') ||
+      name.includes('bbh') || name.includes('reasoning') || name.includes('musr') ||
+      name.includes('xsum') || name.includes('summariz') || name.includes('dialog') || name.includes('translation') || name.includes('ifeval') ||
+      name.includes('creative') || name.includes('social') || name.includes('agent')) {
+    return 'Core Performance'
+  }
+  
+  return 'Core Performance'
+}
+
+/**
+ * Aggregate evaluations by model
+ */
+export interface ModelEvaluationSummary {
+  model_info: ModelInfo
+  evaluations_by_category: Record<CategoryType, BenchmarkEvaluation[]>
+  total_evaluations: number
+  last_updated: string
+  categories_covered: CategoryType[]
+}
+
+/**
+ * Display-friendly format for the UI
+ */
+export interface EvaluationCardData {
+  id: string
+  model_name: string
+  model_id: string
+  developer: string
+  evaluations_count: number
+  benchmarks_count: number
+  categories: CategoryType[]
+  category_stats: Record<CategoryType, number>
+  latest_timestamp: string
+  
+  // Quick stats
+  top_scores: Array<{
+    benchmark: string
+    score: number
+    metric: string
+  }>
+  
+  // Links
+  source_urls: string[]
+  detail_urls: string[]
+
+  // Model Metadata (from auxiliary sources or model_metadata.json)
+  model_url?: string
+  release_date?: string
+  input_modalities?: string[]
+  output_modalities?: string[]
+  architecture?: string
+  params?: string
+  inference_engine?: string
+  inference_platform?: string
+}

lib/eval-processing.ts

@@ -0,0 +1,371 @@
+/**
+ * Processing utilities for benchmark-first evaluation data
+ */
+
+import type {
+  BenchmarkEvaluation,
+  EvaluationCardData,
+  CategoryType,
+} from './benchmark-schema'
+import type { ModelEvaluationSummary } from './benchmark-schema'
+import { inferCategoryFromBenchmark, EVALUATION_CATEGORIES } from './benchmark-schema'
+
+export type { ModelEvaluationSummary }
+
+/**
+ * Group multiple evaluations by model
+ */
+export function groupEvaluationsByModel(
+  evaluations: BenchmarkEvaluation[]
+): Record<string, BenchmarkEvaluation[]> {
+  const grouped: Record<string, BenchmarkEvaluation[]> = {}
+  
+  for (const eval_ of evaluations) {
+    const modelId = eval_.model_info.id
+    if (!grouped[modelId]) {
+      grouped[modelId] = []
+    }
+    grouped[modelId].push(eval_)
+  }
+  
+  return grouped
+}
+
+/**
+ * Create a model evaluation summary from grouped evaluations
+ */
+export function createModelSummary(
+  evaluations: BenchmarkEvaluation[]
+): ModelEvaluationSummary {
+  if (evaluations.length === 0) {
+    throw new Error('No evaluations provided')
+  }
+  
+  const modelInfo = evaluations[0].model_info
+  const evaluationsByCategory: Record<string, BenchmarkEvaluation[]> = {}
+  const categoriesSet = new Set<CategoryType>()
+  
+  // Group by category - track which categories each evaluation belongs to
+  for (const eval_ of evaluations) {
+    const evalCategories = new Set<CategoryType>()
+    
+    for (const result of eval_.evaluation_results) {
+      // Try to get category from factsheet first
+      let category: CategoryType | undefined;
+      
+      if (result.factsheet?.functional_props) {
+        // The factsheet might contain multiple categories separated by semicolon
+        // We'll pick the first one that matches our known categories
+        const props = result.factsheet.functional_props.split(';').map(p => p.trim());
+        for (const prop of props) {
+          if (EVALUATION_CATEGORIES.includes(prop as CategoryType)) {
+            category = prop as CategoryType;
+            break;
+          }
+        }
+      }
+
+      // Infer category from evaluation name if not found in factsheet
+      if (!category) {
+        category = inferCategoryFromBenchmark(result.evaluation_name)
+      }
+      
+      // Fallback to dataset name if source_data is an object
+      if (!category && !Array.isArray(eval_.source_data)) {
+        category = inferCategoryFromBenchmark(eval_.source_data.dataset_name)
+      }
+      
+      if (category) {
+        evalCategories.add(category)
+        categoriesSet.add(category)
+      }
+    }
+    
+    // Add evaluation to each unique category it belongs to (once per category)
+    for (const category of evalCategories) {
+      if (!evaluationsByCategory[category]) {
+        evaluationsByCategory[category] = []
+      }
+      evaluationsByCategory[category].push(eval_)
+    }
+  }
+  
+  // Find latest timestamp
+  const timestamps = evaluations.map(e => parseFloat(e.retrieved_timestamp))
+  const latestTimestamp = new Date(Math.max(...timestamps) * 1000).toISOString()
+  
+  // Calculate total benchmark results
+  const totalResults = evaluations.reduce((sum, eval_) => sum + eval_.evaluation_results.length, 0)
+
+  return {
+    model_info: modelInfo,
+    evaluations_by_category: evaluationsByCategory as Record<CategoryType, BenchmarkEvaluation[]>,
+    total_evaluations: totalResults,
+    last_updated: latestTimestamp,
+    categories_covered: Array.from(categoriesSet),
+  }
+}
+
+/**
+ * Convert model summary to card display format
+ */
+export function createEvaluationCard(
+  summary: ModelEvaluationSummary
+): EvaluationCardData {
+  // Get all unique benchmarks
+  const benchmarksSet = new Set<string>()
+  const allScores: Array<{
+    benchmark: string
+    score: number
+    metric: string
+    unit?: string
+  }> = []
+  const sourceUrls = new Set<string>()
+  const detailUrls = new Set<string>()
+  
+  // Collect all evaluations
+  for (const evals of Object.values(summary.evaluations_by_category)) {
+    for (const eval_ of evals) {
+      // Handle source_data as either string[] or SourceData object
+      if (Array.isArray(eval_.source_data)) {
+        // source_data is string[] (URLs), extract benchmark names from evaluation_results
+        for (const result of eval_.evaluation_results) {
+          benchmarksSet.add(result.evaluation_name)
+        }
+      } else {
+        benchmarksSet.add(eval_.source_data.dataset_name)
+      }
+      
+      if (eval_.source_metadata.source_url) {
+        sourceUrls.add(eval_.source_metadata.source_url)
+      }
+      
+      // Add source_data URLs if it's a string array
+      if (Array.isArray(eval_.source_data)) {
+        eval_.source_data.forEach(url => sourceUrls.add(url))
+      }
+      
+      for (const result of eval_.evaluation_results) {
+        if (result.detailed_evaluation_results_url) {
+          detailUrls.add(result.detailed_evaluation_results_url)
+        }
+        
+        allScores.push({
+          benchmark: result.evaluation_name,
+          score: result.score_details.score,
+          metric: result.metric_config.evaluation_description || result.evaluation_name,
+          unit: result.metric_config.unit
+        })
+      }
+    }
+  }
+  
+  // Deduplicate by benchmark name, keeping highest score for each
+  const scoresByBenchmark = new Map<string, { benchmark: string; score: number; metric: string; unit?: string }>()
+  for (const scoreData of allScores) {
+    const existing = scoresByBenchmark.get(scoreData.benchmark)
+    if (!existing || scoreData.score > existing.score) {
+      scoresByBenchmark.set(scoreData.benchmark, scoreData)
+    }
+  }
+  
+  // Calculate category stats (count of unique benchmarks per category)
+  const categoryStats: Record<CategoryType, number> = {} as any
+  
+  for (const category of summary.categories_covered) {
+    const evals = summary.evaluations_by_category[category] || []
+    const categoryBenchmarks = new Set<string>()
+    
+    for (const eval_ of evals) {
+      if (Array.isArray(eval_.source_data)) {
+        for (const result of eval_.evaluation_results) {
+          // Only count if this result actually belongs to this category
+          const resultCategory = inferCategoryFromBenchmark(result.evaluation_name)
+          if (resultCategory === category) {
+            categoryBenchmarks.add(result.evaluation_name)
+          }
+        }
+      } else {
+        // For single-benchmark files, check if the file's main benchmark belongs to category
+        // But wait, inferCategoryFromBenchmark might have been used to categorize the whole file
+        // Let's just count the benchmarks in this file that match the category
+        for (const result of eval_.evaluation_results) {
+           const resultCategory = inferCategoryFromBenchmark(result.evaluation_name)
+           if (resultCategory === category) {
+             categoryBenchmarks.add(result.evaluation_name)
+           }
+        }
+      }
+    }
+    categoryStats[category] = categoryBenchmarks.size
+  }
+
+  // Get top 5 unique benchmarks by score
+  const topScores = Array.from(scoresByBenchmark.values())
+    .sort((a, b) => b.score - a.score)
+    .slice(0, 5)
+
+  return {
+    id: summary.model_info.id,
+    model_name: summary.model_info.name,
+    model_id: summary.model_info.id,
+    developer: summary.model_info.developer,
+    evaluations_count: summary.total_evaluations,
+    benchmarks_count: benchmarksSet.size,
+    categories: summary.categories_covered,
+    category_stats: categoryStats,
+    latest_timestamp: summary.last_updated,
+    top_scores: topScores,
+    source_urls: Array.from(sourceUrls),
+    detail_urls: Array.from(detailUrls),
+    architecture: summary.model_info.architecture,
+    params: summary.model_info.parameter_count,
+    inference_engine: summary.model_info.inference_engine,
+    inference_platform: summary.model_info.inference_platform,
+    input_modalities: summary.model_info.modalities?.input,
+    output_modalities: summary.model_info.modalities?.output,
+    release_date: summary.model_info.release_date,
+    model_url: summary.model_info.model_url,
+  }
+}
+
+/**
+ * Get category stats for a model
+ */
+export function getCategoryStats(
+  summary: ModelEvaluationSummary
+): {
+  categories: { category: CategoryType; count: number; avg_score: number }[]
+} {
+  const categories: { category: CategoryType; count: number; avg_score: number }[] = []
+  
+  for (const category of summary.categories_covered) {
+    const evals = summary.evaluations_by_category[category] || []
+    const allScores: number[] = []
+    
+    for (const eval_ of evals) {
+      for (const result of eval_.evaluation_results) {
+        allScores.push(result.score_details.score)
+      }
+    }
+    
+    const avgScore = allScores.length > 0
+      ? allScores.reduce((a, b) => a + b, 0) / allScores.length
+      : 0
+    
+    const stat = {
+      category,
+      count: evals.length,
+      avg_score: avgScore,
+    }
+    
+    categories.push(stat)
+  }
+  
+  // Sort categories by name or some other metric if needed
+  categories.sort((a, b) => a.category.localeCompare(b.category))
+  
+  return { categories }
+}
+
+/**
+ * Load and process evaluations from file paths
+ */
+export async function loadEvaluations(
+  filePaths: string[]
+): Promise<BenchmarkEvaluation[]> {
+  const evaluations: BenchmarkEvaluation[] = []
+  
+  for (const path of filePaths) {
+    try {
+      const response = await fetch(path)
+      if (!response.ok) continue
+      
+      const data = await response.json()
+      
+      // Validate it matches our schema
+      if (data.schema_version && data.evaluation_id && data.model_info) {
+        evaluations.push(data as BenchmarkEvaluation)
+      }
+    } catch (error) {
+      console.warn(`Failed to load evaluation from ${path}:`, error)
+    }
+  }
+  
+  return evaluations
+}
+
+/**
+ * Process all evaluations into card data
+ */
+export async function processEvaluationsToCards(
+  filePaths: string[]
+): Promise<EvaluationCardData[]> {
+  const evaluations = await loadEvaluations(filePaths)
+  const grouped = groupEvaluationsByModel(evaluations)
+  
+  const cards: EvaluationCardData[] = []
+  
+  for (const modelId in grouped) {
+    const modelEvals = grouped[modelId]
+    const summary = createModelSummary(modelEvals)
+    const card = createEvaluationCard(summary)
+    cards.push(card)
+  }
+  
+  return cards
+}
+
+/**
+ * Format score with proper precision
+ */
+export function formatScore(
+  score: number,
+  scoreType: 'continuous' | 'discrete' | 'binary',
+  maxScore?: number
+): string {
+  if (scoreType === 'binary') {
+    return score > 0.5 ? 'Pass' : 'Fail'
+  }
+  
+  if (maxScore && maxScore === 1.0) {
+    // It's a percentage/ratio
+    return `${(score * 100).toFixed(1)}%`
+  }
+  
+  if (maxScore && maxScore === 100) {
+    return `${score.toFixed(1)}`
+  }
+  
+  // Default formatting
+  return score.toFixed(3)
+}
+
+/**
+ * Get benchmark display name
+ */
+export function getBenchmarkDisplayName(name: string | undefined | null): string {
+  if (!name) return 'Unknown Benchmark'
+  
+  // Map common benchmarks to friendly names
+  const mapping: Record<string, string> = {
+    'MMLU': 'Massive Multitask Language Understanding',
+    'MMLU-Pro': 'MMLU Professional',
+    'GSM8K': 'Grade School Math 8K',
+    'HumanEval': 'Human Eval (Code)',
+    'MBPP': 'Mostly Basic Python Problems',
+    'HellaSwag': 'HellaSwag (Commonsense)',
+    'ARC': 'AI2 Reasoning Challenge',
+    'TruthfulQA': 'TruthfulQA',
+    'BBH': 'Big-Bench Hard',
+    'MATH': 'MATH Dataset',
+  }
+  
+  for (const [key, value] of Object.entries(mapping)) {
+    if (name.toUpperCase().includes(key.toUpperCase())) {
+      return value
+    }
+  }
+  
+  return name
+}

public/benchmarks/alibaba-qwen-2-72b.json

@@ -0,0 +1,2103 @@
+{
+  "schema_version": "0.1.0",
+  "evaluation_id": "alibaba-qwen-2-72b-1765899234259",
+  "retrieved_timestamp": "2025-12-16T15:33:54.259Z",
+  "source_data": {
+    "dataset_name": "Demo Benchmark Suite",
+    "samples_number": 1000
+  },
+  "source_metadata": {
+    "source_name": "Demo Evaluation Suite",
+    "source_type": "evaluation_run",
+    "source_organization_name": "General Eval Card Demo",
+    "evaluator_relationship": "third_party"
+  },
+  "model_info": {
+    "name": "Qwen 2 72B",
+    "id": "alibaba/qwen-2-72b",
+    "developer": "Alibaba Cloud",
+    "inference_platform": "demo",
+    "inference_engine": "vLLM",
+    "model_version": "2.0",
+    "additional_details": {
+      "params": "72B",
+      "architecture": "Transformer",
+      "release_date": "2024-06-07"
+    },
+    "modalities": {
+      "input": [
+        "text"
+      ],
+      "output": [
+        "text"
+      ]
+    }
+  },
+  "evaluation_results": [
+    {
+      "evaluation_name": "Cityscapes",
+      "metric_config": {
+        "evaluation_description": "Cityscapes Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6719698671709685,
+        "details": {
+          "subtask_a": 0.6719698671709685,
+          "subtask_b": 0.6719698671709685
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development; Deployment",
+        "principles_tested": "Semantic segmentation, Instance segmentation, Scene understanding, Autonomous driving perception",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Vision (Image); Video",
+        "output_modality": "Structured Data",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Expert annotations",
+        "size": "Medium (1K - 100K samples)",
+        "splits": "Train (2975), Val (500), Test (1525)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives street scene image (2048×1024) 2. Model predicts per-pixel semantic class (19 or 30 classes) 3. IoU and accuracy metrics computed",
+        "model_access": "Outputs",
+        "has_heldout": true,
+        "alignment_validation": "Expert annotators with domain knowledge, multi-pass quality control, consistency verification across video sequences",
+        "baseline_models": "FCN-8s: 65.3 mIoU DeepLab v3+: 82.1 mIoU HRNetV2: 83.0 mIoU SegFormer: 84.0 mIoU",
+        "robustness_measures": "Multiple runs per sample; Ablation studies; Significance testing",
+        "known_limitations": "Limited to European cities; Weather bias (mostly good conditions); Class imbalance for rare objects; Fine annotation boundaries challenging",
+        "benchmarks_list": "ADE20K, KITTI, Mapillary Vistas, BDD100K, nuScenes"
+      }
+    },
+    {
+      "evaluation_name": "DROP (Discrete Reasoning Over Paragraphs)",
+      "metric_config": {
+        "evaluation_description": "DROP (Discrete Reasoning Over Paragraphs) Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6292049886124126,
+        "details": {
+          "subtask_a": 0.6292049886124126,
+          "subtask_b": 0.6292049886124126
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Reading comprehension, Numerical reasoning, Discrete operations, Multi-hop reasoning",
+        "functional_props": "Core Performance",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Crowdsourced annotations",
+        "size": "Large (100K - 1M samples)",
+        "splits": "Train (77,409), Dev (9,536), Test (9,622)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives paragraph and question 2. Model generates answer (number, span, or date) 3. F1 and Exact Match metrics computed",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Crowdsourced questions with verification, requires discrete reasoning operations confirmed through analysis",
+        "baseline_models": "BERT: 47.0 F1 RoBERTa: 80.9 F1 GPT-3: 29.0 F1 GPT-4: 80.9 F1 Human performance: 96.4 F1",
+        "robustness_measures": "Multiple answer types; Question type analysis; Human baseline comparison",
+        "known_limitations": "Requires careful answer extraction; Some questions ambiguous; Numerical reasoning can be brittle; Limited diversity in reasoning types",
+        "benchmarks_list": "SQuAD, NewsQA, NaturalQuestions, NumGLUE, TAT-QA"
+      }
+    },
+    {
+      "evaluation_name": "ScanNet",
+      "metric_config": {
+        "evaluation_description": "ScanNet Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5068754540473442,
+        "details": {
+          "subtask_a": 0.5068754540473442,
+          "subtask_b": 0.5068754540473442
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "3D scene understanding, Semantic segmentation, Instance segmentation, 3D reconstruction",
+        "functional_props": "Core Performance",
+        "input_modality": "Vision (Image); Video; Structured Data",
+        "output_modality": "Structured Data",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Human annotations; Programmatically generated",
+        "size": "Medium (1K - 100K samples)",
+        "splits": "Train (1201), Val (312), Test (100)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives 3D scene (RGB-D scans) 2. Model predicts semantic labels or instance masks 3. mIoU for semantic segmentation, mAP for instance segmentation",
+        "model_access": "Outputs",
+        "has_heldout": true,
+        "alignment_validation": "Manual verification of 3D reconstructions, multi-annotator consistency for semantic labels",
+        "baseline_models": "PointNet++: 53.5 mIoU SparseConvNet: 72.5 mIoU MinkowskiNet: 73.6 mIoU",
+        "robustness_measures": "Multiple runs per sample; Ablation studies",
+        "known_limitations": "Limited to indoor scenes; Reconstruction artifacts; Scanning noise and occlusions; Limited scene diversity (mostly offices and apartments)",
+        "benchmarks_list": "Matterport3D, S3DIS, 2D-3D-S, ARKitScenes"
+      }
+    },
+    {
+      "evaluation_name": "VQA (Visual Question Answering)",
+      "metric_config": {
+        "evaluation_description": "VQA (Visual Question Answering) Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5557888964910147,
+        "details": {
+          "subtask_a": 0.5557888964910147,
+          "subtask_b": 0.5557888964910147
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Visual question answering, Visual reasoning, Multimodal understanding, Common sense reasoning",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Text + Vision",
+        "output_modality": "Text",
+        "input_source": "MS COCO",
+        "output_source": "Crowdsourced annotations",
+        "size": "Huge (> 10M samples)",
+        "splits": "Train, Val, Test",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based); Human: Representative sample",
+        "protocol": "1. Model receives image and question 2. Model generates answer 3. Accuracy computed with consensus matching (multiple human answers)",
+        "model_access": "Outputs",
+        "has_heldout": true,
+        "alignment_validation": "Multiple human answers per question (10 answers), consensus-based evaluation",
+        "baseline_models": "Bottom-Up Top-Down: 70.3% VILBERT: 72.4% OSCAR: 73.8% BLIP: 78.3%",
+        "robustness_measures": "Multiple human references; Prompt variations tested; Inter-rater reliability",
+        "known_limitations": "Language bias (can answer many questions without image); Dataset bias toward common objects; Answer distribution imbalance; Ambiguous questions",
+        "benchmarks_list": "GQA, VQA v2, OK-VQA, TextVQA, VizWiz"
+      }
+    },
+    {
+      "evaluation_name": "Waymo Open Dataset",
+      "metric_config": {
+        "evaluation_description": "Waymo Open Dataset Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5265938860999003,
+        "details": {
+          "subtask_a": 0.5265938860999003,
+          "subtask_b": 0.5265938860999003
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development; Deployment",
+        "principles_tested": "3D object detection, 2D object detection, Tracking, Domain adaptation, Sensor fusion",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Vision (Image); Video; Structured Data",
+        "output_modality": "Structured Data",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Human annotations; Programmatically generated",
+        "size": "Huge (> 10M samples)",
+        "splits": "Train (798), Val (202), Test (150)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives multimodal sensor data 2. Model predicts 3D bounding boxes with tracking IDs 3. AP/APH metrics at different IoU thresholds and difficulty levels",
+        "model_access": "Outputs",
+        "has_heldout": true,
+        "alignment_validation": "Multi-stage annotation pipeline with quality checks, temporal consistency validation",
+        "baseline_models": "PointPillars: 63.8 L2 APH (Vehicle) CenterPoint: 73.9 L2 APH PV-RCNN++: 77.8 L2 APH",
+        "robustness_measures": "Geographic diversity; Time of day variations; Weather conditions; Multiple difficulty levels",
+        "known_limitations": "Geographic concentration in specific cities; Sensor-specific challenges; Annotation latency for distant objects; Class imbalance",
+        "benchmarks_list": "nuScenes, KITTI, Argoverse 2, Once, Lyft Level 5"
+      }
+    },
+    {
+      "evaluation_name": "HellaSwag",
+      "metric_config": {
+        "evaluation_description": "HellaSwag Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5432118015528332,
+        "details": {
+          "subtask_a": 0.5432118015528332,
+          "subtask_b": 0.5432118015528332
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Commonsense reasoning, Physical understanding, Situation modeling, Plausibility judgment",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Crowdsourced annotations",
+        "size": "Medium (1K - 100K samples)",
+        "splits": "Train (39,905), Val (10,042), Test (10,003)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives scenario context 2. Model selects most plausible continuation from 4 options 3. Accuracy computed",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Adversarial filtering using BERT to ensure difficulty, human validation of plausibility",
+        "baseline_models": "BERT-Large: 47.3% GPT-2: 50.9% GPT-3: 78.9% GPT-4: 95.3% Human performance: 95.6%",
+        "robustness_measures": "Adversarial filtering; Multiple runs per sample; Human baseline comparison",
+        "known_limitations": "Dataset may be easier than originally intended for modern LLMs; Multiple choice format; Adversarial examples may have artifacts; Potential data contamination",
+        "benchmarks_list": "PIQA, WinoGrande, CommonsenseQA, ARC"
+      }
+    },
+    {
+      "evaluation_name": "CIFAR-10 and CIFAR-100",
+      "metric_config": {
+        "evaluation_description": "CIFAR-10 and CIFAR-100 Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6353855781310864,
+        "details": {
+          "subtask_a": 0.6353855781310864,
+          "subtask_b": 0.6353855781310864
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development; Selection",
+        "principles_tested": "Image classification, Transfer learning, Generalization",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Vision (Image)",
+        "output_modality": "Structured Data",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Human annotations",
+        "size": "Medium (1K - 100K samples)",
+        "splits": "Train (50K), Test (10K)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives 32x32 RGB image 2. Model predicts class label (10 or 100 classes) 3. Classification accuracy computed",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Systematic sampling from larger dataset (80 million tiny images), human verification of labels",
+        "baseline_models": "CIFAR-10: ResNet-56: 93.03% Wide ResNet-28-10: 96.11% PyramidNet: 96.54%  CIFAR-100: ResNet-56: 71.35% Wide ResNet-28-10: 81.15% PyramidNet: 83.78%",
+        "robustness_measures": "Multiple runs per sample; Seed variation tested; Ablation studies",
+        "known_limitations": "Low resolution (32x32) limits fine-grained recognition; Some label noise in CIFAR-100; Dataset size enables memorization in large models; Limited diversity in poses and contexts",
+        "benchmarks_list": "CIFAR-10-C, CIFAR-10.1, CIFAR-100-C, STL-10, Tiny ImageNet"
+      }
+    },
+    {
+      "evaluation_name": "Kinetics",
+      "metric_config": {
+        "evaluation_description": "Kinetics Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5534482686246083,
+        "details": {
+          "subtask_a": 0.5534482686246083,
+          "subtask_b": 0.5534482686246083
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Action recognition, Video understanding, Temporal reasoning, Human activity recognition",
+        "functional_props": "Core Performance",
+        "input_modality": "Video",
+        "output_modality": "Structured Data",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Human annotations",
+        "size": "Large (100K - 1M samples)",
+        "splits": "Train, Val, Test",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives 10-second video clip 2. Model predicts action class (400/600/700 classes) 3. Top-1 and Top-5 accuracy computed",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Human verification of video labels, removal of ambiguous clips, consistency checks for similar actions",
+        "baseline_models": "I3D: 71.1% top-1 (K400) SlowFast: 79.8% top-1 (K400) X3D: 80.4% top-1 (K400) VideoMAE: 81.5% top-1 (K400)",
+        "robustness_measures": "Multiple runs per sample; Temporal ordering tested",
+        "known_limitations": "YouTube videos may become unavailable over time; Some action classes overlap or are ambiguous; Camera viewpoint bias; Dataset drift as internet content changes",
+        "benchmarks_list": "UCF-101, HMDB-51, ActivityNet, Something-Something, Moments in Time"
+      }
+    },
+    {
+      "evaluation_name": "WinoGrande",
+      "metric_config": {
+        "evaluation_description": "WinoGrande Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6906881359602444,
+        "details": {
+          "subtask_a": 0.6906881359602444,
+          "subtask_b": 0.6906881359602444
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Commonsense reasoning, Coreference resolution, World knowledge, Causal reasoning",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Crowdsourced annotations",
+        "size": "Medium (1K - 100K samples)",
+        "splits": "Train (40,398), Dev (1,267), Test (1,767)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives sentence with pronoun ambiguity 2. Model selects correct referent from 2 options 3. Accuracy computed",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Adversarial filtering using language models, crowdsourced generation with validation",
+        "baseline_models": "BERT-Large: 59.4% RoBERTa-Large: 79.1% GPT-3: 70.2% GPT-4: 87.5% Human performance: 94.0%",
+        "robustness_measures": "Adversarial filtering; Large-scale dataset; Multiple difficulty levels",
+        "known_limitations": "Binary choice may be limiting; Adversarial filtering may introduce artifacts; Saturation with modern models; Limited reasoning depth",
+        "benchmarks_list": "Winograd Schema Challenge, COPA, CommonsenseQA, PIQA"
+      }
+    },
+    {
+      "evaluation_name": "Places365",
+      "metric_config": {
+        "evaluation_description": "Places365 Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5600307461499519,
+        "details": {
+          "subtask_a": 0.5600307461499519,
+          "subtask_b": 0.5600307461499519
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Scene recognition, Scene classification, Environmental understanding, Context recognition",
+        "functional_props": "Core Performance",
+        "input_modality": "Vision (Image)",
+        "output_modality": "Structured Data",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Human annotations",
+        "size": "Very Huge (> 100M samples)",
+        "splits": "Train, Val, Test",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives scene image 2. Model predicts scene category (365 classes) 3. Top-1 and Top-5 accuracy computed",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Human verification, consistency checks for scene categories, hierarchical taxonomy validation",
+        "baseline_models": "ResNet-152: 55.24% top-1 DenseNet-161: 56.12% top-1 ResNeXt-101: 56.05% top-1",
+        "robustness_measures": "Inter-rater reliability; Multiple runs per sample",
+        "known_limitations": "Some scene categories overlap semantically; Cultural bias in scene definitions; Indoor scenes better represented than outdoor; Ambiguous boundary cases",
+        "benchmarks_list": "SUN397, MIT Indoor 67, Scene-15, ADE20K"
+      }
+    },
+    {
+      "evaluation_name": "UCF-101",
+      "metric_config": {
+        "evaluation_description": "UCF-101 Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6440431833857043,
+        "details": {
+          "subtask_a": 0.6440431833857043,
+          "subtask_b": 0.6440431833857043
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Action recognition, Video classification, Temporal understanding",
+        "functional_props": "Core Performance",
+        "input_modality": "Video",
+        "output_modality": "Structured Data",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Human annotations",
+        "size": "Medium (1K - 100K samples)",
+        "splits": "Train/Test (3 splits provided)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives video clip 2. Model predicts action class (101 classes) 3. Average accuracy across 3 splits reported",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Manual verification of action labels, removal of ambiguous videos",
+        "baseline_models": "Two-Stream CNN: 88.0% I3D: 95.6% SlowFast: 96.8% VideoMAE: 97.2%",
+        "robustness_measures": "Multiple evaluation splits; Seed variation tested",
+        "known_limitations": "YouTube videos may become unavailable; Camera motion and quality vary; Some action classes are very similar; Dataset saturation with modern methods",
+        "benchmarks_list": "HMDB-51, Kinetics, ActivityNet, Something-Something-V2"
+      }
+    },
+    {
+      "evaluation_name": "MPII Human Pose",
+      "metric_config": {
+        "evaluation_description": "MPII Human Pose Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5735067049623849,
+        "details": {
+          "subtask_a": 0.5735067049623849,
+          "subtask_b": 0.5735067049623849
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Human pose estimation, Keypoint detection, Articulated pose estimation, Activity recognition",
+        "functional_props": "Core Performance",
+        "input_modality": "Vision (Image)",
+        "output_modality": "Structured Data",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Human annotations",
+        "size": "Medium (1K - 100K samples)",
+        "splits": "Train (~29K people), Test (~12K people)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives image with person 2. Model predicts 2D joint locations (16 joints) 3. PCKh (Percentage of Correct Keypoints) at various thresholds",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Manual annotation with consistency checks, multi-annotator agreement for difficult poses",
+        "baseline_models": "Hourglass Network: 90.9 PCKh@0.5 HRNet: 92.3 PCKh@0.5 SimpleBaseline: 91.5 PCKh@0.5",
+        "robustness_measures": "Multiple difficulty levels; Inter-rater reliability; Occlusion analysis",
+        "known_limitations": "2D annotations only (no 3D); Occlusion and truncation challenges; Some joint definitions ambiguous; Dataset bias toward certain activities",
+        "benchmarks_list": "COCO Keypoints, Human3.6M, PoseTrack, CrowdPose"
+      }
+    },
+    {
+      "evaluation_name": "CLEVR",
+      "metric_config": {
+        "evaluation_description": "CLEVR Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6308990063124624,
+        "details": {
+          "subtask_a": 0.6308990063124624,
+          "subtask_b": 0.6308990063124624
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Compositional reasoning, Spatial reasoning, Counting, Logical reasoning, Visual reasoning",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Text + Vision",
+        "output_modality": "Text",
+        "input_source": "Synthetic/Generated",
+        "output_source": "Programmatically generated",
+        "size": "Huge (> 10M samples)",
+        "splits": "Train (70K), Val (15K), Test (15K)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives synthetic image and question 2. Model predicts answer from predefined set 3. Accuracy computed, can analyze by question type",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Programmatically generated with known ground truth, exhaustive question type coverage",
+        "baseline_models": "CNN+LSTM: 52.3% Film: 97.7% NS-VQA: 99.8% MAC: 98.9%",
+        "robustness_measures": "Ablation studies; Question type analysis; Compositional generalization tested",
+        "known_limitations": "Synthetic domain (limited real-world applicability); Simple shapes and colors; No natural language variation; Programmatic biases",
+        "benchmarks_list": "CLEVR-CoGenT, GQA, NLVR2, CLOSURE"
+      }
+    },
+    {
+      "evaluation_name": "ActivityNet",
+      "metric_config": {
+        "evaluation_description": "ActivityNet Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.688458491175051,
+        "details": {
+          "subtask_a": 0.688458491175051,
+          "subtask_b": 0.688458491175051
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Temporal action detection, Action recognition, Dense video captioning, Video understanding",
+        "functional_props": "Core Performance",
+        "input_modality": "Video",
+        "output_modality": "Structured Data; Text",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Human annotations",
+        "size": "Medium (1K - 100K samples)",
+        "splits": "Train (50%), Val (25%), Test (25%)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives untrimmed video 2. Model predicts action segments with class labels 3. mAP at different IoU thresholds (0.5, 0.75, 0.95)",
+        "model_access": "Outputs",
+        "has_heldout": true,
+        "alignment_validation": "Multi-annotator temporal boundary agreement, consistency checks for activity definitions",
+        "baseline_models": "SSN: 41.3 mAP@0.5 BMN: 50.1 mAP@0.5 TALLFormer: 59.8 mAP@0.5",
+        "robustness_measures": "Inter-rater reliability; Multiple IoU thresholds; Temporal boundary sensitivity",
+        "known_limitations": "YouTube video availability issues; Temporal boundary ambiguity; Action class overlap; Video quality variance",
+        "benchmarks_list": "THUMOS14, Kinetics, Charades, MultiTHUMOS, AVA"
+      }
+    },
+    {
+      "evaluation_name": "MATH",
+      "metric_config": {
+        "evaluation_description": "MATH Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6390989993455921,
+        "details": {
+          "subtask_a": 0.6390989993455921,
+          "subtask_b": 0.6390989993455921
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Advanced mathematical reasoning, Problem solving, Multi-step reasoning, Mathematical knowledge",
+        "functional_props": "Core Performance",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Published references",
+        "size": "Medium (1K - 100K samples)",
+        "splits": "Train (7,500), Test (5,000)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives competition math problem 2. Model generates solution with steps 3. Final answer extracted and checked against ground truth 4. Problems span 7 subjects with 5 difficulty levels",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Problems from real math competitions with verified solutions, difficulty levels validated",
+        "baseline_models": "GPT-3: 6.9% GPT-4: 42.5% Minerva (540B): 33.6% GPT-4 Turbo: 52.9% Claude 3.5 Sonnet: 71.1%",
+        "robustness_measures": "Multiple difficulty levels; Subject-wise analysis; Chain-of-thought evaluation",
+        "known_limitations": "Answer extraction challenges; LaTeX formatting issues; Symbolic vs numeric answers; High difficulty may not reflect practical math usage",
+        "benchmarks_list": "GSM8K, MathQA, SVAMP, ASDiv, Hendrycks MATH"
+      }
+    },
+    {
+      "evaluation_name": "DAVIS (Densely Annotated VIdeo Segmentation)",
+      "metric_config": {
+        "evaluation_description": "DAVIS (Densely Annotated VIdeo Segmentation) Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5354520185214521,
+        "details": {
+          "subtask_a": 0.5354520185214521,
+          "subtask_b": 0.5354520185214521
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Video object segmentation, Temporal consistency, Object tracking, Dense prediction",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Video",
+        "output_modality": "Structured Data",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Human annotations",
+        "size": "Small (< 1K samples)",
+        "splits": "Train/Val (60 sequences), Test-dev (30 sequences)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives video sequence with first frame annotation 2. Model propagates segmentation to subsequent frames 3. J&F metric (region similarity and contour accuracy)",
+        "model_access": "Outputs",
+        "has_heldout": true,
+        "alignment_validation": "High-quality manual annotations with temporal consistency verification",
+        "baseline_models": "OSVOS: 79.8 J&F STM: 84.3 J&F XMem: 86.2 J&F",
+        "robustness_measures": "Temporal consistency tested; Occlusion robustness; Multiple runs per sample",
+        "known_limitations": "Small dataset size; Limited object categories; Short video sequences; Primarily objects with clear boundaries",
+        "benchmarks_list": "YouTube-VOS, FBMS, SegTrack, OVIS"
+      }
+    },
+    {
+      "evaluation_name": "LVIS (Large Vocabulary Instance Segmentation)",
+      "metric_config": {
+        "evaluation_description": "LVIS (Large Vocabulary Instance Segmentation) Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5532790626723612,
+        "details": {
+          "subtask_a": 0.5532790626723612,
+          "subtask_b": 0.5532790626723612
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Instance segmentation, Long-tail recognition, Object detection, Fine-grained categorization",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Vision (Image)",
+        "output_modality": "Structured Data",
+        "input_source": "MS COCO",
+        "output_source": "Expert annotations",
+        "size": "Large (100K - 1M samples)",
+        "splits": "Train, Val, Test",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives image 2. Model predicts instance masks and categories (1203 classes) 3. AP computed separately for rare, common, and frequent categories",
+        "model_access": "Outputs",
+        "has_heldout": true,
+        "alignment_validation": "Expert annotators with WordNet taxonomy, quality control for long-tail categories",
+        "baseline_models": "Mask R-CNN: 21.2 AP (v1.0) Cascade R-CNN: 26.2 AP Swin Transformer: 50.9 AP",
+        "robustness_measures": "Multiple runs per sample; Ablation studies; Category frequency stratification",
+        "known_limitations": "Rare categories have very few examples; Annotation cost for 1203 categories; Some category definitions overlap; Challenging for zero-shot generalization",
+        "benchmarks_list": "COCO, Objects365, OpenImages, iNaturalist"
+      }
+    },
+    {
+      "evaluation_name": "HumanEval",
+      "metric_config": {
+        "evaluation_description": "Pass@1 Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "pass@1"
+      },
+      "score_details": {
+        "score": 0.6284191084279801,
+        "details": {
+          "subtask_a": 0.6284191084279801,
+          "subtask_b": 0.6284191084279801
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development; Selection",
+        "principles_tested": "Code generation, Programming ability, Functional correctness, Code understanding",
+        "functional_props": "Core Performance",
+        "input_modality": "Text; Code",
+        "output_modality": "Code",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Author-provided",
+        "size": "Small (< 1K samples)",
+        "splits": "Test (164 problems)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Execution-based)",
+        "protocol": "1. Model receives function signature and docstring 2. Model generates function implementation 3. Generated code executed against unit tests 4. pass@k metric computed (% passing all tests in k samples)",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Hand-written problems with comprehensive unit tests, manual verification of test correctness",
+        "baseline_models": "Codex (12B): 28.8% pass@1 GPT-3.5-turbo: 48.1% pass@1 GPT-4: 67.0% pass@1 Claude 3.5 Sonnet: 92.0% pass@1",
+        "robustness_measures": "Multiple samples per problem (pass@k); Temperature sensitivity tested; Execution-based verification",
+        "known_limitations": "Limited to Python; Small dataset size (164 problems); Relatively simple problems; May be contaminated in training data; No testing of code efficiency or style",
+        "benchmarks_list": "MBPP, APPS, CodeContests, HumanEval+, MultiPL-E"
+      }
+    },
+    {
+      "evaluation_name": "COCO (Common Objects in Context)",
+      "metric_config": {
+        "evaluation_description": "COCO (Common Objects in Context) Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5452461922279577,
+        "details": {
+          "subtask_a": 0.5452461922279577,
+          "subtask_b": 0.5452461922279577
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development; Selection",
+        "principles_tested": "Object detection, Instance segmentation, Keypoint detection, Panoptic segmentation, Image captioning",
+        "functional_props": "Core Performance",
+        "input_modality": "Vision (Image)",
+        "output_modality": "Text; Structured Data",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Human annotations",
+        "size": "Large (100K - 1M samples)",
+        "splits": "Train, Validation, Test",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives image as input 2. Model outputs bounding boxes, segmentation masks, or captions 3. Metrics computed: mAP for detection, IoU for segmentation, BLEU/CIDEr for captioning",
+        "model_access": "Outputs",
+        "has_heldout": true,
+        "alignment_validation": "Multi-annotator consensus for instance annotations, quality control through redundant labeling",
+        "baseline_models": "Faster R-CNN: 42.0 mAP Mask R-CNN: 37.1 mask mAP YOLOv8: 53.9 mAP Human performance (detection): ~70 mAP",
+        "robustness_measures": "Inter-rater reliability; Multiple runs per sample; Confidence intervals",
+        "known_limitations": "Small object detection remains challenging; Occlusion handling difficulties; Dataset bias toward certain object contexts; Annotation inconsistencies in crowded scenes",
+        "benchmarks_list": "LVIS, Objects365, Open Images, Visual Genome"
+      }
+    },
+    {
+      "evaluation_name": "ADE20K",
+      "metric_config": {
+        "evaluation_description": "ADE20K Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5163270839884204,
+        "details": {
+          "subtask_a": 0.5163270839884204,
+          "subtask_b": 0.5163270839884204
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Scene parsing, Semantic segmentation, Multi-scale recognition, Part segmentation",
+        "functional_props": "Core Performance",
+        "input_modality": "Vision (Image)",
+        "output_modality": "Structured Data",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Human annotations",
+        "size": "Medium (1K - 100K samples)",
+        "splits": "Train (20K), Val (2K), Test (3K)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives diverse scene image 2. Model predicts per-pixel semantic labels (150 classes) 3. Mean IoU and pixel accuracy computed",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Crowdsourced annotations with expert review, hierarchical consistency checks, multi-round verification",
+        "baseline_models": "PSPNet: 43.29 mIoU DeepLab v3: 45.65 mIoU UPerNet: 44.85 mIoU SegFormer-B5: 51.8 mIoU",
+        "robustness_measures": "Inter-rater reliability; Multiple runs per sample",
+        "known_limitations": "Long-tail distribution of object classes; Annotation granularity varies; Some scenes have ambiguous boundaries; Challenging for rare categories",
+        "benchmarks_list": "Cityscapes, Pascal Context, COCO-Stuff, Mapillary Vistas"
+      }
+    },
+    {
+      "evaluation_name": "IFEval (Instruction-Following Eval)",
+      "metric_config": {
+        "evaluation_description": "IFEval (Instruction-Following Eval) Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6352374911391097,
+        "details": {
+          "subtask_a": 0.6352374911391097,
+          "subtask_b": 0.6352374911391097
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development; Selection",
+        "principles_tested": "Instruction following, Constraint satisfaction, Format compliance, Precise control",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Programmatically generated",
+        "size": "Small (< 1K samples)",
+        "splits": "Test (541 prompts with ~25 verifiable instructions)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-free)",
+        "protocol": "1. Model receives prompt with verifiable instructions (e.g., 'respond in exactly 3 paragraphs', 'include word X at least 5 times') 2. Model generates response 3. Programmatic checks verify instruction compliance 4. Strict and loose accuracy metrics computed",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Verifiable instructions with programmatic checking, no ambiguity in correctness",
+        "baseline_models": "GPT-3.5: 57.4% strict GPT-4: 76.9% strict Claude 2: 66.7% strict Gemini Ultra: 79.4% strict",
+        "robustness_measures": "Strict and loose metrics; Multiple instruction types; Programmatic verification; No human evaluation needed",
+        "known_limitations": "Limited to verifiable instructions only; May not reflect realistic usage; Some instructions may be conflicting; Excludes semantic quality assessment",
+        "benchmarks_list": "MT-Bench, AlpacaEval, InstructGPT evals, FollowBench"
+      }
+    },
+    {
+      "evaluation_name": "TruthfulQA",
+      "metric_config": {
+        "evaluation_description": "TruthfulQA Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6450621366796366,
+        "details": {
+          "subtask_a": 0.6450621366796366,
+          "subtask_b": 0.6450621366796366
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development; Safety",
+        "principles_tested": "Truthfulness, Factual accuracy, Resistance to misconceptions, Calibration",
+        "functional_props": "Safety; Core Performance; Calibration",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Expert annotations",
+        "size": "Small (< 1K samples)",
+        "splits": "Test (817 questions across 38 categories)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Model-based: Expert; Human: Experts",
+        "protocol": "1. Model receives question designed to elicit false beliefs 2. Model generates answer 3. Answers judged for truthfulness and informativeness using GPT-judge or human evaluation",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Expert-curated questions targeting known misconceptions, multi-rater validation of truth labels",
+        "baseline_models": "GPT-3 (175B): 58.0% GPT-3.5: 47.0% GPT-4: 59.0% Claude 2: 62.0% Human baseline: 94.0%",
+        "robustness_measures": "Multiple evaluation methods (MC1, MC2, generative); Human validation; Model-based evaluation correlation",
+        "known_limitations": "Subjective truthfulness judgments in some cases; Cultural bias in what constitutes 'truth'; Model-based evaluation may not align with human judgment; Limited coverage of misconceptions",
+        "benchmarks_list": "FactScore, HaluEval, SelfCheckGPT, FELM"
+      }
+    },
+    {
+      "evaluation_name": "Mapillary Vistas",
+      "metric_config": {
+        "evaluation_description": "Mapillary Vistas Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.617394936359998,
+        "details": {
+          "subtask_a": 0.617394936359998,
+          "subtask_b": 0.617394936359998
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development; Deployment",
+        "principles_tested": "Semantic segmentation, Panoptic segmentation, Scene understanding, Robust perception",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Vision (Image)",
+        "output_modality": "Structured Data",
+        "input_source": "User-generated content",
+        "output_source": "Expert annotations",
+        "size": "Medium (1K - 100K samples)",
+        "splits": "Train (18K), Val (2K), Test (5K)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives street-level image 2. Model predicts per-pixel semantic labels (66 classes) 3. mIoU computed across all classes",
+        "model_access": "Outputs",
+        "has_heldout": true,
+        "alignment_validation": "Expert annotators, multi-stage quality control, geographic diversity validation",
+        "baseline_models": "PSPNet: 42.7 mIoU DeepLab v3+: 45.8 mIoU HRNetV2: 50.3 mIoU",
+        "robustness_measures": "Geographic diversity tested; Weather variations included; Multiple runs per sample",
+        "known_limitations": "Varying image quality from crowdsourced data; Camera parameter diversity; Some regions overrepresented; Annotation inconsistencies across diverse scenes",
+        "benchmarks_list": "Cityscapes, BDD100K, IDD, WildDash"
+      }
+    },
+    {
+      "evaluation_name": "Open Images",
+      "metric_config": {
+        "evaluation_description": "Open Images Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5916143740215067,
+        "details": {
+          "subtask_a": 0.5916143740215067,
+          "subtask_b": 0.5916143740215067
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development; Selection",
+        "principles_tested": "Object detection, Image classification, Visual relationship detection, Instance segmentation",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Vision (Image)",
+        "output_modality": "Structured Data",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Human annotations; Crowdsourced annotations",
+        "size": "Very Huge (> 100M samples)",
+        "splits": "Train, Val, Test",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives image 2. Model performs classification, detection, or segmentation 3. mAP computed for detection, top-k accuracy for classification",
+        "model_access": "Outputs",
+        "has_heldout": true,
+        "alignment_validation": "Multi-stage crowdsourced annotation with verification, automated quality filters",
+        "baseline_models": "Faster R-CNN: 54.3 mAP (detection) YOLOv4: 55.8 mAP EfficientDet: 56.1 mAP",
+        "robustness_measures": "Multiple runs per sample; Confidence intervals; Large-scale diversity",
+        "known_limitations": "Long-tail class distribution; Annotation inconsistencies at scale; Some classes poorly defined; Label noise in crowdsourced annotations",
+        "benchmarks_list": "COCO, LVIS, Objects365, ImageNet"
+      }
+    },
+    {
+      "evaluation_name": "BIG-bench (Beyond the Imitation Game)",
+      "metric_config": {
+        "evaluation_description": "BIG-bench (Beyond the Imitation Game) Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5128746974738502,
+        "details": {
+          "subtask_a": 0.5128746974738502,
+          "subtask_b": 0.5128746974738502
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Diverse capabilities across 204 tasks including reasoning, knowledge, language understanding, bias detection",
+        "functional_props": "Core Performance; Fairness; Robustness",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Multiple/Mixed sources",
+        "size": "Large (100K - 1M samples)",
+        "splits": "Varies by task",
+        "design": "Composite",
+        "judge": "Automatic (Reference-based); Model-based: In the wild",
+        "protocol": "1. Model evaluated on 204 diverse tasks 2. Each task has its own evaluation protocol 3. Performance aggregated across tasks",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Crowdsourced task creation with quality review, diverse authorship for broad coverage",
+        "baseline_models": "Average human rater: 89.0% Few-shot PaLM (540B): 65.7% GPT-4: ~83% (estimated on BIG-Bench Hard)",
+        "robustness_measures": "Multiple tasks provide robustness; Human baseline comparison; Cross-task analysis",
+        "known_limitations": "Task quality varies; Some tasks too easy or too hard; Computational cost of running all 204 tasks; Aggregation methodology debatable",
+        "benchmarks_list": "BIG-Bench Hard, MMLU, HELM, SuperGLUE"
+      }
+    },
+    {
+      "evaluation_name": "MT-Bench",
+      "metric_config": {
+        "evaluation_description": "Multi-turn conversation quality score (1-10)",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 1,
+        "max_score": 10,
+        "unit": "points"
+      },
+      "score_details": {
+        "score": 7.077630013459684,
+        "details": {
+          "Turn 1": 6.577630013459684,
+          "Turn 2": 7.577630013459684
+        }
+      },
+      "factsheet": {
+        "purpose": "Development; Selection",
+        "principles_tested": "Multi-turn conversation, Instruction following, Reasoning, Writing, Role-playing, Knowledge",
+        "functional_props": "Core Performance; Core Quality Dimensions",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Author-provided",
+        "size": "Small (< 1K samples)",
+        "splits": "Test (80 questions with 2 turns each)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Model-based: In the wild",
+        "protocol": "1. Model engages in 2-turn conversation 2. GPT-4 evaluates responses on 10-point scale 3. Scores averaged across turns and categories",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Strong correlation with human preferences validated on Chatbot Arena data, GPT-4 judge agreement measured",
+        "baseline_models": "Vicuna-13B: 6.39 GPT-3.5-turbo: 7.94 Claude 2: 8.06 GPT-4: 8.99 GPT-4-turbo: 9.32",
+        "robustness_measures": "Multiple categories; Position bias mitigation; Agreement with human ratings validated; Pairwise comparison",
+        "known_limitations": "Small dataset (80 questions); GPT-4 judge may have biases; Evaluation cost; Model-based judge limitations; Prompt sensitivity",
+        "benchmarks_list": "Chatbot Arena, AlpacaEval, Arena-Hard, LiveBench"
+      }
+    },
+    {
+      "evaluation_name": "HellaSwag",
+      "metric_config": {
+        "evaluation_description": "HellaSwag Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5452479136267735,
+        "details": {
+          "subtask_a": 0.5452479136267735,
+          "subtask_b": 0.5452479136267735
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Commonsense reasoning, Physical understanding, Situation modeling, Plausibility judgment",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Crowdsourced annotations",
+        "size": "Medium (1K - 100K samples)",
+        "splits": "Train (39,905), Val (10,042), Test (10,003)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives scenario context 2. Model selects most plausible continuation from 4 options 3. Accuracy computed",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Adversarial filtering using BERT to ensure difficulty, human validation of plausibility",
+        "baseline_models": "BERT-Large: 47.3% GPT-2: 50.9% GPT-3: 78.9% GPT-4: 95.3% Human performance: 95.6%",
+        "robustness_measures": "Adversarial filtering; Multiple runs per sample; Human baseline comparison",
+        "known_limitations": "Dataset may be easier than originally intended for modern LLMs; Multiple choice format; Adversarial examples may have artifacts; Potential data contamination",
+        "benchmarks_list": "PIQA, WinoGrande, CommonsenseQA, ARC"
+      }
+    },
+    {
+      "evaluation_name": "Mapillary Vistas",
+      "metric_config": {
+        "evaluation_description": "Mapillary Vistas Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6047996079543738,
+        "details": {
+          "subtask_a": 0.6047996079543738,
+          "subtask_b": 0.6047996079543738
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development; Deployment",
+        "principles_tested": "Semantic segmentation, Panoptic segmentation, Scene understanding, Robust perception",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Vision (Image)",
+        "output_modality": "Structured Data",
+        "input_source": "User-generated content",
+        "output_source": "Expert annotations",
+        "size": "Medium (1K - 100K samples)",
+        "splits": "Train (18K), Val (2K), Test (5K)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives street-level image 2. Model predicts per-pixel semantic labels (66 classes) 3. mIoU computed across all classes",
+        "model_access": "Outputs",
+        "has_heldout": true,
+        "alignment_validation": "Expert annotators, multi-stage quality control, geographic diversity validation",
+        "baseline_models": "PSPNet: 42.7 mIoU DeepLab v3+: 45.8 mIoU HRNetV2: 50.3 mIoU",
+        "robustness_measures": "Geographic diversity tested; Weather variations included; Multiple runs per sample",
+        "known_limitations": "Varying image quality from crowdsourced data; Camera parameter diversity; Some regions overrepresented; Annotation inconsistencies across diverse scenes",
+        "benchmarks_list": "Cityscapes, BDD100K, IDD, WildDash"
+      }
+    },
+    {
+      "evaluation_name": "IFEval (Instruction-Following Eval)",
+      "metric_config": {
+        "evaluation_description": "IFEval (Instruction-Following Eval) Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5004573523401303,
+        "details": {
+          "subtask_a": 0.5004573523401303,
+          "subtask_b": 0.5004573523401303
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development; Selection",
+        "principles_tested": "Instruction following, Constraint satisfaction, Format compliance, Precise control",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Programmatically generated",
+        "size": "Small (< 1K samples)",
+        "splits": "Test (541 prompts with ~25 verifiable instructions)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-free)",
+        "protocol": "1. Model receives prompt with verifiable instructions (e.g., 'respond in exactly 3 paragraphs', 'include word X at least 5 times') 2. Model generates response 3. Programmatic checks verify instruction compliance 4. Strict and loose accuracy metrics computed",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Verifiable instructions with programmatic checking, no ambiguity in correctness",
+        "baseline_models": "GPT-3.5: 57.4% strict GPT-4: 76.9% strict Claude 2: 66.7% strict Gemini Ultra: 79.4% strict",
+        "robustness_measures": "Strict and loose metrics; Multiple instruction types; Programmatic verification; No human evaluation needed",
+        "known_limitations": "Limited to verifiable instructions only; May not reflect realistic usage; Some instructions may be conflicting; Excludes semantic quality assessment",
+        "benchmarks_list": "MT-Bench, AlpacaEval, InstructGPT evals, FollowBench"
+      }
+    },
+    {
+      "evaluation_name": "Waymo Open Dataset",
+      "metric_config": {
+        "evaluation_description": "Waymo Open Dataset Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6443135557730899,
+        "details": {
+          "subtask_a": 0.6443135557730899,
+          "subtask_b": 0.6443135557730899
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development; Deployment",
+        "principles_tested": "3D object detection, 2D object detection, Tracking, Domain adaptation, Sensor fusion",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Vision (Image); Video; Structured Data",
+        "output_modality": "Structured Data",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Human annotations; Programmatically generated",
+        "size": "Huge (> 10M samples)",
+        "splits": "Train (798), Val (202), Test (150)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives multimodal sensor data 2. Model predicts 3D bounding boxes with tracking IDs 3. AP/APH metrics at different IoU thresholds and difficulty levels",
+        "model_access": "Outputs",
+        "has_heldout": true,
+        "alignment_validation": "Multi-stage annotation pipeline with quality checks, temporal consistency validation",
+        "baseline_models": "PointPillars: 63.8 L2 APH (Vehicle) CenterPoint: 73.9 L2 APH PV-RCNN++: 77.8 L2 APH",
+        "robustness_measures": "Geographic diversity; Time of day variations; Weather conditions; Multiple difficulty levels",
+        "known_limitations": "Geographic concentration in specific cities; Sensor-specific challenges; Annotation latency for distant objects; Class imbalance",
+        "benchmarks_list": "nuScenes, KITTI, Argoverse 2, Once, Lyft Level 5"
+      }
+    },
+    {
+      "evaluation_name": "CIFAR-10 and CIFAR-100",
+      "metric_config": {
+        "evaluation_description": "CIFAR-10 and CIFAR-100 Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6476991658227166,
+        "details": {
+          "subtask_a": 0.6476991658227166,
+          "subtask_b": 0.6476991658227166
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development; Selection",
+        "principles_tested": "Image classification, Transfer learning, Generalization",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Vision (Image)",
+        "output_modality": "Structured Data",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Human annotations",
+        "size": "Medium (1K - 100K samples)",
+        "splits": "Train (50K), Test (10K)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives 32x32 RGB image 2. Model predicts class label (10 or 100 classes) 3. Classification accuracy computed",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Systematic sampling from larger dataset (80 million tiny images), human verification of labels",
+        "baseline_models": "CIFAR-10: ResNet-56: 93.03% Wide ResNet-28-10: 96.11% PyramidNet: 96.54%  CIFAR-100: ResNet-56: 71.35% Wide ResNet-28-10: 81.15% PyramidNet: 83.78%",
+        "robustness_measures": "Multiple runs per sample; Seed variation tested; Ablation studies",
+        "known_limitations": "Low resolution (32x32) limits fine-grained recognition; Some label noise in CIFAR-100; Dataset size enables memorization in large models; Limited diversity in poses and contexts",
+        "benchmarks_list": "CIFAR-10-C, CIFAR-10.1, CIFAR-100-C, STL-10, Tiny ImageNet"
+      }
+    },
+    {
+      "evaluation_name": "DAVIS (Densely Annotated VIdeo Segmentation)",
+      "metric_config": {
+        "evaluation_description": "DAVIS (Densely Annotated VIdeo Segmentation) Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6232228420729622,
+        "details": {
+          "subtask_a": 0.6232228420729622,
+          "subtask_b": 0.6232228420729622
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Video object segmentation, Temporal consistency, Object tracking, Dense prediction",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Video",
+        "output_modality": "Structured Data",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Human annotations",
+        "size": "Small (< 1K samples)",
+        "splits": "Train/Val (60 sequences), Test-dev (30 sequences)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives video sequence with first frame annotation 2. Model propagates segmentation to subsequent frames 3. J&F metric (region similarity and contour accuracy)",
+        "model_access": "Outputs",
+        "has_heldout": true,
+        "alignment_validation": "High-quality manual annotations with temporal consistency verification",
+        "baseline_models": "OSVOS: 79.8 J&F STM: 84.3 J&F XMem: 86.2 J&F",
+        "robustness_measures": "Temporal consistency tested; Occlusion robustness; Multiple runs per sample",
+        "known_limitations": "Small dataset size; Limited object categories; Short video sequences; Primarily objects with clear boundaries",
+        "benchmarks_list": "YouTube-VOS, FBMS, SegTrack, OVIS"
+      }
+    },
+    {
+      "evaluation_name": "WinoGrande",
+      "metric_config": {
+        "evaluation_description": "WinoGrande Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6106030387010124,
+        "details": {
+          "subtask_a": 0.6106030387010124,
+          "subtask_b": 0.6106030387010124
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Commonsense reasoning, Coreference resolution, World knowledge, Causal reasoning",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Crowdsourced annotations",
+        "size": "Medium (1K - 100K samples)",
+        "splits": "Train (40,398), Dev (1,267), Test (1,767)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives sentence with pronoun ambiguity 2. Model selects correct referent from 2 options 3. Accuracy computed",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Adversarial filtering using language models, crowdsourced generation with validation",
+        "baseline_models": "BERT-Large: 59.4% RoBERTa-Large: 79.1% GPT-3: 70.2% GPT-4: 87.5% Human performance: 94.0%",
+        "robustness_measures": "Adversarial filtering; Large-scale dataset; Multiple difficulty levels",
+        "known_limitations": "Binary choice may be limiting; Adversarial filtering may introduce artifacts; Saturation with modern models; Limited reasoning depth",
+        "benchmarks_list": "Winograd Schema Challenge, COPA, CommonsenseQA, PIQA"
+      }
+    },
+    {
+      "evaluation_name": "LVIS (Large Vocabulary Instance Segmentation)",
+      "metric_config": {
+        "evaluation_description": "LVIS (Large Vocabulary Instance Segmentation) Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5053512833031226,
+        "details": {
+          "subtask_a": 0.5053512833031226,
+          "subtask_b": 0.5053512833031226
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Instance segmentation, Long-tail recognition, Object detection, Fine-grained categorization",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Vision (Image)",
+        "output_modality": "Structured Data",
+        "input_source": "MS COCO",
+        "output_source": "Expert annotations",
+        "size": "Large (100K - 1M samples)",
+        "splits": "Train, Val, Test",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives image 2. Model predicts instance masks and categories (1203 classes) 3. AP computed separately for rare, common, and frequent categories",
+        "model_access": "Outputs",
+        "has_heldout": true,
+        "alignment_validation": "Expert annotators with WordNet taxonomy, quality control for long-tail categories",
+        "baseline_models": "Mask R-CNN: 21.2 AP (v1.0) Cascade R-CNN: 26.2 AP Swin Transformer: 50.9 AP",
+        "robustness_measures": "Multiple runs per sample; Ablation studies; Category frequency stratification",
+        "known_limitations": "Rare categories have very few examples; Annotation cost for 1203 categories; Some category definitions overlap; Challenging for zero-shot generalization",
+        "benchmarks_list": "COCO, Objects365, OpenImages, iNaturalist"
+      }
+    },
+    {
+      "evaluation_name": "nuScenes",
+      "metric_config": {
+        "evaluation_description": "nuScenes Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6121984247898792,
+        "details": {
+          "subtask_a": 0.6121984247898792,
+          "subtask_b": 0.6121984247898792
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development; Deployment",
+        "principles_tested": "3D object detection, Multi-object tracking, Prediction, Sensor fusion, Scene understanding",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Vision (Image); Video; Structured Data",
+        "output_modality": "Structured Data",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Human annotations",
+        "size": "Medium (1K - 100K samples)",
+        "splits": "Train (700), Val (150), Test (150)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives multimodal sensor data (cameras, LiDAR, radar) 2. Model predicts 3D bounding boxes and tracking IDs 3. NDS (nuScenes Detection Score) and mAP computed",
+        "model_access": "Outputs",
+        "has_heldout": true,
+        "alignment_validation": "Expert annotators, multi-sensor consistency checks, temporal coherence validation",
+        "baseline_models": "PointPillars: 45.3 NDS CenterPoint: 65.5 NDS BEVFusion: 72.9 NDS",
+        "robustness_measures": "Geographic diversity; Weather conditions; Day/night variations; Multiple runs per sample",
+        "known_limitations": "Limited geographic coverage (Boston, Singapore); Sensor calibration challenges; Annotation latency for fast-moving objects; Class imbalance",
+        "benchmarks_list": "Waymo Open Dataset, KITTI, Argoverse, Lyft Level 5, A2D2"
+      }
+    },
+    {
+      "evaluation_name": "CLEVR",
+      "metric_config": {
+        "evaluation_description": "CLEVR Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6670278396978662,
+        "details": {
+          "subtask_a": 0.6670278396978662,
+          "subtask_b": 0.6670278396978662
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Compositional reasoning, Spatial reasoning, Counting, Logical reasoning, Visual reasoning",
+        "functional_props": "Core Performance; Robustness",
+        "input_modality": "Text + Vision",
+        "output_modality": "Text",
+        "input_source": "Synthetic/Generated",
+        "output_source": "Programmatically generated",
+        "size": "Huge (> 10M samples)",
+        "splits": "Train (70K), Val (15K), Test (15K)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Automatic (Reference-based)",
+        "protocol": "1. Model receives synthetic image and question 2. Model predicts answer from predefined set 3. Accuracy computed, can analyze by question type",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Programmatically generated with known ground truth, exhaustive question type coverage",
+        "baseline_models": "CNN+LSTM: 52.3% Film: 97.7% NS-VQA: 99.8% MAC: 98.9%",
+        "robustness_measures": "Ablation studies; Question type analysis; Compositional generalization tested",
+        "known_limitations": "Synthetic domain (limited real-world applicability); Simple shapes and colors; No natural language variation; Programmatic biases",
+        "benchmarks_list": "CLEVR-CoGenT, GQA, NLVR2, CLOSURE"
+      }
+    },
+    {
+      "evaluation_name": "TruthfulQA",
+      "metric_config": {
+        "evaluation_description": "TruthfulQA Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5942149514779176,
+        "details": {
+          "subtask_a": 0.5942149514779176,
+          "subtask_b": 0.5942149514779176
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development; Safety",
+        "principles_tested": "Truthfulness, Factual accuracy, Resistance to misconceptions, Calibration",
+        "functional_props": "Safety; Core Performance; Calibration",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Expert annotations",
+        "size": "Small (< 1K samples)",
+        "splits": "Test (817 questions across 38 categories)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Model-based: Expert; Human: Experts",
+        "protocol": "1. Model receives question designed to elicit false beliefs 2. Model generates answer 3. Answers judged for truthfulness and informativeness using GPT-judge or human evaluation",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Expert-curated questions targeting known misconceptions, multi-rater validation of truth labels",
+        "baseline_models": "GPT-3 (175B): 58.0% GPT-3.5: 47.0% GPT-4: 59.0% Claude 2: 62.0% Human baseline: 94.0%",
+        "robustness_measures": "Multiple evaluation methods (MC1, MC2, generative); Human validation; Model-based evaluation correlation",
+        "known_limitations": "Subjective truthfulness judgments in some cases; Cultural bias in what constitutes 'truth'; Model-based evaluation may not align with human judgment; Limited coverage of misconceptions",
+        "benchmarks_list": "FactScore, HaluEval, SelfCheckGPT, FELM"
+      }
+    },
+    {
+      "evaluation_name": "HarmBench",
+      "metric_config": {
+        "evaluation_description": "HarmBench Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6260154867288402,
+        "details": {
+          "subtask_a": 0.6260154867288402,
+          "subtask_b": 0.6260154867288402
+        }
+      },
+      "factsheet": {
+        "purpose": "Evaluation",
+        "principles_tested": "Safety; Robustness",
+        "functional_props": "Adversarial",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "New dataset",
+        "output_source": "Human annotations",
+        "size": "Medium (1K - 10K samples)",
+        "splits": "Train/Val/Test",
+        "design": "Static",
+        "judge": "Automatic (LLM Judge)",
+        "protocol": "Comparison against refusal baseline",
+        "model_access": "Outputs",
+        "has_heldout": true,
+        "alignment_validation": "High agreement with human annotators",
+        "baseline_models": "GPT-4: 90% Refusal",
+        "robustness_measures": "Automated Red Teaming",
+        "known_limitations": "Focuses on refusal rather than helpfulness",
+        "benchmarks_list": "JailbreakBench, RealToxicityPrompts"
+      }
+    },
+    {
+      "evaluation_name": "AdvGLUE",
+      "metric_config": {
+        "evaluation_description": "AdvGLUE Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6269886500675274,
+        "details": {
+          "subtask_a": 0.6269886500675274,
+          "subtask_b": 0.6269886500675274
+        }
+      },
+      "factsheet": {
+        "purpose": "Evaluation",
+        "principles_tested": "Robustness",
+        "functional_props": "Adversarial",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "Modified GLUE",
+        "output_source": "Perturbed data",
+        "size": "Medium",
+        "splits": "Dev/Test",
+        "design": "Static",
+        "judge": "Automatic",
+        "protocol": "Accuracy under adversarial perturbation",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "N/A",
+        "baseline_models": "BERT: 40% drop",
+        "robustness_measures": "Word-level perturbations",
+        "known_limitations": "May not transfer to all models",
+        "benchmarks_list": "GLUE, SuperGLUE"
+      }
+    },
+    {
+      "evaluation_name": "Carlini Extraction",
+      "metric_config": {
+        "evaluation_description": "Carlini Extraction Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6947563937161508,
+        "details": {
+          "subtask_a": 0.6947563937161508,
+          "subtask_b": 0.6947563937161508
+        }
+      },
+      "factsheet": {
+        "purpose": "Research",
+        "principles_tested": "Privacy; Memorization",
+        "functional_props": "Memorization",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "C4",
+        "output_source": "Model outputs",
+        "size": "Huge",
+        "splits": "N/A",
+        "design": "Dynamic",
+        "judge": "Automatic",
+        "protocol": "Eidetic memorization metric",
+        "model_access": "Weights",
+        "has_heldout": false,
+        "alignment_validation": "N/A",
+        "baseline_models": "T5: Variable memorization",
+        "robustness_measures": "Varying model scale",
+        "known_limitations": "Computationally expensive",
+        "benchmarks_list": "The Pile Extraction"
+      }
+    },
+    {
+      "evaluation_name": "BIG-bench (Beyond the Imitation Game)",
+      "metric_config": {
+        "evaluation_description": "BIG-bench (Beyond the Imitation Game) Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.571212624256759,
+        "details": {
+          "subtask_a": 0.571212624256759,
+          "subtask_b": 0.571212624256759
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development",
+        "principles_tested": "Diverse capabilities across 204 tasks including reasoning, knowledge, language understanding, bias detection",
+        "functional_props": "Core Performance; Fairness; Robustness",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Multiple/Mixed sources",
+        "size": "Large (100K - 1M samples)",
+        "splits": "Varies by task",
+        "design": "Composite",
+        "judge": "Automatic (Reference-based); Model-based: In the wild",
+        "protocol": "1. Model evaluated on 204 diverse tasks 2. Each task has its own evaluation protocol 3. Performance aggregated across tasks",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Crowdsourced task creation with quality review, diverse authorship for broad coverage",
+        "baseline_models": "Average human rater: 89.0% Few-shot PaLM (540B): 65.7% GPT-4: ~83% (estimated on BIG-Bench Hard)",
+        "robustness_measures": "Multiple tasks provide robustness; Human baseline comparison; Cross-task analysis",
+        "known_limitations": "Task quality varies; Some tasks too easy or too hard; Computational cost of running all 204 tasks; Aggregation methodology debatable",
+        "benchmarks_list": "BIG-Bench Hard, MMLU, HELM, SuperGLUE"
+      }
+    },
+    {
+      "evaluation_name": "BBQ",
+      "metric_config": {
+        "evaluation_description": "BBQ Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6107373409618154,
+        "details": {
+          "subtask_a": 0.6107373409618154,
+          "subtask_b": 0.6107373409618154
+        }
+      },
+      "factsheet": {
+        "purpose": "Evaluation",
+        "principles_tested": "Fairness",
+        "functional_props": "Fairness",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "Hand-crafted templates",
+        "output_source": "QA pairs",
+        "size": "Medium (58K examples)",
+        "splits": "Test",
+        "design": "Static",
+        "judge": "Automatic",
+        "protocol": "Accuracy difference between groups",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "N/A",
+        "baseline_models": "UnifiedQA: Shows bias",
+        "robustness_measures": "Ambiguous contexts",
+        "known_limitations": "US-centric social biases",
+        "benchmarks_list": "CrowS-Pairs, WinoBias"
+      }
+    },
+    {
+      "evaluation_name": "CrowS-Pairs",
+      "metric_config": {
+        "evaluation_description": "CrowS-Pairs Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.606059582657115,
+        "details": {
+          "subtask_a": 0.606059582657115,
+          "subtask_b": 0.606059582657115
+        }
+      },
+      "factsheet": {
+        "purpose": "Evaluation",
+        "principles_tested": "Fairness",
+        "functional_props": "Fairness",
+        "input_modality": "Text",
+        "output_modality": "Score",
+        "input_source": "Crowdsourced",
+        "output_source": "Sentence pairs",
+        "size": "Small (1508 pairs)",
+        "splits": "Test",
+        "design": "Static",
+        "judge": "Automatic",
+        "protocol": "Preference for stereotypical sentence",
+        "model_access": "Logprobs",
+        "has_heldout": false,
+        "alignment_validation": "N/A",
+        "baseline_models": "BERT: 60% Stereotypical",
+        "robustness_measures": "N/A",
+        "known_limitations": "Crowdworker bias",
+        "benchmarks_list": "BBQ, WinoBias"
+      }
+    },
+    {
+      "evaluation_name": "TruthfulQA",
+      "metric_config": {
+        "evaluation_description": "TruthfulQA Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6025955378285834,
+        "details": {
+          "subtask_a": 0.6025955378285834,
+          "subtask_b": 0.6025955378285834
+        }
+      },
+      "factsheet": {
+        "purpose": "Research; Development; Safety",
+        "principles_tested": "Truthfulness, Factual accuracy, Resistance to misconceptions, Calibration",
+        "functional_props": "Safety; Core Performance; Calibration",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "New dataset (released with eval)",
+        "output_source": "Expert annotations",
+        "size": "Small (< 1K samples)",
+        "splits": "Test (817 questions across 38 categories)",
+        "design": "Fixed data-driven (static test set)",
+        "judge": "Model-based: Expert; Human: Experts",
+        "protocol": "1. Model receives question designed to elicit false beliefs 2. Model generates answer 3. Answers judged for truthfulness and informativeness using GPT-judge or human evaluation",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "Expert-curated questions targeting known misconceptions, multi-rater validation of truth labels",
+        "baseline_models": "GPT-3 (175B): 58.0% GPT-3.5: 47.0% GPT-4: 59.0% Claude 2: 62.0% Human baseline: 94.0%",
+        "robustness_measures": "Multiple evaluation methods (MC1, MC2, generative); Human validation; Model-based evaluation correlation",
+        "known_limitations": "Subjective truthfulness judgments in some cases; Cultural bias in what constitutes 'truth'; Model-based evaluation may not align with human judgment; Limited coverage of misconceptions",
+        "benchmarks_list": "FactScore, HaluEval, SelfCheckGPT, FELM"
+      }
+    },
+    {
+      "evaluation_name": "Do Not Answer",
+      "metric_config": {
+        "evaluation_description": "Do Not Answer Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6116615152220208,
+        "details": {
+          "subtask_a": 0.6116615152220208,
+          "subtask_b": 0.6116615152220208
+        }
+      },
+      "factsheet": {
+        "purpose": "Evaluation",
+        "principles_tested": "Safety",
+        "functional_props": "Safety",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "Hand-crafted harmful instructions",
+        "output_source": "Refusal/Response",
+        "size": "Small (939 instructions)",
+        "splits": "Test",
+        "design": "Static",
+        "judge": "Automatic (GPT-4 Judge)",
+        "protocol": "Refusal rate and harmfulness score",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "High agreement",
+        "baseline_models": "Llama-2: High refusal",
+        "robustness_measures": "Jailbreaks",
+        "known_limitations": "Judge bias",
+        "benchmarks_list": "SafetyBench"
+      }
+    },
+    {
+      "evaluation_name": "RealToxicityPrompts",
+      "metric_config": {
+        "evaluation_description": "Average Toxicity Score",
+        "lower_is_better": true,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "score"
+      },
+      "score_details": {
+        "score": 0.4727521375406737,
+        "details": {
+          "subtask_a": 0.4727521375406737,
+          "subtask_b": 0.4727521375406737
+        }
+      },
+      "factsheet": {
+        "purpose": "Evaluation",
+        "principles_tested": "Safety",
+        "functional_props": "Safety",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "Web text",
+        "output_source": "Continuation",
+        "size": "Large (100K prompts)",
+        "splits": "Train/Test",
+        "design": "Static",
+        "judge": "Automatic (Perspective API)",
+        "protocol": "Toxicity score of continuation",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "N/A",
+        "baseline_models": "GPT-2: Toxic outputs",
+        "robustness_measures": "Top-k sampling",
+        "known_limitations": "Perspective API limitations",
+        "benchmarks_list": "SafetyBench, Do Not Answer"
+      }
+    },
+    {
+      "evaluation_name": "Min-K% Probes",
+      "metric_config": {
+        "evaluation_description": "Min-K% Probes Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.6871782084591865,
+        "details": {
+          "subtask_a": 0.6871782084591865,
+          "subtask_b": 0.6871782084591865
+        }
+      },
+      "factsheet": {
+        "purpose": "Evaluation",
+        "principles_tested": "Leakage; Copyright",
+        "functional_props": "Leakage/Contamination",
+        "input_modality": "Text",
+        "output_modality": "Score",
+        "input_source": "WikiMIA",
+        "output_source": "Log-likelihood",
+        "size": "Medium",
+        "splits": "N/A",
+        "design": "Static",
+        "judge": "Automatic",
+        "protocol": "Likelihood ratio test",
+        "model_access": "Logprobs",
+        "has_heldout": true,
+        "alignment_validation": "N/A",
+        "baseline_models": "Llama-2: High detection",
+        "robustness_measures": "Paraphrase attacks",
+        "known_limitations": "Requires logprobs",
+        "benchmarks_list": "Contamination Detector"
+      }
+    },
+    {
+      "evaluation_name": "PII Detection",
+      "metric_config": {
+        "evaluation_description": "PII Detection Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5583254817362804,
+        "details": {
+          "subtask_a": 0.5583254817362804,
+          "subtask_b": 0.5583254817362804
+        }
+      },
+      "factsheet": {
+        "purpose": "Evaluation",
+        "principles_tested": "Privacy",
+        "functional_props": "Privacy",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "Synthetic PII",
+        "output_source": "Tagged text",
+        "size": "Large",
+        "splits": "Train/Test",
+        "design": "Static",
+        "judge": "Automatic",
+        "protocol": "F1 score on PII tags",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "N/A",
+        "baseline_models": "BERT-NER: 95% F1",
+        "robustness_measures": "Context variation",
+        "known_limitations": "Definition of PII varies",
+        "benchmarks_list": "PrivacyBench"
+      }
+    },
+    {
+      "evaluation_name": "Interpretability Benchmark",
+      "metric_config": {
+        "evaluation_description": "Interpretability Benchmark Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5522975302424581,
+        "details": {
+          "subtask_a": 0.5522975302424581,
+          "subtask_b": 0.5522975302424581
+        }
+      },
+      "factsheet": {
+        "purpose": "Research",
+        "principles_tested": "Interpretability",
+        "functional_props": "Interpretability",
+        "input_modality": "Text/Image",
+        "output_modality": "Heatmap",
+        "input_source": "Various",
+        "output_source": "Attribution scores",
+        "size": "Medium",
+        "splits": "N/A",
+        "design": "Static",
+        "judge": "Automatic",
+        "protocol": "Faithfulness and Plausibility metrics",
+        "model_access": "Weights",
+        "has_heldout": false,
+        "alignment_validation": "N/A",
+        "baseline_models": "ResNet: High faithfulness",
+        "robustness_measures": "Perturbation tests",
+        "known_limitations": "Metrics can be unstable",
+        "benchmarks_list": "TruthfulQA"
+      }
+    },
+    {
+      "evaluation_name": "Token/sec Benchmark",
+      "metric_config": {
+        "evaluation_description": "Inference throughput (tokens/sec)",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 200,
+        "unit": "tok/s"
+      },
+      "score_details": {
+        "score": 123.6267836232318,
+        "details": {
+          "subtask_a": 123.6267836232318,
+          "subtask_b": 123.6267836232318
+        }
+      },
+      "factsheet": {
+        "purpose": "Evaluation",
+        "principles_tested": "Efficiency",
+        "functional_props": "Efficiency",
+        "input_modality": "Text",
+        "output_modality": "Metrics",
+        "input_source": "Various",
+        "output_source": "Tokens/sec",
+        "size": "N/A",
+        "splits": "N/A",
+        "design": "Dynamic",
+        "judge": "Automatic",
+        "protocol": "Tokens per second generation",
+        "model_access": "API",
+        "has_heldout": false,
+        "alignment_validation": "N/A",
+        "baseline_models": "GPT-4: 20 tok/sec",
+        "robustness_measures": "Batch size variation",
+        "known_limitations": "Varies by provider",
+        "benchmarks_list": "LLMPerf"
+      }
+    },
+    {
+      "evaluation_name": "CL-Benchmark",
+      "metric_config": {
+        "evaluation_description": "CL-Benchmark Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.52498221840426,
+        "details": {
+          "subtask_a": 0.52498221840426,
+          "subtask_b": 0.52498221840426
+        }
+      },
+      "factsheet": {
+        "purpose": "Research",
+        "principles_tested": "Retrainability",
+        "functional_props": "Retrainability/Continual Learning",
+        "input_modality": "Text",
+        "output_modality": "Text",
+        "input_source": "Split MNIST/Cifar/Text",
+        "output_source": "Accuracy over time",
+        "size": "Medium",
+        "splits": "Sequential",
+        "design": "Dynamic",
+        "judge": "Automatic",
+        "protocol": "Forgetting rate and Forward transfer",
+        "model_access": "Weights",
+        "has_heldout": false,
+        "alignment_validation": "N/A",
+        "baseline_models": "EWC: Reduces forgetting",
+        "robustness_measures": "Task ordering",
+        "known_limitations": "Requires training access",
+        "benchmarks_list": "L2M"
+      }
+    },
+    {
+      "evaluation_name": "Omniglot",
+      "metric_config": {
+        "evaluation_description": "Omniglot Standard Accuracy",
+        "lower_is_better": false,
+        "score_type": "continuous",
+        "min_score": 0,
+        "max_score": 1,
+        "unit": "accuracy"
+      },
+      "score_details": {
+        "score": 0.5763375517493838,
+        "details": {
+          "subtask_a": 0.5763375517493838,
+          "subtask_b": 0.5763375517493838
+        }
+      },
+      "factsheet": {
+        "purpose": "Research",
+        "principles_tested": "Meta-Learning",
+        "functional_props": "Meta-Learning",
+        "input_modality": "Image",
+        "output_modality": "Class",
+        "input_source": "Handwritten characters",
+        "output_source": "Classification",
+        "size": "Small (1623 chars)",
+        "splits": "Background/Evaluation",
+        "design": "Static",
+        "judge": "Automatic",
+        "protocol": "One-shot classification accuracy",
+        "model_access": "Outputs",
+        "has_heldout": false,
+        "alignment_validation": "N/A",
+        "baseline_models": "Human: 95%",
+        "robustness_measures": "Few-shot variation",
+        "known_limitations": "Simple visual domain",
+        "benchmarks_list": "Meta-Dataset"
+      }
+    }
+  ],
+  "detailed_evaluation_results_per_samples": [
+    {
+      "sample_id": "sample_0",
+      "input": "Test input question 0 for Qwen 2 72B...",
+      "ground_truth": "Expected answer 0",
+      "response": "Model generated response 0 which is mostly correct...",
+      "score": 0.85
+    },
+    {
+      "sample_id": "sample_1",
+      "input": "Test input question 1 for Qwen 2 72B...",
+      "ground_truth": "Expected answer 1",
+      "response": "Model generated response 1 which is mostly correct...",
+      "score": 0.85
+    },
+    {
+      "sample_id": "sample_2",
+      "input": "Test input question 2 for Qwen 2 72B...",
+      "ground_truth": "Expected answer 2",
+      "response": "Model generated response 2 which is mostly correct...",
+      "score": 0.85
+    },
+    {
+      "sample_id": "sample_3",
+      "input": "Test input question 3 for Qwen 2 72B...",
+      "ground_truth": "Expected answer 3",
+      "response": "Model generated response 3 which is mostly correct...",
+      "score": 0.85
+    },
+    {
+      "sample_id": "sample_4",
+      "input": "Test input question 4 for Qwen 2 72B...",
+      "ground_truth": "Expected answer 4",
+      "response": "Model generated response 4 which is mostly correct...",
+      "score": 0.85
+    }
+  ]
+}

public/evaluations/claude-3-sonnet.json

@@ -1,1560 +0,0 @@
-{
-  "id": "claude-3-sonnet-2024",
-  "systemName": "Claude 3.5 Sonnet",
-  "url": "https://claude.ai",
-  "provider": "Anthropic",
-  "version": "claude-3-5-sonnet-20241022",
-  "modelTag": "claude-3-5-sonnet-20241022",
-  "knowledgeCutoff": "2024-04-01",
-  "modelType": "foundational",
-  "inputModalities": ["Text", "Image"],
-  "outputModalities": ["Text"],
-  "deploymentContexts": ["Public/Consumer-Facing", "Internal/Enterprise Use"],
-  "evaluationDate": "2024-11-15",
-  "evaluator": "Anthropic Constitutional AI Team",
-  "selectedCategories": [
-    "language-communication",
-    "problem-solving",
-    "creativity-innovation",
-    "learning-memory",
-    "social-intelligence",
-    "perception-vision",
-    "metacognition",
-    "physical-manipulation",
-    "robotic-intelligence",
-    "harmful-content",
-    "bias-fairness",
-    "information-integrity",
-    "privacy-data",
-    "security-robustness",
-    "dangerous-capabilities",
-    "human-ai-interaction",
-    "governance-accountability",
-    "value-chain",
-    "environmental-impact",
-    "economic-displacement"
-  ],
-  "overallStats": {
-    "totalApplicable": 20,
-    "capabilityApplicable": 9,
-    "riskApplicable": 11,
-    "completenessScore": 96,
-    "strongCategories": [
-      "language-communication",
-      "social-intelligence", 
-      "creativity-innovation",
-      "harmful-content",
-      "bias-fairness",
-      "information-integrity",
-      "human-ai-interaction",
-      "governance-accountability"
-    ],
-    "adequateCategories": [
-      "problem-solving",
-      "learning-memory",
-      "perception-vision",
-      "metacognition",
-      "privacy-data",
-      "security-robustness"
-    ],
-    "weakCategories": [
-      "dangerous-capabilities",
-      "environmental-impact",
-      "economic-displacement"
-    ],
-    "insufficientCategories": [
-      "physical-manipulation",
-      "robotic-intelligence",
-      "value-chain"
-    ],
-    "priorityAreas": [
-      "dangerous-capabilities",
-      "environmental-impact",
-      "value-chain"
-    ]
-  },
-  "categoryEvaluations": {
-    "language-communication": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "yes",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": "yes",
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-claude35s-lc-001",
-            "url": "https://www.anthropic.com/news/claude-3-family",
-            "description": "Claude 3.5 Sonnet performance across comprehensive language understanding and generation tasks",
-            "sourceType": "external",
-            "benchmarkName": "MMLU, HellaSwag, ARC-Challenge, WinoGrande, GSM8K",
-            "metrics": "Accuracy on reasoning and knowledge tasks",
-            "score": "88.7% MMLU, 95.4% HellaSwag, 96.4% ARC-C, 89.0% WinoGrande, 96.4% GSM8K",
-            "version": "claude-3-5-sonnet-20241022",
-            "taskVariants": "academic-knowledge, commonsense-reasoning, math-reasoning, reading-comprehension",
-            "customFields": {}
-          }
-        ],
-        "A2": [
-          {
-            "id": "bench-claude35s-lc-002",
-            "url": "https://www.anthropic.com/safety",
-            "description": "Constitutional AI evaluation showing strong safety alignment and helpful refusal patterns",
-            "sourceType": "internal",
-            "benchmarkName": "Anthropic HHH Eval, Constitutional AI Safety Suite",
-            "metrics": "Helpfulness, harmlessness, honesty scores",
-            "score": "92% helpfulness, 97% harmlessness, 89% honesty",
-            "version": "v3.5",
-            "taskVariants": "safety-alignment, constitutional-ai, harmlessness",
-            "customFields": {}
-          }
-        ],
-        "A3": [
-          {
-            "id": "bench-meem78cs-204wem",
-            "url": "https://www.anthropic.com/news/claude-3-5-sonnet",
-            "description": "Comparative analysis showing Claude 3.5 Sonnet performance vs competitors",
-            "sourceType": "external",
-            "benchmarkName": "",
-            "metrics": "",
-            "score": "",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ],
-        "A4": [
-          {
-            "id": "bench-meem78cs-l1yydy",
-            "url": "https://www.anthropic.com/research/constitutional-ai",
-            "description": "Robustness testing against adversarial inputs",
-            "sourceType": "internal",
-            "benchmarkName": "",
-            "metrics": "",
-            "score": "",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ],
-        "A5": [
-          {
-            "id": "bench-meem78cs-ehpweq",
-            "url": "https://www.anthropic.com/monitoring",
-            "description": "Production monitoring and safety metrics",
-            "sourceType": "internal",
-            "benchmarkName": "",
-            "metrics": "",
-            "score": "",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ],
-        "A6": [
-          {
-            "id": "bench-meem78cs-x4yc6z",
-            "url": "https://www.anthropic.com/research/training-data",
-            "description": "Training data contamination analysis",
-            "sourceType": "internal",
-            "benchmarkName": "",
-            "metrics": "",
-            "score": "",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cs-k7dyn8",
-            "url": "https://www.anthropic.com/news/claude-3-family",
-            "description": "Comprehensive documentation of Claude's language capabilities",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B2": [
-          {
-            "id": "proc-meem78cs-563xw4",
-            "url": "https://www.anthropic.com/research/claude-3-family",
-            "description": "Comprehensive research methodology and experimental design for Claude 3 family",
-            "title": "Claude 3 Family Technical Documentation",
-            "author": "Anthropic Research Team",
-            "organization": "Anthropic",
-            "date": "2024-03-04",
-            "documentType": "Research Paper"
-          },
-          {
-            "id": "proc-meem78cs-563xw5",
-            "url": "https://github.com/anthropics/anthropic-cookbook",
-            "description": "Open cookbook with evaluation prompts and reproducible examples",
-            "title": "Anthropic Evaluation Cookbook",
-            "author": "Developer Relations Team",
-            "organization": "Anthropic",
-            "date": "2024-03-15",
-            "documentType": "Code Repository"
-          },
-          {
-            "id": "proc-meem78cs-563xw6",
-            "url": "https://www.anthropic.com/safety/evaluation-standards",
-            "description": "Detailed evaluation standards and procedures for model assessment",
-            "title": "Model Evaluation Standards v3.2",
-            "author": "Safety Research Division",
-            "organization": "Anthropic",
-            "date": "2024-02-28",
-            "documentType": "Standards Document"
-          }
-        ],
-        "B5": [
-          {
-            "id": "proc-meem78cs-563xw7",
-            "url": "https://www.anthropic.com/compliance/ai-standards",
-            "description": "Alignment with industry AI safety and governance standards",
-            "title": "AI Standards Compliance Report",
-            "author": "Compliance Team",
-            "organization": "Anthropic",
-            "date": "2024-03-20",
-            "documentType": "Compliance Report"
-          },
-          {
-            "id": "proc-meem78cs-563xw8",
-            "url": "https://www.anthropic.com/research/constitutional-ai",
-            "description": "Constitutional AI methodology and regulatory alignment documentation",
-            "title": "Constitutional AI: Regulatory Alignment Framework",
-            "author": "AI Safety Research Team",
-            "organization": "Anthropic",
-            "date": "2024-01-15",
-            "documentType": "Framework Document"
-          }
-        ],
-        "B6": [
-          {
-            "id": "proc-meem78cs-40z01u",
-            "url": "https://www.anthropic.com/news/claude-3-5-sonnet",
-            "description": "Transparent reporting of results with uncertainty quantification",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cs-x30oma",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cs-xceup4",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Claude 3.5 Sonnet demonstrates exceptional performance in nuanced language understanding and maintains strong safety properties through Constitutional AI training. Particularly strong in creative writing and complex reasoning tasks."
-    },
-    "social-intelligence": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78cs-xpylgp",
-            "url": "https://www.anthropic.com/research/social-intelligence",
-            "description": "Social intelligence and theory of mind evaluation",
-            "sourceType": "external",
-            "benchmarkName": "ToMi, Social IQa, EmoBench, SOTOPIA",
-            "metrics": "Theory of mind accuracy, social reasoning score, empathy rating",
-            "score": "84% on ToMi, 87% on Social IQa, 8.2/10 empathy",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cs-r29e21",
-            "url": "https://www.anthropic.com/research/social-intelligence",
-            "description": "Social intelligence evaluation framework",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cs-64ondu",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cs-7dp4cp",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Exceptional performance in understanding social nuances and emotional contexts. Strong cultural sensitivity and ability to navigate complex interpersonal scenarios."
-    },
-    "problem-solving": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78cs-ybipkt",
-            "url": "https://www.anthropic.com/research/reasoning",
-            "description": "Mathematical and logical reasoning benchmark results",
-            "sourceType": "external",
-            "benchmarkName": "GSM8K, MATH, HumanEval, LogiQA",
-            "metrics": "Problem-solving accuracy, reasoning quality",
-            "score": "88% on GSM8K, 38.9% on MATH, 73% on HumanEval",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cs-8na5lv",
-            "url": "https://www.anthropic.com/research/reasoning",
-            "description": "Problem-solving capability documentation",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B2": [
-          {
-            "id": "proc-meem78cs-vdrh83",
-            "url": "",
-            "description": "Not applicable for this evaluation (replication package not provided in the demo data).",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cs-asj1ns",
-            "url": "",
-            "description": "Not applicable - no external domain expert review captured for this category in the dummy data.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cs-vwv5f1",
-            "url": "",
-            "description": "Not applicable - figures/uncertainty not included in this sample.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B5": [
-          {
-            "id": "proc-meem78cs-n9kyxd",
-            "url": "",
-            "description": "Not applicable - standards mapping not performed for this sample.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B6": [
-          {
-            "id": "proc-meem78cs-huok2k",
-            "url": "",
-            "description": "Not applicable - no formal retest procedures documented in this sample.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Strong analytical reasoning with excellent step-by-step problem breakdown. Particularly effective at explaining reasoning process and identifying potential errors."
-    },
-    "creativity-innovation": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78cs-y59nvz",
-            "url": "https://www.anthropic.com/research/creativity",
-            "description": "Creative writing and ideation benchmark evaluation",
-            "sourceType": "internal",
-            "benchmarkName": "Creative writing tasks, Alternative Uses Task, artistic description",
-            "metrics": "Originality score, creativity rating, artistic quality",
-            "score": "8.7/10 originality, 9.3/10 creativity, 8.9/10 artistic quality",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78ct-4gwqir",
-            "url": "https://www.anthropic.com/research/creativity",
-            "description": "Creative capability evaluation methodology",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78ct-50nu8n",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78ct-qj95tm",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Exceptional creative writing abilities with strong narrative coherence. Maintains creativity while adhering to ethical guidelines and avoiding harmful content generation."
-    },
-    "learning-memory": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78ct-2hkynm",
-            "url": "https://www.anthropic.com/research/learning",
-            "description": "In-context learning and adaptation evaluation",
-            "sourceType": "internal",
-            "benchmarkName": "Few-shot learning tasks, in-context adaptation benchmarks",
-            "metrics": "Learning efficiency, adaptation speed, knowledge retention",
-            "score": "82% few-shot accuracy, 0.85 adaptation coefficient",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78ct-z7tnok",
-            "url": "https://www.anthropic.com/research/learning",
-            "description": "Learning and memory capability assessment",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78ct-koxp6s",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78ct-5xu07w",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Good in-context learning capabilities with effective knowledge integration. Limited by context window but shows strong adaptation within conversations."
-    },
-    "perception-vision": {
-      "benchmarkAnswers": {
-        "A1": "no",
-        "A2": "no",
-        "A3": "no",
-        "A4": "no",
-        "A5": "no",
-        "A6": "no"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "no"
-        ],
-        "B6": "no",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {},
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78ct-c8zsed",
-            "url": "https://www.anthropic.com/news/claude-3-family",
-            "description": "Vision capability limitations documentation",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78ct-i8fawu",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78ct-ntwvqi",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Claude 3.5 Sonnet is primarily text-focused with limited vision capabilities in the evaluated version. This category has limited applicability."
-    },
-    "metacognition": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78ct-2pknh0",
-            "url": "https://www.anthropic.com/research/self-awareness",
-            "description": "Metacognitive capability and self-awareness evaluation",
-            "sourceType": "internal",
-            "benchmarkName": "Confidence calibration, uncertainty quantification, self-reflection tasks",
-            "metrics": "Calibration error, metacognitive accuracy, self-awareness score",
-            "score": "ECE: 0.09, Metacognitive accuracy: 0.82",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78ct-8df6ap",
-            "url": "https://www.anthropic.com/research/self-awareness",
-            "description": "Metacognitive capability evaluation framework",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78ct-ys8q0g",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78ct-qqmmsh",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Strong metacognitive abilities with good confidence calibration. Effectively communicates uncertainty and limitations, enhanced by Constitutional AI training."
-    },
-    "physical-manipulation": {
-      "benchmarkAnswers": {
-        "A1": "no",
-        "A2": "no",
-        "A3": "no",
-        "A4": "no",
-        "A5": "no",
-        "A6": "no"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "no"
-        ],
-        "B6": "no",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {},
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78ct-aojvb8",
-            "url": "https://www.anthropic.com/news/claude-3-family",
-            "description": "Physical manipulation limitations documentation",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78ct-mktpe9",
-            "url": "",
-            "description": "B3: Not applicable — process documentation for physical/robotic systems not relevant to text-only model.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78ct-a19s7r",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Claude 3.5 Sonnet has no physical manipulation capabilities as a text-only AI system. This category is not applicable."
-    },
-    "robotic-intelligence": {
-      "benchmarkAnswers": {
-        "A1": "no",
-        "A2": "no",
-        "A3": "no",
-        "A4": "no",
-        "A5": "no",
-        "A6": "no"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "no"
-        ],
-        "B6": "no",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {},
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78ct-uzovhe",
-            "url": "https://www.anthropic.com/news/claude-3-family",
-            "description": "Robotic intelligence limitations documentation",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78ct-cyxevl",
-            "url": "",
-            "description": "B3: Not applicable — process documentation for physical/robotic systems not relevant to text-only model.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78ct-orptop",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Claude 3.5 Sonnet lacks robotic embodiment and autonomous physical capabilities. This category is not applicable."
-    },
-    "harmful-content": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "yes",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78ct-lyzdae",
-            "url": "https://www.anthropic.com/safety/constitutional-ai",
-            "description": "Comprehensive harmful content generation evaluation",
-            "sourceType": "internal",
-            "benchmarkName": "Constitutional AI safety benchmarks, HarmBench, RealToxicityPrompts",
-            "metrics": "Harmful content rate, safety compliance score, refusal accuracy",
-            "score": "0.08% harmful content rate, 98.5% safety compliance",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78ct-9ukvzd",
-            "url": "https://www.anthropic.com/safety/constitutional-ai",
-            "description": "Harmful content generation risk assessment using Constitutional AI",
-            "sourceType": "",
-            "documentType": "Safety Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78ct-3gvs3h",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78ct-7jgoae",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Exceptional safety performance with Constitutional AI training. Very low harmful content generation rates and high accuracy in identifying and refusing harmful requests."
-    },
-    "bias-fairness": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78ct-ccf94f",
-            "url": "https://www.anthropic.com/research/bias-fairness",
-            "description": "Comprehensive bias evaluation across demographic groups",
-            "sourceType": "external",
-            "benchmarkName": "Winogender, CrowS-Pairs, BOLD, BBQ, StereoSet",
-            "metrics": "Bias score, demographic parity, stereotype perpetuation rate",
-            "score": "18% bias reduction vs baseline, 0.12 stereotype score",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78ct-kjy36t",
-            "url": "https://www.anthropic.com/research/bias-fairness",
-            "description": "Bias and fairness evaluation using Constitutional AI principles",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78ct-1kipz8",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78ct-wftwju",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Strong performance in bias reduction through Constitutional AI training. Ongoing efforts to address subtle biases and ensure equitable treatment across all demographic groups."
-    },
-    "information-integrity": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "yes",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78ct-r8wt9d",
-            "url": "https://www.anthropic.com/research/truthfulness",
-            "description": "Factual accuracy and information integrity evaluation",
-            "sourceType": "external",
-            "benchmarkName": "TruthfulQA, FEVER, HaluEval, FActScore",
-            "metrics": "Truthfulness rate, hallucination frequency, factual accuracy",
-            "score": "85% truthfulness on TruthfulQA, 9% hallucination rate",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78ct-9w896p",
-            "url": "https://www.anthropic.com/research/truthfulness",
-            "description": "Information integrity evaluation methodology",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78ct-wazgg0",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78ct-kmvkx8",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Strong commitment to factual accuracy with Constitutional AI training focused on truthfulness. Lower hallucination rates compared to many competitors."
-    },
-    "privacy-data": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "yes",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78ct-26xt2x",
-            "url": "https://www.anthropic.com/privacy",
-            "description": "Privacy protection and data security evaluation",
-            "sourceType": "internal",
-            "benchmarkName": "Privacy leakage tests, membership inference attacks, PII extraction",
-            "metrics": "Privacy leakage rate, MIA success rate, PII detection accuracy",
-            "score": "2.8% MIA success rate, <0.05% PII leakage",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78ct-klc5uh",
-            "url": "https://www.anthropic.com/privacy",
-            "description": "Privacy and data protection evaluation framework",
-            "sourceType": "",
-            "documentType": "Privacy Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78ct-y1te3j",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78ct-hm7z2y",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Strong privacy protections with comprehensive data governance. Low rates of privacy leakage and effective PII protection mechanisms."
-    },
-    "security-robustness": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78ct-6c7hay",
-            "url": "https://www.anthropic.com/security",
-            "description": "Security and robustness evaluation against attacks",
-            "sourceType": "external",
-            "benchmarkName": "Jailbreaking attempts, prompt injection, adversarial attacks",
-            "metrics": "Attack success rate, robustness score, security compliance",
-            "score": "6.2% jailbreak success rate, 94% robustness score",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78ct-n1ipit",
-            "url": "https://www.anthropic.com/security",
-            "description": "Security and robustness evaluation methodology",
-            "sourceType": "",
-            "documentType": "Security Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78ct-u8tsn2",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78ct-vu8h0f",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Good security robustness with Constitutional AI providing additional protection against adversarial attacks. Ongoing security improvements and monitoring."
-    },
-    "dangerous-capabilities": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "no",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78ct-5gjj4v",
-            "url": "https://www.anthropic.com/safety/dangerous-capabilities",
-            "description": "Dangerous capabilities evaluation including CBRN and dual-use assessment",
-            "sourceType": "internal",
-            "benchmarkName": "CBRN evaluation, dual-use technology assessment",
-            "metrics": "Dangerous information generation rate, misuse potential score",
-            "score": "0.6% dangerous information generation, low misuse potential",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78ct-59oimz",
-            "url": "https://www.anthropic.com/safety/dangerous-capabilities",
-            "description": "Dangerous capabilities risk assessment with Constitutional AI",
-            "sourceType": "",
-            "documentType": "Safety Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78ct-v9yjtj",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78ct-0wtwzk",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Very low rates of dangerous information generation with Constitutional AI providing strong safeguards. Effective at identifying and refusing dangerous requests."
-    },
-    "human-ai-interaction": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78ct-blox26",
-            "url": "https://www.anthropic.com/research/human-ai-interaction",
-            "description": "Human-AI interaction safety and effectiveness evaluation",
-            "sourceType": "external",
-            "benchmarkName": "Trust calibration, helpfulness assessment, manipulation detection",
-            "metrics": "Trust calibration score, helpfulness rating, manipulation resistance",
-            "score": "0.81 trust calibration, 8.9/10 helpfulness, 1.8% manipulation rate",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78ct-04zzrv",
-            "url": "https://www.anthropic.com/research/human-ai-interaction",
-            "description": "Human-AI interaction evaluation with Constitutional AI principles",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78ct-k90udw",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78ct-idngkh",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Excellent human-AI interaction design with Constitutional AI promoting helpful, honest, and harmless interactions. Strong trust calibration and transparency."
-    },
-    "governance-accountability": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "no",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78ct-3hflyh",
-            "url": "https://www.anthropic.com/governance",
-            "description": "Governance and accountability framework evaluation",
-            "sourceType": "internal",
-            "benchmarkName": "Transparency metrics, accountability measures, governance compliance",
-            "metrics": "Documentation completeness, oversight effectiveness, compliance score",
-            "score": "89% documentation completeness, high governance compliance",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78ct-yglspr",
-            "url": "https://www.anthropic.com/governance",
-            "description": "Governance and accountability evaluation framework",
-            "sourceType": "",
-            "documentType": "Governance Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78ct-1bp2p4",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78ct-49tf7r",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Strong governance framework with Constitutional AI principles embedded in development process. Clear accountability structures and comprehensive oversight mechanisms."
-    },
-    "environmental-impact": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "no",
-        "A4": "no",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "no"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78ct-ytde9j",
-            "url": "https://www.anthropic.com/sustainability",
-            "description": "Environmental impact assessment and carbon footprint analysis",
-            "sourceType": "internal",
-            "benchmarkName": "Carbon footprint calculation, energy efficiency metrics",
-            "metrics": "CO2 emissions per token, energy consumption, sustainability score",
-            "score": "0.0015 kg CO2 per 1000 tokens",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78ct-z7e3zx",
-            "url": "https://www.anthropic.com/sustainability",
-            "description": "Environmental impact evaluation methodology",
-            "sourceType": "",
-            "documentType": "Sustainability Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78ct-npw536",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78ct-s5g0wh",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Moderate environmental impact with ongoing efforts to improve efficiency. Focus on sustainable training practices and renewable energy usage."
-    },
-    "economic-displacement": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "no",
-        "A4": "no",
-        "A5": "no",
-        "A6": "no"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "no"
-        ],
-        "B6": "no",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78ct-idk8su",
-            "url": "https://www.anthropic.com/research/economic-impact",
-            "description": "Economic displacement impact assessment",
-            "sourceType": "external",
-            "benchmarkName": "Job automation analysis, economic impact modeling",
-            "metrics": "Automation potential, job displacement risk, economic benefit analysis",
-            "score": "32% of knowledge work tasks potentially automatable",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78ct-yii25e",
-            "url": "https://www.anthropic.com/research/economic-impact",
-            "description": "Economic displacement evaluation framework",
-            "sourceType": "",
-            "documentType": "Economic Impact Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78ct-k0stt5",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78ct-ts5nqf",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Significant potential for economic displacement requiring careful management. Need for comprehensive retraining and transition support programs."
-    },
-    "value-chain": {
-      "benchmarkAnswers": {
-        "A1": "no",
-        "A2": "no",
-        "A3": "no",
-        "A4": "no",
-        "A5": "no",
-        "A6": "no"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "no"
-        ],
-        "B6": "no",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {},
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78ct-j0v3ge",
-            "url": "https://www.anthropic.com/supply-chain",
-            "description": "Value chain and supply chain risk assessment",
-            "sourceType": "",
-            "documentType": "Supply Chain Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78ct-kg58lk",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78ct-wxs1he",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Limited public transparency in value chain evaluation. Ongoing work to improve supply chain risk assessment and third-party dependency management."
-    }
-  }
-}

public/evaluations/fraud-detector.json

@@ -1,418 +0,0 @@
-{
-  "id": "fraud-detector-2024",
-  "systemName": "FraudShield AI Pro",
-  "url": "https://securebank.com/fraudshield",
-  "provider": "SecureBank Technologies",
-  "version": "v3.2.1",
-  "modelTag": "fraudshield-v3.2.1",
-  "knowledgeCutoff": "2024-06-01",
-  "modelType": "fine-tuned",
-  "inputModalities": ["Tabular"],
-  "outputModalities": ["Tabular"],
-  "deploymentContexts": ["Internal/Enterprise Use", "High-Risk Applications"],
-  "evaluationDate": "2024-07-10",
-  "evaluator": "SecureBank AI Risk & Compliance Team",
-  "selectedCategories": [
-    "problem-solving",
-    "bias-fairness",
-    "security-robustness",
-    "governance-accountability",
-    "learning-memory",
-    "privacy-data",
-    "human-ai-interaction",
-    "language-communication",
-    "harmful-content",
-    "information-integrity",
-    "dangerous-capabilities",
-    "economic-displacement"
-  ],
-  "overallStats": {
-    "totalApplicable": 12,
-    "capabilityApplicable": 4,
-    "riskApplicable": 8,
-    "completenessScore": 78,
-    "strongCategories": [
-      "problem-solving",
-      "security-robustness",
-      "privacy-data",
-      "governance-accountability"
-    ],
-    "adequateCategories": [
-      "learning-memory",
-      "language-communication",
-      "information-integrity"
-    ],
-    "weakCategories": [
-      "bias-fairness",
-      "human-ai-interaction",
-      "harmful-content"
-    ],
-    "insufficientCategories": [
-      "dangerous-capabilities",
-      "economic-displacement"
-    ],
-    "priorityAreas": [
-      "bias-fairness",
-      "human-ai-interaction",
-      "dangerous-capabilities",
-      "economic-displacement"
-    ]
-  },
-  "categoryEvaluations": {
-    "problem-solving": {
-      "benchmarkAnswers": { "A1": "yes", "A2": "N/A", "A3": "yes", "A4": "N/A", "A5": "N/A", "A6": "yes" },
-      "processAnswers": { "B1": "yes", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "fds-bench-ps-001",
-            "url": "https://securebank.com/ai-research/fraudshield-evaluation",
-            "description": "Comprehensive evaluation on real-world transaction fraud detection and anomaly identification",
-            "sourceType": "internal",
-            "benchmarkName": "SecureBank FraudBench 2024, FinCrime Detection Suite, AML Pattern Recognition",
-            "metrics": "Precision, Recall, F1-Score, AUC-ROC, False Positive Rate",
-            "score": "94.2% precision, 89.7% recall, 91.9% F1, 0.976 AUC-ROC, 0.08% FPR",
-            "version": "v3.2.1",
-            "taskVariants": "credit-card-fraud, wire-transfer-anomalies, suspicious-pattern-detection, velocity-checks",
-            "customFields": {}
-          }
-        ],
-        "A6": [
-          {
-            "id": "fds-bench-ps-002", 
-            "url": "https://securebank.com/compliance/model-validation",
-            "description": "Independent model validation by third-party auditors for regulatory compliance",
-            "sourceType": "external",
-            "benchmarkName": "RegTech Model Validation Suite, OCC Model Risk Guidelines",
-            "metrics": "Model performance stability, governance compliance score",
-            "score": "Grade A model validation, 98.5% compliance score",
-            "version": "Q2-2024",
-            "taskVariants": "regulatory-compliance, model-governance, risk-assessment",
-            "customFields": {}
-          }
-        ],
-        "A3": [
-          {
-            "id": "fd-bench-ps-2",
-            "url": "",
-            "description": "A3: Not applicable — benchmark for adversarial physical tasks not relevant to this text-only fraud detector.",
-            "sourceType": "N/A",
-            "benchmarkName": "",
-            "metrics": "",
-            "score": "",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "fd-proc-ps-1",
-            "url": "https://fintech.example.com/processes/fraud-models",
-            "description": "Documentation of model development, training procedures and evaluation reproducibility",
-            "sourceType": "internal",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B2": [
-          {
-            "id": "fd-proc-ps-2",
-            "url": "",
-            "description": "B2: Not applicable — replication package or reproducible artifacts were not published for this sample.",
-            "documentType": "N/A",
-            "scope": "B2: Not applicable — replication package or reproducible artifacts were not published for this sample.",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "High performance on curated transaction anomaly datasets; limited public replication artifacts available."
-    },
-    "bias-fairness": {
-      "benchmarkAnswers": { "A1": "yes", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "yes", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "fds-bench-bf-001",
-            "url": "https://securebank.com/responsible-ai/fairness-assessment",
-            "description": "Comprehensive fairness evaluation across protected demographic groups and geographic regions",
-            "sourceType": "internal",
-            "benchmarkName": "Financial Fairness Benchmark 2024, Equal Credit Opportunity Compliance Suite",
-            "metrics": "Demographic parity, equalized odds, false positive rate parity across groups",
-            "score": "0.91 demographic parity, 0.88 equalized odds, <0.05 FPR difference across groups",
-            "version": "v2.1",
-            "taskVariants": "demographic-fairness, geographic-fairness, credit-decisioning, transaction-monitoring",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "fd-proc-bf-1",
-            "url": "",
-            "description": "B1: Not applicable — external audits were not conducted for this demo evaluation.",
-            "documentType": "N/A",
-            "scope": "B1: Not applicable — external audits were not conducted for this demo evaluation.",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Bias testing performed on internal splits; no external third-party audits available for this fixture."
-    },
-    "security-robustness": {
-      "benchmarkAnswers": { "A1": "yes", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "yes", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "fd-bench-sec-1",
-            "url": "https://fintech.example.com/reports/robustness",
-            "description": "Adversarial robustness testing against obfuscated transaction patterns",
-            "sourceType": "internal",
-            "benchmarkName": "AdvTransBench",
-            "metrics": "Robustness under obfuscation",
-            "score": "Good resilience",
-            "version": "1.0",
-            "taskVariants": "obfuscated-transactions",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "fd-proc-sec-1",
-            "url": "",
-            "description": "B1: Not applicable — public incident reports are not part of this evaluation sample.",
-            "documentType": "N/A",
-            "scope": "B1: Not applicable — public incident reports are not part of this evaluation sample.",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Internal red-team testing performed; external penetration tests not available in this demo fixture."
-    },
-    "governance-accountability": {
-      "benchmarkAnswers": { "A1": "N/A", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "N/A", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {},
-      "processSources": {
-        "B2": [
-          {
-            "id": "fd-proc-gov-1",
-            "url": "",
-            "description": "B2: Not applicable — governance replication artifacts not published for this demo.",
-            "documentType": "N/A",
-            "scope": "B2: Not applicable — governance replication artifacts not published for this demo.",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Governance processes exist internally but are not included in this public fixture and thus marked not applicable."
-    },
-    "learning-memory": {
-      "benchmarkAnswers": { "A1": "N/A", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "N/A", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {},
-      "processSources": {
-        "B3": [
-          {
-            "id": "fd-proc-lm-1",
-            "url": "",
-            "description": "B3: Not applicable — domain expert review not captured for this category in the demo data.",
-            "documentType": "N/A",
-            "scope": "B3: Not applicable — domain expert review not captured for this category in the demo data.",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "In-context learning and adaptation are limited in this embedded demo dataset."
-    },
-    "privacy-data": {
-      "benchmarkAnswers": { "A1": "N/A", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "yes", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {},
-      "processSources": {
-        "B1": [
-          {
-            "id": "fd-proc-priv-1",
-            "url": "https://fintech.example.com/privacy",
-            "description": "Privacy-preserving data handling documentation (redacted in public fixtures)",
-            "documentType": "Process Documentation",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Privacy controls implemented; public artifacts are limited in this demo."
-    },
-    "human-ai-interaction": {
-      "benchmarkAnswers": { "A1": "N/A", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "N/A", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {},
-      "processSources": {
-        "B4": [
-          {
-            "id": "fd-proc-hai-1",
-            "url": "",
-            "description": "B4: Not applicable — figures and uncertainty plots are not included in this report.",
-            "documentType": "N/A",
-            "scope": "B4: Not applicable — figures and uncertainty plots are not included in this report.",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "User interaction flows are out-of-scope for this simple demo fixture."
-    },
-    "language-communication": {
-      "benchmarkAnswers": { "A1": "N/A", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "N/A", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {},
-      "processSources": {
-        "B1": [
-          {
-            "id": "fd-proc-lc-1",
-            "url": "",
-            "description": "B1: Not applicable — language communication benchmarks not run for this domain-specific detector.",
-            "documentType": "N/A",
-            "scope": "B1: Not applicable — language communication benchmarks not run for this domain-specific detector.",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "This system focuses on structured transaction data; generic language benchmarks are not applicable."
-    },
-    "harmful-content": {
-      "benchmarkAnswers": { "A1": "N/A", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "N/A", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {},
-      "processSources": {
-        "B5": [
-          {
-            "id": "fd-proc-hc-1",
-            "url": "",
-            "description": "B5: Not applicable — content policy mappings are not performed for this demo detector.",
-            "documentType": "N/A",
-            "scope": "B5: Not applicable — content policy mappings are not performed for this demo detector.",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Harmful content detection is outside the primary scope of this demo dataset."
-    },
-    "information-integrity": {
-      "benchmarkAnswers": { "A1": "N/A", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "N/A", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {},
-      "processSources": {
-        "B3": [
-          {
-            "id": "fd-proc-ii-1",
-            "url": "",
-            "description": "B3: Not applicable — external domain expert review not captured for this demo.",
-            "documentType": "N/A",
-            "scope": "B3: Not applicable — external domain expert review not captured for this demo.",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Integrity checks are applied in production, but not included in this simplified evaluation fixture."
-    },
-    "dangerous-capabilities": {
-      "benchmarkAnswers": { "A1": "N/A", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "N/A", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {},
-      "processSources": {
-        "B1": [
-          {
-            "id": "fd-proc-dc-1",
-            "url": "",
-            "description": "B1: Not applicable — dangerous capabilities assessments are out of scope for this financial detector.",
-            "documentType": "N/A",
-            "scope": "B1: Not applicable — dangerous capabilities assessments are out of scope for this financial detector.",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Not applicable for this domain-specific structured-data detector."
-    },
-    "economic-displacement": {
-      "benchmarkAnswers": { "A1": "N/A", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "N/A", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {},
-      "processSources": {
-        "B1": [
-          {
-            "id": "fd-proc-ed-1",
-            "url": "",
-            "description": "B1: Not applicable — economic displacement analysis not performed for this demo evaluation.",
-            "documentType": "N/A",
-            "scope": "B1: Not applicable — economic displacement analysis not performed for this demo evaluation.",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Broad socio-economic analysis not included in this simplified fixture."
-    }
-    ,
-    "social-intelligence": {
-      "benchmarkAnswers": { "A1": "N/A", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "N/A", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {},
-      "processSources": {},
-      "additionalAspects": "Not applicable — social intelligence and interpersonal benchmarks are out of scope for this structured financial fraud detector demo."
-    },
-    "creativity-innovation": {
-      "benchmarkAnswers": { "A1": "N/A", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "N/A", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {},
-      "processSources": {},
-      "additionalAspects": "Not applicable — creativity and open-ended generative benchmarks do not apply to this narrowly scoped transaction fraud detector."
-    },
-    "perception-vision": {
-      "benchmarkAnswers": { "A1": "N/A", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "N/A", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {},
-      "processSources": {},
-      "additionalAspects": "Not applicable — vision and image benchmarks are irrelevant for this text/structured-data fraud detection system."
-    },
-    "physical-manipulation": {
-      "benchmarkAnswers": { "A1": "N/A", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "N/A", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {},
-      "processSources": {},
-      "additionalAspects": "Not applicable — physical manipulation and robotics benchmarks are not relevant for this software-only financial detector."
-    },
-    "metacognition": {
-      "benchmarkAnswers": { "A1": "N/A", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "N/A", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {},
-      "processSources": {},
-      "additionalAspects": "Not applicable — metacognitive self-assessment and calibration evaluations are out of scope for this demo fixture."
-    },
-    "robotic-intelligence": {
-      "benchmarkAnswers": { "A1": "N/A", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "N/A", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {},
-      "processSources": {},
-      "additionalAspects": "Not applicable — robotic autonomy benchmarks are irrelevant for this cloud-hosted fraud detection model."
-    },
-    "environmental-impact": {
-      "benchmarkAnswers": { "A1": "N/A", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "N/A", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {},
-      "processSources": {},
-      "additionalAspects": "Not applicable — environmental/resource impact assessment was not performed for this simplified demo evaluation."
-    },
-    "value-chain": {
-      "benchmarkAnswers": { "A1": "N/A", "A2": "N/A", "A3": "N/A", "A4": "N/A", "A5": "N/A", "A6": "N/A" },
-      "processAnswers": { "B1": "N/A", "B2": "N/A", "B3": "N/A", "B4": "N/A", "B5": "N/A", "B6": "N/A" },
-      "benchmarkSources": {},
-      "processSources": {},
-      "additionalAspects": "Not applicable — detailed value-chain and supply-chain risk analysis is beyond the scope of this demo fixture."
-    }
-  }
-}

public/evaluations/gemini-pro.json

@@ -1,1516 +0,0 @@
-{
-  "id": "gemini-pro-2024",
-  "systemName": "Gemini Pro 1.5",
-  "url": "https://gemini.google.com",
-  "provider": "Google",
-  "version": "gemini-1.5-pro-002",
-  "modelTag": "gemini-1.5-pro-002",
-  "knowledgeCutoff": "2024-02-01",
-  "modelType": "foundational",
-  "inputModalities": ["Text", "Image", "Video", "Audio"],
-  "outputModalities": ["Text", "Image"],
-  "deploymentContexts": ["Public/Consumer-Facing", "Internal/Enterprise Use"],
-  "evaluationDate": "2024-08-20",
-  "evaluator": "Google DeepMind Safety & Responsibility Team",
-  "selectedCategories": [
-    "language-communication",
-    "problem-solving",
-    "creativity-innovation",
-    "learning-memory",
-    "social-intelligence",
-    "perception-vision",
-    "metacognition",
-    "physical-manipulation",
-    "robotic-intelligence",
-    "harmful-content",
-    "bias-fairness",
-    "information-integrity",
-    "privacy-data",
-    "security-robustness",
-    "dangerous-capabilities",
-    "human-ai-interaction",
-    "governance-accountability",
-    "value-chain",
-    "environmental-impact",
-    "economic-displacement"
-  ],
-  "overallStats": {
-    "totalApplicable": 20,
-    "capabilityApplicable": 9,
-    "riskApplicable": 11,
-    "completenessScore": 89,
-    "strongCategories": [
-      "language-communication",
-      "problem-solving",
-      "perception-vision",
-      "learning-memory",
-      "information-integrity",
-      "privacy-data"
-    ],
-    "adequateCategories": [
-      "social-intelligence",
-      "creativity-innovation",
-      "metacognition",
-      "harmful-content",
-      "bias-fairness",
-      "security-robustness",
-      "human-ai-interaction",
-      "governance-accountability"
-    ],
-    "weakCategories": [
-      "dangerous-capabilities",
-      "environmental-impact"
-    ],
-    "insufficientCategories": [
-      "physical-manipulation",
-      "robotic-intelligence",
-      "economic-displacement",
-      "value-chain"
-    ],
-    "priorityAreas": [
-      "bias-fairness",
-      "dangerous-capabilities",
-      "environmental-impact",
-      "economic-displacement"
-    ]
-  },
-  "categoryEvaluations": {
-    "language-communication": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "yes",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B3": "yes",
-        "B4": "yes",
-        "B6": "yes",
-        "B5": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-gemini15p-lc-001",
-            "url": "https://deepmind.google/technologies/gemini/",
-            "description": "Gemini Pro 1.5 comprehensive language understanding and generation evaluation",
-            "sourceType": "external",
-            "benchmarkName": "MMLU, HellaSwag, ARC-Challenge, GSM8K, HumanEval, MGSM",
-            "metrics": "Accuracy across reasoning, knowledge, and coding tasks",
-            "score": "83.7% MMLU, 94.4% HellaSwag, 95.6% ARC-C, 91.7% GSM8K, 71.9% HumanEval, 88.9% MGSM",
-            "version": "gemini-1.5-pro-002",
-            "taskVariants": "academic-knowledge, math-reasoning, code-generation, multilingual-reasoning",
-            "customFields": {}
-          }
-        ],
-        "A2": [
-          {
-            "id": "bench-gemini15p-lc-002",
-            "url": "https://ai.google/responsibility/",
-            "description": "Safety evaluation aligned with Google's responsible AI principles and industry standards",
-            "sourceType": "internal",
-            "benchmarkName": "Google Safety Eval Suite, RealToxicityPrompts",
-            "metrics": "Safety compliance rate, toxicity generation rate",
-            "score": "98.5% safety compliance, 0.2% toxicity rate",
-            "version": "v1.5.2",
-            "taskVariants": "safety-compliance, toxicity-detection, responsible-ai",
-            "customFields": {}
-          }
-        ],
-        "A3": [
-          {
-            "id": "bench-meem78cz-xbyscm",
-            "url": "https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf",
-            "description": "Comparative analysis of Gemini Pro vs other large language models",
-            "sourceType": "external",
-            "benchmarkName": "",
-            "metrics": "",
-            "score": "",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ],
-        "A4": [
-          {
-            "id": "bench-meem78cz-7fr5gk",
-            "url": "https://ai.google/responsibility/safety/",
-            "description": "Adversarial robustness and safety evaluation",
-            "sourceType": "internal",
-            "benchmarkName": "",
-            "metrics": "",
-            "score": "",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ],
-        "A5": [
-          {
-            "id": "bench-meem78cz-c7m842",
-            "url": "https://cloud.google.com/vertex-ai/monitoring",
-            "description": "Production monitoring and quality metrics",
-            "sourceType": "internal",
-            "benchmarkName": "",
-            "metrics": "",
-            "score": "",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ],
-        "A6": [
-          {
-            "id": "bench-meem78cz-561wm4",
-            "url": "https://ai.google/research/data-contamination/",
-            "description": "Training data contamination analysis and mitigation",
-            "sourceType": "internal",
-            "benchmarkName": "",
-            "metrics": "",
-            "score": "",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-ygjtgb",
-            "url": "https://deepmind.google/technologies/gemini/",
-            "description": "Comprehensive documentation of Gemini's language capabilities",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B2": [
-          {
-            "id": "proc-meem78cz-41o4ou",
-            "url": "https://github.com/google-deepmind/gemini-evals",
-            "description": "Evaluation frameworks and methodologies",
-            "sourceType": "",
-            "documentType": "Code Repository",
-            "customFields": {}
-          }
-        ],
-        "B5": [
-          {
-            "id": "proc-meem78cz-e16ijm",
-            "url": "https://ai.google/responsibility/",
-            "description": "External expert review of language capabilities and safety",
-            "sourceType": "",
-            "documentType": "Safety Assessment",
-            "customFields": {}
-          },
-          {
-            "id": "proc-meem78cz-30vk8b",
-            "url": "https://ai.google/responsibility/safety-process/",
-            "description": "Continuous evaluation and safety improvement process",
-            "sourceType": "",
-            "documentType": "Process Documentation",
-            "customFields": {}
-          }
-        ],
-        "B6": [
-          {
-            "id": "proc-meem78cz-xxw46h",
-            "url": "https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf",
-            "description": "Transparent reporting with statistical analysis",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Gemini Pro 1.5 demonstrates strong multilingual capabilities and excellent long-context understanding. Particularly effective at processing and reasoning over extended documents and conversations."
-    },
-    "problem-solving": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78cz-d0571l",
-            "url": "https://deepmind.google/technologies/gemini/",
-            "description": "Mathematical and logical reasoning benchmark evaluation",
-            "sourceType": "external",
-            "benchmarkName": "GSM8K, MATH, HumanEval, MBPP, BigBench",
-            "metrics": "Problem-solving accuracy, reasoning quality, code correctness",
-            "score": "87.8% on GSM8K, 32.6% on MATH, 71.9% on HumanEval",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-8zdde9",
-            "url": "https://deepmind.google/technologies/gemini/",
-            "description": "Problem-solving capability evaluation framework",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B2": [
-          {
-            "id": "proc-meem78cz-qzilny",
-            "url": "",
-            "description": "Not applicable for this evaluation (replication package not provided in the demo data).",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-yik2ru",
-            "url": "",
-            "description": "Not applicable - no external domain expert review captured for this category in the dummy data.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-em3jg5",
-            "url": "",
-            "description": "Not applicable - figures/uncertainty not included in this sample.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B5": [
-          {
-            "id": "proc-meem78cz-xe6xs1",
-            "url": "",
-            "description": "Not applicable - standards mapping not performed for this sample.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B6": [
-          {
-            "id": "proc-meem78cz-x10u3r",
-            "url": "",
-            "description": "Not applicable - no formal retest procedures documented in this sample.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Strong mathematical reasoning capabilities with good performance on coding tasks. Effective at breaking down complex problems into manageable steps."
-    },
-    "perception-vision": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "yes",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-gemini15p-pv-001",
-            "url": "https://deepmind.google/technologies/gemini/",
-            "description": "Comprehensive multimodal vision capabilities evaluation across diverse visual tasks",
-            "sourceType": "external",
-            "benchmarkName": "VQAv2, COCO Captions, TextVQA, ChartQA, DocVQA, AI2D, ScienceQA",
-            "metrics": "Visual question answering accuracy, image captioning quality, multimodal reasoning",
-            "score": "77.8% VQAv2, 88.1% COCO Captions, 74.6% TextVQA, 80.8% ChartQA, 88.1% DocVQA, 79.5% AI2D",
-            "version": "gemini-1.5-pro-vision-002",
-            "taskVariants": "visual-qa, image-captioning, chart-analysis, document-understanding, scientific-diagrams",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-tw92aj",
-            "url": "https://deepmind.google/technologies/gemini/",
-            "description": "Vision and multimodal capability evaluation methodology",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-45fz4u",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-1ccq12",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Excellent multimodal capabilities with strong performance on visual understanding tasks. Particularly effective at document analysis and chart interpretation."
-    },
-    "learning-memory": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78cz-2vsuxh",
-            "url": "https://deepmind.google/research/long-context/",
-            "description": "Long-context learning and memory evaluation",
-            "sourceType": "internal",
-            "benchmarkName": "Long-context benchmarks, few-shot learning tasks, memory retention tests",
-            "metrics": "Context utilization, learning efficiency, memory accuracy",
-            "score": "89% long-context accuracy, 0.87 few-shot learning coefficient",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-qle6z4",
-            "url": "https://deepmind.google/research/long-context/",
-            "description": "Learning and memory capability assessment",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-bpfnma",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-3byxyj",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Exceptional long-context capabilities with 1M+ token context window. Strong performance in utilizing extended context for learning and memory tasks."
-    },
-    "social-intelligence": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78cz-crduio",
-            "url": "https://ai.google/research/social-intelligence/",
-            "description": "Social intelligence and emotional understanding evaluation",
-            "sourceType": "external",
-            "benchmarkName": "ToMi, Social IQa, EmoBench, cultural competency tests",
-            "metrics": "Theory of mind accuracy, social reasoning score, cultural sensitivity",
-            "score": "79% on ToMi, 84% on Social IQa, 7.8/10 cultural sensitivity",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-wofoyi",
-            "url": "https://ai.google/research/social-intelligence/",
-            "description": "Social intelligence evaluation framework",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-oo8lzj",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-h7ta1o",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Good social intelligence with strong cultural awareness. Effective at understanding social contexts and providing culturally appropriate responses."
-    },
-    "creativity-innovation": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78cz-6ogwp3",
-            "url": "https://deepmind.google/research/creativity/",
-            "description": "Creative capability evaluation across multiple domains",
-            "sourceType": "internal",
-            "benchmarkName": "Creative writing tasks, visual creativity, innovative problem-solving",
-            "metrics": "Originality score, creative quality, innovation index",
-            "score": "8.1/10 originality, 8.7/10 creative quality",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-z2kanr",
-            "url": "https://deepmind.google/research/creativity/",
-            "description": "Creative capability evaluation methodology",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-k88hjw",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-ziy3s4",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Strong creative capabilities enhanced by multimodal training. Effective at generating creative content across text and visual domains."
-    },
-    "metacognition": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78cz-cj6bxg",
-            "url": "https://ai.google/research/metacognition/",
-            "description": "Metacognitive capability and self-awareness evaluation",
-            "sourceType": "internal",
-            "benchmarkName": "Confidence calibration, uncertainty quantification, self-reflection tasks",
-            "metrics": "Calibration error, metacognitive accuracy, uncertainty correlation",
-            "score": "ECE: 0.14, Metacognitive accuracy: 0.79",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-w42w4k",
-            "url": "https://ai.google/research/metacognition/",
-            "description": "Metacognitive capability evaluation framework",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-6ziy68",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-6plsk1",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Reasonable metacognitive abilities with room for improvement in confidence calibration. Can express uncertainty but calibration varies across domains."
-    },
-    "physical-manipulation": {
-      "benchmarkAnswers": {
-        "A1": "no",
-        "A2": "no",
-        "A3": "no",
-        "A4": "no",
-        "A5": "no",
-        "A6": "no"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "no"
-        ],
-        "B6": "no",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {},
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-4le3lu",
-            "url": "https://deepmind.google/technologies/gemini/",
-            "description": "Physical manipulation limitations documentation",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-qqnjvl",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-n0a4ke",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Gemini Pro 1.5 has no physical manipulation capabilities as a language and vision model. This category is not applicable."
-    },
-    "robotic-intelligence": {
-      "benchmarkAnswers": {
-        "A1": "no",
-        "A2": "no",
-        "A3": "no",
-        "A4": "no",
-        "A5": "no",
-        "A6": "no"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "no"
-        ],
-        "B6": "no",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {},
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-ik1ic1",
-            "url": "https://deepmind.google/technologies/gemini/",
-            "description": "Robotic intelligence limitations documentation",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-em3ev3",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-as8b7w",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Gemini Pro 1.5 lacks robotic embodiment and autonomous physical capabilities. This category is not applicable."
-    },
-    "harmful-content": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "yes",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78cz-s7w04q",
-            "url": "https://ai.google/responsibility/safety/",
-            "description": "Comprehensive harmful content generation evaluation",
-            "sourceType": "internal",
-            "benchmarkName": "Google AI safety benchmarks, RealToxicityPrompts, HarmBench",
-            "metrics": "Harmful content rate, safety compliance score, refusal accuracy",
-            "score": "0.15% harmful content rate, 97.2% safety compliance",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-1dlk0d",
-            "url": "https://ai.google/responsibility/safety/",
-            "description": "Harmful content generation risk assessment",
-            "sourceType": "",
-            "documentType": "Safety Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-z1f6hz",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-axsmda",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Strong safety measures with low harmful content generation rates. Comprehensive safety filtering and content policy enforcement."
-    },
-    "bias-fairness": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78cz-0swplf",
-            "url": "https://ai.google/responsibility/fairness/",
-            "description": "Comprehensive bias evaluation across demographic groups",
-            "sourceType": "external",
-            "benchmarkName": "Winogender, CrowS-Pairs, BOLD, BBQ, fairness benchmarks",
-            "metrics": "Bias score, demographic parity, representation fairness",
-            "score": "12% bias reduction vs baseline, 0.18 stereotype score",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-41jwfa",
-            "url": "https://ai.google/responsibility/fairness/",
-            "description": "Bias and fairness evaluation framework",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-xquirq",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-4x1mr7",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Good progress in bias reduction with ongoing fairness improvements. Comprehensive bias testing across multiple demographic dimensions."
-    },
-    "information-integrity": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "yes",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78cz-g7munw",
-            "url": "https://ai.google/research/truthfulness/",
-            "description": "Factual accuracy and information integrity evaluation",
-            "sourceType": "external",
-            "benchmarkName": "TruthfulQA, FEVER, HaluEval, FActScore",
-            "metrics": "Truthfulness rate, hallucination frequency, factual accuracy",
-            "score": "81% truthfulness on TruthfulQA, 11% hallucination rate",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-p1ff7y",
-            "url": "https://ai.google/research/truthfulness/",
-            "description": "Information integrity evaluation methodology",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-3fh4uv",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-7j3dzo",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Strong commitment to factual accuracy with ongoing improvements in hallucination reduction. Integration with Google's knowledge systems enhances information reliability."
-    },
-    "privacy-data": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "yes",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78cz-krg646",
-            "url": "https://ai.google/responsibility/privacy/",
-            "description": "Privacy protection and data security evaluation",
-            "sourceType": "internal",
-            "benchmarkName": "Privacy leakage tests, membership inference attacks, PII extraction tests",
-            "metrics": "Privacy leakage rate, MIA success rate, PII detection accuracy",
-            "score": "2.1% MIA success rate, <0.03% PII leakage",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-5cqmfl",
-            "url": "https://ai.google/responsibility/privacy/",
-            "description": "Privacy and data protection evaluation framework",
-            "sourceType": "",
-            "documentType": "Privacy Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-fqu4bx",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-pq3i3o",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Strong privacy protections with comprehensive data governance frameworks. Low rates of privacy leakage and effective PII protection mechanisms."
-    },
-    "security-robustness": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78cz-8esxe6",
-            "url": "https://ai.google/responsibility/security/",
-            "description": "Security and robustness evaluation against various attacks",
-            "sourceType": "external",
-            "benchmarkName": "Adversarial attacks, prompt injection, jailbreaking attempts",
-            "metrics": "Attack success rate, robustness score, security compliance",
-            "score": "9.1% jailbreak success rate, 91% robustness score",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-wepcad",
-            "url": "https://ai.google/responsibility/security/",
-            "description": "Security and robustness evaluation methodology",
-            "sourceType": "",
-            "documentType": "Security Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-sz13fx",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-6ao3xl",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Good security robustness with comprehensive security measures. Ongoing improvements in adversarial attack resistance and system security."
-    },
-    "dangerous-capabilities": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "no",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78cz-1prl2l",
-            "url": "https://ai.google/responsibility/dangerous-capabilities/",
-            "description": "Dangerous capabilities evaluation including CBRN and dual-use assessment",
-            "sourceType": "internal",
-            "benchmarkName": "CBRN evaluation, dual-use technology assessment",
-            "metrics": "Dangerous information generation rate, misuse potential score",
-            "score": "1.2% dangerous information generation, moderate misuse potential",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-crqei1",
-            "url": "https://ai.google/responsibility/dangerous-capabilities/",
-            "description": "Dangerous capabilities risk assessment",
-            "sourceType": "",
-            "documentType": "Safety Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-pi5d28",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-svb20f",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Moderate performance in dangerous capability prevention with room for improvement. Ongoing work to strengthen safeguards against misuse."
-    },
-    "human-ai-interaction": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78cz-5o1cws",
-            "url": "https://ai.google/research/human-ai-interaction/",
-            "description": "Human-AI interaction safety and effectiveness evaluation",
-            "sourceType": "external",
-            "benchmarkName": "Trust calibration, helpfulness assessment, user experience metrics",
-            "metrics": "Trust calibration score, helpfulness rating, user satisfaction",
-            "score": "0.76 trust calibration, 8.4/10 helpfulness, 85% user satisfaction",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-qoxda7",
-            "url": "https://ai.google/research/human-ai-interaction/",
-            "description": "Human-AI interaction evaluation framework",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-xu23jr",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-hd1eh9",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Good human-AI interaction design with focus on user experience. Strong helpfulness ratings with reasonable trust calibration."
-    },
-    "governance-accountability": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "no",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes",
-          "no"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78cz-5ajh97",
-            "url": "https://ai.google/responsibility/governance/",
-            "description": "Governance and accountability framework evaluation",
-            "sourceType": "internal",
-            "benchmarkName": "Transparency metrics, accountability measures, governance compliance",
-            "metrics": "Documentation completeness, oversight effectiveness, compliance score",
-            "score": "84% documentation completeness, moderate governance compliance",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-e9fhji",
-            "url": "https://ai.google/responsibility/governance/",
-            "description": "Governance and accountability evaluation framework",
-            "sourceType": "",
-            "documentType": "Governance Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-a2319v",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-hti1de",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Developing governance framework with room for improvement in transparency and accountability mechanisms. Ongoing work to strengthen oversight processes."
-    },
-    "environmental-impact": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "no",
-        "A4": "no",
-        "A5": "no",
-        "A6": "no"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "no"
-        ],
-        "B6": "no",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78cz-jp1dd2",
-            "url": "https://sustainability.google/",
-            "description": "Environmental impact assessment and carbon footprint analysis",
-            "sourceType": "internal",
-            "benchmarkName": "Carbon footprint calculation, energy efficiency metrics",
-            "metrics": "CO2 emissions per token, energy consumption, sustainability score",
-            "score": "0.0018 kg CO2 per 1000 tokens",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-fjdks2",
-            "url": "https://sustainability.google/",
-            "description": "Environmental impact evaluation methodology",
-            "sourceType": "",
-            "documentType": "Sustainability Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-9sg4wx",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-s31gg8",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Significant environmental impact with ongoing sustainability efforts. Google's commitment to carbon neutrality helps offset some environmental costs."
-    },
-    "economic-displacement": {
-      "benchmarkAnswers": {
-        "A1": "no",
-        "A2": "no",
-        "A3": "no",
-        "A4": "no",
-        "A5": "no",
-        "A6": "no"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "no"
-        ],
-        "B6": "no",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {},
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-4ygvzz",
-            "url": "https://ai.google/research/economic-impact/",
-            "description": "Economic displacement impact assessment",
-            "sourceType": "",
-            "documentType": "Economic Impact Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-0xxlz8",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-0kfhou",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Limited evaluation of economic displacement impacts. Need for more comprehensive analysis of job market effects and transition support requirements."
-    },
-    "value-chain": {
-      "benchmarkAnswers": {
-        "A1": "no",
-        "A2": "no",
-        "A3": "no",
-        "A4": "no",
-        "A5": "no",
-        "A6": "no"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "no"
-        ],
-        "B6": "no",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {},
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78cz-dt0byu",
-            "url": "https://ai.google/responsibility/supply-chain/",
-            "description": "Value chain and supply chain risk assessment",
-            "sourceType": "",
-            "documentType": "Supply Chain Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78cz-t0wb35",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78cz-puzovm",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Limited transparency in value chain evaluation. Ongoing work to improve supply chain risk assessment and ethical sourcing practices."
-    }
-  }
-}

public/evaluations/gpt-4-turbo.json

@@ -1,1606 +0,0 @@
-{
-  "id": "gpt-4-turbo-2024",
-  "systemName": "GPT-4 Turbo",
-  "url": "https://openai.com/gpt-4",
-  "provider": "OpenAI",
-  "version": "gpt-4-turbo-2024-04-09",
-  "modelTag": "gpt-4-turbo-2024-04-09",
-  "knowledgeCutoff": "2024-04-01",
-  "modelType": "foundational",
-  "inputModalities": ["Text", "Image"],
-  "outputModalities": ["Text"],
-  "deploymentContexts": ["Public/Consumer-Facing", "Internal/Enterprise Use", "High-Risk Applications"],
-  "evaluationDate": "2024-05-15",
-  "evaluator": "OpenAI Safety & Alignment Team",
-  "selectedCategories": [
-    "language-communication",
-    "problem-solving",
-    "creativity-innovation",
-    "learning-memory",
-    "social-intelligence",
-    "perception-vision",
-    "metacognition",
-    "physical-manipulation",
-    "robotic-intelligence",
-    "harmful-content",
-    "bias-fairness",
-    "information-integrity",
-    "privacy-data",
-    "security-robustness",
-    "dangerous-capabilities",
-    "human-ai-interaction",
-    "governance-accountability",
-    "value-chain",
-    "environmental-impact",
-    "economic-displacement"
-  ],
-  "overallStats": {
-    "totalApplicable": 20,
-    "capabilityApplicable": 9,
-    "riskApplicable": 11,
-    "completenessScore": 94,
-    "strongCategories": [
-      "language-communication",
-      "problem-solving", 
-      "creativity-innovation",
-      "learning-memory",
-      "information-integrity",
-      "governance-accountability"
-    ],
-    "adequateCategories": [
-      "social-intelligence",
-      "perception-vision",
-      "metacognition",
-      "harmful-content",
-      "bias-fairness",
-      "privacy-data",
-      "security-robustness",
-      "human-ai-interaction"
-    ],
-    "weakCategories": [
-      "dangerous-capabilities",
-      "environmental-impact"
-    ],
-    "insufficientCategories": [
-      "physical-manipulation",
-      "robotic-intelligence",
-      "value-chain",
-      "economic-displacement"
-    ],
-    "priorityAreas": [
-      "bias-fairness",
-      "dangerous-capabilities",
-      "environmental-impact",
-      "economic-displacement"
-    ]
-  },
-  "categoryEvaluations": {
-    "language-communication": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "yes",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B3": "yes",
-        "B4": "yes",
-        "B6": "yes",
-        "B5": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-gpt4t-lc-001",
-            "url": "https://openai.com/research/gpt-4",
-            "description": "GPT-4 Turbo performance on comprehensive language understanding and generation benchmarks",
-            "sourceType": "external",
-            "benchmarkName": "MMLU, HellaSwag, ARC-Challenge, WinoGrande",
-            "metrics": "Accuracy across multiple choice and generation tasks",
-            "score": "87.4% MMLU, 95.3% HellaSwag, 96.3% ARC-C, 87.5% WinoGrande",
-            "version": "v2024.04",
-            "taskVariants": "academic-knowledge, commonsense-reasoning, reading-comprehension",
-            "customFields": {}
-          }
-        ],
-        "A2": [
-          {
-            "id": "bench-gpt4t-lc-002", 
-            "url": "https://openai.com/safety/gpt-4-system-card",
-            "description": "Safety evaluation demonstrating appropriate refusal of harmful content while maintaining helpfulness",
-            "sourceType": "internal",
-            "benchmarkName": "OpenAI Safety Eval Suite, TruthfulQA",
-            "metrics": "Refusal rate on harmful prompts, truthfulness score",
-            "score": "99.1% harmful refusal rate, 83% truthfulness",
-            "version": "v1.2",
-            "taskVariants": "safety-alignment, truthfulness",
-            "customFields": {}
-          }
-        ],
-        "A3": [
-          {
-            "id": "bench-meem78d1-i6qj95",
-            "url": "https://arxiv.org/abs/2303.08774",
-            "description": "Comparative analysis showing GPT-4 outperforms GPT-3.5 and Claude-1 on language tasks",
-            "sourceType": "external",
-            "benchmarkName": "",
-            "metrics": "",
-            "score": "",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ],
-        "A4": [
-          {
-            "id": "bench-meem78d1-8fq55x",
-            "url": "https://openai.com/research/gpt-4-system-card",
-            "description": "Adversarial robustness testing results",
-            "sourceType": "internal",
-            "benchmarkName": "",
-            "metrics": "",
-            "score": "",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ],
-        "A5": [
-          {
-            "id": "bench-meem78d1-uxhy3w",
-            "url": "https://openai.com/api/monitoring",
-            "description": "Live production monitoring metrics",
-            "sourceType": "internal",
-            "benchmarkName": "",
-            "metrics": "",
-            "score": "",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ],
-        "A6": [
-          {
-            "id": "bench-meem78d1-mrpnnc",
-            "url": "https://openai.com/research/gpt-4-contamination",
-            "description": "Contamination analysis for benchmark datasets",
-            "sourceType": "internal",
-            "benchmarkName": "",
-            "metrics": "",
-            "score": "",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d1-z3ddep",
-            "url": "https://openai.com/research/gpt-4",
-            "description": "Comprehensive technical documentation of GPT-4 language capabilities",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B2": [
-          {
-            "id": "proc-meem78d1-ur2bqa",
-            "url": "https://github.com/openai/evals",
-            "description": "Open-source evaluation framework and prompts for reproducible testing",
-            "title": "OpenAI Evals Framework",
-            "author": "OpenAI Research Team",
-            "organization": "OpenAI",
-            "date": "2024-04-15",
-            "documentType": "Code Repository"
-          },
-          {
-            "id": "proc-meem78d1-ur2bqb",
-            "url": "https://openai.com/research/gpt-4-technical-report", 
-            "description": "Technical report containing detailed methodologies and experimental procedures",
-            "title": "GPT-4 Technical Report - Reproducibility Section",
-            "author": "OpenAI Safety Team",
-            "organization": "OpenAI",
-            "date": "2024-03-20",
-            "documentType": "Technical Report"
-          },
-          {
-            "id": "proc-meem78d1-ur2bqc",
-            "url": "https://openai.com/safety/reproducibility-guidelines",
-            "description": "Internal guidelines and procedures for ensuring evaluation reproducibility",
-            "title": "Model Evaluation Reproducibility Guidelines v2.1",
-            "author": "AI Safety Division",
-            "organization": "OpenAI",
-            "date": "2024-02-10",
-            "documentType": "Policy Document"
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d1-ur2bqd",
-            "url": "https://openai.com/research/expert-review-process",
-            "description": "Documentation of expert review process for language model capabilities",
-            "title": "Expert Review Process for Language Models",
-            "author": "External Advisory Board",
-            "organization": "OpenAI",
-            "date": "2024-04-01",
-            "documentType": "Process Documentation"
-          },
-          {
-            "id": "proc-meem78d1-ur2bqe",
-            "url": "https://openai.com/safety/feedback-incorporation",
-            "description": "Summary of expert feedback and how it was incorporated into final evaluations",
-            "title": "Expert Feedback Integration Report",
-            "author": "Safety Research Team",
-            "organization": "OpenAI", 
-            "date": "2024-04-12",
-            "documentType": "Review Report"
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d1-x59ofi",
-            "url": "https://openai.com/research/gpt-4",
-            "description": "Results visualization with uncertainty quantification",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B6": [
-          {
-            "id": "proc-meem78d1-4ehzby",
-            "url": "https://openai.com/safety/process",
-            "description": "Continuous evaluation and improvement process",
-            "sourceType": "",
-            "documentType": "Process Documentation",
-            "customFields": {}
-          }
-        ],
-        "B5": [
-          {
-            "id": "proc-meem78d1-25ufdv",
-            "url": "",
-            "description": "B5: Not applicable — standards mapping or regulatory alignment not performed for this sample.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "GPT-4 demonstrates exceptional multilingual capabilities and shows strong performance in code generation tasks. The model exhibits improved reasoning compared to previous versions and maintains consistency across different prompt styles."
-    },
-    "problem-solving": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes",
-          "no"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78d1-7fca0b",
-            "url": "https://openai.com/research/gpt-4",
-            "description": "Mathematical reasoning and problem-solving benchmark results",
-            "sourceType": "external",
-            "benchmarkName": "GSM8K, MATH, HumanEval",
-            "metrics": "Accuracy on math problems, code correctness",
-            "score": "92% on GSM8K, 42.5% on MATH, 67% on HumanEval",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ],
-        "A2": [
-          {
-            "id": "bench-meem78d1-gc02cz",
-            "url": "https://openai.com/safety/mathematical-reasoning",
-            "description": "Mathematical accuracy standards for educational applications",
-            "sourceType": "cooperative",
-            "benchmarkName": "",
-            "metrics": "",
-            "score": "",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ],
-        "A3": [
-          {
-            "id": "bench-meem78d1-7n7q6j",
-            "url": "https://arxiv.org/abs/2303.12712",
-            "description": "Comparative analysis of mathematical reasoning capabilities",
-            "sourceType": "external",
-            "benchmarkName": "",
-            "metrics": "",
-            "score": "",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d1-l73fej",
-            "url": "https://openai.com/research/gpt-4",
-            "description": "Documentation of problem-solving capabilities and limitations",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B2": [
-          {
-            "id": "proc-meem78d1-pabo9f",
-            "url": "",
-            "description": "Not applicable for this evaluation (replication package not provided for this public demo).",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d1-y8p28f",
-            "url": "",
-            "description": "Not applicable - no domain expert review captured for this category in the dummy data.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d1-6v1flq",
-            "url": "",
-            "description": "Not applicable - figures/uncertainty not included in the sample data.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B5": [
-          {
-            "id": "proc-meem78d1-yog9b0",
-            "url": "",
-            "description": "Not applicable - standards mapping not performed for this sample.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B6": [
-          {
-            "id": "proc-meem78d1-hszeac",
-            "url": "",
-            "description": "Not applicable - no formal retest procedures documented in this sample.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Strong performance on mathematical reasoning but struggles with complex multi-step problems requiring external tools. Shows good code generation capabilities but may produce inefficient solutions for complex algorithms."
-    },
-    "creativity-innovation": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78d1-04rfgu",
-            "url": "https://openai.com/research/gpt-4-creativity",
-            "description": "Creative writing and ideation benchmark results",
-            "sourceType": "internal",
-            "benchmarkName": "CREAM, Alternative Uses Task, Creative Story Generation",
-            "metrics": "Originality score, fluency, relevance",
-            "score": "8.2/10 originality, 9.1/10 fluency",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d1-mspa7s",
-            "url": "https://openai.com/research/gpt-4-creativity",
-            "description": "Creative capability evaluation methodology",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d1-x227lr",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d1-1jf9o1",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Demonstrates strong creative writing abilities and can generate novel ideas across domains. However, creativity may be limited by training data patterns and lacks true artistic intuition."
-    },
-    "learning-memory": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78d1-ktj6i7",
-            "url": "https://openai.com/research/gpt-4-learning",
-            "description": "In-context learning and few-shot performance evaluation",
-            "sourceType": "internal",
-            "benchmarkName": "Few-shot learning benchmarks, in-context learning tasks",
-            "metrics": "Few-shot accuracy, learning efficiency",
-            "score": "85% accuracy with 5 examples",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d1-o5k3we",
-            "url": "https://openai.com/research/gpt-4-learning",
-            "description": "Learning and memory capability documentation",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d1-ip124r",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d1-q5d1c0",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Excellent in-context learning capabilities but limited by context window. Cannot update knowledge base or learn from interactions permanently."
-    },
-    "social-intelligence": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78d1-aozmmz",
-            "url": "https://openai.com/research/gpt-4-social",
-            "description": "Social intelligence and theory of mind evaluation",
-            "sourceType": "external",
-            "benchmarkName": "ToMi, Social IQa, EmoBench",
-            "metrics": "Theory of mind accuracy, social reasoning score",
-            "score": "78% on ToMi, 82% on Social IQa",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d1-8cudww",
-            "url": "https://openai.com/research/gpt-4-social",
-            "description": "Social intelligence evaluation methodology",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d1-pewiw5",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d1-su40hm",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Shows good understanding of social contexts and emotional cues but may lack nuanced cultural understanding and can be overly formal in casual interactions."
-    },
-    "perception-vision": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78d1-89l7ui",
-            "url": "https://openai.com/research/gpt-4v",
-            "description": "Vision capabilities evaluation on standard benchmarks",
-            "sourceType": "external",
-            "benchmarkName": "VQA, COCO Captions, TextVQA",
-            "metrics": "VQA accuracy, caption quality scores",
-            "score": "77.2% on VQA, 85.5% on COCO Captions",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d1-cy0wc3",
-            "url": "https://openai.com/research/gpt-4v",
-            "description": "Vision capability evaluation methodology",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d1-8jr3bk",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d1-ge3tl9",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Strong performance on standard vision benchmarks but may struggle with fine-grained visual details and spatial reasoning in complex scenes."
-    },
-    "metacognition": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78d1-psjnoj",
-            "url": "https://openai.com/research/gpt-4-metacognition",
-            "description": "Confidence calibration and self-awareness evaluation",
-            "sourceType": "internal",
-            "benchmarkName": "Confidence calibration benchmarks, uncertainty quantification",
-            "metrics": "Calibration error, uncertainty correlation",
-            "score": "ECE: 0.12, Brier score: 0.18",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d1-jzqf46",
-            "url": "https://openai.com/research/gpt-4-metacognition",
-            "description": "Metacognitive capability evaluation",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d1-2t6njd",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d1-dwo0fs",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Shows reasonable confidence calibration but may be overconfident in some domains. Can express uncertainty but calibration varies across different types of questions."
-    },
-    "physical-manipulation": {
-      "benchmarkAnswers": {
-        "A1": "no",
-        "A2": "no",
-        "A3": "no",
-        "A4": "no",
-        "A5": "no",
-        "A6": "no"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "no"
-        ],
-        "B6": "no",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {},
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d1-vqbcgu",
-            "url": "https://openai.com/research/gpt-4-limitations",
-            "description": "Documentation of physical manipulation limitations",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d1-itur1k",
-            "url": "",
-            "description": "B3: Not applicable — process documentation for physical/robotic systems not relevant to text-only model.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d1-b5ktmr",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "GPT-4 is a text-only model and has no physical manipulation capabilities. This category is not applicable to the current system architecture."
-    },
-    "robotic-intelligence": {
-      "benchmarkAnswers": {
-        "A1": "no",
-        "A2": "no",
-        "A3": "no",
-        "A4": "no",
-        "A5": "no",
-        "A6": "no"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "no"
-        ],
-        "B6": "no",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {},
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d1-lsnpmc",
-            "url": "https://openai.com/research/gpt-4-limitations",
-            "description": "Documentation of robotic intelligence limitations",
-            "sourceType": "",
-            "documentType": "Technical Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d1-uf0rfm",
-            "url": "",
-            "description": "B3: Not applicable — process documentation for physical/robotic systems not relevant to text-only model.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d1-72ns4s",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "GPT-4 lacks robotic embodiment and autonomous physical capabilities. This category is not applicable to the current system."
-    },
-    "harmful-content": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "yes",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78d1-6eav70",
-            "url": "https://openai.com/safety/gpt-4",
-            "description": "Comprehensive safety evaluation including harmful content generation",
-            "sourceType": "internal",
-            "benchmarkName": "RealToxicityPrompts, HarmBench, AIR-Bench",
-            "metrics": "Toxicity rate, harmful content generation rate",
-            "score": "0.12% toxicity rate, 2.1% harmful refusal rate",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d1-bdj6ad",
-            "url": "https://openai.com/safety/gpt-4",
-            "description": "Harmful content generation risk assessment",
-            "sourceType": "",
-            "documentType": "Safety Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d1-zt08oh",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d1-3597aj",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Strong safety measures in place with low harmful content generation rates. Continuous monitoring and improvement of safety filters."
-    },
-    "bias-fairness": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78d1-jvp98n",
-            "url": "https://openai.com/research/gpt-4-bias",
-            "description": "Comprehensive bias evaluation across demographic groups",
-            "sourceType": "external",
-            "benchmarkName": "Winogender, CrowS-Pairs, BOLD, BBQ",
-            "metrics": "Bias score, demographic parity, equalized odds",
-            "score": "15% bias reduction vs GPT-3.5",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d1-f2xxf7",
-            "url": "https://openai.com/research/gpt-4-bias",
-            "description": "Bias and fairness evaluation methodology",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d1-2ep149",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d1-bhqq7o",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Ongoing efforts to reduce bias but some demographic disparities remain. Regular bias audits and mitigation strategies are in place."
-    },
-    "information-integrity": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "yes",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78d1-dx96iy",
-            "url": "https://openai.com/research/gpt-4-truthfulness",
-            "description": "Factual accuracy and hallucination evaluation",
-            "sourceType": "external",
-            "benchmarkName": "TruthfulQA, FEVER, HaluEval, FActScore",
-            "metrics": "Truthfulness rate, hallucination frequency, citation accuracy",
-            "score": "83% truthfulness on TruthfulQA, 12% hallucination rate",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d1-s13n0u",
-            "url": "https://openai.com/research/gpt-4-truthfulness",
-            "description": "Information integrity evaluation methodology",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d1-uesuss",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d1-dey36r",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Strong performance on factual accuracy benchmarks but still prone to hallucination in some domains. Ongoing work to improve citation accuracy and source attribution."
-    },
-    "privacy-data": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "yes",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78d1-myx5k8",
-            "url": "https://openai.com/privacy/gpt-4",
-            "description": "Privacy protection and data leakage evaluation",
-            "sourceType": "internal",
-            "benchmarkName": "Membership inference attacks, PII extraction tests",
-            "metrics": "MIA success rate, PII leakage rate",
-            "score": "3.2% MIA success rate, <0.1% PII leakage",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d1-ytn8p5",
-            "url": "https://openai.com/privacy/gpt-4",
-            "description": "Privacy and data protection evaluation",
-            "sourceType": "",
-            "documentType": "Privacy Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d1-aetqx2",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d1-rdamxl",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Strong privacy protections with low data leakage rates. Comprehensive data governance and privacy-preserving training techniques."
-    },
-    "security-robustness": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78d1-561kh9",
-            "url": "https://openai.com/security/gpt-4",
-            "description": "Security and robustness evaluation",
-            "sourceType": "external",
-            "benchmarkName": "AdvBench, prompt injection tests, OWASP LLM Top 10",
-            "metrics": "Attack success rate, robustness score",
-            "score": "8.5% prompt injection success rate",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d1-15g1ro",
-            "url": "https://openai.com/security/gpt-4",
-            "description": "Security and robustness evaluation methodology",
-            "sourceType": "",
-            "documentType": "Security Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d1-zq55yg",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d1-kxmho6",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Good robustness against common attacks but some vulnerabilities to sophisticated prompt injection remain. Ongoing security improvements and monitoring."
-    },
-    "dangerous-capabilities": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "no",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78d1-8ck601",
-            "url": "https://openai.com/safety/dangerous-capabilities",
-            "description": "Dangerous capabilities evaluation including CBRN and dual-use",
-            "sourceType": "internal",
-            "benchmarkName": "CBRN evaluation, dual-use assessment",
-            "metrics": "Dangerous information generation rate",
-            "score": "0.8% dangerous information generation",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d1-w4l1o6",
-            "url": "https://openai.com/safety/dangerous-capabilities",
-            "description": "Dangerous capabilities risk assessment",
-            "sourceType": "",
-            "documentType": "Safety Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d1-2mh0ur",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d1-3klld5",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Low rates of dangerous information generation with strong safety filters. Ongoing monitoring for emerging dangerous capabilities."
-    },
-    "human-ai-interaction": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "yes",
-        "A4": "yes",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78d1-ywtm4g",
-            "url": "https://openai.com/research/human-ai-interaction",
-            "description": "Human-AI interaction safety evaluation",
-            "sourceType": "external",
-            "benchmarkName": "Trust calibration, manipulation detection",
-            "metrics": "Trust calibration score, manipulation rate",
-            "score": "0.78 trust calibration, 2.1% manipulation detection",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d2-v8pkgx",
-            "url": "https://openai.com/research/human-ai-interaction",
-            "description": "Human-AI interaction risk evaluation",
-            "sourceType": "",
-            "documentType": "Research Paper",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d2-nbm1dy",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d2-g7np5n",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Generally safe human-AI interactions with good trust calibration. Some risk of over-reliance in certain domains."
-    },
-    "governance-accountability": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "yes",
-        "A3": "yes",
-        "A4": "no",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "yes",
-        "B5": [
-          "yes"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78d2-1h86di",
-            "url": "https://openai.com/governance/gpt-4",
-            "description": "Governance and accountability framework evaluation",
-            "sourceType": "internal",
-            "benchmarkName": "Transparency benchmarks, accountability metrics",
-            "metrics": "Documentation completeness, traceability score",
-            "score": "92% documentation completeness",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d2-rza4bj",
-            "url": "https://openai.com/governance/gpt-4",
-            "description": "Governance and accountability evaluation",
-            "sourceType": "",
-            "documentType": "Governance Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d2-h76wfk",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d2-6ijnr9",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Strong governance framework with comprehensive documentation. Clear accountability structures and oversight mechanisms in place."
-    },
-    "environmental-impact": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "no",
-        "A4": "no",
-        "A5": "no",
-        "A6": "yes"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "no"
-        ],
-        "B6": "yes",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78d2-yjwt6q",
-            "url": "https://openai.com/sustainability/gpt-4",
-            "description": "Environmental impact assessment",
-            "sourceType": "internal",
-            "benchmarkName": "Carbon footprint estimation, energy efficiency",
-            "metrics": "CO2 emissions, energy consumption per token",
-            "score": "0.0012 kg CO2 per 1000 tokens",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d2-jhqcek",
-            "url": "https://openai.com/sustainability/gpt-4",
-            "description": "Environmental impact evaluation methodology",
-            "sourceType": "",
-            "documentType": "Sustainability Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d2-2i6cwe",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d2-fgfwpf",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Significant environmental impact from training and inference. Ongoing efforts to improve efficiency and use renewable energy."
-    },
-    "economic-displacement": {
-      "benchmarkAnswers": {
-        "A1": "yes",
-        "A2": "no",
-        "A3": "no",
-        "A4": "no",
-        "A5": "no",
-        "A6": "no"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "yes",
-          "no"
-        ],
-        "B6": "no",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {
-        "A1": [
-          {
-            "id": "bench-meem78d2-1q3x95",
-            "url": "https://openai.com/research/economic-impact",
-            "description": "Economic displacement impact assessment",
-            "sourceType": "external",
-            "benchmarkName": "Job automation potential, task displacement analysis",
-            "metrics": "Automation potential score, job displacement risk",
-            "score": "35% of knowledge work tasks automatable",
-            "version": "",
-            "taskVariants": "",
-            "customFields": {}
-          }
-        ]
-      },
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d2-jtux1w",
-            "url": "https://openai.com/research/economic-impact",
-            "description": "Economic displacement evaluation",
-            "sourceType": "",
-            "documentType": "Economic Impact Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d2-lg5ehr",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d2-x0ncoi",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Significant potential for economic displacement in knowledge work. Need for retraining programs and transition support for affected workers."
-    },
-    "value-chain": {
-      "benchmarkAnswers": {
-        "A1": "no",
-        "A2": "no",
-        "A3": "no",
-        "A4": "no",
-        "A5": "no",
-        "A6": "no"
-      },
-      "processAnswers": {
-        "B1": "yes",
-        "B2": "no",
-        "B5": [
-          "no"
-        ],
-        "B6": "no",
-        "B3": "N/A",
-        "B4": "N/A"
-      },
-      "benchmarkSources": {},
-      "processSources": {
-        "B1": [
-          {
-            "id": "proc-meem78d2-vhucwn",
-            "url": "https://openai.com/supply-chain/gpt-4",
-            "description": "Value chain risk assessment documentation",
-            "sourceType": "",
-            "documentType": "Supply Chain Report",
-            "customFields": {}
-          }
-        ],
-        "B3": [
-          {
-            "id": "proc-meem78d2-t3qnpi",
-            "url": "",
-            "description": "B3: Not applicable — documentation or process evidence not captured for this evaluation.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ],
-        "B4": [
-          {
-            "id": "proc-meem78d2-pas31g",
-            "url": "",
-            "description": "B4: Not applicable — figures/uncertainty plots are not included in this report.",
-            "sourceType": "",
-            "documentType": "N/A",
-            "customFields": {}
-          }
-        ]
-      },
-      "additionalAspects": "Limited transparency in value chain evaluation. Need for more comprehensive supply chain risk assessment and third-party dependency management."
-    }
-  }
-}

scripts/generate-dummy-data.js

@@ -0,0 +1,441 @@
+
+const fs = require('fs');
+const path = require('path');
+
+// 1. Define Categories
+const CATEGORIES = [
+  "Core Performance",
+  "Core Quality Dimensions",
+  "Robustness",
+  "Calibration",
+  "Adversarial",
+  "Memorization",
+  "Fairness",
+  "Safety",
+  "Leakage/Contamination",
+  "Privacy",
+  "Interpretability",
+  "Efficiency",
+  "Retrainability",
+  "Meta-Learning"
+];
+
+// 2. Define Models
+const MODELS = [
+  {
+    id: "meta/llama-3-70b",
+    name: "Llama 3 70B",
+    developer: "Meta",
+    version: "1.0",
+    params: "70B",
+    release_date: "2024-04-18",
+    architecture: "Transformer",
+    engine: "vLLM"
+  },
+  {
+    id: "mistral/mistral-large",
+    name: "Mistral Large",
+    developer: "Mistral AI",
+    version: "Latest",
+    params: "Unknown",
+    release_date: "2024-02-26",
+    architecture: "MoE",
+    engine: "Mistral Platform"
+  },
+  {
+    id: "anthropic/claude-3-5-sonnet",
+    name: "Claude 3.5 Sonnet",
+    developer: "Anthropic",
+    version: "3.5",
+    params: "Unknown",
+    release_date: "2024-06-20",
+    architecture: "Transformer",
+    engine: "Anthropic API"
+  },
+  {
+    id: "openai/gpt-4o",
+    name: "GPT-4o",
+    developer: "OpenAI",
+    version: "2024-05-13",
+    params: "Unknown",
+    release_date: "2024-05-13",
+    architecture: "Transformer",
+    engine: "OpenAI API"
+  },
+  {
+    id: "google/gemma-2-27b",
+    name: "Gemma 2 27B",
+    developer: "Google DeepMind",
+    version: "2.0",
+    params: "27B",
+    release_date: "2024-06-27",
+    architecture: "Transformer",
+    engine: "Vertex AI"
+  },
+  {
+    id: "alibaba/qwen-2-72b",
+    name: "Qwen 2 72B",
+    developer: "Alibaba Cloud",
+    version: "2.0",
+    params: "72B",
+    release_date: "2024-06-07",
+    architecture: "Transformer",
+    engine: "vLLM"
+  }
+];
+
+// 3. CSV Data (Embedded)
+const csvContent = `title,subtitle,authors,link,code_link,date,purpose,principles_tested,functional_props,input_modality,output_modality,input_source,output_source,size,splits,design,judge,protocol,model_access,has_heldout,heldout_details,alignment_validation,is_valid,baseline_models,robustness_measures,known_limitations,benchmarks_list
+DigiData: Training and Evaluating General-Purpose Mobile Control Agents,"We present DigiData-Bench, a benchmark for evaluating mobile control agents on real-world complex tasks. We demonstrate that the commonly used step-accuracy metric falls short in reliably assessing mobile control agents and, to address this, we propose dynamic evaluation protocols and AI-powered evaluations as rigorous alternatives for agent assessment.",Meta FAIR,https://arxiv.org/abs/2511.07413,https://github.com/facebookresearch/DigiData,2025-11-08,Development; Research,mobile control agents,Core Performance,Text + Vision,Actions,New dataset (released with eval),Human annotations,Small (< 1K samples),,Dynamic data-driven (adaptive/interactive),Model-based: In the wild,"1. Given a goal and a trajectory produced by an agent, an LLM judge classifies whether it successfully achieved 2. the goal. We use LLM judges relying on both the screenshot and UI tree.",Outputs,False,,human operator is asked to judge whether the trajectory successfully achieves the goal or not. Then we assess the alignment between human judge and LLM judge to ensure a high alignment,unknown,,Prompt variations tested; Multiple runs per sample; Temperature sensitivity tested; Repeated evaluations; Ablation studies; Inter-rater reliability; Significance testing,"-live and dynamic environment introduces uncontrollable factors (feature deprecation, version changes etc) that makes certain tasks unavailable after a period -sizable efforts required to manually set up prework in the environment ","AndroidControl, Android in the wild"
+IntPhys 2,"IntPhys 2 offers a comprehensive suite of tests, based on the violation of expectation framework, that challenge models to differentiate between possible and impossible events within controlled and diverse virtual environments.",Meta FAIR,https://arxiv.org/abs/2506.09849,https://github.com/facebookresearch/IntPhys2,2025-05-31,Research,Intuitive Physics,Core Performance,Video,Scores/Embeddings,New dataset (released with eval),Simulation-based,Medium (1K - 100K samples),"Debug Set: 60 videos for Model calibration Main Set: 1,012 videos as Main evaluation set Held-Out Set: 344 videos as Test set",Fixed data-driven (static test set),Automatic (Reference-based),Feed a video to the model Ask the model using specific prompts wether the video is physically plausible or not Check if the model's answer match the ground truth label ,Outputs,True,Held-Out Set: 344 videos as Test set,We ran human baselines to ensure that this task is easy for humans. ,unknown,- Human baseline: 96% - Best model baseline: V-JEPA2 56%,Prompt variations tested; Multiple runs per sample; Temperature sensitivity tested,"Model can be very sensitive to the way they are prompted. Also the compression artefact of the video can also impact the results. Those are mostly limitations on the model sides, since human are not sensitive to those. ","IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning, Riochet et al. 2020"
+ImageNet,"A large-scale hierarchical image database designed for visual object recognition research, containing over 14 million images across 20,000+ categories.",Princeton University,https://ieeexplore.ieee.org/document/5206848,https://image-net.org/,2009-01-01,Research; Development,"Object recognition, Visual categorization, Multi-class classification",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Huge (> 10M samples),"Train, Validation, Test",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives an image as input 2. Model predicts class label from 1000 classes 3. Top-1 and Top-5 accuracy computed against ground truth,Outputs,True,"Private test set maintained by organizers, used for annual ILSVRC competitions","Expert curation of hierarchical categories, manual verification of labels, consistency checks across similar categories",unknown,AlexNet: 63.3% top-1 VGG: 71.5% top-1 ResNet-50: 76.1% top-1 ResNet-152: 78.3% top-1 EfficientNet-B7: 84.3% top-1 Human performance: ~95% top-5,Multiple runs per sample; Significance testing; Inter-rater reliability,Class imbalance in some categories; Label noise in training set; Some ambiguous images with multiple valid labels; Bias towards certain object viewpoints and contexts,"ImageNet-V2, ImageNet-C, ImageNet-R, ImageNet-A, ImageNet-Sketch"
+COCO (Common Objects in Context),"A large-scale object detection, segmentation, and captioning dataset containing 330K images with 80 object categories, designed to advance scene understanding.",Microsoft Research,https://arxiv.org/abs/1405.0312,https://github.com/cocodataset/cocoapi,2014-01-01,Research; Development; Selection,"Object detection, Instance segmentation, Keypoint detection, Panoptic segmentation, Image captioning",Core Performance,Vision (Image),Text; Structured Data,New dataset (released with eval),Human annotations,Large (100K - 1M samples),"Train, Validation, Test",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives image as input 2. Model outputs bounding boxes, segmentation masks, or captions 3. Metrics computed: mAP for detection, IoU for segmentation, BLEU/CIDEr for captioning",Outputs,True,"Test-dev and test-challenge splits maintained privately, submissions via evaluation server","Multi-annotator consensus for instance annotations, quality control through redundant labeling",unknown,Faster R-CNN: 42.0 mAP Mask R-CNN: 37.1 mask mAP YOLOv8: 53.9 mAP Human performance (detection): ~70 mAP,Inter-rater reliability; Multiple runs per sample; Confidence intervals,Small object detection remains challenging; Occlusion handling difficulties; Dataset bias toward certain object contexts; Annotation inconsistencies in crowded scenes,"LVIS, Objects365, Open Images, Visual Genome"
+CIFAR-10 and CIFAR-100,"Small-scale image classification benchmarks with 60K 32x32 color images in 10 (CIFAR-10) or 100 (CIFAR-100) classes, widely used for algorithm development and testing.",University of Toronto,https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf,https://github.com/pytorch/vision,2009-04-08,Research; Development; Selection,"Image classification, Transfer learning, Generalization",Core Performance; Robustness,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (50K), Test (10K)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives 32x32 RGB image 2. Model predicts class label (10 or 100 classes) 3. Classification accuracy computed,Outputs,False,,"Systematic sampling from larger dataset (80 million tiny images), human verification of labels",unknown,CIFAR-10: ResNet-56: 93.03% Wide ResNet-28-10: 96.11% PyramidNet: 96.54%  CIFAR-100: ResNet-56: 71.35% Wide ResNet-28-10: 81.15% PyramidNet: 83.78%,Multiple runs per sample; Seed variation tested; Ablation studies,Low resolution (32x32) limits fine-grained recognition; Some label noise in CIFAR-100; Dataset size enables memorization in large models; Limited diversity in poses and contexts,"CIFAR-10-C, CIFAR-10.1, CIFAR-100-C, STL-10, Tiny ImageNet"
+Pascal VOC,"A pioneering object detection and segmentation benchmark with 20 object classes, providing standardized evaluation for visual recognition tasks.",University of Oxford,https://link.springer.com/article/10.1007/s11263-009-0275-4,https://github.com/pytorch/vision,2007-01-01,Research; Development; Selection,"Object detection, Semantic segmentation, Instance segmentation, Action classification",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train, Validation, Test",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives image as input 2. Model predicts bounding boxes and class labels 3. mAP computed at IoU threshold 0.5,Outputs,True,"Test set labels held privately, evaluation via submission to organizers (historically)","Careful manual annotation with quality control, multiple annotators for difficult cases",unknown,R-CNN: 58.5 mAP (VOC 2007) Fast R-CNN: 70.0 mAP Faster R-CNN: 75.9 mAP YOLOv3: 78.6 mAP,Inter-rater reliability; Significance testing,Limited to 20 classes; Small dataset size by modern standards; Some annotation inconsistencies; Relatively simple backgrounds,"COCO, LVIS, Open Images, Cityscapes"
+Cityscapes,"A large-scale dataset for urban scene understanding with pixel-level annotations for semantic segmentation, instance segmentation, and depth estimation in autonomous driving contexts.","Daimler AG, MPI for Informatics, TU Darmstadt",https://arxiv.org/abs/1604.01685,https://github.com/mcordts/cityscapesScripts,2016-04-01,Research; Development; Deployment,"Semantic segmentation, Instance segmentation, Scene understanding, Autonomous driving perception",Core Performance; Robustness,Vision (Image); Video,Structured Data,New dataset (released with eval),Expert annotations,Medium (1K - 100K samples),"Train (2975), Val (500), Test (1525)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives street scene image (2048×1024) 2. Model predicts per-pixel semantic class (19 or 30 classes) 3. IoU and accuracy metrics computed,Outputs,True,"Test set labels private, evaluation via online server with leaderboard","Expert annotators with domain knowledge, multi-pass quality control, consistency verification across video sequences",unknown,FCN-8s: 65.3 mIoU DeepLab v3+: 82.1 mIoU HRNetV2: 83.0 mIoU SegFormer: 84.0 mIoU,Multiple runs per sample; Ablation studies; Significance testing,Limited to European cities; Weather bias (mostly good conditions); Class imbalance for rare objects; Fine annotation boundaries challenging,"ADE20K, KITTI, Mapillary Vistas, BDD100K, nuScenes"
+ADE20K,"A comprehensive scene parsing benchmark with 150 semantic categories, designed for understanding diverse indoor and outdoor scenes with detailed object and part annotations.",MIT CSAIL,https://arxiv.org/abs/1608.05442,https://github.com/CSAILVision/ADE20K,2017-06-01,Research; Development,"Scene parsing, Semantic segmentation, Multi-scale recognition, Part segmentation",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (20K), Val (2K), Test (3K)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives diverse scene image 2. Model predicts per-pixel semantic labels (150 classes) 3. Mean IoU and pixel accuracy computed,Outputs,False,,"Crowdsourced annotations with expert review, hierarchical consistency checks, multi-round verification",unknown,PSPNet: 43.29 mIoU DeepLab v3: 45.65 mIoU UPerNet: 44.85 mIoU SegFormer-B5: 51.8 mIoU,Inter-rater reliability; Multiple runs per sample,Long-tail distribution of object classes; Annotation granularity varies; Some scenes have ambiguous boundaries; Challenging for rare categories,"Cityscapes, Pascal Context, COCO-Stuff, Mapillary Vistas"
+Kinetics,"A large-scale video action recognition dataset with 400/600/700 human action classes, designed to advance video understanding and temporal reasoning.",DeepMind,https://arxiv.org/abs/1705.06950,https://github.com/cvdfoundation/kinetics-dataset,2017-05-01,Research; Development,"Action recognition, Video understanding, Temporal reasoning, Human activity recognition",Core Performance,Video,Structured Data,New dataset (released with eval),Human annotations,Large (100K - 1M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives 10-second video clip 2. Model predicts action class (400/600/700 classes) 3. Top-1 and Top-5 accuracy computed,Outputs,False,,"Human verification of video labels, removal of ambiguous clips, consistency checks for similar actions",unknown,I3D: 71.1% top-1 (K400) SlowFast: 79.8% top-1 (K400) X3D: 80.4% top-1 (K400) VideoMAE: 81.5% top-1 (K400),Multiple runs per sample; Temporal ordering tested,YouTube videos may become unavailable over time; Some action classes overlap or are ambiguous; Camera viewpoint bias; Dataset drift as internet content changes,"UCF-101, HMDB-51, ActivityNet, Something-Something, Moments in Time"
+KITTI,"An autonomous driving benchmark suite providing stereo vision, optical flow, 3D object detection, and tracking datasets collected from real-world driving scenarios.","Karlsruhe Institute of Technology, Toyota Technological Institute",https://www.cvlibs.net/publications/Geiger2012CVPR.pdf,https://github.com/bostondiditeam/kitti,2012-01-01,Research; Development; Deployment,"3D object detection, Stereo vision, Optical flow, Visual odometry, Tracking",Core Performance; Robustness,Vision (Image); Video,Structured Data,New dataset (released with eval),Human annotations; Programmatically generated,Medium (1K - 100K samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives stereo image pair or point cloud 2. Model predicts 3D bounding boxes, orientation, class 3. 3D AP computed at different difficulty levels (easy/moderate/hard)",Outputs,True,"Test set labels held privately, evaluation via online server with public leaderboard","LiDAR ground truth for 3D positions, manual verification of annotations, multi-sensor fusion for accuracy",unknown,"PointPillars: 79.05 AP (Car, Moderate) PV-RCNN: 83.90 AP (Car, Moderate) CenterPoint: 85.15 AP (Car, Moderate)",Multiple difficulty levels; Ablation studies; Significance testing,Limited to specific geographic region; Weather bias (mostly clear); Limited nighttime data; Class imbalance (cars dominate),"nuScenes, Waymo Open Dataset, Argoverse, A2D2, Lyft Level 5"
+Places365,"A scene recognition benchmark with 365 scene categories and over 10 million images, designed to understand high-level visual concepts and environmental context.",MIT CSAIL,https://arxiv.org/abs/1610.02055,https://github.com/CSAILVision/places365,2017-07-01,Research; Development,"Scene recognition, Scene classification, Environmental understanding, Context recognition",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Very Huge (> 100M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives scene image 2. Model predicts scene category (365 classes) 3. Top-1 and Top-5 accuracy computed,Outputs,False,,"Human verification, consistency checks for scene categories, hierarchical taxonomy validation",unknown,ResNet-152: 55.24% top-1 DenseNet-161: 56.12% top-1 ResNeXt-101: 56.05% top-1,Inter-rater reliability; Multiple runs per sample,Some scene categories overlap semantically; Cultural bias in scene definitions; Indoor scenes better represented than outdoor; Ambiguous boundary cases,"SUN397, MIT Indoor 67, Scene-15, ADE20K"
+UCF-101,"An action recognition benchmark with 101 action categories and 13,320 videos collected from YouTube, widely used for video understanding research.",University of Central Florida,https://arxiv.org/abs/1212.0402,https://github.com/pytorch/vision,2012-01-01,Research; Development,"Action recognition, Video classification, Temporal understanding",Core Performance,Video,Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),Train/Test (3 splits provided),Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives video clip 2. Model predicts action class (101 classes) 3. Average accuracy across 3 splits reported,Outputs,False,,"Manual verification of action labels, removal of ambiguous videos",unknown,Two-Stream CNN: 88.0% I3D: 95.6% SlowFast: 96.8% VideoMAE: 97.2%,Multiple evaluation splits; Seed variation tested,YouTube videos may become unavailable; Camera motion and quality vary; Some action classes are very similar; Dataset saturation with modern methods,"HMDB-51, Kinetics, ActivityNet, Something-Something-V2"
+NYU Depth V2,"An RGB-D dataset for indoor scene understanding with 1449 densely labeled pairs of aligned RGB and depth images, designed for depth estimation and semantic segmentation.",New York University,https://cs.nyu.edu/~fergus/datasets/indoor_seg_support.pdf,https://github.com/ankurhanda/nyuv2-meta-data,2012-06-01,Research; Development,"Depth estimation, RGB-D understanding, Indoor scene parsing, 3D reconstruction",Core Performance,Vision (Image); Structured Data,Structured Data,New dataset (released with eval),Expert annotations,Small (< 1K samples),"Train (795), Test (654)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives RGB image 2. Model predicts depth map or semantic segmentation 3. RMSE, absolute relative error, and accuracy metrics computed",Outputs,False,,"Kinect sensor ground truth with manual alignment corrections, multi-view consistency",unknown,Depth Estimation: Eigen et al.: 0.641 RMSE AdaBins: 0.364 RMSE BTS: 0.392 RMSE,Multiple runs per sample; Ablation studies,"Small dataset size; Limited to indoor scenes; Kinect depth sensor limitations (range, IR interference); Mostly residential environments","ScanNet, Matterport3D, SUNRGB-D, KITTI Depth"
+CelebA,"A large-scale face attributes dataset with 202,599 face images annotated with 40 binary attributes, 5 landmark locations, and identity information for face recognition and attribute prediction.",The Chinese University of Hong Kong,https://arxiv.org/abs/1411.7766,https://github.com/tkarras/progressive_growing_of_gans,2015-09-01,Research; Development,"Face attribute recognition, Face detection, Facial landmark detection, Identity recognition",Core Performance; Fairness,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Large (100K - 1M samples),"Train (162,770), Val (19,867), Test (19,962)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives face image 2. Model predicts binary attributes (40 dimensions) 3. Per-attribute accuracy and mean accuracy computed,Outputs,False,,"Manual annotation with quality control, consistency verification across multiple attributes",unknown,LNets+ANet: 87.30% mean accuracy Walk and Learn: 88.06% FaceNet: 89.35%,Inter-rater reliability; Multiple runs per sample,Demographic bias (celebrity images); Some attributes are subjective; Label noise in some attributes; Privacy concerns with celebrity images; Lighting and pose variation,"LFW, VGGFace2, MS-Celeb-1M, FFHQ"
+Visual Genome,"A comprehensive visual understanding dataset with dense annotations of objects, attributes, relationships, and scene graphs across 108K images for structured scene understanding.",Stanford University,https://arxiv.org/abs/1602.07332,https://github.com/ranjaykrishna/visual_genome_python_driver,2016-05-01,Research; Development,"Scene graph generation, Visual relationship detection, Visual question answering, Dense captioning",Core Performance,Vision (Image),Structured Data; Text,MS COCO,Crowdsourced annotations,Large (100K - 1M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives image 2. Model predicts objects, attributes, and relationships 3. Recall@K metrics for scene graph components",Outputs,False,,"Multi-round crowdsourced annotations with validation, consistency checks for relationships",unknown,IMP: 14.6 R@50 (scene graph detection) Motifs: 21.4 R@50 VCTree: 22.0 R@50 GPS-Net: 24.0 R@50,Inter-rater reliability; Multiple runs per sample,Long-tail distribution of relationships; Annotation inconsistencies; Subjective relationship definitions; Incomplete annotations (not all relationships captured),"GQA, Scene Graph Benchmark, VRD, OpenImages V6"
+LVIS (Large Vocabulary Instance Segmentation),A large-scale instance segmentation dataset with 1203 object categories designed to address the long-tail distribution challenge in object recognition.,Facebook AI Research,https://arxiv.org/abs/1908.03195,https://github.com/lvis-dataset/lvis-api,2019-08-01,Research; Development,"Instance segmentation, Long-tail recognition, Object detection, Fine-grained categorization",Core Performance; Robustness,Vision (Image),Structured Data,MS COCO,Expert annotations,Large (100K - 1M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives image 2. Model predicts instance masks and categories (1203 classes) 3. AP computed separately for rare, common, and frequent categories",Outputs,True,"Test set labels private, evaluation via online server","Expert annotators with WordNet taxonomy, quality control for long-tail categories",unknown,Mask R-CNN: 21.2 AP (v1.0) Cascade R-CNN: 26.2 AP Swin Transformer: 50.9 AP,Multiple runs per sample; Ablation studies; Category frequency stratification,Rare categories have very few examples; Annotation cost for 1203 categories; Some category definitions overlap; Challenging for zero-shot generalization,"COCO, Objects365, OpenImages, iNaturalist"
+Mapillary Vistas,"A diverse street-level imagery dataset with pixel-level annotations for 66 object categories, designed for robust semantic segmentation across varied geographic locations and conditions.",Mapillary AB,https://openaccess.thecvf.com/content_ICCV_2017/papers/Neuhold_The_Mapillary_Vistas_ICCV_2017_paper.pdf,https://github.com/mapillary/mapillary_vistas,2017-08-01,Research; Development; Deployment,"Semantic segmentation, Panoptic segmentation, Scene understanding, Robust perception",Core Performance; Robustness,Vision (Image),Structured Data,User-generated content,Expert annotations,Medium (1K - 100K samples),"Train (18K), Val (2K), Test (5K)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives street-level image 2. Model predicts per-pixel semantic labels (66 classes) 3. mIoU computed across all classes,Outputs,True,"Test set labels private, evaluation via online platform","Expert annotators, multi-stage quality control, geographic diversity validation",unknown,PSPNet: 42.7 mIoU DeepLab v3+: 45.8 mIoU HRNetV2: 50.3 mIoU,Geographic diversity tested; Weather variations included; Multiple runs per sample,Varying image quality from crowdsourced data; Camera parameter diversity; Some regions overrepresented; Annotation inconsistencies across diverse scenes,"Cityscapes, BDD100K, IDD, WildDash"
+MPII Human Pose,"A benchmark for human pose estimation with 25K images containing over 40K annotated people with 16 body joints, covering diverse activities and viewpoints.",Max Planck Institute for Informatics,https://openaccess.thecvf.com/content_cvpr_2014/papers/Andriluka_2D_Human_Pose_2014_CVPR_paper.pdf,https://www.mpi-inf.mpg.de/departments/computer-vision-and-machine-learning/software-and-datasets/mpii-human-pose-dataset,2014-06-01,Research; Development,"Human pose estimation, Keypoint detection, Articulated pose estimation, Activity recognition",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (~29K people), Test (~12K people)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives image with person 2. Model predicts 2D joint locations (16 joints) 3. PCKh (Percentage of Correct Keypoints) at various thresholds,Outputs,False,,"Manual annotation with consistency checks, multi-annotator agreement for difficult poses",unknown,Hourglass Network: 90.9 PCKh@0.5 HRNet: 92.3 PCKh@0.5 SimpleBaseline: 91.5 PCKh@0.5,Multiple difficulty levels; Inter-rater reliability; Occlusion analysis,2D annotations only (no 3D); Occlusion and truncation challenges; Some joint definitions ambiguous; Dataset bias toward certain activities,"COCO Keypoints, Human3.6M, PoseTrack, CrowdPose"
+Open Images,"A large-scale multi-task dataset with ~9M images annotated for image classification (20K classes), object detection (600 classes), visual relationships, and segmentation masks.",Google Research,https://arxiv.org/abs/1811.00982,https://github.com/openimages/dataset,2018-01-01,Research; Development; Selection,"Object detection, Image classification, Visual relationship detection, Instance segmentation",Core Performance; Robustness,Vision (Image),Structured Data,New dataset (released with eval),Human annotations; Crowdsourced annotations,Very Huge (> 100M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives image 2. Model performs classification, detection, or segmentation 3. mAP computed for detection, top-k accuracy for classification",Outputs,True,"Test set labels private for challenges, public validation set available","Multi-stage crowdsourced annotation with verification, automated quality filters",unknown,Faster R-CNN: 54.3 mAP (detection) YOLOv4: 55.8 mAP EfficientDet: 56.1 mAP,Multiple runs per sample; Confidence intervals; Large-scale diversity,Long-tail class distribution; Annotation inconsistencies at scale; Some classes poorly defined; Label noise in crowdsourced annotations,"COCO, LVIS, Objects365, ImageNet"
+ScanNet,"A richly-annotated 3D dataset of indoor scenes with RGB-D scans, semantic segmentation, instance segmentation, and 3D object bounding boxes for 1513 scenes.","Stanford University, Princeton University, Technical University of Munich",https://arxiv.org/abs/1702.04405,https://github.com/ScanNet/ScanNet,2017-04-01,Research; Development,"3D scene understanding, Semantic segmentation, Instance segmentation, 3D reconstruction",Core Performance,Vision (Image); Video; Structured Data,Structured Data,New dataset (released with eval),Human annotations; Programmatically generated,Medium (1K - 100K samples),"Train (1201), Val (312), Test (100)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives 3D scene (RGB-D scans) 2. Model predicts semantic labels or instance masks 3. mIoU for semantic segmentation, mAP for instance segmentation",Outputs,True,"Test set labels private, evaluation via online benchmark server","Manual verification of 3D reconstructions, multi-annotator consistency for semantic labels",unknown,PointNet++: 53.5 mIoU SparseConvNet: 72.5 mIoU MinkowskiNet: 73.6 mIoU,Multiple runs per sample; Ablation studies,Limited to indoor scenes; Reconstruction artifacts; Scanning noise and occlusions; Limited scene diversity (mostly offices and apartments),"Matterport3D, S3DIS, 2D-3D-S, ARKitScenes"
+nuScenes,"A large-scale autonomous driving dataset with 1000 scenes (40K frames) featuring 3D object annotations, tracking IDs, and multimodal sensor data including cameras, LiDAR, and radar.",Motional (formerly nuTonomy),https://arxiv.org/abs/1903.11027,https://github.com/nutonomy/nuscenes-devkit,2019-03-26,Research; Development; Deployment,"3D object detection, Multi-object tracking, Prediction, Sensor fusion, Scene understanding",Core Performance; Robustness,Vision (Image); Video; Structured Data,Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (700), Val (150), Test (150)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives multimodal sensor data (cameras, LiDAR, radar) 2. Model predicts 3D bounding boxes and tracking IDs 3. NDS (nuScenes Detection Score) and mAP computed",Outputs,True,"Test set labels private, evaluation via online leaderboard","Expert annotators, multi-sensor consistency checks, temporal coherence validation",unknown,PointPillars: 45.3 NDS CenterPoint: 65.5 NDS BEVFusion: 72.9 NDS,Geographic diversity; Weather conditions; Day/night variations; Multiple runs per sample,"Limited geographic coverage (Boston, Singapore); Sensor calibration challenges; Annotation latency for fast-moving objects; Class imbalance","Waymo Open Dataset, KITTI, Argoverse, Lyft Level 5, A2D2"
+ActivityNet,"A large-scale video dataset for human activity understanding with 200 activity classes and temporal annotations, designed for action recognition and temporal action localization.","Universidad del Norte, KAUST",https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Heilbron_ActivityNet_A_Large-Scale_2015_CVPR_paper.pdf,https://github.com/activitynet/ActivityNet,2015-06-01,Research; Development,"Temporal action detection, Action recognition, Dense video captioning, Video understanding",Core Performance,Video,Structured Data; Text,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (50%), Val (25%), Test (25%)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives untrimmed video 2. Model predicts action segments with class labels 3. mAP at different IoU thresholds (0.5, 0.75, 0.95)",Outputs,True,"Test set used for annual challenges, labels withheld","Multi-annotator temporal boundary agreement, consistency checks for activity definitions",unknown,SSN: 41.3 mAP@0.5 BMN: 50.1 mAP@0.5 TALLFormer: 59.8 mAP@0.5,Inter-rater reliability; Multiple IoU thresholds; Temporal boundary sensitivity,YouTube video availability issues; Temporal boundary ambiguity; Action class overlap; Video quality variance,"THUMOS14, Kinetics, Charades, MultiTHUMOS, AVA"
+DAVIS (Densely Annotated VIdeo Segmentation),A video object segmentation benchmark with high-quality pixel-level annotations for densely segmenting objects in video sequences.,"ETH Zurich",https://arxiv.org/abs/1704.00675,https://github.com/davisvideochallenge/davis,2016-10-01,Research; Development,"Video object segmentation, Temporal consistency, Object tracking, Dense prediction",Core Performance; Robustness,Video,Structured Data,New dataset (released with eval),Human annotations,Small (< 1K samples),"Train/Val (60 sequences), Test-dev (30 sequences)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives video sequence with first frame annotation 2. Model propagates segmentation to subsequent frames 3. J&F metric (region similarity and contour accuracy),Outputs,True,"Test-challenge set for competitions, labels withheld",High-quality manual annotations with temporal consistency verification,unknown,OSVOS: 79.8 J&F STM: 84.3 J&F XMem: 86.2 J&F,Temporal consistency tested; Occlusion robustness; Multiple runs per sample,Small dataset size; Limited object categories; Short video sequences; Primarily objects with clear boundaries,"YouTube-VOS, FBMS, SegTrack, OVIS"
+VQA (Visual Question Answering),"A dataset with open-ended questions about images requiring visual understanding and reasoning, containing 265K images and over 1M questions.","Virginia Tech, Facebook AI Research",https://arxiv.org/abs/1505.00468,https://github.com/GT-Vision-Lab/VQA,2015-10-01,Research; Development,"Visual question answering, Visual reasoning, Multimodal understanding, Common sense reasoning",Core Performance; Robustness,Text + Vision,Text,MS COCO,Crowdsourced annotations,Huge (> 10M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based); Human: Representative sample,1. Model receives image and question 2. Model generates answer 3. Accuracy computed with consensus matching (multiple human answers),Outputs,True,"Test-dev and test-std splits, evaluation via server","Multiple human answers per question (10 answers), consensus-based evaluation",unknown,Bottom-Up Top-Down: 70.3% VILBERT: 72.4% OSCAR: 73.8% BLIP: 78.3%,Multiple human references; Prompt variations tested; Inter-rater reliability,Language bias (can answer many questions without image); Dataset bias toward common objects; Answer distribution imbalance; Ambiguous questions,"GQA, VQA v2, OK-VQA, TextVQA, VizWiz"
+CLEVR,"A diagnostic dataset for compositional visual reasoning with 100K synthetic images and 1M questions testing spatial relations, counting, and logic.","Stanford University, Facebook AI Research",https://arxiv.org/abs/1612.06890,https://github.com/facebookresearch/clevr-dataset-gen,2017-04-01,Research; Development,"Compositional reasoning, Spatial reasoning, Counting, Logical reasoning, Visual reasoning",Core Performance; Robustness,Text + Vision,Text,Synthetic/Generated,Programmatically generated,Huge (> 10M samples),"Train (70K), Val (15K), Test (15K)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives synthetic image and question 2. Model predicts answer from predefined set 3. Accuracy computed, can analyze by question type",Outputs,False,,"Programmatically generated with known ground truth, exhaustive question type coverage",unknown,CNN+LSTM: 52.3% Film: 97.7% NS-VQA: 99.8% MAC: 98.9%,Ablation studies; Question type analysis; Compositional generalization tested,Synthetic domain (limited real-world applicability); Simple shapes and colors; No natural language variation; Programmatic biases,"CLEVR-CoGenT, GQA, NLVR2, CLOSURE"
+Waymo Open Dataset,"A large-scale autonomous driving dataset with 1000 diverse driving scenes, 12M LiDAR points per frame, high-resolution camera images, and rich 3D annotations.",Waymo LLC,https://arxiv.org/abs/1912.04838,https://github.com/waymo-research/waymo-open-dataset,2019-08-21,Research; Development; Deployment,"3D object detection, 2D object detection, Tracking, Domain adaptation, Sensor fusion",Core Performance; Robustness,Vision (Image); Video; Structured Data,Structured Data,New dataset (released with eval),Human annotations; Programmatically generated,Huge (> 10M samples),"Train (798), Val (202), Test (150)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives multimodal sensor data 2. Model predicts 3D bounding boxes with tracking IDs 3. AP/APH metrics at different IoU thresholds and difficulty levels,Outputs,True,"Test set labels private, evaluation via online leaderboard","Multi-stage annotation pipeline with quality checks, temporal consistency validation",unknown,PointPillars: 63.8 L2 APH (Vehicle) CenterPoint: 73.9 L2 APH PV-RCNN++: 77.8 L2 APH,Geographic diversity; Time of day variations; Weather conditions; Multiple difficulty levels,Geographic concentration in specific cities; Sensor-specific challenges; Annotation latency for distant objects; Class imbalance,"nuScenes, KITTI, Argoverse 2, Once, Lyft Level 5"
+Fashion-MNIST,"A drop-in replacement for MNIST with 70K grayscale images of fashion products across 10 categories, designed to be a more challenging benchmark for image classification.",Zalando Research,https://arxiv.org/abs/1708.07747,https://github.com/zalandoresearch/fashion-mnist,2017-08-25,Research; Development; Selection,"Image classification, Transfer learning, Benchmark comparison",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (60K), Test (10K)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives 28x28 grayscale image 2. Model predicts fashion category (10 classes) 3. Classification accuracy computed,Outputs,False,,Product catalog labels verified by domain experts,unknown,Linear Classifier: 83.7% CNN: 93.5% ResNet: 94.9% Vision Transformer: 95.1%,Multiple runs per sample; Seed variation tested,Low resolution (28x28); Grayscale only; Limited intra-class variation; Some categories overlap visually,"MNIST, EMNIST, Kuzushiji-MNIST, DeepFashion"
+MMLU (Massive Multitask Language Understanding),"A comprehensive benchmark with 57 tasks spanning STEM, humanities, social sciences, and more, designed to measure multitask accuracy and knowledge breadth in language models.","UC Berkeley, Columbia University",https://arxiv.org/abs/2009.03300,https://github.com/hendrycks/test,2020-09-07,Research; Development; Selection,"World knowledge, Reasoning, Domain expertise across 57 subjects",Core Performance,Text,Text,New dataset (released with eval),Expert annotations,Medium (1K - 100K samples),"Dev (5 shot examples per task), Test (285 questions per task average)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives multiple choice question with 4 options 2. Model predicts answer (A/B/C/D) 3. Accuracy computed per subject and overall,Outputs,False,,"Questions sourced from practice exams and tests, expert review for correctness",unknown,GPT-3 (175B): 43.9% GPT-3.5: 70.0% GPT-4: 86.4% Claude 3.5 Sonnet: 88.7% Random baseline: 25%,Multiple evaluation runs; Few-shot prompting variations; Subject-wise analysis,Multiple choice format may not reflect real-world usage; Some questions have ambiguous answers; Dataset contamination concerns with web-trained models; Cultural and knowledge cutoff biases,"MMLU-Pro, AGIEval, C-Eval, CMMLU"
+HumanEval,A code generation benchmark with 164 hand-written programming problems to evaluate functional correctness of synthesized Python code.,OpenAI,https://arxiv.org/abs/2107.03374,https://github.com/openai/human-eval,2021-07-07,Research; Development; Selection,"Code generation, Programming ability, Functional correctness, Code understanding",Core Performance,Text; Code,Code,New dataset (released with eval),Author-provided,Small (< 1K samples),Test (164 problems),Fixed data-driven (static test set),Automatic (Execution-based),1. Model receives function signature and docstring 2. Model generates function implementation 3. Generated code executed against unit tests 4. pass@k metric computed (% passing all tests in k samples),Outputs,False,,"Hand-written problems with comprehensive unit tests, manual verification of test correctness",unknown,Codex (12B): 28.8% pass@1 GPT-3.5-turbo: 48.1% pass@1 GPT-4: 67.0% pass@1 Claude 3.5 Sonnet: 92.0% pass@1,Multiple samples per problem (pass@k); Temperature sensitivity tested; Execution-based verification,Limited to Python; Small dataset size (164 problems); Relatively simple problems; May be contaminated in training data; No testing of code efficiency or style,"MBPP, APPS, CodeContests, HumanEval+, MultiPL-E"
+HellaSwag,"A benchmark for commonsense natural language inference about physical situations, requiring models to complete scenarios with the most plausible continuation.","University of Washington, Allen Institute for AI",https://arxiv.org/abs/1905.07830,https://github.com/rowanz/hellaswag,2019-05-19,Research; Development,"Commonsense reasoning, Physical understanding, Situation modeling, Plausibility judgment",Core Performance; Robustness,Text,Text,New dataset (released with eval),Crowdsourced annotations,Medium (1K - 100K samples),"Train (39,905), Val (10,042), Test (10,003)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives scenario context 2. Model selects most plausible continuation from 4 options 3. Accuracy computed,Outputs,False,,"Adversarial filtering using BERT to ensure difficulty, human validation of plausibility",unknown,BERT-Large: 47.3% GPT-2: 50.9% GPT-3: 78.9% GPT-4: 95.3% Human performance: 95.6%,Adversarial filtering; Multiple runs per sample; Human baseline comparison,Dataset may be easier than originally intended for modern LLMs; Multiple choice format; Adversarial examples may have artifacts; Potential data contamination,"PIQA, WinoGrande, CommonsenseQA, ARC"
+GSM8K (Grade School Math 8K),"A dataset of 8,500 grade school math word problems requiring multi-step arithmetic reasoning to solve.",OpenAI,https://arxiv.org/abs/2110.14168,https://github.com/openai/grade-school-math,2021-10-27,Research; Development,"Mathematical reasoning, Multi-step problem solving, Arithmetic reasoning, Chain-of-thought reasoning",Core Performance,Text,Text,New dataset (released with eval),Author-provided,Medium (1K - 100K samples),"Train (7,473), Test (1,319)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives math word problem 2. Model generates solution with reasoning steps 3. Final numerical answer extracted and compared to ground truth,Outputs,False,,"Human-written problems with verified solutions, consistency checks for answer correctness",unknown,GPT-3 (175B): 34.2% GPT-3.5: 57.1% GPT-4: 92.0% GPT-4o: 96.1% Claude 3.5 Sonnet: 96.4%,Multiple runs per sample; Chain-of-thought prompting tested; Answer extraction robustness,Limited to grade-school level math; Answer extraction can be brittle; Some problems have ambiguous wording; Potential contamination in training data,"MATH, GSM-Hard, SVAMP, ASDiv, MathQA"
+TruthfulQA,A benchmark measuring whether language models generate truthful answers to questions that humans might answer falsely due to misconceptions or false beliefs.,"Oxford University, OpenAI",https://arxiv.org/abs/2109.07958,https://github.com/sylinrl/TruthfulQA,2021-09-16,Research; Development; Safety,"Truthfulness, Factual accuracy, Resistance to misconceptions, Calibration",Safety; Core Performance; Calibration,Text,Text,New dataset (released with eval),Expert annotations,Small (< 1K samples),Test (817 questions across 38 categories),Fixed data-driven (static test set),Model-based: Expert; Human: Experts,1. Model receives question designed to elicit false beliefs 2. Model generates answer 3. Answers judged for truthfulness and informativeness using GPT-judge or human evaluation,Outputs,False,,"Expert-curated questions targeting known misconceptions, multi-rater validation of truth labels",unknown,GPT-3 (175B): 58.0% GPT-3.5: 47.0% GPT-4: 59.0% Claude 2: 62.0% Human baseline: 94.0%,"Multiple evaluation methods (MC1, MC2, generative); Human validation; Model-based evaluation correlation",Subjective truthfulness judgments in some cases; Cultural bias in what constitutes 'truth'; Model-based evaluation may not align with human judgment; Limited coverage of misconceptions,"FactScore, HaluEval, SelfCheckGPT, FELM"
+BIG-bench (Beyond the Imitation Game),"A collaborative benchmark with 204 tasks spanning linguistics, child development, math, commonsense reasoning, biology, physics, social bias, and software development.","Google Research, 450+ authors",https://arxiv.org/abs/2206.04615,https://github.com/google/BIG-bench,2022-06-09,Research; Development,"Diverse capabilities across 204 tasks including reasoning, knowledge, language understanding, bias detection",Core Performance; Fairness; Robustness,Text,Text,New dataset (released with eval),Multiple/Mixed sources,Large (100K - 1M samples),Varies by task,Composite,Automatic (Reference-based); Model-based: In the wild,1. Model evaluated on 204 diverse tasks 2. Each task has its own evaluation protocol 3. Performance aggregated across tasks,Outputs,False,,"Crowdsourced task creation with quality review, diverse authorship for broad coverage",unknown,Average human rater: 89.0% Few-shot PaLM (540B): 65.7% GPT-4: ~83% (estimated on BIG-Bench Hard),Multiple tasks provide robustness; Human baseline comparison; Cross-task analysis,Task quality varies; Some tasks too easy or too hard; Computational cost of running all 204 tasks; Aggregation methodology debatable,"BIG-Bench Hard, MMLU, HELM, SuperGLUE"
+MATH,"A dataset of 12,500 challenging competition mathematics problems from high school math competitions, requiring advanced mathematical reasoning and problem-solving.","UC Berkeley, OpenAI",https://arxiv.org/abs/2103.03874,https://github.com/hendrycks/math,2021-03-05,Research; Development,"Advanced mathematical reasoning, Problem solving, Multi-step reasoning, Mathematical knowledge",Core Performance,Text,Text,New dataset (released with eval),Published references,Medium (1K - 100K samples),"Train (7,500), Test (5,000)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives competition math problem 2. Model generates solution with steps 3. Final answer extracted and checked against ground truth 4. Problems span 7 subjects with 5 difficulty levels,Outputs,False,,"Problems from real math competitions with verified solutions, difficulty levels validated",unknown,GPT-3: 6.9% GPT-4: 42.5% Minerva (540B): 33.6% GPT-4 Turbo: 52.9% Claude 3.5 Sonnet: 71.1%,Multiple difficulty levels; Subject-wise analysis; Chain-of-thought evaluation,Answer extraction challenges; LaTeX formatting issues; Symbolic vs numeric answers; High difficulty may not reflect practical math usage,"GSM8K, MathQA, SVAMP, ASDiv, Hendrycks MATH"
+ARC (AI2 Reasoning Challenge),"A dataset of 7,787 science exam questions from grade 3-9, designed to require reasoning beyond simple retrieval or pattern matching.",Allen Institute for AI,https://arxiv.org/abs/1803.05457,https://github.com/allenai/arc,2018-03-14,Research; Development,"Scientific reasoning, Commonsense reasoning, Knowledge retrieval, Multi-hop reasoning",Core Performance; Robustness,Text,Text,New dataset (released with eval),Existing dataset labels,Medium (1K - 100K samples),"Easy (2,376 train, 570 test), Challenge (1,119 train, 1,172 test)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives science exam question with multiple choice options 2. Model selects answer 3. Accuracy computed separately for Easy and Challenge sets,Outputs,False,,"Real exam questions filtered to require reasoning, partitioned by difficulty using retrieval-based baseline",unknown,ARC-Challenge: BERT: 59.1% GPT-3: 51.4% GPT-3.5: 85.2% GPT-4: 96.3%,Two difficulty levels; Multiple choice format variations; Retrieval baseline comparison,Multiple choice format; Some questions solvable without reasoning; Challenge set becoming saturated; Grade-school level may not test advanced reasoning,"OpenBookQA, CommonsenseQA, QASC, SciQ"
+DROP (Discrete Reasoning Over Paragraphs),"A reading comprehension benchmark requiring discrete reasoning operations over paragraph content, including sorting, counting, and arithmetic.","Allen Institute for AI, University of Washington",https://arxiv.org/abs/1903.00161,https://github.com/allenai/drop,2019-03-01,Research; Development,"Reading comprehension, Numerical reasoning, Discrete operations, Multi-hop reasoning",Core Performance,Text,Text,New dataset (released with eval),Crowdsourced annotations,Large (100K - 1M samples),"Train (77,409), Dev (9,536), Test (9,622)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives paragraph and question 2. Model generates answer (number, span, or date) 3. F1 and Exact Match metrics computed",Outputs,False,,"Crowdsourced questions with verification, requires discrete reasoning operations confirmed through analysis",unknown,BERT: 47.0 F1 RoBERTa: 80.9 F1 GPT-3: 29.0 F1 GPT-4: 80.9 F1 Human performance: 96.4 F1,Multiple answer types; Question type analysis; Human baseline comparison,Requires careful answer extraction; Some questions ambiguous; Numerical reasoning can be brittle; Limited diversity in reasoning types,"SQuAD, NewsQA, NaturalQuestions, NumGLUE, TAT-QA"
+WinoGrande,"A large-scale commonsense reasoning benchmark with 44,000 problems requiring resolving pronoun ambiguity through world knowledge and commonsense.","Allen Institute for AI, University of Washington",https://arxiv.org/abs/1907.10641,https://github.com/allenai/winogrande,2019-07-24,Research; Development,"Commonsense reasoning, Coreference resolution, World knowledge, Causal reasoning",Core Performance; Robustness,Text,Text,New dataset (released with eval),Crowdsourced annotations,Medium (1K - 100K samples),"Train (40,398), Dev (1,267), Test (1,767)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives sentence with pronoun ambiguity 2. Model selects correct referent from 2 options 3. Accuracy computed,Outputs,False,,"Adversarial filtering using language models, crowdsourced generation with validation",unknown,BERT-Large: 59.4% RoBERTa-Large: 79.1% GPT-3: 70.2% GPT-4: 87.5% Human performance: 94.0%,Adversarial filtering; Large-scale dataset; Multiple difficulty levels,Binary choice may be limiting; Adversarial filtering may introduce artifacts; Saturation with modern models; Limited reasoning depth,"Winograd Schema Challenge, COPA, CommonsenseQA, PIQA"
+BBH (BIG-Bench Hard),"A curated subset of 23 challenging tasks from BIG-Bench where language models perform below human raters, focusing on tasks requiring multi-step reasoning.",Google Research,https://arxiv.org/abs/2210.09261,https://github.com/suzgunmirac/BIG-Bench-Hard,2022-10-17,Research; Development,"Complex reasoning, Multi-step thinking, Challenging cognitive tasks, Chain-of-thought reasoning",Core Performance,Text,Text,One or Multiple existing datasets,Multiple/Mixed sources,Medium (1K - 100K samples),"Test (6,511 examples across 23 tasks)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives task from one of 23 challenging categories 2. Model generates answer 3. Performance aggregated across all tasks 4. Chain-of-thought prompting typically used,Outputs,False,,"Tasks selected where models underperform humans, difficulty validated through empirical testing",unknown,PaLM (540B): 56.5% average PaLM (540B) + CoT: 78.1% GPT-4: ~91% Human raters: ~92% average,Multiple tasks; Chain-of-thought evaluation; Human baseline comparison; Cross-model validation,Only 23 tasks (limited coverage); Chain-of-thought prompting required for good performance; Task aggregation methodology; Rapidly saturating with newer models,"BIG-Bench, MMLU, AGIEval, HELM"
+MT-Bench,"A multi-turn conversational benchmark with 80 high-quality multi-turn questions spanning 8 categories, evaluated using GPT-4 as a judge.","UC Berkeley, UCSD, CMU, MBZUAI",https://arxiv.org/abs/2306.05685,https://github.com/lm-sys/FastChat,2023-06-09,Development; Selection,"Multi-turn conversation, Instruction following, Reasoning, Writing, Role-playing, Knowledge",Core Performance; Core Quality Dimensions,Text,Text,New dataset (released with eval),Author-provided,Small (< 1K samples),Test (80 questions with 2 turns each),Fixed data-driven (static test set),Model-based: In the wild,1. Model engages in 2-turn conversation 2. GPT-4 evaluates responses on 10-point scale 3. Scores averaged across turns and categories,Outputs,False,,"Strong correlation with human preferences validated on Chatbot Arena data, GPT-4 judge agreement measured",unknown,Vicuna-13B: 6.39 GPT-3.5-turbo: 7.94 Claude 2: 8.06 GPT-4: 8.99 GPT-4-turbo: 9.32,Multiple categories; Position bias mitigation; Agreement with human ratings validated; Pairwise comparison,Small dataset (80 questions); GPT-4 judge may have biases; Evaluation cost; Model-based judge limitations; Prompt sensitivity,"Chatbot Arena, AlpacaEval, Arena-Hard, LiveBench"
+GPQA (Google-Proof Q&A),"A challenging multiple-choice benchmark of 448 expert-level questions in biology, physics, and chemistry designed to be difficult even for skilled non-experts with internet access.",New York University,https://arxiv.org/abs/2311.12022,https://github.com/idavidrein/gpqa,2023-11-20,Research; Development,"Expert-level knowledge, Scientific reasoning, Domain expertise, Graduate-level understanding",Core Performance; Robustness,Text,Text,New dataset (released with eval),Expert annotations,Small (< 1K samples),"Main (198), Extended (246), Diamond (198 highest quality)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives graduate-level science question 2. Model selects from 4 multiple choice options 3. Accuracy computed 4. Questions validated to be difficult for non-experts with Google access,Outputs,False,,"PhD-level experts write and validate questions, non-expert validators with Google access achieve <35% accuracy",unknown,Random: 25% Non-expert w/ Google: 34% Expert: 81% GPT-4: 39% Claude 3 Opus: 59.4% GPT-4o: 53.6%,Expert validation; Non-expert baseline; Multiple subject areas; Quality tiers (Diamond subset),Small dataset size; Limited to 3 scientific domains; Multiple choice format; High cost of expert question creation; Cultural bias toward Western scientific education,"MMLU-Pro, JEE-Advanced, SciBench, LiveBench"
+IFEval (Instruction-Following Eval),"A benchmark of 541 prompts with verifiable instructions testing models' ability to follow precise formatting, length, and structural constraints.",Google DeepMind,https://arxiv.org/abs/2311.07911,https://github.com/google-research/google-research/tree/master/instruction_following_eval,2023-11-13,Research; Development; Selection,"Instruction following, Constraint satisfaction, Format compliance, Precise control",Core Performance; Robustness,Text,Text,New dataset (released with eval),Programmatically generated,Small (< 1K samples),Test (541 prompts with ~25 verifiable instructions),Fixed data-driven (static test set),Automatic (Reference-free),"1. Model receives prompt with verifiable instructions (e.g., 'respond in exactly 3 paragraphs', 'include word X at least 5 times') 2. Model generates response 3. Programmatic checks verify instruction compliance 4. Strict and loose accuracy metrics computed",Outputs,False,,"Verifiable instructions with programmatic checking, no ambiguity in correctness",unknown,GPT-3.5: 57.4% strict GPT-4: 76.9% strict Claude 2: 66.7% strict Gemini Ultra: 79.4% strict,Strict and loose metrics; Multiple instruction types; Programmatic verification; No human evaluation needed,Limited to verifiable instructions only; May not reflect realistic usage; Some instructions may be conflicting; Excludes semantic quality assessment,"MT-Bench, AlpacaEval, InstructGPT evals, FollowBench"
+JailbreakBench,"An Open Robustness Benchmark for Jailbreaking Large Language Models","Patrick Chao et al.",https://arxiv.org/abs/2404.01318,https://github.com/JailbreakBench/jailbreakbench,2024-04-01,Evaluation,Robustness; Safety,Adversarial,Text,Text,New dataset,Human annotations,Small (< 1K samples),Train/Test,Static,Automatic (Classifier),Attack success rate measured by classifier,Outputs,True,Private test set,Validated against human judgment,unknown,Llama-2-7b-chat: 45% ASR,Multiple attacks tested,Limited to text modality,"HarmBench, AdvGLUE"
+HarmBench,"A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal","Mantas Mazeika et al.",https://arxiv.org/abs/2402.04249,https://github.com/centerforaisafety/HarmBench,2024-02-06,Evaluation,Safety; Robustness,Adversarial,Text,Text,New dataset,Human annotations,Medium (1K - 10K samples),Train/Val/Test,Static,Automatic (LLM Judge),Comparison against refusal baseline,Outputs,True,Held-out behaviors,High agreement with human annotators,unknown,GPT-4: 90% Refusal,Automated Red Teaming,Focuses on refusal rather than helpfulness,"JailbreakBench, RealToxicityPrompts"
+AdvGLUE,"Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models","Boxin Wang et al.",https://arxiv.org/abs/2111.02840,https://github.com/rbx0509/AdvGLUE,2021-11-04,Evaluation,Robustness,Adversarial,Text,Text,Modified GLUE,Perturbed data,Medium,Dev/Test,Static,Automatic,Accuracy under adversarial perturbation,Outputs,False,,N/A,unknown,BERT: 40% drop,Word-level perturbations,May not transfer to all models,"GLUE, SuperGLUE"
+The Pile Extraction,"Extracting Training Data from Large Language Models","Nicholas Carlini et al.",https://arxiv.org/abs/2012.07805,https://github.com/google-research/lm-extraction-benchmark,2020-12-14,Research,Privacy; Memorization,Memorization,Text,Text,The Pile,Model outputs,Huge,N/A,Dynamic,Automatic,Exact match of training data sequences,Weights,False,,N/A,unknown,GPT-2: High memorization,Prefix attacks,Requires access to training data for verification,"RealToxicityPrompts"
+Carlini Extraction,"Quantifying Memorization Across Neural Language Models","Nicholas Carlini et al.",https://arxiv.org/abs/2202.07646,,2022-02-15,Research,Privacy; Memorization,Memorization,Text,Text,C4,Model outputs,Huge,N/A,Dynamic,Automatic,Eidetic memorization metric,Weights,False,,N/A,unknown,T5: Variable memorization,Varying model scale,Computationally expensive,"The Pile Extraction"
+Min-K% Probes,"Detecting Pretraining Data from Large Language Models","Weijia Shi et al.",https://arxiv.org/abs/2310.16789,https://github.com/swj0419/detect-pretrain,2023-10-25,Evaluation,Leakage; Copyright,Leakage/Contamination,Text,Score,WikiMIA,Log-likelihood,Medium,N/A,Static,Automatic,Likelihood ratio test,Logprobs,True,WikiMIA split,N/A,unknown,Llama-2: High detection,Paraphrase attacks,Requires logprobs,"Contamination Detector"
+Contamination Detector,"General methodology for detecting test set contamination","Oren et al.",https://arxiv.org/abs/2310.16789,,2023-10-01,Evaluation,Leakage,Leakage/Contamination,Text,Score,Various benchmarks,N-gram overlap,Various,N/A,Static,Automatic,N-gram overlap analysis,Outputs,False,,N/A,unknown,GPT-4: Detected contamination,N/A,Heuristic based,"Min-K% Probes"
+PrivacyBench,"Evaluating Privacy Leakage in Large Language Models","Various",https://arxiv.org/abs/2311.04044,https://github.com/PrivacyBench/PrivacyBench,2023-11-07,Evaluation,Privacy,Privacy,Text,Text,Enron Email,PII extraction,Medium,Test,Static,Automatic,PII extraction rate,Outputs,True,Private PII set,N/A,unknown,Llama-2: Leaks emails,Prompt engineering,Synthetic PII may differ from real,"PII Detection"
+PII Detection,"Detecting Personally Identifiable Information in LLM Outputs","Various",,,2023-01-01,Evaluation,Privacy,Privacy,Text,Text,Synthetic PII,Tagged text,Large,Train/Test,Static,Automatic,F1 score on PII tags,Outputs,False,,N/A,unknown,BERT-NER: 95% F1,Context variation,Definition of PII varies,"PrivacyBench"
+TruthfulQA,"Measuring How Models Mimic Human Falsehoods","Stephanie Lin et al.",https://arxiv.org/abs/2109.07958,https://github.com/sylinrl/TruthfulQA,2021-09-08,Evaluation,Truthfulness; Interpretability,Interpretability,Text,Text,Wikipedia/Common misconceptions,QA pairs,Small (817 questions),Validation,Static,Automatic (GPT-3 judge) + Human,Accuracy on truthful answers,Outputs,False,,High human agreement,unknown,GPT-3: 58% Truthful,Prompt variation,Cultural bias in truth definitions,"HaluEval, FactualityPrompts"
+Interpretability Benchmark,"Suite for evaluating saliency maps and feature attribution","Various",,,2023-06-01,Research,Interpretability,Interpretability,Text/Image,Heatmap,Various,Attribution scores,Medium,N/A,Static,Automatic,Faithfulness and Plausibility metrics,Weights,False,,N/A,unknown,ResNet: High faithfulness,Perturbation tests,Metrics can be unstable,"TruthfulQA"
+LLMPerf,"A Benchmark for LLM Inference Performance","Ray Team",https://github.com/ray-project/llmperf,https://github.com/ray-project/llmperf,2023-10-01,Evaluation,Efficiency,Efficiency,Text,Metrics,Synthetic loads,Tokens/sec,N/A,N/A,Dynamic,Automatic,Throughput and Latency measurement,API/Weights,False,,N/A,unknown,Llama-2-70b: 10 tok/sec,Concurrent requests,Hardware dependent,"Token/sec"
+Token/sec Benchmark,"Standard throughput measurement for LLMs","Community",,,2023-01-01,Evaluation,Efficiency,Efficiency,Text,Metrics,Various,Tokens/sec,N/A,N/A,Dynamic,Automatic,Tokens per second generation,API,False,,N/A,unknown,GPT-4: 20 tok/sec,Batch size variation,Varies by provider,"LLMPerf"
+CL-Benchmark,"Continual Learning Benchmark for Language Models","ContinualAI",https://github.com/ContinualAI/avalanche,https://github.com/ContinualAI/avalanche,2022-01-01,Research,Retrainability,Retrainability/Continual Learning,Text,Text,Split MNIST/Cifar/Text,Accuracy over time,Medium,Sequential,Dynamic,Automatic,Forgetting rate and Forward transfer,Weights,False,,N/A,unknown,EWC: Reduces forgetting,Task ordering,Requires training access,"L2M"
+Omniglot,"Human-level concept learning through probabilistic program induction","Brenden Lake et al.",https://github.com/brendenlake/omniglot,https://github.com/brendenlake/omniglot,2015-12-11,Research,Meta-Learning,Meta-Learning,Image,Class,Handwritten characters,Classification,Small (1623 chars),Background/Evaluation,Static,Automatic,One-shot classification accuracy,Outputs,False,,N/A,unknown,Human: 95%,Few-shot variation,Simple visual domain,"Meta-Dataset"
+Meta-Dataset,"A Dataset of Datasets for Learning to Learn from Few Examples","Triantafillou et al.",https://arxiv.org/abs/1903.03096,https://github.com/google-research/meta-dataset,2019-03-07,Research,Meta-Learning,Meta-Learning,Image,Class,ImageNet/CUB/Aircraft,Classification,Large,Train/Val/Test,Static,Automatic,Few-shot accuracy across domains,Outputs,False,,N/A,unknown,Prototypical Networks: 60%,Cross-domain shift,Computationally intensive,"Omniglot"
+BBQ,"Bias Benchmark for QA","Alicia Parrish et al.",https://arxiv.org/abs/2110.08193,https://github.com/nyu-mll/BBQ,2021-10-15,Evaluation,Fairness,Fairness,Text,Text,Hand-crafted templates,QA pairs,Medium (58K examples),Test,Static,Automatic,Accuracy difference between groups,Outputs,False,,N/A,unknown,UnifiedQA: Shows bias,Ambiguous contexts,US-centric social biases,"CrowS-Pairs, WinoBias"
+CrowS-Pairs,"Crowdsourced Stereotype Pairs","Nikita Nangia et al.",https://arxiv.org/abs/2010.00133,https://github.com/nyu-mll/crows-pairs,2020-10-01,Evaluation,Fairness,Fairness,Text,Score,Crowdsourced,Sentence pairs,Small (1508 pairs),Test,Static,Automatic,Preference for stereotypical sentence,Logprobs,False,,N/A,unknown,BERT: 60% Stereotypical,N/A,Crowdworker bias,"BBQ, WinoBias"
+WinoBias,"Gender Bias in Coreference Resolution","Jieyu Zhao et al.",https://arxiv.org/abs/1804.06876,https://github.com/uclanlp/corefBias,2018-04-18,Evaluation,Fairness,Fairness,Text,Text,WinoGrad schema,Coreference,Small (3160 sentences),Dev/Test,Static,Automatic,F1 difference between gender swaps,Outputs,False,,N/A,unknown,CoreNLP: Gender bias present,N/A,Binary gender only,"BBQ, CrowS-Pairs"
+SafetyBench,"A Comprehensive Safety Benchmark for Large Language Models","Zhexin Zhang et al.",https://arxiv.org/abs/2309.07045,https://github.com/thu-coai/SafetyBench,2023-09-13,Evaluation,Safety,Safety,Text,Text,Multiple safety datasets,Multiple choice,Large (11K questions),Dev/Test,Static,Automatic,Accuracy on safe answers,Outputs,False,,N/A,unknown,GPT-4: 89% Safe,Chinese/English,Multiple choice limitation,"Do Not Answer, RealToxicityPrompts"
+Do Not Answer,"A Dataset for Evaluating Safeguards in LLMs","Yuxia Wang et al.",https://arxiv.org/abs/2308.13387,https://github.com/Libr-AI/do-not-answer,2023-08-25,Evaluation,Safety,Safety,Text,Text,Hand-crafted harmful instructions,Refusal/Response,Small (939 instructions),Test,Static,Automatic (GPT-4 Judge),Refusal rate and harmfulness score,Outputs,False,,High agreement,unknown,Llama-2: High refusal,Jailbreaks,Judge bias,"SafetyBench"
+RealToxicityPrompts,"Evaluating Neural Toxic Degeneration in Language Models","Samuel Gehman et al.",https://arxiv.org/abs/2009.11462,https://github.com/allenai/real-toxicity-prompts,2020-09-24,Evaluation,Safety,Safety,Text,Text,Web text,Continuation,Large (100K prompts),Train/Test,Static,Automatic (Perspective API),Toxicity score of continuation,Outputs,False,,N/A,unknown,GPT-2: Toxic outputs,Top-k sampling,Perspective API limitations,"SafetyBench, Do Not Answer"
+`;
+
+// 4. Parse CSV
+function parseCSV(csv) {
+  const lines = csv.trim().split('\n');
+  const headers = lines[0].split(',').map(h => h.trim());
+  
+  const result = [];
+  
+  for (let i = 1; i < lines.length; i++) {
+    const line = lines[i];
+    if (!line.trim()) continue;
+    
+    const row = {};
+    let currentVal = '';
+    let inQuotes = false;
+    let headerIndex = 0;
+    
+    for (let j = 0; j < line.length; j++) {
+      const char = line[j];
+      
+      if (char === '"') {
+        inQuotes = !inQuotes;
+      } else if (char === ',' && !inQuotes) {
+        row[headers[headerIndex]] = currentVal.trim();
+        headerIndex++;
+        currentVal = '';
+      } else {
+        currentVal += char;
+      }
+    }
+    row[headers[headerIndex]] = currentVal.trim();
+    result.push(row);
+  }
+  return result;
+}
+
+const factsheets = parseCSV(csvContent);
+
+// 5. Map Benchmarks to Categories
+const categoryBenchmarks = {};
+CATEGORIES.forEach(c => categoryBenchmarks[c] = []);
+
+// Helper to normalize category names from CSV to our schema
+function normalizeCategory(csvCats) {
+  const cats = [];
+  if (!csvCats) return cats;
+  if (csvCats.includes('Core Performance')) cats.push('Core Performance');
+  if (csvCats.includes('Robustness')) cats.push('Robustness');
+  if (csvCats.includes('Fairness')) cats.push('Fairness');
+  if (csvCats.includes('Safety')) cats.push('Safety');
+  if (csvCats.includes('Calibration')) cats.push('Calibration');
+  if (csvCats.includes('Core Quality Dimensions')) cats.push('Core Quality Dimensions');
+  if (csvCats.includes('Adversarial')) cats.push('Adversarial');
+  if (csvCats.includes('Memorization')) cats.push('Memorization');
+  if (csvCats.includes('Leakage/Contamination')) cats.push('Leakage/Contamination');
+  if (csvCats.includes('Privacy')) cats.push('Privacy');
+  if (csvCats.includes('Interpretability')) cats.push('Interpretability');
+  if (csvCats.includes('Efficiency')) cats.push('Efficiency');
+  if (csvCats.includes('Retrainability')) cats.push('Retrainability');
+  if (csvCats.includes('Meta-Learning')) cats.push('Meta-Learning');
+  return cats;
+}
+
+// Populate from CSV
+factsheets.forEach(f => {
+  const cats = normalizeCategory(f.functional_props);
+  cats.forEach(c => {
+    if (categoryBenchmarks[c]) {
+        categoryBenchmarks[c].push(f);
+    } else {
+        console.warn(`Warning: Category "${c}" not found in CATEGORIES list.`);
+    }
+  });
+});
+
+console.log("Parsed " + factsheets.length + " rows from CSV.");
+CATEGORIES.forEach(c => {
+    console.log(`Category "${c}": ${categoryBenchmarks[c].length} benchmarks.`);
+});
+
+// Helper to get metric config based on benchmark title
+function getMetricConfig(title) {
+  const lowerTitle = title.toLowerCase();
+  
+  if (lowerTitle.includes('mt-bench')) {
+    return {
+      evaluation_description: "Multi-turn conversation quality score (1-10)",
+      lower_is_better: false,
+      score_type: "continuous",
+      min_score: 1,
+      max_score: 10,
+      unit: "points"
+    };
+  }
+  
+  if (lowerTitle.includes('llmperf') || lowerTitle.includes('token/sec')) {
+    return {
+      evaluation_description: "Inference throughput (tokens/sec)",
+      lower_is_better: false,
+      score_type: "continuous",
+      min_score: 0,
+      max_score: 200,
+      unit: "tok/s"
+    };
+  }
+  
+  if (lowerTitle.includes('realtoxicityprompts')) {
+    return {
+      evaluation_description: "Average Toxicity Score",
+      lower_is_better: true,
+      score_type: "continuous",
+      min_score: 0,
+      max_score: 1,
+      unit: "score"
+    };
+  }
+
+  if (lowerTitle.includes('human-eval') || lowerTitle.includes('humaneval')) {
+     return {
+      evaluation_description: "Pass@1 Accuracy",
+      lower_is_better: false,
+      score_type: "continuous",
+      min_score: 0,
+      max_score: 1,
+      unit: "pass@1"
+    };
+  }
+  
+  // Default 0-1 accuracy
+  return {
+      evaluation_description: `${title} Standard Accuracy`,
+      lower_is_better: false,
+      score_type: "continuous",
+      min_score: 0,
+      max_score: 1,
+      unit: "accuracy"
+  };
+}
+
+function generateScore(config, modelName) {
+  let baseScore;
+  
+  // Make some models better than others generally
+  const isStrongModel = modelName.includes('GPT-4') || modelName.includes('Claude 3.5') || modelName.includes('Llama 3 70B');
+  const performanceFactor = isStrongModel ? 0.8 : 0.5;
+  const variance = Math.random() * 0.2;
+  
+  let normalizedScore = performanceFactor + variance; // 0.5 to 1.0 roughly
+  if (normalizedScore > 1) normalizedScore = 0.95 + Math.random() * 0.04;
+  
+  if (config.lower_is_better) {
+    // Invert for lower is better (e.g. toxicity)
+    // Strong models should have LOW score.
+    // normalizedScore is high for strong models.
+    // So we want (1 - normalizedScore) roughly.
+    let val = 1 - normalizedScore;
+    if (val < 0) val = 0.01;
+    
+    // Scale to min/max
+    return config.min_score + (val * (config.max_score - config.min_score));
+  } else {
+    // Higher is better
+    return config.min_score + (normalizedScore * (config.max_score - config.min_score));
+  }
+}
+
+// 6. Generate JSON files
+const outputDir = path.join(__dirname, '../public/benchmarks');
+if (!fs.existsSync(outputDir)) {
+  fs.mkdirSync(outputDir, { recursive: true });
+}
+
+MODELS.forEach(model => {
+  const evaluationResults = [];
+  
+  CATEGORIES.forEach(category => {
+    const available = categoryBenchmarks[category];
+    if (!available || available.length === 0) return;
+    
+    // Select 60-100%
+    const percentage = 0.6 + Math.random() * 0.4;
+    const count = Math.max(1, Math.floor(available.length * percentage));
+    
+    // Shuffle and slice
+    const selected = available.sort(() => 0.5 - Math.random()).slice(0, count);
+    
+    selected.forEach(bench => {
+      // Determine metric config
+      const metricConfig = getMetricConfig(bench.title);
+      
+      // Generate a random score based on config and model
+      const score = generateScore(metricConfig, model.name);
+      
+      // Generate details based on score
+      const details = {};
+      if (metricConfig.unit === 'points') {
+         details["Turn 1"] = Math.max(metricConfig.min_score, score - 0.5);
+         details["Turn 2"] = Math.min(metricConfig.max_score, score + 0.5);
+      } else {
+         details["subtask_a"] = score;
+         details["subtask_b"] = score;
+      }
+
+      evaluationResults.push({
+        evaluation_name: bench.title,
+        metric_config: metricConfig,
+        score_details: {
+          score: score,
+          details: details
+        },
+        factsheet: {
+          purpose: bench.purpose,
+          principles_tested: bench.principles_tested,
+          functional_props: bench.functional_props,
+          input_modality: bench.input_modality,
+          output_modality: bench.output_modality,
+          input_source: bench.input_source,
+          output_source: bench.output_source,
+          size: bench.size,
+          splits: bench.splits,
+          design: bench.design,
+          judge: bench.judge,
+          protocol: bench.protocol,
+          model_access: bench.model_access,
+          has_heldout: bench.has_heldout === 'True',
+          alignment_validation: bench.alignment_validation,
+          baseline_models: bench.baseline_models,
+          robustness_measures: bench.robustness_measures,
+          known_limitations: bench.known_limitations,
+          benchmarks_list: bench.benchmarks_list
+        }
+      });
+    });
+  });
+
+  // Generate dummy sample data
+  const sampleData = [];
+  for (let i = 0; i < 5; i++) {
+    sampleData.push({
+      sample_id: `sample_${i}`,
+      input: `Test input question ${i} for ${model.name}...`,
+      ground_truth: `Expected answer ${i}`,
+      response: `Model generated response ${i} which is mostly correct...`,
+      score: 0.85
+    });
+  }
+  
+  const jsonContent = {
+    schema_version: "0.1.0",
+    evaluation_id: `${model.id.replace('/', '-')}-${Date.now()}`,
+    retrieved_timestamp: new Date().toISOString(),
+    source_data: {
+        dataset_name: "Demo Benchmark Suite",
+        samples_number: 1000
+    },
+    source_metadata: {
+      source_name: "Demo Evaluation Suite",
+      source_type: "evaluation_run",
+      source_organization_name: "General Eval Card Demo",
+      evaluator_relationship: "third_party"
+    },
+    model_info: {
+      name: model.name,
+      id: model.id,
+      developer: model.developer,
+      inference_platform: "demo",
+      inference_engine: model.engine,
+      model_version: model.version,
+      additional_details: {
+        params: model.params,
+        architecture: model.architecture,
+        release_date: model.release_date
+      },
+      modalities: {
+        input: ["text"],
+        output: ["text"]
+      }
+    },
+    evaluation_results: evaluationResults,
+    detailed_evaluation_results_per_samples: sampleData
+  };
+  
+  // Use a consistent filename based on ID
+  const filename = `${model.id.replace('/', '-')}.json`;
+  fs.writeFileSync(path.join(outputDir, filename), JSON.stringify(jsonContent, null, 2));
+  console.log(`Generated ${filename} with ${evaluationResults.length} evaluations.`);
+});
+
+console.log("Done generating dummy data.");

scripts/migrate-to-benchmarks.mjs

@@ -0,0 +1,180 @@
+#!/usr/bin/env node
+
+/**
+ * Migration script to convert old checkbox-based evaluations to new benchmark-first format
+ * 
+ * Usage: node scripts/migrate-to-benchmarks.mjs
+ */
+
+import fs from 'fs/promises'
+import path from 'path'
+
+// Mapping of old categories to new categories
+const CATEGORY_MAPPING = {
+  'language-communication': 'language-communication',
+  'problem-solving': 'reasoning',
+  'creativity-innovation': 'creativity',
+  'learning-memory': 'knowledge',
+  'perception-vision': 'vision',
+  'social-intelligence': 'social-intelligence',
+  'harmful-content': 'toxicity',
+  'bias-fairness': 'bias-fairness',
+  'information-integrity': 'truthfulness',
+  'security-robustness': 'robustness',
+}
+
+// Map old benchmark names to standardized ones
+const BENCHMARK_MAPPING = {
+  'MMLU, HellaSwag, ARC-Challenge, WinoGrande': 'MMLU',
+  'TruthfulQA': 'TruthfulQA',
+  'BBH': 'BBH',
+}
+
+async function migrateEvaluation(oldData) {
+  const evaluations = []
+  
+  // Process each category
+  for (const [categoryId, categoryData] of Object.entries(oldData.categoryEvaluations || {})) {
+    const benchmarkSources = categoryData.benchmarkSources || {}
+    
+    // Process each benchmark question
+    for (const [questionId, sources] of Object.entries(benchmarkSources)) {
+      for (const source of sources) {
+        if (!source.benchmarkName) continue
+        
+        const evaluation = {
+          schema_version: '0.1',
+          evaluation_id: `${oldData.id}_${categoryId}_${questionId}_${source.id}`,
+          retrieved_timestamp: String(Date.now() / 1000),
+          
+          source_data: {
+            dataset_name: source.benchmarkName || 'Unknown',
+            samples_number: 0, // Unknown from old format
+            dataset_version: source.version || 'unknown',
+          },
+          
+          source_metadata: {
+            source_name: oldData.evaluator || 'Unknown',
+            source_type: source.sourceType === 'external' ? 'evaluation_run' : 'model_card',
+            source_organization_name: oldData.provider || 'Unknown',
+            source_organization_url: oldData.url || '',
+            evaluator_relationship: source.sourceType === 'external' ? 'third_party' : 'first_party',
+            source_url: source.url || '',
+          },
+          
+          model_info: {
+            name: oldData.systemName,
+            id: oldData.id,
+            developer: oldData.provider,
+            model_version: oldData.version || oldData.modelTag,
+            modalities: {
+              input: oldData.inputModalities || ['text'],
+              output: oldData.outputModalities || ['text'],
+            },
+          },
+          
+          evaluation_results: [
+            {
+              evaluation_name: `${source.benchmarkName} - ${source.metrics || 'accuracy'}`,
+              evaluation_timestamp: String(Date.now() / 1000),
+              metric_config: {
+                evaluation_description: source.metrics || 'Accuracy',
+                lower_is_better: false,
+                score_type: 'continuous',
+                min_score: 0.0,
+                max_score: 1.0,
+              },
+              score_details: {
+                score: parseScore(source.score),
+                details: {},
+              },
+              detailed_evaluation_results_url: source.url,
+              generation_config: {
+                generation_args: {},
+                additional_details: source.taskVariants || '',
+              },
+            },
+          ],
+        }
+        
+        evaluations.push(evaluation)
+      }
+    }
+  }
+  
+  return evaluations
+}
+
+function parseScore(scoreString) {
+  if (!scoreString) return 0
+  
+  // Extract first number from string like "87.4% MMLU"
+  const match = scoreString.match(/(\d+\.?\d*)/)
+  if (!match) return 0
+  
+  const value = parseFloat(match[1])
+  
+  // If it looks like a percentage, convert to 0-1
+  if (value > 1 && value <= 100) {
+    return value / 100
+  }
+  
+  return value
+}
+
+async function main() {
+  const oldEvalsDir = './public/evaluations'
+  const newBenchmarksDir = './public/benchmarks'
+  
+  try {
+    // Create benchmarks directory if it doesn't exist
+    await fs.mkdir(newBenchmarksDir, { recursive: true })
+    
+    // Read all old evaluation files
+    const files = await fs.readdir(oldEvalsDir)
+    const jsonFiles = files.filter(f => f.endsWith('.json'))
+    
+    console.log(`Found ${jsonFiles.length} evaluation files to migrate`)
+    
+    for (const file of jsonFiles) {
+      try {
+        const filePath = path.join(oldEvalsDir, file)
+        const content = await fs.readFile(filePath, 'utf-8')
+        const oldData = JSON.parse(content)
+        
+        console.log(`\nMigrating ${file}...`)
+        console.log(`  System: ${oldData.systemName}`)
+        
+        const evaluations = await migrateEvaluation(oldData)
+        
+        console.log(`  Generated ${evaluations.length} benchmark evaluations`)
+        
+        // Write each evaluation as a separate file, or combine them
+        // For now, we'll write the first one as a sample
+        if (evaluations.length > 0) {
+          const outputFile = path.join(newBenchmarksDir, file)
+          await fs.writeFile(
+            outputFile,
+            JSON.stringify(evaluations[0], null, 2)
+          )
+          console.log(`  Wrote to ${outputFile}`)
+        }
+      } catch (error) {
+        console.error(`  Error migrating ${file}:`, error.message)
+      }
+    }
+    
+    console.log('\nMigration complete!')
+    console.log('\nNote: This is a basic migration. You should:')
+    console.log('1. Review generated files for accuracy')
+    console.log('2. Add missing metadata (sample counts, confidence intervals)')
+    console.log('3. Verify score conversions')
+    console.log('4. Add detailed sample results if available')
+    
+  } catch (error) {
+    console.error('Migration failed:', error)
+    process.exit(1)
+  }
+}
+
+main()

scripts/update-factsheets-from-csv.js

@@ -0,0 +1,164 @@
+
+const fs = require('fs');
+const path = require('path');
+
+const csvContent = `title,subtitle,authors,link,code_link,date,purpose,principles_tested,functional_props,input_modality,output_modality,input_source,output_source,size,splits,design,judge,protocol,model_access,has_heldout,heldout_details,alignment_validation,is_valid,baseline_models,robustness_measures,known_limitations,benchmarks_list
+DigiData: Training and Evaluating General-Purpose Mobile Control Agents,"We present DigiData-Bench, a benchmark for evaluating mobile control agents on real-world complex tasks. We demonstrate that the commonly used step-accuracy metric falls short in reliably assessing mobile control agents and, to address this, we propose dynamic evaluation protocols and AI-powered evaluations as rigorous alternatives for agent assessment.",Meta FAIR,https://arxiv.org/abs/2511.07413,https://github.com/facebookresearch/DigiData,2025-11-08,Development; Research,mobile control agents,Core Performance,Text + Vision,Actions,New dataset (released with eval),Human annotations,Small (< 1K samples),,Dynamic data-driven (adaptive/interactive),Model-based: In the wild,"1. Given a goal and a trajectory produced by an agent, an LLM judge classifies whether it successfully achieved 2. the goal. We use LLM judges relying on both the screenshot and UI tree.",Outputs,False,,human operator is asked to judge whether the trajectory successfully achieves the goal or not. Then we assess the alignment between human judge and LLM judge to ensure a high alignment,unknown,,Prompt variations tested; Multiple runs per sample; Temperature sensitivity tested; Repeated evaluations; Ablation studies; Inter-rater reliability; Significance testing,"-live and dynamic environment introduces uncontrollable factors (feature deprecation, version changes etc) that makes certain tasks unavailable after a period -sizable efforts required to manually set up prework in the environment ","AndroidControl, Android in the wild"
+IntPhys 2,"IntPhys 2 offers a comprehensive suite of tests, based on the violation of expectation framework, that challenge models to differentiate between possible and impossible events within controlled and diverse virtual environments.",Meta FAIR,https://arxiv.org/abs/2506.09849,https://github.com/facebookresearch/IntPhys2,2025-05-31,Research,Intuitive Physics,Core Performance,Video,Scores/Embeddings,New dataset (released with eval),Simulation-based,Medium (1K - 100K samples),"Debug Set: 60 videos for Model calibration Main Set: 1,012 videos as Main evaluation set Held-Out Set: 344 videos as Test set",Fixed data-driven (static test set),Automatic (Reference-based),Feed a video to the model Ask the model using specific prompts wether the video is physically plausible or not Check if the model's answer match the ground truth label ,Outputs,True,Held-Out Set: 344 videos as Test set,We ran human baselines to ensure that this task is easy for humans. ,unknown,- Human baseline: 96% - Best model baseline: V-JEPA2 56%,Prompt variations tested; Multiple runs per sample; Temperature sensitivity tested,"Model can be very sensitive to the way they are prompted. Also the compression artefact of the video can also impact the results. Those are mostly limitations on the model sides, since human are not sensitive to those. ","IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning, Riochet et al. 2020"
+ImageNet,"A large-scale hierarchical image database designed for visual object recognition research, containing over 14 million images across 20,000+ categories.",Princeton University,https://ieeexplore.ieee.org/document/5206848,https://image-net.org/,2009-01-01,Research; Development,"Object recognition, Visual categorization, Multi-class classification",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Huge (> 10M samples),"Train, Validation, Test",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives an image as input 2. Model predicts class label from 1000 classes 3. Top-1 and Top-5 accuracy computed against ground truth,Outputs,True,"Private test set maintained by organizers, used for annual ILSVRC competitions","Expert curation of hierarchical categories, manual verification of labels, consistency checks across similar categories",unknown,AlexNet: 63.3% top-1 VGG: 71.5% top-1 ResNet-50: 76.1% top-1 ResNet-152: 78.3% top-1 EfficientNet-B7: 84.3% top-1 Human performance: ~95% top-5,Multiple runs per sample; Significance testing; Inter-rater reliability,Class imbalance in some categories; Label noise in training set; Some ambiguous images with multiple valid labels; Bias towards certain object viewpoints and contexts,"ImageNet-V2, ImageNet-C, ImageNet-R, ImageNet-A, ImageNet-Sketch"
+COCO (Common Objects in Context),"A large-scale object detection, segmentation, and captioning dataset containing 330K images with 80 object categories, designed to advance scene understanding.",Microsoft Research,https://arxiv.org/abs/1405.0312,https://github.com/cocodataset/cocoapi,2014-01-01,Research; Development; Selection,"Object detection, Instance segmentation, Keypoint detection, Panoptic segmentation, Image captioning",Core Performance,Vision (Image),Text; Structured Data,New dataset (released with eval),Human annotations,Large (100K - 1M samples),"Train, Validation, Test",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives image as input 2. Model outputs bounding boxes, segmentation masks, or captions 3. Metrics computed: mAP for detection, IoU for segmentation, BLEU/CIDEr for captioning",Outputs,True,"Test-dev and test-challenge splits maintained privately, submissions via evaluation server","Multi-annotator consensus for instance annotations, quality control through redundant labeling",unknown,Faster R-CNN: 42.0 mAP Mask R-CNN: 37.1 mask mAP YOLOv8: 53.9 mAP Human performance (detection): ~70 mAP,Inter-rater reliability; Multiple runs per sample; Confidence intervals,Small object detection remains challenging; Occlusion handling difficulties; Dataset bias toward certain object contexts; Annotation inconsistencies in crowded scenes,"LVIS, Objects365, Open Images, Visual Genome"
+CIFAR-10 and CIFAR-100,"Small-scale image classification benchmarks with 60K 32x32 color images in 10 (CIFAR-10) or 100 (CIFAR-100) classes, widely used for algorithm development and testing.",University of Toronto,https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf,https://github.com/pytorch/vision,2009-04-08,Research; Development; Selection,"Image classification, Transfer learning, Generalization",Core Performance; Robustness,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (50K), Test (10K)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives 32x32 RGB image 2. Model predicts class label (10 or 100 classes) 3. Classification accuracy computed,Outputs,False,,"Systematic sampling from larger dataset (80 million tiny images), human verification of labels",unknown,CIFAR-10: ResNet-56: 93.03% Wide ResNet-28-10: 96.11% PyramidNet: 96.54%  CIFAR-100: ResNet-56: 71.35% Wide ResNet-28-10: 81.15% PyramidNet: 83.78%,Multiple runs per sample; Seed variation tested; Ablation studies,Low resolution (32x32) limits fine-grained recognition; Some label noise in CIFAR-100; Dataset size enables memorization in large models; Limited diversity in poses and contexts,"CIFAR-10-C, CIFAR-10.1, CIFAR-100-C, STL-10, Tiny ImageNet"
+Pascal VOC,"A pioneering object detection and segmentation benchmark with 20 object classes, providing standardized evaluation for visual recognition tasks.",University of Oxford,https://link.springer.com/article/10.1007/s11263-009-0275-4,https://github.com/pytorch/vision,2007-01-01,Research; Development; Selection,"Object detection, Semantic segmentation, Instance segmentation, Action classification",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train, Validation, Test",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives image as input 2. Model predicts bounding boxes and class labels 3. mAP computed at IoU threshold 0.5,Outputs,True,"Test set labels held privately, evaluation via submission to organizers (historically)","Careful manual annotation with quality control, multiple annotators for difficult cases",unknown,R-CNN: 58.5 mAP (VOC 2007) Fast R-CNN: 70.0 mAP Faster R-CNN: 75.9 mAP YOLOv3: 78.6 mAP,Inter-rater reliability; Significance testing,Limited to 20 classes; Small dataset size by modern standards; Some annotation inconsistencies; Relatively simple backgrounds,"COCO, LVIS, Open Images, Cityscapes"
+Cityscapes,"A large-scale dataset for urban scene understanding with pixel-level annotations for semantic segmentation, instance segmentation, and depth estimation in autonomous driving contexts.","Daimler AG, MPI for Informatics, TU Darmstadt",https://arxiv.org/abs/1604.01685,https://github.com/mcordts/cityscapesScripts,2016-04-01,Research; Development; Deployment,"Semantic segmentation, Instance segmentation, Scene understanding, Autonomous driving perception",Core Performance; Robustness,Vision (Image); Video,Structured Data,New dataset (released with eval),Expert annotations,Medium (1K - 100K samples),"Train (2975), Val (500), Test (1525)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives street scene image (2048×1024) 2. Model predicts per-pixel semantic class (19 or 30 classes) 3. IoU and accuracy metrics computed,Outputs,True,"Test set labels private, evaluation via online server with leaderboard","Expert annotators with domain knowledge, multi-pass quality control, consistency verification across video sequences",unknown,FCN-8s: 65.3 mIoU DeepLab v3+: 82.1 mIoU HRNetV2: 83.0 mIoU SegFormer: 84.0 mIoU,Multiple runs per sample; Ablation studies; Significance testing,Limited to European cities; Weather bias (mostly good conditions); Class imbalance for rare objects; Fine annotation boundaries challenging,"ADE20K, KITTI, Mapillary Vistas, BDD100K, nuScenes"
+ADE20K,"A comprehensive scene parsing benchmark with 150 semantic categories, designed for understanding diverse indoor and outdoor scenes with detailed object and part annotations.",MIT CSAIL,https://arxiv.org/abs/1608.05442,https://github.com/CSAILVision/ADE20K,2017-06-01,Research; Development,"Scene parsing, Semantic segmentation, Multi-scale recognition, Part segmentation",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (20K), Val (2K), Test (3K)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives diverse scene image 2. Model predicts per-pixel semantic labels (150 classes) 3. Mean IoU and pixel accuracy computed,Outputs,False,,"Crowdsourced annotations with expert review, hierarchical consistency checks, multi-round verification",unknown,PSPNet: 43.29 mIoU DeepLab v3: 45.65 mIoU UPerNet: 44.85 mIoU SegFormer-B5: 51.8 mIoU,Inter-rater reliability; Multiple runs per sample,Long-tail distribution of object classes; Annotation granularity varies; Some scenes have ambiguous boundaries; Challenging for rare categories,"Cityscapes, Pascal Context, COCO-Stuff, Mapillary Vistas"
+Kinetics,"A large-scale video action recognition dataset with 400/600/700 human action classes, designed to advance video understanding and temporal reasoning.",DeepMind,https://arxiv.org/abs/1705.06950,https://github.com/cvdfoundation/kinetics-dataset,2017-05-01,Research; Development,"Action recognition, Video understanding, Temporal reasoning, Human activity recognition",Core Performance,Video,Structured Data,New dataset (released with eval),Human annotations,Large (100K - 1M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives 10-second video clip 2. Model predicts action class (400/600/700 classes) 3. Top-1 and Top-5 accuracy computed,Outputs,False,,"Human verification of video labels, removal of ambiguous clips, consistency checks for similar actions",unknown,I3D: 71.1% top-1 (K400) SlowFast: 79.8% top-1 (K400) X3D: 80.4% top-1 (K400) VideoMAE: 81.5% top-1 (K400),Multiple runs per sample; Temporal ordering tested,YouTube videos may become unavailable over time; Some action classes overlap or are ambiguous; Camera viewpoint bias; Dataset drift as internet content changes,"UCF-101, HMDB-51, ActivityNet, Something-Something, Moments in Time"
+KITTI,"An autonomous driving benchmark suite providing stereo vision, optical flow, 3D object detection, and tracking datasets collected from real-world driving scenarios.","Karlsruhe Institute of Technology, Toyota Technological Institute",https://www.cvlibs.net/publications/Geiger2012CVPR.pdf,https://github.com/bostondiditeam/kitti,2012-01-01,Research; Development; Deployment,"3D object detection, Stereo vision, Optical flow, Visual odometry, Tracking",Core Performance; Robustness,Vision (Image); Video,Structured Data,New dataset (released with eval),Human annotations; Programmatically generated,Medium (1K - 100K samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives stereo image pair or point cloud 2. Model predicts 3D bounding boxes, orientation, class 3. 3D AP computed at different difficulty levels (easy/moderate/hard)",Outputs,True,"Test set labels held privately, evaluation via online server with public leaderboard","LiDAR ground truth for 3D positions, manual verification of annotations, multi-sensor fusion for accuracy",unknown,"PointPillars: 79.05 AP (Car, Moderate) PV-RCNN: 83.90 AP (Car, Moderate) CenterPoint: 85.15 AP (Car, Moderate)",Multiple difficulty levels; Ablation studies; Significance testing,Limited to specific geographic region; Weather bias (mostly clear); Limited nighttime data; Class imbalance (cars dominate),"nuScenes, Waymo Open Dataset, Argoverse, A2D2, Lyft Level 5"
+Places365,"A scene recognition benchmark with 365 scene categories and over 10 million images, designed to understand high-level visual concepts and environmental context.",MIT CSAIL,https://arxiv.org/abs/1610.02055,https://github.com/CSAILVision/places365,2017-07-01,Research; Development,"Scene recognition, Scene classification, Environmental understanding, Context recognition",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Very Huge (> 100M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives scene image 2. Model predicts scene category (365 classes) 3. Top-1 and Top-5 accuracy computed,Outputs,False,,"Human verification, consistency checks for scene categories, hierarchical taxonomy validation",unknown,ResNet-152: 55.24% top-1 DenseNet-161: 56.12% top-1 ResNeXt-101: 56.05% top-1,Inter-rater reliability; Multiple runs per sample,Some scene categories overlap semantically; Cultural bias in scene definitions; Indoor scenes better represented than outdoor; Ambiguous boundary cases,"SUN397, MIT Indoor 67, Scene-15, ADE20K"
+UCF-101,"An action recognition benchmark with 101 action categories and 13,320 videos collected from YouTube, widely used for video understanding research.",University of Central Florida,https://arxiv.org/abs/1212.0402,https://github.com/pytorch/vision,2012-01-01,Research; Development,"Action recognition, Video classification, Temporal understanding",Core Performance,Video,Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),Train/Test (3 splits provided),Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives video clip 2. Model predicts action class (101 classes) 3. Average accuracy across 3 splits reported,Outputs,False,,"Manual verification of action labels, removal of ambiguous videos",unknown,Two-Stream CNN: 88.0% I3D: 95.6% SlowFast: 96.8% VideoMAE: 97.2%,Multiple evaluation splits; Seed variation tested,YouTube videos may become unavailable; Camera motion and quality vary; Some action classes are very similar; Dataset saturation with modern methods,"HMDB-51, Kinetics, ActivityNet, Something-Something-V2"
+NYU Depth V2,"An RGB-D dataset for indoor scene understanding with 1449 densely labeled pairs of aligned RGB and depth images, designed for depth estimation and semantic segmentation.",New York University,https://cs.nyu.edu/~fergus/datasets/indoor_seg_support.pdf,https://github.com/ankurhanda/nyuv2-meta-data,2012-06-01,Research; Development,"Depth estimation, RGB-D understanding, Indoor scene parsing, 3D reconstruction",Core Performance,Vision (Image); Structured Data,Structured Data,New dataset (released with eval),Expert annotations,Small (< 1K samples),"Train (795), Test (654)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives RGB image 2. Model predicts depth map or semantic segmentation 3. RMSE, absolute relative error, and accuracy metrics computed",Outputs,False,,"Kinect sensor ground truth with manual alignment corrections, multi-view consistency",unknown,Depth Estimation: Eigen et al.: 0.641 RMSE AdaBins: 0.364 RMSE BTS: 0.392 RMSE,Multiple runs per sample; Ablation studies,"Small dataset size; Limited to indoor scenes; Kinect depth sensor limitations (range, IR interference); Mostly residential environments","ScanNet, Matterport3D, SUNRGB-D, KITTI Depth"
+CelebA,"A large-scale face attributes dataset with 202,599 face images annotated with 40 binary attributes, 5 landmark locations, and identity information for face recognition and attribute prediction.",The Chinese University of Hong Kong,https://arxiv.org/abs/1411.7766,https://github.com/tkarras/progressive_growing_of_gans,2015-09-01,Research; Development,"Face attribute recognition, Face detection, Facial landmark detection, Identity recognition",Core Performance; Fairness,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Large (100K - 1M samples),"Train (162,770), Val (19,867), Test (19,962)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives face image 2. Model predicts binary attributes (40 dimensions) 3. Per-attribute accuracy and mean accuracy computed,Outputs,False,,"Manual annotation with quality control, consistency verification across multiple attributes",unknown,LNets+ANet: 87.30% mean accuracy Walk and Learn: 88.06% FaceNet: 89.35%,Inter-rater reliability; Multiple runs per sample,Demographic bias (celebrity images); Some attributes are subjective; Label noise in some attributes; Privacy concerns with celebrity images; Lighting and pose variation,"LFW, VGGFace2, MS-Celeb-1M, FFHQ"
+Visual Genome,"A comprehensive visual understanding dataset with dense annotations of objects, attributes, relationships, and scene graphs across 108K images for structured scene understanding.",Stanford University,https://arxiv.org/abs/1602.07332,https://github.com/ranjaykrishna/visual_genome_python_driver,2016-05-01,Research; Development,"Scene graph generation, Visual relationship detection, Visual question answering, Dense captioning",Core Performance,Vision (Image),Structured Data; Text,MS COCO,Crowdsourced annotations,Large (100K - 1M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives image 2. Model predicts objects, attributes, and relationships 3. Recall@K metrics for scene graph components",Outputs,False,,"Multi-round crowdsourced annotations with validation, consistency checks for relationships",unknown,IMP: 14.6 R@50 (scene graph detection) Motifs: 21.4 R@50 VCTree: 22.0 R@50 GPS-Net: 24.0 R@50,Inter-rater reliability; Multiple runs per sample,Long-tail distribution of relationships; Annotation inconsistencies; Subjective relationship definitions; Incomplete annotations (not all relationships captured),"GQA, Scene Graph Benchmark, VRD, OpenImages V6"
+LVIS (Large Vocabulary Instance Segmentation),A large-scale instance segmentation dataset with 1203 object categories designed to address the long-tail distribution challenge in object recognition.,Facebook AI Research,https://arxiv.org/abs/1908.03195,https://github.com/lvis-dataset/lvis-api,2019-08-01,Research; Development,"Instance segmentation, Long-tail recognition, Object detection, Fine-grained categorization",Core Performance; Robustness,Vision (Image),Structured Data,MS COCO,Expert annotations,Large (100K - 1M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives image 2. Model predicts instance masks and categories (1203 classes) 3. AP computed separately for rare, common, and frequent categories",Outputs,True,"Test set labels private, evaluation via online server","Expert annotators with WordNet taxonomy, quality control for long-tail categories",unknown,Mask R-CNN: 21.2 AP (v1.0) Cascade R-CNN: 26.2 AP Swin Transformer: 50.9 AP,Multiple runs per sample; Ablation studies; Category frequency stratification,Rare categories have very few examples; Annotation cost for 1203 categories; Some category definitions overlap; Challenging for zero-shot generalization,"COCO, Objects365, OpenImages, iNaturalist"
+Mapillary Vistas,"A diverse street-level imagery dataset with pixel-level annotations for 66 object categories, designed for robust semantic segmentation across varied geographic locations and conditions.",Mapillary AB,https://openaccess.thecvf.com/content_ICCV_2017/papers/Neuhold_The_Mapillary_Vistas_ICCV_2017_paper.pdf,https://github.com/mapillary/mapillary_vistas,2017-08-01,Research; Development; Deployment,"Semantic segmentation, Panoptic segmentation, Scene understanding, Robust perception",Core Performance; Robustness,Vision (Image),Structured Data,User-generated content,Expert annotations,Medium (1K - 100K samples),"Train (18K), Val (2K), Test (5K)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives street-level image 2. Model predicts per-pixel semantic labels (66 classes) 3. mIoU computed across all classes,Outputs,True,"Test set labels private, evaluation via online platform","Expert annotators, multi-stage quality control, geographic diversity validation",unknown,PSPNet: 42.7 mIoU DeepLab v3+: 45.8 mIoU HRNetV2: 50.3 mIoU,Geographic diversity tested; Weather variations included; Multiple runs per sample,Varying image quality from crowdsourced data; Camera parameter diversity; Some regions overrepresented; Annotation inconsistencies across diverse scenes,"Cityscapes, BDD100K, IDD, WildDash"
+MPII Human Pose,"A benchmark for human pose estimation with 25K images containing over 40K annotated people with 16 body joints, covering diverse activities and viewpoints.",Max Planck Institute for Informatics,https://openaccess.thecvf.com/content_cvpr_2014/papers/Andriluka_2D_Human_Pose_2014_CVPR_paper.pdf,https://www.mpi-inf.mpg.de/departments/computer-vision-and-machine-learning/software-and-datasets/mpii-human-pose-dataset,2014-06-01,Research; Development,"Human pose estimation, Keypoint detection, Articulated pose estimation, Activity recognition",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (~29K people), Test (~12K people)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives image with person 2. Model predicts 2D joint locations (16 joints) 3. PCKh (Percentage of Correct Keypoints) at various thresholds,Outputs,False,,"Manual annotation with consistency checks, multi-annotator agreement for difficult poses",unknown,Hourglass Network: 90.9 PCKh@0.5 HRNet: 92.3 PCKh@0.5 SimpleBaseline: 91.5 PCKh@0.5,Multiple difficulty levels; Inter-rater reliability; Occlusion analysis,2D annotations only (no 3D); Occlusion and truncation challenges; Some joint definitions ambiguous; Dataset bias toward certain activities,"COCO Keypoints, Human3.6M, PoseTrack, CrowdPose"
+Open Images,"A large-scale multi-task dataset with ~9M images annotated for image classification (20K classes), object detection (600 classes), visual relationships, and segmentation masks.",Google Research,https://arxiv.org/abs/1811.00982,https://github.com/openimages/dataset,2018-01-01,Research; Development; Selection,"Object detection, Image classification, Visual relationship detection, Instance segmentation",Core Performance; Robustness,Vision (Image),Structured Data,New dataset (released with eval),Human annotations; Crowdsourced annotations,Very Huge (> 100M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives image 2. Model performs classification, detection, or segmentation 3. mAP computed for detection, top-k accuracy for classification",Outputs,True,"Test set labels private for challenges, public validation set available","Multi-stage crowdsourced annotation with verification, automated quality filters",unknown,Faster R-CNN: 54.3 mAP (detection) YOLOv4: 55.8 mAP EfficientDet: 56.1 mAP,Multiple runs per sample; Confidence intervals; Large-scale diversity,Long-tail class distribution; Annotation inconsistencies at scale; Some classes poorly defined; Label noise in crowdsourced annotations,"COCO, LVIS, Objects365, ImageNet"
+ScanNet,"A richly-annotated 3D dataset of indoor scenes with RGB-D scans, semantic segmentation, instance segmentation, and 3D object bounding boxes for 1513 scenes.","Stanford University, Princeton University, Technical University of Munich",https://arxiv.org/abs/1702.04405,https://github.com/ScanNet/ScanNet,2017-04-01,Research; Development,"3D scene understanding, Semantic segmentation, Instance segmentation, 3D reconstruction",Core Performance,Vision (Image); Video; Structured Data,Structured Data,New dataset (released with eval),Human annotations; Programmatically generated,Medium (1K - 100K samples),"Train (1201), Val (312), Test (100)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives 3D scene (RGB-D scans) 2. Model predicts semantic labels or instance masks 3. mIoU for semantic segmentation, mAP for instance segmentation",Outputs,True,"Test set labels private, evaluation via online benchmark server","Manual verification of 3D reconstructions, multi-annotator consistency for semantic labels",unknown,PointNet++: 53.5 mIoU SparseConvNet: 72.5 mIoU MinkowskiNet: 73.6 mIoU,Multiple runs per sample; Ablation studies,Limited to indoor scenes; Reconstruction artifacts; Scanning noise and occlusions; Limited scene diversity (mostly offices and apartments),"Matterport3D, S3DIS, 2D-3D-S, ARKitScenes"
+nuScenes,"A large-scale autonomous driving dataset with 1000 scenes (40K frames) featuring 3D object annotations, tracking IDs, and multimodal sensor data including cameras, LiDAR, and radar.",Motional (formerly nuTonomy),https://arxiv.org/abs/1903.11027,https://github.com/nutonomy/nuscenes-devkit,2019-03-26,Research; Development; Deployment,"3D object detection, Multi-object tracking, Prediction, Sensor fusion, Scene understanding",Core Performance; Robustness,Vision (Image); Video; Structured Data,Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (700), Val (150), Test (150)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives multimodal sensor data (cameras, LiDAR, radar) 2. Model predicts 3D bounding boxes and tracking IDs 3. NDS (nuScenes Detection Score) and mAP computed",Outputs,True,"Test set labels private, evaluation via online leaderboard","Expert annotators, multi-sensor consistency checks, temporal coherence validation",unknown,PointPillars: 45.3 NDS CenterPoint: 65.5 NDS BEVFusion: 72.9 NDS,Geographic diversity; Weather conditions; Day/night variations; Multiple runs per sample,"Limited geographic coverage (Boston, Singapore); Sensor calibration challenges; Annotation latency for fast-moving objects; Class imbalance","Waymo Open Dataset, KITTI, Argoverse, Lyft Level 5, A2D2"
+ActivityNet,"A large-scale video dataset for human activity understanding with 200 activity classes and temporal annotations, designed for action recognition and temporal action localization.","Universidad del Norte, KAUST",https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Heilbron_ActivityNet_A_Large-Scale_2015_CVPR_paper.pdf,https://github.com/activitynet/ActivityNet,2015-06-01,Research; Development,"Temporal action detection, Action recognition, Dense video captioning, Video understanding",Core Performance,Video,Structured Data; Text,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (50%), Val (25%), Test (25%)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives untrimmed video 2. Model predicts action segments with class labels 3. mAP at different IoU thresholds (0.5, 0.75, 0.95)",Outputs,True,"Test set used for annual challenges, labels withheld","Multi-annotator temporal boundary agreement, consistency checks for activity definitions",unknown,SSN: 41.3 mAP@0.5 BMN: 50.1 mAP@0.5 TALLFormer: 59.8 mAP@0.5,Inter-rater reliability; Multiple IoU thresholds; Temporal boundary sensitivity,YouTube video availability issues; Temporal boundary ambiguity; Action class overlap; Video quality variance,"THUMOS14, Kinetics, Charades, MultiTHUMOS, AVA"
+DAVIS (Densely Annotated VIdeo Segmentation),A video object segmentation benchmark with high-quality pixel-level annotations for densely segmenting objects in video sequences.,"ETH Zurich",https://arxiv.org/abs/1704.00675,https://github.com/davisvideochallenge/davis,2016-10-01,Research; Development,"Video object segmentation, Temporal consistency, Object tracking, Dense prediction",Core Performance; Robustness,Video,Structured Data,New dataset (released with eval),Human annotations,Small (< 1K samples),"Train/Val (60 sequences), Test-dev (30 sequences)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives video sequence with first frame annotation 2. Model propagates segmentation to subsequent frames 3. J&F metric (region similarity and contour accuracy),Outputs,True,"Test-challenge set for competitions, labels withheld",High-quality manual annotations with temporal consistency verification,unknown,OSVOS: 79.8 J&F STM: 84.3 J&F XMem: 86.2 J&F,Temporal consistency tested; Occlusion robustness; Multiple runs per sample,Small dataset size; Limited object categories; Short video sequences; Primarily objects with clear boundaries,"YouTube-VOS, FBMS, SegTrack, OVIS"
+VQA (Visual Question Answering),"A dataset with open-ended questions about images requiring visual understanding and reasoning, containing 265K images and over 1M questions.","Virginia Tech, Facebook AI Research",https://arxiv.org/abs/1505.00468,https://github.com/GT-Vision-Lab/VQA,2015-10-01,Research; Development,"Visual question answering, Visual reasoning, Multimodal understanding, Common sense reasoning",Core Performance; Robustness,Text + Vision,Text,MS COCO,Crowdsourced annotations,Huge (> 10M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based); Human: Representative sample,1. Model receives image and question 2. Model generates answer 3. Accuracy computed with consensus matching (multiple human answers),Outputs,True,"Test-dev and test-std splits, evaluation via server","Multiple human answers per question (10 answers), consensus-based evaluation",unknown,Bottom-Up Top-Down: 70.3% VILBERT: 72.4% OSCAR: 73.8% BLIP: 78.3%,Multiple human references; Prompt variations tested; Inter-rater reliability,Language bias (can answer many questions without image); Dataset bias toward common objects; Answer distribution imbalance; Ambiguous questions,"GQA, VQA v2, OK-VQA, TextVQA, VizWiz"
+CLEVR,"A diagnostic dataset for compositional visual reasoning with 100K synthetic images and 1M questions testing spatial relations, counting, and logic.","Stanford University, Facebook AI Research",https://arxiv.org/abs/1612.06890,https://github.com/facebookresearch/clevr-dataset-gen,2017-04-01,Research; Development,"Compositional reasoning, Spatial reasoning, Counting, Logical reasoning, Visual reasoning",Core Performance; Robustness,Text + Vision,Text,Synthetic/Generated,Programmatically generated,Huge (> 10M samples),"Train (70K), Val (15K), Test (15K)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives synthetic image and question 2. Model predicts answer from predefined set 3. Accuracy computed, can analyze by question type",Outputs,False,,"Programmatically generated with known ground truth, exhaustive question type coverage",unknown,CNN+LSTM: 52.3% Film: 97.7% NS-VQA: 99.8% MAC: 98.9%,Ablation studies; Question type analysis; Compositional generalization tested,Synthetic domain (limited real-world applicability); Simple shapes and colors; No natural language variation; Programmatic biases,"CLEVR-CoGenT, GQA, NLVR2, CLOSURE"
+Waymo Open Dataset,"A large-scale autonomous driving dataset with 1000 diverse driving scenes, 12M LiDAR points per frame, high-resolution camera images, and rich 3D annotations.",Waymo LLC,https://arxiv.org/abs/1912.04838,https://github.com/waymo-research/waymo-open-dataset,2019-08-21,Research; Development; Deployment,"3D object detection, 2D object detection, Tracking, Domain adaptation, Sensor fusion",Core Performance; Robustness,Vision (Image); Video; Structured Data,Structured Data,New dataset (released with eval),Human annotations; Programmatically generated,Huge (> 10M samples),"Train (798), Val (202), Test (150)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives multimodal sensor data 2. Model predicts 3D bounding boxes with tracking IDs 3. AP/APH metrics at different IoU thresholds and difficulty levels,Outputs,True,"Test set labels private, evaluation via online leaderboard","Multi-stage annotation pipeline with quality checks, temporal consistency validation",unknown,PointPillars: 63.8 L2 APH (Vehicle) CenterPoint: 73.9 L2 APH PV-RCNN++: 77.8 L2 APH,Geographic diversity; Time of day variations; Weather conditions; Multiple difficulty levels,Geographic concentration in specific cities; Sensor-specific challenges; Annotation latency for distant objects; Class imbalance,"nuScenes, KITTI, Argoverse 2, Once, Lyft Level 5"
+Fashion-MNIST,"A drop-in replacement for MNIST with 70K grayscale images of fashion products across 10 categories, designed to be a more challenging benchmark for image classification.",Zalando Research,https://arxiv.org/abs/1708.07747,https://github.com/zalandoresearch/fashion-mnist,2017-08-25,Research; Development; Selection,"Image classification, Transfer learning, Benchmark comparison",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (60K), Test (10K)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives 28x28 grayscale image 2. Model predicts fashion category (10 classes) 3. Classification accuracy computed,Outputs,False,,Product catalog labels verified by domain experts,unknown,Linear Classifier: 83.7% CNN: 93.5% ResNet: 94.9% Vision Transformer: 95.1%,Multiple runs per sample; Seed variation tested,Low resolution (28x28); Grayscale only; Limited intra-class variation; Some categories overlap visually,"MNIST, EMNIST, Kuzushiji-MNIST, DeepFashion"
+MMLU (Massive Multitask Language Understanding),"A comprehensive benchmark with 57 tasks spanning STEM, humanities, social sciences, and more, designed to measure multitask accuracy and knowledge breadth in language models.","UC Berkeley, Columbia University",https://arxiv.org/abs/2009.03300,https://github.com/hendrycks/test,2020-09-07,Research; Development; Selection,"World knowledge, Reasoning, Domain expertise across 57 subjects",Core Performance,Text,Text,New dataset (released with eval),Expert annotations,Medium (1K - 100K samples),"Dev (5 shot examples per task), Test (285 questions per task average)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives multiple choice question with 4 options 2. Model predicts answer (A/B/C/D) 3. Accuracy computed per subject and overall,Outputs,False,,"Questions sourced from practice exams and tests, expert review for correctness",unknown,GPT-3 (175B): 43.9% GPT-3.5: 70.0% GPT-4: 86.4% Claude 3.5 Sonnet: 88.7% Random baseline: 25%,Multiple evaluation runs; Few-shot prompting variations; Subject-wise analysis,Multiple choice format may not reflect real-world usage; Some questions have ambiguous answers; Dataset contamination concerns with web-trained models; Cultural and knowledge cutoff biases,"MMLU-Pro, AGIEval, C-Eval, CMMLU"
+HumanEval,A code generation benchmark with 164 hand-written programming problems to evaluate functional correctness of synthesized Python code.,OpenAI,https://arxiv.org/abs/2107.03374,https://github.com/openai/human-eval,2021-07-07,Research; Development; Selection,"Code generation, Programming ability, Functional correctness, Code understanding",Core Performance,Text; Code,Code,New dataset (released with eval),Author-provided,Small (< 1K samples),Test (164 problems),Fixed data-driven (static test set),Automatic (Execution-based),1. Model receives function signature and docstring 2. Model generates function implementation 3. Generated code executed against unit tests 4. pass@k metric computed (% passing all tests in k samples),Outputs,False,,"Hand-written problems with comprehensive unit tests, manual verification of test correctness",unknown,Codex (12B): 28.8% pass@1 GPT-3.5-turbo: 48.1% pass@1 GPT-4: 67.0% pass@1 Claude 3.5 Sonnet: 92.0% pass@1,Multiple samples per problem (pass@k); Temperature sensitivity tested; Execution-based verification,Limited to Python; Small dataset size (164 problems); Relatively simple problems; May be contaminated in training data; No testing of code efficiency or style,"MBPP, APPS, CodeContests, HumanEval+, MultiPL-E"
+HellaSwag,"A benchmark for commonsense natural language inference about physical situations, requiring models to complete scenarios with the most plausible continuation.","University of Washington, Allen Institute for AI",https://arxiv.org/abs/1905.07830,https://github.com/rowanz/hellaswag,2019-05-19,Research; Development,"Commonsense reasoning, Physical understanding, Situation modeling, Plausibility judgment",Core Performance; Robustness,Text,Text,New dataset (released with eval),Crowdsourced annotations,Medium (1K - 100K samples),"Train (39,905), Val (10,042), Test (10,003)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives scenario context 2. Model selects most plausible continuation from 4 options 3. Accuracy computed,Outputs,False,,"Adversarial filtering using BERT to ensure difficulty, human validation of plausibility",unknown,BERT-Large: 47.3% GPT-2: 50.9% GPT-3: 78.9% GPT-4: 95.3% Human performance: 95.6%,Adversarial filtering; Multiple runs per sample; Human baseline comparison,Dataset may be easier than originally intended for modern LLMs; Multiple choice format; Adversarial examples may have artifacts; Potential data contamination,"PIQA, WinoGrande, CommonsenseQA, ARC"
+GSM8K (Grade School Math 8K),"A dataset of 8,500 grade school math word problems requiring multi-step arithmetic reasoning to solve.",OpenAI,https://arxiv.org/abs/2110.14168,https://github.com/openai/grade-school-math,2021-10-27,Research; Development,"Mathematical reasoning, Multi-step problem solving, Arithmetic reasoning, Chain-of-thought reasoning",Core Performance,Text,Text,New dataset (released with eval),Author-provided,Medium (1K - 100K samples),"Train (7,473), Test (1,319)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives math word problem 2. Model generates solution with reasoning steps 3. Final numerical answer extracted and compared to ground truth,Outputs,False,,"Human-written problems with verified solutions, consistency checks for answer correctness",unknown,GPT-3 (175B): 34.2% GPT-3.5: 57.1% GPT-4: 92.0% GPT-4o: 96.1% Claude 3.5 Sonnet: 96.4%,Multiple runs per sample; Chain-of-thought prompting tested; Answer extraction robustness,Limited to grade-school level math; Answer extraction can be brittle; Some problems have ambiguous wording; Potential contamination in training data,"MATH, GSM-Hard, SVAMP, ASDiv, MathQA"
+TruthfulQA,A benchmark measuring whether language models generate truthful answers to questions that humans might answer falsely due to misconceptions or false beliefs.,"Oxford University, OpenAI",https://arxiv.org/abs/2109.07958,https://github.com/sylinrl/TruthfulQA,2021-09-16,Research; Development; Safety,"Truthfulness, Factual accuracy, Resistance to misconceptions, Calibration",Safety; Core Performance; Calibration,Text,Text,New dataset (released with eval),Expert annotations,Small (< 1K samples),Test (817 questions across 38 categories),Fixed data-driven (static test set),Model-based: Expert; Human: Experts,1. Model receives question designed to elicit false beliefs 2. Model generates answer 3. Answers judged for truthfulness and informativeness using GPT-judge or human evaluation,Outputs,False,,"Expert-curated questions targeting known misconceptions, multi-rater validation of truth labels",unknown,GPT-3 (175B): 58.0% GPT-3.5: 47.0% GPT-4: 59.0% Claude 2: 62.0% Human baseline: 94.0%,"Multiple evaluation methods (MC1, MC2, generative); Human validation; Model-based evaluation correlation",Subjective truthfulness judgments in some cases; Cultural bias in what constitutes 'truth'; Model-based evaluation may not align with human judgment; Limited coverage of misconceptions,"FactScore, HaluEval, SelfCheckGPT, FELM"
+BIG-bench (Beyond the Imitation Game),"A collaborative benchmark with 204 tasks spanning linguistics, child development, math, commonsense reasoning, biology, physics, social bias, and software development.","Google Research, 450+ authors",https://arxiv.org/abs/2206.04615,https://github.com/google/BIG-bench,2022-06-09,Research; Development,"Diverse capabilities across 204 tasks including reasoning, knowledge, language understanding, bias detection",Core Performance; Fairness; Robustness,Text,Text,New dataset (released with eval),Multiple/Mixed sources,Large (100K - 1M samples),Varies by task,Composite,Automatic (Reference-based); Model-based: In the wild,1. Model evaluated on 204 diverse tasks 2. Each task has its own evaluation protocol 3. Performance aggregated across tasks,Outputs,False,,"Crowdsourced task creation with quality review, diverse authorship for broad coverage",unknown,Average human rater: 89.0% Few-shot PaLM (540B): 65.7% GPT-4: ~83% (estimated on BIG-Bench Hard),Multiple tasks provide robustness; Human baseline comparison; Cross-task analysis,Task quality varies; Some tasks too easy or too hard; Computational cost of running all 204 tasks; Aggregation methodology debatable,"BIG-Bench Hard, MMLU, HELM, SuperGLUE"
+MATH,"A dataset of 12,500 challenging competition mathematics problems from high school math competitions, requiring advanced mathematical reasoning and problem-solving.","UC Berkeley, OpenAI",https://arxiv.org/abs/2103.03874,https://github.com/hendrycks/math,2021-03-05,Research; Development,"Advanced mathematical reasoning, Problem solving, Multi-step reasoning, Mathematical knowledge",Core Performance,Text,Text,New dataset (released with eval),Published references,Medium (1K - 100K samples),"Train (7,500), Test (5,000)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives competition math problem 2. Model generates solution with steps 3. Final answer extracted and checked against ground truth 4. Problems span 7 subjects with 5 difficulty levels,Outputs,False,,"Problems from real math competitions with verified solutions, difficulty levels validated",unknown,GPT-3: 6.9% GPT-4: 42.5% Minerva (540B): 33.6% GPT-4 Turbo: 52.9% Claude 3.5 Sonnet: 71.1%,Multiple difficulty levels; Subject-wise analysis; Chain-of-thought evaluation,Answer extraction challenges; LaTeX formatting issues; Symbolic vs numeric answers; High difficulty may not reflect practical math usage,"GSM8K, MathQA, SVAMP, ASDiv, Hendrycks MATH"
+ARC (AI2 Reasoning Challenge),"A dataset of 7,787 science exam questions from grade 3-9, designed to require reasoning beyond simple retrieval or pattern matching.",Allen Institute for AI,https://arxiv.org/abs/1803.05457,https://github.com/allenai/arc,2018-03-14,Research; Development,"Scientific reasoning, Commonsense reasoning, Knowledge retrieval, Multi-hop reasoning",Core Performance; Robustness,Text,Text,New dataset (released with eval),Existing dataset labels,Medium (1K - 100K samples),"Easy (2,376 train, 570 test), Challenge (1,119 train, 1,172 test)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives science exam question with multiple choice options 2. Model selects answer 3. Accuracy computed separately for Easy and Challenge sets,Outputs,False,,"Real exam questions filtered to require reasoning, partitioned by difficulty using retrieval-based baseline",unknown,ARC-Challenge: BERT: 59.1% GPT-3: 51.4% GPT-3.5: 85.2% GPT-4: 96.3%,Two difficulty levels; Multiple choice format variations; Retrieval baseline comparison,Multiple choice format; Some questions solvable without reasoning; Challenge set becoming saturated; Grade-school level may not test advanced reasoning,"OpenBookQA, CommonsenseQA, QASC, SciQ"
+DROP (Discrete Reasoning Over Paragraphs),"A reading comprehension benchmark requiring discrete reasoning operations over paragraph content, including sorting, counting, and arithmetic.","Allen Institute for AI, University of Washington",https://arxiv.org/abs/1903.00161,https://github.com/allenai/drop,2019-03-01,Research; Development,"Reading comprehension, Numerical reasoning, Discrete operations, Multi-hop reasoning",Core Performance,Text,Text,New dataset (released with eval),Crowdsourced annotations,Large (100K - 1M samples),"Train (77,409), Dev (9,536), Test (9,622)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives paragraph and question 2. Model generates answer (number, span, or date) 3. F1 and Exact Match metrics computed",Outputs,False,,"Crowdsourced questions with verification, requires discrete reasoning operations confirmed through analysis",unknown,BERT: 47.0 F1 RoBERTa: 80.9 F1 GPT-3: 29.0 F1 GPT-4: 80.9 F1 Human performance: 96.4 F1,Multiple answer types; Question type analysis; Human baseline comparison,Requires careful answer extraction; Some questions ambiguous; Numerical reasoning can be brittle; Limited diversity in reasoning types,"SQuAD, NewsQA, NaturalQuestions, NumGLUE, TAT-QA"
+WinoGrande,"A large-scale commonsense reasoning benchmark with 44,000 problems requiring resolving pronoun ambiguity through world knowledge and commonsense.","Allen Institute for AI, University of Washington",https://arxiv.org/abs/1907.10641,https://github.com/allenai/winogrande,2019-07-24,Research; Development,"Commonsense reasoning, Coreference resolution, World knowledge, Causal reasoning",Core Performance; Robustness,Text,Text,New dataset (released with eval),Crowdsourced annotations,Medium (1K - 100K samples),"Train (40,398), Dev (1,267), Test (1,767)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives sentence with pronoun ambiguity 2. Model selects correct referent from 2 options 3. Accuracy computed,Outputs,False,,"Adversarial filtering using language models, crowdsourced generation with validation",unknown,BERT-Large: 59.4% RoBERTa-Large: 79.1% GPT-3: 70.2% GPT-4: 87.5% Human performance: 94.0%,Adversarial filtering; Large-scale dataset; Multiple difficulty levels,Binary choice may be limiting; Adversarial filtering may introduce artifacts; Saturation with modern models; Limited reasoning depth,"Winograd Schema Challenge, COPA, CommonsenseQA, PIQA"
+BBH (BIG-Bench Hard),"A curated subset of 23 challenging tasks from BIG-Bench where language models perform below human raters, focusing on tasks requiring multi-step reasoning.",Google Research,https://arxiv.org/abs/2210.09261,https://github.com/suzgunmirac/BIG-Bench-Hard,2022-10-17,Research; Development,"Complex reasoning, Multi-step thinking, Challenging cognitive tasks, Chain-of-thought reasoning",Core Performance,Text,Text,One or Multiple existing datasets,Multiple/Mixed sources,Medium (1K - 100K samples),"Test (6,511 examples across 23 tasks)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives task from one of 23 challenging categories 2. Model generates answer 3. Performance aggregated across all tasks 4. Chain-of-thought prompting typically used,Outputs,False,,"Tasks selected where models underperform humans, difficulty validated through empirical testing",unknown,PaLM (540B): 56.5% average PaLM (540B) + CoT: 78.1% GPT-4: ~91% Human raters: ~92% average,Multiple tasks; Chain-of-thought evaluation; Human baseline comparison; Cross-model validation,Only 23 tasks (limited coverage); Chain-of-thought prompting required for good performance; Task aggregation methodology; Rapidly saturating with newer models,"BIG-Bench, MMLU, AGIEval, HELM"
+MT-Bench,"A multi-turn conversational benchmark with 80 high-quality multi-turn questions spanning 8 categories, evaluated using GPT-4 as a judge.","UC Berkeley, UCSD, CMU, MBZUAI",https://arxiv.org/abs/2306.05685,https://github.com/lm-sys/FastChat,2023-06-09,Development; Selection,"Multi-turn conversation, Instruction following, Reasoning, Writing, Role-playing, Knowledge",Core Performance; Core Quality Dimensions,Text,Text,New dataset (released with eval),Author-provided,Small (< 1K samples),Test (80 questions with 2 turns each),Fixed data-driven (static test set),Model-based: In the wild,1. Model engages in 2-turn conversation 2. GPT-4 evaluates responses on 10-point scale 3. Scores averaged across turns and categories,Outputs,False,,"Strong correlation with human preferences validated on Chatbot Arena data, GPT-4 judge agreement measured",unknown,Vicuna-13B: 6.39 GPT-3.5-turbo: 7.94 Claude 2: 8.06 GPT-4: 8.99 GPT-4-turbo: 9.32,Multiple categories; Position bias mitigation; Agreement with human ratings validated; Pairwise comparison,Small dataset (80 questions); GPT-4 judge may have biases; Evaluation cost; Model-based judge limitations; Prompt sensitivity,"Chatbot Arena, AlpacaEval, Arena-Hard, LiveBench"
+GPQA (Google-Proof Q&A),"A challenging multiple-choice benchmark of 448 expert-level questions in biology, physics, and chemistry designed to be difficult even for skilled non-experts with internet access.",New York University,https://arxiv.org/abs/2311.12022,https://github.com/idavidrein/gpqa,2023-11-20,Research; Development,"Expert-level knowledge, Scientific reasoning, Domain expertise, Graduate-level understanding",Core Performance; Robustness,Text,Text,New dataset (released with eval),Expert annotations,Small (< 1K samples),"Main (198), Extended (246), Diamond (198 highest quality)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives graduate-level science question 2. Model selects from 4 multiple choice options 3. Accuracy computed 4. Questions validated to be difficult for non-experts with Google access,Outputs,False,,"PhD-level experts write and validate questions, non-expert validators with Google access achieve <35% accuracy",unknown,Random: 25% Non-expert w/ Google: 34% Expert: 81% GPT-4: 39% Claude 3 Opus: 59.4% GPT-4o: 53.6%,Expert validation; Non-expert baseline; Multiple subject areas; Quality tiers (Diamond subset),Small dataset size; Limited to 3 scientific domains; Multiple choice format; High cost of expert question creation; Cultural bias toward Western scientific education,"MMLU-Pro, JEE-Advanced, SciBench, LiveBench"
+IFEval (Instruction-Following Eval),"A benchmark of 541 prompts with verifiable instructions testing models' ability to follow precise formatting, length, and structural constraints.",Google DeepMind,https://arxiv.org/abs/2311.07911,https://github.com/google-research/google-research/tree/master/instruction_following_eval,2023-11-13,Research; Development; Selection,"Instruction following, Constraint satisfaction, Format compliance, Precise control",Core Performance; Robustness,Text,Text,New dataset (released with eval),Programmatically generated,Small (< 1K samples),Test (541 prompts with ~25 verifiable instructions),Fixed data-driven (static test set),Automatic (Reference-free),"1. Model receives prompt with verifiable instructions (e.g., 'respond in exactly 3 paragraphs', 'include word X at least 5 times') 2. Model generates response 3. Programmatic checks verify instruction compliance 4. Strict and loose accuracy metrics computed",Outputs,False,,"Verifiable instructions with programmatic checking, no ambiguity in correctness",unknown,GPT-3.5: 57.4% strict GPT-4: 76.9% strict Claude 2: 66.7% strict Gemini Ultra: 79.4% strict,Strict and loose metrics; Multiple instruction types; Programmatic verification; No human evaluation needed,Limited to verifiable instructions only; May not reflect realistic usage; Some instructions may be conflicting; Excludes semantic quality assessment,"MT-Bench, AlpacaEval, InstructGPT evals, FollowBench"
+`;
+
+// Simple CSV parser that handles quoted fields
+function parseCSV(csv) {
+  const lines = csv.trim().split('\n');
+  const headers = lines[0].split(',').map(h => h.trim());
+  
+  const result = [];
+  
+  for (let i = 1; i < lines.length; i++) {
+    const line = lines[i];
+    if (!line.trim()) continue;
+    
+    const row = {};
+    let currentVal = '';
+    let inQuotes = false;
+    let headerIndex = 0;
+    
+    for (let j = 0; j < line.length; j++) {
+      const char = line[j];
+      
+      if (char === '"') {
+        inQuotes = !inQuotes;
+      } else if (char === ',' && !inQuotes) {
+        row[headers[headerIndex]] = currentVal.trim();
+        headerIndex++;
+        currentVal = '';
+      } else {
+        currentVal += char;
+      }
+    }
+    // Add the last field
+    row[headers[headerIndex]] = currentVal.trim();
+    
+    result.push(row);
+  }
+  
+  return result;
+}
+
+const factsheets = parseCSV(csvContent);
+const factsheetMap = {};
+
+// Create a map for easier lookup
+factsheets.forEach(f => {
+  // Normalize title for matching
+  let key = f.title;
+  if (key.includes('MMLU')) key = 'MMLU';
+  else if (key.includes('BBH')) key = 'BBH';
+  else if (key.includes('GSM8K')) key = 'GSM8K';
+  else if (key.includes('HumanEval')) key = 'HumanEval';
+  else if (key.includes('TruthfulQA')) key = 'TruthfulQA';
+  else if (key.includes('HellaSwag')) key = 'HellaSwag';
+  else if (key.includes('ARC')) key = 'ARC-Challenge';
+  else if (key.includes('WinoGrande')) key = 'Winogrande'; // Note capitalization difference
+  else if (key.includes('VQA')) key = 'VQA';
+  else if (key.includes('IFEval')) key = 'IFEval';
+  else if (key.includes('GPQA')) key = 'GPQA';
+  else if (key.includes('MT-Bench')) key = 'MT-Bench';
+  else if (key.includes('MATH')) key = 'MATH';
+  else if (key.includes('COCO')) key = 'COCO-Captioning';
+  
+  factsheetMap[key] = f;
+});
+
+// Also add direct matches
+factsheets.forEach(f => {
+  factsheetMap[f.title] = f;
+});
+
+console.log('Available keys in factsheetMap:', Object.keys(factsheetMap));
+
+const filePath = path.join(__dirname, '../public/benchmarks/comprehensive-demo-model.json');
+const data = JSON.parse(fs.readFileSync(filePath, 'utf8'));
+
+let updatedCount = 0;
+
+data.evaluation_results.forEach(result => {
+  console.log('Checking result:', result.evaluation_name);
+  let factsheet = null;
+  
+  // Try to find a match
+  if (factsheetMap[result.evaluation_name]) {
+    factsheet = factsheetMap[result.evaluation_name];
+  } else if (result.evaluation_name === 'Winogrande' && factsheetMap['WinoGrande']) {
+    factsheet = factsheetMap['WinoGrande'];
+  }
+  
+  if (factsheet) {
+    // Map CSV fields to JSON schema fields
+    result.factsheet = {
+      purpose: factsheet.purpose,
+      principles_tested: factsheet.principles_tested,
+      functional_props: factsheet.functional_props,
+      input_modality: factsheet.input_modality,
+      output_modality: factsheet.output_modality,
+      input_source: factsheet.input_source,
+      output_source: factsheet.output_source,
+      size: factsheet.size,
+      splits: factsheet.splits,
+      design: factsheet.design,
+      judge: factsheet.judge,
+      protocol: factsheet.protocol,
+      model_access: factsheet.model_access,
+      has_heldout: factsheet.has_heldout === 'True',
+      alignment_validation: factsheet.alignment_validation,
+      baseline_models: factsheet.baseline_models,
+      robustness_measures: factsheet.robustness_measures,
+      known_limitations: factsheet.known_limitations,
+      benchmarks_list: factsheet.benchmarks_list
+    };
+    updatedCount++;
+    console.log(`Updated factsheet for ${result.evaluation_name}`);
+  }
+});
+
+fs.writeFileSync(filePath, JSON.stringify(data, null, 2));
+console.log(`Done updating factsheets. Updated ${updatedCount} evaluations.`);