| # Resource Automation |
|
|
| This repository uses a semi-automated process to keep Pashto resources current while preserving human review. |
|
|
| ## Goals |
| - Discover new Pashto-relevant resources from trusted public endpoints. |
| - Keep a machine-readable canonical catalog. |
| - Prevent unreviewed low-confidence resources from directly entering verified lists. |
|
|
| ## Covered source types |
| - Kaggle datasets |
| - Hugging Face datasets |
| - Hugging Face models |
| - Hugging Face Spaces (projects) |
| - GitHub repositories (projects and code) |
| - GitLab repositories (projects and code) |
| - Zenodo records |
| - Dataverse datasets |
| - DataCite DOI records |
| - Research-paper endpoints (arXiv, Semantic Scholar, OpenAlex, Crossref) |
|
|
| ## Files involved |
| - Canonical verified catalog: [../resources/catalog/resources.json](../resources/catalog/resources.json) |
| - Candidate feed: [../resources/catalog/pending_candidates.json](../resources/catalog/pending_candidates.json) |
| - Catalog schema: [../resources/schema/resource.schema.json](../resources/schema/resource.schema.json) |
| - Search export: [search/resources.json](search/resources.json) |
|
|
| ## Scripts |
| - Validate catalog: `python scripts/validate_resource_catalog.py` |
| - Generate markdown and search index: `python scripts/generate_resource_views.py` |
| - Sync new candidates: `python scripts/sync_resources.py --limit 20` |
| - Full run wrapper: `python scripts/run_resource_cycle.py --limit 25` |
|
|
| ## GitHub Actions |
| - CI (`.github/workflows/ci.yml`) enforces: |
| - catalog validation |
| - generated file consistency |
| - markdown link checks |
| - tests |
| - Resource Sync (`.github/workflows/resource_sync.yml`) runs daily and opens a PR with candidate updates. |
|
|
| ## Review flow |
| 1. Inspect candidate entries in `resources/catalog/pending_candidates.json`. |
| 2. Select useful items and move them into `resources/catalog/resources.json`. |
| 3. Set `status` to `verified` only after checking evidence and license. |
| 4. Run: |
| - `python scripts/validate_resource_catalog.py` |
| - `python scripts/generate_resource_views.py` |
| 5. Commit and open PR. |
|
|
| ## Runbook |
| - Reusable process guide: [resource_cycle_runbook.md](resource_cycle_runbook.md) |
|
|
