| --- |
| language: |
| - en |
| license: wtfpl |
| library_name: transformers |
| tags: |
| - code |
| - text-generation-inference |
| datasets: |
| - flytech/python-codes-25k |
| - espejelomar/code_search_net_python_10000_examples |
| metrics: |
| - accuracy |
| pipeline_tag: text-generation |
| --- |
| # Model Card for GPT_2_CODE |
| -Goal is to create a small GPT2 python coder |
| # Table of Contents |
| - [Model Card for GPT_2_CODE](#model-card-for--model_id-) |
| - [Table of Contents](#table-of-contents) |
| - [Table of Contents](#table-of-contents-1) |
| - [Model Details](#model-details) |
| - [Model Description](#model-description) |
| - [Uses](#uses) |
| - [Direct Use](#direct-use) |
| - [Downstream Use [Optional]](#downstream-use-optional) |
| - [Out-of-Scope Use](#out-of-scope-use) |
| - [Bias, Risks, and Limitations](#bias-risks-and-limitations) |
| - [Recommendations](#recommendations) |
| - [Training Details](#training-details) |
| - [Training Data](#training-data) |
| - [Training Procedure](#training-procedure) |
| - [Preprocessing](#preprocessing) |
| - [Speeds, Sizes, Times](#speeds-sizes-times) |
| - [Evaluation](#evaluation) |
| - [Testing Data, Factors & Metrics](#testing-data-factors--metrics) |
| - [Testing Data](#testing-data) |
| - [Factors](#factors) |
| - [Metrics](#metrics) |
| - [Results](#results) |
| - [Model Examination](#model-examination) |
| - [Environmental Impact](#environmental-impact) |
| - [Technical Specifications [optional]](#technical-specifications-optional) |
| - [Model Architecture and Objective](#model-architecture-and-objective) |
| - [Compute Infrastructure](#compute-infrastructure) |
| - [Hardware](#hardware) |
| - [Software](#software) |
| - [Citation](#citation) |
| - [Glossary [optional]](#glossary-optional) |
| - [More Information [optional]](#more-information-optional) |
| - [Model Card Authors [optional]](#model-card-authors-optional) |
| - [Model Card Contact](#model-card-contact) |
| - [How to Get Started with the Model](#how-to-get-started-with-the-model) |
| # Model Details |
| ## Model Description |
| WIP,Goal is to create a small GPT2 python coder |
| - **Developed by:** C, o, d, e, M, o, n, k, e, y |
| - **Shared by [Optional]:** More information needed |
| - **Model type:** Language model |
| - **Language(s) (NLP):** eng |
| - **License:** wtfpl |
| - **Parent Model:** More information needed |
| - **Resources for more information:** More information needed |
| - [GitHub Repo](None) |
| - [Associated Paper](None) |
| # Uses |
| coding assistant |
| ## Direct Use |
| generate python code snippets |
| ## Downstream Use [Optional] |
| semi auto coder |
| ## Out-of-Scope Use |
| describe code |
| Keep Finetuning on question/python datasets |
| # Training Details |
|
|
| ## Training Data |
| flytech/python-codes-25k |
| espejelomar/code_search_net_python_10000_examples |
| ## Training Procedure |
| Train/Val/Scheduler |
| ### Preprocessing |
| More information needed |
| ### Speeds, Sizes, Times |
| Epochs 3 |
| # "flytech/python-codes-25k" |
| Training Loss: 0.4007 |
| Validation Loss: 0.5526 |
| Epochs 3 |
| |
| # "espejelomar/code_search_net_python_10000_examples" |
| --Starting Loss: 2.0862 |
| -Epoch 1/4 | Training Loss: 1.5355 | Validation Loss: 1.1723 |
| -Epoch 2/4 | Training Loss: 1.0501 | Validation Loss: 1.0702 |
| -Epoch 3/4 | Training Loss: 0.9804 | Validation Loss: 1.0798 |
| -Epoch 4/4 | Training Loss: 0.9073 | Validation Loss: 1.0772 |
|
|
| # Evaluation |
| Manual comparison with base model |
| ### Testing Data |
| flytech/python-codes-25k |
| espejelomar/code_search_net_python_10000_examples |
| ### Factors |
| 80/20 train/val |
| ### Metrics |
| train/validate |
| lr scheduling |
| ## Results |
| Better in python code generation as base gpt2-medium model |
| # Model Examination |
| More information needed |
| # Environmental Impact |
| - **Hardware Type:** CPU and Colab T4 |
| - **Hours used:** 4 |
| - **Cloud Provider:** Google Colab |
| - **Compute Region:** NL |
| ## Model Architecture and Objective |
| gpt2 |
| ## Compute Infrastructure |
| More information needed |
| ### Hardware |
| CPU and Colab T4 |
| ### Software |
| pytorch, custom python |
| # More Information [optional] |
| Experimental |
| # Model Card Authors [optional] |
| CodeMonkeyXL |
| # Model Card Contact |
| K00B404 huggingface |
| # How to Get Started with the Model |
| Use the code below to get started with the model. |
