Spaces:
Running
Running
adding readme.md
Browse files
app.py
CHANGED
|
@@ -1,6 +1,7 @@
|
|
| 1 |
import streamlit as st
|
| 2 |
import os
|
| 3 |
import torch
|
|
|
|
| 4 |
from transformers import AutoTokenizer
|
| 5 |
from jax import numpy as jnp
|
| 6 |
import json
|
|
@@ -89,3 +90,10 @@ if query:
|
|
| 89 |
)
|
| 90 |
|
| 91 |
st.image(image_paths)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
import streamlit as st
|
| 2 |
import os
|
| 3 |
import torch
|
| 4 |
+
from pathlib import Path
|
| 5 |
from transformers import AutoTokenizer
|
| 6 |
from jax import numpy as jnp
|
| 7 |
import json
|
|
|
|
| 90 |
)
|
| 91 |
|
| 92 |
st.image(image_paths)
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
def read_markdown_file(markdown_file):
|
| 96 |
+
return Path(markdown_file).read_text()
|
| 97 |
+
|
| 98 |
+
intro_markdown = read_markdown_file("readme.md")
|
| 99 |
+
st.markdown(intro_markdown, unsafe_allow_html=True)
|
readme.md
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Italian CLIP
|
| 2 |
+
|
| 3 |
+
....
|
| 4 |
+
|
| 5 |
+
# Novel Contributions
|
| 6 |
+
The original CLIP model was trained on 400millions text-image pairs; this amount of data is not available for Italian and the only datasets for captioning in the literature are MSCOCO-IT (translated version of MSCOCO) and WIT. To get competitive results we follewed three directions: 1) more data 2) better augmentation and 3) better training.
|
| 7 |
+
|
| 8 |
+
## More Data
|
| 9 |
+
|
| 10 |
+
## Better Augmentations
|
| 11 |
+
|
| 12 |
+
## Better Training
|
| 13 |
+
|
| 14 |
+
different optimizer and backbone freezing
|
| 15 |
+
|
| 16 |
+
# Scientific Validity
|
| 17 |
+
To better understand how well our clip-italian model works we run an experimental evaluation. Since this is the first clip-based model in Italian, we used the multilingual CLIP model as a comparison baseline.
|
| 18 |
+
|
| 19 |
+
We selected two different tasks:
|
| 20 |
+
+ image-retrieval
|
| 21 |
+
+ zero-shot classification
|
| 22 |
+
|
| 23 |
+
## Image Retrieval
|
| 24 |
+
|
| 25 |
+
## Zero-shot classification
|
| 26 |
+
|
| 27 |
+
# Broader Outlook
|