Spaces:

hesamation
/

primer-llm-embedding

Running

App Files Files Community

hesamation commited on Mar 27, 2025

Commit

d8283b1

1 Parent(s): 099cdb9

Update bibliography and index.html to reflect new title and description for the LLM embeddings guide, add a new citation for word2vec, and enhance image alt texts for better clarity.

Browse files

Files changed (3) hide show

assets/images/word2vec_tfep.jpg +3 -0
src/bibliography.bib +9 -0
src/index.html +30 -18

assets/images/word2vec_tfep.jpg ADDED Viewed

Git LFS Details

SHA256: d9f4fc0688c293fc5bc9a9b4fe007baa180bc506c7355bd0f111d07766112807
Pointer size: 131 Bytes
Size of remote file: 230 kB

src/bibliography.bib CHANGED Viewed

@@ -61,3 +61,12 @@
   url={https://github.com/chrishayuk/embeddings/tree/main},
   publisher={GitHub}
 }

   url={https://github.com/chrishayuk/embeddings/tree/main},
   publisher={GitHub}
 }
+@misc{Mikolov_Chen_Corrado_Dean_2013,
+  title={Efficient estimation of word representations in vector space},
+  url={https://arxiv.org/abs/1301.3781},
+  journal={arXiv.org},
+  author={Mikolov, Tomas and Chen, Kai and Corrado, Greg and Dean, Jeffrey},
+  year={2013},
+  month=jan
+}

src/index.html CHANGED Viewed

@@ -8,15 +8,15 @@
     <meta name="viewport" content="width=device-width, initial-scale=1">
     <meta charset="utf8">
     <base target="_blank">
-    <title>A Primer on LLM Embeddings: The Semantic Backbone of AI</title>
     <link rel="stylesheet" href="style.css">
 </head>
 <body>
     <d-front-matter>
         <script id='distill-front-matter' type="text/json">{
-        "title": "A Primer on LLM Embeddings: The Semantic Backbone of AI",
-        "description": "The role of embeddings in traditional NLP, and their application to LLMs.",
         "published": "Mar 18, 2025",
         "affiliation": {
             "name": "University of Bologna"
@@ -36,7 +36,7 @@
     </script>
     </d-front-matter>
     <d-title>
-        <h1 class="l-page" style="text-align: center;">A Primer on LLM Embeddings:<br>The Semantic Backbone of AI</h1>
         <div id="title-plot" class="main-plot-container l-screen" style="overflow-x: hidden; width: 100%; text-align: center;">
             <div style="display: flex; justify-content: center; position: relative;">
                 {{{fragment-banner}}}
@@ -71,6 +71,10 @@
             </div>
             <figcaption>Figure 1: Overview of different word embedding techniques.</figcaption>
         </figure>
         <h2>What Makes a Good Embedding?</h2>
@@ -86,9 +90,6 @@
         <p>What should be the size of an embedding vector, 15, 50, 300? Striking the right balance is key. Smaller vectors (lower dimensions) are more efficient to keep in memory or to process, while bigger vectors (higher dimensions) can capture intricate relationships, but are prone to overfitting. For reference, GPT-2 model family has an embedding size of at least 768.</p>
-        <aside>
-            <span style="font-style: italic; font-weight: bold;">NOTE:</span> When reading about embeddings you may come across static vs. dynamic/contextualized embeddings. Static embeddings have a fixed representation for each word or token, regardless of the context it appears in. For example, the word "tear" has very different meanings in "Tears fell down from her eyes" and "tearing the page out", but dynamic embeddings change this representation based on the context of the word or token.
-        </aside>
         <h2>Traditional Embedding Techniques</h2>
@@ -158,7 +159,7 @@
             <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
               {{{fragment-tfidf-emb}}}
             </div>
-            <figcaption>TF-IDF example</figcaption>
         </figure>
         <p>There are two things noticeable about this embedding space:</p>
@@ -172,13 +173,13 @@
         <h2>word2vec</h2>
-        <p>A deep-learning based approach that is more modern than TF-IDF is word2vec. As can be assumed by the name, it is a network that aims to convert words into embedding vectors. It achieves this by defining a side goal, something to optimize the network for. For example, in CBOW (continuous bag of words), the word2vec network is trained to predict a missing word when its given the neighbors of that word as input. The intuition is that you can infer the embeddings of a word given the words around it.</p>
         <p>The word2vec architecture is pretty simple: one hidden layer that we extract the embeddings from, and one output layer which predicts the probabilities of all words in the vocabulary. On the surface, the network is trained to predict the right missing word given its neighbors, but in reality, this is an excuse to train the hidden layer of the network and find the right embeddings for each word. After the network is trained, the last layer can be tossed out the window because figuring out the embeddings is the real goal of the network.</p>
         <figure class="fullscreen">
             <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
-                <img src="assets/images/word2vec.png" alt="word2vec architecture diagram">
             </div>
             <figcaption>word2vec architecture showing the input layer, hidden layer (embeddings), and output layer</figcaption>
         </figure>
@@ -191,7 +192,7 @@
         <figure class="fullscreen">
             <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
-                <img src="assets/images/word2vec2.png" alt="word2vec visualization">
             </div>
             <figcaption>word2vec embeddings visualization</figcaption>
         </figure>
@@ -209,8 +210,19 @@
                 </div>
         </details>
         <p>The semantic relationship is a fun topic to explore and word2vec is a simple setup for your experiments. You can explore the biases of society or the data, or explore how words have evolved overtime by studying the embeddings of older manuscripts.</p>
         <h2>BERT (Bidirectional encoder representations from transformers)</h2>
         <p>Wherever you look in the world of NLP, you will see BERT. It's a good idea to do yourself a favor and learn about BERT once and for all, as it is the source of many ideas and techniques when it comes to LLMs. Here's a good video to get started. <d-cite bibtex-key="codeemporium2020bert"></d-cite></p>
@@ -237,7 +249,7 @@
         <figure class="fullscreen">
             <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
-                <img src="assets/images/BERT.png" alt="BERT architecture visualization">
             </div>
             <figcaption>BERT architecture overview. (Image source: <d-cite bibtex-key="devlin2018bert"></d-cite>)</figcaption>
         </figure>
@@ -261,7 +273,7 @@
         <figure class="fullscreen">
             <div id="image-as-graph" style="display: grid; position: relative; justify-content: center; width: 70%;">
-                <img src="assets/images/llm_embedding_overview.png" alt="Overview of embeddings in LLMs">
             </div>
             <figcaption>Overview of how embeddings fit into the LLM architecture</figcaption>
         </figure>
@@ -276,7 +288,7 @@
         <figure class="fullscreen">
             <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
-                <img src="assets/images/embedding_layer.png" alt="Embedding layer visualization">
             </div>
             <figcaption>Visualization of how the embedding layer works as a lookup table. (Image source: <d-cite bibtex-key="manning2024llm"></d-cite>)</figcaption>
         </figure>
@@ -285,7 +297,7 @@
         <figure class="fullscreen">
             <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
-                <img src="assets/images/emb_lookup.jpg" alt="Embedding lookup table visualization">
             </div>
             <figcaption>Visualization of the embedding lookup process. (Image source: <d-cite bibtex-key="manning2024llm"></d-cite>)</figcaption>
         </figure>
@@ -461,10 +473,10 @@
         <h3 id="citation">Citation</h3>
         <p>For attribution in academic contexts, please cite this work as</p>
         <pre
-            class="citation short">"A Primer on LLM Embeddings: The Semantic Backbone of AI", 2025.</pre>
         <p>BibTeX citation</p>
-        <pre class="citation long">@misc{a_primer_on_llm_embeddings,
-        title={A Primer on LLM Embeddings: The Semantic Backbone of AI},
         author={Hesam Sheikh Hessani},
         year={2025},
         }</pre>

     <meta name="viewport" content="width=device-width, initial-scale=1">
     <meta charset="utf8">
     <base target="_blank">
+    <title>LLM Embeddings Explained: A Visual and Intuitive Guide</title>
     <link rel="stylesheet" href="style.css">
 </head>
 <body>
     <d-front-matter>
         <script id='distill-front-matter' type="text/json">{
+        "title": "LLM Embeddings Explained: A Visual and Intuitive Guide",
+        "description": "How Language Models Turn Text into Meaning, From Traditional Embeddings to Modern LLMs",
         "published": "Mar 18, 2025",
         "affiliation": {
             "name": "University of Bologna"
     </script>
     </d-front-matter>
     <d-title>
+        <h1 class="l-page" style="text-align: center;">LLM Embeddings Explained:<br>A Visual and Intuitive Guide</h1>
         <div id="title-plot" class="main-plot-container l-screen" style="overflow-x: hidden; width: 100%; text-align: center;">
             <div style="display: flex; justify-content: center; position: relative;">
                 {{{fragment-banner}}}
             </div>
             <figcaption>Figure 1: Overview of different word embedding techniques.</figcaption>
         </figure>
+        <p>When reading about embeddings you may come across static vs. dynamic/contextualized embeddings. Static embeddings have a fixed representation for each word or token, regardless of the context they appear in.</p>
+        <p>Embeddings are the semantic backbone of LLMs because they work as the initial transformation from raw tokens into numeric vectors that models can understand. However, it's important to distinguish between token embeddings—which are the fixed vectors assigned to input tokens at the very beginning—and the contextual representations produced by the deeper layers of the model. While both are technically embeddings, token embeddings are static, whereas the intermediate hidden states evolve as they pass through each layer, incorporating the full context of the input. In some literature, these contextual outputs are also referred to as "embeddings," which can be confusing.</p>
         <h2>What Makes a Good Embedding?</h2>
         <p>What should be the size of an embedding vector, 15, 50, 300? Striking the right balance is key. Smaller vectors (lower dimensions) are more efficient to keep in memory or to process, while bigger vectors (higher dimensions) can capture intricate relationships, but are prone to overfitting. For reference, GPT-2 model family has an embedding size of at least 768.</p>
         <h2>Traditional Embedding Techniques</h2>
             <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
               {{{fragment-tfidf-emb}}}
             </div>
+            <figcaption>TF-IDF example showing word embeddings plotted in 2D space after dimensionality reduction</figcaption>
         </figure>
         <p>There are two things noticeable about this embedding space:</p>
         <h2>word2vec</h2>
+        <p>Originally proposed in <d-cite bibtex-key="Mikolov_Chen_Corrado_Dean_2013"></d-cite>, is a more modern technique than TF-IDF. As can be assumed by the name, it is a network that aims to convert words into embedding vectors. It achieves this by defining a side goal, something to optimize the network for. For example, in CBOW (continuous bag of words), the word2vec network is trained to predict a missing word when its given the neighbors of that word as input. The intuition is that you can infer the embeddings of a word given the words around it.</p>
         <p>The word2vec architecture is pretty simple: one hidden layer that we extract the embeddings from, and one output layer which predicts the probabilities of all words in the vocabulary. On the surface, the network is trained to predict the right missing word given its neighbors, but in reality, this is an excuse to train the hidden layer of the network and find the right embeddings for each word. After the network is trained, the last layer can be tossed out the window because figuring out the embeddings is the real goal of the network.</p>
         <figure class="fullscreen">
             <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
+                <img src="assets/images/word2vec.png" alt="Diagram showing word2vec architecture with input layer, hidden layer (embeddings), and output layer. Input words are encoded as one-hot vectors, passed through a hidden layer to generate embeddings, and finally through an output layer that predicts probabilities for each word in vocabulary.">
             </div>
             <figcaption>word2vec architecture showing the input layer, hidden layer (embeddings), and output layer</figcaption>
         </figure>
         <figure class="fullscreen">
             <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
+                <img src="assets/images/word2vec2.png" alt="Visualization of word2vec embeddings in 2D space, showing how semantically similar words cluster together. Words with related meanings appear closer to each other in the embedding space.">
             </div>
             <figcaption>word2vec embeddings visualization</figcaption>
         </figure>
                 </div>
         </details>
+        <p>To train word2vec efficiently, especially with large vocabularies, an optimization technique called negative sampling is used. Instead of computing the full softmax over the entire vocabulary (which is computationally expensive), negative sampling simplifies the task by updating only a small number of negative examples (i.e., randomly selected words not related to the context) along with the positive ones. This makes training faster and more scalable.</p>
         <p>The semantic relationship is a fun topic to explore and word2vec is a simple setup for your experiments. You can explore the biases of society or the data, or explore how words have evolved overtime by studying the embeddings of older manuscripts.</p>
+        <p>You can actually visualize and play with word2vec embeddings with <a href="https://projector.tensorflow.org/">Tensorflow Embedding Projector</a>.</p>
+        <figure class="fullscreen">
+            <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
+                <img src="assets/images/word2vec_tfep.jpg" alt="Screenshot of TensorFlow Embedding Projector showing interactive 3D visualization of word2vec embeddings, allowing exploration of semantic relationships between words in the embedding space.">
+            </div>
+            <figcaption>TensorFlow Embedding Projector's interactive exploration of word2vec embeddings in 3D/2D space.</figcaption>
+        </figure>
         <h2>BERT (Bidirectional encoder representations from transformers)</h2>
         <p>Wherever you look in the world of NLP, you will see BERT. It's a good idea to do yourself a favor and learn about BERT once and for all, as it is the source of many ideas and techniques when it comes to LLMs. Here's a good video to get started. <d-cite bibtex-key="codeemporium2020bert"></d-cite></p>
         <figure class="fullscreen">
             <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
+                <img src="assets/images/BERT.png" alt="BERT architecture diagram showing the model's components: input layer with tokenization, embedding layer, multiple transformer encoder blocks with self-attention mechanisms, and task-specific output heads. Special tokens [CLS] and [SEP] are highlighted.">
             </div>
             <figcaption>BERT architecture overview. (Image source: <d-cite bibtex-key="devlin2018bert"></d-cite>)</figcaption>
         </figure>
         <figure class="fullscreen">
             <div id="image-as-graph" style="display: grid; position: relative; justify-content: center; width: 70%;">
+                <img src="assets/images/llm_embedding_overview.png" alt="Diagram showing how embeddings fit into LLM architecture: input text is tokenized, converted to token embeddings, combined with positional embeddings, and processed through transformer layers to generate contextual representations.">
             </div>
             <figcaption>Overview of how embeddings fit into the LLM architecture</figcaption>
         </figure>
         <figure class="fullscreen">
             <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
+                <img src="assets/images/embedding_layer.png" alt="Illustration of embedding layer as a lookup table: input tokens are mapped to their corresponding embedding vectors through a matrix multiplication operation. Shows how token IDs index into rows of the embedding matrix.">
             </div>
             <figcaption>Visualization of how the embedding layer works as a lookup table. (Image source: <d-cite bibtex-key="manning2024llm"></d-cite>)</figcaption>
         </figure>
         <figure class="fullscreen">
             <div id="image-as-graph" style="display: grid; position: relative; justify-content: center;">
+                <img src="assets/images/emb_lookup.jpg" alt="Detailed visualization of embedding lookup process: shows how token IDs are used to select specific rows from the embedding weight matrix, resulting in embedding vectors for each input token.">
             </div>
             <figcaption>Visualization of the embedding lookup process. (Image source: <d-cite bibtex-key="manning2024llm"></d-cite>)</figcaption>
         </figure>
         <h3 id="citation">Citation</h3>
         <p>For attribution in academic contexts, please cite this work as</p>
         <pre
+            class="citation short">"LLM Embeddings Explained: A Visual and Intuitive Guide", 2025.</pre>
         <p>BibTeX citation</p>
+        <pre class="citation long">@misc{llm_embeddings_explained,
+        title={LLM Embeddings Explained: A Visual and Intuitive Guide},
         author={Hesam Sheikh Hessani},
         year={2025},
         }</pre>