Visualizing Word Embeddings
January 29, 2025
AI POWERED APPLICATION IN ____, LLMs for ____, MACHINE LEARNING IN ____, we’ve all seen it before a million times. AI powers our apps, writes our notes, and does our homework. It’s everywhere in our lives. But despite its rumoured world-transforming potential, I’ve found myself curious about AI’s humbler intricacies and mechanisms.
After all, the entire field of Deep Learning is predicated on math, primarily partial derivative computations and matrix multiplications. You could literally write an equation for neural networks. It’s crazy. How can math comprehend the questions I ask it; how can it comprehend language? Is ChatGPT fluent in English?
No. Rather, these models use word embeddings, mathematical representations of words as high-dimensional vectors. Obtained through extensive training, every (most) English word has an associated 200-300 dimensional vector (jargon for a list of 200-300 numbers) associated with it, allowing them to be directly inputted as mathematical objects.
But these lists of 200-300 numbers aren’t just random; exploration of word embedding vector space has captured semantic relationships among vectors. Converting words to vectors allows for vector space operations; we can subtract/add vectors. Using this, it has been found that (man – woman) ≈ (king – queen), the commonality here being the concept of gender.
However, extracting semantic meaning from word embeddings is not always so straightforward; high dimensional vector space is tedious to analyze, a phenomenon coined as “The Curse of Dimensionality” by Richard Bellman. We need to “reduce” the dimension, projecting embeddings into lower dimensional space while retaining as much information as possible.
Dimensionality reduction techniques come in many shapes and forms, t-SNE, PCA, UMAP just to name a few, each with their pros and cons. Throughout my project, I’m planning on testing out various dimensionality reduction techniques and their effects on word embedding space, analyzing their effects, and evaluating their performance.
I apologize for throwing so much jargon in blog post 0, but I’m excited to have you all with me on this journey! See you next blog post!
Leave a Reply
You must be logged in to post a comment.