Visualizing Word Embeddings Using Dimensionality Reduction Techniques
Advisor: Ms. Apra
You’d think that ChatGPT, the thing that writes all your English essays, would be fluent in English. But it’s not. Transformer networks (GPT stands for Generative Pretrained Transformer) are entirely mathematical models, incapable of processing literal strings of text as input; they need to be converted into a language it can understand: math. Word embeddings, mathematical representations of words in vector space, were created to solve this problem, culminating in an “embedding matrix” that acts as the Transformer’s “dictionary” for English words. They’ve been known to capture some semantic meaning and relationships among words. For example, king - man + woman = queen demonstrates the concept of gender being encapsulated across different words. However, word embeddings are high-dimensional vectors (lists of around 200-300, to even 800 numbers), which makes interpretations and visualization of them, as we are limited to 2d and 3d representations of them, incredibly difficult. This problem is often referred to as “the curse of dimensionality,” a term coined by Richard Bellman. As a result, interpretation of the embeddings themselves are difficult, and analogies like the king queen example Aare often not always consistent. I hope to explore various “dimensionality reduction” techniques (reducing the list of numbers down to 2 or 3 while preserving as much information as possible), to gain insight into the most effective methods for maintaining semantic relationships in lower-dimensional space and review What each technique can teach us about word embeddings, hoping to contribute to the intuitiveness of AI.
Project Blog
Post 1: Visualizing Word Embeddings
AI POWERED APPLICATION IN ____, LLMs for ____, MACHINE LEARNING IN ____, we’ve all seen it before a million times. AI powers our apps, writes our notes, and does our homework. It’s everywhere in our lives. But despite its rumoured world-transforming potential, I’ve found myself curious about AI’s humbler intricacies and mechanisms.
After all, the entire field of Deep Learning is predicated on math, primarily partial derivative computations and matrix multiplications. You could literally write an equation for neural networks. It’s crazy. How can math comprehend the questions I ask it; how can it comprehend language? Is ChatGPT fluent in English?
No. Rather, these models use word embeddings, mathematical representations of words as high-dimensional vectors. Obtained through extensive training, every (most) English word has an associated 200-300 dimensional vector (jargon for a list of 200-300 numbers) associated with it, allowing them to be directly inputted as mathematical objects.
But these lists of 200-300 numbers aren’t just random; exploration of word embedding vector space has captured semantic relationships among vectors. Converting words to vectors allows for vector space operations; we can subtract/add vectors. Using this, it has been found that (man - woman) ≈ (king - queen), the commonality here being the concept of gender.
However, extracting semantic meaning from word embeddings is not always so straightforward; high dimensional vector space is tedious to analyze, a phenomenon coined as “The Curse of Dimensionality” by Richard Bellman. We need to “reduce” the dimension, projecting embeddings into lower dimensional space while retaining as much information as possible.
Dimensionality reduction techniques come in many shapes and forms, t-SNE, PCA, UMAP just to name a few, each with their pros and cons. Throughout my project, I’m planning on testing out various dimensionality reduction techniques and their effects on word embedding space, analyzing their effects, and evaluating their performance.
I apologize for throwing so much jargon in blog post 0, but I’m excited to have you all with me on this journey! See you next blog post!
Post 2: Project Setup!
Hello everybody and welcome to my week 1 blog post! Shaking off the ski week rust while working from home was not easy; one of my 3 hour shifts started at 9:30 PM. But nonetheless, I actually made some pretty good progress on my project this week. It will only get easier from here.
My first two days of work were spent downloading packages, creating virtual environments, downloading vector databases for the word embeddings online, and battling with terminal errors on my computer (this took 90% of the time). I almost lost my mind, but I was eventually able to get everything onto a locally hosted Jupyter notebook, where I can now run all the code I want for testing. I downloaded two different types of word embeddings: Word2Vec from Google, mostly used for general NLP tasks, and NumberBatch from ConceptNet, a set of embeddings constructed from a knowledge graph, which allows it to incorporate structured knowledge to better capture semantic relationships. After processing the raw data, I’m now able to convert any word I want into its corresponding vector in Word2Vec and Numberbatch. The image below is the vector for “woman.” (from Numberbatch and Word2Vec).
I have also finalized my test set of words that I will be performing analysis on. I wanted to get a broad list of words that spanned various concepts and ideas, like emotional states, foods, colors, occupations, etc. After finalizing everything and figuring out a balance for the different ideas I wanted to cover, I ended up with a list of 140 words, which I now have the vector representation for each.
With the rest of the time I had in the week, I turned to research papers and started studying the dimensionality reduction techniques that I will use later on.
Overall, I've completed the setup for my project, having attained my test set of words and their corresponding vectors. Next week, I plan on implementing the actual dimensionality reductions on my test set, and analyzing their effects. See y'all next week!
Post 3: Blog post 2 - reductions
Hello everybody! It was even harder returning to work after having a TON of fun at Costa Rica, but I got my flow going once more, getting some solid progress done this week.
The first thing I did was return to my battles with the terminal on my computer, spending a bit of time to remind myself how this all worked again. Once I got it to go, I spent a lot of time researching the actual mathematics behind three dimensionality reduction techniques: PCA, t-SNE, and Kernel PCA.
The idea behind PCA is to find principal components (like a certain axis on a plane), such that when you project higher dimensional points onto this principal component, the sum of the squares of the distances the points have to move is minimized. An extremely simple case could be like a 2d plot of data that represents a linear trend being represented as a single one dimensional line, that line being the principal component of the data, going from 2d to 1d. This same exact principle is applied to higher dimensional data, in our case, 150 dimensions to 2 dimensions.
Kernel PCA has a slightly different approach. Firstly, a nonlinear kernel function is applied to the data, transforming it into a higher dimensional feature space, where normal PCA is applied. The intuition of the feature space is to make linear separation of features easier to deduce in a higher dimensional space.
Overall, this week was some really good progress, as I figured out the gist of doing the actual dimensionality reductions on my data wordset. Next week, I hope to implement more techniques, including UMAP, SVM, and others!
Week 3: blog - reducing!
Hey guys!! After a well needed weekend, I returned to my battles with my beloved computer terminal. My battles have granted me the familiarity to run commands on terminal without having to search them up beforehand; I feel like a hacker in those movie scenes when I run a command to open my locally hosted Jupyter Notebooks.
Anyways, this week I continued what was being done last week, which is figuring out the actual dimensionality reductions on word embeddings. This required a lot of online research, as some dimensionality reduction techniques are not directly available from well known libraries sklearn (which I used for PCA, Kernel KCA, and t-sne)
One of such techniques is known as UMAP (Uniform Manifold Approximation and Projection). Without diving into mathematical nuance, the idea of UMAP is to first process the data in it’s high dimensions first before going to lower ones. It creates something known as a k-NN graph, or the k nearest neighbors graphs. If every data point was a house in a neighborhood, nearby houses are more strongly connected (maybe better friends), whereas farther data points are more weakly connected. The distance from data points is used in computing a probability, where a higher probability is associated with a stronger connection. UMAP, to provide further nuance, uses something called fuzzy simplicial set construction, which basically weakens the friendship in “crowded cities,” since there are so many neighbors to go around, and strengthens friendships in the “countryside” (more sparse), where more distant neighborhoods can still be close friends. The entire construction is like a fuzzy “friendship network” of the data points, which will be used as the reference when reconstructing the data points in lower dimensional space.
In lower dimensional space, where it needs to construct a graph, it first scatters all the houses randomly. Over many iterations, houses will be pushed and pulled until the ideal configuration is achieved. Houses that were close get pulled towards each other, houses that were far get pushed away; the magnitude of this pull/push strength depends on the probability value calculated in the previous step. After many iterations, UMAP is complete with its lower dimensional representation of the data. Here are UMAP’s representations of our data!
I not only got UMAP done, but various other techniques such as Laplacian Eigenmaps and locally linear embedding. I won't explain these in detail in this post, partly due to space, partly due to mathematical complexity. Here are views of their projections:
You can see how each technique produces drastically different results for the embeddings. It's really cool to such different interpretations of the same data.
But yea! That's what I worked on this week, and I plan on finishing the rest of the dimensionality reductions techniques next week. See you then!
Week 4: DrLIM
Hey everyone! This week’s work focused primarily on a single dimensionality reduction technique, namely Dimensionality reduction by Learning an Invariant Mapping (DrLIM). In the original paper, the authors noted that DrLIM was created to solve the limitations of LLE, Hessian Eigenmaps, and other classic dimensionality reduction techniques.
DrLIM uses a Siamese neural network along with a contrastive loss function to learn a nonlinear function that takes high dimensional data to lower dimensions. The siamese network architecture has two identical sub neural networks that share the same weights and parameters. These two sub networks get two different inputs (along with a “similarity score”), and their respective outputs are called feature vectors. The two feature vectors are compared with some distance metric. Ideally, pairs of data points that were deemed “similar” in the dataset would still be similar (small distance metric difference) after being spit out by the sub neural networks. This sort of architecture is key to things like facial recognition, but in our case we are applying it to word embeddings
Using an online dataset named SimLex-999, which has pairs of words along with an assigned similarity score to each pair, I managed to train my own Siamese network architecture to perform dimensionality reductions to my own dataset. THIS WAS REALLY HARD AND ANNOYING TO DO. Processing the dataset on its own took a whole day; the other four days were spent battling with tensorflow errors. Even now, though I have the actual projections for my words, there is still more to be done. For typical networks, you require tens of thousands of examples to properly train a network to the point where it can generalize. I only had around 600 examples. As a result, the final projections don’t make much sense.
However, using NumberBatch’s word embeddings, some better results are obtained.
This week concludes my testing of dimensionality reduction techniques. I have experimented with 7 techniques: PCA, Kernel PCA, t-SNE, LLE, Laplacian Eigenmaps, and UMAP.
Next week I’ll dive into figuring out the differences in what each technique offers, through data analysis and statistical analysis. See you then!
Week 5: Data analysis!
Hey! Welcome back. I’m writing this right before Prom; needless to say I’m not in the mood for working. But despite that, a good amount of work was done this week. This week, it’s all about data analysis!
The first day was all about cleanup; standardizing variable names for the bajillion different kinds of dimensionality reduction techniques I was using and organizing everything to be easy to work with. Turns out I was redefining the same variable for each dimensionality reduction technique, I had to give unique names to each one and put them into a dictionary for processing later. It was boring, tedious work, but it had to be done.
The rest of the week was spent coding the statistical analysis algorithms; I decided on silhouette score, Davies Bouldin Index, and Neighbor Consistency as my metrics. But first, I had to cluster my data, separating words into groups that shared semantic characteristics. Emotions like “love” and “hate” were in a group; colors, numbers, and animals are other examples of groups. I then painstakingly labelled each of my 140 words with their respective group assignments. These groups would serve as a measure for how well semantic groups were preserved in a projection from 300 dimensions to 2d.
I also used an unsupervised learning algorithm known as KMeans clustering, which forms clusters on unlabelled data simply by minimizing distances to a common centroid, to provide some alternative groupings. I then coded the algorithms for silhouette score, Davies Bouldin Index, and Neighbor Consistency (thank god a lot of these algorithms are included in packages), which I then tested on each dimensionality reduction technique.
Starting to get some results! Next week, I'll begin writing my literature review paper, while also taking a deeper dive into the data. I've also started experimenting using 3 dimensions instead of 2. Lots of stuff to be excited about! See you next week.
Weeks 6-3: D??
Hey everybody! Welcome back. I came across an idea this week that sidetracked my initial goals, but provided some cool insights to word embedding projections. I found a library that did 3d visualizations in a way that was almost identical to doing 2d visualizations with the standard matplotlib library. I began by doing 3d visualizations of UMAP, since UMAP was performing the best on most metrics.
After doing that for both Word2Vec and NumberBatch embeddings, I began separating semantic clusters into their own groups, visualizing them separately to see more clearly where these concepts ended up. I did this for 2d and 3d.
You can see how all the different colors are all bunched up in a group!
With this, I started doing some more case by case analysis with each cluster. I started by looking closely into the infamous man, woman, king, queen example.
As you can see, if you look at king-queen, and man-woman, the resulting vectors are almost identical. This is a well known result, and it’s interesting that this sort of result is still preserved even under UMAP’s nonlinear ways of dimensionality reduction.
I’ll continue in much more rigor this sort of analysis next week, and begin to get some good progress on my literature review paper!
Week 7: writing
This past week, I shifted my focus from visualizations to the final product of my research: the literature review.
I began organizing my paper into three main sections:
Overview of Word Embeddings – covering models like Word2Vec and NumberBatch, and how they encode semantic information. Understanding the intuition behind how word embeddings are formed in the first place is essential to a good understanding of this topic
Dimensionality Reduction Techniques – diving into each of the dimensionality reduction techniques and their mathematical foundations, gaining an understanding of them in the general context before looking at them in the word embedding context.
Evaluating Visualization Quality – looking at current methods for evaluating reduced-dimension projections, including metrics like cluster cohesion, neighborhood preservation, and interpretability. There’s also the subjective aspect of linguistic analysis that I have to consider (what is even considered to be in a semantic cluster)
While working on the paper, I revisited a key distinction between dimensionality reduction techniques that I would have to consider in my analysis: linearity. PCA and LLE are linear methods, which has its linear algebra definition, but would intuitively mean that global structures would be forced to be preserved under the linear constraint. T-SNE, Kernel PCA, and especially UMAP, are non-linear methods, meant to capture the more complex geometry of high dimensional space, capturing local semantic structure better than linear methods.
What’s been tricky—but interesting—is navigating conflicting opinions in the literature. Some researchers are skeptical of visualizations entirely, suggesting they can mislead more than they reveal. That tension has actually helped me sharpen the purpose of my project: not just making “pretty plots,” but understanding what they really say about the embeddings we rely on.
Next week, my plan is to finalize a full draft of the literature review and start systematically comparing how each reduction technique performs across a few controlled semantic tasks. I’ll probably include new visualizations that illustrate both strengths and weaknesses of each method.
Stay tuned—things are getting more rigorous and reflective, and I’m excited to see how this all comes together into a solid final paper!
Week 8: More Writing!
This week was frankly, really boring. Transcribing jargon and the research I’ve been doing for the past few months into something that’s both rigorous but understandable is quite the task. This week was a full dive into academic writing; I’ve had enough of staring at overleaf.
At first, it was a bit intimidating. I was scared to write statements, fearing the lack of rigor behind my understanding of things. I cross referenced my understanding with the abstracts and introductions of other academic papers, making sure I wasn’t making incorrect assumptions, and then incorporating it into a writing style that’s friendly to those without technical background. I ended up dividing the paper into the standard research paper sections: Introduction, Methodology, Results, Discussion, Future work, etc.
What I found most interesting in this was dissecting the “personality” of each dimensionality reduction technique. After all, these techniques were invented with trade-offs in mind, whether it be linearity, computational efficiency, neighbor consistency, or others. I noticed t-SNE was the most “aesthetic,” having the nicest clusters, but often messing up distances between clusters. As the statistical analysis suggested, UMAP had kind of the best balance between clustering and cluster distance. This is incredibly imprecise terminology, but it’s a blog so I can say such things
The more I write, the more I think about future work in this field. I’m imagining playing around with contextual models instead of standalone word embeddings, comparing the same word’s embedding in different contexts. Or maybe looking into using more predefined labels or clustering to drive our understanding of linguistics into the word embedding vector space. There’s a lot to think about here!
Overall though, this week was really just a step back from everything, and looking at what this project is ultimately about: demystifying word embedding space. Lot’s of typing this week, and i look forward to more typing next week! See y’all then!
Week 9: LaTeX is terrible
Hey everybody! Nothing interesting this week–just more academic paper writing.
I will say though, this week I felt much more confident writing compared to last week. Once you get started, everything else flows from there. I found myself reflecting back to the summer of my junior year, back when I was learning about sequence models and the overall field of deep learning through a course online called Deep Learning Specialization. That course was the most fascinating academic experience of my life, really igniting a strong interest in the field of AI. But it wasn't the world-transforming implications that excited, but rather the intricate, but clever mechanisms by which AI operated upon.
I learned, from each mathematical step, just how all of deep learning worked. The buildup of complexity from simple operations was fascinating, and I remember being particularly fascinated about the idea of assigning a vector representation to words. Learning how innovation in this field translated to mathematics and eventually implementation in coding was just the coolest thing to me, and the feeling of wonder from back then translated to me starting this project. As I write on, I find myself often pulling intuitions and concepts from my learnings back then, a reminder of why I even chose to do this project.
But not all of the week was filled with pleasant nostalgia. In fact, the majority of this week was spent dealing with my new nemesis, LaTeX. I hate it so much now, and spending all this time just figuring out why the formatting looks off has become the bane of my existence. Nevertheless, good progress was made, most of the paper has been written and put into LaTeX code, and it’s looking good so far. Not much to say this week, but next week I’m looking to finalize everything for this paper, and complete my presentation + poster for the Senior Project Symposium!
Week 10: Wrapping up
This week marks the official wrap-up of my senior project. My senior research paper titled “Visualizing Word Embeddings Using Dimensionality” is finally completed. What began as a vague curiosity about how AI “understands” words transformed into this deep-dive into high-dimensional vector space and dimensionality reduction techniques.
During this last stretch, I put my foot on the pedal and finished an astonishing amount of writing. Finalizing touches to my writing, making sure the mathematical formulations are correct, and going over the paper a few times, it felt so good when I finally exported the paper as a pdf.
Looking back, I felt so lost at times during this project. Back when it was just me and my computer’s terminal, it felt hard to visualize a whole research paper coming out of this. But week by week, I’d build this project piece by piece. It’d be a few cells in my jupyter notebook completed, it’d be some more notes down in my overleaf project, or it’d be just a better foundational understanding of a dimensionality reduction technique. But progress was steady, and I’m proud of the work I’ve done over the past 10 weeks.
Working through all this answered some questions I had about the topic, but brought upon so many more. What happens when we use context-based embeddings? Are static embeddings a statistical amalgamation of the different definitions a word can have? How will clustering change? One of the biggest unanswered questions was the usage of DrLiM, dimensionality reduction by learning an invariant mapping. If we could learn a sophisticated enough mapping from high dimensions to low dimensions using tons of data, what results could we achieve? With only 999 training examples in our experiments, I wonder if performance could improve with way more. But these questions are for another project, another time. For now, I am signing off!
-Kevin Zhou