Week 2: Data Analysis

Hi everyone! Welcome back to my blog. This week, I began working with the data that will be used in my project. Since my goal is to study echo chambers, I need a dataset that can help me analyze the conversations that people have online. To collect this data, I used the X API that gives me access to posts and comments made on the site.

Using the API, I collected a random assortment of comments from different online discussions. Because of usage limits, I have initially gathered 2000 comments, but I will increase the amount of comments I get in the future.

Preparing and Structuring the Data

After collecting the data, I spent time cleaning and organizing it so that it could be properly analyzed. In this step, I formatted the data, removed empty or incomplete comments and prepared the text through creating embeddings, so it could be processed by my model.

After the data was cleaned, I converted it into a network structure. In this network, each comment acts as a node, and connections between nodes symbolize the similarity between the content of comments. For now, I have chosen to calculate this similarity using cosine similarity rather than Euclidean, but I will look into other similarity metrics in the following weeks which I will talk more about at the end of this blog.

To build this graph, I used the Python library igraph, which is great for building large networks. This library makes it easy and more efficient to analyze graphs, especially as I use the Leiden Algorithm to detect communities.

Detecting Communities in the Network

After building the graph, the next step was identifying communities in it. Communities are groups of comments that are more strongly connected to each other than the rest of the graph.

To detect these clusters, I used the Leiden Algorithm, which is a widely used method to find communities in networks. The Leiden Algorithm works by grouping nodes together in a way that maximizes how densely connected they are within the group compared to the rest of the network.

Running this algorithm helps me automate cluster detection that I will then analyze to see if they really are echo chambers.

Debugging the Community Detection

While running the Leiden Algorithm, I came across an interesting issue. When I tested it on my dataset of 2000 comments, the algorithm found 268 communities. This number is a lot higher than expected, since meaningful communities are meant to have a lot of comments in them. After looking at the graph, I realized the issue came from the network being too split into disconnected pieces. Since the algorithm can’t add comments that aren’t connected into the same community, it struggled to make big clusters.

To fix this issue, I chose to focus on the largest connected groups in the graph using the following line of code:

This line finds all the connected groups in the graph and keeps only the largest ones, which is referred to as the giant component. This made my community detection much more reasonable, with the Leiden Algorithm outputting 13 communities as opposed to 268.

Looking Ahead

Next work, I want to improve how I calculate similarities between comments. Right now, I use cosine similarity to measure similarity based on embeddings. This kind of similarity is commonly used with Natural Language Processing (NLP) because it calculates the angle between two vectors.

However, one of my main goals is to experiment with different similarity metrics to see which one can be most accurate as I detect communities. For example, I could test Euclidean Distance, which measured the straight line distance between two vectors. I could also try Manhattan Distance or Jaccard Similarity, which are all viable techniques for calculating similarity values between texts. If you have any ideas for similarity metrics I could use, I am open to suggestions as well!

Thank you for reading, and I will see you all next week,

Harish