Week 10: Analysis and the Frontend

Hello everybody and welcome to Week 10 of my Senior Project Blog! There are just 2 weeks left until the final presentation, and I’m about 95% done with my project. However, I still have a couple of action items left to check off, and I’ll go over that in today’s blog.

Analyzing the Data

In last week’s post I attached rubric scores showing how well each variant performed across the five personas (if you want to see those click the hyperlink for reference). I’ve decided to leave it to just those five personas rather than create additional personas through a Prolific survey, as the timescope that would take is beyond the range of this project.

Also, because I forgot to mention it within the contents of the post, Variant A is a simple zero-shot prompt (asking for advice on the persona and nothing else), Variant B allows the model to have access to the two grounding documents, heuristics.md and biases.md, Variant C is the sequential agent pipeline, and Variant D is the hierarchical agent pipeline. I will refer to them as such in the rest of the blog.

When designing the rubric scores, it was inherently difficult to put an objective metric on a subject matter. In fact, this is true generally when dealing with Large Language Models. Typically, there are datasets known as “benchmarks” which are used to evaluate the performance of LLMs. Creating benchmarks and evaluating models on them is a whole research domain of itself. If you would like to read more about benchmarks, I recommend this article in Google Deepmind. In essence, I’m creating my own mini benchmark by testing these 5 personas as datapoints, however this is nowhere near robust enough to count as a true evaluation. However, for the purposes of this project, it will suffice.

On average across the five personas, Variant A had an average rubric score of 39%, Variant B had an average rubric score of 61%, Variant C had an average rubric score of 89%, and Variant D had an average rubric score of 85%. The jump from Variant A to Variant B of 22% is a significant gap, and it shows that by simply giving the LLM additional context, the quality of the advice significantly increases. The largest difference between two variants, however, was between Variant B and C, with a 28% jump between the two. In fact, Variant C was the highest performing of the four, with an average rubric score of 89%, 4 percent higher than the hierarchical. As a result, this proves the original thesis of the research that agentic structures do in fact create better advice.

Variant D and Variant C are generally extremely close, with Variant D actually slightly outperforming Variant C on the personas that were meant to be more complex (Personas 3, 4, and 5). Each of these scenarios has either high net worth, poor asset allocation, or is a near-retirement situation, which are the scenarios where applying baseline heuristics actually hinders overall performance. As a result, the ability to decide not to attempt to apply the heuristics created more robust advice. However, this was a downside for the simpler personas, where the linear structure of Variant C outperformed Variant D.

Behavioral Realism was where all models struggled. Variants C and D never received above a 4, and frequently received 3’s. In fact, based on realism, Variants C and D weren’t actually that much better than Variants A and B. The reason for this was the strict structure that Variants C and D enforced. When you try to look at things overly analytically, you’re bound to forgo realistic factors such as human perception, stress, and lack of motivation. As a result, the advice given by Variant A especially was much more targeted toward humans and didn’t feel as robotic as the advice in Variants C and D. 

Heuristic Appropriateness was another factor that tended to cause some issues with Variants C and D. Generally, both variants had a tendency to overclassify many heuristics rather than flag more as inapplicable. This would result in the model attempting to apply up to 8 heuristics at once, which is physically impossible and infeasible in a non-ideal scenario. 

Variant B was the variant with the largest range across personas, and the least consistent. Generally, when given more complex personas, it tended to outperform Variant A significantly as the added context of the heuristics, when to apply them, and when not to apply them, was key knowledge in allowing the model to identify scenarios and provide the right action items. However, when the persona was more simple, it tended to lean closer to Variant A’s scores than to Variant C’s scores.

Variant C and D performed the best when there was a very clear and obvious constraint in the persona. This shows that when given a more “textbook” example of a case, the Variant is able to identify the missing link and provide the right advice. When that missing link is more vague the model tends to struggle slightly more.

I’ll create graphs displaying all this on the slides I present in my mock presentation next week, so stay tuned for that!

Web App Time

The final product for this project was always to have a functional web app at the end of it. As a result, I’ve started designing the code for that. I’m not done with it yet, but by the end of Week 11 I will have a functional web app ready. However, I will go over some of the architecture to design it.

Firstly, I had to wrap the backend in FastAPI. FastAPI is a service that allows my agent calls to run asynchronously. If you think about how API calls work, it runs one call at a time in order. If 3 people want to run the sequential agent at the same time, the API immediately gets overloaded with 12 simultaneous API calls, causing an overload error and crashing the system. In order to prevent this, I had to define each system thread asynchronously. 

Once I finished that, I began working on the ReAct frontend. ReAct is a JavaScript framework that works seamlessly with FastAPI to create a good-looking frontpage for your application. I’m currently in the process of implementing this, but I will provide more information when it is complete next week.

Conclusion

Overall, I made great progress this week! I’m excited to complete this project and complete the final presentation, and get ready for the symposium on May 16. Thanks for reading and I’ll see you guys next week.