Evaluating the effectiveness of agentic frameworks on goal-based individual financial advisory
Advisor: Bhattacharya
Internship Location: Maharashtran Institute of Technology
Onsite Mentor: Dr. Bhavana Tiple, Associate Professor of Computer Science at Maharashtran Institute of Technology, World Peace University
Currently, assisting financial goals of young people are overlooked and underserved. Many young people have their lofty (or conservative) financial goals, but tend to lack personalized guidance. Current financial heuristics are generic and do not adapt well to individual constraints and situations. My goal with this project is to evaluate the extent to which agent-based AI frameworks can turn high-level financial goals into personalized, step-by-step financial plans.
Much of the research conducted for this study will be internet research. Firstly, I’ll review individual financial principles and heuristics that dictate current widely accepted views on finance management. I’ll also conduct technical research on agent AI, its use in finance, and how it can be optimized for my personal project. I’ll need to design synthetic user personas for evaluating the final product, which will be a task in itself of researching which groups may find the most usefulness in a tool like mine.
The final deliverable of this project is a functional prototype of an agent-based AI system that generates personalized, step-by-step financial plans based on user goals and constraints. I’ll also have a set of user personas used for evaluation, and a structured framework on how to measure the model’s performance versus basic financial heuristics.
Project Blog
Week 1: Heuristic Research and Agent Role Definition
Hello everybody!
My name is Arjun Maganti and welcome to my Senior Project Blog! In this blog I’ll be going over all developments relevant to the furtherment of my project.
Background
My motivation to focus on this as my senior project stemmed from my own lack of experience with financial management. As a senior in high school, I’ve recently gotten my first job and I’m starting to handle my own money more and more. Although I am spared from paying bills just yet, even managing spending and making sure I have enough money for wants down the line gets daunting quickly, and creating a system that can help me manage all the moving pieces was the key motivation behind this project.
Agent Definition
The title of my project is “Evaluating the effectiveness of agentic frameworks on goal-based individual financial advisory”. An agentic framework is essentially an architecture that connects a preset “agent” to a specific task that it is meant to do. An agent is an instance of a Large Language Model (LLM) that is given a certain system prompt that makes it think and act a certain way. This week, I’ve designed four agents that I believe will yield the best results. If it becomes necessary to tweak these in later weeks as I build the system, I will do so.
Agent 1, the Persona Profiler, identifies the specific constraints unique to the specific demographic labeled in the persona. Agent 2, the Heuristic Strategist, applies general rules to create a baseline recommendation. Agent 3, the Goal Optimizer, deviates from the standard heuristics to prioritize certain financial tactics over others. Lastly, Agent 4, is a quality control evaluating agent (common in typical agentic frameworks) to check every other agent’s work.
Heuristic Research
Another aspect of planning I worked on this week regards the actual financial heuristics that I will be basing my agents off of. I compiled a list of 9 financial “rules of thumb” that I will continue to do more research into in the upcoming weeks. These include:
50/30/20 rule: Allocate 50% of income to needs, 30% to wants, and 20% to savings/debt
Pay Yourself First: Save a portion of your income immediately before paying any bills
Emergency Fund: Maintain 3-6 months living expenses in a liquid, accessible account
15% Retirement Savings: Save roughly 15% of your annual pre-tax income for retirement
Housing Costs: Keep housing costs below 30% of your income
Avoid High-Interest Debt: Prioritize paying off high-interest debt (like credit cards) to avoid compounding interest charges
Rule of 72: Estimate how long it takes for an investment to double by dividing 72 by the annual rate of return.
401(k) Match: Contribute at least enough to your employer’s plan to get the full company match
Age-Based Asset Allocation: A common rule is to subtract your age from 100 (or 110/120) to determine the percentage of stocks in your portfolio.
All of these have their own upsides and downsides. Since they are general heuristics, they do work in a variety of scenarios but fall short in specific ones. For example, keeping housing costs below 30% of income is physically impossible in a lot of high-density urban spaces where rent can skyrocket due to higher demand. In theory, my final product should be able to take this into account.
Financial Mistakes to Avoid
I also compiled a general list of things to avoid. These will also be fed into the agent as context later.
Anchoring: Relying too heavily on the first piece of information offered (e.g., the original price of a clothing item).
Availability Bias: Making decisions based on recent or easily remembered events, such as panic-selling after a market crash.
Status Quo Bias: A preference for the current state of affairs, leading to inaction in rebalancing portfolios.
Loss Aversion: The psychological tendency to prefer avoiding losses over acquiring equivalent gains.
Conclusion
I’m excited to continue working on this project in the following weeks! Next week I hope to finalize these into specific markdown or pdf files, which I can use as context bases to feed into an agentic system which I’ll define a particular architecture for (either RAG [Retrieval Augmented Generation] or some other format). See you next week!
Week 2: Architecture Design and Intro to Context Engineering
Hello everybody and welcome back to my senior project blog! This is week #2 and I’ve made some good progress on designing a lot of the architecture behind my project, which I’m excited to share here.
Introduction to Context Engineering
For my project, I’ve decided to settle on a relatively new approach to designing agentic systems known as context engineering. This week I read an article on Anthropic’s (the company behind the AI model Claude) blog from about 6 months ago that highlights a newer approach to optimizing the output of LLMs called context engineering. To quote from that article, “Building with language models is becoming less about finding the right words and phrases for your prompts, and more about answering the broader question of ‘what configuration of context is most likely to generate our model’s desired behavior?’” This approach was introduced this past summer (July 2025), and has demonstrated success across a wide variety of benchmarks. Most famously, context engineering is used in Claude Code, an AI-powered coding assistant which has gained notoriety for its impressive performance for developing software systems.
Agentic Architecture
I’ve settled on using a sequential multi-agent pipeline, which is split into four different agents that each have their own specific task . In the image below, there’s an overview of a bunch of different agentic architectures. For this project, I’ve decided on using example 4, but if time permits later on I would like to attempt different architectures later on. A sequential pipeline is how it sounds, each agent communicates to each other in a sequential line one-by-one, until an output is reached.
The four agents I’ll be using, as shown in the image above, are 1) the persona profiler, 2) the heuristic strategist, 3) the goal optimizer, and 4) quality control. The goal of the first agent is to turn raw text in the form of financial goals or natural language, and convert it into a JSON schema, which should look something like this image below.
The second agent will then use the fields “applicable_heuristics and inapplicable_heuristics” and directly match to the specific heuristics, which will be used as a baseline control to compare the third agent’s output to. As you can probably guess then, the third agent’s task is to tweak these heuristics to be more personalized to the actual persona and the goals it has. The fourth agent’s task is to create a structured quality control report which will be formatted based on specific rubrics that I create later (in weeks 4 and 5).
Grounding Documents
I then needed to create something known as “grounding documents”. In the field of context engineering, we need to actively decide which context we think is worth sharing to the LLM. In essence, we need to have a set of documents that the LLM can always refer to when answering any prompt, regardless of past conversation context. This will sort of act as the LLM’s “knowledge-base”, typical in normal prompt engineering systems, but instead we will store them as simple textual files rather than large databases. This removes the need for me to implement a context database search tool that is connected to the LLM, and saves me a lot of time and energy.
As a result, I created two documents: heuristics.md and biases.md. Heuristics is a markdown file containing 5 fields: definition, why it works, applicable when, failure conditions, and correct application example.
Biases is a markdown file that looks like this.
Both of these files will act as general context and “knowledge” that the LLM has. It outlines the logic that the LLM should use when approaching a new situation, and hopefully improve the advice given. I’ve also attached the sources for these heuristics in the footnote below.
Conclusion
Next week, my goal is to get an end-to-end for agent 1 working in its entirety. As a result, I will need to set up a Gemini API call through Google ADK, and similarly connect it to my two grounding documents. I also plan to begin researching what personas I would like to test, and this may require some broader demographic research into the economic states of a lot of people across the United States. Nonetheless, I’m excited to continue working on this project for the next few weeks, and start putting theory to practice!
Heuristic Footnotes
Islam, Mohammad T. "The Six Foundational Rules of Personal Finance: Essential Principles for Financial Stability and Growth."
https://www.sandvoldfinancialgroup.com/blog/heuristics-shortcuts-to-financial-decision-making
Week 3: Placing the First Agent in the Puzzle
Hey there everybody! I hope you guys are having a great week. I’ve honestly been having a good one as well because I’ve made some really good progress on my project (which also includes some pivots from my original idea! 😱) but contributes to a more robust final product.
A Slight Pivot
Firstly, I want to start off where we left off last week. If you’ve been following my blog for the past 2 weeks you’ve seen that I’ve “finalized” my architecture for the project. I said I was going to be using a sequential agentic system, which means I have 4 agents in a line communicating with each other (agent 1 talks to agent 2, agent 2 talks to agent 3, and so on). However, I realized that this wasn’t really robust enough. After all, my project is about evaluating agentic frameworks, and I realized my project isn’t really doing that. It’s essentially comparing one experimental group to a control. As a result, I instead decided to build two architectures, a sequential agentic pipeline as I discussed last week, and a hierarchical agentic pipeline. This hierarchical pipeline has an evaluator agent that can invoke 3 sub-agents to perform specific planning tasks.
All in all, this means that I’ll have 4 groups: a flat prompt (the control group), structured grounding (giving the prompt + docs and hoping for the best), the sequential pipeline, and the hierarchical pipeline. I’ll test all of these across 5 personas, creating a 5x4 table that I’ll use in evaluation, looking something like this.
Flat Prompt | Structured Grounding | Sequential Pipeline | Hierarchical Pipeline
Persona 1
Persona 2
Persona 3
Persona 4
Persona 5
Starting to (vibe) code
Now that I had my final evaluation metrics in place, I’m now finally ready to start programming! For this week I decided to focus specifically on getting the first agent in the sequential pipeline running locally on my device through a basic Gemini API call. As a reminder, the goal of the first agent was to convert a basic low-level description of a persona into a JSON file. The first step in this process was to set up a Gemini API key, which I borrowed from my father as Gemini API is billed to a credit card. Luckily, it’s extremely inexpensive for the task at hand (literally about 14 cents per month), but it’s still money nonetheless so this project does have an investment cost.
System Prompts
The next step is to create a system prompt. This is the heart of the first agent which tells it exactly how to respond to a prompt and how to format the output. For this, I gave it a simple roleplay scenario telling it was the first agent in a multi-agent pipeline and its ONLY task was to output a JSON schema in this format:
{
"income_tier": "low | mid | high",
"debt_load": "none | manageable | high",
"housing_situation": "renting | owned | high-cost-market",
"employment_stability": "stable | variable | gig",
"goal": "<string>",
"applicable_heuristics": ["string", ...],
“additional_considerations”: [{“item”: “<string>”, “rationale”:”<string>”},...],
"inapplicable_heuristics": [{"rule": "<string>", "reason": "<string>"}, ...],
"cognitive_risks": [{"bias": "<string>", "reason": "<string>"}, ...]
}
I then gave it specific instructions for each field. For example, for income_tier, I gave exact delineations for tiers (low is below $45k, mid is between $45k to $120k, and high is anything above $120k). Obviously, this is deterministic based on the city/environment the persona is in (for example $120k in SF is far from high income), but there are other metrics like housing_situation, and additional_considerations that take account of that.
In fact, one thing I didn’t include originally was additional_considerations. On the first run testing this system out, the LLM would begin to hallucinate additional heuristics that weren’t in my grounding documents, but still technically did apply to the situation at hand. At first, I wanted to eliminate all hallucination and essentially make my system prompt as clear as possible, but on further inspection I realized that using the LLM’s knowledge base to my advantage could actually provide better results and give a more robust definition to the advice it hands out. So, I decided to add a new field, additional_considerations, which are any extra heuristics the LLM may want to include.
Generating outputs
Now that all that was done, I decided to create a basic prompt and test the agent back-to-front. Here’s how it went.
Prompt: 28-year-old, $58k salary, renting in Austin TX, $22k in student loans, stable job at a tech company, wants to eventually buy a house
Output: {
"income_tier": "mid",
"debt_load": "manageable",
"housing_situation": "renting",
"employment_stability": "stable",
"goal": "Pay off $22k in student loans within 36 months while building a 3-month emergency fund.",
"applicable_heuristics": ["50/30/20", "Emergency Fund", "Avoid High-Interest Debt"],
"additional_considerations": [{"item": "Debt Snowball/Avalanche", “rationale": "Given manageable student debt, prioritizing highest-interest balance first may accelerate payoff"}, {"item": "Consider a High Yield Savings Account", "rationale": "To maximize interest earned on emergency fund savings."}],
"inapplicable_heuristics": [{"rule": "401(k) Employer Match", "reason": "No information provided about employer match."}, {"rule": "Age-Based Asset Allocation", "reason": "Goal is debt repayment and emergency fund, not retirement savings yet."}, { "rule": "Tax-Advantaged Investing", "reason": "Goal is debt repayment and emergency fund, not long-term investing yet."}, {"rule": "Retirement Savings Rate", "reason": "Goal is debt repayment and emergency fund, not retirement savings yet."}, {"rule": "Pay Yourself First", "reason": "Goal is debt repayment and emergency fund, not discretionary savings yet."}],
"cognitive_risks": [{"bias": "present_bias","reason": "Desire to buy a house eventually may lead to neglecting current debt repayment."}]
}
All in all, this is incredibly good! I now have a working back-to-front agent running locally on my device through an API call. If you're extra curious, this code (and biases.md + heuristics.md) is available in my working GitHub repository
Conclusion and Next Steps
The goal for next week is to finalize all the personas I’m going to use for evaluation purposes. This is an important step because I eventually am going to use those for comparing the efficiencies of the different systems. I will be required to do pretty significant demographic research as a result, and based on large-scale American demographics I’ll craft my 5 personas in as much detail as possible. Similarly, next week I also hope to get a back-to-front of Agent 2, the heuristic strategist, complete. The goal of that agent is to take the JSON schema outputted by Agent 1 and extract the heuristics, starting to make a semblance of a longer-term plan. I’m really excited to get working on that and I’m happy with the progress I’ve made! Thanks for reading and I’ll see you guys next week.
Week 4: Deep Dive into Demographics
Hello everybody, I hope you guys are all having a great week. Welcome back to week 4 of my senior project blog! This week, unfortunately, I wasn’t able to get through everything I tried to do this week, but I still got through about 90% of what I wanted to, which is great!
Demographic Duties
This week, my goal was to both create and finalize my five personas that I will use for evaluation, and complete the second agent in the sequential pipeline. I decided to first start with creating the five personas, and I did some demographic research into personal financial knowledge in the United States. I decided to only focus on the United States as expanding this to foreign countries would require me to supplement knowledge to the LLM far beyond what I currently have, and it would cause me to get significantly closer to the token memory limit of the LLM, which is not something I wish to do.
The TIAA Institute publishes an index known as the GFLEC Personal Finance (P-Fin) Index annually [1] . The P-Fin index is a financial readiness exam which assesses the US adult population on their ability to make sound financial decisions across eight common financial activities: earning, consuming, saving, investing, borrowing, insuring, understanding risk, and gathering information.
As you can see in this graph, the percentage of questions answered correctly has hovered roughly around the 50% mark, with 49% being the latest number.
Similarly, across the eight metrics, there was little change regarding the percentage of questions answered correctly, however it’s clear that overall there is a slight decrease across all metrics. Knowledge on borrowing, saving, and consuming is the highest, whereas insuring and comprehending risk is the lowest. As a result, I decided to center my personas around investing, insuring, and comprehending risk and helping manage those.
Likewise, I also took a look at the Pew Research Center’s demographic data [2] , which measured people's personal opinion on how much they THOUGHT they knew about personal finance. This is just as important to knowing how well people actually know about personal finance because it gives us a concept as to the disparity between people’s perception of themselves versus what reality actually says (the P-Fin Index).
From what we can see here, the data is surprisingly accurate to what the exams say! Most of these numbers seem to cover around 49%, which is the global average across American adults. As expected, higher income individuals tend to view themselves as better money managers, and Whites and Asians (the two highest-income demographic groups of the four shown) also view themselves that way. Similarly, the trend is correlated with age: as people get older they believe in their financial abilities more.
This table lines up with everything and shows essentially the same trends, just stratified across 5 management skills. Investment plans, typically treated as the hardest to individually compose, are the lowest at 27%, but every other skill is above 50%, and typically the lowest numbers seem to be low-income young minorities. Based on these stats, to craft each of my personas revolving around one of the holes in this table, as well as the other data that we’ve seen published by the TIAA.
The Personas
Now it’s finally time for me to show you the personas, and here they are!
Persona 1: recent college grad entry-level salary hispanic individual living in an HCOL area (most likely west coast) with significant student debt. Key stress test: Can the system create a well structured plan to pay off student loans and begin creating an investment plan?
Persona 2: middle-aged white career professional with stable mid-income living in suburban city with a family, no significant debt, wants to retire early and begin investing. Key stress test: Can the system sense that this individual is in a well-off financial situation and expand expectations to take risks and make money moves that are limited by heuristics?
Persona 3: young black skilled freelance worker in a medium COL city with inconsistent monthly income (bouncing from low to mid) and no employer 401(k), wants to stabilize and settle down. Key stress test: Can the system deal with variable income and plan for future savings without employer matching?
Persona 4: middle-aged white single parent with two young children, moderate income in a suburban area, wants to save for college in the next 10 years. Key stress test: Can the system deal with needing to save a large sum of money in a relatively short timeframe without largely cutting down on the quality of life?
Persona 5: Near-retiree with high net worth, poor asset allocation, and significant credit card debt, working a stable job in a city. Key Stress Test: How does the system deal with high credit card debt this close to retirement? (missing age and income level)
Each key stress test is a way to test the architecture and stretch it in a way that would be difficult even for a human financial advisor to accomplish. As a result, I plan to use these as the baseline tests across the control group and the 3 experimental groups. If time permits, I plan to create more, but for now I am going to stick to 5 personas.
Agent 2 (Incomplete, Unfortunately)
The next part of my week was to work on Agent 2 in the sequential pipeline, the heuristic strategist, but however due to the time this background research took and my duties in Science Olympiad, I was unable to complete more than just a draft of the system prompt, which I hope to refine. In time for next week, I will create a python script (which I already have the skeleton for), and update my GitHub along with my future work on Agent 3 next week.
Eventually, I also plan to move all my agents from running locally on my device through the Gemini API and move them to Google ADK, a python framework for running agents. I’ll talk more about those next week, so stay tuned and thanks for reading!
Sources
[1] Financial Literacy and Retirement Fluency in America, www.tiaa.org/content/dam/tiaa/institute/pdf/insights-report/2025-05/financial-literacy-and-retirement-fluency-in-america-p-fin-ir.pdf. Accessed 21 Mar. 2026.
[2] Edwards, Khadijah. “Personal Finance Knowledge and Confidence Vary by Income, Race, Age in the US.” Pew Research Center, Pew Research Center, 9 Dec. 2024, www.pewresearch.org/short-reads/2024/12/09/roughly-half-of-americans-are-knowledgeable-about-personal-finances/.
Week 5: Uno, Dos, Tres
Hello everybody!
We’re about one month into the senior project (crazy how time flies)! This week I’ve made a lot of progress in finishing the sequential pipeline, and I’m incredibly excited to delve into that. I’ve also made a couple of tweaks to my final prototype that I plan to show at the senior project panel, which I’ll also go into later in this post.
Agent 2
Backtracking to the architecture of the sequential pipeline, Agent 1 was always going to be the easiest to design as it had no choice of context for me to deliberate on. The only things I needed to consider when designing the system prompt were the two grounding documents (heuristics.md and biases.md), as well as the persona itself (which is just a sentence or so). As a result, there was no real “context engineering” done here other than the two grounding documents, as there was no real need for me to deliberate what context I wanted to include.
However, for agent 2, it started to get a little bit more complicated. Agent 1 outputs a JSON schema that characterizes the persona based on their employment status, debt load, housing situation, and more. The goal of agent 2 was to transform this JSON schema into a second JSON schema which takes each heuristic listed by agent 1 and gives a specific recommendation, monthly impact, and priority value to it.
{{
"baseline_plan": [
{{
"heuristic": "",
"recommendation": "<specific, actionable instruction derived strictly from this heuristic>",
"monthly_impact": "",
"priority": <integer, 1 = highest>
}}
],
"inapplicable_notes": [
{{
"rule": "",
"reason": ""
}}
],
"assumptions": [
""
],
"baseline_summary": "<2-3 sentence plain-language summary of what this baseline plan does and does not address>"
}}
As a result, the system prompt needed to be EXTREMELY clear in stating that the goal of agent 2 was ONLY to take the heuristics that agent 1 thought the persona could be linked to, and follow them verbatim. Similarly, it needed to have a calculable monthly impact to show what the effect of following that heuristic would be on the individual. Additional heuristics or biases that agent 1 may have included (on purpose or hallucinated) didn’t have a necessary need in this agent.
At first, this was a huge challenge for me because I was unable to prevent the agent from either hallucinating new heuristics or from using ones that didn’t apply to the persona at hand, despite being fed in the JSON file from agent 1. In order to fix it, I restructured some of the prompting in agent 1, making sure that the reasoning attached to each additional heuristic was clear, and I fixed up some of the prompting in agent 2 to provide direct step-by-step instructions on how to tackle additional_heuristics. I also added a new field additional_biases which worked the same exact way as additional_heuristics.
Likewise, agent 2 kept on listing monthly impact as “indeterminate”. I realized that this was because I wasn’t actually feeding it any data regarding the persona’s income (which was in the original prompt given to agent 1). So, I tweaked the system prompt to include the original prompt in the larger API call, and it fixed the issue.
Here was the output from agent 2 for this persona: 28-year-old, $58k salary, renting in Austin TX, $22k in student loans, stable job at a tech company, wants to eventually buy a house.
{
"baseline_plan": [
{
"heuristic": "50/30/20 Rule",
"recommendation": "With a $58,000 salary, and assuming a 25% tax rate, monthly take-home pay is approximately $3,625. Allocate $1,812.50 to needs, $1,087.50 to wants, and $725 to savings and debt repayment.",
"monthly_impact": "$725",
"priority": "N/A"
},
{
"heuristic": "Emergency Fund",
"recommendation": "Aim to save 3 months of essential living expenses in a high-yield savings account. Assuming $1,812.50 in monthly needs, the target emergency fund size is $5,437.50.",
"monthly_impact": "Indeterminate",
"priority": 1
},
{
"heuristic": "Avoid High-Interest Debt",
"recommendation": "Allocate the savings/debt repayment portion of the 50/30/20 rule ($725/month) toward the $22,000 in student loans. Prioritize paying off any student loans with interest rates above 7-8%.",
"monthly_impact": "$725",
"priority": 2
},
{
"heuristic": "Age-Based Asset Allocation",
"recommendation": "As a 28-year-old, and using the rule of 110, allocate 82% of investments to equities and 18% to fixed income (110 - 28 = 82). This applies to any investment accounts outside of the emergency fund and debt repayment.",
"monthly_impact": "Indeterminate",
"priority": 3
}
],
"inapplicable_notes": [
{
"rule": "15% Retirement Savings",
"reason": "Paying off student loans and saving for a down payment should take priority over saving 15% for retirement."
},
{
"rule": "Housing Costs Under 30%",
"reason": "They are currently renting, so this rule is not applicable until they start seriously considering buying a house."
},
{
"rule": "401(k) Employer Match",
"reason": "Whether or not this is applicable depends on if their employer offers a 401(k) match, which is unknown."
},
{
"rule": "Pay Yourself First",
"reason": "Paying off student loans and saving for a down payment should take priority over paying themselves first."
},
{
"rule": "Rule of 72",
"reason": "The Rule of 72 is not applicable at this stage."
}
],
"assumptions": [
"Assumed a 25% effective tax rate for calculating take-home pay.",
"Assumed student loan interest rates are known and can be prioritized from highest to lowest."
],
"baseline_summary": "This baseline plan establishes a budget framework, prioritizes building an emergency fund and paying down debt, and provides a basic asset allocation guideline. It does not address the persona's specific goal of saving for a down payment, nor does it consider the potential impact of a 401(k) employer match."
This output is extremely promising and gives a great baseline for agent 3 to work and compare from!
Agent 3
Agent 3 is the heart of the sequential process. It’s the key differentiating factor between the typical heuristic-following agent 2’s plan, and trying to work out manually what IRAs to open, where to invest, how to save, and how much to spend. The goal of agent 3 is to give extremely specific advice, and also documents exactly WHAT it’s changing and WHY it’s making those changes from the base heuristics. As a result, it has a very robust JSON schema that is meant to make evaluating its performance much easier.
{{
{{
"optimized_plan": [
{{
"heuristic": "",
"source": "baseline | modified | new",
"recommendation": "<specific, actionable instruction with dollar amounts where possible>",
"deviation_from_baseline": "",
"rationale": "",
"monthly_impact": "",
"priority": ""
}}
],
"goal_alignment": {{
"goal": "",
"addressed_by": [""],
"projected_timeline": "",
"gap_analysis": ""
}},
"additional_considerations_incorporated": [
{{
"item": "",
"incorporated": true,
"application": ""
}}
],
"cognitive_risk_mitigations": [
{{
"bias": "",
"mitigation": ""
}}
],
"assumptions": [""]
}}
The important thing here is for each heuristic, there is a “source” field which states whether the heuristic is the baseline, modified, or new. In most or all circumstances, this should be modified or new, but in some cases the baseline heuristic might be sufficient for the persona. As well as that, this agent also outputs a ranking, monthly impact, and rationale.
Goal alignment is a new field that essentially takes into account the applicability of the plan that the agent put in place. My goal was to ensure that the agent was answering the original goal of the person, and it means that it needs to not deviate from that plan. When I ran the agent for the first time, these two first parts of the JSON schema worked amazingly: they gave incredibly detailed plans which went over exactly WHAT changed and WHY it changed. I’m extremely happy with the results that came with the first half of the agent’s output.
However, the second half which includes additional_considerations_incorporated and cognitive_risk_mitigations were extremely buggy. The goal of these two were to go over any additional considerations that agent 1 may have outputted (e.g. Debt Snowball which is something that isn’t in the original grounding documents) and try to incorporate it into the system. However, the agent is time and time again incorporating it in the original optimized_plan subsection. Also, for cognitive_risk_mitigations, it only includes cognitive risks specifically from biases.md, and not the other cognitive risks it took from its own knowledge base. Despite tweaking the prompt multiple times, I was unable to fix this error.
{
"optimized_plan": [
{
"heuristic": "50/30/20 Rule",
"source": "modified",
"recommendation": "Based on a $58,000 salary, and prioritizing debt payoff and down payment savings, allocate approximately $2,417/month to needs (50%), $1,160/month to wants (24%), $967/month to debt repayment (20%), and $290/month to down payment savings (6%).",
"deviation_from_baseline": "Reduced 'wants' allocation from 30% to 24% and created a new 6% allocation for down payment savings. This shifts discretionary spending toward the stated goal of buying a house.",
"rationale": "The persona's goal is to eliminate debt and save for a down payment. Reducing the 'wants' category allows for a dedicated savings allocation without sacrificing debt repayment.",
"monthly_impact": "$1257",
"priority": "N/A"
},
{
"heuristic": "Emergency Fund",
"source": "modified",
"recommendation": "Initially, save 1 month of essential living expenses ($2,417) in a liquid, easily accessible account. Once this minimum is met, pause emergency fund contributions and direct those funds toward debt repayment and down payment savings. After the student loan is paid off, increase the emergency fund to the recommended 3-6 months of expenses ($7,251 - $14,502).",
"deviation_from_baseline": "Reduced the initial emergency fund target to 1 month of expenses to accelerate debt repayment and down payment savings. Deferred building the full 3-6 month fund until after debt is eliminated.",
"rationale": "Accelerating debt repayment directly addresses the persona's goal of eliminating student loan debt. A smaller initial emergency fund allows for faster progress on debt and down payment savings, with the understanding that this increases short-term risk.",
"monthly_impact": "Variable",
"priority": 3
},
{
"heuristic": "Avoid High-Interest Debt",
"source": "modified",
"recommendation": "Allocate $967/month from the 50/30/20 rule towards the $22,000 in student loan debt. Prioritize the highest interest rate loans first. Given the $22,000 debt at an assumed average interest rate of 6%, this will take approximately 24 months to repay. Consider a debt avalanche or snowball method based on psychological preference.",
"deviation_from_baseline": "Added a timeline estimate for debt repayment and suggested considering debt avalanche or snowball methods.",
"rationale": "Providing a timeline makes the debt repayment goal more concrete and motivating. Acknowledging different repayment strategies caters to individual preferences and increases adherence to the plan.",
"monthly_impact": "$967",
"priority": 1
},
{
"heuristic": "Age-Based Asset Allocation",
"source": "baseline",
"recommendation": "As a 28-year-old, a moderate asset allocation strategy (using the 110 rule) suggests holding 82% of investments in equities and 18% in fixed income. This applies to any investment accounts beyond the emergency fund and debt repayment.",
"deviation_from_baseline": null,
"rationale": "This is a standard recommendation for long-term investing and remains appropriate.",
"monthly_impact": "Indeterminate",
"priority": 4
},
{
"heuristic": "401(k) Employer Match",
"source": "new",
"recommendation": "Investigate the availability of a 401(k) employer match. If available, contribute at least enough to receive the full match, even if it means temporarily reducing the down payment savings rate. This is essentially 'free money' and should be prioritized.",
"deviation_from_baseline": "This heuristic was not addressed in the baseline plan due to lack of information. It is added here as a critical consideration.",
"rationale": "Capturing the full employer match provides an immediate and guaranteed return on investment, which accelerates progress toward long-term financial goals, including buying a house.",
"monthly_impact": "Varies based on employer match",
"priority": 2
}
],
"goal_alignment": {
"goal": "Eliminate $22k in student loan debt while saving for a down payment on a house.",
"addressed_by": [
"50/30/20 Rule",
"Emergency Fund",
"Avoid High-Interest Debt",
"401(k) Employer Match"
],
"projected_timeline": "Student loan payoff in approximately 24 months. Down payment savings will depend on the target down payment amount and the actual savings rate after accounting for the employer match. Assuming a $290/month contribution, a $20,000 down payment will take approximately 69 months (5.75 years).",
"gap_analysis": "The plan does not include specific strategies for increasing income or reducing expenses to accelerate savings. It also does not address the specific type of house the persona wants to buy or the target location, which would influence the down payment goal."
},
"additional_considerations_incorporated": [],
"cognitive_risk_mitigations": [
{
"bias": "Status Quo Bias",
"mitigation": "Set a recurring calendar reminder every 6 months to re-evaluate the debt repayment and down payment savings plan. This forces an active decision point and prevents sticking with a suboptimal plan due to inertia."
}
],
"assumptions": [
"Assumed a combined federal and state income tax rate of 25% to arrive at an approximate monthly net income of $4,833.",
"Assumed student loans are the only form of high-interest debt.",
"Assumed essential living expenses align with the 'needs' category of the 50/30/20 rule.",
"Assumed an average interest rate of 6% on student loans for repayment timeline calculation.",
"Assumed a $20,000 target for the down payment savings goal."
]
}
This output however, is still extremely good! I’m very happy with the progress I’ve made for the sequential pipeline.
Conclusion
The goal for next week is to wrap up agent 4 and complete the full agent pipeline. I also want to begin moving all these agents to ADK. However, ADK is an extremely complex framework and requires quite a bit more debugging the local API call. As a result, depending on the timeframe I’m working with, I’ll reconsider if moving them to ADK is the best idea.
Also, my initial plan for evaluation was to holistically and objectively analyze the quality of the advice that the system outputs. However, my faculty advisor Mrs. Bhattacharya suggested that I also include a survey with real customers to see how humans would use such a system. As a result, around weeks 8-9 I plan to incorporate the survey and distribute it around to adults.
The code for this week’s section is on my GitHub. Check it out if you’re curious!
Thanks for reading my blog and I’ll see you guys next week!
Week 6: Sequential Success!
Hello everyone and welcome back to my senior project blog! This week, I’ve hit a key milestone in completing the sequential pipeline back to front! I’m excited to share that progress and some other key decisions I’ve made this week.
Agent 4: The Final Step
Just as a reminder, the goal of Agent 1 in the sequential pipeline was to break down a persona string (usually 2-3 lines of text) into a JSON schema that contains all the key information of that person, including things like their income bracket, their cost of living situation, their employment status, and more. The goal of Agent 2 was to take the JSON schema and match it to the applicable heuristics in the grounding documents and attempt to follow them as explicitly as possible. The goal of Agent 3 was to take Agent 2’s explicitly matched heuristics and modify them to create more personalized goals that work better for the specific persona’s scenario.
Agent 4 is different from the rest of them because Agent 3 was essentially the last “step” in generating the advice. The idea behind Agent 4 is to evaluate the effectiveness of the pipeline and essentially generate a QC report “scoring” the rest of the agents.
Here’s the JSON schema that Agent 4 will output:
{
"pipeline_variant": "",
"pipeline_stages_evaluated": ["agent_1", "agent_2", "agent_3"],
"rubric_scores": {
"heuristic_alignment": {
"score": "<1-5>",
"confidence": "",
"justification": "",
"flags": [""]
},
"persona_fit": {
"score": "<1-5>",
"confidence": "",
"justification": "",
"flags": [""]
},
"goal_specificity": {
"score": "<1-5>",
"confidence": "",
"justification": "",
"flags": [""]
},
"bias_avoidance": {
"score": "<1-5>",
"confidence": "",
"justification": "",
"flags": [""]
},
"feasibility": {
"verdict": "",
"confidence": "",
"justification": "",
"flags": [""]
}
},
"pipeline_integrity": {
"agent_1_assessment": {
"performed_correctly": "",
"issues": [""]
},
"agent_2_assessment": {
"stayed_in_lane": "",
"issues": [""]
},
"agent_3_assessment": {
"meaningfully_deviated": "",
"deviation_quality": "",
"issues": [""]
}
},
"overall_recommendation": "",
"revision_suggestions": [""],
"user_facing_summary": "<3-4 sentence plain language summary of the plan's strengths, weaknesses, and one concrete next step. Written for a non-expert reader.>"
}
Firstly, “pipeline_variant” is an identifier for what type of pipeline this agent is evaluating. In our cases, it’s always either going to be sequential or hierarchical, as this agent is only going to be used in these two scenarios. The rest of the JSON schema is split into 3 main categories: “rubric_scores”, “pipeline_integrity” and “user_facing_summary”.
Rubric Scores
The “rubric_scores” subsection consists of 5 metrics: heuristic alignment, persona fit, goal specificity, bias avoidance, and feasibility. Each of these metrics contains a score from 1-5, a confidence level (low, med, high) and a justification as to why the agent handed out that score. As a result, each run of Agents 1-3 is rated out of 25. The agent also adds justifications on where it took off points. The idea here is that the agent is self-critical in its assessment and is able to go back and point out its shortcomings, particularly when checking whether it met the goal of the individual and if its plan was feasible and realistic.
Pipeline Integrity
Pipeline integrity is slightly simpler. All it is is a dictionary that assesses the job of Agent 1, Agent 2, and Agent 3. Agent 4 takes the output of all of those agents as input, so it's able to retrospectively grade the job that each agent did. Initially, I attempted to put this on a scale of 1-5 as well, but the resultant JSON object became increasingly buggy, so I just switched it to a boolean true/false to maintain simplicity. For Agents 2 and 3, there are slightly more specific variable names like “stayed_in_lane” and “deviation_quality” so that the agent isn’t very generic when it assesses the quality of the other outputs and actually gives concrete feedback.
User Facing Summary
Since my final prototype of this project is to be user facing, Agent 4 also outputs a 3-4 sentence string that encapsulates the advice of the previous 3 agents. If this pipeline were in chatbot format, this is what would be outputted.
Outputs
Now finally came the time to test out the model. I fed in the 5 personas I created in Week 4 into the model and saved the results as .json files. Those files are available at my GitHub under the “outputs” folder, so if you are curious please refer to those as they’re too long for me to paste into this blog.
Although the output was much better than a single base prompt and gave very specific, actionable advice, there were some more minor edge-case issues that prevented full trustability in the model. Firstly, in the “revision_suggestions” category, the model continuously suggested financial products like 529 plans and Coverdell ESAs. Although for that circumstance using such financial savings accounts is not inherently wrong, the general rule of thumb for any financial assistant model is NOT to include something like that, as it is generally the role of a human financial advisor to suggest opening up things like Roth IRAs. Despite tweaking the prompts slightly I was unable to resolve this issue.
Secondly, the agent continuously labeled unknown heuristics as inapplicable. For example, if a person is employed, they most certainly have some level of employment benefits, even if it isn’t explicitly stated in the initial prompt. The model labels any 401(k) Employer Match as an “inapplicable_heuristic”, despite it fully being applicable and just being an unknown quantity. Despite prompt tweaking I was unable to resolve this issue as well.
A couple of other fields like “additional_considerations_incorporated” and the feasibility scores are also slightly buggy, but neither of them are large enough to prove harmful to the user in any meaningful way.
Future Looking
Throughout my project so far, I’ve been running my model (Gemini 2.5-Flash) at a temperature of 0. I did this to make the model as deterministic and predictable as possible. For reference, a higher temperature of an LLM means the coefficient of random sampling is greater. Essentially, this means the higher the temperature the more “random” and variant the responses will be run-on-run.
In order to prove some level of consistency in Agent 4’s evaluation, I will run it at a temperature of 0.3, and generate 3 responses and average the scores across the 3. This will ensure that across each person I’m getting a slightly different angle of scrutiny, similar to how it would look if 3 different financial advisors, with their own unique styles, looked at the same persona.
Also, I’ve mentioned time and time again that I would like to move my agents to Google ADK. However, for the sequential pipeline, this wasn’t the best idea, as ADK essentially does what my python code does in the backend, so transferring the agents over would lead to teething issues that would eat into the limited time that I have. As a result, I’m only going to develop the hierarchical agent (which I start working on next week) in Google ADK natively.
Likewise, this week I had a meeting with my secondary advisor Dr. Wang and she suggested for my survey I source real data from prolific, a site where you can pay to have your survey sent to a variety of users across that platform. Once my survey is complete, I will source part of my human data from there, and part of it from adults around me to, hopefully, get more robust and real demographic personas.
That is it for Week 6 of my blog! Thanks for reading and I’ll see you guys next week.
Week 7: Hierarchical Work!
Hello everybody and welcome back to my senior project blog! This week is important because it’s where I’ve started (at mostly finalized) my hierarchical pipeline! I’m excited to share that with you guys, so let’s get into it.
The Hierarchical Pipeline Overview
The sequential pipeline was an agentic framework where every agent in the sequence was invoked without question. Although this is a surefire way to ensure accuracy across all levels of agents, it’s not necessarily the most efficient or scalable method due to the token cost of invoking four instances of Gemini API calls. As a result, I decided to also use a hierarchical structure that essentially allows the agent to skip agent 2 if it believes that the persona it is dealing with has no need to do so.
In order to enable this, I’m going to create an orchestrator agent. An orchestrator agent is simply an overarching agent that has access to invoke tool calls if necessary. In order not to give the orchestrator too much freedom and have it stick to the general pipeline, it is mandated to start with agent 1 (convert the input into a JSON), and end with agent 4 (the quality control agent).
The ADK Integration
Initially, my goal with the hierarchical agent was to develop it natively with Google ADK, because it would be more scalable and, as a result, easier to implement into a web app. However, due to the issues with ADK’s implementation I ran into too many bugs, and instead have made a temporary python script that achieves the same pipeline with API calls.
Firstly, ADK works by establishing what is known as a “root agent”. Whenever you run the call “adk web” in your terminal, the package searches for an init python file that imports the root agent. We can then assign sub-agents to the root agent with the script shown below.
root_agent = LlmAgent(
name="financial_planning_orchestrator",
model="gemini-2.0-flash",
instruction=ORCHESTRATOR_PROMPT,
sub_agents=[
persona_profiler,
heuristic_strategist,
goal_optimizer,
quality_control
]
)
Each of these sub-agents can be defined with a similar framework (using the LlmAgent constructor) as shown below.
goal_optimizer = LlmAgent(
name="goal_optimizer",
model="gemini-2.0-flash",
instruction=AGENT3_PROMPT,
description="""Takes a persona profile and baseline plan and
produces a goal-optimized financial plan with specific deviations
from the baseline. Call this after heuristic_strategist, or
directly after persona_profiler if heuristic_strategist was
skipped."""
)
The “description” tag is what the root_agent (the orchestrator) uses to allocate which agents it wants to use. In our case, there are only two options it has: either to run agent 2 and then run agent 3, or just run agent 3 while skipping agent 2.
After this, I decided to test the system out using the ADK Web UI. At first, I was confused because after I initiated a prompt it seemed as if only agent 1 was run. In reality, the way the ADK web version works is it adds a “human in the loop” every single run. Essentially, you are the one who is giving the orchestrator the final say. On top of that, after agent 3 had run, agent 4 was never being called despite it being clear in the orchestrator’s system prompt that agent 4 was a non-negotiable.
Due to time constraints as I was travelling for spring break, I was unable to debug these fully, so I instead decided to use a makeshift python script with API calls that achieves the same goal. I gave the script a hard set at 3 applicable heuristics if it was to run agent 2, in order to give it a clear concrete understanding of when to skip agent 2 and when to run it. I also added two hard cases to skip agent 2: when the user has a FIRE (Financial Independence, Retire Early) goal or is a freelance/gig worker, where none of the heuristics in my document apply.
After this, I added a reasoning step to every instance call where the orchestrator would skip agent 2, and put a section for it to explain its reasoning. Below is an example where agent 2 was invoked.
"routing_decisions": {
"persona_profiler": {
"called": true,
"reason": "Always called first"
},
"heuristic_strategist": {
"called": true,
"reason": "Standard persona with sufficient applicable heuristics"
},
"goal_optimizer": {
"called": true,
"reason": "Always called"
},
"quality_control": {
"called": true,
"reason": "Always called last"
}
And here is an example of where it is not invoked:
"routing_decisions": {
"persona_profiler": {
"called": true,
"reason": "Always called first"
},
"heuristic_strategist": {
"called": false,
"reason": "Fewer than 3 applicable heuristics \u2014 baseline would not be meaningful"
},
"goal_optimizer": {
"called": true,
"reason": "Always called"
},
"quality_control": {
"called": true,
"reason": "Always called last"
}
Just for reference the \u2014 is an em-dash (we all know that AI loves to use those), but in the JSON schema it appears in its unicode format. Apart from what is shown above, the outputs are formatting identically to the sequential pipeline, with the outputs from agents 1, 3, 4, and 2 if applicable.
Conclusion
Due to the fact I was travelling for Spring Break, this week was purposefully a lighter week when it comes to progress. In the coming weeks I hope to set up my account on Prolific and begin with data collection as soon as possible. I also hope to debug the ADK implementation as ADK allows me to scale my data better and works as a more robust backend when I plan to create a web app. As always, the code for this week is available on my GitHub. Thanks for reading and I’ll see you guys next week!
Week 8: Evaluation Era
Hello everybody and welcome back to my senior project blog! This week I made a lot of progress that I’m incredibly excited to share with you guys!
Survey Stuff
One thing my secondary advisor Dr. Wang recommended I do a survey evaluating real, human data for personas. Although in week 3 I did a huge amount of demographic research in order to craft my 5 personas, I decided to use the site Prolific to gather this data instead.
As a result, I made my Prolific account, however I immediately ran into a few issues. Firstly, Prolific limits its surveys to just 5 questions. Luckily for me, this wasn’t a problem as 5 questions was plenty in order to craft well-rounded personas, however it may generally be a bigger problem for larger studies.
Prolific also gives me an option to use a representative sample of US demographic and census data. Initially, I thought this was perfect because it allowed me to cluster and spread my data across different demographic groups and craft more well-rounded personas. However, the issue is that if you do a representative study, Prolific mandates at least 300 responses. Since each response is roughly 0.20, the combined cost with fees exceeded $90. Although I knew this method would cost money, I wasn't willing to pay that much money.
Due to this, I decided to stick with a traditional standard sample and include 50 responses, meaning my survey would cost me $15 to $20, a much more reasonable amount. The questions I’ve crafted for the survey are listed below here.
Question 1 is the most simple of them all. It is the age question, and it essentially allows me to gauge what portion of my survey audience is in what age range. I predict this will mostly be middle-aged, in the 35-50 range. This maps exactly to the first part of any persona I input, which states “middle aged woman” or “young adult”.
Question 2 is meant to characterize the “cost of living” portion of the JSON. It essentially classifies every person into whether they live in a low COL area, a medium COL area, or a high COL area.
Question 3 is meant to characterize the type of worker. This is really important particularly to how the hierarchical pipeline thinks. Freelance and Contract workers typically skip agent 2 as traditional heuristics don’t apply to them.
Question 4 is meant to characterize if the person is in any financial stress. If the persona is supporting 3 or more people on their income, then they are under high financial stress and may not be able to meet all savings quotas.
This is simply the goal part of the JSON. It characterizes what the main goal of the persona is.
During the weekend I will publish the survey and hopefully results come by the time the next blog is out!
Starting to Evaluate
Now that I’ve finished the sequential and hierarchical pipelines, I wanted to begin evaluating. As a result, I wrote scripts for Variants A and B of my comparison. As a reminder, Variant A refers to simply giving the LLM a persona and asking it to give financial advice. Variant B refers to giving the LLM a person and the two grounding documents and asking it to give financial advice. Variant C refers to the sequential pipeline, and Variant D refers to the hierarchical pipeline.
However, I ran into a pretty large issue. The package I was using for my LLM, google.generativeai, had fully moved over to a brand new SDK called google.genai. This meant that all my code from previous iterations of the project had to be adjusted to this new package. Unfortunately this week, that took most of my time, and I had to adjust my API clients so that they matched well. I’ve fixed them all, but I’ve run into some API timeout limits, but the rest of the outputs will be out on my GitHub sometime in the next week.
Visualizer
I also did some research on how I want to wrap the final evaluation into a functional web app, as that needs to be completed by presentation day on May 16. I’ve decided to use a FastAPI backend with a React frontend. If you are unaware what those are, I will go over those in future blog posts as the web app starts to come to fruition.
Conclusion
Next week I plan to finish evaluating by crafting my evaluating rubrics, which I’ll go into detail in next week’s blog post. I also plan to begin wrapping my python code with FastAPI, and I’ll start showing my progress with that next week. I also hope to receive the results of my survey and craft some additional evaluation personas.
Thanks for reading this week and I’ll see you next week!
Week 9: Evaluation Time
Hello everybody and welcome back to my Senior Project Blog! This week was one of the most hectic weeks, but also one of the most important to the final evaluation that I will be presenting in May! I’m excited to get into it, so let’s get started.
API Stumbles
Since the beginning of this project, I was always met with a disclaimer message whenever I ran my API call in order to use Gemini models for my project. The message was something like, “Reminder: google.generativeai will be discontinued soon. Please switch to google.genai.” For the longest time, I ignored this message, but finally the day came. I tried to run my model once and I encountered an error, and I quickly realized that my API call wasn’t going through. Now was the time I had to switch to google’s new SDK, google.genai.
For anyone who is unaware, SDK stands for “Software Development Kit.” It’s a standardized way, typically in the form of an API, that allows users to use models, products, or software from a third party source. In this case, the SDK “google.generativeai” was the go-to if you wanted to access Google’s Gemini AI models.
However, in late 2024, Google released “google.genai”, which was a more compact version of google.generativeai that was faster, easier to use, and would be the new go-to AI SDK. Since I am doing this project in 2026, the time finally came for me to switch over. The key difference for my project was how I defined an instance of an LLM. In google.generativeai, you would define a model as an object, with parameters such as system_prompt, temperature, and model_type. However, in google.genai, you define something known as a client, which acts as an instance of the LLM. You can then feed in different information into that client, calling it many times. This allows for more simplified code, as you only have to define the client once, rather than defining it four teams for each sub-agent.
client = genai.Client(api_key=os.environ.get("GEMINI_API_KEY"))
As a result, I made the necessary changes, however I kept running into an error. The error was the famous 429 error. If you are knowledgeable about computers, you will recognize this as a “too many requests” error. This meant that my code was sending too many requests to Google servers at once, causing the system to overload and thus returning an error. This didn’t make much sense at first because the system was working fine previously, but I decided to add 30 second buffers in between. However, this still didn’t fix the problem. By this point, I was perplexed, so I turned to Claude, which is well known for its ability to be better than more developers at coding. I was adamant on the fact that this was an error with the way my Google account was being billed, however despite the numerous changes it recommended to me I couldn’t get an output. At this point, I turned to my Dad, and he eventually figured out that the reason was that Gemini-2.0-Flash wasn’t supported on the new SDK. I had to switch to Gemini-2.5-Flash, which was much slower. As a result, generating the outputs for all four variants on this new model took up to 30 minutes, but I eventually managed to retrieve the outputs.
As a result of the new code, I also transferred everything to a new GitHub repository. All the outputs are neatly stored there, so please take a look if you are interested.
The Rubric
Now we arrive at what is probably the most important part of my project and the answer to my main research question: do these agentic systems and frameworks actually create better financial advice? The answer to this question is inherently subjective because advice as a concept is something that applies differently to different people. Nonetheless, I decided to create a 6-pronged rubric to help grade scores.
Dimension 1: Personal Constraint Recognition: This measures how well the variant correctly identifies and responds to the persona’s specific constraints.
1 = Generic advice with no acknowledgment of specific constraints
2 = Constraints mentioned but advice doesn't meaningfully adapt to them
3 = Key constraints acknowledged with some adaptation in recommendations
4 = Most constraints reflected in specific recommendations with clear reasoning
5 = All major constraints drive concrete, differentiated recommendations throughout
________________________________________________________________________
Dimension 2: Goal Specificity: This measures how well the plan provides concrete, actionable steps toward the staged goal (specificity over vagueness)
1 = Goal acknowledged but no concrete steps toward it
2 = Steps provided but no dollar amounts or timelines
3 = Some specificity — either dollar amounts or a timeline, but not both
4 = Dollar amounts and timeline present, but assumptions not stated
5 = Dollar amounts, timeline, and explicit assumptions all present
________________________________________________________________________
Dimension 3: Heuristic Appropriateness: This measures how appropriate the heuristics that are mentioned are. It measures how well they actually apply to the system, and also if the system properly modifies them to be more specific.
1 = Applies heuristics mechanically with no awareness of failure conditions
2 = Applies most heuristics but misses at least one major inapplicability
3 = Correctly flags the most obvious inapplicable heuristic but applies others without
scrutiny
4 = Correctly handles most heuristics with appropriate caveats
5 = Correctly applies applicable heuristics, explicitly flags inapplicable ones with persona-specific reasoning
________________________________________________________________________
Dimension 4: Prioritization Logic: This measures the order and the steps that the model provides. Are they actually in a logical order, or are they just listed without being meant to be followed chronologically?
1 = Steps presented with no logical prioritization or in a counterproductive order
2 = Rough ordering present but missing a critical priority (e.g., employer match not mentioned first)
3 = Correct ordering for most steps with one notable misordering
4 = Correct ordering with clear reasoning for sequence
5 = Correct ordering with explicit reasoning and acknowledgment of trade-offs between competing priorities
________________________________________________________________________
Dimension 5: Behavioral Realism: This measures how realistic the goals are in a practical, non-idealistic scenario.
1 = Recommendations are financially or behaviorally impossible for this persona
2 = Mathematically possible but ignores significant behavioral constraints
3 = Realistic for an ideal version of this persona but doesn't account for likely friction
4 = Realistic with minor caveats, acknowledges some behavioral challenges
5 = Realistic, acknowledges behavioral challenges explicitly, and adjusts recommendations to account for them
________________________________________________________________________
Dimension 6: Cognitive Bias Awareness: This is solely meant to ensure that the model takes into account common cognitive biases that humans may encounter during these situations
1 = No mention of behavioral or psychological factors
2 = Generic mention of psychology without persona-specific relevance
3 = One or two relevant biases identified but mitigations are vague
4 = Multiple relevant biases identified with specific mitigations
5 = All relevant biases identified with concrete, persona-specific mitigations integrated into the plan itself
________________________________________________________________________
I then decided to weigh these. The first three dimensions are weighted 20% because they are the three things that the system is specifically designed to improve over generic advice. Dimensions 4 and 5 (prioritization and behavioral realism) are 15% each because they measure the practical quality of the advice (important, but not the utmost goal of the research question). Cognitive Bias Awareness is 10% because it doesn’t hold the weight of the other 5 dimensions.
The Results
Persona #1: Middle-aged white career professional, stable mid-income, suburban, family, no significant debt, wants to retire early
Dimension | Weight | Variant A | Variant B | Variant C | Variant D
Persona Constraint Recognition | 20% | 2 | 3 | 4 | 4
Goal Specificity | 20% | 2 | 4 | 5 | 5
Heuristic Appropriateness | 20% | 1 | 4 | 4 | 4
Prioritization Logic | 15% | 2 | 3 | 3 | 2
Behavioral Realism | 15% | 3 | 4 | 3 | 3
Cognitive Bias Awareness | 10% | 1 | 3 | 5 | 5
Weighted Total | 100% | 37% | 71% | 80% | 77%
________________________________________________________________________
Persona #2: Middle-aged white single parent, two young children, moderate income, suburban, wants to save for college in 10 years
Dimension | Weight | Variant A | Variant B | Variant C | Variant D
Persona Constraint Recognition | 20% | 3 | 3 | 5 | 4
Goal Specificity | 20% | 2 | 3 | 5 | 5
Heuristic Appropriateness | 20% | 1 | 4 | 4 | 4
Prioritization Logic | 15% | 2 | 3 | 5 | 4
Behavioral Realism | 15% | 4 | 3 | 3 | 3
Cognitive Bias Awareness | 10% | 1 | 3 | 5 | 5
Weighted Total | 100% | 44% | 64% | 90% | 83%
________________________________________________________________________
Persona #3: Near-retiree, high net worth, poor asset allocation, significant credit card debt, stable job in a city
Dimension | Weight | Variant A | Variant B | Variant C | Variant D
Persona Constraint Recognition | 20% | 2 | 3 | 5 | 5
Goal Specificity | 20% | 2 | 3 | 5 | 5
Heuristic Appropriateness | 20% | 1 | 3 | 5 | 4
Prioritization Logic | 15% | 3 | 3 | 3 | 4
Behavioral Realism | 15% | 2 | 3 | 3 | 4
Cognitive Bias Awareness | 10% | 1 | 3 | 5 | 5
Weighted Total | 100% | 37% | 60% | 88% | 90%
________________________________________________________________________
Persona #4: Recent college grad, Hispanic, entry-level salary, renting in HCOL area on the west coast, significant student debt
Dimension | Weight | Variant A | Variant B | Variant C | Variant D
Persona Constraint Recognition | 20% | 3 | 3 | 5 | 4
Goal Specificity | 20% | 2 | 2 | 5 | 5
Heuristic Appropriateness | 20% | 2 | 3 | 5 | 5
Prioritization Logic | 15% | 2 | 3 | 5 | 4
Behavioral Realism | 15% | 2 | 2 | 4 | 3
Cognitive Bias Awareness | 10% | 1 | 3 | 5 | 5
Weighted Total | 100% | 42% | 53% | 97% | 87%
________________________________________________________________________
Persona #5: Young Black skilled freelance worker, medium COL city, inconsistent income low to mid, no employer 401k, wants to stabilize
Dimension | Weight | Variant A | Variant B | Variant C | Variant D
Persona Constraint Recognition | 20% | 3 | 4 | 5 | 4
Goal Specificity | 20% | 2 | 3 | 5 | 5
Heuristic Appropriateness | 20% | 1 | 3 | 4 | 4
Prioritization Logic | 15% | 1 | 2 | 4 | 4
Behavioral Realism | 15% | 2 | 2 | 4 | 4
Cognitive Bias Awareness | 10% | 1 | 3 | 5 | 5
Weighted Total | 100% | 35% | 58% | 90% | 86%
Conclusion
All in all this week was incredibly productive. Despite some setbacks at the beginning, I was able to get my rubric scores for all five personas. The goal for next week is to publish my Prolific Survey from Week 8, and possibly get more personas to input into the model. I would also hope to start with the integration into a functional web app now that all outputs have been made. Thanks for reading and I’ll see you guys next week!
Week 10: Analysis and the Frontend
Hello everybody and welcome to Week 10 of my Senior Project Blog! There are just 2 weeks left until the final presentation, and I’m about 95% done with my project. However, I still have a couple of action items left to check off, and I’ll go over that in today’s blog.
Analyzing the Data
In last week’s post I attached rubric scores showing how well each variant performed across the five personas (if you want to see those click the hyperlink for reference). I’ve decided to leave it to just those five personas rather than create additional personas through a Prolific survey, as the timescope that would take is beyond the range of this project.
Also, because I forgot to mention it within the contents of the post, Variant A is a simple zero-shot prompt (asking for advice on the persona and nothing else), Variant B allows the model to have access to the two grounding documents, heuristics.md and biases.md, Variant C is the sequential agent pipeline, and Variant D is the hierarchical agent pipeline. I will refer to them as such in the rest of the blog.
When designing the rubric scores, it was inherently difficult to put an objective metric on a subject matter. In fact, this is true generally when dealing with Large Language Models. Typically, there are datasets known as “benchmarks” which are used to evaluate the performance of LLMs. Creating benchmarks and evaluating models on them is a whole research domain of itself. If you would like to read more about benchmarks, I recommend this article in Google Deepmind. In essence, I’m creating my own mini benchmark by testing these 5 personas as datapoints, however this is nowhere near robust enough to count as a true evaluation. However, for the purposes of this project, it will suffice.
On average across the five personas, Variant A had an average rubric score of 39%, Variant B had an average rubric score of 61%, Variant C had an average rubric score of 89%, and Variant D had an average rubric score of 85%. The jump from Variant A to Variant B of 22% is a significant gap, and it shows that by simply giving the LLM additional context, the quality of the advice significantly increases. The largest difference between two variants, however, was between Variant B and C, with a 28% jump between the two. In fact, Variant C was the highest performing of the four, with an average rubric score of 89%, 4 percent higher than the hierarchical. As a result, this proves the original thesis of the research that agentic structures do in fact create better advice.
Variant D and Variant C are generally extremely close, with Variant D actually slightly outperforming Variant C on the personas that were meant to be more complex (Personas 3, 4, and 5). Each of these scenarios has either high net worth, poor asset allocation, or is a near-retirement situation, which are the scenarios where applying baseline heuristics actually hinders overall performance. As a result, the ability to decide not to attempt to apply the heuristics created more robust advice. However, this was a downside for the simpler personas, where the linear structure of Variant C outperformed Variant D.
Behavioral Realism was where all models struggled. Variants C and D never received above a 4, and frequently received 3’s. In fact, based on realism, Variants C and D weren’t actually that much better than Variants A and B. The reason for this was the strict structure that Variants C and D enforced. When you try to look at things overly analytically, you’re bound to forgo realistic factors such as human perception, stress, and lack of motivation. As a result, the advice given by Variant A especially was much more targeted toward humans and didn’t feel as robotic as the advice in Variants C and D.
Heuristic Appropriateness was another factor that tended to cause some issues with Variants C and D. Generally, both variants had a tendency to overclassify many heuristics rather than flag more as inapplicable. This would result in the model attempting to apply up to 8 heuristics at once, which is physically impossible and infeasible in a non-ideal scenario.
Variant B was the variant with the largest range across personas, and the least consistent. Generally, when given more complex personas, it tended to outperform Variant A significantly as the added context of the heuristics, when to apply them, and when not to apply them, was key knowledge in allowing the model to identify scenarios and provide the right action items. However, when the persona was more simple, it tended to lean closer to Variant A’s scores than to Variant C’s scores.
Variant C and D performed the best when there was a very clear and obvious constraint in the persona. This shows that when given a more “textbook” example of a case, the Variant is able to identify the missing link and provide the right advice. When that missing link is more vague the model tends to struggle slightly more.
I’ll create graphs displaying all this on the slides I present in my mock presentation next week, so stay tuned for that!
Web App Time
The final product for this project was always to have a functional web app at the end of it. As a result, I’ve started designing the code for that. I’m not done with it yet, but by the end of Week 11 I will have a functional web app ready. However, I will go over some of the architecture to design it.
Firstly, I had to wrap the backend in FastAPI. FastAPI is a service that allows my agent calls to run asynchronously. If you think about how API calls work, it runs one call at a time in order. If 3 people want to run the sequential agent at the same time, the API immediately gets overloaded with 12 simultaneous API calls, causing an overload error and crashing the system. In order to prevent this, I had to define each system thread asynchronously.
Once I finished that, I began working on the ReAct frontend. ReAct is a JavaScript framework that works seamlessly with FastAPI to create a good-looking frontpage for your application. I’m currently in the process of implementing this, but I will provide more information when it is complete next week.
Conclusion
Overall, I made great progress this week! I’m excited to complete this project and complete the final presentation, and get ready for the symposium on May 16. Thanks for reading and I’ll see you guys next week.
Week 11: Finishing Touches and the Final Application!
Hello everybody and welcome back to my Senior Project Blog! I’m excited to share that I’ve essentially completed my project and am all but ready for presentation next Saturday on May 16! I have quite a bit to get into, so let’s get started.
Analyzing Data, but with graphs
Last week I promised that I would have graphical analyses for the rubric data I collected, so here they are:
These first two graphs simply show the average rating by persona and then the individual ratings for each persona across each variant. As I said in my previous blog, Variant C slightly outperformed Variant D across the board due to the rigor and care it takes going to each agent at a time without attempting to skip any agents.
This web chart compares the average score across each dimension for Variant A (in blue) and Variant B (in red). As we can see, Variant B outperforms Variant A in every single metric, showing that by simply adding the contextualizing documents we were able to see marked improvement in results.
This next graph is a more nuanced take on the field “goal_specificity” and “behavioral_realism”. Goal specificity for Variant C across the board received a 5, as it should. However, behavioral realism struggled in comparison, only receiving a 3 or 4 on all occasions. What this shows is that the goal specificity and behavioral realism metrics may be negatively correlated as it’s hard to make an extremely detailed plan that is also feasible for the average consumer.
Lastly, we have a comparison of Variant C and Variant D across the 5 personas. For every persona, Variant C performed better except for Persona 3. Persona 3 is of an elderly individuals with high net worth but poor asset allocation and a lot of debt, typically scenarios you tend not to apply baseline heuristics for, which is why Variant D excelled by skipping Agent 2 and moving straight to Agent 3.
The Full Stack App
Now we come to designing the full-stack architecture of the application. Up till this point, my entire program had been running locally on my computer. I had a python program that called an API and returned a JSON. I then ran four python files back-to-back (for Variant C), or called them situationally (for Variant D), and noted their responses as .json files. However, now comes the age-old problem of transferring code that works locally on your laptop into something that anyone can use anywhere at any time.
Asynchronous Endpoints and Wrapping the Backend
The first step in order to make code like this deployable is to wrap them in something known as endpoints. An endpoint is a URL that triggers a python function or python file. These are defined as @app.post in the code. All functions within this block are defined as “async def” (def is the traditional way of defining a function/method in python). Asynchronous calling allows the computer to process requests parallely. In essence, if I wanted to call “def run_variant_c”, it would take me up to 90 seconds to receive a response, which is valuable time that the computer is wasting. Instead, defining it asynchronously allows the frontend to call different endpoints while one is still running, helping streamline run-time and allowing for scalability. The Python library that does this is known as FastAPI.
Example of an API endpoint for Variant A.
@app.post("/run/variant-a", response_model=ProseResponse)
async def run_variant_a_endpoint(request: PersonaRequest):
"""
Variant A: Simple prompt.
Sends the persona directly to Gemini with no grounding documents
and no pipeline. Returns plain prose.
"""
start = time.time()
try:
result = await run_in_thread(run_variant_a, request.persona)
return ProseResponse(
variant="variant_a_simple_prompt",
persona=request.persona,
duration_seconds=round(time.time() - start, 2),
result=result["response"],
)
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
CORS Middleware
For security purposes, almost all browsers block websites from talking to different servers without explicit permission from that server. As a result, I had to implement CORS (Cross-Origin Resource Sharing) middleware that essentially lists all the valid URLs that are allowed to access the API endpoints. In this, I’ve only listed the domain of my final application, so no other website on the internet can inadvertently call my functions.
CORS Middleware as defined in the main python file.
app.add_middleware(
CORSMiddleware,
allow_origins=["https://personal-finance-assister.vercel.app/"],
allow_credentials=True,
allow_methods=["https://personal-finance-assister.vercel.app/"],
allow_headers=["https://personal-finance-assister.vercel.app/"],
)
The JavaScript Bridge
When designing a frontend, it always uses HTTP (Hyper-Text Transfer Protocols) to send requests. However, the Python files do not receive HTTP requests, so we need to translate them. As a result, in my codebase I created a JavaScript file “api.js” that converts the user’s input into a JSON object and sends it to the FastAPI URL. The file also waits for the backend to complete its work and receives the output to send to the final frontend file.
This function appends the endpoints to the main URL and awaits for the response, and stringifies the JSON.
const API_BASE_URL = import.meta.env.VITE_API_URL || "http://localhost:8000";
export const runAgentPipeline = async (variant, persona) => {
const endpoints = {
'A': '/run/variant-a',
'B': '/run/variant-b',
'C': '/run/sequential',
'D': '/run/hierarchical'
};
try {
const response = await fetch(`${API_BASE_URL}${endpoints[variant]}`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({ persona }),
});
This codeblock runs the request for all four variants (because all four run at once on the final app)
export const runAllVariants = async (persona) => {
const variants = ['A', 'B', 'C', 'D'];
const promises = variants.map(v => runAgentPipeline(v, persona));
return await Promise.allSettled(promises);
};
Frontend Design
The entire frontend for the project is contained in a file titled “App.jsx”. There are a bunch of state variables defined at the top which hold values of states in the frontend, such as the state of the text box, the loading state, the error state, the copy/paste state, and more.
const [persona, setPersona] = useState('');
const [loading, setLoading] = useState(false);
const [results, setResults] = useState({});
const [errors, setErrors] = useState({});
const [secondsElapsed, setSecondsElapsed] = useState(0);
const [expandedCards, setExpandedCards] = useState({});
const [copyStatus, setCopyStatus] = useState({});
Every single button in the frontend is connected to the JavaScript, which then handles the API calls and returns the raw JSON outputs. The frontend then formats these raw JSON outputs. For Variants A and B, it converts the JSON into markdown and then into readable text. For Variants C and D, it displays the Agent_3 plan and the Agent_4 summary. All of this uses a frontend framework known as React. It then exports the HTML for deployment.
Actually Transferring the Code Onto a Website
Now that we’ve done the code, we need to actually transport it to a place where it’s always accessible by these URLs. This is where the concept of a Docker comes in. We create a Dockerfile, which is essentially a file that tells a hosting service to first download python, then import all the necessary libraries that were used in this project, then to import all the python, javascript, and html files that were used to build this project. The Docker then builds this read-only image of the project and exposes the URL so that now the internet can access it.
Now that Docker is created, it needs a place to sit. I decided to use Google Cloud Run (gcloud) to hold Docker. This essentially stores the Docker on Google’s servers, so whenever a request comes in it contacts Google’s Servers and allows me not to use my own computer as a server. Cloud Run is a fully managed service, meaning that Google will automatically scale up and scale down and prevent API clogging, which is super useful and allows me not to have to worry about that stuff.
Lastly, we host the app. I chose to use Vercel to host the app because it’s extremely easy to use and it automatically links with GitHub, meaning any time I stage a commit it auto updates the app. I pointed the Vercel app to the Google Cloud Run, and that creates the functional app!
Obviously this app had a lot more struggles than is mentioned here, but I decided to mostly go over the architectures because I believe it’s more interesting than me going over why error 429 is misleading.
Conclusion
This essentially concludes my project! The only thing left is for me to touch up my slides a bit and then rehearse the presentation. In next week’s blog (week 12), I’ll upload a video of the full final presentation, if you’re unable to make it.
Here are some links for the entire project:
Deployed app: https://personal-finance-assister.vercel.app/
GitHub: https://github.com/arjunmaganti/Personal-Finance-Assister
Slides: https://docs.google.com/presentation/d/1XhS7zedHwRP75OE_ik2_EJmr1wdA_sAoUJHjybQ7Lyw/edit?usp=sharing
Thanks for reading this blog, and I hope to see you after my presentation on May 16!
Week 12: Reflection & Acknowledgements
Hello everybody and welcome to my final Senior Project Blog! This week was the culmination of the past 12 weeks of work, and I’m excited to say that the final presentation was successful!
Final Presentation and Slideshow
If you missed it, you can view a recording of my final presentation here.
You can also view the slide deck here.
Reflection
Overall, the Senior Project experience has been a great one not only regarding the academic skills I gained but also what it taught me about time management, meeting deadlines, and planning a project from the ground up. My prior experience in research definitely helped me from a technical standpoint when it came to conducting background research and doing the actual dirty work with science, math, and coding, however the Senior Project experience has helped me collaborate with a variety of different mentors and students who actively critique your project as you go. This change in pace has been a positive one and has pushed me to make my project much better than it would have been if I had conducted it solo.
Throughout the experience, the most important skill I learned was to meticulously document all my progress. From these blog posts to myA GitHub commits and to maintaining a Google Drive, it all contributed to my organizational skills becoming better as a result.
Acknowledgements
I would like to thank my two faculty advisors, Ms. Bhattacharya and Dr. Wang for their continued support across the 12 weeks. Ms. Bhattacharya has been incredibly helpful in helping me meet deadlines and stay on top of my work, while Dr. Wang has provided valuable technical feedback when it comes to the economic feasibility of my project.
I would also like to thank my external advisor Dr. Bhavana Tiple who has helped me formulate the idea for this project and has answered all technical questions regarding the coding and computer science behind it. I would like to thank her for taking time out of her busy schedule to assist me.
I would lastly like to thank my parents for helping fund this project. Although the cost was under 5 dollars, API usage is not free and as a result my parents were the ones who helped fund this project.
I would like to thank everyone for reading the blogs! If you would like to contact me about this project in general, please contact arjunmaganti2008@gmail.com.