Week 5: Uno, Dos, Tres

Hello everybody!

We’re about one month into the senior project (crazy how time flies)! This week I’ve made a lot of progress in finishing the sequential pipeline, and I’m incredibly excited to delve into that. I’ve also made a couple of tweaks to my final prototype that I plan to show at the senior project panel, which I’ll also go into later in this post.

Agent 2

Backtracking to the architecture of the sequential pipeline, Agent 1 was always going to be the easiest to design as it had no choice of context for me to deliberate on. The only things I needed to consider when designing the system prompt were the two grounding documents (heuristics.md and biases.md), as well as the persona itself (which is just a sentence or so). As a result, there was no real “context engineering” done here other than the two grounding documents, as there was no real need for me to deliberate what context I wanted to include.

However, for agent 2, it started to get a little bit more complicated. Agent 1 outputs a JSON schema that characterizes the persona based on their employment status, debt load, housing situation, and more. The goal of agent 2 was to transform this JSON schema into a second JSON schema which takes each heuristic listed by agent 1 and gives a specific recommendation, monthly impact, and priority value to it. 

{{
 "baseline_plan": [
   {{
     "heuristic": "",
     "recommendation": "<specific, actionable instruction derived strictly from this heuristic>",
     "monthly_impact": "",
     "priority": <integer, 1 = highest>
   }}
 ],
 "inapplicable_notes": [
   {{
     "rule": "",
     "reason": ""
   }}
 ],
 "assumptions": [
   ""
 ],
 "baseline_summary": "<2-3 sentence plain-language summary of what this baseline plan does and does not address>"
}}

As a result, the system prompt needed to be EXTREMELY clear in stating that the goal of agent 2 was ONLY to take the heuristics that agent 1 thought the persona could be linked to, and follow them verbatim. Similarly, it needed to have a calculable monthly impact to show what the effect of following that heuristic would be on the individual. Additional heuristics or biases that agent 1 may have included (on purpose or hallucinated) didn’t have a necessary need in this agent.

At first, this was a huge challenge for me because I was unable to prevent the agent from either hallucinating new heuristics or from using ones that didn’t apply to the persona at hand, despite being fed in the JSON file from agent 1. In order to fix it, I restructured some of the prompting in agent 1, making sure that the reasoning attached to each additional heuristic was clear, and I fixed up some of the prompting in agent 2 to provide direct step-by-step instructions on how to tackle additional_heuristics. I also added a new field additional_biases which worked the same exact way as additional_heuristics. 

Likewise, agent 2 kept on listing monthly impact as “indeterminate”. I realized that this was because I wasn’t actually feeding it any data regarding the persona’s income (which was in the original prompt given to agent 1). So, I tweaked the system prompt to include the original prompt in the larger API call, and it fixed the issue.

Here was the output from agent 2 for this persona: 28-year-old, $58k salary, renting in Austin TX, $22k in student loans, stable job at a tech company, wants to eventually buy a house.

{
  "baseline_plan": [
    {
      "heuristic": "50/30/20 Rule",
      "recommendation": "With a $58,000 salary, and assuming a 25% tax rate, monthly take-home pay is approximately $3,625. Allocate $1,812.50 to needs, $1,087.50 to wants, and $725 to savings and debt repayment.",
      "monthly_impact": "$725",
      "priority": "N/A"
    },
    {
      "heuristic": "Emergency Fund",
      "recommendation": "Aim to save 3 months of essential living expenses in a high-yield savings account. Assuming $1,812.50 in monthly needs, the target emergency fund size is $5,437.50.",
      "monthly_impact": "Indeterminate",
      "priority": 1
    },
    {
      "heuristic": "Avoid High-Interest Debt",
      "recommendation": "Allocate the savings/debt repayment portion of the 50/30/20 rule ($725/month) toward the $22,000 in student loans. Prioritize paying off any student loans with interest rates above 7-8%.",
      "monthly_impact": "$725",
      "priority": 2
    },
    {
      "heuristic": "Age-Based Asset Allocation",
      "recommendation": "As a 28-year-old, and using the rule of 110, allocate 82% of investments to equities and 18% to fixed income (110 - 28 = 82). This applies to any investment accounts outside of the emergency fund and debt repayment.",
      "monthly_impact": "Indeterminate",
      "priority": 3
    }
  ],
  "inapplicable_notes": [
    {
      "rule": "15% Retirement Savings",
      "reason": "Paying off student loans and saving for a down payment should take priority over saving 15% for retirement."
    },
    {
      "rule": "Housing Costs Under 30%",
      "reason": "They are currently renting, so this rule is not applicable until they start seriously considering buying a house."
    },
    {
      "rule": "401(k) Employer Match",
      "reason": "Whether or not this is applicable depends on if their employer offers a 401(k) match, which is unknown."
    },
    {
      "rule": "Pay Yourself First",
      "reason": "Paying off student loans and saving for a down payment should take priority over paying themselves first."
    },
    {
      "rule": "Rule of 72",
      "reason": "The Rule of 72 is not applicable at this stage."
    }
  ],
  "assumptions": [
    "Assumed a 25% effective tax rate for calculating take-home pay.",
    "Assumed student loan interest rates are known and can be prioritized from highest to lowest."
  ],
  "baseline_summary": "This baseline plan establishes a budget framework, prioritizes building an emergency fund and paying down debt, and provides a basic asset allocation guideline. It does not address the persona's specific goal of saving for a down payment, nor does it consider the potential impact of a 401(k) employer match."

This output is extremely promising and gives a great baseline for agent 3 to work and compare from!

Agent 3

Agent 3 is the heart of the sequential process. It’s the key differentiating factor between the typical heuristic-following agent 2’s plan, and trying to work out manually what IRAs to open, where to invest, how to save, and how much to spend. The goal of agent 3 is to give extremely specific advice, and also documents exactly WHAT it’s changing and WHY it’s making those changes from the base heuristics. As a result, it has a very robust JSON schema that is meant to make evaluating its performance much easier.

{{
{{
 "optimized_plan": [
   {{
     "heuristic": "",
     "source": "baseline | modified | new",
     "recommendation": "<specific, actionable instruction with dollar amounts where possible>",
     "deviation_from_baseline": "",
     "rationale": "",
     "monthly_impact": "",
     "priority": ""
   }}
 ],
 "goal_alignment": {{
   "goal": "",
   "addressed_by": [""],
   "projected_timeline": "",
   "gap_analysis": ""
 }},
 "additional_considerations_incorporated": [
   {{
     "item": "",
     "incorporated": true,
     "application": ""
   }}
 ],
 "cognitive_risk_mitigations": [
   {{
     "bias": "",
     "mitigation": ""
   }}
 ],
 "assumptions": [""]
}}

The important thing here is for each heuristic, there is a “source” field which states whether the heuristic is the baseline, modified, or new. In most or all circumstances, this should be modified or new, but in some cases the baseline heuristic might be sufficient for the persona. As well as that, this agent also outputs a ranking, monthly impact, and rationale.

Goal alignment is a new field that essentially takes into account the applicability of the plan that the agent put in place. My goal was to ensure that the agent was answering the original goal of the person, and it means that it needs to not deviate from that plan. When I ran the agent for the first time, these two first parts of the JSON schema worked amazingly: they gave incredibly detailed plans which went over exactly WHAT changed and WHY it changed. I’m extremely happy with the results that came with the first half of the agent’s output.

However, the second half which includes additional_considerations_incorporated and cognitive_risk_mitigations were extremely buggy. The goal of these two were to go over any additional considerations that agent 1 may have outputted (e.g. Debt Snowball which is something that isn’t in the original grounding documents) and try to incorporate it into the system. However, the agent is time and time again incorporating it in the original optimized_plan subsection. Also, for cognitive_risk_mitigations, it only includes cognitive risks specifically from biases.md, and not the other cognitive risks it took from its own knowledge base. Despite tweaking the prompt multiple times, I was unable to fix this error. 

{
  "optimized_plan": [
    {
      "heuristic": "50/30/20 Rule",
      "source": "modified",
      "recommendation": "Based on a $58,000 salary, and prioritizing debt payoff and down payment savings, allocate approximately $2,417/month to needs (50%), $1,160/month to wants (24%), $967/month to debt repayment (20%), and $290/month to down payment savings (6%).",
      "deviation_from_baseline": "Reduced 'wants' allocation from 30% to 24% and created a new 6% allocation for down payment savings. This shifts discretionary spending toward the stated goal of buying a house.",
      "rationale": "The persona's goal is to eliminate debt and save for a down payment. Reducing the 'wants' category allows for a dedicated savings allocation without sacrificing debt repayment.",
      "monthly_impact": "$1257",
      "priority": "N/A"
    },
    {
      "heuristic": "Emergency Fund",
      "source": "modified",
      "recommendation": "Initially, save 1 month of essential living expenses ($2,417) in a liquid, easily accessible account. Once this minimum is met, pause emergency fund contributions and direct those funds toward debt repayment and down payment savings. After the student loan is paid off, increase the emergency fund to the recommended 3-6 months of expenses ($7,251 - $14,502).",
      "deviation_from_baseline": "Reduced the initial emergency fund target to 1 month of expenses to accelerate debt repayment and down payment savings. Deferred building the full 3-6 month fund until after debt is eliminated.",
      "rationale": "Accelerating debt repayment directly addresses the persona's goal of eliminating student loan debt. A smaller initial emergency fund allows for faster progress on debt and down payment savings, with the understanding that this increases short-term risk.",
      "monthly_impact": "Variable",
      "priority": 3
    },
    {
      "heuristic": "Avoid High-Interest Debt",
      "source": "modified",
      "recommendation": "Allocate $967/month from the 50/30/20 rule towards the $22,000 in student loan debt. Prioritize the highest interest rate loans first. Given the $22,000 debt at an assumed average interest rate of 6%, this will take approximately 24 months to repay. Consider a debt avalanche or snowball method based on psychological preference.",
      "deviation_from_baseline": "Added a timeline estimate for debt repayment and suggested considering debt avalanche or snowball methods.",
      "rationale": "Providing a timeline makes the debt repayment goal more concrete and motivating. Acknowledging different repayment strategies caters to individual preferences and increases adherence to the plan.",
      "monthly_impact": "$967",
      "priority": 1
    },
    {
      "heuristic": "Age-Based Asset Allocation",
      "source": "baseline",
      "recommendation": "As a 28-year-old, a moderate asset allocation strategy (using the 110 rule) suggests holding 82% of investments in equities and 18% in fixed income. This applies to any investment accounts beyond the emergency fund and debt repayment.",
      "deviation_from_baseline": null,
      "rationale": "This is a standard recommendation for long-term investing and remains appropriate.",
      "monthly_impact": "Indeterminate",
      "priority": 4
    },
    {
      "heuristic": "401(k) Employer Match",
      "source": "new",
      "recommendation": "Investigate the availability of a 401(k) employer match. If available, contribute at least enough to receive the full match, even if it means temporarily reducing the down payment savings rate. This is essentially 'free money' and should be prioritized.",
      "deviation_from_baseline": "This heuristic was not addressed in the baseline plan due to lack of information. It is added here as a critical consideration.",
      "rationale": "Capturing the full employer match provides an immediate and guaranteed return on investment, which accelerates progress toward long-term financial goals, including buying a house.",
      "monthly_impact": "Varies based on employer match",
      "priority": 2
    }
  ],
  "goal_alignment": {
    "goal": "Eliminate $22k in student loan debt while saving for a down payment on a house.",
    "addressed_by": [
      "50/30/20 Rule",
      "Emergency Fund",
      "Avoid High-Interest Debt",
      "401(k) Employer Match"
    ],
    "projected_timeline": "Student loan payoff in approximately 24 months. Down payment savings will depend on the target down payment amount and the actual savings rate after accounting for the employer match. Assuming a $290/month contribution, a $20,000 down payment will take approximately 69 months (5.75 years).",
    "gap_analysis": "The plan does not include specific strategies for increasing income or reducing expenses to accelerate savings. It also does not address the specific type of house the persona wants to buy or the target location, which would influence the down payment goal."
  },
  "additional_considerations_incorporated": [],
  "cognitive_risk_mitigations": [
    {
      "bias": "Status Quo Bias",
      "mitigation": "Set a recurring calendar reminder every 6 months to re-evaluate the debt repayment and down payment savings plan. This forces an active decision point and prevents sticking with a suboptimal plan due to inertia."
    }
  ],
  "assumptions": [
    "Assumed a combined federal and state income tax rate of 25% to arrive at an approximate monthly net income of $4,833.",
    "Assumed student loans are the only form of high-interest debt.",
    "Assumed essential living expenses align with the 'needs' category of the 50/30/20 rule.",
    "Assumed an average interest rate of 6% on student loans for repayment timeline calculation.",
    "Assumed a $20,000 target for the down payment savings goal."
  ]
}

This output however, is still extremely good! I’m very happy with the progress I’ve made for the sequential pipeline.

Conclusion

The goal for next week is to wrap up agent 4 and complete the full agent pipeline. I also want to begin moving all these agents to ADK. However, ADK is an extremely complex framework and requires quite a bit more debugging the local API call. As a result, depending on the timeframe I’m working with, I’ll reconsider if moving them to ADK is the best idea.

Also, my initial plan for evaluation was to holistically and objectively analyze the quality of the advice that the system outputs. However, my faculty advisor Mrs. Bhattacharya suggested that I also include a survey with real customers to see how humans would use such a system. As a result, around weeks 8-9 I plan to incorporate the survey and distribute it around to adults.

The code for this week’s section is on my GitHub. Check it out if you’re curious!

Thanks for reading my blog and I’ll see you guys next week!