Week 6: Model Finalization

This week at MITRE, we continued adding to the live memory demonstration and started working towards encrypting the LLM. The web chat interface and memory scanner are working reliably now, so we’ve been testing them with different types of conversations and more to make sure the scanner consistently gets the results we want. We also started documenting how the demonstration works to explain the technical setup, what the attack looks like step by step, and what the results show.

I also spent time this week learning more about how web applications handle real-time data updates. For the chat reader interface from the RAM, we needed the conversation to update smoothly as messages come in, which meant understanding how to structure the frontend code to look for new data from the back without constantly refreshing the page. Similar to last week, this is also really relevant to my research project since the AI readiness tool will need to process user input and display results dynamically.

On the research side, I finished building the Random Forest models this week. After testing different hyperparameter configurations last week. I settled on 100 trees with a maximum depth of 15 based on initial cross-validation results during training. The models are built and trained for both micro-businesses (1-9 employees) and larger small businesses (10-49 employees), but I still need to run them on the test set to get final performance numbers and compare them to each other.

Right now, I’m making sure everything is working correctly before I evaluate the final results. So I’m checking that the models are loading the right data, that the predictions are outputting in the correct format, and that the evaluation metrics are calculating properly. I also need to verify that the train/test split happened correctly and that there’s no data leakage (where information from the test set accidentally gets used during training, which would make the results unreliable).