Week 7: Testing Encryption and Models

This week at MITRE, we were focused on setting up the encrypted memory environment to complete the side-by-side demonstration. Setting up hardware-level memory encryption is more complex than I initially expected. We need to configure a virtual machine with encrypting technologies like AMD SEV (Secure Encrypted Virtualization) or Intel TDX (Trust Domain Extensions) enabled, which means we need specific hardware support and BIOS/firmware settings that are harder to implement with all of MITRE’s restrictions. The encrypted VM needs to run the same setup as the unencrypted (the language model, the web interface, and allow the memory scanner to run), but when the scanner tries to read memory, it should only see encrypted gibberish instead of plaintext.

I’m also starting to prepare for the end-of-spring presentation at MITRE. This is when I present what the demonstration shows, why it matters from a security perspective, and how hardware-level encryption protects against memory-reading attacks.

On the research side, I started running the Random Forest models on the test set this week. Last week I finished building the models and verifying that everything was set up correctly, making sure the data was loading properly, the predictions were outputting in the right format, and there was no data leaking between the training and test sets. This week was about actually running the evaluation.

I’m also calculating different things to understand how well the models perform. I’m using accuracy to see the percentage of businesses the model classifies correctly overall. Precision to see how many of the businesses the model predicted would adopt AI that actually did adopt it (important for avoiding false positives). And I’m using recall to see how many of the actual AI adopters the model successfully identified (important for not missing businesses that would succeed). Finally, ROC-AUC gives me an overall score from 0 to 1 showing how well the model distinguishes between adopters and non-adopters.

I’m running these separately for the micro-business model (1-9 employees) and the larger small business model (10-49 employees), and comparing both to the logistic regression baseline I set from Week 4. The process is taking longer than expected because I’m also checking for issues like class imbalance (when there is more non-adopters than adopters in the dataset) and making sure the results are statistically reliable.