Week 9: Closing the Gap, One Hyperparameter at a Time

From last week’s case study, I identified ResNet-50 as the strongest base model, with approximately 89% test accuracy and a clear set of failure modes concentrated on two similar fault pairs of early extension versus no hip rotation and inside-out versus outside-in path. This week, I am working to close those gaps through hyperparameter tuning.

The first decision was what to tune and in what order. The three axes I prioritized were learning rate and schedule, data augmentation strategy, and the depth of fine-tuning within the pretrained backbone. These three control the most consequential tradeoffs for a transfer learning setup on a small dataset, namely, how aggressively the model updates its features, how much synthetic variation it sees during training, and how many of the pretrained layers it is allowed to modify.

For learning rate tuning, I swept over five values from 5e-6 to 5e-4 and tested two scheduling strategies: a cosine annealing schedule that smoothly decays the learning rate to near zero, and a step decay that reduces it by a factor of 10 at fixed epoch milestones. The cosine schedule produced the best validation accuracy, likely because the smooth decay avoids the abrupt transitions of step decay that can destabilize fine-tuning on a small dataset. Although the difference was modest, at 1.5 percentage points relative to the step-decay, it was consistent across multiple runs.

Data augmentation was where I saw the most significant gains. Since spectrograms are not natural images, I had to think carefully about which augmentations are physically meaningful. I introduced small random time shifts (up to 0.3 seconds in either direction) to simulate variation in where the swing falls within the 5-second recording window, slight intensity jitter (±2 dB) to mimic variation in radar return strength across players and sessions, and SpecAugment-style frequency masking, which randomly zeroes out narrow horizontal bands of the spectrogram to force the model to rely on a broader set of frequency features rather than overfitting to a single dominant band. This configuration improved validation accuracy by 3 percentage points over the baseline augmentation, with the largest gains on the two fault pairs that ResNet-50 previously confused. The frequency masking was particularly effective. By randomly hiding portions of the low-frequency region where both early extension and no-hip-rotation produce their signatures, the model was forced to learn complementary features in the mid-frequency range, where these two faults diverge.

For fine-tuning depth, I tested three configurations: freezing everything except the final classification head, unfreezing the last residual block (layer4 in the ResNet-50 architecture), and unfreezing the last two blocks (layer3 and layer4). Unfreezing only the last block produced the best results. Keeping the earlier layers frozen preserved the general-purpose feature detectors learned from ImageNet, such as edge, texture, and gradient filters, which transfer well to spectrogram data. Unfreezing the last block allowed the model to adapt its highest-level feature representations to the specific patterns in the spectrograms without the risk of forgetting that comes with unfreezing too many layers on a small dataset. Unfreezing two blocks led to slight overfitting, with training accuracy climbing above 98% while validation accuracy plateaued five percentage points below the single-block configuration.

After combining the best settings from each axis, the tuned ResNet-50 achieved approximately 93% overall test accuracy on the player-stratified split, a meaningful improvement over the 89% baseline from last week. The per-class results were encouraging. Correct swings and no shoulder rotation both achieved above 95% accuracy, consistent with their visually distinct spectrograms. The inside-out and outside-in path faults, which the baseline model frequently confused, improved to around 90% each, with the remaining errors split evenly between the two. Early extension and no hip rotation, the most biomechanically similar pair, reached approximately 88% and 86% accuracy, respectively. The confusion between them decreased substantially but did not disappear entirely, which I suspect reflects a genuine overlap in their spectral signatures when viewed at 45°. A multi-angle radar setup would likely resolve this, but that falls outside the scope of a single-sensor system.

I also ran an ablation study to confirm that each tuning choice contributed independently. Removing the augmentation improvements while keeping the optimized learning rate and fine-tuning depth dropped accuracy back to about 90%. Reverting to the original learning rate while keeping the augmentations brought it to 91%. This confirmed that the gains were not redundant and that the augmentation strategy, particularly the frequency masking, was the single most impactful change.