Federated Whisper Aggregation Pipeline

Problem

Speech datasets are often too sensitive to centralise, which rules out conventional fine-tuning. Federated training removes the data-sharing requirement but introduces the question of which aggregation strategy actually preserves model quality.

Approach

• 01Engineered a federated pipeline fine-tuning whisper-large-v3-turbo across decentralised speech datasets using Flower on a Linux VM.
• 02Implemented weighted checkpoint averaging across more than ten aggregation strategies, with automated orchestration across configurations.
• 03Evaluated on HuggingFace datasets, logging WER and BLEU per strategy.
• 04Validated exported state dicts directly against Whisper's native inference backend.

Outcome

A reproducible aggregation and evaluation stack that compares strategies under zero data sharing.