Google launches VaultGemma: privacy AI without compromising performance

"Google presents VaultGemma, an AI model that protects sensitive data without compromising performance. The 1 billion-parameter model uses differential privacy and will be available as open source. Differential privacy works by adding controlled noise to datasets. This makes it impossible to retrieve specific information while maintaining overall usability. VaultGemma was built from the ground up and trained with a differential privacy framework to ensure that it cannot remember or leak sensitive data."

"Traditional scaling laws for AI models do not apply when differential privacy is applied. Google therefore developed new "DP Scaling Laws" that take into account added noise and larger batch sizes. This breakthrough enables the development of larger and more powerful private language models. The team adapted the training protocols to counteract the instability caused by noise addition. Private models require batch sizes with millions of examples to train stably. Google found ways to reduce these computational costs without undermining privacy guarantees."

"In evaluations on benchmarks such as MMLU and Big-Bench, VaultGemma performs comparably to non-private Gemma models with the same number of parameters. This is remarkable because previous differential private models always performed significantly worse. VaultGemma uses a decoder-only transformer architecture with 26 layers and Multi-Query Attention. The sequence length is limited to 1,024 tokens to keep the intensive computational requirements of private training manageable."

VaultGemma is a 1 billion-parameter language model that applies differential privacy during training to prevent memorization or leakage of sensitive data. Differential privacy is implemented by adding controlled noise to datasets, preserving aggregate utility while blocking retrieval of specific records. The model was built from the ground up with a DP training framework and uses a decoder-only transformer architecture with 26 layers, Multi-Query Attention, and a 1,024-token context window. New DP Scaling Laws account for added noise and very large batch sizes, enabling larger private models and improved training stability. Optimizations reduced computational costs without weakening privacy guarantees. VaultGemma matches non-private Gemma performance on benchmarks and will be released open source via Hugging Face.

Read at Techzine Global