#attention-mechanisms

#attention-mechanisms

The Chinese artificial intelligence startup DeepSeek AI, which stunned the world in January with claims of dramatic cost efficiency for generative AI, is back with the latest twist on its use of the technology to drive down the price of computing. Last week, DeepSeek unveiled its latest research, DeepSeek-V3.2-Exp. On its corporate blog, the company claims the new model can cut the cost of making predictions, known as inference, by 75%, from $1.68 per million tokens to 42 cents.

Dong et al. (2019) and Tay et al. (2022) train on a mixture of denoising tasks with different attention masks (full, causal and prefix attention) to bridge the performance gap with next token pretraining on generative tasks.