#attention-mechanisms

#attention-mechanisms

Yoshua Bengio is a computer scientist at the University of Montreal in Canada. In 2019, he won an A. M. Turing Award - considered the most prestigious honour in computer science - for pioneering the 'deep learning' techniques that are now making artificial intelligence (AI) ubiquitous. Last month, he also became the first person to top 1 million citations on Google Scholar.

The Chinese artificial intelligence startup DeepSeek AI, which stunned the world in January with claims of dramatic cost efficiency for generative AI, is back with the latest twist on its use of the technology to drive down the price of computing. Last week, DeepSeek unveiled its latest research, DeepSeek-V3.2-Exp. On its corporate blog, the company claims the new model can cut the cost of making predictions, known as inference, by 75%, from $1.68 per million tokens to 42 cents.

Dong et al. (2019) and Tay et al. (2022) train on a mixture of denoising tasks with different attention masks (full, causal and prefix attention) to bridge the performance gap with next token pretraining on generative tasks.