Meta releases PyTorch inference framework for edge devices

"The framework enables developers to take any PyTorch-based model from any domain-large language models (LLM), vision-language models (VLM), image segmentation, image detection, audio, and more-and deploy it directly onto edge devices without the need to convert to other formats or rewrite the model. The team said ExecuTorch already is powering real-world applications including Instagram, WhatsApp, Messenger, and Facebook, accelerating innovation and adoption of on-device AI for billions of users."

"Traditional on-device AI examples include running computer vision algorithms on mobile devices for photo editing and processing. But recently there has been rapid growth in new use cases driven by advances in hardware and AI models, such as local agents powered by LLMs and ambient AI applications in smart glasses and wearables, the PyTorch Team said."

"However, when deploying these novel models to on-device production environments such as mobile, desktop, and embedded applications, models often had to be converted to other runtimes and formats. These conversions are time-consuming for machine learning engineers and often become bottlenecks in the production deployment process due to issues such as numerical mismatches and loss of debug information during conversion."

ExecuTorch enables developers to deploy any PyTorch-based model — including LLMs, VLMs, image segmentation, detection, and audio — directly to edge devices without converting formats or rewriting models. ExecuTorch already powers applications on Instagram, WhatsApp, Messenger, and Facebook, accelerating on-device AI adoption for billions of users. Advances in hardware and AI models have expanded use cases toward local agents and ambient AI in wearables and smart glasses. Converting models to other runtimes often causes numerical mismatches, loss of debug information, and deployment bottlenecks. ExecuTorch provides familiar PyTorch tools optimized for edge devices to eliminate those conversions; a beta released a year ago.

Read at InfoWorld