Kimi K3 officially open source: 2.8 trillion parameter model weights, technical reports released simultaneously

The Zhitong Finance App learned that in the middle of the night of July 23, Moonshot AI (Moonshot AI) announced the weight of the open source Kimi K3 model, released a technical report, and simultaneously opened the three critical infrastructure (Infra) technologies MoonEP, FlashKDA, and AgentEnv. This release covers models, training methods, and underlying training systems. Developers can download models for deployment and secondary development. The specific scope of use is subject to the Kimi K3 license.

According to the official presentation, Kimi K3 uses a hybrid expert (MoE) architecture, has a total parameter scale of 2.8 trillion yuan, has native visual understanding capabilities, and supports 1 million token context windows. Compared with the previous generation Kimi K2.5, the Kimi K3 parameter scale has increased by about 3 times. Dark Side of the Moon said that with limited computing power resources, the large-scale efficiency of the model was increased by about 2.5 times using technologies such as Kimi Delta Attention, Attention Residuals, and MoonEP.

The Kimi K3 technical report was also released at the same time as the model weights. According to public content, the report revealed a number of technical details of model training and architecture design, including KDA and Gated MLA hybrid attention structure, Stable LatentMoe expert routing mechanism, MoonVit-V2 visual encoder training scheme, and post-model training and evaluation methods. Among them, the model uses an MoE design with 896 routing experts, 16 experts per token activation, and covers training and evaluation in various competency directions such as general reasoning, agents, and programming agents.

In addition to the model itself, Dark Side of the Moon also announced open source key infrastructure technology to support Kimi K3 training. Among them, MoonEP is a high-performance expert parallel communication library for large-scale MoE models; FlashKDA is a high-performance computing operator corresponding to Kimi Delta Attention. Official data shows that its prefill speed on the Nvidia H20 is 1.72 to 2.22 times higher than the Flash-Linear-Attention baseline; agentEnv is an agent sandbox system developed by Dark Side of the Moon in collaboration with KVCache.AI. It can support fast snapshots, recovery, and Fork and other capabilities are used in large-scale agent training environments.

Dark Side of the Moon said that the purpose of this opening is to promote the development of an open weight model ecosystem. It hopes to lower the threshold for model deployment and development through open models, training methods, and underlying infrastructure, and provide more reusable technical resources for the AI community and related research.