Phi-4
Overview
Phi-4 is a 14-billion-parameter language model developed by Microsoft Research, released on December 12, 2024. It is the fourth iteration in Microsoft's Phi series, designed to excel in complex reasoning tasks, particularly in mathematics and STEM fields. The model employs a dense decoder-only transformer architecture and utilizes a context window of 16,384 tokens. Phi-4 was trained on 9.8 trillion tokens over 21 days using 1,920 NVIDIA H100 GPUs, incorporating a diverse dataset that includes high-quality synthetic data, filtered web content, and academic resources. This training approach emphasizes data quality and advanced reasoning capabilities, enabling Phi-4 to perform competitively with larger models despite its smaller size. The model is open-sourced under the MIT License, facilitating unrestricted commercial use and derivative works. Its strengths include efficient reasoning, suitability for edge deployments, and a transparent training methodology. Phi-4 is particularly well-suited for applications requiring complex reasoning, such as educational tools, on-device assistants, and research in synthetic data training methodologies. However, it has limitations, including a shorter context window compared to some peers and a focus on text-based tasks without support for multimodal inputs. Benchmark performances as of December 2024 include an MMLU score of approximately 85%, GSM8K accuracy around 91%, and HumanEval performance at about 83%. These results demonstrate Phi-4's competitive edge in reasoning-intensive applications relative to its parameter count and computational requirements.