Overview AI news Technical blog Live AI events Inference explained See our approach Inference explained See our approach Products Red Hat AI Enterprise Red Hat AI Inference Red Hat Enterprise Linux AI Red Hat OpenShift AI Explore Red Hat AI Red Hat AI Enterprise Red Hat AI Inference Red Hat Enterprise Linux AI Red Hat OpenShift AI Explore Red Hat AI Engage & learn Learning hub AI topics AI partners Services for AI Services for AI Platform solutions Artificial intelligence Build, deploy, and monitor AI models and apps. Linux standardization Get consistency across operating environments. Application development Simplify the way you build, deploy, and manage apps. Automation Scale automation and unite tech, teams, and environments. Platform solutions Artificial intelligence Build, deploy, and monitor AI models and apps. Build, deploy, and monitor AI models and apps. Linux standardization Get consistency across operating environments. Get consistency across operating environments. Application development Simplify the way you build, deploy, and manage apps. Simplify the way you build, deploy, and manage apps. Automation Scale automation and unite tech, teams, and environments. Scale automation and unite tech, teams, and environments. Use cases Virtualization Modernize operations for virtualized and containerized workloads. Digital sovereignty Control and protect critical infrastructure. Security Code, build, deploy, and monitor security-focused software. Edge computing Deploy workloads closer to the source with edge technology. Virtualization Modernize operations for virtualized and containerized workloads. Modernize operations for virtualized and containerized workloads. Digital sovereignty Control and protect critical infrastructure. Control and protect critical infrastructure. Security Code, build, deploy, and monitor security-focused software. Code, build, deploy, and monitor security-focused software. Edge computing Deploy workloads closer to the source with edge technology. Deploy workloads closer to the source with edge technology. Explore solutions Solutions by industry Automotive Financial services Healthcare Industrial sector Media and entertainment Public sector (Global) Public sector (U.S.) Telecommunications Solutions by industry Financial services Industrial sector Media and entertainment Public sector (Global) Public sector (U.S.) Telecommunications Discover cloud technologies Learn how to use our cloud products and solutions at your own pace in the Red Hat® Hybrid Cloud Console. Platforms Red Hat AI Develop and deploy AI solutions across the hybrid cloud. Red Hat Enterprise Linux Support hybrid cloud innovation on a flexible operating system. Red Hat OpenShift Build, modernize, and deploy apps at scale. Red Hat Ansible Automation Platform Implement enterprise-wide automation. New version Red Hat AI Develop and deploy AI solutions across the hybrid cloud. Develop and deploy AI solutions across the hybrid cloud. Red Hat Enterprise Linux Support hybrid cloud innovation on a flexible operating system. Support hybrid cloud innovation on a flexible operating system. Red Hat OpenShift Build, modernize, and deploy apps at scale. Build, modernize, and deploy apps at scale. Red Hat Ansible Automation Platform Implement enterprise-wide automation. New version Implement enterprise-wide automation. Featured Lightwell Red Hat AI Enterprise Red Hat OpenShift Virtualization Engine Red Hat Desktop See all products Red Hat AI Enterprise Red Hat OpenShift Virtualization Engine Red Hat Desktop See all products Try & buy Start a trial Buy online Integrate with major cloud providers Integrate with major cloud providers Services & support Consulting Product support Services for AI Technical Account Management Explore services Services & support Product support Services for AI Technical Account Management Explore services Training & certification Courses and exams Certifications Skills assessments Red Hat Academy Learning subscription Explore training Training & certification Courses and exams Skills assessments Red Hat Academy Learning subscription Explore training Featured Red Hat Certified System Administrator exam Red Hat System Administration I Red Hat Learning Subscription trial (No cost) Red Hat Certified Engineer exam Red Hat Certified OpenShift Administrator exam Red Hat Certified System Administrator exam Red Hat System Administration I Red Hat Learning Subscription trial (No cost) Red Hat Certified Engineer exam Red Hat Certified OpenShift Administrator exam Services Consulting Partner training Product support Services for AI Technical Account Management Partner training Product support Services for AI Technical Account Management Build your skills Documentation Hands-on labs Hybrid cloud learning hub Interactive demos Training and certification Build your skills Hybrid cloud learning hub Interactive demos Training and certification More ways to learn Blog Events and webinars Podcasts and video series Red Hat TV Resource library More ways to learn Events and webinars Podcasts and video series Resource library Discover resources and tools to help you build, deliver, and manage cloud-native applications and services. For customers Our partners Red Hat Ecosystem Catalog Find a partner Red Hat Ecosystem Catalog For partners Partner Connect Become a partner Training Support Access the partner portal Partner Connect Become a partner Access the partner portal Build solutions powered by trusted partners Find solutions from our collaborative community of experts and technologies in the Red Hat® Ecosystem Catalog. Buy a learning subscription Manage subscriptions Contact customer service See Red Hat jobs Developer resources Architecture center Security updates Customer support I want to learn more about: Application modernization Cloud-native applications We'll recommend resources you may like as you browse. Try these suggestions for now. Product trial center Courses and exams Resource library Get more with a Red Hat account Event registration Training & trials World-class support A subscription may be required for some services. Products & documentation Red Hat AI A platform of products and services for the development and deployment of AI across the hybrid cloud. Red Hat AI Enterprise Build, develop, and deploy AI-powered applications across the hybrid cloud. Documentation Try it Red Hat AI Inference Optimize model performance with an integrated stack for fast, consistent, and cost-effective inference at scale. Documentation Try it Red Hat Enterprise Linux AI Develop, test, and run generative AI models to power enterprise applications. Documentation Try it How to buy Red Hat OpenShift AI Build and deploy AI-enabled applications and models at scale across hybrid environments. Documentation Try it MCP servers A platform of products and services for the development and deployment of AI across the hybrid cloud. Red Hat AI Enterprise Build, develop, and deploy AI-powered applications across the hybrid cloud. Red Hat AI Inference Optimize model performance with an integrated stack for fast, consistent, and cost-effective inference at scale. Red Hat Enterprise Linux AI Develop, test, and run generative AI models to power enterprise applications. Red Hat OpenShift AI Build and deploy AI-enabled applications and models at scale across hybrid environments. Learn Basics What's new in Red Hat AI Why you should care about AI inference What is vLLM? What is AI inference? What is Agentic AI? What is AgentOps? What is llm-d? See all AI topics See all AI blogs Why Red Hat AI Fast, efficient inference Connecting models to data and agents Agentic AI Scaling AI Technical Learning hub Training & certification Point-and-click demos Current Red Hat AI customers AI quickstarts What's new in Red Hat AI Why you should care about AI inference What is AI inference? What is Agentic AI? What is AgentOps? See all AI topics See all AI blogs Fast, efficient inference Connecting models to data and agents

Training & certification Point-and-click demos Current Red Hat AI customers AI partners Hardware & infrastructure NVIDIA AMD Intel Dell Lenovo Cloud AWS Microsoft Azure Google Cloud IBM Cloud System integrators Accenture IBM Kyndryl See all AI partners Hardware & infrastructure Microsoft Azure System integrators Simply put, there’s no AI without inference. Inference is at the core of generative AI. But when big models have to execute even bigger strategies, things can get complicated. That’s why we’re breaking down the challenges and opportunities that come with AI inference—from model optimization with vLLM to the latest open source, distributed frameworks like llm-d. Jump to section Why is inference so important? Inference is the final step in a long and complex machine learning process, when a model delivers the desired output. Most importantly, it’s a necessary function for AI to be successful. That’s why the hardware and software that support your inference capabilities can make or break your AI strategy. AI inference 101 What happens after the prompt? Scale AI with open source Getting started with AI inference What’s holding you back from scaling? Inference gets a lot of pressure from models that keep growing bigger. As models get more complex, inference becomes slower. For inference to be successful, AI models need to do a lot of math in a short period of time. So, factors like model size, high user volume, and latency can all limit performance. When models require more data and more memory, hardware and accelerators struggle to keep up. Push the boundaries of LLM inference with Marlin How AI accelerators strengthen inference Faster inference with speculative decoding Deploy a lightweight AI model AI computing resources expected to be consumed by inference in 2026, up from 33% in 2023 and 50% in 2025. 1 So, how do you make inference better? When you optimize inference, AI models can run faster and smarter. Optimization methods include processing GPUs more efficiently, speculative decoding, sparsity, compressing models with quantization techniques, and distributed inference. Tools like LLM Compressor use the latest model compression research to make LLMs smaller, more energy efficient, and faster. This reduces hardware requirements and improves efficiency—without sacrificing accuracy. Optimizations like these help AI inference stay cost effective, so it can scale with your teams as you go. LLM Compressor: Optimize LLMs for low-latency deployments The economics of LLM Compressor LLM Compressor in production Check out the open source project Accuracy preserved during optimizations with LLM Compressor. 2 More computational throughput using compressed models, without sacrificing accuracy. 3 Cost savings without sacrificing performance when optimizing models with LLM Compressor. 4 How does vLLM optimize inference? Optimizing models is only half the battle. You also need a high-performing inference engine. That’s where vLLM can help. Traditional LLM memory management systems don’t organize memory in the most efficient way, which makes LLMs move slowly. vLLM uses PagedAttention, a memory management technique that identifies repetitive key values to reduce extra work for the LLM. This allows vLLM to make better use of GPU memory and speed up generative AI inference. It maximizes throughput (tokens processed per second) to serve many users at once. Using accelerators more efficiently means models can do more math in less time, so teams can serve more users and agents faster. Optimize LLM inference with vLLM vLLM: 3 real-world use cases Build more efficient AI with vLLM Parameters reduced when using sparsity structure. 5 Inference latency decreased with speculative decoding techniques. 6 Higher throughput performance with vLLM compared to competitors. 7 Why is vLLM so popular? vLLM has helped address the core issues around efficient GPU utilization, unlocking lower cost per token, stable latency at scale, and doing it with an open, portable deployment approach. That’s why the vLLM community is active and vibrant. Contributions come from passionate groups like Hugging Face, UC Berkeley, NVIDIA, Red Hat, and many more. The community consistently challenges and improves the software in the open source project. With Day 0 support for all major models and accelerators, its accessibility is attractive to both industries and academia. Join the vLLM community Register for a vLLM meetup vLLM Office Hours * Commits are updates, changes, and saves made to the open source project as contributors adjust vLLM to work for their use cases. vLLM GitHub commits*—an increase of over 200%—in 2025. The vLLM community today GPUs deployed 24/7 8 Different accelerator types 9 Supported model architectures 9 Unique contributors 9 Where does distributed inference fit in? Distributed inference allows AI models to divide the labor of inference across a group of interconnected devices. When a model can fulfill different requests—all at the same time—it significantly reduces the necessary hardware and increases inference efficiency. Distributed inference uses techniques like tensor parallelism, intelligent inference scheduling, and disaggregation. When layered with vLLM, inference becomes a very efficient, multitasking machine. This helps inference stay observable, scalable, and consistent. What is distributed inference? Intro to distributed inference More token throughput using tensor parallelism, a distributed inference architecture. 10 Is there an open source community for that? Yes, it’s called llm-d. llm-d is an open source framework that gives developers a blueprint for building distributed inference at scale. Its modular architecture supports the complex resource demands of sophisticated LLMs and replaces manual, fragmented processes with integrated well-lit paths, speeding up the time from pilot to production. llm-d brings inference to Kubernetes, providing a standardized tool-kit that helps apply distributed inference to your unique enterprise use cases. Inside distributed inference and llm-d Why do we need llm-d? Get started quickly with llm-d’s well-lit paths Baseline of Queries Per Second (QPS) sustained by llm-d. 11 More AI resources Red Hat AI experts explain inference Agentic AI systems with Red Hat AI Unlock smarter AI: inference- time scaling Build more efficient AI with vLLM What is generative AI? How to scale AI at the enterprise Why compressed models lead to cheaper inference Explore Red Hat AI Inference Server Kubernetes-native distributed inferencing Ollama vs. vLLM Build on vLLM with llm-d Platform engineering for AI agents Autoscaling vLLM with OpenShift AI Build a production- ready AI toolbox Ireland’s next steps for effective AI delivery Driving healthcare discoveries with AI Red Hat AI Inference Built on vLLM, our enterprise-grade inference engine enables faster inference without sacrificing performance. Scale across the hybrid cloud with your preferred and optimized gen AI model, on any AI accelerator, in any cloud environment. [1] “ Why AI’s Next Phase Will Likely Demand More Computing Power—Not Less .”The Wall Street Journal, 22 Jan. 2026. [2] Kurtić, Eldar, et al. “ We ran over half a million evaluations on quantized LLMs—here's what we found. ” Red Hat Developer Blog, 17 Oct. 2024. [3] Condado, Carlos. “ A strategic approach to AI inference performance. ” Red Hat Blog, 15 Sept. 2025. [4] Zelenović, Saša. “ Unleash the full potential of LLMs: Optimize for performance with vLLM. ”Red Hat Blog, 27 Feb. 2025. [5] Kurtić, Eldar, et al. “ 2:4 Sparse Llama: Smaller models for efficient GPU inference. ” Red Hat Developer Blog, 28 Feb, 2025. [6] Marques, Alexandre, et al. “ Fly Eagle(3) fly: Faster inference with vLLM & speculative decoding. ”Red Hat Developer Blog, 1 July 2025. [7] Kwon, Woosuk, et al. “ vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention. ” vLLM Blog, 20 June 2023. [8] Goin, Michael. “ [vLLM Office Hours #38] vLLM 2025 Retrospective & 2026 Roadmap - December 18, 2025. ” YouTube, Dec. 8, 2025. [9] Kwon, Woosuk. “ Today, vLLM supports 500+ model architectures, runs on 200+ accelerator types, and powers inference at global scale. ” X, Jan. 26, 2026. [10] Goin, Michael. “ Distributed inference with vLLM. ” Red Hat Developer, 6 Feb. 2025. [11] Shaw, Robert. “ llm-d: Kubernetes-native distributed inferencing. ” Red Hat Developers, 20 May, 2025. Red Hat Enterprise Linux Red Hat OpenShift Red Hat Ansible Automation Platform See all products Training and certification Customer support Developer resources Red Hat Ecosystem Catalog Try, buy, & sell Product trial center Buy online (Japan) Contact customer service Contact training Red Hat is an open hybrid cloud technology leader, delivering a consistent, comprehensive foundation for transformative IT and artificial intelligence (AI) applications in the enterprise. As a trusted adviser to the Fortune 500 , Red Hat offers cloud, developer, Linux, automation, and application platform technologies, as well as award-winning services. Customer success stories Analyst relations Open source commitments Our social impact Change page language Red Hat legal and privacy links Contact Red Hat Inclusion at Red Hat Cool Stuff Store Red Hat legal and privacy links Privacy statement All policies and guidelines Digital accessibility