AI Model Co-Design: Hardware-Friendly LLM Design
AI performance depends on three dimensions: accuracy, throughput, and interactivity. This post focuses on throughput and interactivity, examining how model-design choices can optimize both without sacrificing accuracy, aiming to push the Pareto frontier outward.
AI performance comes down to three dimensions:
Accuracy: How well the model reasons and produces outputs
Throughput: How many tokens per second a datacenter can generate
Interactivity: How responsive the model feels to a user, dominated by latency
Deployments must balance all three: High accuracy is wasted if responses are slow, and raw throughput means little if each user’s experience is laggy. Practical systems therefore optimize accuracy, throughput, and interactivity together.
This post focuses on throughput and interactivity, and how model-design choices shape both without sacrificing accuracy (we flag accuracy trade-offs where they arise).
Holding accuracy fixed, the problem becomes a two-dimensional Pareto frontier: improving one usually costs the other. The goal is to push the whole frontier outward, maximizing the area under the curve (see Figure 1, below).
Figure 1. System throughput versus interactivity Pareto frontier