Respan releases Span-02 weights that run only on its Rust and CUDA server
Respan's 15 October post releases Span-02. The card says the weights run only on its Rust and CUDA engine, on Blackwell GPUs.
Topic · 51 stories · page 2 of 3
51 stories tagged inference on AI Tamer, newest first.
Respan's 15 October post releases Span-02. The card says the weights run only on its Rust and CUDA engine, on Blackwell GPUs.
Thinking Machines' 15 October note marks a year of Tinker. Compatible sampling is already in beta. Three models retire on 23 October 2026.
At OCP, Intel named Crescent Island the GPU D-series. GPU D270 is a 350 W air-cooled PCIe card. Specs are Intel's. No price or ship date.
NVIDIA and Tri Dao's lab describe ReplaySSM, which caches recent SSM inputs instead of writing full state every step. The speedups are the authors' benchmarks.
Scott Guthrie says Azure Maia 300, Microsoft's third in-house inference chip, begins production deployments in 2027 and is meant to be the company's main inference system. The efficiency figures are Microsoft's.
River Inference is serving DeepSeek V4.1 Flash and GLM 5.3 Flash. Its public card is 20% under DeepSeek's off-peak rates and Z.ai's list prices. A second homepage table is much higher.
MIG cuts a supported NVIDIA GPU into up to seven instances with their own memory, cache and compute. How to create them and pick a profile for a model server.
Dropout changes a model’s forward pass according to module mode. Learn how masking, scaling, evaluation mode and gradient recording affect a reliable inference path.
Mamba changes a recurrent state according to each input. That keeps generation memory bounded by the model state, while making recall depend on what the state retained.
Grouped-query attention gives several query heads one key-value head pair. That shrinks decoding caches, with speed and quality gains that depend on the model and workload.
Mirai's open-source Rust engine uzu uses tree speculative decoding tuned for M5 Macs. A creator run shows 92 tokens/s on a 9B model versus 22 for llama.cpp, with caveats on quants, chips and workload.
Long prompts can fill GPU memory through the key-value cache. Moving that cache to the CPU saves GPU space, with a possible speed cost.
A small model can draft tokens for a larger model to verify. The speed gain depends on how quickly it drafts and how often its proposals are accepted.
A model’s weights are only part of its memory needs. The attention cache grows with the conversation, and saving space can slow generation.
Continuous batching lets a serving system replace finished requests while longer responses keep generating. Here is how that changes GPU use and what limits it.
Memory requests place a Pod. Limits constrain its containers. Learn how to tell a container out-of-memory kill from node-pressure eviction and size inference workloads accordingly.
Use a Kubernetes init container and a shared volume to put model files in place before an inference server starts. Plan for retries, storage use, and model loading.
A PodDisruptionBudget can limit how many inference pods a node drain evicts at once. Its allowed-disruptions count shows when the workload has room for another eviction.
Inference requests can wait while CPU utilization stays below its scaling target. Measure pending work and queue delay to decide when more workers are useful.
A sudden inference surge meets several gates before new Pods serve traffic. Follow the autoscaler's measurement loop, replica rules, Node capacity, and readiness.
Both PyTorch modes skip gradient recording. Inference mode also changes the tensors it creates, which matters when later code needs autograd.
A small model can draft the next words while a larger model checks them. Speculative decoding can shorten the wait when enough drafts survive the check.
Modal (Oct 1, 2026) made Clusters generally available via @modal.clustered for multi-node GPU jobs. RDMA is optional; Volumes, Cloud Bucket Mounts, and Queues stay in the loop. No dollar rates on the post.
EricLBuehler/mistral.rs (not Mistral AI) merged PR #2462 on 2026-10-01: Qwen3.8-Flash-Next from safetensors and GGUF with built-in MTP. Not a new release tag. Paged attention requires CUDA; FP8 KV / TP / GGUF MTP / external drafters unsupported.