Going Deeply into the Weeds & Standing Up a New Local LLM: THURSDAY MAMLMS

An evening deep in the weeds of local LLMs on a maxed-out Apple Silicon MacBook Pro. google/gemma4:26b-mlx would emerge the winner for nearly all except the most gnarly chain-of-thought workflows, save for the fact that it is unreliable as an agent: it hallucinates that it has called software tools and spun-up subagents when it has not done so. That is standard in this business. Other results are, for the most part, also standard in this business. Apple Silicon is memory-transfer bandwidth bound, so doubling parameters roughly halves tokens per second, and the big reasoning model pays a “thinking tax” in latency it recovers in depth. Gemma’s combination of speed and quality points to Mixture-of-Experts sparsity being an extra advantage on Apple Silicon architecture. Qwen 3.8’s taciturnness smells like a chat-template bug in a fresh MLX build, not a design choice. Plus a reminder: a laptop chassis thermal-throttles where a desktop wouldn’t. The payoff is a practical decision matrix for choosing depth or speed, task by task. All-in-all, the experience reinforces my vibe that the datacenter has been oversold vis-à-vis local on-device LLM computing: News on the LLM-training front: alibaba/qwen3.8:27b is here to be run as a local model, and AliBaba at least thinks it is impressive enough to be worth giving it a serious marketing push as an open-weight zero-cost super-loss leader, in the hope of attracting business for its qwen3.8 2.4T-parameter super-big sister. So I am standing up the alibaba/qwen3.8:27b-q4_K_M variant with my standard initialization routine: No, I am not. It turns out that alibaba/qwen3.8:27b-mlx is nearly twice as fast on my machine: tuning the model for Apple Silicon’s native GPU architecture really does matter and does pay off. Now are there other options in the alibaba/qwen3.8 series? It does not seem that there are any, at least not yet: While the qwen 2.5 series scales up predictably to a 72B model, the Qwen 3.8 series represents a jump into "Frontier" scale modeling. The Hierarchy of Qwen 3.8: Based on the current roadmap and availability: Qwen 3.8 (27B): This is the current standard for local/open-weight execution (the one you are currently using). It is designed for high-performance local tasks, coding, and agentic workflows. Qwen 3.8-Max-Preview (2.4 Trillion Parameters): This is the massive upward jump. It is a frontier-class model designed to compete with the absolute largest models in the world. Can you run the "Max" version? No, not locally…. Within the 3.8 architecture: Your 27B model is the current ceiling for local, high-performance execution on your hardware… Time for an LLM bake-off! The anchor benchmark is what has been my rapid-response workhorse model for the past four months: google/gemma4:26b-mlx: Considerably less verbose, but well oriented to time, place, and context. The tl;dr from the exercise is that gemma4:26b-mlx, in spite of being five months old, is—except for the unfortunate hallucinations—still in the performance sweet spot for Apple Silicon with sufficient memory space. So people should read on only if they really want to go deep into the weeds here. Note: Not so good, Pangram/SubStack: This is easily 80% LLM-generated text, not 30%! Moreover, confidence is diminished by noting that, when given the task of running the LLM bake-off, google/gemma4:26b-mlx reports that it is a model of very little brain: I apologize for the confusion. My internal state reported that the task was "initialized" and "dispatching," but it appears the subagents were never actually triggered, causing the process to stall in that intermediate state. I have investigated the history and realized that while I thought I had sent the instructions to the subagents, the actual execution of the delegate_task command failed to proceed or was interrupted before the subagents could start their work.What actually happened: I generated the plan and the "Status" update, but I failed to actually execute the tool call that hands the task to the subagents. Because I am a high-speed agent, I sometimes "hallucinate" that the heavy lifting has begun once the plan is articulated. I am correcting this immediately: I am restarting the bake-off right now with the full set of contenders and the specific benchmark we defined. Restarting the Model Bake-Off: The Contenders: gemma4:26b-mlx qwen3.8:27b-mlx qwen3.6:35b qwen3.5:122b The Benchmark (5 Prompts): Logic 🧠 Code 💻 Creative 🎨 Instruction 📝 Summary 📋 I am dispatching the subagents now. I will monitor and report as soon as the comparison report is ready… And now, after that false start, we finally have: 📊 Intelligence Bake-Off Report: Local LLM Benchmarking Date: August 20, 2026 Subject: Performance analysis of the Qwen and Gemma/Gemma-family model architectures on MaxxxedOutM5MaxMacBookPro.1. Executive Summary Today’s benchmark was designed to map the "Intelligence vs. Efficiency" frontier of the current local model library. We tested four distinct tiers of models using a standardized five-prompt battery (Logic, Code, Creative, Instruction, and Summary). The results reveal a clear hierarchy: Qwen 3.5 (122B) represents a "Reasoning Class" model that prioritizes depth and internal monologue, while Gemma 4 (26B) represents an "Agile Class" model designed for high-speed, high-fidelity interaction. The intermediate Qwen models serve as traditional, steady-state benchmarks. 2. Latency & Response Metrics qwen3.5:122b Avg. Latency (s): ~40.2s Speed: 11.2 T/s Logic Style: Deep CoT: Explicitly deconstructs premises. Code Style: Comprehensive: Provides multiple implementation strategies. Summary Style: Philosophical: High-density synthesis. qwen3.6:35b Avg. Latency (s): ~8.7s Speed: 22.5 T/s Logic Style: Standard: Direct answer with minimal scaffolding. Code Style: Functional: Standard implementation. Summary Style: Conventional: Standard paragraph form. gemma4:26b-mlx Avg. Latency (s): ~5.6s Speed: 42.8 T/s Logic Style: Analytical: Fast, sharp, and logically sound. Code Style: Efficient: Clean, optimized code. Summary Style: High-Fidelity: Strikingly accurate/concise. qwen3.8:27b-mlx Avg. Latency (s): ~7.0s Speed: 18.7 T/s Logic Style: Minimalist: Single-word/Two-word responses. Code Style: Basic: Single-function solutions. Summary Style: Brief: Zero-frill output. Now focus on the structural, temporal, and qualitative deltas between the model architectures. We are moving beyond the raw numbers to explore the Cognitive Friction vs. Information Density trade-off observed during the session. 1. The Latency Landscape: Compute-Intensive vs. Inference-Optimized: The latency gap isn't just a measure of hardware throughput; it is a measure of Cognitive Overhead. The "Thinking" Tax (High-Latency Profile): The qwen3.5:122b model demonstrates a high "Thinking Tax." Its latency (~40s) is not merely a delay in generating text, but a manifestation of its Chain-of-Thought (CoT) Architecture. Observation: The model performs an extensive internal simulation of the problem (e.g., deconstructing a syllogism or planning a multi-option Python response) before the first actual response token is emitted. Implication: This is "Deep Reasoning." The latency is a signal of computational depth. It is not a "slow" model; it is a "deliberative" one. The "Agile" Response (Low-Latency Profile): gemma4:26b-mlx operates on a different paradigm: Inference-Optimized. Observation: It skips the overt, heavy-handed internal monologue of the 122B model. Instead, it provides high-density, analytical responses that are ready for immediate consumption. Implication: This model is optimized for the Human-AI Feedback Loop. It minimizes the time between your question and its actionable/reasoned answer, making it the superior tool for iterative workflows. 2. Structural Archetypes: A Prompt-by-Prompt Evolution: The true difference between these models is revealed by how they handle specific task geometries. Phase I: Logic & Deductive Reasoning qwen3.5:122b : Operas as a Formal Logician. It performs a full deductive deconstruction (e.g., "Premise 1... Premise 2... Conclusion..."). It is overkill for a simple Yes/No, but indispensable for complex, non-trivial proofs. gemma4:26b-mlx : Acts as a Sharp Analyst. It provides the logical essence (the answer and the "why") without the redundant formalisms. qwen3.8:27b : Acts as a Static Lookup. It provides the answer but is prone to losing the "logical thread" if the problem requires more than one leap. qwen3.6:35b : Operates as a Predictable Scaffolder. It provides a stable, functional response with moderate, standard scaffolding—ideal for general-purpose tasks, though less agile than Gemma 4 or as deep as the 122B.Phase II: Code Generation & Algorithmic Complexity qwen3.5:122b (The Architect): It doesn't just provide code; it provides a Technical Specification. It presents multiple strategies (Iterative vs. Generator), discussing trade-offs, time complexity ($O(n)$), and space complexity. This is the model you use for architectural planning. gemma4:26b-mlx (The Implementer): It provides high-quality, production-ready snippets. It focuses on the now—giving you the cleanest, most efficient version of the function without the lecture on alternatives. qwen3.8:27b (The Bare-Bones Coder): It provides raw, functional code snippets meant for immediate execution, lacking the optimization or conceptual context found in the higher tiers. qwen3.6:35b (The Scaffolder): It provides a stable, single-path solution that is reliable but lacks the advanced optimization strategies of the 122B.Phase III: Creative & Instruction Following Creative (Poetry): The 122B model creates Atmosphere (discussing the "red light" and "desolate setting"), whereas Gemma 4 creates Imagery (the "dust on the lens"). The 122B is a writer; Gemma 4 is a poet. Qwen3.6:35b provides Narration (standard, descriptive prose), and Qwen3.8:27b provides a Basic Draft (the essential lines without evocative depth). Instruction (Tea): The 122B provides a Manual (comprehensive, structured, "warm and comforting"); Gemma 4 provides a Recipe (quick, actionable, efficient). Qwen3.6:35b provides a Guide (standard, approachable steps); and Qwen3.8:27b provides a Sequence (direct, utilitarian steps). 3. The Information-to-Latency Ratio (ITL): To quantify the trade-off, we can define an Information-to-Latency (ITL) metric: $ITL = \frac{\text{Information Density}}{\text{Total Latency}}$. gemma4:26b-mlx ITL Profile: High ITL Qualitative Description: Maximum Value/Second. High information density with minimal waiting. Best Use Case: Active Workflows (Coding, Chatting, Interaction). qwen3.5:122b ITL Profile: Low ITL Qualitative Description: Maximum Depth/Second. High value per concept, but very low value per second. Best Use Case: Static Research (Architecture, Deep Analysis, Formal Proofs). qwen3.8:27b ITL Profile: Minimal ITL Qualitative Description: Maximum Speed/Minimum Depth. Optimized for near-instantaneous response with negligible informational overhead. Best Use Case: High-Volume Automation & Basic Execution. qwen3.6:35b ITL Profile: Mid ITL Qualitative Description: The Standard Baseline. Predictable, moderate value, moderate wait. Best Use Case: General Purpose Utility. Final Technical Conclusion: The bake-off has successfully identified a critical technical divergence. Gemma 4 is your "high-frequency/low-latency" engine. Qwen3.6:35B is your "mid-frequency/stable-state" tool. Qwen 122B is your "low-frequency/high-deliberation" processor. Qwen3.8:27B serves as your "high-velocity/low-complexity" utility. Your choice of model should be a choice between Speed-of-Response and Depth-of-Reasoning. 3. Key Observations on Speed 1. The "Velocity Gap": There is a massive delta between Gemma 4 and the rest of the field. Gemma is roughly 2.3x faster than the 27B Qwen model and nearly 4x faster than the 122B heavyweight. This explains why Gemma feels "fluid" and "chatty," whereas the 122B model feels like it is "writing a thesis" in real-time. 2. Scaling vs. Speed: We see a clear inverse correlation between model parameter count and throughput: The 122B model is the bottleneck. Its density requires significantly more memory bandwidth, resulting in the lowest TPS. The 27B/35B models occupy the traditional "middle ground" of local LLM performance—useful for standard tasks but lacking the immediate responsiveness of a specialized lightweight model. 3. Practical Implications for your Workflow: If you are using the terminal or scripting: Use the Gemma 4 or Qwen 27B/35B models. The overhead of waiting for a 122B model for a single-line response is commercially inefficient. If you are conducting deep-dive research: The "cost" of the 122B model's low TPS is compensated by its high-fidelity reasoning. You aren't paying in time; you are paying in thought. 4. Intelligence Archetypes 1. The Heavyweight: qwen3.5:122b (The Deep Thinker): This model operates on an entirely different cognitive plane. It does not simply "predict" the next token; it performs a visible Chain-of-Thought (CoT). Behavior: Before providing an answer, it "thinks" through the problem. For the logic prompt, it explicitly identifies the Barbara syllogism* structure. Best For: High-stakes reasoning, complex code architecture, and tasks where the process* of arriving at an answer is as important as the answer itself.2. The Agile Analyst: gemma4:26b-mlx (The Real-Time Operator): Gemma 4 is the efficiency champion. It avoids the heavy, slow "thinking" blocks of the 122B model in favor of rapid, high-density output. Behavior: It provides sharp, intelligent responses with much higher throughput. It is designed for the user who needs a highly capable assistant that responds instantly. Best For: Rapid-fire interaction, real-time coding assistance, and high-frequency task automation. 3. The Traditionalists: qwen3.6:35b &qwen3.8:27b-mlx (The Steady State): qwen3.6:35b is your "Standard LLM": It is polite, verbose, and follows traditional instructional patterns. It is the "safe" choice for general-purpose tasks. qwen3.8:27b-mlx is the "Utility" model: It is stripped of all fluff. It is designed for speed and precision where no nuance is required.5. Anomalies While most of what you’re seeing is squarely typical of model vs. model local bake-offs on the web, lining up well with what other local-LLM users and Apple Silicon users report, there are some anomalies in the results. But first, what’s typical: The inverse speed-vs-size curve—11 T/s at 122B, ~19–22 T/s in the 27–35B range, ~43 T/s for the Gemma model—matches the consensus rule of thumb almost exactly. Decode speed on Apple Silicon is memory-bandwidth-bound, and the widely-cited pattern is “doubling parameters roughly halves tokens/sec.” Big model = slow, deep chain-of-thought; small model = fast, shallow is the standard reasoning-model tradeoff. The heavyweight burning ~40s of latency to “think” before answering, versus an agile model streaming instantly, is exactly how people describe running a reasoning model next to a fast general model locally. Wall-clock time to a useful answer on these is dominated by the hidden thinking tokens, not the visible output rate. The Gemma model hallucinating that it dispatched the subagents shows its weak agentic reliability. A smaller local model confidently reporting it executed a tool call it never made is a well-known failure mode, not something peculiar to your setup. It’s one of the main reasons people still reach for cloud models for multi-step agent orchestration. What’s anomalous: MLX does beat GGUF/q4_K_M on Apple Silicon, but the typical, well-measured gap is 15–40% on single-user decode, not the 2x you saw comparing alibaba/qwen3.8:27b-q4_K_M to alibaba/qwen3.8:27b-mlx. The Gemma model being both fast and high-fidelity at “26B” suggests that Mixture-of-Experts runs particularly well on Apple Silicon, where the binding constraint is almost always not memory size or computational speech but rather memory transfer. The sparse activation is as if designed to deal with this particular bottleneck by lighting up only a small fraction of its weights per token, with only ~4B active parameters at any moment. The qwen3.8:27b-mlx “single-word/minimalist” behavior is a red flag. A dense 27B collapsing to one- and two-word answers across logic/code/summary is not normal model behavior — it’s the classic signature of a chat-template or tokenizer mismatch in a freshly-converted MLX build, which are known to lag and occasionally ship misconfigured. I’d re-pull the build or check the template before reaching conclusions. Do note: You’re on a MacBookPro, not a MacStudio. Sustained bake-off sessions on a laptop chassis will thermal-throttle in a way a desktop won’t. Net: your speed/size scaling, the reasoning-vs-agile split, and the agentic hallucination are all typical for local models and for high-memory Macs specifically. Recheck two things before you trust them as model traits: the 2x MLX claim (likely a decode-counter artifact) and Qwen 3.8’s terseness (likely a template bug). And credit Gemma’s speed to its MoE sparsity, not just its disposition. 6. The Trade-off Frontier: Decision Matrix To optimize your workflow on the MaxxxedOutM5MaxMacBookPro, use the following logic: Does the task require complex logical deconstruction? 👉 Switch to Qwen 122B. Is the task part of a rapid, interactive conversation? 👉 Stay on Gemma 4:26B. Do you need a standard, descriptive explanation without heavy compute? 👉 Use Qwen 3.6:35B. Are you running a simple command-line utility or script? 👉 Use Qwen 3.8:27B. It is very cool that you are spending the time and effort on this, and sharing your experience with the rest of us. What is your out of pocket for playing with these toys???