Community evidence snapshot
Hermes Agent local models
Browse what community members report using, then compare the topology, model, runner, context, and caveats without turning an anecdote into a compatibility promise.
// Explore
Start with primary and general reports. Auxiliary uses remain available through the role filter. K-only context values keep their reported wording and are approximate only for numeric filtering.
Primary report
The author reports good Hermes tool handling and long-form workflows.
Major limitation Q4 is described as faster than Q6, and the source does not connect either quantization to this context setting; the remote Mac client is not inference hardware. Parallel sessions are not usable and prefill speed drops severely.
Reported 96k context with the single RTX 5090. The source names thinkingcap-qwen3.8-27b@q6_k as a configured ID and also reports Q4_K_M for daily use without pairing either quantization to this context.
Q4 is described as pretty fast; Q6 is slower.
Method: Author's everyday-use observation; no benchmark or tokens-per-second measurement.
Workload: Email sorting, tool use, browser/navigation workflows, and file handling.
Community local-model report Hardware: Windows PC • Intel Core Ultra 9 285K • 1 × NVIDIA RTX 5090 Topology: 64 GB system RAM. This is the author's single-GPU context configuration; a separate MacBook Pro is the remote Hermes client, not inference hardware. Model: ThinkingCap-Qwen3.8-27B Quantization: Q6_K; Q4_K_M is also reported for daily Hermes use, but the source does not associate either quantization with a context configuration. Runner: LM Studio on Windows Reported context: 96k Outcome: Works with caveats — The author reports good Hermes tool handling and long-form workflows. Major limitation: Q4 is described as faster than Q6, and the source does not connect either quantization to this context setting; the remote Mac client is not inference hardware. Parallel sessions are not usable and prefill speed drops severely. Source: u/Jonathan_Rivera — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pd9mygi/ (updated 2026-10-01)
Primary report · dual-GPU context configuration
The author reports good Hermes tool handling and long-form workflows when both GPUs are used.
Major limitation The author reports instability with the two-GPU setup and a BSOD about once a day; parallel sessions are not usable and prefill speed drops severely. The two VRAM capacities must remain separate rather than being treated as pooled memory.
Reported 156k context when both GPUs are used. The source explicitly describes the 5090 and 5070 Ti as separate GPUs, not one pooled device.
Q4 is described as pretty fast; Q6 is slower.
Method: Author's everyday-use observation; no benchmark or tokens-per-second measurement.
Workload: Email sorting, tool use, browser/navigation workflows, and file handling.
Community local-model report Hardware: Windows PC • Intel Core Ultra 9 285K • RTX 5090 + RTX 5070 Ti Topology: 64 GB system RAM plus 32 GB on the RTX 5090 and 16 GB on the RTX 5070 Ti. Separate GPUs are not one pooled 48 GB device. Model: ThinkingCap-Qwen3.8-27B Quantization: Q6_K; Q4_K_M is also reported for daily Hermes use, but the source does not associate either quantization with a context configuration. Runner: LM Studio on Windows Reported context: 156k Outcome: Works with caveats — The author reports good Hermes tool handling and long-form workflows when both GPUs are used. Major limitation: The author reports instability with the two-GPU setup and a BSOD about once a day; parallel sessions are not usable and prefill speed drops severely. The two VRAM capacities must remain separate rather than being treated as pooled memory. Source: u/Jonathan_Rivera — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pd9mygi/ (updated 2026-10-01)
General report
The author describes the stack as fast and smooth.
Major limitation Hardware, runner, model, and context details are a curator transcription of the linked screenshot; the comment supplies no benchmark or direct Hermes-tool result.
The curator-transcribed screenshot explicitly reports OLLAMA_CONTEXT_LENGTH=64000 in Ollama 0.32.14 on native Windows. Planned power changes and a planned second GPU are excluded from this configuration.
Fast and smooth.
Method: Author's subjective description; no benchmark supplied.
Workload: Unspecified local LLM use.
Community local-model report Hardware: Windows 10 Pro • Ryzen 9 5900X • MSI MAG X570 TOMAHAWK WIFI • ASUS TUF RTX 3090 Topology: 32 GB DDR4 system RAM. A later RTX 3080 addition was planned in a follow-up comment and is not part of this reported configuration. Model: qwen3.8:27b-q4_K_M Quantization: Q4_K_M Runner: Ollama 0.32.14 (native Windows) Reported context: 64000 Outcome: Works with caveats — The author describes the stack as fast and smooth. Major limitation: Hardware, runner, model, and context details are a curator transcription of the linked screenshot; the comment supplies no benchmark or direct Hermes-tool result. Source: u/piranhamoose2323 — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pd9wl5v/ (updated 2026-10-01)
Primary report · daily driver
The author reports good tool use and uses Qwen as the primary model for two of four profiles.
Major limitation Occasional malformed tool calls are reported as retried by the vLLM grammar layer; the source describes this host as one of three separate inference machines.
Reported 262K context with medium reasoning effort, TP2, MTP speculative decoding, and GPU memory utilization reported as 0.92.
55.4 average tokens/s at 1K/512 and 60.2 tokens/s at short/512.
Method: Timed API completions.
Workload: Qwen daily-driver use; the source does not provide a comparable benchmark protocol.
Community local-model report Hardware: Ryzen 5 9600X host • 2 × refurbished RTX 3090 Topology: 32 GB RAM and 48 GB VRAM across two separate cards. This record does not turn the cards into a single pooled-memory device. Model: lued/Qwen3.8-27B-INT8-W8A16-MTP Quantization: INT8 W8A16 (Marlin), FP8 KV Runner: vLLM in Docker with TP2 and MTP speculative decoding Reported context: 262K (full native) Outcome: Works with caveats — The author reports good tool use and uses Qwen as the primary model for two of four profiles. Major limitation: Occasional malformed tool calls are reported as retried by the vLLM grammar layer; the source describes this host as one of three separate inference machines. Source: u/ConvertibleConflict — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pdbifai/ (updated 2026-10-02)
Primary report · heavy/long-context
The author uses DSV4 as the primary model for two profiles when deeper reasoning or more than about 250K tokens is needed.
Major limitation The two-node cluster has a reported cold boot warmup of about 47 layers; its model and memory topology are separate from the Qwen host.
Reported 1,048,576-token context on a two-node cluster; the source says the weights are shared over the cluster fabric.
43.4, 47.1, and 47.6 tokens/s after a first call at 21.8.
Method: Three 256-token completions using the author's stated protocol.
Workload: Per-stream decode; not comparable with other reports using different hardware or workloads.
Community local-model report Hardware: 2-node DGX Spark cluster • Grace-Blackwell GB10 Topology: Two separate 121 GiB nodes are described; 242 GiB is their cluster aggregate, not one device or a pooled-memory guarantee. Model: deepseek-ai/DeepSeek-V4-Flash-Vision-Exp Quantization: Native weights; no quantization reported Runner: Two-node vLLM cluster (head and worker) Reported context: 1,048,576 (1M) Outcome: Works with caveats — The author uses DSV4 as the primary model for two profiles when deeper reasoning or more than about 250K tokens is needed. Major limitation: The two-node cluster has a reported cold boot warmup of about 47 layers; its model and memory topology are separate from the Qwen host. Source: u/ConvertibleConflict — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pdbifai/ (updated 2026-10-02)
Auxiliary report · Hindsight memory-system auxiliary
The author reports this as the Hindsight memory system's brain rather than a primary Hermes chat model.
Major limitation This is an auxiliary memory-system model, not a primary Hermes/tool-success report; prompts occasionally hit a hard 32K wall.
Reported 32K context on the separate utility host. This record is auxiliary and excluded from the default primary/general view.
About 115 tokens/s decode in the supplied live benchmark sequence.
Method: Three 256-token completions.
Workload: Auxiliary memory-system work; not comparable with primary chat reports.
Community local-model report Hardware: Utility host • RTX 5060 Ti 16 GB Topology: The source identifies this as a separate utility inference host, distinct from the dual-3090 and two-node cluster records. Model: ggml-org/gpt-oss-20b-GGUF Quantization: Native MXFP4 Runner: llama.cpp in Docker Reported context: 32K Outcome: Works with caveats — The author reports this as the Hindsight memory system's brain rather than a primary Hermes chat model. Major limitation: This is an auxiliary memory-system model, not a primary Hermes/tool-success report; prompts occasionally hit a hard 32K wall. Source: u/ConvertibleConflict — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pdbifai/ (updated 2026-10-02)
Primary report
The author says Qwen feels great and is the usual choice for the reported work.
Major limitation Runner and measurement method are unknown; the reported speed is an anecdote, not a comparable ranking or compatibility guarantee.
The source reports 262k context on this two-card configuration and does not provide a runner name.
Around 90 tokens/s.
Method: Author's report; measurement method not stated.
Workload: Mostly Qwen use; the source contrasts it with an Ornith alternative for fast/read-only questions.
Community local-model report Hardware: 5800xt (reported) • X570 ASUS • 2 × RTX 3090 FTW3 Topology: 48 GB VRAM total is reported across two cards with NVLink and 64 GB DDR4 system RAM; the source does not state a per-card capacity or establish one pooled device. Model: Qwen3.8-27b-Int8 Quantization: Int8 Runner: Unknown Reported context: 262k Outcome: Works with caveats — The author says Qwen feels great and is the usual choice for the reported work. Major limitation: Runner and measurement method are unknown; the reported speed is an anecdote, not a comparable ranking or compatibility guarantee. Source: u/Cold_Tree190 — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pdaa93p/ (updated 2026-10-01)
General report · read-only/fast-answer alternative
The author reports using Ornith for fast answers and read-only questions.
Major limitation This is a source-described alternative on the same two-card host; runner and measurement method are unknown.
The source reports 512k context for Ornith on the same two-card configuration.
Around 150 tokens/s.
Method: Author's report; measurement method not stated.
Workload: Fast answers and read-only questions; not comparable with other reports.
Community local-model report Hardware: 5800xt (reported) • X570 ASUS • 2 × RTX 3090 FTW3 Topology: 48 GB VRAM total is reported across two cards with NVLink and 64 GB DDR4 system RAM; the source does not state a per-card capacity or establish one pooled device. Model: Ornith 1.0 35b Quantization: Int8 Runner: Unknown Reported context: 512k Outcome: Works with caveats — The author reports using Ornith for fast answers and read-only questions. Major limitation: This is a source-described alternative on the same two-card host; runner and measurement method are unknown. Source: u/Cold_Tree190 — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pdaa93p/ (updated 2026-10-01)
Primary report
QFN is reported as the primary Hermes model and as covering about 90% of the author's desired work.
Major limitation Part of the workload is offloaded to CPU and NVMe; coding is the stated exception, and the R9700 memory total is not split into invented per-card values.
QFN is reported as primary for Hermes with CPU and NVMe offload for part of the workload. The source names VLLM for QFN and does not give a context value.
About 3k prefill, 71 tokens/s across eight concurrent streams, and 77 tokens/s average decode.
Method: Measured with better bench, as reported by the author.
Workload: QFN across eight concurrent streams; not comparable with single-stream anecdotes.
Community local-model report Hardware: Ryzen 9 3900X • X570 • 2 × R9700 + 1 × RTX 3060 Topology: The source reports 64 GB RAM, 64 GB VRAM across the two R9700 devices without a per-card split, plus a separate 12 GB RTX 3060. The devices remain mixed and separate. Model: tcclaviger/Qwen3.8-Flash-Next-MXFP4-FP8-GPTQ Quantization: QFN MXFP4 FP8 mixed quant Runner: vLLM for QFN Reported context: Unknown Outcome: Works with caveats — QFN is reported as the primary Hermes model and as covering about 90% of the author's desired work. Major limitation: Part of the workload is offloaded to CPU and NVMe; coding is the stated exception, and the R9700 memory total is not split into invented per-card values. Source: u/Ecstatic-Wash-7667 — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pda77h7/ (updated 2026-10-01)
Primary report
The author reports fast use in both directions and good multi-step Hermes tool calls; this is a source report, not a general recommendation.
Major limitation A contiguous reservation above roughly 53 GB is reported to fault high-address HBM; the author keeps GPU memory utilization at about 0.84 or below. The source reports bfloat16 as the compute/runtime dtype and FP8 E4M3 for KV, not as the weight quantization.
Reported 262,144-token context with Freaksterz/Qwen3.8-27B-SmoothQuant-W8A8-INT8 W8A8 SmoothQuant weights, --dtype bfloat16 as the compute/runtime dtype, FP8 E4M3 KV, GPU memory utilization about 0.84, automatic tool choice, Qwen XML parsing, and MTP-4. These are reported settings, not executable instructions.
About 2,000 tokens/s median uncached prefill, about 3,500 maximum, and about 56–65 tokens/s sustained decode with MTP-4.
Method: vLLM metrics and per-window logger lines over the author's last three days.
Workload: Agent turns with prefix-cache hits and short prompts; the source says these measurements are workload-dependent.
Community local-model report Hardware: NVIDIA CMP 170HX • GA100/SM80 Topology: All 64 GB is reported as VRAM; the author reports no CPU offload and no swap. Model: Qwen3.8-27B Quantization: W8A8 SmoothQuant Runner: vLLM 0.28.0 with FlashInfer Reported context: 262,144 Outcome: Works as reported — The author reports fast use in both directions and good multi-step Hermes tool calls; this is a source report, not a general recommendation. Major limitation: A contiguous reservation above roughly 53 GB is reported to fault high-address HBM; the author keeps GPU memory utilization at about 0.84 or below. The source reports bfloat16 as the compute/runtime dtype and FP8 E4M3 for KV, not as the weight quantization. Source: u/couperd — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pdatpon/ (updated 2026-10-01)
Primary report
The author reports few Hermes-tool issues and successful overnight use with repeated compaction.
Major limitation Crashes are reported when a game allocates about 1 GB of VRAM on the Intel card; the report also includes a display-only RTX 3070.
Reported -c 120000 context with SYCL-backed llama.cpp inference optimized for the B60; the source command reports Q4_K_M while the prose labels the quantization Q4 KM. The RTX 3070 is described as display-only. The executable launch command is not reproduced.
About 40–45 tokens/s at empty context, slowing to about 20–25 at 100k; prefill about 750 down to about 500.
Method: Llama-bench runs and incremental-build observations reported by the author.
Workload: Context-length observations; not comparable with other reports using different runtimes or workloads.
Community local-model report Hardware: Windows • Ryzen 5 5600X • Intel Arc Pro B60 Topology: 64 GB RAM and 24 GB VRAM are reported for the B60. The RTX 3070 is described as display-connected; the author says inference is running on the B60. Family is classified as Intel, not mixed. Model: Swift 1.5 Qwen3.8-27B Quantization: Q4 KM Runner: llama.cpp with SYCL backend Reported context: 120000 Outcome: Works with caveats — The author reports few Hermes-tool issues and successful overnight use with repeated compaction. Major limitation: Crashes are reported when a game allocates about 1 GB of VRAM on the Intel card; the report also includes a display-only RTX 3070. Source: u/Hairy_Ad_7347 — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pd9rdqg/ (updated 2026-10-01)
Primary report
The author reports amazing everyday use, fine Hermes-tool handling, and SysAdmin work on this setup.
Major limitation The performance figures are source-reported oMLX benchmark evidence for a specific workload, not a comparable ranking or compatibility guarantee.
Reported 262144 context with oMLX and oQ5e quantization on a Mac Studio / M5U / 64c with 256 GB unified memory.
The oMLX benchmark reports 2108.8 tok/s prompt processing and 141.1 tok/s generation at pp1024/tg128, with 122.24 GB peak memory; additional pp/tg measurements are reported through pp200000/tg128.
Method: oMLX single-request benchmark with Code (Python) context and the author's pp/tg test sizes.
Workload: Prompt-processing and generation tests from pp1024/tg128 through pp200000/tg128; values are workload-specific.
Community local-model report Hardware: Mac Studio / M5U / 64c Topology: 256 GB unified memory is reported for the Mac Studio / M5U / 64c. Model: Qwen3.8-Flash-Next-oQ5e-mtp Quantization: oQ5e Runner: oMLX Reported context: 262144 Outcome: Works with caveats — The author reports amazing everyday use, fine Hermes-tool handling, and SysAdmin work on this setup. Major limitation: The performance figures are source-reported oMLX benchmark evidence for a specific workload, not a comparable ranking or compatibility guarantee. Source: u/Only-An-Egg — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pd9sytw/ (updated 2026-10-01)
Primary report
The author reports no tool issues so far while continuing to tune parallelization.
Major limitation Quantization is unknown, and the author reports occasional context-compression failures while tuning two independent boxes.
Reported 64K context with parallelization settings still under tuning across two independent hosts.
Not reported.
Community local-model report Hardware: 2 × AMD Ryzen 9 (8060S video) hosts • Ubuntu Topology: Each independent host reports 128 GB unified memory with 96 GB assigned to VRAM. This is not a pooled 256 GB inference device. Model: Qwen3.8-27B Quantization: Unknown Runner: Lemonade → Hermes Agent → Hermes Desktop Reported context: 64K Outcome: Works with caveats — The author reports no tool issues so far while continuing to tune parallelization. Major limitation: Quantization is unknown, and the author reports occasional context-compression failures while tuning two independent boxes. Source: u/corio3 — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pdafhps/ (updated 2026-10-01)
General report · incomplete report
The source contains a broad positive capability description but not a reproducible Hermes setup report.
Major limitation Runner, quantization, context, and Hermes tool handling are unknown; broad performance and privacy claims from the source are not validated here and are not a compatibility or privacy guarantee.
Not reported.
Not reported.
Not reported.
Community local-model report Hardware: 128 GB Strix Halo Halogen Flash server Topology: 128 GB reported; memory type and device topology are not specified. Model: Qwen 3.8 Flash Next Quantization: Unknown Runner: Unknown Reported context: Unknown Outcome: Incomplete report — The source contains a broad positive capability description but not a reproducible Hermes setup report. Major limitation: Runner, quantization, context, and Hermes tool handling are unknown; broad performance and privacy claims from the source are not validated here and are not a compatibility or privacy guarantee. Source: u/cunasmoker69420 — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pdb7fqp/ (updated 2026-10-01)
Primary report
The author reports a fast always-on assistant that handles everyday tasks and Hermes tools.
Major limitation The reported 16,000 maximum generated-context value is separate from the 262,000 context setting; the author uses a cloud model for complex coding plans.
Reported 262,000 context, 16,000 maximum generated context, and three concurrent sessions. These are separate limits.
60.8–64.2 decode tokens/s, with short-prompt bursts up to about 105; prefill about 1,900 tokens/s.
Method: Author's reported decode, prefill, and warm/cached TTFT observations.
Workload: Short-prompt bursts, cached and uncached pack loads, and everyday assistant use; not comparable with other reports.
Community local-model report Hardware: M5 Max Mac Studio Topology: 128 GB unified memory is reported for the Mac Studio. Model: Custom Swift1.5-Qwen Flash Next build Quantization: Mixed 4–8 bit with oQ4e smart quantization; 32 GB n-grams remain unchanged Runner: mlx-serve Reported context: 262000 context with 16000 maximum generated context Outcome: Works with caveats — The author reports a fast always-on assistant that handles everyday tasks and Hermes tools. Major limitation: The reported 16,000 maximum generated-context value is separate from the 262,000 context setting; the author uses a cloud model for complex coding plans. Source: u/rokrdev — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pd9yvm4/ (updated 2026-10-01)
Primary report
The author reports everyday use for server management, coding, and personal-assistant tasks.
Major limitation The source says max context but gives no numeric setting; the custom V100 build and self-quantization are part of this report and not a general recipe.
The source reports max context and two concurrent requests but supplies no numeric context value.
40–45 tokens/s base and 100–110 tokens/s with dflash2 at 7 and 44–56% acceptance.
Method: Author's report; measurement protocol not stated.
Workload: Two concurrent requests in everyday assistant tasks; not comparable with other reports.
Community local-model report Hardware: 2 × NVIDIA V100 32 GB SXM2 Topology: 96 GiB system RAM and two separate 32 GB V100 devices are reported; no pooled-memory claim is made. Model: Qwen 3.8 27B ThinkingCap version Quantization: Self-quantized for V100; partial quantization from bfloat16 Runner: Special V100 vLLM build with dflash2 Reported context: max context Outcome: Works with caveats — The author reports everyday use for server management, coding, and personal-assistant tasks. Major limitation: The source says max context but gives no numeric setting; the custom V100 build and self-quantization are part of this report and not a general recipe. Source: u/Drakaner — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pdd0lrp/ (updated 2026-10-02)
Primary report · local agent
It's fast enough for practical everyday use. For example, it can summarize my unread emails without taking 60 minutes, which makes it genuinely usable as a local agent. It follows instructions well
Major limitation I sometimes need to be fairly specific in my prompts to get the exact behavior I want, especially when specifying which tools or skills it should use/read.
quantized to Q8_0, 1 parallel slot Hermes tools: It handles Hermes tools quite well. It can use the tools reliably, with hallucinations mainly becoming an issue in very large contexts.
Around 67 tok/s prompt processing and 10 tok/s generation , measured with a 31,000-token context and the model consistently running at high effort .
Method: measured with a 31,000-token context
Workload: the model consistently running at high effort
Community local-model report Hardware: Ryzen 5 4600G with integrated Vega 7 Topology: 16 GB DDR4, with ~9 GB available for the model Model: peculiar-ragdoll/Sharp-Spark-X2.5-4B-GGUF Quantization: Q5_K_XL Runner: llama.cpp + Quartermaster Reported context: 80,000-token context, quantized to Q8_0, 1 parallel slot Outcome: Works with caveats — It's fast enough for practical everyday use. For example, it can summarize my unread emails without taking 60 minutes, which makes it genuinely usable as a local agent. It follows instructions well Major limitation: I sometimes need to be fairly specific in my prompts to get the exact behavior I want, especially when specifying which tools or skills it should use/read. Source: u/Icy_Progress_4660 — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pdi8qfu/ (updated 2026-10-02)
Primary report · full blown chat with cloud as backup & big tasks
still testing, but clearly better than Ollama
Major limitation no offloading
MTP speculative decoding (3 draft tokens), FP8 KV cache, vision enabled Hermes tools: good, no issues so far
~127 t/s on a real Hermes turn (this one); ~145 t/s on a fresh short prompt. Both measured from streaming usage.completion_tokens — not SSE chunk count, which undercounts ~2.3x because NInfer groups tokens per chunk.
Method: Both measured from streaming usage.completion_tokens — not SSE chunk count, which undercounts ~2.3x because NInfer groups tokens per chunk.
Workload: a real Hermes turn (this one); ~145 t/s on a fresh short prompt
Community local-model report Hardware: NVIDIA RTX 5090 Topology: 32GB VRAM + 186GB system RAM Model: Qwen3.8-27B (NVFP4) Quantization: NVFP4 Runner: NInfer Reported context: 231k (231200) Outcome: Works with caveats — still testing, but clearly better than Ollama Major limitation: no offloading Source: u/bizz101 — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pdg928c/ (updated 2026-10-02)
Primary report
good - better if on mature profile (use a better cloud/hosted model initially for configuration/skill building)
Major limitation Major limitation was not reported.
allowed wired memory increased a bit to 24GB
Not reported.
None reported.
~10 tok/s
Community local-model report Hardware: M5 Topology: 32GB unified Model: Qwen3.8-27B Quantization: Q4_K_M Runner: llama.cpp Reported context: at least 64k (smooth) - 128k (tight) Outcome: Works as reported — good - better if on mature profile (use a better cloud/hosted model initially for configuration/skill building) Major limitation: Major limitation was not reported. Source: u/mfowle3 — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pdfrt93/ (updated 2026-10-02)
Primary report
Great, but just not so fast
Major limitation Major limitation was not reported.
Hermes tools: I can't complain
None reported.
Post-Processing: 2000tk/s Token Generation: 30tk/s (single session) up to 160tk/s (10 sessions). Measured wird llama-benchy
Method: llama-benchy
Workload: single session) up to 160tk/s (10 sessions
Community local-model report Hardware: Lenovo ThinkStation PGX (DGX Spark hardware: NVIDIA GB10) Topology: 128GB unified memory Model: Qwen3.8-27b Quantization: NVFP4 Runner: SGLang Reported context: 256k Outcome: Works as reported — Great, but just not so fast Major limitation: Major limitation was not reported. Source: u/kla_sch — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pdetbrl/ (updated 2026-10-02)
General report
Seems to be good, kind of just started so I don't have any context for a comparison.
Major limitation There have been a few errors that it has troubleshooted and fixed itself.
Hermes tools: Seems fine.
Qwen 3.8 = ~110 t/s Method: llama bench
Method: llama bench
Community local-model report Hardware: 1 Physical PC 9950X + 64GB System RAM + RTX 6000 Pro 96GB VRAM Topology: 96GB VRAM Model: Qwen 3.8 27B Quantization: Q8 Runner: llama.cpp Reported context: 262144 Outcome: Works with caveats — Seems to be good, kind of just started so I don't have any context for a comparison. Major limitation: There have been a few errors that it has troubleshooted and fixed itself. Source: u/invizen — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pdniw4h/ (updated 2026-10-03)
General report
Seems to be good, kind of just started so I don't have any context for a comparison.
Major limitation There have been a few errors that it has troubleshooted and fixed itself.
Hermes tools: Seems fine.
gemma 4 = ~220 t/s Method: llama bench
Method: llama bench
Community local-model report Hardware: 1 VM 9955HX 16vCPU + 64 GB Virtual System RAM + RTX 4090 24GB VRAM Topology: 24GB VRAM Model: gemma 4 26B-A4B Quantization: Q4 Runner: llama.cpp Reported context: 131072 Outcome: Works with caveats — Seems to be good, kind of just started so I don't have any context for a comparison. Major limitation: There have been a few errors that it has troubleshooted and fixed itself. Source: u/invizen — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pdniw4h/ (updated 2026-10-03)
Primary report · Primary daily driver for homelab ops
Snappy and rock solid. Having dflash speculative decoding + FP8 KV cache makes multi-turn reasoning and agentic back-and-forth feel almost instant compared to waiting on dense decodes.
Major limitation Zero CPU offloading or disk swap.
dflash speculative decoding + FP8 KV cache Hermes tools: Remarkably well. I didn't expect a lower level quant to perform this well but I'm quite surprised.
Not reported.
Community local-model report Hardware: AMD Ryzen Threadripper PRO 5955WX (16c/32t), 2× NVIDIA RTX PRO 6000 Blackwell Workstation Edition (PCIe) Topology: 128 GB DDR4 system RAM + 192 GB VRAM (2× 96 GB) Model: brandonmusic/GLM-5.3-Flash-tr3-4bpw Quantization: 4bpw EXL3 (TR3 MCG codebook on routed experts; native BF16 non-routed layers) Runner: https://github.com/tpurtell/glm-5.3-flash-ext3-2x-rtx Reported context: 1,048,576 (1M native context limit configured, typically running within 64k–256k in active sessions) Outcome: Works with caveats — Snappy and rock solid. Having dflash speculative decoding + FP8 KV cache makes multi-turn reasoning and agentic back-and-forth feel almost instant compared to waiting on dense decodes. Major limitation: Zero CPU offloading or disk swap. Source: u/anitamaxwynnn69 — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pe5cyzy/ (updated 2026-10-06)
Primary report
Does it work decently? Yes.
Major limitation Curator note: hardware identified as the Intel Core Ultra 5 125H (Meteor Lake laptop processor with integrated Arc graphics) from the reported 125H label; the author did not name the vendor or say whether inference runs on the CPU or integrated GPU.
enabling and disabling thinking on use case Hermes tools: External memory with open viking selfhosted
Not reported.
Not reported.
Community local-model report Hardware: Intel Core Ultra 5 125H Topology: Memory type and capacity were not reported. Model: qwen 3.5 9b Quantization: Unknown Runner: ollama Reported context: Unknown Outcome: Works with caveats — Does it work decently? Yes. Major limitation: Curator note: hardware identified as the Intel Core Ultra 5 125H (Meteor Lake laptop processor with integrated Arc graphics) from the reported 125H label; the author did not name the vendor or say whether inference runs on the CPU or integrated GPU. Source: u/_st_daime_ — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pdhv3la/ (updated 2026-10-02)
Primary report
not exactly usable
Major limitation Major limitation was not reported.
Not reported.
Not reported.
None reported.
it was at 1.1tok/sec so not exactly usable
Community local-model report Hardware: M4 Mac mini Topology: 24gb ram Model: flash next Quantization: Unknown Runner: Unknown Reported context: Unknown Outcome: Works as reported — not exactly usable Major limitation: Major limitation was not reported. Source: u/eatoff — https://www.reddit.com/r/hermesagent/comments/1wv8amt/what_local_model_are_you_running_with_hermes_and/pek8e9q/ (updated 2026-10-08)
Primary report
No everyday-use result was reported.
Major limitation Major limitation was not reported.
Not reported.
Not reported.
None reported.
Not reported.
Community local-model report Hardware: MacStudio Topology: Memory type and capacity were not reported. Model: GLM 5.3 Flash Quantization: 4 bit Runner: MLX Reported context: Unknown Outcome: Incomplete report — No everyday-use result was reported. Major limitation: Major limitation was not reported. Source: u/Zorogozano — https://www.reddit.com/r/hermesagent/comments/1xgss06/what_local_model_are_you_running_with_hermes_and/19rqicporpq1v/ (updated 2026-10-27)
Primary report · local coding
Averaging ~40t/s under real-world usage with oMLX.
Major limitation I only have room to load one large model at a time
Not reported.
Averaging ~40t/s under real-world usage with oMLX.
Workload: real-world usage
Community local-model report Hardware: M3 Ultra Topology: Memory type and capacity were not reported. Model: GLM-5.3-Flash Quantization: Unknown Runner: oMLX Reported context: Unknown Outcome: Works with caveats — Averaging ~40t/s under real-world usage with oMLX. Major limitation: I only have room to load one large model at a time Source: u/CatchDublinSurprise — https://www.reddit.com/r/hermesagent/comments/1xgss06/what_local_model_are_you_running_with_hermes_and/19rqio3je7482/ (updated 2026-10-27)
Primary report
No everyday-use result was reported.
Major limitation Major limitation was not reported.
Not reported.
Not reported.
None reported.
Not reported.
Community local-model report Hardware: RTX 5090 Topology: Memory type and capacity were not reported. Model: Qwen 3.8 27B Quantization: Unknown Runner: Unknown Reported context: Unknown Outcome: Incomplete report — No everyday-use result was reported. Major limitation: Major limitation was not reported. Source: u/Mean-Loquat-7982 — https://www.reddit.com/r/hermesagent/comments/1xgss06/what_local_model_are_you_running_with_hermes_and/19rqibxy8sy01/ (updated 2026-10-27)
General report
No everyday-use result was reported.
Major limitation Major limitation was not reported.
Not reported.
Not reported.
None reported.
Not reported.
Community local-model report Hardware: 3080-10GB Topology: 10GB Model: Ornith-1.5-35B-A3B Quantization: Q4_K_M Runner: Unknown Reported context: Unknown Outcome: Incomplete report — No everyday-use result was reported. Major limitation: Major limitation was not reported. Source: u/XCherryCokeO — https://www.reddit.com/r/hermesagent/comments/1xgss06/what_local_model_are_you_running_with_hermes_and/19rqhqy7o8gtd/ (updated 2026-10-27)
Primary report
No everyday-use result was reported.
Major limitation Major limitation was not reported.
Not reported.
Not reported.
None reported.
Not reported.
Community local-model report Hardware: Mac Studio M4 Max 36GB Topology: 36GB Model: Gemma 4 26B A4B QAT Quantization: QAT Runner: Unknown Reported context: Unknown Outcome: Incomplete report — No everyday-use result was reported. Major limitation: Major limitation was not reported. Source: u/Objective-Towel5542 — https://www.reddit.com/r/hermesagent/comments/1xgss06/what_local_model_are_you_running_with_hermes_and/19rqgqi5zr8kj/ (updated 2026-10-27)
Primary report · my best daily assistant
GOOD, my buddy is my best daily assistant.
Major limitation Major limitation was not reported.
750k+ kv cache pool for 3x 256k agent running at same time or 10 agent queue running. Hermes tools: GOOD.
None reported.
(n=1) code ~224 / real-job ~158 / mixed ~86 / prose ~66 / numeric ~272 tok/s (avg ~144); (n=8) ~320 tok/s per req; prefill ~2,765 (1k) / ~2,307 (64k) tok/s; TTFT p99 0.22s (n=1) / 3.97s (n=8); DFlash2-W8 K=7, avg acceptance 0.324. Measured with custom bench_suite.py sendding chat req to vllm.
Method: Measured with custom bench_suite.py sendding chat req to vllm.
Workload: code ~224 / real-job ~158 / mixed ~86 / prose ~66 / numeric ~272 tok/s
Community local-model report Hardware: 3090 24gb x 4 Topology: 96GB VRAM Model: Qwen3.8 27b Quantization: int8 with fp8 kv Runner: patched vLLM 0.28.1 + LMcache Reported context: 750k pool. Outcome: Works as reported — GOOD, my buddy is my best daily assistant. Major limitation: Major limitation was not reported. Source: u/Old_Ad_6033 — https://www.reddit.com/r/hermesagent/comments/1xgss06/what_local_model_are_you_running_with_hermes_and/19rqjrj7xnyg4/ (updated 2026-10-27)
Primary report
Very clever and useable
Major limitation Major limitation was not reported.
Hermes tools: No issue, never had a problem
None reported.
85t/s between 4k and 256k ctx, very stable
Workload: between 4k and 256k ctx
Community local-model report Hardware: Intel 10940x + X299 + 2x RTX3090 Topology: 192GB Quad channel DDR4 3600 + 48Gb VRAM Model: Huihui-Qwen3.8-Flash-Next-abliterated-v2-GGUF Quantization: Q8 Runner: Strata 0.2.1 Reported context: 512k Outcome: Works as reported — Very clever and useable Major limitation: Major limitation was not reported. Source: u/GaelOffMySoul — https://www.reddit.com/r/hermesagent/comments/1xgss06/what_local_model_are_you_running_with_hermes_and/19rqjc79a592a/ (updated 2026-10-27)
Primary report · Hermes agent's brain 24/7
This box is my Hermes agent's brain 24/7, so it lives under continuous agent load, not benchmark pings.
Major limitation No crashes on the current config.
262,144 tokens, cache_size 262k, MTP drafting from the main model + hybrid drafting (16 tokens drafted), recurrent checkpoint interval 2048, plus a 50 GB disk cache tier that persists KV pages and checkpoints across restarts. Hermes tools: Tool calling is solid, including multi-step chains, and draft acceptance holds at 64% across all real agent traffic (1.2M drafted tokens).
Decode median ~180 tok/s (middle 50%: 151–202). Prefill median ~2,100 T/s cold, ~3,450 T/s on cache hits. Draft acceptance 64%. Extracted from ExLlamaV3's own request logs over ~15 h of continuous real agent traffic (1,464 generations of 100+ tokens), no benchmark harness. Workload: a cold 40k-token prefill is ~20 s, and warm requests after cache hits are near-instant.
Method: Extracted from ExLlamaV3's own request logs over ~15 h of continuous real agent traffic (1,464 generations of 100+ tokens), no benchmark harness.
Workload: a cold 40k-token prefill is ~20 s, and warm requests after cache hits are near-instant.
Community local-model report Hardware: AMD Ryzen 9 9950X (16c), 2× RTX 5090 (32 GB each) Topology: 64 GB VRAM Model: Qwen3.8-Flash-Next (hybrid/GDN MoE) Quantization: EXL3, ~3.05 bits per weight on average, 87 GB on disk Runner: TabbyAPI + ExLlamaV3 1.6.5 Reported context: 262,144 tokens, cache_size 262k Outcome: Works with caveats — This box is my Hermes agent's brain 24/7, so it lives under continuous agent load, not benchmark pings. Major limitation: No crashes on the current config. Source: u/_Scorpoon_ — https://www.reddit.com/r/hermesagent/comments/1xgss06/what_local_model_are_you_running_with_hermes_and/19rrgwbc4r602/ (updated 2026-10-28)
Primary report
160t/s, quick as hell
Major limitation Engram offloading is off (table in VRAM)
Hermes tools: Great, of course.
~160-181t/s, measure at the vLLM health/metric api.
Method: vLLM health/metric api
Community local-model report Hardware: 2x RTX 6000 Pros Topology: 192GB VRAM + 512GB 6400 DDR5 Model: qwen3.8-flash-next-nvfp4 Quantization: NVFP4 Runner: vLLM Reported context: 524k Outcome: Works with caveats — 160t/s, quick as hell Major limitation: Engram offloading is off (table in VRAM) Source: u/drake90001 — https://www.reddit.com/r/hermesagent/comments/1xgss06/what_local_model_are_you_running_with_hermes_and/19rr4uilngf1g/ (updated 2026-10-27)
Primary report
Runs near instant with all my Hermes sessions.
Major limitation Major limitation was not reported.
200GB session cache, cache the last 50 sessions, @95% cache hit. Hermes tools: Hermes
None reported.
Getting over 110 tk/s prose and 1300 tk/s on prefill, even at long context.
Community local-model report Hardware: M3 Ultra Mac Studio Topology: Memory type and capacity were not reported. Model: Qwen3.8-Next-Flash Quantization: Q4/Q8 mixed Runner: MLX-Serve engine Reported context: even at long context Outcome: Works as reported — Runs near instant with all my Hermes sessions. Major limitation: Major limitation was not reported. Source: u/toomanypubes — https://www.reddit.com/r/hermesagent/comments/1xgss06/what_local_model_are_you_running_with_hermes_and/19rqtdx9711yc/ (updated 2026-10-27)
Primary report · qwen4exp
I don't wait on it. Tool-call turns come back instantly; long agentic sessions with 100k+ token contexts run all day.
Major limitation expert tiering to host RAM is the design, not a fallback
KV cache at `k6_v4`. Hermes tools: 28/28 single-call + 3/3 multi-turn on my internal tool suite
BetterBench harness (weighted decode across task categories, spec-decode on): 173–266 t/s engine-level under bench concurrency; prefill ~6,678 t/s at 64k.
Method: BetterBench harness (weighted decode across task categories, spec-decode on)
Workload: weighted decode across task categories
Community local-model report Hardware: 2× Radeon AI PRO R9700 (gfx1201, RDNA4, 32 GB each) on an i5-14600K / Z790 board Topology: 2×32 GB VRAM Model: Qwen3.8-Flash-Next MoE Quantization: fp8 trunk + iq4r experts, 16 protected experts, Hessian-tuned recipe Runner: Radiance 1.12.0 Reported context: 262,144 max-model-len, 8 concurrent seqs, KV ceiling ~8 full-length sessions Outcome: Works with caveats — I don't wait on it. Tool-call turns come back instantly; long agentic sessions with 100k+ token contexts run all day. Major limitation: expert tiering to host RAM is the design, not a fallback Source: u/dschmitz1987 — https://www.reddit.com/r/hermesagent/comments/1xgss06/what_local_model_are_you_running_with_hermes_and/19rxj4ojxp0dc/ (updated 2026-10-31)
Auxiliary report · failover
If it dies, the gateway fails over to Qwen3.6-35B-A3B on a 16 GB card in ~1s.
Major limitation Major limitation was not reported.
Not reported.
Not reported.
None reported.
Not reported.
Community local-model report Hardware: ember RX 9070 Topology: 16 GB card Model: Qwen3.6-35B-A3B Quantization: Unknown Runner: Unknown Reported context: Unknown Outcome: Works as reported — If it dies, the gateway fails over to Qwen3.6-35B-A3B on a 16 GB card in ~1s. Major limitation: Major limitation was not reported. Source: u/dschmitz1987 — https://www.reddit.com/r/hermesagent/comments/1xgss06/what_local_model_are_you_running_with_hermes_and/19rxj4ojxp0dc/ (updated 2026-10-31)
Primary report
Pretty snappy I guess. Simple questions are answered instantly.
Major limitation None so far.
Context setting: 128k Hermes tools: I'd say at least as good as Jundot/Qwen3.6-35B-A3B-oQ4e-mtp
Around 30 t/s
Community local-model report Hardware: Mac mini M6 Topology: 32 GB Model: Ornith-1.5-35B-A3B-MLX-4bit Quantization: 4bit Runner: oMLX v0.7.3rc1 Reported context: 128k Outcome: Works with caveats — Pretty snappy I guess. Simple questions are answered instantly. Major limitation: None so far. Source: u/Successful_Ad8602 — https://www.reddit.com/r/hermesagent/comments/1xgss06/what_local_model_are_you_running_with_hermes_and/19s37jr6rw9ia/ (updated 2026-11-03)
Auxiliary report · for vision
This trio of models work well together so far.
Major limitation Major limitation was not reported.
Not reported.
None reported.
Not reported.
Community local-model report Hardware: Mac mini M6 Topology: 32 GB Model: mlx-community/gemma-4-e4b-it-4bit Quantization: 4bit Runner: oMLX v0.7.3rc1 Reported context: Unknown Outcome: Works as reported — This trio of models work well together so far. Major limitation: Major limitation was not reported. Source: u/Successful_Ad8602 — https://www.reddit.com/r/hermesagent/comments/1xgss06/what_local_model_are_you_running_with_hermes_and/19s37jr6rw9ia/ (updated 2026-11-03)
Auxiliary report · for STT
This trio of models work well together so far.
Major limitation Major limitation was not reported.
Not reported.
None reported.
Not reported.
Community local-model report Hardware: Mac mini M6 Topology: 32 GB Model: mlx-community/whisper-large-v3-turbo-asr-4bit Quantization: 4bit Runner: oMLX v0.7.3rc1 Reported context: Unknown Outcome: Works as reported — This trio of models work well together so far. Major limitation: Major limitation was not reported. Source: u/Successful_Ad8602 — https://www.reddit.com/r/hermesagent/comments/1xgss06/what_local_model_are_you_running_with_hermes_and/19s37jr6rw9ia/ (updated 2026-11-03)
No matching report is not evidence of incompatibility. Reset the filters or ask the community through the original thread.
// Add evidence
Use the original Reddit thread to contribute. The template is only a blank plain-text prompt; this page never submits automatically and does not fetch Reddit at runtime.
// Method
Reports are paraphrased from one public thread snapshot and retain the source author, permalink, and source-updated date. Hardware topology stays separate: system RAM, VRAM, unified memory, hosts, and display-only devices are not silently pooled.
Performance anecdotes are method- and workload-dependent and not comparable rankings; each report retains its stated method and workload. Unknowns stay unknown, and approximate K values are not silently converted into exact binary token counts.
