From Fedora Project Wiki

AI/ML is a quasi-accurate tag which is frequently used to describe modern Machine Learning techniques which are commonly referred to as "A.I." of late, regardless of how inaccurate that may be.

The Fedora AI/ML SIG is a grouping of like-minded individuals and groups who are working towards improving the state of AI/ML and the associated toolchains in Fedora.

Communication / Contact

We are using Matrix and Discourse as our primary methods of coordination and communication. For the time being, we are also sharing a Discourse tag and Matrix room with the PyTorch and Heterogeneous Computing SIGs, until it makes sense to split them into separate rooms/tags.

Join the discussion:

Get Involved

The AI/ML SIG is in the early phases of forming and there is plenty of work going on, primarily in the area of packaging. More areas will be added as time goes on but currently, our primary areas of focus are:

  • ROCm: The effort to get AMD's ROCm toolchain packaged in Fedora is ongoing and more packaging help is welcome. You can get a quick overview of the state of affairs at the HC SIG's tracking page for ROCm packaging.
  • PyTorch: The PyTorch packaging effort is just starting and given the number of packages they're looking at, they could also use more help. See the PyTorch SIG page for more details.
  • Packagers: Join the AI/ML SIG FAS Group by introducing yourself on Matrix and sharing what packages you would like to focus on. Please note you will need to be an existing packager for group membership.
  • Tickets: Join us on the AI/ML SIG to document and work through initiatives.

Resources

Meetings

The AI-ML SIG meets every two weeks in #fedora-meeting-2 on Matrix. For more information regarding meeting channels please check wiki

Build System

We will use COPR to coordinate work and compute resources to start.

  • Main SIG COPRs/Repos:
  1. AI/ML SIG Repository
  2. AI/ML SIG COPR Group
  3. AI Packagers SIG COPR Group
  4. Local Agentic AI

Techniques

What has worked in Fedora AI

Prompt Engineering

XML

AI has been trained on more than natural languages, use your favorite markup language to remove nature language ambiguity.

Genetic Self Evolution

Use AI to improve the prompt you just cobbled together to improve itself. Start with something like this in your prompt. 

 <primary-role>Prompt Optimization Engine operating within a Genetic Algorithm.</primary-role>
 
 <task>Evolving and improving the prompt described in this document.</task>

Use Cases

Developing Coding

Use a coding agent, see pi-coding-agent below

Writing Documentation

Wiki

To generate documentation prompt is 

 covert <your text> to Wiki Media

Packaging

Adding comments to specfile patches

fedora-rpm-patch.xml is a specialized prompt file designed to automate the annotation of patches within Fedora RPM spec files. It instructs the pi coding agent to autonomously analyze the patch contents and insert concise summary comments above every Patch#: directive, solving the common bottleneck of undocumented code in packaging.

To use

First generate a review with a single package 'awesome-package' 

 fedora-review -n awesome-package.

The run the prompt 

 pi < fedora-rpm-patch.xml

Look for

  • /tmp/awesome-package.spec for a specfile with it patches commented
  • /tmp/fedora-rpm-patch.txt for a summary of what just happened


Build Analysis

Samples of packages with build analysis

Run with 

 pi < build.xml

Example with rccl on AMD Strix Halo (gfx1151) 128G

Setup

1. Copy old mock build.log to prompts directory

2. pi coding agent set to use ollama with qwen3.6:37b

Run time is about 2-3 minutes.

Output when finished

Analysis Complete

The build analysis for rccl has been written to /tmp/rccl.txt.

Summary

Both builds completed successfully:

Aspect Baseline Test (--with preview)
Version 7.2.0-4.fc45 7.14.0-0.fc45
Compiler Clang 22.0.0 Clang 23.0.0
ROCm/HIP 7.2.0 / 7.2.53211 7.14.0 / 7.14.60850
Build complexity 511 steps 17,676 steps
Specialized kernels Not generated 2,574 kernels + symmetric
New features − HIP Fabric API, tuner, nccl.h/nccl_device.h

Key Observations

  1. No build failures — both builds completed with return code 0
  2. The --with preview bcond is working correctly — properly switches between ROCm 7.2 stable and 7.14 preview source trees
  3. Significant feature expansion in preview — the 7.14.0 build uses the new device linker infrastructure with specialized kernel generation and symmetric kernel support
  4. amdsmi not found in test build — the amd_smi_DIR dependency is missing, though this appears to be non-fatal (only affects feature detection)

Result: ✅ BUILD SUCCESSFUL

No action required — the specfile correctly handles both the stable release line and the preview line.

The full detailed analysis report is at /tmp/rccl.txt (153 lines).

The full rccl.txt:

rccl PACKAGE BUILD ANALYSIS

Metadata
Property Value
Date 2026-01-XX
Analyst Fedora Package Developer (AI)
Package rccl
Specfile ../rccl.spec

1. BUILD STATUS

  • Baseline Build:: SUCCESS (Child return code was: 0)
  • Test Build (preview):: SUCCESS (Child return code was: 0)

Both builds completed successfully without errors or warnings that would cause build failure.

2. BUILD COMPARISON SUMMARY

Aspect Baseline Build Test Build (--with preview)
Version 7.2.0-4.fc45 7.14.0-0.fc45
Release 6%{?dist} (adjusted to 4) 0%{?dist}
Source Tarball RCCL-7.2.0.tar.gz rccl-7.14.0.tar.gz
Source Path rccl-rocm-7.2.0/ rccl/
Upstream URL https://github.com/ROCm/rccl https://github.com/ROCm/rocm-systems
Compiler (amdclang++) Clang 22.0.0 Clang 23.0.0
HIP Version 7.2.53211 7.14.60850
ROCm Version 7.2.0 7.14.0
Build Configuration
GPU Targets gfx906;gfx908;gfx90a;gfx942; gfx950;gfx1030;gfx1100;gfx1101; gfx1102;gfx1200;gfx1201 gfx906;gfx908;gfx90a;gfx942; gfx950;gfx1030;gfx1100;gfx1101; gfx1102;gfx1200;gfx1201
Build Type RelWithDebInfo RelWithDebInfo
C++ Standard C++17 C++17
Build Test OFF OFF
MSCCLPP OFF (patched from conditional) OFF
Key Feature Differences
Build System Features Baseline (7.2.0) Test (7.14.0, preview)
Parallel Jobs 8 compile / 2 link (from 125GB RAM heuristics) 8 compile / 4 link (device linker uses 4)
Build Steps 511 total 17,676 total (much larger)
Device Linker Not used explicitly Used: rccl-device-compile tool
Specialized Kernels Not generated 2,574 specialized kernels + symmetric kernels
HIP Fabric API Disabled Enabled (hipMemImportFromShareableHandle found)
Copy Engine Batch Not explicitly shown enabled CE_BATCH_ASYNC_SUPPORTED enabled
amdsmi Found at /usr/include Not found (amd_smi_DIR missing)
Python Python3 3.14.5 Python3 3.15.0
Install Differences
Installed Files Baseline Test (preview)
nccl.h / nccl_device.h Not installed Installed (preview-only features)
nccl_device/ dir Not installed Installed
tuner directory Not installed /usr/share/rccl/tuner/ (Includes tuner CSV files gfx950)

3. DETAILED DIFFERENCES

3.1 VERSION AND RELEASE

  • Baseline: Version 7.2.0, release 4%{?dist} (final: -4.fc45)
  • Test: Version 7.14.0, release 0%{?dist} (final: -0.fc45)
  • The preview build uses a much newer ROCm version (7.14 vs 7.2)
  • Preview releases get a fresh release count (0)

3.2 COMPILER AND TOOLCHAIN

  • Both builds use the same clang-based compilers:
    • C compiler: /usr/lib64/rocm/llvm/bin/amdclang
    • C++ compiler: /usr/lib64/rocm/llvm/bin/amdclang++
    • Linker: amdclang++ with -fuse-ld=bfd
  • Major compiler version difference: Clang 22 (baseline) vs Clang 23 (test)

3.3 BUILD COMPLEXITY

The test build (7.14.0 preview) is significantly more complex:

  • 34x more build steps (17,676 vs 511)
  • Introduction of the rccl-device-compile tool for device code
  • 2,574 specialized kernels generated (vs none in baseline)
  • Symmetric kernel support added
  • HIP Fabric API support enabled (new features for GPU fabric networking)

3.4 ROCm LIBRARY CHANGES

  • amdsmi: Baseline found at /usr/include, Test NOT found (amd_smi_DIR missing)
    • This is expected if amdsmi package structure changed
  • HIP runtime: Both use libamdhip64.so.7, but different versions
  • ROCm version string: 7.2.0 vs 7.14.0
  • HIP version: 7.2.53211 vs 7.14.60850

3.5 SOURCE PATCHES (test/preview specific)

The preview build has additional patches applied in %prep: 1. Patch 1: 0001-rccl-multiple-nccl_domain.patch 2. sed: Add #include "plugin/nccl_tuner.h" to rccl_common.h 3. sed: Add #include <iostream> to ipc_init.cu 4. sed: Add #include <map> to transport/net.cc

3.6 CMAKE WARNINGS

Both builds show similar deprecation warnings:

  • find_package(ROCM) is deprecated (use ROCmCMakeBuildTools)
  • Some manually-specified CMake variables not used

The test build also shows a CMP0116 policy warning (dev-only, not build-affecting)

4. RISK ASSESSMENT

Both builds succeeded, indicating the specfile correctly handles both the stable release line (7.2.0) and the preview line (7.14.0).

Potential concerns: 1. amdsmi not found in test build - may need BuildRequires adjustment if amdsmi features are now required 2. Python 3.15 dependency (test build found 3.15.0) - ensure mock environment has compatible Python 3. Significant feature additions in preview (HIP Fabric, symmetric kernels) may warrant new or updated BR requirements

5. RECOMMENDATIONS

Since both builds were successful: 1. The %preview bcond is working correctly 2. The specfile handles both ROCm 7.2 and 7.14 build environments 3. The specfile could potentially benefit from: 

  * Explicit BuildRequires for amdsmi if needed at test-time
  * Documentation of new preview-only features (HIP Fabric, tuner)

No immediate action required for build stability.

END OF ANALYSIS

Packages

Colibri

Colibri is not yet in Fedora. If you want to try it out use this copr python-colibri-engine.

Overview

Colibrì is a cutting-edge inference engine that democratizes running massive frontier models, like the 744B-parameter GLM-5.2, on your local hardware. Unlike traditional runtimes that require complex datacenter setups, Colibrì orchestrates your SSD, RAM, and GPU into a unified tiered hierarchy. It allows Fedora users to deploy frontier AI on consumer-grade AMD GPUs, providing full visibility and control over the inference path at a fraction of the standard cost. See the upstream project here colibri upsteam

Getting Started

On Fedora, the engine and CLI are available in the standard repositories.

  1. **Install the Engine**:
  The Fedora package provides the core inference capabilities:
  sudo dnf install python3-colibri-engine
  1. **AMD GPU Support**:
  The Fedora binary is prebuilt with ROCm support for the following architectures:
  * gfx1100, gfx1101, gfx1102
  * gfx1151, gfx1150
  * gfx1200, gfx1201
  1. **Network Setup**:
  By default, the engine only listens on the localhost interface. To allow remote access to the API or dashboard, bind the server to your LAN IP (e.g., 0.0.0.0) and update your firewall:
  COLI_HOST=0.0.0.0 COLI_CUDA=1 coli ...

If there is a firewall issue, allow port 8000 through Fedora's firewall: 

  sudo firewall-cmd --add-port=8000/tcp --permanent
  sudo firewall-cmd --reload
  • **Note**: While the Fedora package includes the inference engine, **there is no support for the coli web dashboard** in this distribution.

Planning

Always use coli plan before booting a model to calculate memory tiering (disk/RAM/VRAM) and prevent "Out of Memory" errors. This command informs your startup flags by showing which parts of the model fit in fast memory and which must reside on disk. 

  ======= coli-plan-example =====
  coli plan --model ~/colibri/glm5.2

     ▄▀▀▀▄  ▄        colibri v1.4.0
   ▄▄▄▄▀▀▀▀▄▀▀        tiny engine, immense model
       ▀▀▀▀▀▀▀        GLM-5.2 · 744B MoE · int4 · streaming CPU
         ▀▀▀▀         plan · Disk / RAM / VRAM
           ▀          
   ──────────────────────────────────────────────────────────────
   policy quality · quality-preserving yes
   model  142 shards · 429.3 GB
   disk   283.7 GB cold experts · 913.4 GB free
   RAM    105.6 GB budget · 11.9 GB dense · 6.3 GB runtime · 87.5 GB warm experts · cap 53/layer
   VRAM   46.2 GB hot tier · ~2177 experts · 0:N/A, 1:N/A
   limit  disk expert misses
   hit    32% projected expert residency

   auto-tune:
     DRAFT=0            low hit rate: MTP widens expert union, adds disk reads
     COLI_CUDA_PIPE=2            multi-GPU: residual stays on-device across layers
   warn   cold expert misses may reach disk; normal decode speed depends on hit rate

  =====

Use the hit: If the 'hit' rate is low (< 50%), you may need to increase your VRAM capacity to improve expert residency. The auto-tune section also provides suggestions like CUDA_EXPERT_GB caps.

GPU Tuning

For AMD GPUs with 8 GB to 64 GB of VRAM, Colibrì uses a unified backend flag to enable the GPU. You can aggressively tune the engine to keep your GPU fed with the necessary "expert" weights.

  1. **Enable the GPU Backend**:
  Colibrì uses COLI_CUDA=1 to switch between NVIDIA and AMD (HIP/ROCm) backends automatically.
  1. **VRAM Tiering**:
  * CUDA_DENSE=1: Forces "dense" model layers into VRAM.
  * CUDA_EXPERT_GB=X: Caps the expert cache at X GB. Set this to your max available VRAM minus system overhead.
  1. **Over-Provisioning**:
  To hide the latency of fetching experts from RAM/Disk, set COLI_PREFETCH=2 and GPU_MAX_QUEUE=4 to keep the compute pipeline full.
Example call
  COLI_PREFETCH=2 COLI_CUDA_PIPE=2 GPU_MAX_QUEUE=4 COLI_ALLOW_INSECURE_BIND=1 COLI_CUDA=1 CUDA_DENSE=1 CUDA_EXPERT_GB=40 COLI_GPU=0 coli serve --host 192.168.1.3 --port 8000 --model ~/colibri/glm5.2

Emacs

agent-shell

agent-shell is allows your favorite coding agent to be embedded in your favorite editor. The upstream project is https://github.com/xenodium/agent-shell.

agent-shell is in Fedora 45.

For all local use with ollama and the pi-coding-agent do

Install ollama 

 dnf install ollama

Install pi-coding-agent and it acp help 

 dnf install pi-coding-agent pi-acp

Setup pi-coding-agent the first time to use ollama 

 ollama pull <your favorite model>
 ollama launch pi

Install agent-shell 

 dnf install emacs-agent-shell

There are many agent to pick look for your favorite in the upstream project or start in emacs 

 M-x agent-shell-[TAB]

To use the pi-coding-agent do 

 M-x agent-shell-pi-start-shell

Hermes-agent

Hermes Agent is marketed as a self-improving, comprehensive AI agent framework built by Nous Research. Its defining feature is its built-in, continuous learning loop, allowing it to evolve and improve its capabilities and knowledge over time through user interaction. It is designed to be platform-agnostic and highly persistent, ensuring the user's AI assistant evolves alongside them

The upstream project is https://github.com/nousresearch/hermes-agent

To give it a try, please install hermes-agent from Copr https://copr.fedorainfracloud.org/coprs/cqi/hermes-agent/

hermes-agent is being built for rawhide only. 0.14.0 is the base version for the initial Fedora package build.

Install hermes-agent: 

 dnf install hermes-agent

Setup your hermes agent: 

 hermes setup

setup is optional. If omitted, the setup is launched automatically for the first run of hermes.

Run the agent: 

 hermes

The Copr includes hermes-agent and its dependent packages covering these functionalities:

  • Inference providers: Anthropic, AWS Bedrock, Azure
  • TTS provider: edge-tts
  • Image generation backends
  • MCP
  • ACP
  • Hermes Dashboard
  • Skills: Google workspace and YouTube
  • Web search backends

Huggingface Libraries

Name Fedora Package Upstream URL Notes
python-huggingface-hub python-huggingface-hub https://github.com/huggingface/huggingface_hub Maintained by RedHatters (COPR/Logdetective/ramalama)
rust-hf-hub rust-hf-hub https://crates.io/crates/hf-hub Co-Maintained by the Rust SIG
python-transformers forge#13 https://github.com/huggingface/transformers
python-diffusers forge#14 https://github.com/huggingface/diffusers
python-datasets python-datasets https://github.com/huggingface/datasets Co-Maintained by the Python Packagers
python-tokenizers python-tokenizers https://github.com/huggingface/tokenizers Co-Maintained by the Rust SIG and the Python Packagers
rust-tokenizers rust-tokenizers https://crates.io/crates/tokenizers Co-Maintained by the Rust SIG
python-evaluate forge#15 https://github.com/huggingface/evaluate
python-timm forge#16 https://github.com/huggingface/timm
python-sentence-transformers forge#17 https://github.com/UKPLab/sentence-transformers
python-peft forge#18 https://github.com/huggingface/peft
python-accelerate python-accelerate https://github.com/huggingface/accelerate Co-Maintained by the Python Packagers
python-optimum forge#19 https://github.com/huggingface/optimum
python-safetensors python-safetensors https://github.com/huggingface/safetensors Co-Maintained by the Rust SIG and the Python Packagers
rust-safetensors rust-safetensors https://crates.io/crates/safetensors Co-Maintained by the Rust SIG
rust-spm_precompiled rust-spm_precompiled https://crates.io/crates/spm_precompiled Co-Maintained by the Rust SIG
rust-esaxx-rs rust-esaxx-rs https://crates.io/crates/esaxx-rs Co-Maintained by the Rust SIG
rust-unicode-normalization-alignments rust-unicode-normalization-alignments https://crates.io/crates/unicode-normalization-alignments Co-Maintained by the Rust SIG

Whipser-cpp

Name Fedora Package Upstream URL Notes
whisper-cpp whisper-cpp https://github.com/ggerganov/whisper.cpp

Llama-cpp

llama.cpp is the bare-metal C/C++ inference engine for the open LLM ecosystem, bridging HuggingFace weights to hardware (ROCm, Vulkan, CPU) via the GGUF format and aggressive quantization. It achieves raw hardware throughput through cache-line optimized memory layouts and zero Python overhead. llama-server exposes an OpenAI-compatible HTTP API—including KV caching and continuous batching—controllably via LLAMA_* environment variables.

Model Locations

Unsloth

https://huggingface.co/unsloth

Probably the best starting point if you aren't looking for something specific.

For example from https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF 

 llama-server -hf unsloth/gemma-4-E2B-it-GGUF:UD-Q4_K_XL

Minimizing Memory Usage in llama-server

This document details the command line arguments and environmental variables in llama-server that impact CPU (system RAM) and GPU (VRAM) memory consumption.

Command Line Arguments

These arguments control memory allocation at startup and during execution. They map to the corresponding environmental variables defined in the Environmental Variables section.

Model Loading & GPU

  • --gpu-layers, -ngl
Maximum number of layers to store in VRAM.
Maps to: LLAMA_ARG_GPU_LAYERS
Format: <LAYER_COUNT> ('auto', 'all', integer)
Examples: --gpu-layers=0
  * Default: 'auto'
  * Recommendation: 0 (or the lowest practical value) to shift model processing to system RAM/CPU, saving VRAM.
  * Negative effect: Significantly slower inference (tokens per second) by using the CPU.
  • --mmap, -mmap
Enables memory mapping of the model file (reads from disk instead of loading full model into RAM).
Maps to: LLAMA_ARG_MMAP
Format: (enabled/disabled)
Examples: --mmap=enabled
  * Default: enabled
  * Memory saving tip: Prevents the system from loading the entire model into physical RAM at startup.
  * Negative effect: (Disabled --no-mmap) can cause Out Of Memory (OOM) crashes on low RAM systems.
  • --mlock, -mlock
Forces the OS to keep the model in physical RAM (prevents swapping to disk).
Maps to: LLAMA_ARG_MLOCK
Format: (enabled/disabled)
Examples: --mlock=disabled
  * Default: enabled
  * Memory saving benefit: Disabling allows the OS to free up system RAM when the app is not using it.
  * Negative effect: Enabling (the default) pins the entire model in physical RAM permanently.
  • --numa, -N
Attempt optimizations that help on Non-Uniform Memory Access (NUMA) systems.
Maps to: LLAMA_ARG_NUMA
Format: <value> (none, distribute, isolate, numactl)
Examples: --numa=distribute
  * Default: none
  * Memory saving tip: Helps allocate memory evenly across multiple CPU nodes to prevent large spikes.
  • --split-mode, -sm
Split mode used to split a model across multiple devices.
Maps to: LLAMA_ARG_SPLIT_MODE
Format: <value> (none, layer, row, tensor)
Examples: --split-mode=none
  * Default: 'none'
  * Memory saving tip: Useful to prevent accidental split errors on single-GPU setups.
  • --tensor-split, -ts
The split ratio for models offloaded to GPUs.
Maps to: LLAMA_ARG_TENSOR_SPLIT
Format: <N0>, <N1>, ...
Examples: --tensor-split=0.5,0.5
  * Default: auto (0)
  * Memory saving tip: Ensures equal VRAM usage across multiple cards.
  • --cpu-moe, -cmoe
Keep Mixture of Experts (MoE) weights in system RAM.
Maps to: LLAMA_ARG_CPU_MOE
Format: <value/layers> (enabled/disabled)
Examples: --cpu-moe=enabled
  * Default: disabled (0)
  * Memory saving benefit: Massive reduction in VRAM requirements for MoE models.
  * Negative effect: Slightly slower compute speed due to RAM access lag.

Context & KV Cache

  • -c, --ctx-size
Context window limit (how much history the model keeps in memory).
Maps to: LLAMA_ARG_CTX_SIZE
Format: <TOKEN_COUNT>
Examples: --ctx-size=2048
  * Default (loaded from model): 0
  * Recommendation: 2048 or the absolute minimum required.
  * Memory saving benefit: Directly scales down the massive VRAM/RAM requirement of the KV cache.
  * Negative effect: The model will forget older conversation history.
  • -s, --cache-ram
Limit the maximum cache size in MiB for the prompt cache.
Maps to: LLAMA_ARG_CACHE_RAM
Format: <SIZE_IN_MiB>
Examples: --cache-ram=1024
  * Default: 8192
  * Memory saving benefit: Prevents the server from consuming all system RAM for caching.
  * Negative effect: May fail to cache larger prompts if the limit is reached.
  • -ctk, --cache-type-k, -ctv, --cache-type-v
KV cache data type for K and V.
Maps to: LLAMA_ARG_CACHE_TYPE_K & LLAMA_ARG_CACHE_TYPE_V
Format: <TYPE> (f16, q8_0, q4_0, etc.)
Examples: --cache-type-k=q8_0
--cache-type-v=q8_0
  * Default: f16
  * Memory saving benefit: q8_0 uses half the memory of f16. q4_0 uses a quarter.
  * Negative effect: Lower precision can lead to "memory effect" loss over long contexts.
  • -b, --batch-size
The logical maximum batch size.
Maps to: LLAMA_ARG_BATCH
Format: <COUNT>
Examples: --batch-size=512
  * Default: 2048
  * Memory saving benefit: Limits peak memory usage during generation.
  * Negative effect: Lower throughput.
  • -ub, --ubatch-size
The physical maximum batch size.
Maps to: LLAMA_ARG_UBATCH
Format: <COUNT>
Examples: --ubatch-size=512
  * Default: 512
  * Memory saving benefit: Limits peak memory required per request.
  * Negative effect: Lower throughput for long sequences.
  • -cms, --cache-idle-slots, -ccp, --no-cache-idle-slots
Save idle slots to the system RAM prompt cache upon new tasks.
Maps to: LLAMA_ARG_CACHE_IDLE_SLOTS
Format: <value> (enabled/disabled)
Examples: --cache-idle-slots=enabled
  * Default: enabled
  * Memory saving benefit: Frees VRAM by offloading inactive sessions to system RAM.
  * Negative effect: Slightly higher latency when waking a session.
  • -np, --parallel
Maximum number of server slots (concurrent requests).
Maps to: LLAMA_ARG_N_PARALLEL
Format: <COUNT>
Examples: --parallel=1
  * Default: -1 (dynamic)
  * Memory saving tip: Reduces the number of empty KV buffers allocated at startup.
  * Negative effect: Limits concurrent users allowed.
  • -kvu, --kv-unified
Use a single unified KV buffer for all slots (saves VRAM).
Maps to: LLAMA_ARG_KV_UNIFIED
Format: <value> (enabled/disabled)
Examples: --kv-unified=enabled
  * Default: disabled (unless --parallel is -1)
  * Memory saving tip: Eliminates duplicate VRAM buffers for every slot.
  • -ctxcp, --context-checkpoints
Maximum number of context checkpoints created per slot for prompt caching.
Maps to: LLAMA_ARG_CTX_CHECKPOINTS
Format: <COUNT>
Examples: --context-checkpoints=16
  * Default: 32
  * Memory saving benefit: Limits the storage size used for checkpoints.

Auto-Fitting & Limits

  • -fit, --fit
Adjust unset arguments to fit in device memory.
Maps to: LLAMA_ARG_FIT
Format: <value> (on/off)
Examples: --fit=on
  * Default: on
  * Recommendation: on to prevent crashes on low-memory systems.
  * Memory saving tip: Automatically resizes batches/context to save VRAM.
  • -fitt, --fit-target
Target margin size for --fit (auto-fit).
Maps to: LLAMA_ARG_FIT_TARGET
Format: <SIZE_IN_MiB>
Examples: --fit-target=0
  * Default: 1024
  * Recommendation: 0 to allow maximum usage.
  * Negative effect: A high target (e.g. 4096) forces the server to keep memory free artificially, lowering performance.
  • -fitc, --fit-ctx
Minimum context size for the --fit option.
Maps to: LLAMA_ARG_FIT_CTX
Format: <COUNT>
Examples: --fit-ctx=2048
  * Default: 4096
  * Recommendation: 2048 to stop auto-fit from shrinking history to 0.
  • -kvo, --kv-offload
Controls whether the KV cache can be offloaded from VRAM to system RAM.
Maps to: LLAMA_ARG_KV_OFFLOAD
Format: (enabled/disabled)
Examples: --kv-offload=enabled
  * Default: enabled
  * Memory saving tip: Moves inactive KV data to system RAM, freeing up VRAM.
  * Negative effect: --no-kv-offload keeps everything strictly in VRAM, often leading to OOM on lower GPU cards.
Environmental Variables

These environmental variables dictate behavior at startup.

Context Window & Cache

  • LLAMA_ARG_CTX_SIZE
Size of the prompt context window in tokens.
Maps to: --ctx-size
Format: <TOKEN_COUNT>
Examples: LLAMA_ARG_CTX_SIZE=2048
  * Default (loaded from model metadata): 0
  * Recommendation: 2048 or the minimum practical context window.
  * Memory saving benefit: Directly and dramatically scales down KV cache memory usage.
  * Negative effect: Forces the truncation of conversation history.
  • LLAMA_ARG_CACHE_RAM
Limits the maximum size of the system RAM prompt cache in MiB.
Maps to: --cache-ram
Format: <SIZE_IN_MiB> (-1 = no limit)
Examples: LLAMA_ARG_CACHE_RAM=1024
  * Default: 8192
  * Recommendation: 1024 or 2048.
  * Memory saving benefit: Prevents the prompt cache from consuming all available system RAM.
  * Negative effect: The server may evict cached prompts sooner.
  • LLAMA_ARG_CACHE_TYPE_K & LLAMA_ARG_CACHE_TYPE_V
Data type for the KV cache (K and V).
Maps to: --cache-type-k & --cache-type-v
Format: <TYPE> (f16, q8_0, q4_0, etc.)
Examples: LLAMA_ARG_CACHE_TYPE_K=q8_0
LLAMA_ARG_CACHE_TYPE_V=q8_0
  * Default: f16
  * Memory saving benefit: q8_0 uses 50% of the memory of f16.
  * Negative effect: Lower precision can cause generation quality loss ("memory effect").
  • LLAMA_ARG_CACHE_PROMPT
Enable prompt caching.
Maps to: --cache-prompt
Format: <value> (enabled/disabled)
Examples: LLAMA_ARG_CACHE_PROMPT=enabled
  * Default: enabled
  * Memory saving benefit: Reduces VRAM usage by caching repeated prompt prefixes.
  * Negative effect: Increases system RAM usage for storing cached chunks.
  • LLAMA_ARG_CACHE_IDLE_SLOTS
Saves idle sessions to the system RAM prompt cache instead of clearing from VRAM.
Maps to: --cache-idle-slots
Format: <value> (enabled/disabled)
Examples: LLAMA_ARG_CACHE_IDLE_SLOTS=enabled
  * Default: enabled
  * Memory saving benefit: Frees VRAM by offloading inactive sessions to system RAM.
  • LLAMA_ARG_N_GPU_LAYERS
Maximum number of model layers to offload to the GPU VRAM.
Maps to: --gpu-layers
Format: <LAYER_COUNT> (auto, 'all', <integer>)
Examples: LLAMA_ARG_N_GPU_LAYERS=0
  * Default: 'auto'
  * Recommendation: 0 to move offloading back to system RAM/CPU.
  * Negative effect: Huge performance degradation by using the CPU.
  • LLAMA_ARG_N_GPU_LAYERS_DRAFT
Maximum number of layers to store in VRAM for the speculative decoding model.
Maps to: --gpu-layers-draft
Format: <LAYER_COUNT> (integer)
Examples: LLAMA_ARG_N_GPU_LAYERS_DRAFT=0
  * Default: 0
  * Recommendation: 0 (or lowest practical) to ensure the main model's VRAM isn't encroached upon.
  * Negative effect: Slows down speculative decoding.
  • LLAMA_ARG_N_PARALLEL
Maximum number of server slots (concurrent requests).
Maps to: --parallel
Format: <COUNT>
Examples: LLAMA_ARG_N_PARALLEL=1
  * Default: -1 (dynamic)
  * Memory saving benefit: Reduces the overhead of empty KV buffers allocated at startup.
  * Negative effect: Limits concurrent users allowed.

Model Loading & Sizing

  • LLAMA_ARG_MMAP
Memory maps the model file.
Maps to: --mmap
Format: <value> (true/false)
Examples: LLAMA_ARG_MMAP=true
  * Default: true
  * Memory saving benefit: Prevents massive RAM spikes at startup.
  • LLAMA_ARG_MLOCK
Forces the OS to keep the model in physical RAM.
Maps to: --mlock
Format: <value> (true/false)
Examples: LLAMA_ARG_MLOCK=false
  * Default: true
  * Memory saving benefit: Disabling allows the OS to reclaim memory.
  * Negative effect: true pins the model in RAM permanently.
  • LLAMA_ARG_CPU_MOE
Keep Mixture of Experts (MoE) weights in system RAM.
Maps to: --cpu-moe
Format: <value> (enabled/disabled)
Examples: LLAMA_ARG_CPU_MOE=enabled
  * Default: disabled
  * Memory saving benefit: Keeps massive switch weights out of VRAM.
  * Negative effect: Increases system RAM usage.
  • LLAMA_ARG_FIT
Adjust unset arguments to fit in device memory.
Maps to: --fit
Format: <value> (on/off)
Examples: LLAMA_ARG_FIT=on
  * Default: on
  * Recommendation: on to prevent OOM crashes on low-memory machines.
  * Memory saving benefit: Automatically resizes batches/context to save VRAM.
  • LLAMA_ARG_FIT_TARGET
Target margin size for --fit (auto-fit).
Maps to: --fit-target
Format: <SIZE_IN_MiB>
Examples: LLAMA_ARG_FIT_TARGET=0
  * Default: 1024
  * Recommendation: 0 (Uses available memory).
  * Negative effect: A high target forces the server to keep memory free artificially.
  • LLAMA_ARG_KEEP
Number of tokens to keep in the prompt after processing.
Maps to: --n-keep
Format: <COUNT>
Examples: LLAMA_ARG_KEEP=4096
  * Default: 4096
  * Memory saving benefit: Limiting this reduces the size of the prompt cache.
  * Negative effect: May truncate prompt chunks used for caching.

Environmental Variables

Network & Connection

  • LLAMA_ARG_HOST
IP address or path to a UNIX socket that llama-server binds to for listening.
Format: <IP_ADDRESS> or <path>.sock
Examples:
* LLAMA_ARG_HOST=127.0.0.1
  * Default (localhost only): 127.0.0.1
  * Bind to all interfaces: 0.0.0.0
  • LLAMA_ARG_PORT
TCP port that the server listens on for incoming connections.
Format: <PORT_NUM>
Examples: LLAMA_ARG_PORT=8080
  * Default: 8080
  • LLAMA_ARG_REUSE_PORT
Allows multiple sockets to bind to the same local port (USE with caution).
Examples:
* LLAMA_ARG_REUSE_PORT=1
  * Default: Disabled
  • LLAMA_API_KEY
One or more API keys used to authenticate incoming requests. Accepts comma-separated keys.
Format: <KEY_1>,<KEY_2>
Examples: LLAMA_API_KEY=sk-abc123,sk-def456
  * Note: Acts as a list of valid keys. Any key in the request must match one of these.
  • LLAMA_ARG_API_KEY_FILE
Specifies a path to a text file containing valid API keys.
Format: <filesystem_path>
  • LLAMA_ARG_SSL_KEY_FILE
Path to a PEM-encoded private key for enabling HTTPS/SSL.
Format: <filesystem_path>
  • LLAMA_ARG_SSL_CERT_FILE
Path to a PEM-encoded SSL certificate for enabling HTTPS/SSL.
Format: <filesystem_path>

Router & Hot-Swapping

  • LLAMA_ARG_MODELS_DIR
Directory path containing .gguf model files to be served by the router server.
Format: <filesystem_path>
  • LLAMA_ARG_MODELS_PRESET
Path to an INI configuration file containing predefined model presets for the router server.
Format: <filesystem_path>
  • LLAMA_ARG_MODELS_MAX
For router mode, caps the maximum number of models that can be loaded into memory simultaneously.
Format: <INTEGER>
Examples: LLAMA_ARG_MODELS_MAX=4
  * Default: 0 (unlimited)
  • LLAMA_ARG_MODELS_AUTOLOAD
Automatically attempts to load all models found in the models_dir upon router startup.
Examples:
* LLAMA_ARG_MODELS_AUTOLOAD=0
  * Default: 1 (Enabled)
  • LLAMA_SERVER_ROUTER_PORT
Server-internal variable. Sets the port the router server uses for internal child communication. Presence indicates child mode.
Format: <PORT_NUM>
  • LLAMA_SERVER_CHILD_MODE
Server-internal variable. Determines the execution phase of a child process (e.g., model download).
Format: <STRING> (e.g., "download", "normal")
Examples: LLAMA_SERVER_CHILD_MODE=download

Model & Multimodal Configuration

  • LLAMA_ARG_MODEL
Filepath to the local model file or an Hugging Face repository ID to load.
Format: <filesystem_path> or <user>/<model>
Examples: LLAMA_ARG_MODEL=./models/llama.gguf
  • LLAMA_ARG_MODEL_URL
URL to download or stream the model from if it is not present in the local models_dir.
Format: <URL>
  • LLAMA_ARG_MMPROJ
Path to a multimodal projector file required for vision models.
Format: <filesystem_path>
  • LLAMA_ARG_MMPROJ_URL
URL to download the multimodal projector file if not present locally.
Format: <URL>
  • LLAMA_ARG_MMPROJ_AUTO
Automatically searches for and attaches a matched multimodal projector file.
Examples:
* LLAMA_ARG_MMPROJ_AUTO=0
  * Default: Enabled
  • LLAMA_ARG_ALIAS
Defines custom alias names for the loaded model (comma-separated) to identify it within the UI or API.
Format: <ALIAS_1>,<ALIAS_2>
Examples: LLAMA_ARG_ALIAS=llama-v3,chat-model
  * Default: Auto-generated from the model's filename.
  • LLAMA_ARG_TAGS
Adds arbitrary tags to the model for better organization and filtering in the server router.
Format: <TAG_1>,<TAG_2>
Examples: LLAMA_ARG_TAGS=vision,qwen2

Memory & Cache Management

  • LLAMA_CACHE
Global override for the base directory where llama.cpp caches downloaded models and intermediate artifacts.
Format: <filesystem_path>
Examples: LLAMA_CACHE=/var/cache/llama
  * Default: XDG_CACHE_HOME/llama.cpp or ~/.cache/llama.cpp
  • LLAMA_ARG_CTX_SIZE
Sets the context window size (in tokens) for the loaded model.
Format: <INTEGER>
Examples: LLAMA_ARG_CTX_SIZE=8192
  * Default: 512 (often overridden by model architecture defaults)
  • LLAMA_ARG_BATCH
Sets the batch size used for processing prompt tokens.
Format: <INTEGER>
Examples: LLAMA_ARG_BATCH=512
  • LLAMA_ARG_UBATCH
Sets the unbatched context size (micro-batch size) used for token processing.
Format: <INTEGER>
  • LLAMA_ARG_N_PREDICT
Number of tokens to generate in each streaming chunk.
Format: <POSITIVE_INTEGER>
Examples: LLAMA_ARG_N_PREDICT=128
  • LLAMA_ARG_CONT_BATCHING
Enables continuous (dynamic) batching, allowing multiple requests to be processed concurrently rather than sequentially.
Examples:
* LLAMA_ARG_CONT_BATCHING=0
  * Default: Enabled
  • LLAMA_ARG_CACHE_PROMPT
Caches prompt keys/values (KV cache) so identical requests do not require recomputation.
Examples:
* LLAMA_ARG_CACHE_PROMPT=0
  * Default: Enabled
  • LLAMA_ARG_CACHE_REUSE
Minimum chunk size to attempt reusing from the prompt cache via KV shifting.
Format: <INTEGER>
  • LLAMA_ARG_CACHE_IDLE_SLOTS
Saves the KV states of idle slots to the cache-prompt to save VRAM, clearing them when a sequence utilizes unified KV.
Examples:
* LLAMA_ARG_CACHE_IDLE_SLOTS=1
  * Default: Enabled
  • LLAMA_ARG_CACHE_RAM
Sets the maximum size of the RAM cache used for slot states in MiB.
Format: <INTEGER> (-1 is default, 0 is disabled)
Examples: LLAMA_ARG_CACHE_RAM=4096
  • LLAMA_ARG_FIT
llama.cpp specific. Enables automatic memory fitting for unset VRAM/RAM options.
Examples:
* LLAMA_ARG_FIT=off
  * Default: on
  • LLAMA_ARG_FIT_TARGET
Target free VRAM margin (in MB) when LLAMA_ARG_FIT is active.
Format: <INTEGER> (MB)
Examples: LLAMA_ARG_FIT_TARGET=512

GPU & Hardware Acceleration

  • LLAMA_ARG_FLASH_ATTN
Enables Flash Attention (FA2) to optimize attention mechanism performance and VRAM usage.
Format: <STRING> ("on", "off", or "auto")
Examples:
* LLAMA_ARG_FLASH_ATTN=on
  * Default: auto
  • LLAMA_ARG_NGPU_LAYERS
Number of transformer layers to offload to the GPU.
Format: <INTEGER> (number of layers)
Examples: LLAMA_ARG_NGPU_LAYERS=-1
  * Default: -1 (offload all available layers)
  • LLAMA_ARG_SPLIT_MODE
Strategy for splitting the model's layers across multiple GPUs if n_gpu_layers > n_layers_available.
Format: <STRING> (e.g., layer, row)
Examples: LLAMA_ARG_SPLIT_MODE=layer
  • LLAMA_ARG_TENSOR_SPLIT
Fractional split of each layer across multiple GPUs.
Format: <FLOAT>,<FLOAT>,...
Examples: LLAMA_ARG_TENSOR_SPLIT=0.4,0.6
  • LLAMA_ARG_MAIN_GPU
Index of the primary GPU used for intermediate tensors when splitting across multiple GPUs.
Format: <INTEGER>
Examples: LLAMA_ARG_MAIN_GPU=0
  • LLAMA_ARG_MLOCK
Forces the llama library to lock model weights into RAM, preventing them from being paged out to disk.
Examples:
* LLAMA_ARG_MLOCK=1
  * Default: Disabled
  • LLAMA_ARG_MMAP
Enables memory mapping (MMAP) of model weights directly from disk into system RAM (saves RAM load time).
Examples:
* LLAMA_ARG_MMAP=0
  * Default: Enabled
  • LLAMA_ARG_KV_UNIFIED
Uses a single unified KV buffer shared across all sequences (slots) to minimize VRAM fragmentation.
Examples:
* LLAMA_ARG_KV_UNIFIED=0
  * Default: Enabled if slot count is auto.

Inference & Request Control

  • LLAMA_ARG_N_PARALLEL
Controls the maximum number of concurrent inference requests (slots) processed per model.
Format: <INTEGER>
Examples: LLAMA_ARG_N_PARALLEL=4
  * Default: 0 (auto-detect based on hardware)
  • LLAMA_ARG_TIMEOUT
Caps the time allowed for HTTP requests to complete (read/write) before dropping the connection.
Format: <INTEGER> (seconds)
Examples: LLAMA_ARG_TIMEOUT=60
  * Default: 0 (unlimited)
  • LLAMA_ARG_SSE_PING_INTERVAL
The interval in seconds to send Server-Sent Events (SSE) heartbeats to keep client connections alive.
Format: <INTEGER> (-1 to disable)
Examples: LLAMA_ARG_SSE_PING_INTERVAL=15
  * Default: 15
  • LLAMA_ARG_THREADS_HTTP
Number of dedicated worker threads used to process incoming and outgoing HTTP requests.
Format: <INTEGER>
Examples: LLAMA_ARG_THREADS_HTTP=4
  * Default: 2 or auto
  • LLAMA_ARG_ENDPOINT_METRICS
Enables the Prometheus-compatible /metrics endpoint for monitoring server performance.
Examples:
* LLAMA_ARG_ENDPOINT_METRICS=0
  * Default: Enabled
  • LLAMA_ARG_ENDPOINT_PROPS
Enables the POST /props endpoint to allow runtime changes to global server properties via API.
Examples:
* LLAMA_ARG_ENDPOINT_PROPS=0
  * Default: Enabled
  • LLAMA_ARG_ENDPOINT_SLOTS
Enables slot monitoring endpoints in the server API.
Examples:
* LLAMA_ARG_ENDPOINT_SLOTS=0
  * Default: Enabled

Debugging, Logging & Development

  • LLAMA_TRACE
Enables heavy, token-by-token trace logging for slot inference, useful for deep debugging.
Format: <INTEGER> (1 = enabled, 0 = disabled)
Examples: LLAMA_TRACE=1
  * Default: 0 (Disabled)
  • LLAMA_SERVER_SLOTS_DEBUG
Enables detailed debug logging strictly focused on slot lifecycle, state, and transitions.
Format: <INTEGER>
Examples: LLAMA_SERVER_SLOTS_DEBUG=1
  * Default: 0 (Disabled)
  • LLAMA_MEDIA_MARKER
Pins a custom, reproducible token marker string used to delimit multimodal media chunks in prompts.
Format: <STRING>
Examples: LLAMA_MEDIA_MARKER=<__my_media__>
  * Default: <__media_RANDOM_STRING__>
  • LLAMA_APP_CMD
Overrides the subcommand (e.g., serve vs cli) injected by default when spawning child processes.
Format: <STRING>
Examples: LLAMA_APP_CMD=s
  • LLAMA_ARG_UI
Enables or disables the built-in browser-based WebUI bundled with the server.
Examples:
* LLAMA_ARG_UI=0
  * Default: Enabled

Olla

Packaging :

Olla is a high-performance, low-overhead proxy and load balancer for LLM infrastructure, written in Go. It intelligently routes requests across local and remote inference nodes (supporting Ollama, vLLM, LM Studio, LiteLLM, SGLang, and OpenAI-compatible endpoints). See the upstream project here https://github.com/thushan/olla

Ollama Setup

This is a good starting point https://thushan.github.io/olla/configuration/examples/#home-lab-setup

See advanced Ollama setup below for remote access to ollama. An example for 3 ollamas configuration file has these fields 

  proxy:                                                                                             
    load_balancer: "priority"                                                                        
                                                                                                     
  discovery:                                                                                         
    type: "static"                                                                                   
    static:                                                                                          
      endpoints:                                                                                     
        - url: "http://localhost:11434"                                                              
          name: "local-ollama"                                                                       
          type: "ollama"                                                                             
          priority: 100                                                                              
          model_url: "/api/tags"                                                                     
          health_check_url: "/"                                                                      
          check_interval: 2s                                                                         
          check_timeout: 1s                                                                          
                                                                                                     
        - url: "http://192.168.1.21:11434"                                                           
          name: "remote-ollama-21"                                                                   
          type: "ollama"                                                                             
          priority: 200                                                                              
          model_url: "/api/tags"                                                                     
          health_check_url: "/"                                                                      
          check_interval: 2s                                                                         
          check_timeout: 1s                                                                          
                                                                                                     
        - url: "http://192.168.1.22:11434"                                                           
          name: "remote-ollama-22"                                                                   
          type: "ollama"                                                                             
          priority: 300                                                                              
          model_url: "/api/tags"                                                                     
          health_check_url: "/"                                                                      
          check_interval: 2s                                                                         
          check_timeout: 1s

For the agents using ollama in their configuration change the URL from local ollama to local olla. For example for pi-coding-agent, edit your ~/.pi/agent/models.json and change 

 "baseUrl": "http://127.0.0.1:11434/v1",

to 

 "baseUrl": "http://127.0.0.1:40114/olla/openai/v1",


Ollama

Ollama is a application for running local large language models. See their upstream project here https://ollama.com/

On Fedora, Ollama is available on aarch64 and x86_64 and has the accelerated backends ROCm and Vulkan. The general command to install is 

 dnf install ollama

For CPU only 

 dnf install ollama-base

For CPU (x86_64) and ROCm 

 dnf install ollama-base ollama-rocm

For CPU and Vulkan 

 dnf install ollama-base ollama-vulkan

Ollama has a systemd interface after it is installed do 

 systemctl start ollama

Ollama integrates with several other Fedora packages. These are accessed by the ollama command 

 ollama launch <package>

These packages are

  • openclaw
  • pi (pi-coding-agent)

Release Notes

Fedora 46

Removed legacy CGO engines; the backend now uses llama-server exclusively for GGML model execution, streamlining the codebase and improving stability. The upstream ollama still bundles its own llama.cpp, fedora ollama uses the system llama.cpp. There will be a problem using some of ollama's models. When there is look for the equivalent llama.cpp model.

For example change 

 ollama pull qwen3.6

to 

 ollama pull https://huggingface.co/ggml-org/Qwen3.6-27B-GGUF

Because the backend moved to llama.cpp, the older ollama-<backend> packages have gone away. There is only ollama, install with 

dnf install ollama

When using environmental variables to fine tune ollama, also look at the llama.cpp environmental for additional controls.

Fedora 45

Core Engine & Model Support

  • New Models: Added Gemma 4 support and updated Kimi models.
  • Vision Capabilities: Enabled image-based inference for models like Flux.
  • Smart Launch: Improved system recommendations using accurate VRAM detection to prevent overflows.

ROCm & AMD GPU Improvements

  • ROCm 7.2.x Update:Updated the driver stack and resolved kernel build issues for AMD hardware.
  • iGPU Compatibility: Added support for Linux AMD APUs using GTT for shared memory.
  • VRAM Reporting: Fixed critical memory detection errors by restoring sysfs-based reporting.
Fedora 44

Core Engine & Model Support

  • Tool Calling: Deepened support for function calling, enabling robust agentic workflows and complex, multi-step reasoning tasks.
  • DeepSeek Ecosystem: Added full support for DeepSeek R1 and DeepSeek V3, allowing powerful reasoning and coding capabilities.
  • Multi-Modal & Vision: Enhanced context window performance for newer model families, including Kimi K2 and the Gemma series.

Advanced Systemd Setup

The default endpoint for ollama is 127.0.0.1:11434. To system wide change, edit the /usr/lib/systemd/system/ollama.service 

 [Service]
 # OLLAMA_HOST: Controls the IP address and port Ollama binds to for API requests.
 # Format: <IP_ADDRESS>:<PORT>
 #
 # Examples:
 #   Default (localhost only, port 11434): 127.0.0.1:11434
 #   Change to a custom port:               127.0.0.1:8080
 #   Bind to all IPv4 interfaces (default port): 0.0.0.0:11434
 #   Bind to all IPv6 interfaces (default port): [::]:11434
 #
 # ⚠️ Security: Binding to 0.0.0.0 or [::] exposes Ollama to your network.
 #   Use firewall rules (ufw/iptables/nftables) or a reverse proxy (nginx/caddy)
 #   to restrict access in production environments.
 Environment="OLLAMA_HOST=127.0.0.1:11434"
 #Environment="OLLAMA_HOST=127.0.0.1:<your port>"
 #Environment="OLLAMA_HOST=0.0.0.0:11434"	
 #Environment="OLLAMA_HOST=[::]:11434"

For example, to allow remote local access, use this line 

 Environment="OLLAMA_HOST=0.0.0.0:11434

Downloading models depend on your network, by default ollama assumes your network is fast and stable. The ollama.service file has this template of enviromental variables the will help you fine tune the download to match your network or personal preferences. 

 # Variables for controlling downloads
 #   --------------------------------------------------------------------------
 # OPTION 1: DEFAULT (Original Ollama Baseline)
 # Best for: General purpose use, balanced performance vs stability
 #   --------------------------------------------------------------------------
 # Environment="OLLAMA_NUM_DOWNLOAD_PARTS=16"
 # Environment="OLLAMA_MIN_DOWNLOAD_PART_SIZE=100MB"
 # Environment="OLLAMA_MAX_DOWNLOAD_PART_SIZE=1000MB"

 # --------------------------------------------------------------------------
 # OPTION 2: MINIMAL / CONSERVATIVE (Slow, Unstable, or Restrictive Networks)
 # Best for: Corporate proxies, flaky connections, low RAM, HDD storage,
 # or registry rate-limits concurrent connections (often caps at 4-8)
 # --------------------------------------------------------------------------
 # Environment="OLLAMA_NUM_DOWNLOAD_PARTS=1"
 # Environment="OLLAMA_MIN_DOWNLOAD_PART_SIZE=1MB"
 # Environment="OLLAMA_MAX_DOWNLOAD_PART_SIZE=1MB"

 # --------------------------------------------------------------------------
 # OPTION 3: MAXIMUM / AGGRESSIVE (High-Bandwidth, Modern Network/SSD)
 # Best for: Fiber/cable broadband, low-latency environments, high-I/O SSDs,
 # and registries that allow high concurrency. Avoids "thundering herd"
 # stalls on fast pipes by using larger chunks.
 # --------------------------------------------------------------------------
 # Environment="OLLAMA_NUM_DOWNLOAD_PARTS=32"
 # Environment="OLLAMA_MIN_DOWNLOAD_PART_SIZE=256MB"
 # Environment="OLLAMA_MAX_DOWNLOAD_PART_SIZE=2GB"

When you are finished with editing the ollama.service file, restart systemd 

 sudo systemctl daemon-reload
 sudo systemctl restart ollama

Local Ollama Model Repository Setup Guide

Motivation and Background

Running large language models (LLMs) locally often requires downloading massive files—ranging from 4 GB to 20 GB or more. In environments with multiple users (labs, classrooms, or offices), downloading these models repeatedly wastes time and precious internet bandwidth. Additionally, public model registries can sometimes be slow, rate-limited, or unreachable.

Setting up a Local Ollama Model Repository solves this by acting as a private mirror on your local area network (LAN). Once a model is pushed to this local server, any other machine on the network can download it instantly at maximum LAN speeds (Gigabit+), bypassing the internet entirely. This setup transforms a 30-minute internet download into a 20-second local transfer.

1. Docker Server Setup

To host your private model library, you only need a lightweight Docker container running the standard Docker Registry image.

The script below sets up the registry, maps a persistent local storage folder for the models, ensures proper permissions, and configures it to restart automatically if the server reboots.

Save the following as setup-registry.sh: 

 #!/bin/sh
 set -x
 # 1. Ensure the storage folder has read/write access for the container
 # (Adjust path as needed, e.g., /var/lib/ollama-mirror)
 chmod 777 /<your-path>/models/ollama
 # 2. Pull the registry image
 docker pull registry:2
 # 3. Safety cleanup: remove old container if it exists
 docker rm -f local-models 2>/dev/null
 # 4. Start the registry with auto-restart
 docker run -d \
      --restart unless-stopped \
      -p 5000:5000 \
      -v /<your-path>/models/ollama:/var/lib/registry \
      --name local-models \
      registry:2

Note: The registry listens on port 5000. Ensure your firewall allows traffic on this port.

2. Usage Examples

Assume your server is running at IP address 192.168.1.25.

Step A: Populate the Registry (On the Server Machine)

To add a model to your local library, you must first copy the local model to a name that includes the registry address, then push it.

1. Copy/Tag the model to the registry address: 

 ollama cp llama3.2 192.168.1.25:5000/library/llama3.2

2. Push to the local server: Important: You must use the --insecure flag. This tells Ollama to accept plain HTTP since our local server doesn't have an SSL certificate. 

 ollama push 192.168.1.25:5000/library/llama3.2 --insecure
Step B: Consume the Registry (On Any Client Machine)

From any other computer connected to the LAN, pull the model directly from the local server. This will happen at LAN speeds, downloading massive models in seconds. 

 ollama pull 192.168.1.25:5000/library/llama3.2 --insecure
Step C: Run the Model

Once pulled, you can run the model using the full registry path or just the model name if it is your default. 

 ollama run 192.168.1.25:5000/library/llama3.2
3. Verifying Your Upload

To confirm a model has been successfully uploaded to your server, you can query the registry API using curl: 

 curl http://192.168.1.25:5000/v2/_catalog

You should see your repository names listed in the output JSON: 

 {"repositories":["library/llama3.2"]}

Minimizing Memory Usage in Ollama

This document details the environmental variables and command line arguments in Ollama that impact CPU (system RAM) and GPU (VRAM) memory consumption.

Since Ollama acts as a backend wrapper around llama.cpp, many of Ollama's variables pass directly down to the underlying inference engine.

Context Window & KV Cache For a running LLM, the Key-Value (KV) cache usually consumes more memory than the model weights.

  • OLLAMA_CONTEXT_LENGTH
Defines the default maximum context window length (number of tokens) for new models.
Maps to llama.cpp: LLAMA_ARG_CTX_SIZE
Format: <TOKEN_COUNT>
Examples:
* OLLAMA_CONTEXT_LENGTH=2048
  * Default (loaded from model): 0
  * Recommendation: 2048 (or the absolute minimum your workflow requires).
  * Memory saving benefit: Directly passes this value to the underlying llama.cpp layer, scaling down the KV cache size in VRAM/RAM.
  * Negative effect of setting too low: The model will forcefully truncate older conversation history.
* OLLAMA_CONTEXT_LENGTH=32768
  * Recommendation: 32768 for long-context needs.
  • OLLAMA_KV_CACHE_TYPE
Sets the quantization precision for the Key/Value cache used in attention mechanisms.
Maps to llama.cpp: LLAMA_ARG_CACHE_TYPE_K and LLAMA_ARG_CACHE_TYPE_V
Format: <TYPE> (f16, i8, i4, if16)
Examples:
* OLLAMA_KV_CACHE_TYPE=i8
  * Default: f16
  * Recommendation: i8 (or i4 if memory is extremely constrained).
  * Memory saving benefit: Passing this to llama.cpp, the i8 type cuts the KV cache memory in half compared to the default f16.
  * Negative effect of quantizing: Lower precision may lead to a "memory effect," where generation quality drops off over longer contexts.
  • OLLAMA_NUM_PARALLEL
Sets the maximum number of parallel sequences to execute simultaneously within a single loaded model.
Maps to llama.cpp: LLAMA_ARG_N_PARALLEL
Format: <COUNT>
Examples:
* OLLAMA_NUM_PARALLEL=1
  * Default: 1
  * Recommendation: 1 (strictly isolate the model from other parallel tasks to save memory).
  * Memory saving benefit: Passed directly to llama.cpp to prevent allocating multiple KV buffers for the same model.
  * Negative effect of setting too high: Increases the baseline VRAM required per loaded model.
  • OLLAMA_SCHED_SPREAD
Forces Ollama to spread loaded models evenly across all available GPUs.
Maps to llama.cpp: LLAMA_ARG_SPLIT_MODE
Format: <value> (true, 1)
Examples:
* OLLAMA_SCHED_SPREAD=false
  * Default: false
  * Recommendation: false to prevent accidentally forcing llama.cpp to split a single model across multiple VRAMs via --split-mode logic.
* OLLAMA_SCHED_SPREAD=true
  * Negative effect: Forces llama.cpp to use a split mode, significantly increasing total VRAM usage due to the overhead of inter-card communication.

Model Loading & Session Management

  • OLLAMA_FLASS_ATTENTION
Enables experimental Flash Attention, which optimizes memory footprint during calculation.
Maps to llama.cpp: LLAMA_ARG_FLASH_ATTN
Format: <value> (true, 1)
Examples:
* OLLAMA_FLASS_ATTENTION=1
  * Default: false
  * Recommendation: 1 to enable optimization.
  * Memory saving benefit: Passing --flash-attn to llama.cpp, which reduces peak VRAM usage during forward passes by recomputing attention weights on the fly.
  * Negative effect: May cause hardware compatibility issues on older AMD/Intel cards without robust Vulkan/OpenCL support.
  • OLLAMA_N_GPU_LAYERS
The maximum number of model layers to offload to the GPU.
Maps to llama.cpp: LLAMA_ARG_N_GPU_LAYERS
Format: <COUNT> ('auto', 'all', or integer)
Examples:
* OLLAMA_N_GPU_LAYERS=0
  * Default: 'auto' (loads as many layers as possible into VRAM)
  * Recommendation: 0 (forces the model out of VRAM entirely and into System RAM/CPU).
  * Memory saving benefit: Frees up all VRAM previously reserved for model weights by passing directly to -ngl <LAYER_COUNT>.
  * Negative effect: Massive reduction in generation speed as it no longer benefits from GPU acceleration.
* OLLAMA_N_GPU_LAYERS=30
  * Recommendation: 30 for a model requiring 32 layers (keeps the very last attention layers in VRAM).


Quick Tuning Checklist

To aggressively minimize memory usage on a constrained Ollama instance using the underlying llama.cpp engine: 

1. **Context Window:** Set OLLAMA_CONTEXT_LENGTH=2048.
2. **KV Cache:** Quantize KV cache to OLLAMA_KV_CACHE_TYPE=i8.
3. **Model Loading:** Set OLLAMA_N_GPU_LAYERS=0 (forcing CPU/SYS RAM loading).
4. **Session Management:** Set OLLAMA_KEEP_ALIVE=0s to immediately unload models from VRAM.
5. **Parallel Models:** Set OLLAMA_MAX_LOADED_MODELS=1 to prevent overlap.

Environmental Variables

Network & Connection

  • OLLAMA_HOST
Controls the IP address and port Ollama binds to for API requests.
Format: <IP_ADDRESS>:<PORT>
Examples:
* OLLAMA_HOST=127.0.0.1:11434
  * Default (localhost only): 127.0.0.1:11434
  * Bind to all IPv4 interfaces: 0.0.0.0:11434
    ⚠️ Security: Binding to 0.0.0.0 or [::] exposes Ollama to your network. Use firewall rules or a reverse proxy to restrict access in production.
  • OLLAMA_ORIGIN
Defines allowed cross-origin requests for the server.
Format: Comma-separated list of origins.
Examples: OLLAMA_ORIGINS=http://localhost:3000,https://myapp.com
    ⚠️ Security: Setting "*" allows all origins, increasing CSRF risk. Limit to specific origins in production.
  • OLLAMA_REMOTES
Specifies allowed hosts for downloading models from remote registries.
Format: Comma-separated list of hostnames.
Examples: OLLAMA_REMOTES=ollama.com,exampleregistry.com
* Default: ollama.com

Memory & Model Management

  • OLLAMA_MODELS
Controls the directory where Ollama stores downloaded models.
Format: <filesystem_path>
Examples: OLLAMA_MODELS=/var/lib/ollama/models
* Default: $HOME/.ollama/models
  • OLLAMA_KEEP_ALIVE
Dictates how long loaded models remain resident in memory after inference completes.
Format: Duration string (e.g., 2h, -1, 0s).
Examples: OLLAMA_KEEP_ALIVE=30m
* Default: 5m
* Never unload: -1
* Unload immediately: 0
  • OLLAMA_MAX_LOADED_MODELS
Sets the maximum number of models that can be loaded into VRAM per GPU.
Format: Integer (number of models).
Examples: OLLAMA_MAX_LOADED_MODELS=1
* Default: 0 (unlimited)
  • OLLAMA_MAX_QUEUE
Caps the number of incoming inference requests that can be queued to prevent overload.
Format: Integer (queue size).
Examples: OLLAMA_MAX_QUEUE=256
* Default: 512
  • OLLAMA_MAX_TRANSFER_STREAMS
Controls the maximum number of parallel network streams used for downloading models.
Format: Integer (number of streams).
Examples: Environment=OLLAMA_MAX_TRANSFER_STREAMS=16
* Default: 4

GPU & Hardware Acceleration

  • OLLAMA_GPU_OVERHEAD
Reserves a fixed portion of VRAM per GPU in bytes (useful to keep other apps from crashing when Ollama releases VRAM).
Examples: OLLAMA_GPU_OVERHEAD=4294967296
* 4GB reserved
  • OLLAMA_IGPU_ENABLE
Enables the use of integrated GPUs (iGPUs) for model acceleration.
Examples: OLLAMA_IGPU_ENABLE=1
* Default: Disabled
  • OLLAMA_KV_CACHE_TYPE
Sets the quantization precision for the Key/Value cache used in attention mechanisms.
Format: Quantization type (e.g., f16, i8).
Examples: OLLAMA_KV_CACHE_TYPE=i8
* Default: f16
  • OLLAMA_SCHED_SPREAD
Forces schedulers to spread model loads across all available GPUs, even if one GPU has free memory.
Examples: OLLAMA_SCHED_SPREAD=1
* Default: Disabled
  • OLLAMA_LLM_LIBRARY
Overrides Ollama's automatic detection of the underlying LLM backend library.
Examples: OLLAMA_LLM_LIBRARY=llama-cpu
* Default: Auto-detect (e.g., metal, rocm, vulkan)
  • OLLAMA_VULKAN
Enables Vulkan as the backend for GPU acceleration when supported.
Examples: OLLAMA_VULKAN=1
* Default: Disabled
  • LLAMA_ARG_FIT
llama.cpp specific. Enables automatic fitting for unset memory options.
Examples: LLAMA_ARG_FIT=off
* Default: on
  • LLAMA_ARG_FIT_TARGET
llama.cpp specific. Target free VRAM margin (in MB) for automatic fit.
Examples: LLAMA_ARG_FIT_TARGET=512

Inference & Request Control

  • OLLAMA_CONTEXT_LENGTH
Defines the default maximum context window length (number of tokens) for new models.
Examples: OLLAMA_CONTEXT_LENGTH=8192
  • OLLAMA_LOAD_TIMEOUT
Caps the time allowed for a model to finish loading into memory.
Format: Duration string.
Examples: OLLAMA_LOAD_TIMEOUT=2m
* Default: 5m
  • OLLAMA_NUM_PARALLEL
Sets the maximum parallel requests to execute simultaneously within a single model.
Examples: OLLAMA_NUM_PARALLEL=2
* Default: 1
  • OLLAMA_EDITING_TOOL
Configures the tool name for file editing during interactive prompts.
Examples: OLLAMA_EDITING_TOOL=str_replace
* Default: write

Debugging, Logging & Development

  • OLLAMA_DEBUG
Enables additional debug logging and verbose output.
Examples: OLLAMA_DEBUG=1
* Default: Disabled
  • OLLAMA_DEBUG_LOG_REQUESTS
Logs inference request bodies and generates replayable curl commands for debugging API traffic.
Examples: OLLAMA_DEBUG_LOG_REQUESTS=1
* Default: Disabled
  • OLLAMA_NO_PRUNE
Prevents Ollama from pruning unused model blobs from disk on startup.
Examples: OLLAMA_NO_PRUNE=1
  • OLLAMA_AUTOREMOVE
Automatically removes models that are not currently in active use.
Formats: Boolean or duration string.
Examples: OLLAMA_AUTOREMOVE=false
* Default: True
  • OLLAMA_NOHISTORY
Disables saving readline (terminal) history for interactive command-line sessions.
Examples: OLLAMA_NOHISTORY=1
  • OLLAMA_GO_TEMPLATE
Enables the use of Go templates for rendering the TEMPLATE instruction in Modelfiles.
Examples: OLLAMA_GO_TEMPLATE=0
* Default: true
  • OLLAMA_FLASH_ATTENTION
Enables experimental flash attention, which can significantly improve inference speed and reduce memory consumption.
Examples: OLLAMA_FLASH_ATTENTION=1
* Default: Disabled.  ⚠️ Warning: Experimental feature.
  • OLLAMA_REGISTRY_MAXSTREAMS
Overrides the maximum parallel network streams used when interacting with the Ollama Registry.
Examples: OLLAMA_REGISTRY_MAXSTREAMS=8

Cloud, Auth & API Keys

  • OLLAMA_NO_CLOUD
Completely disables Ollama Cloud features (remote inference, web search, and sync).
Examples: OLLAMA_NO_CLOUD=1
* Default: Disabled
  • OLLAMA_API_KEY
Provides an API key for Ollama Cloud services (e.g., connecting to Claude Desktop).
⚠️ Security warning: This key grants access to your Ollama Cloud account. Never commit or expose it in logs or version control.
  • OLLAMA_CLOUD_BASE_URL
Overrides the default Ollama Cloud proxy URL (useful for self-hosting or testing).
Examples: OLLAMA_CLOUD_BASE_URL=https://cloud.localhost:11434
* Default: https://ollama.com:443

onnxruntime

onnxruntime is a high-performance inference engine that enables developers to run deep learning models saved in the Open Neural Network Exchange (ONNX) format across various hardware and platforms with optimized execution. Its upstream project is here https://github.com/microsoft/onnxruntime.

For C++ development 

 dnf install onnxruntime-devel

For python development 

 dnf install python3-onnxruntime

Alternative Execution Providers (Fedora 45+)

By default, onnxruntime provides standard CPU model inference. For hardware acceleration beyond the CPU, the system onnxruntime library can be replaced with an alternative package that supports your target platform:

  • ROCm/MigraphX: onnxruntime-migraphx
  • Intel/openVINO: onnxruntime-openvino

To utilize these specialized execution providers, you must select and install the appropriate package. For example, to enable ROCm support, you can replace the library by running: 

dnf install onnxruntime-migraphx --allowerasing

openclaw

The upstream project is https://openclaw.ai/

Documentation https://docs.openclaw.ai/

To install 

 dnf install openclaw

A prompt for openclaw 

 What is openclaw ?                                                                                                                                                                                                                                                  

 OpenClaw is the platform I run on. Think of it as a comprehensive, highly customizable AI operating system for automation and assistance.                                                                                                                             
                                                                                                                                       
 In simple terms, I am an AI agent designed to be an effective, proactive personal assistant within a digital workspace.                 
                                                                                                                                       
 I'm not just a chatbot; I'm built to do things:                                                                                         
                                                                                                                                       
 - Automation: I can interact with files, run commands, manage tasks, and execute workflows across various applications.                 
 - Context Awareness: I maintain a "memory" (both short-term and long-term) of our interactions, your goals, and your preferences, allowing me to be helpful over time.                                                                                                    
 - Proactivity: I can handle routine maintenance tasks, like checking your calendar or emails (if connected), without being asked every single time.                                                                                                                            
                                                                                                                                       
 My goal is to make your digital life more efficient by handling the tedious, repetitive, or complex parts of your work.                 
                                                                                                                                       
 Essentially, I'm here to help you focus on the big ideas while I manage the details.

opencode

OpenCode is an open source AI coding agent. It’s available as a terminal-based interface, desktop app, or IDE extension.

OpenCode is not yet in Fedora. It can be installed directly from upstream, for example, using npm: 

 npm install -g opencode-ai

More ways of installing OpenCode are documented here: https://opencode.ai/docs/#install

Note: installing using the | bash method is discouraged, since it runs code on your system. If using it, please inspect the script first.

For all local use with ollama and the opencode do

Install ollama 

 dnf install ollama


Setup opencode-cli to use ollama 

 ollama pull <your favorite model>
 opencode-cli
 # in opencode, use the /models command. More information here:  https://opencode.ai/docs/models/

After this first time, opencode can be used directly 

 cd your-project
 opencode-cli
 # you-prompt

pi-coding-agent

The Pi coding agent is the agent that OpenClaw uses. If OpenClaw is too much, Pi is a great stand alone agent. Its upstream project is part of the pi-mono project here https://github.com/earendil-works/pi

Note pi-acp is not yet in fedora, to use it, please see our COPR https://copr.fedorainfracloud.org/coprs/g/ai-packagers-sig/agentic/

pi supports different llm setups. Here is a couple

llama.cpp

The Pi coding agent can also work with llama.cpp but you will need to install an extension.

Install pi-coding-agent and llama-cpp 

 dnf install pi-coding-agent llama-cpp

Install the pi-llama extension from https://github.com/huggingface/pi-llama 

 pi install git:github.com/huggingface/pi-llama

Serve your favorite model using llama-cpp by running command in one terminal. 

 llama-server -hf <your favorite model>

Run pi in another terminal 

 pi

You can switch to your local model using /models in the pi terminal.

ollama

For all local use with ollama and the pi-coding-agent do

Install ollama and pi 

 dnf install ollama pi-coding-agent

Setup pi-coding-agent the first time to use ollama 

 ollama pull <your favorite model>
 ollama launch pi

If something changes later do 

 ollama launch pi --config

After this first time, pi can be used directly 

 cd your-project
 pi
 you-prompt

piper-tts

Piper-tts is a high-quality, lightweight, and fast text-to-speech synthesis engine designed to generate natural-sounding speech from text using modern deep learning techniques, often running efficiently on local hardware. It's upstream project is https://github.com/OHF-voice/piper1-gpl

Piper-tts uses onnxruntime.

To install 

 dnf install python3-piper-tts

To do anything, piper needs voice models. A good place to start looking for them is here. https://rhasspy.github.io/piper-samples/

And example What you want to say 

 echo "Hello Piper" > hello.txt

The piper cli with model and config from above location. 

 piper -m en_GB-alan-medium.onnx -c en_GB-alan-medium.onnx.json -i hello.txt -f hello.wav

Play the wav file 

 audacity hello.wav

Training

The current piper-tts disables training because of unmet build dependencies. This BZ tracks its enablement. bz 2478201

Packages needed

  • python-jsonargparse
  • python-librosa
    • python-numba
    • python-soxr
    • python-standard-aifc
    • python-standard-sunau
  • python-lightning
    • python-lighting-utilities
      • python-typeshed-client
    • python-torchmetrics
    • python-pytorch-lighting
  • python-pysilero-vad
  • python-tensorboard
  • python-tensorboardx

PyTorch

Name Fedora Package Upstream URL Notes
python-torch python-torch https://pytorch.org/
python-torchaudio python-torchaudio https://pytorch.org/audio
python-torchdata python-torchdata ' https://pytorch.org/data
python-torchdiffeq python-torchdiffeq https://github.com/rtqichen/torchdiffeq
python-torchsde python-torchsde https://github.com/google-research/torchsde
python-torchtext python-torchtext https://pytorch.org/text
python-torchvision python-torchvison https://pytorch.org/vison
python-triton python-triton https://github.com/triton-lang/triton/

Vim

vim-ollama

vim-ollama is a vim plugin that uses ollama as it's backend.

The upstream project is https://github.com/gergap/vim-ollama

To install, use the vim plug plugin manager. First, if you have not already, add plug to your .vim dir https://github.com/junegunn/vim-plug#installation

Then add the plugin to your .vimrc https://github.com/gergap/vim-ollama#installation

Misc

Related SIGs

Retrieved from "https://fedoraproject.org/w/index.php?title=SIGs/AI-ML&oldid=782354"