Simple reference page for some AI stuff…

Sizing

While everyone is looking at VRAM sizing, including

• size of model when loaded (with or without some external “mmproj” file for VL models without in-built projection files)
• context length
• KV cache (whether quantised or not)
• temporal memory

I often forget to sync the front-end and inference engine settings, resulting in truncated results and such.

AnythingLLM

• have to remember to set the LLM, Embedder and Text Splitter & Chunking settings

Formatting and <think> Tags

• with the myriad of models and front-ends, the two do not always align – try the following, which was needed from AnythingLLM + llama.cpp using Qwen3.6 27B model:
  • using system prompts, assuming the model is a “thinking” model:

  You are a helpful AI assistant. ALWAYS:
  1. Begin responses with <think>.
  2. Output internal reasoning process within <think> tags.
  3. CRITICAL: Explicitly close the thinking block with </think> before answering.
  4. After closing the tag, immediately provide the final answer .
  5. DO NOT output "The response is ready" or meta-commentary; just output the answer.

  • forcing inference engine with some “chat template” (if supported):
    • llama.cpp --chat-template chatml

Git RAG

• When attempting to RAG from Git, ensure to exclude the following:
  • *.exe, *.dll, *.so, *.dylib, *.bin, *.out, *.app, *.o, *.obj, *.a, *.lib, *.pyc, *.pyd, *.class, *.jar, *.war, *.ear, *.png, *.jpg, *.jpeg, *.gif, *.ico, *.svg, *.webp, *.mp4, *.mp3, *.wav, *.woff, *.woff2, *.ttf, *.eot, *.po, *.pot, *.mo, *.min.js, *.min.css, *.map, *.test, *.pdb, *.ilk, *.gch, *.pch, *.suo, *.user, target/, debug/, release/, x64/, x86/, ipch/, bin/, vendor/, __pycache__/, .venv/, venv/, env/, .pytest_cache/, node_modules/, dist/, build/, out/, .next/, .nuxt/, package-lock.json, yarn.lock, pnpm-lock.yaml, .gradle/, .mvn/, obj/, ClientBin/, .DS_Store, Thumbs.db, .idea/, .vscode/