AI Application Development Mastery Part 18: Advanced Topics

1. Fine-Tuning vs Prompting

The single most important decision in advanced AI development is when to fine-tune and when to stick with prompting. Getting this wrong can waste months of engineering effort and thousands of dollars in compute costs — or, conversely, leave significant performance on the table.

1.1 When to Fine-Tune

Fine-tuning is the process of taking a pre-trained foundation model and continuing its training on a smaller, task-specific dataset. The model's weights are updated to specialize in your domain. Here is the decision matrix:

1.2 Cost-Benefit Analysis

Before committing to fine-tuning, calculate the total cost of ownership:

# Cost comparison framework
# No external dependencies required

def compare_approaches(
    daily_queries: int,
    prompt_tokens_per_query: int,
    fine_tuned_tokens_per_query: int,
    days: int = 30
):
    """Compare prompting vs fine-tuning costs over time."""

    # Prompting approach: longer prompts (system prompt + few-shot examples)
    prompt_cost_per_1k = 0.03  # GPT-4 input tokens
    monthly_prompt_cost = (
        daily_queries * prompt_tokens_per_query / 1000
        * prompt_cost_per_1k * days
    )

    # Fine-tuned approach: shorter prompts (model already knows the task)
    ft_cost_per_1k = 0.012  # Fine-tuned GPT-3.5 input tokens
    training_cost = 500  # One-time training cost
    monthly_ft_cost = (
        daily_queries * fine_tuned_tokens_per_query / 1000
        * ft_cost_per_1k * days
    )

    # Break-even analysis
    monthly_savings = monthly_prompt_cost - monthly_ft_cost
    if monthly_savings > 0:
        breakeven_months = training_cost / monthly_savings
        print(f"Break-even in {breakeven_months:.1f} months")
    else:
        print("Fine-tuning is MORE expensive - stick with prompting")

    return {
        "monthly_prompt_cost": monthly_prompt_cost,
        "monthly_ft_cost": monthly_ft_cost,
        "training_cost": training_cost,
        "monthly_savings": monthly_savings
    }

# Example: 10K queries/day, 2000 tokens prompted vs 500 fine-tuned
result = compare_approaches(
    daily_queries=10000,
    prompt_tokens_per_query=2000,
    fine_tuned_tokens_per_query=500
)

1.3 Decision Framework

The industry trend is clear: prompting and RAG handle 80-90% of use cases. Fine-tuning is reserved for the remaining 10-20% where task-specific behavior, cost optimization at scale, or edge deployment demand a customized model.

2. LoRA & QLoRA

Low-Rank Adaptation (LoRA) is the breakthrough that democratized fine-tuning. Instead of updating all model parameters (which requires massive GPU memory), LoRA freezes the original weights and trains small, low-rank matrices that are injected into specific layers. This reduces trainable parameters by 99%+ while achieving comparable performance to full fine-tuning.

2.1 Low-Rank Adaptation Theory

The core insight of LoRA is that weight updates during fine-tuning have a low intrinsic rank. Instead of updating a weight matrix W (d x d), we decompose the update into two smaller matrices: A (d x r) and B (r x d), where r is much smaller than d (typically 4-64).

# LoRA conceptual implementation
# pip install torch

import torch
import torch.nn as nn

class LoRALayer(nn.Module):
    """Low-Rank Adaptation layer."""

    def __init__(self, in_features, out_features, rank=8, alpha=16):
        super().__init__()
        self.rank = rank
        self.alpha = alpha
        self.scaling = alpha / rank

        # Original frozen weight
        self.weight = nn.Parameter(
            torch.randn(out_features, in_features),
            requires_grad=False  # Frozen!
        )

        # LoRA trainable matrices
        self.lora_A = nn.Parameter(torch.randn(rank, in_features))
        self.lora_B = nn.Parameter(torch.zeros(out_features, rank))

        # Initialize A with Kaiming, B with zeros
        nn.init.kaiming_uniform_(self.lora_A)
        nn.init.zeros_(self.lora_B)

    def forward(self, x):
        # Original path (frozen)
        original = x @ self.weight.T

        # LoRA path (trainable) - low-rank update
        lora_update = x @ self.lora_A.T @ self.lora_B.T * self.scaling

        return original + lora_update

# Memory comparison for Llama 2 7B:
# Full fine-tuning: ~56 GB VRAM (needs 4x A100 80GB)
# LoRA (rank=8):    ~8 GB VRAM  (fits on single RTX 4090)
# QLoRA (rank=8):   ~5 GB VRAM  (fits on RTX 3090)

2.2 QLoRA: Quantized Fine-Tuning

QLoRA combines LoRA with 4-bit quantization (NormalFloat4 / NF4), allowing you to fine-tune a 65B parameter model on a single 48GB GPU. It introduces three innovations: NF4 data type, double quantization, and paged optimizers.

# QLoRA fine-tuning with Hugging Face + PEFT
# pip install transformers peft trl bitsandbytes torch accelerate

import torch
from transformers import (
    AutoModelForCausalLM, AutoTokenizer,
    BitsAndBytesConfig, TrainingArguments
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer

# 4-bit quantization config (QLoRA)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",           # NormalFloat4
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True        # Double quantization
)

# Load base model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

# Prepare model for k-bit training
model = prepare_model_for_kbit_training(model)

# LoRA configuration
lora_config = LoraConfig(
    r=16,                     # Rank
    lora_alpha=32,            # Scaling factor
    target_modules=[          # Which layers to adapt
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# Apply LoRA
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 18,874,368 || all params: 6,756,425,728 || 0.28%

# Training arguments
training_args = TrainingArguments(
    output_dir="./qlora-output",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    warmup_steps=100,
    logging_steps=10,
    save_strategy="epoch",
    fp16=True,
    optim="paged_adamw_32bit"  # Paged optimizer for memory
)

2.3 Practical LoRA Training

3. Instruction Tuning

Instruction tuning is the process of fine-tuning a base language model on a dataset of (instruction, response) pairs. This is what transforms a raw language model (which merely predicts next tokens) into an assistant that follows instructions. Models like ChatGPT, Claude, and Llama-Chat all went through instruction tuning.

3.1 Supervised Fine-Tuning Pipeline

Supervised fine-tuning (SFT) adapts a pre-trained model to follow instructions by training on curated input-output pairs. The standard format is Alpaca-style: each example has an instruction, optional input context, and the desired output. Combined with QLoRA (quantized low-rank adaptation), you can fine-tune models with billions of parameters on a single GPU by updating only a small fraction of the weights.

# Instruction tuning dataset format (Alpaca-style)
# pip install trl datasets transformers peft
# Requires: model, tokenizer, training_args, lora_config from QLoRA block above

training_examples = [
    {
        "instruction": "Summarize the following legal contract clause.",
        "input": "The Licensee shall not sublicense, transfer, or assign...",
        "output": "This clause prohibits the licensee from sharing, transferring..."
    },
    {
        "instruction": "Extract all medication names from this clinical note.",
        "input": "Patient prescribed metformin 500mg BID, lisinopril 10mg...",
        "output": "Medications: metformin 500mg, lisinopril 10mg, atorvastatin 20mg"
    }
]

# ChatML format (preferred for chat models)
chat_examples = [
    {
        "messages": [
            {"role": "system", "content": "You are a medical coding assistant."},
            {"role": "user", "content": "Code this diagnosis: Type 2 diabetes..."},
            {"role": "assistant", "content": "ICD-10: E11.9 - Type 2 diabetes..."}
        ]
    }
]

# SFT Training with TRL
from trl import SFTTrainer
from datasets import load_dataset

dataset = load_dataset("json", data_files="training_data.jsonl")

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset["train"],
    tokenizer=tokenizer,
    max_seq_length=2048,
    formatting_func=lambda example: format_chat(example),
    args=training_args,
    peft_config=lora_config  # Use LoRA for efficiency
)

trainer.train()

3.2 Dataset Preparation

The quality of your instruction-tuning dataset is the single largest factor determining fine-tuning success. Here are the key principles:

# Data quality pipeline
# No external dependencies required

import json
from typing import List, Dict

class DatasetValidator:
    """Validate and clean instruction-tuning datasets."""

    def __init__(self):
        self.issues = []
        self.all_examples = []  # Set via validate_dataset()
        self.write_count = 0

    def validate_dataset(self, examples: List[Dict]) -> List[Dict]:
        """Validate all examples and return only valid ones."""
        self.all_examples = examples
        self.write_count = 0
        return [ex for ex in examples if self.validate_example(ex)]

    def validate_example(self, example: Dict) -> bool:
        """Check a single training example for quality issues."""
        issues = []

        # Check instruction quality
        if len(example.get("instruction", "")) < 10:
            issues.append("Instruction too short")

        # Check response quality
        response = example.get("output", "")
        if len(response) < 20:
            issues.append("Response too short")
        if response.startswith("I "):
            issues.append("Response starts with 'I' - may be too conversational")

        # Check for contamination
        if "as an AI" in response.lower():
            issues.append("Contains 'as an AI' - likely generated, not curated")

        # Check for diversity
        if example.get("instruction", "").lower().startswith("write"):
            self.write_count = getattr(self, 'write_count', 0) + 1
            if self.write_count > len(self.all_examples) * 0.3:
                issues.append("Too many 'write' instructions - needs diversity")

        if issues:
            self.issues.append({"example": example, "issues": issues})
            return False
        return True

    def dedup_dataset(self, examples: List[Dict]) -> List[Dict]:
        """Remove near-duplicate examples."""
        seen_instructions = set()
        unique_examples = []

        for ex in examples:
            # Simple dedup by instruction prefix
            prefix = ex["instruction"][:100].lower().strip()
            if prefix not in seen_instructions:
                seen_instructions.add(prefix)
                unique_examples.append(ex)

        print(f"Removed {len(examples) - len(unique_examples)} duplicates")
        return unique_examples

4. RLHF & DPO

Instruction tuning teaches a model what to say. Alignment teaches it how to say it — safely, helpfully, and in accordance with human preferences. The two dominant approaches are Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO).

4.1 RLHF Pipeline

RLHF is the technique that made ChatGPT feel fundamentally different from GPT-3. The three-stage pipeline works as follows:

# RLHF with TRL (simplified)
# pip install trl transformers torch

from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead
from transformers import AutoTokenizer, AutoModelForSequenceClassification

# Step 1: Load SFT model (already instruction-tuned)
model = AutoModelForCausalLMWithValueHead.from_pretrained("my-sft-model")
tokenizer = AutoTokenizer.from_pretrained("my-sft-model")

# Step 2: Load reward model
reward_model = AutoModelForSequenceClassification.from_pretrained(
    "my-reward-model"
)

# Step 3: PPO Training
ppo_config = PPOConfig(
    learning_rate=1.41e-5,
    batch_size=16,
    mini_batch_size=4,
    gradient_accumulation_steps=1,
    optimize_cuda_cache=True,
    ppo_epochs=4,
    kl_penalty="kl",           # KL divergence penalty
    init_kl_coef=0.2,          # Prevents model from diverging too far
    target_kl=6.0,
    cliprange=0.2
)

ppo_trainer = PPOTrainer(
    model=model,
    config=ppo_config,
    tokenizer=tokenizer,
    dataset=prompt_dataset
)

# Training loop
for batch in ppo_trainer.dataloader:
    # Generate responses
    query_tensors = batch["input_ids"]
    response_tensors = ppo_trainer.generate(query_tensors)

    # Score responses with reward model
    rewards = reward_model(response_tensors)

    # Update policy with PPO
    stats = ppo_trainer.step(query_tensors, response_tensors, rewards)

4.2 DPO: Direct Preference Optimization

DPO eliminates the need for a separate reward model and the complexity of RL training. Instead, it directly optimizes the language model on preference pairs (chosen/rejected responses) using a simple classification-like loss function.

# DPO Training with TRL — dramatically simpler than RLHF
# pip install trl datasets transformers

from trl import DPOTrainer, DPOConfig
from datasets import load_dataset

# Preference dataset format
# Each example has: prompt, chosen (preferred), rejected (dispreferred)
preference_data = [
    {
        "prompt": "Explain quantum entanglement simply.",
        "chosen": "Quantum entanglement is when two particles become linked...",
        "rejected": "Quantum entanglement is a QM phenomenon described by..."
    }
]

# Load preference dataset
dataset = load_dataset("json", data_files="preferences.jsonl")

# DPO config
dpo_config = DPOConfig(
    beta=0.1,                   # Temperature for DPO loss
    learning_rate=5e-7,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    warmup_ratio=0.1,
    logging_steps=10,
    output_dir="./dpo-output",
    max_length=1024,
    max_prompt_length=512,
)

# DPO Trainer — no reward model needed!
dpo_trainer = DPOTrainer(
    model=model,
    ref_model=ref_model,        # Frozen copy of SFT model
    args=dpo_config,
    train_dataset=dataset["train"],
    tokenizer=tokenizer,
)

dpo_trainer.train()

4.3 RLHF vs DPO Comparison

5. Tool Learning

While prompting can teach models to use tools at inference time (as we covered in Parts 7-8), tool learning trains the model itself to understand when and how to invoke external tools. This produces more reliable tool use with lower latency and fewer tokens.

5.1 Teaching Models to Use Tools

Tool-use training teaches models when to call external tools and how to format the call correctly. Training data uses a structured message format where the assistant response includes a tool_calls field with the function name and arguments, followed by a tool response message containing the result. This teaches the model the complete tool invocation lifecycle.

# Tool-use training data format
# Training data showing the model when and how to call tools

tool_training_examples = [
    {
        "messages": [
            {"role": "system", "content": "You have access to these tools:\n"
                "- search(query: str) -> str: Search the web\n"
                "- calculate(expression: str) -> float: Evaluate math\n"
                "- get_weather(city: str) -> dict: Get current weather"},
            {"role": "user", "content": "What's 15% tip on a $67.50 bill?"},
            {"role": "assistant", "content": None, "tool_calls": [
                {"name": "calculate", "arguments": {"expression": "67.50 * 0.15"}}
            ]},
            {"role": "tool", "content": "10.125", "name": "calculate"},
            {"role": "assistant", "content": "A 15% tip on $67.50 would be $10.13."}
        ]
    },
    {
        "messages": [
            {"role": "system", "content": "You have access to these tools:..."},
            {"role": "user", "content": "Who won the latest Nobel Prize in Physics?"},
            {"role": "assistant", "content": None, "tool_calls": [
                {"name": "search", "arguments": {"query": "Nobel Prize Physics 2025 winner"}}
            ]},
            {"role": "tool", "content": "The 2025 Nobel Prize in Physics was awarded to...", "name": "search"},
            {"role": "assistant", "content": "The 2025 Nobel Prize in Physics was awarded to..."}
        ]
    }
]

# Key insight: train on examples where the model CORRECTLY decides
# NOT to use a tool when it already knows the answer
no_tool_examples = [
    {
        "messages": [
            {"role": "system", "content": "You have access to these tools:..."},
            {"role": "user", "content": "What is the capital of France?"},
            {"role": "assistant", "content": "The capital of France is Paris."}
            # No tool call — the model should know this directly
        ]
    }
]

5.2 Toolformer & Gorilla Approach

Two pioneering approaches to tool learning emerged from research labs:

# Gorilla-style API-aware fine-tuning data
# Training data format pairing API docs with correct usage code

api_training_data = {
    "instruction": "Use the HuggingFace API to perform sentiment analysis",
    "api_documentation": """
    transformers.pipeline(task, model=None, tokenizer=None, ...)
    Parameters:
        task (str): "sentiment-analysis", "text-generation", etc.
        model (str): Model identifier from HuggingFace Hub
    Returns: Pipeline object for inference
    """,
    "output": """
from transformers import pipeline

classifier = pipeline(
    task="sentiment-analysis",
    model="distilbert-base-uncased-finetuned-sst-2-english"
)
result = classifier("This product is amazing!")
print(result)  # [{'label': 'POSITIVE', 'score': 0.9998}]
"""
}

6. Hybrid LLM Systems

Pure LLM approaches have well-known limitations: they hallucinate, struggle with precise computation, cannot guarantee logical consistency, and lack verifiable reasoning chains. Hybrid systems combine the natural language understanding of LLMs with the precision of traditional computing paradigms.

6.1 LLM + Symbolic Reasoning

Hybrid neuro-symbolic systems combine an LLM’s natural language understanding with a symbolic math engine’s exact computation. The LLM extracts mathematical relationships from natural language and constructs sympy expressions, which the symbolic solver then evaluates with guaranteed correctness — eliminating the hallucination risk inherent in asking an LLM to do arithmetic directly.

# Hybrid LLM + Symbolic Reasoning System
# pip install sympy langchain-openai

from sympy import symbols, solve, simplify, latex
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate

# Allowed sympy functions for safe exec
sympy_imports = {"symbols": symbols, "solve": solve, "simplify": simplify}

class HybridMathSolver:
    """Combines LLM natural language understanding with
    SymPy symbolic math for provably correct solutions."""

    def __init__(self):
        self.llm = ChatOpenAI(model="gpt-4", temperature=0)
        self.translate_prompt = ChatPromptTemplate.from_messages([
            ("system", "Convert the following math word problem into a "
             "SymPy expression. Return ONLY valid Python code using sympy. "
             "Define variables with symbols() and create an equation."),
            ("human", "{problem}")
        ])

    def solve(self, problem: str) -> dict:
        # Step 1: LLM translates natural language to symbolic form
        chain = self.translate_prompt | self.llm
        sympy_code = chain.invoke({"problem": problem}).content

        # Step 2: Execute symbolic computation (exact, no hallucination)
        local_vars = {}
        exec(sympy_code, {"__builtins__": {}, **sympy_imports}, local_vars)

        # Step 3: LLM explains the solution in natural language
        explanation = self.llm.invoke(
            f"Explain this math solution step by step: {local_vars['solution']}"
        ).content

        return {
            "symbolic_code": sympy_code,
            "exact_solution": local_vars.get("solution"),
            "explanation": explanation
        }

# Usage
solver = HybridMathSolver()
result = solver.solve(
    "A train leaves Station A at 60 km/h. Another train leaves "
    "Station B (300 km away) at 90 km/h heading towards Station A. "
    "When and where do they meet?"
)

6.2 LLM + Classical ML

Many real-world applications benefit from combining LLMs with classical machine learning models. The LLM handles natural language understanding and generation, while classical ML provides fast, interpretable predictions.

# Hybrid: LLM for feature extraction + Classical ML for prediction
# pip install langchain-openai pandas scikit-learn

import json
import pandas as pd
from sklearn.ensemble import GradientBoostingClassifier
from langchain_openai import ChatOpenAI

class HybridClassifier:
    """LLM extracts semantic features, classical ML makes predictions."""

    def __init__(self):
        self.llm = ChatOpenAI(model="gpt-4", temperature=0)
        self.classifier = GradientBoostingClassifier()

    def extract_features(self, text: str) -> dict:
        """Use LLM to extract structured features from text."""
        response = self.llm.invoke(
            f"Extract these features as JSON from the text:\n"
            f"- sentiment_score (-1 to 1)\n"
            f"- urgency_level (0-10)\n"
            f"- topic_category (billing/technical/general)\n"
            f"- customer_frustration (0-10)\n"
            f"- requires_escalation (true/false)\n\n"
            f"Text: {text}"
        )
        return json.loads(response.content)

    def train(self, texts: list, labels: list):
        """Extract features with LLM, train classical ML model."""
        features = [self.extract_features(t) for t in texts]
        X = pd.DataFrame(features)
        self.classifier.fit(X, labels)

    def predict(self, text: str) -> dict:
        """Predict with feature importances (interpretable!)."""
        features = self.extract_features(text)
        X = pd.DataFrame([features])
        prediction = self.classifier.predict(X)[0]
        probabilities = self.classifier.predict_proba(X)[0]

        return {
            "prediction": prediction,
            "confidence": max(probabilities),
            "features": features,  # Interpretable!
            "feature_importances": dict(zip(
                X.columns, self.classifier.feature_importances_
            ))
        }

6.3 LLM + Knowledge Graphs

Knowledge graphs provide structured, verified facts with explicit relationships. Combining them with LLMs creates systems that can reason over structured knowledge while maintaining the natural language interface.

# LLM + Knowledge Graph (Neo4j) for verified reasoning
# pip install neo4j langchain-openai langchain-community

from neo4j import GraphDatabase
from langchain_openai import ChatOpenAI
from langchain_community.graphs import Neo4jGraph

class KnowledgeGraphQA:
    """Answer questions using verified knowledge graph facts."""

    def __init__(self, neo4j_uri, neo4j_user, neo4j_password):
        self.graph = Neo4jGraph(
            url=neo4j_uri, username=neo4j_user, password=neo4j_password
        )
        self.llm = ChatOpenAI(model="gpt-4", temperature=0)

    def answer(self, question: str) -> dict:
        # Step 1: LLM generates Cypher query from natural language
        cypher = self.llm.invoke(
            f"Convert this question to a Neo4j Cypher query.\n"
            f"Schema: {self.graph.get_schema()}\n"
            f"Question: {question}\n"
            f"Return ONLY the Cypher query."
        ).content

        # Step 2: Execute against knowledge graph (verified facts)
        results = self.graph.query(cypher)

        # Step 3: LLM synthesizes natural language answer
        answer = self.llm.invoke(
            f"Based on these verified facts from our knowledge graph:\n"
            f"{results}\n\n"
            f"Answer the question: {question}\n"
            f"Only use the provided facts. If the facts don't cover "
            f"the question, say so explicitly."
        ).content

        return {
            "answer": answer,
            "cypher_query": cypher,
            "graph_results": results,
            "source": "knowledge_graph",  # Provenance tracking
            "verified": True
        }

7. Model Distillation & Quantization

Production deployment often requires models that are smaller, faster, and cheaper than the original. Distillation transfers knowledge from a large "teacher" model to a smaller "student" model. Quantization reduces the numerical precision of model weights from 16-bit to 8-bit or 4-bit, dramatically shrinking model size and increasing inference speed.

7.1 Model Distillation

Knowledge distillation trains a smaller "student" model to replicate a larger "teacher" model’s behavior. The student learns from soft labels (the teacher’s probability distribution over tokens) rather than hard labels, capturing richer information about the teacher’s uncertainty and reasoning. A temperature parameter controls how much of the teacher’s probability distribution is transferred.

# Knowledge distillation: GPT-4 teacher -> Llama 7B student
# pip install transformers torch

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
import torch.nn.functional as F

class DistillationTrainer:
    """Train a small student model to mimic a large teacher."""

    def __init__(self, teacher_model, student_model, temperature=3.0, alpha=0.5):
        self.teacher = teacher_model
        self.student = student_model
        self.temperature = temperature
        self.alpha = alpha  # Balance between distillation and task loss

    def distillation_loss(self, student_logits, teacher_logits, labels):
        """Combine soft target loss (from teacher) with hard target loss."""

        # Soft targets: teacher's probability distribution (smoothed)
        soft_teacher = F.softmax(teacher_logits / self.temperature, dim=-1)
        soft_student = F.log_softmax(student_logits / self.temperature, dim=-1)
        distill_loss = F.kl_div(
            soft_student, soft_teacher, reduction="batchmean"
        ) * (self.temperature ** 2)

        # Hard targets: standard cross-entropy with ground truth
        hard_loss = F.cross_entropy(student_logits, labels)

        # Combined loss
        return self.alpha * distill_loss + (1 - self.alpha) * hard_loss

# Practical distillation pipeline
# Step 1: Generate teacher outputs for your dataset
def generate_teacher_data(teacher, dataset, batch_size=32):
    """Run teacher model on dataset, save logits and responses."""
    teacher_outputs = []
    for batch in dataset.batch(batch_size):
        with torch.no_grad():
            outputs = teacher(batch["input_ids"])
            teacher_outputs.append({
                "input_ids": batch["input_ids"],
                "teacher_logits": outputs.logits,
                "teacher_text": teacher.generate(batch["input_ids"])
            })
    return teacher_outputs

# Step 2: Train student on teacher outputs
# This is often done using the teacher's TEXT outputs
# (response-based distillation) which is simpler and works well

7.2 Quantization: INT8, INT4, GPTQ, AWQ, GGUF

# GPTQ Quantization with AutoGPTQ
# pip install auto-gptq autoawq transformers

from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

# Quantize a model to 4-bit GPTQ
quantize_config = BaseQuantizeConfig(
    bits=4,                    # 4-bit quantization
    group_size=128,            # Quantize in groups of 128 weights
    desc_act=True,             # Activation-order quantization
    damp_percent=0.1
)

# Load model in full precision
model = AutoGPTQForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantize_config=quantize_config
)

# Calibrate on representative data (128-256 examples is typical)
model.quantize(calibration_dataset)

# Save quantized model — now ~3.5 GB instead of ~14 GB
model.save_quantized("./llama-2-7b-gptq-4bit")

# AWQ Quantization with AutoAWQ
from awq import AutoAWQForCausalLM

model = AutoAWQForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

# AWQ preserves "salient" weights at higher precision
model.quantize(
    tokenizer,
    quant_config={
        "zero_point": True,
        "q_group_size": 128,
        "w_bit": 4,
        "version": "GEMM"     # Optimized GEMM kernels
    }
)

model.save_quantized("./llama-2-7b-awq-4bit")

# GGUF Conversion for llama.cpp
# Convert HuggingFace model to GGUF format
python convert_hf_to_gguf.py \
    ./meta-llama/Llama-2-7b-hf \
    --outfile llama-2-7b.gguf \
    --outtype f16

# Quantize GGUF to various levels
./quantize llama-2-7b.gguf llama-2-7b-Q4_K_M.gguf Q4_K_M
./quantize llama-2-7b.gguf llama-2-7b-Q5_K_M.gguf Q5_K_M
./quantize llama-2-7b.gguf llama-2-7b-Q8_0.gguf Q8_0

# GGUF quantization levels explained:
# Q2_K  - 2-bit, extreme compression, significant quality loss
# Q4_0  - 4-bit, basic quantization
# Q4_K_M - 4-bit, medium quality (best balance for most uses)
# Q5_K_M - 5-bit, good quality, slightly larger
# Q8_0  - 8-bit, near-original quality

8. Edge AI

Running AI models locally — on laptops, servers without internet, or edge devices — is increasingly important for privacy, latency, cost, and reliability. Two tools have made local LLM deployment practical: Ollama and llama.cpp.

8.1 Ollama: Local LLM Server

Ollama provides a Docker-like experience for running LLMs locally. It handles model download, quantization, and serves models via an OpenAI-compatible API.

# Install and run Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull and run models
ollama pull llama3.1:8b          # Llama 3.1 8B (Q4_K_M by default)
ollama pull codellama:13b        # Code-specialized model
ollama pull mistral:7b           # Mistral 7B

# Run interactive chat
ollama run llama3.1:8b

# Ollama serves an OpenAI-compatible API on localhost:11434
# Use it as a drop-in replacement in your apps

# Using Ollama with LangChain — zero code changes needed
# pip install langchain-community langchain-core

from langchain_community.llms import Ollama
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

# Local model — no API key, no internet, no cost per token
llm = Ollama(model="llama3.1:8b", temperature=0.7)

# Works identically to cloud APIs
prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful coding assistant."),
    ("human", "{question}")
])

chain = prompt | llm | StrOutputParser()

# All inference happens locally
result = chain.invoke({"question": "Write a Python quicksort"})

# Ollama also supports embeddings for local RAG
from langchain_community.embeddings import OllamaEmbeddings

embeddings = OllamaEmbeddings(model="nomic-embed-text")
vector = embeddings.embed_query("semantic search locally")

8.2 llama.cpp: Bare-Metal Inference

llama.cpp is a C/C++ inference engine that runs GGUF models with extreme efficiency. It supports CPU inference (AVX2/AVX512), Metal (Apple Silicon), CUDA, and Vulkan acceleration, making it the universal inference backend.

# Build llama.cpp with CUDA support
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make LLAMA_CUDA=1

# Run inference
./main \
    -m models/llama-2-7b-Q4_K_M.gguf \
    -n 512 \
    -p "Explain the concept of retrieval-augmented generation:" \
    --temp 0.7 \
    --top-p 0.9 \
    --threads 8 \
    --n-gpu-layers 35  # Offload layers to GPU

# Start a server (OpenAI-compatible)
./server \
    -m models/llama-2-7b-Q4_K_M.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    --n-gpu-layers 35 \
    --ctx-size 4096

# Performance benchmarks (Llama 2 7B Q4_K_M):
# Apple M2 Pro (CPU):  ~25 tokens/sec
# RTX 4090 (GPU):      ~120 tokens/sec
# RTX 3060 12GB (GPU): ~45 tokens/sec
# Intel i9-13900K (CPU): ~15 tokens/sec

8.3 Edge Deployment Patterns

# Hybrid Edge-Cloud Architecture
# pip install langchain-community langchain-openai
# Requires: Ollama running locally, OPENAI_API_KEY set in environment

from langchain_community.llms import Ollama
from langchain_openai import ChatOpenAI

class HybridInference:
    """Route queries to local or cloud model based on complexity."""

    def __init__(self):
        self.local_model = Ollama(model="llama3.1:8b")
        self.cloud_model = ChatOpenAI(model="gpt-4")
        self.router = Ollama(model="llama3.1:8b")

    def route(self, query: str) -> str:
        """Determine if query needs cloud model."""
        routing = self.router.invoke(
            f"Classify this query as SIMPLE or COMPLEX.\n"
            f"SIMPLE: factual, short answer, basic tasks\n"
            f"COMPLEX: multi-step reasoning, creative, nuanced\n"
            f"Query: {query}\nClassification:"
        )
        return "cloud" if "COMPLEX" in routing.upper() else "local"

    def invoke(self, query: str) -> dict:
        destination = self.route(query)
        if destination == "local":
            response = self.local_model.invoke(query)
            return {"response": response, "source": "local", "cost": 0.0}
        else:
            response = self.cloud_model.invoke(query).content
            return {"response": response, "source": "cloud", "cost": 0.03}

Exercises & Self-Assessment

Conclusion & Next Steps

You now have a comprehensive toolkit of advanced techniques that extend far beyond prompting and API calls. Here are the key takeaways from Part 18:

• Fine-tuning vs prompting — Prompting and RAG handle 80-90% of use cases; fine-tune only when you need consistent format adherence, domain-specific behavior, or cost optimization at scale
• LoRA and QLoRA — Fine-tune billion-parameter models on a single consumer GPU by training only 0.1-0.5% of parameters, with minimal quality loss compared to full fine-tuning
• RLHF and DPO — Align models with human preferences; DPO is simpler and sufficient for most projects, while RLHF offers higher ceilings for frontier model alignment
• Tool learning — Training models to natively invoke tools produces more reliable tool use than pure prompt-based approaches
• Hybrid systems — Combining LLMs with symbolic reasoning, classical ML, and knowledge graphs creates systems that are both intelligent and verifiably correct
• Distillation and quantization — Shrink models by 4-8x with GPTQ, AWQ, or GGUF while retaining 95%+ of original capability
• Edge AI — Ollama and llama.cpp make local deployment practical for privacy-sensitive, latency-critical, and offline applications