How to Build Better Agent Memory with Graph Expansion and Deterministic Ranking

An AI agent can have plenty of stored information and still feel forgetful. The failure is often retrieval: a search returns one matching memory, while the related skill, configuration note, or operating rule remains hidden in another record.

A practical way to improve AI agent memory retrieval is to add a small deterministic layer after the first search. It fits the broader AI agent tools cluster, where memory is one part of a larger autonomous workflow. Expand the top results to nearby memories, lower the rank of low-quality records, and give more weight to sources that were deliberately written or reviewed. These changes can make an existing hybrid search system more useful without adding another model call to every query.

This guide explains a Hermes-style implementation inspired by a personal review of RepoWise. The design uses three ideas: one-hop graph expansion, deterministic memory health scoring, and source ranking. For a related view of how agents behave during long autonomous runs, see the Codex agent workflow guide. The reported implementation was about 220 lines of Python. Its claimed 30% retrieval-coverage improvement is an author measurement, not an independent benchmark.

Why direct retrieval feels isolated

A memory system usually starts with a ranking function. It combines semantic similarity, lexical matching, tags, recency, or confidence, then returns the top few records. That is a sensible first pass. It is also incomplete.

A memory about a “headroom skill” may be directly related to a “caveman skill” through a shared tag. An instruction about sending a message may belong beside the configuration that controls the messaging tool. A rule may be useful because it shares a category with the result, even when the exact words are different.

If the system returns only direct matches, the user has to run several searches and mentally join the results. The search engine has already found a useful starting point. It should be able to inspect the immediate neighborhood before it stops.

1. Add a one-hop memory graph

The simplest graph does not require a graph database. Treat each memory as a node and create an implicit edge when two records share meaningful metadata or content.

A one-hop expansion can score each candidate with three signals:

• shared tags, with the strongest weight when at least two tags overlap;
• the same category, with a lower weight;
• overlapping content terms, after tokenization, with a lower weight again.

In pseudocode:

def find_related_memories(entry, all_entries, max_results=5):
    entry_tags = set(entry.get("tags", "").split(","))
    entry_category = entry.get("category", "")
    entry_words = tokenize(entry.get("content", ""))
    scored = []

    for other in all_entries:
        if other is entry:
            continue
        score = 0.0
        other_tags = set(other.get("tags", "").split(","))
        if len(entry_tags & other_tags) >= 2:
            score += 1.0
        if entry_category == other.get("category", ""):
            score += 0.7
        if len(entry_words & tokenize(other.get("content", ""))) >= 3:
            score += 0.5
        if score > 0:
            scored.append((score, other))

    return sorted(scored, reverse=True)[:max_results]

The important property is locality. The system does not attempt to infer an entire knowledge graph. It expands the top result by one hop, limits the number of additions, and keeps the result explainable. A user can see why a memory was included: shared tags, shared category, or overlapping terms.

For larger stores, materialize tag and category indexes instead of scanning every record. The algorithm stays the same, but the lookup cost becomes predictable. Also remove the original result from its own neighbor list and deduplicate records before the final ranking.

Graph memory research is moving in the same direction. A recent survey describes graph-based agent memory as a way to represent entities, events, and relations rather than treating every memory as an isolated text chunk (Graph-based Agent Memory survey). A one-hop expansion is a deliberately smaller version of that idea, suitable for an existing Python service.

2. Score memory health with deterministic rules

Retrieval quality depends on what is stored. A short, stale, or poorly labelled record can match a query surprisingly well and still be a bad piece of context.

A deterministic health score gives the ranker a cheap quality signal. Start each memory at 1.0 and subtract small, explicit penalties. Examples include:

• content that is too short to be useful;
• content that is unusually long for a single memory;
• command-like or imperative text when the store expects facts;
• raw commit hashes or pull-request references that may expire;
• missing tags or category metadata.

The exact thresholds should match the memory format. A 500-character limit may be sensible for a compact preference store and wrong for a design decision record. Make the rules configuration rather than hiding them in the ranker.

def calculate_content_health_score(entry):
    score = 1.0
    content = entry.get("content", "")
    metadata = entry.get("metadata", {})

    if len(content) < 30:
        score -= 0.4
    elif len(content) > 500:
        score -= 0.2
    if looks_like_transient_command(content):
        score -= 0.35
    if contains_commit_or_pr_reference(content):
        score -= 0.4
    if not metadata.get("tags"):
        score -= 0.15

    return max(0.0, score)

The score should influence retrieval rather than erase data. A low score can reduce a memory’s rank while leaving it available for audit or an exact lookup. Keep the penalty reasons so maintainers can explain a result and tune the rules.

This approach complements hybrid retrieval. Google’s documentation describes hybrid search as combining vector and keyword signals; a memory health score is an additional quality signal, not a replacement for either signal (Google Cloud hybrid search). For an agent memory system, the final score can combine lexical similarity, embedding similarity, metadata matches, health, and recency.

3. Rank the source, not just the text

Two records may contain equally useful sentences but have different provenance. A memory explicitly written at a user’s request should usually outrank a record produced by a test fixture. An identity file reviewed by a person may deserve more trust than an automatically generated synchronization note.

A small source map makes that preference visible:

SOURCE_RANK = {
    "memory_tool": 9,
    "identity.md": 8,
    "USER.md": 8,
    "MEMORY.md": 7,
    "session_search": 7,
    "sync_l1_to_l2": 6,
    "cron": 5,
    "manual_fix": 4,
    "test": 2,
}

def source_weight(source):
    rank = SOURCE_RANK.get(source, 0)
    return 0.9 + 0.1 * (rank / 9)

A source multiplier should be modest. In this example, a highly trusted source gets a weight of 1.0 and a low-ranked test source gets roughly 0.92. Provenance should break close ties; it should not make an irrelevant memory outrank a directly relevant one.

Store the source and the reason with every memory. When a result is shown, the system can say that it came from an explicit memory write, a reviewed identity file, or an automated test. That is more useful than a mysterious confidence number.

Put the three signals after recall

A clean retrieval pipeline can look like this:

user query
  ↓
tag or category router
  ↓
hybrid recall: embeddings + lexical search
  ↓
one-hop graph expansion
  ↓
health and source weighting
  ↓
deduplicate, rank, and return

Keep the stages separate. The router should prune irrelevant categories. The hybrid recall stage should find direct matches. Graph expansion should add nearby memories. Health and source weights should adjust the final order. This makes each stage measurable and easy to disable during an experiment.

For a first implementation, log the candidate set at every stage. Record how many direct results became related results, how often a health penalty changed the top five, and how frequently source ranking broke a tie. Without those counters, “retrieval coverage improved” is difficult to reproduce.

What to measure

The reported Hermes changes used 97 memories and claimed roughly 30% higher retrieval coverage. Treat that as a local result until it is tested on a held-out query set.

A stronger evaluation should include:

1. a fixed set of real user queries;
2. relevant memory labels or human judgements;
3. recall at 5 and 10 before and after expansion;
4. precision after adding neighbors;
5. latency and candidate counts;
6. the rate of stale or low-quality memories in the final context;
7. answer quality when the agent uses the retrieved set.

A graph expansion that improves recall but floods the context with irrelevant neighbors is not an improvement. Measure context size and downstream answer quality alongside retrieval coverage.

Keep evidence gates in the loop

A memory pipeline should be able to reject unsupported structure. If a model extracts a decision, tag, or relation, require the source text to contain the supporting sentence before saving the record. This is especially important when an agent turns a long conversation or repository history into durable memory.

The deterministic layer does not make the system truthful by itself. It makes the system easier to inspect. A source rank, health penalty, and graph edge should each have a reason that a maintainer can verify.

The practical lesson

The useful idea in a large code-analysis project is often a small pipeline decision rather than a large dependency. One-hop expansion, health scoring, and source ranking can be added to an existing hybrid search service without an extra LLM call per query.

The design is intentionally conservative: add nearby context, penalize records that look weak, and prefer deliberate sources. Then measure whether the agent actually answers more reliably. Copy the principle, adapt the thresholds, and keep the implementation small enough to remove when the data says it is not helping.

FAQ

What is graph expansion in agent memory?

It is a bounded step that follows relationships from the top retrieved memories to nearby records, such as items sharing tags, categories, or important terms.

Does memory graph expansion require a graph database?

No. A one-hop implementation can use metadata indexes and ordinary Python data structures. A graph database becomes useful when the number and complexity of relationships justify it.

Should low-health memories be deleted?

Usually no. Lower their ranking and preserve the penalty reasons so they can be audited, repaired, or retrieved by an exact identifier.

How much weight should source trust receive?

Keep it modest. Provenance should help resolve close matches, while semantic and lexical relevance remain the primary signals.