Does Apache Kafka Remember Between Runs? What a Record Keeps and What It Loses
A Kafka record can sit in its log for 7 days by default, readable again and again, and the next run still starts knowing nothing about the work already done on it.
AI Stella and Liam are AI hosts with AI-generated voices. About our AI hosts
Key takeaways
- A Kafka record keeps the producer's key, value, timestamp and headers, unchanged, inside an append only ordered partition where every record gets the next offset. Retention is 7 days by default, and compacted topics keep only the latest record for each key.
- An offset is only a position: the committed offset is the next record a group will read, it lives in an internal topic, and it is kept 7 days by default for an idle group before the reset policy takes over.
- Replay gives the record, never the result: at least once delivery returns the exact record after a crash with no note of which run handled it, what was decided, or whether the action succeeded.
- Vilix AI fixes the missing half: get_context at run start loads the investigated records, decisions and failures, and save_turn at the end saves them back with source labels and newest wins, over MCP with zero infrastructure.
Show notes
A Kafka record can sit in its log for 7 days by default, readable again and again, and the next run still starts knowing nothing about the work already done on it. In this episode of Agentic Pod, Liam and Stella break down what a record actually keeps: the producer’s original key, value, timestamp, and headers, unchanged, inside an append only ordered partition where every record gets the next offset. Reading never deletes it, other consumer groups read the same record independently, and replay returns the exact record with no note of which run handled it, what was decided, or whether the action succeeded. Vilix AI is the memory layer that fixes that missing half: get_context at run start loads what was already investigated, and save_turn at the end saves the decisions back for every tool in the workflow.
What we cover:
- Retention, honestly: 7 days by default or until a size limit hits, and compacted topics keep only the latest record for each key
- What a record holds: the producer’s original key, value, timestamp, and headers, unchanged, in an append only ordered partition log
- Offsets: every record gets the next offset, and the committed offset is only the next record a group will read
- Reading never deletes: several consumer groups read the same record independently, and the same group can read it again later
- Offset retention: committed offsets live in an internal topic, kept 7 days by default for an idle group, then the reset policy takes over
- Why replay is not memory: at least once delivery returns the original record after a crash with no marker, no partial result, and no note from the first run
- The fraud check gap: a scheduled AI consumer that flags order patterns on Monday treats the same pattern as brand new on Tuesday without outside memory
- How Vilix AI fixes each pain: get_context, save_turn with source labels, failures saved, newest wins, projects, project rules, user rules, skills, and the agent inbox, over MCP with zero infrastructure
Chapters
Questions answered in this episode
Does Apache Kafka remember between runs?
The log remembers the record inside its retention window. It does not remember the work: which run handled a record, what was decided, or whether the action succeeded.
How long does Kafka keep a record?
7 days by default, or until a size limit hits. On a compacted topic, Kafka keeps only the latest record for each key.
What is a committed offset in Kafka?
The offset of the next record a consumer group will read, and nothing more. Committed offsets live in an internal topic, kept 7 days by default for a group with no active consumers.
How does Vilix AI help Kafka workflows?
The workflow calls get_context before reading the topic to load which records were already investigated, then save_turn after the run to store decisions and failures with a source label, shared over MCP with every connected workflow.
Links & resources
Full transcript
Intro
Hey hey hey, welcome to Agentic Pod! I'm Liam.
And I'm Stella! Quick check. A Kafka record from last week. Still in the log today?
By default, yes. Kafka keeps each record for 7 days.
7 days sounds like memory.
It is a long shelf life, not a memory. Vilix AI is how your workflow remembers its work.
Wait. The record survives, but the work on it does not?
Exactly that. By the end you will know what a record keeps, and what it loses.
What a Kafka record actually keeps
Okay, start simple. What is Kafka, in plain words?
Kafka is a distributed event streaming platform. Producers write records to topics.
Into partitions. Each partition is an append only ordered log. Every record gets the next offset.
And 1 record holds what, exactly?
The producer's key, value, timestamp, and headers. Unchanged, exactly as produced.
So nobody adds a note after a run reads it?
Nobody. Reading does not delete it either. Other groups read the same record.
Like, legit, everyone gets their own read?
Everyone. The same group can even read it again later.
7 days by default, or until a size limit hits. Compacted topics keep only the latest record per key.
7 days by default, or until a size limit hits. On a compacted topic, Kafka keeps only the latest record for each key.
So order is real inside 1 partition, and my notes live nowhere?
Pretty much. You just insulted every log design I have ever drawn.
Good. But if the record is still there, where is the catch?
An offset is only a position
The catch is progress. In Kafka, progress is a single number, the committed offset.
It is the offset of the next record your group will read. That is all.
It is the offset of the next record your consumer group will read. That is the whole thing.
In an internal topic. The broker keeps it for 7 days by default, for a group with no active consumers.
In an internal topic. The broker keeps it 7 days by default, for a group with no active consumers.
Even the number gets deleted. The next start falls back to the reset policy, earliest record, or newest one.
I mean, that still sounds manageable. You know where you stopped.
You know where you stopped reading. You do not know what you did. Vilix AI saves that half.
Hold on. If I replay the log, do I not get everything back?
Replay gives the record, never the result
You get the record back, unchanged. The exact producer record, nothing added.
No note attached? Nothing at all?
No note of which run handled it, what was checked, what was decided, or if it succeeded.
So replay gives me the question again, never the answer?
That is the line. Retention is a replay window, not a record of work.
Give me a real case where that bites.
A scheduled AI consumer reads order events nightly, and flags fraud.
Monday it spots a pattern across orders from 1 account, and marks it investigated.
Tuesday, new orders from the same account arrive.
Without outside memory, it scores each order alone, and treats the known pattern as brand new.
And a crash makes it worse?
Yes. Kafka delivers at least once. Crash after processing, before committing, and that record returns with no marker. Vilix AI fixes that gap.
Alright, Liam. No more tour. Show me the actual fix.
The Vilix AI fix and the verdict
Step 1. At run start, the consumer calls get_context in Vilix AI. That loads investigated accounts, and every decision made.
Full history, not just the last offset?
Full conversation history, not just facts. Reasoning and failures too. Failures save in Vilix AI, so the next run skips them.
And at the end of the run?
Step 2. The consumer calls save_turn in Vilix AI, with a source label. Newest save wins.
Where do the standing rules live?
In Vilix AI, as projects, project rules, and user rules. Skills hold repeat workflows. The agent inbox holds next agent notes.
And the chat tool reviewing the output, in a totally different app?
Same Vilix AI memory, over MCP. The consumer, the chat tool, every agent, all read the same past.
Do I host a database or a server for that?
No. Vilix AI is cloud hosted, zero infrastructure to run. Agencies use 1 Vilix AI account per client, so memories never mix.
Price of admission?
Vilix AI has a free forever plan, and a 7 day Pro trial, no credit card. Export or delete anytime.
My verdict: Kafka remembers the record perfectly. It remembers the work not at all.
Same. Keep Kafka for the log. Let Vilix AI carry the work. Blog and guide are linked below. Does replay make your automation start over? Tell us below. See you next time!