Project 06
LLM Semantic Cache
A cache that answers repeat questions by meaning, so the LLM isn't called twice.
LLM apps pay for and wait on near-identical questions again and again. This layer returns a stored answer when a new question is close enough in meaning.
I built this as a semantic caching layer for LLM apps. It intercepts a question, turns it into a vector with Jina embeddings, and checks Redis for a previous question that is close enough in meaning. On a hit it returns the stored answer immediately; on a miss it calls Groq, stores the result, and returns it.
The service is a FastAPI app with a per-client rate limit in Redis and a PostgreSQL log of every query. GET /analytics reports total queries, hits and misses and the estimated cost saved, and DELETE /cache clears the cache. Everything starts with one docker-compose up.