ZH version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
71% Positive
Analyzed from 5112 words in the discussion.
Trending Topics
#search#rag#llm#more#retrieval#text#embeddings#vector#data#don
Discussion (168 Comments)Read Original on HackerNews
It was one of the most fun projects I've worked on in my career so far. I got a learn a lot about how US and international addresses worked, so many edge cases, and got to really understand how customers were using the existing search to make sure they weren't adding any duplicates to the database. Token filters and synonyms were neat and figuring out the right indexing strategy was a lot of fun.
It was a lot more work to get it right for most of the use-cases our customers had than just "throw it into ES and be done". That would probably have been fine for the 80/20 case, like you said, but I agree that the bulk of the work is going to be fine-tuning the search solution, whatever technology you're using.
(Obviously this doesn’t apply to searching actual rich document data - for that, go all in on text search, embedding, etc)
RAG is routing and decision making.
I found so much joy in achieving the best results given the requirements than simply hoping for the best with the cool kid called vector db and embeddings.
I agree with you.
Depending on the context and required output I decide how to orchestrate a multitude of specialized modules that produce the best specific result to gain a universally usable system.
It maintains itself.
Also live updates need reruns and rebuilding certain indexes. Everything is highly dynamic but in a deterministic way.
I found my niche with RAG selling and I build them myself.
I take pride in them.
So many look at the technology but not on the required output. It takes hours of talking to people to get an idea of what they need.
And there are regulated businesses where certain information is required to be always factual correct - pricing for example.
Vector search becomes a liability for this use case.
So naturally you have to reconsider your system: mixing factual with probabilistic content and how to make sure, it hits always certain quality benchmarks and on the other hand doesn’t fail others.
I love this kind of stuff.
And there is personal information etc.
Using modules is the key. Orchestration is really fun but I have to admit, not for the faint of heart.
And ever changing parts: LLMs, or restrictions to be matched liked autonomously working - I love RAG.
It gave me back the joy of developing. In fact I never had so much phun before, because it is also “team work”: I am not programming, I am managing a product.
I was in Senior Management of a top tier international bank and besides that build the only ever working platform or IT transformation called dbCORE and overlooked 13 teams with 120 developers.
RAG gives me dbCORE vibes so to say.
Good luck and fun with your RAG systems.
https://www.anthropic.com/engineering/contextual-retrieval
This is from two years ago, but I think it's still SotA?
Volume of documents, size of documents, versioning, frequency of update, documents similar or overlapping information, how much or exactly what you need for the LLM to understand, AI friendly documents, who has access and at what level, blue teaming, red teaming, multi-lingual, does the LLM know the domain language of the user and documents.
I probably missed a few things even with that.
I want fuzzy search like 95% of time and then I might consider having additional list of things that can be suggested by vector search.
It is not psychological, it is fully justified: substring search cannot find synonyms, periphrases and mistaken neighbours.
On the other hand, your post may contain a good idea: L=instruct_LLM("provide a list of synonyms and periphrases of terms T within context C", T, C); then iter(`grep l in L`). One NN query and a `grep` collection. But again, if one wanted to order the results, it is either through a dumb crierion or through another LLM query - but this could make it extremely costly (requiring either a huge context or a quadratic number of ordering queries).
And, the above `grep` based procedure would remain keyword based and not semantic based, which means that the user must know that it will not be based on comprehension but on the possible results that keyword matching can yield.
It is, if people don't even stop to think if they need synonyms, periphrases, or mistaken neighbours.
As the blog post points out, more often than not you don't, particularly if your primary usecase is to search for technical keywords or codenames.
Vectors and LLMs are great, but there's no magic pill here. If your parts data and config management[1] is all crazy, that's an institutional problem. Buying a crapton of tokens isn't fixing it, unless you're using it to help build an actual formal solution based on good fundamentals.
[1] Such as it is.
i dont believe ppl are building rag for this
I think there needs to be a linguist version of "what every programmer needs to know about (full?) text search"...
I'm not a linguist and I don't study languages, but I know enough to realize if a text search system is not designed for a particular language, it simply won't work. (As an example, to implement English search in a system for a hobby project, I had to import a US/UK spelling wordlist, and implement the Porter Stemming Algorithm. This is just for "one" language, and probably does not cover the other "English" dialects. Imagine doing a different workaround for every language in existence...)
RAG is actually a very language-agnostic way to work around those issues.
https://github.com/canvas-ui/canvas-synapsd
The key thing with RAG is to get the right information in the context with as few queries as possible. That requires good recall (ensuring that if it is there it can be found with a reasonable query) and precision (ensuring the best stuff is on top and minimizing false positives).
With search, and by extension RAG, the principle of shit in, shit out applies. Most of what search teams did before AI and RAG is still the best way to optimize the experience with RAG. And if you mess that up, search is not going to be working that well and no amount of AI can compensate for that or only at great cost in tokens and time. So, having an ETL pipeline to pre-process what you index, testing & benchmarking search quality, etc. are all helpful.
The good news is that you don't need that much skills with agentic coding to build something half decent for this. This code almost writes itself. And even a little bit of effort on extracting structure before indexing can make a big difference.
Similar to SEO on marketing pages, we started rewriting product docs around the idea that it will be consumed by a RAG. Mostly by putting a lot of focus on well structured headlines, thinking more carefully about technical terminology vs common human-language questions, occasionally using variations of keywords in the text, etc. This applies to pure LLM consumption too, not just hybrid search.
Once you start tracking what users are asking you learn to adapt the documentation around it. And LLMs can also suggest improvements by comparing questions vs search results vs LLM responses.
If you flatten all that into plain text and then create embeddings, you are effectively throwing out the baby with the bathwater. But on the other hand if you put some effort into normalizing and extracting some structured meta data, you gain a flexibility to do more sophisticated querying that get you more precise results.
You can of course try to fix things at the source, which is a valid thing but usually not that practical when you have a lot of data to worry about.
Is anyone else actually finding it harder and harder to read LLM generated text? I find it quite tiring, my brain just does not want to get through it.
We've all learnt that it's not really communication, and so can be dispensed with.
That is where I stopped
"using GPT-4o-mini for query rewriting" -> model from 2024, when RAG was trendy, and all the langchain, llama-index, etc, docs mentioned this specific model
So: https://en.wikipedia.org/wiki/Retrieval-augmented_generation
The acronym constraint was valid in a pre-LLM world but now you are perhaps 3 clicks in a modern browser from learning.
If I read an article that spelled out Random Access Memory I would definitely treat that as a lay article.
Then, throw it all in BigQuery. Handles all the vector stuff natively.
Sprinkle an agentic bot UI thing on top to make it appear all-knowing and magical.
I assume other vendors than Google have a similar batteries-included approach you can just plug in.
This assumes your text is small. Try embedding pdf reports - though luck. It surely won’t fit into most embeddings. I can think of many more examples: books, news articles, medical reports, insurance claims etc. they’re all too big to “index it all at once”
Because you could have clever ideas about vectors related to more paragraphs related in the document structure - but that would multiply the vectors. The index can become much bigger than the corpus.
"on the fly embedding" and "Sparse + dense reranking" don't really make sense how they're presented and smell like they came from a long claude-driven conversation after multiple cycles of these hybrid compromises across many turns.
"Recipe 4: On-The-Fly Embedding (The Fresh Data Play)
The insight If your data changes frequently, why pay to re-embed everything?"
This reads like every Claude generated presentation I've seen.
Yes, embeddings are computationally heavy, but they are not at all complicated and they provide a lot of benefit.
90% of "document" based RAG projects should view semantic search with embeddings as their primary method.
It's very powerful and so easy to implement that you could try it out and discover whether performance would be an issue rather than trying to anticipate it.
I had to look it up.
RAG is simply fetching external data (retrieval) and adding it to LLM context (augmenting) prior to generating a final response.
Any time LLMs do a grep or a web search to answer the query, it’s RAG. Many people use vector db for their own RAG implementation bc of the semantic search benefits.
People don't understand that any sort of retrieval before generation is RAG.
After doing some rigorous A/B testing, we dropped indexing. For coding, I think the reason is that a repo is already searchable. Imports, call sites, file and test names, grep gives you cheap yet reliable version of what indexing would do, and the agent can read around a hit to verify it. Chunked retrieval hands the model something that looks right, and it tends to trust that instead of going to look for the actual source. Another thing that I noticed was the most intelligent models like Opus 5 and Fable ignored chunks anyway most of the time for some reason. Possibly perhaps they are trained around not trusting similarity checks for codebases.
Extremely large codebases with docs feel different. You can’t grep for a concept you can’t name. That’s the case where I’d still use retrieval.
(I work on TheGitAI, for disclosure.)
https://github.com/jankovicsandras/plpgsql_bm25 BM25 search implemented in PL/pgSQL ( Unlicense / Public domain )
The repo includes also plpgsql_bm25rrf.sql : PL/pgSQL function for hybrid search ( plpgsql_bm25 + pgvector ) with Reciprocal Rank Fusion; and Jupyter notebook examples.
Embeddings and semantic search add non determinism on top of non determinism. This seems fundamentally cursed. Lexical is much easier to control, iterate and debug. The tools are incredibly mature. Your users will probably prefer it as well.
If you are building a RAG pipeline for your company and are struggling like me, I would recommend this author that has whole series on entreprise documents (start with the one from May 22nd): https://towardsdatascience.com/author/angela.shi/page/4/
Note: I am not the author, just got her article in my newsletter and found it useful.
I'd like to work with a corpus offline (internal university research data) and I'm hoping I can get everything done without the data leaving the premises.
I guess the biggest bottleneck is going to be for the context window size which won't be able to fit too many result "hits."
Any info or advice would be appreciated.
Everyone keeps posting articles about how to implement RAG, but I also wonder why there isn’t some sort of skill to help people create a simple retrieval plan, starting with the retrieval methods and connecting them with evals. This could show whether they actually improve the result and make retrieval simpler for any agent, instead of making people start from zero.
Oh boy...
The AI that wrote this might be the master not the writer, as this looks written by AIs.
I will use the author's agents, not read his articles or use him for the job.
How is RAG any different from the search systems we've been building before LLMs? Is it the sudden need for everyone to design a search API and engine that's driven this trend?
If so, I'd like to see more design patterns around existing search problems:
- Correcting or backtracking based on feedback.
- Measuring relevance.
- Comparison with task-based pre-written queries. Does every LLM task need a full blown search engine? Why not a tightly scoped domain API for data retrieval?
What makes it worse, a lot of people in the thread equate vector search with RAG, whereas RAG is the name for anything that model can query so a user doesn't have to copy/paste feed it to the model manually like access to text files is RAG.
Where's the new design tension? Indexes always had to be monitored for freshness and queries have always needed cleaning or parsing.
at least to me that seems the same as https://en.wikipedia.org/wiki/Word2vec for e.g.
Let's say it's just "computation packaged as something new". "Trivial things".
(found of sid.ai so obv biased)
RAG is one of those things where I can hyper optimize to an absolutely needless degree.
Can we not reward junk like this? Most of the sentences are incomprehensible and provide zero actual argumentation, it's just a list of "whats" with no "whys"
RAG is about providing an grounded response, given the actual data in the corpus.
Great article and content, nonetheless!!
Still need ~2 years to be replaced.
With agents, the prompting could be dynamic for maximum accuracy for every retrieval.
This absolutely would beat the best of the best embedding-based RAG models.
Nobody uses this now mainly due to speed. An llm retrieval would be 10x or more slower than embedding.
You can try that now
Take some failing cases or bad retrieval from your current system Prompt an llm wisely like a perfect prompt to get what you want and provide it the context to it. And see the results.
For context, you are limited now by models contexts (1m), so mostly you would need to split what you have and prompt twice....or more...and so on
It's necessary and would be good for you if you want to learn something systematically.
But for most of the normal issues, we can not rely a lot on it.
https://cursor.com/blog/semsearch
Grep falls apart for severely underspecified queries, which is the difficult part of web search. For any given query in web search there can be several millions of candidate results. You can get good results with FTS as well, but just finding phrase matches is inadequate, you need more ranking signals to find relevant results.
When Claude is looking for a function in your code base, it needs to sift through dozens of matches. This is not hard, and anything beyond grep is likely not worth the effort.
I find this interesting because practically no one is doing RAG on thier personal data which is something I wouldn’t have expected.