Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
71% Positive
Analyzed from 2250 words in the discussion.
Trending Topics
#eval#model#evals#test#models#llm#more#hard#simple#frontier
Discussion Sentiment
Analyzed from 2250 words in the discussion.
Trending Topics
Discussion (51 Comments)Read Original on HackerNews
Are there any examples of successful startups doing this?
The market's being split into
1. Longitudinal LLM observability tooling
Most eval startups have gone down the route of something more like being an observability platform for LLM inference. They want to be in your stack and running the inference to collect data on performance of it.
They collect things like how often a model returns JSON that's out of spec or returns values that aren't expected as well as general timing and cost info.
2. Safety Limiting / Pentesting
Say you're doing something in the medical field or that's sensitive in some way and you want to figure out what model has the best outputs for your task that won't fly off the guardrails.
3. Simple cost + performance + quality swapping
This is what my tool does, basically lets you test if you _really_ need to be running that frontier model in a loop across a million records or if you'd be better with an older model or something else.
https://evvl.ai/
Example eval: https://giyd8stidy.evvl.io
My company recently built a tool that is closer to your first category, but it's an API so it doesn't have the security (supply chain) concern of being embedded in your application.
https://endpointevaluator.com
It's built to help people manage the risk of LLMs changing underneath them and drifting from their designed behavior. Traditional deterministic testing probably won't be sufficient for apps that provide nondeterministic output, like a chatbot backed by an LLM.
The point in the linked article about the challenge of selling developer tools to developers is a good one. I think the first reaction to coding agents is "let's build everything ourselves!" but the long tail of maintenance is still there and the pendulum will probably swing back to "let's stick to our knitting."
I haven't looked that hard, but I can't find articles about this type of eval testing, curious to hear if others have approached writing APIs in this way.
For example, a simple eval is a dataset of multiple-choice questions, which each have one correct answer, and scored by accuracy. An example of this kind of eval is the Massive Multitask Language Understanding benchmark (2020) (https://arxiv.org/abs/2009.03300).
A more complex eval is FrontierCode (2026). Questions in FrontierCode represent coding tasks needed for real-world repos and are evaluated against rubrics scoring for correctness, code quality, cleanliness, and other factors. https://cognition.com/blog/frontier-code.
ÂąNote that this is a slightly different definition we used in [0], which used a definition of a fixed input-output correspondence pairs combined with a metric. What's different from 2021 is: models are now given more open-ended inputs (prompts like "find the bug" and a codebase rather than a simple question), have freeform generation (rather than choosing a fixed answer), and are graded in a more complex manner (e.g. beyond correctness, one might care for a coding eval also to grade adherence to coding guidelines, test coverage, etc).
[0] Liao, T., Taori, R., Raji, I. D., & Schmidt, L. (2021, January). Are we learning yet? a meta review of evaluation failures across machine learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). https://thomasliao.com/are_we_learning_yet.pdf
I do agree that the author does not do a good job of introducing the term.
What kind of stupid business is this. Though nothing can beat SEO in that spirit.
What makes it particular for the case of AI is:
- there are many situations where you can’t test using pattern matching
- you don’t only like to test correct answers but voice and tone too (imagine a bank support LLM-based chatbot that answers using slang)
- evals can be used to compare the performance of different implementations; given the costs of LLMs, it’s very important
- running evals is more expensive than running standard tests, because you rely on the LLM calls under test, and many times they use LLMs as a judge. It means that running them in every commit of your CI/CD is very expensive
- Knowing all the possible inputs for the LLM is impossible, so evals can also be run on runtime samples to detect anomalies
AI is not deterministic like regular code, so imagine you use it for "search" (RAG) or for summarizing or for classifying emails etc. How do you know it is giving you the right results? In this context, AI evals are an important idea and very often neglected.
You can use an initial "dataset" to evaluate your prompt and AI calls + code (think test cases), this dataset will of-course be curated by humans. But as the software is used, you want to incorporate, real production data as well and run the evaluation pre and post launch. Sounds simple, but can get complicated specially since this area is new and as the post mentioned there are too many players and options out there (since everyone thought this is a money maker).
However, identifying the right metrics and having the necessary test sets will, at times, be challenging.
To add to this, they have to be developers who aren't already using a fullstack observability solution, since it's fairly straight forward to add the eval startup featureset to an existing observability solution, and easier (plus cost effective) to just keep it all in one place.
The author does have a point around generic benchmarks not being super valuable for companies. But evals should be seen as verifying design/behaviour constraints and can greatly aid product building, golden dataset creations and good software practices.
It's just that the aim should be "how to generate your own good evals, even if it's hard" as not so much "here's some generic evals about models".
Also, model providers are not interested to have their models compared head-to-head under identical conditions. And “Model A is better than Model B” is almost meaningless by itself. Better for what task? With what prompt? What inputs? What budget? What failure tolerance?
It would be nice to have a place where users could run their own benchmarks, define evaluation criteria for their actual use cases, and make those runs verifiable by others.
For years upon years until you get brought out. Then it’s someone else’s problem. Or you IPO and bring in new management to figure out how to make money.
VCs don’t see 20x exits happening for Eval companies, so they have trouble with the losing money for years step
It's probably not gonna be exactly glorious work, but designing expert evals settings and collecting and crunching the data for quality assurance and control is going to be needed.
Therefore your knowledge is better used in training than letting users be slightly better at the token casino. Which is mentioned in this post as well, eval startup people either go to work at frontier labs or finetune startups.
Personally, I agree with the Goodhart problem, but isn't the reason Eval startups fail because they try to sell an 'evaluation service' rather than a 'verification toolchain'? The problem, it seems, is that AI verification toolchains require a model in the end, because they internalize AI and sell it under the name of a 'harness.'
So an AI verification(eval) toolchain would have to be structurally different. Verifying AI code isn't about whether it compiles. AI code can always be made to compile. The issue involves various semantic criticisms, such as overfitting to existing designs and tests. To catch those issues, you ultimately need to build an AI. But building that AI is expensive. So in the end, AI verification companies depend on external model providers for the core components of their verification engine. I think this is a bad business decision
> built the tools to verify, deploy, and build them, such as CI/CD, static analysis tools, and testing frameworks.
Curious. Which company made money with testing frameworks?
It's simple to say but hard to master doing well, and the important thing is that no matter what tool you have the evals don't write themselves.
Aha.
But it sounds like the author does not mean those companies at all since those are actually important in enabling the very Venn diagram he/she describes.
Based on what I assume the author's referring to they are referring to something more like a public benchmark report provider... I would say but yes that's a relatively small total addressable Market space no matter how you look at it