Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
74% Positive
Analyzed from 10046 words in the discussion.
Trending Topics
#more#chips#chip#model#nvidia#hardware#tokens#https#water#models
Discussion Sentiment
Analyzed from 10046 words in the discussion.
Trending Topics
Discussion (354 Comments)Read Original on HackerNews
For example, GPT Sol baked into a custom chip run for $100M that runs 10x as fast and 10x as cheap should pay for itself as long as the chip is useful for long enough.
While 2 years ago nothing was useful more than 1 year long, there are many older models in use now (e.g. Haiku 4.5, GPT-OSS 120b), and I expect this trend to continue.
I know this is what Taalas was doing (acquired by AMD), here was their demo, https://chatjimmy.ai/ which is based on Llama 3.1 8B. It feels like this should start to happen soon.
1. https://matx.com/
2. https://www.d-matrix.ai/
3. https://www.etched.com/
4. https://www.positron.ai/
5. https://hyperaccel.ai/
6. https://axelera.ai/
7. https://www.enchargeai.com/
8. https://furiosa.ai/
Unfortunately, there's some real physical constraints: IIRC, manufacturing a wafer takes on the order of a month, start to finish, for the physical processing.
Maybe once LLM improvements asymptote further?
https://www.eetimes.com/taalas-specializes-to-extremes-for-e...
https://www.turingpost.com/p/taalas
https://cambrian-ai.com/taalas-launches-hardcore-chip-with-i...
Part of the key is that by moving even from 6nm to 3-4nm one could embed a 20-30B model as part of a MoE (or only a subset of activated layers) on a single reticle die (note B300s are already multi-reticle), with a separate predictive/dispatch model controlling them each on a separate chip. This is without even stacking CiM ROM die. Moving the layer activations (and KV cache etc) between die requires relatively high speeds (and low latency), but distributed with multiple die in parallel might well be doable even with standard multilane PCIe. Of course KV cache prefill could also be handled by external GPUs. I'm sure AMD will make some reasonable choices.
If none of that is baked into the chip as now then all the chips are running the latest weights every time.
Even if you could ignore the stuff built into the chip when the time came, at that point you just wasted money on silicon that’s useless in 2-3 months.
This way newly post-trained model can be loaded and served the same day.
The bigger issue seems to be that these chips can’t hold that many weights at the moment.
(I’m curious if chips with large weights in them would be more tolerant or less to yield issues. If you flip a few bits in the weights, does it really matter at scale?)
Yeah, with any luck it would put pressure on Nvidia to charge less, and not just to OpenAI. With a little more luck, we would see all the other players do the same thing, driving down the price of actual GPUs from GPU manufacturers.
Maybe! But it also doesn't require the rate of improvement to slow down. As long as some current model is eventually "good enough" for general use, it could still be a market-killer at a very low marginal price thanks to ASIC. Even if slower, much more expensive models are 10x better, that doesn't actually diminish the utility of the ASIC model, as long as it's "good enough".
The context size is still not nearly as big as it needs to be to store all the code an enterprise needs. And apparently as you increase the context size, there are more defects. So none of this is a solved problem. We are still in early days and there is a lot to be done.
I'm sure there are marketing people who will say "coding is solved" and other such snake oil but none of this is done, far from it!
That being said there is still enormous value in older models especially with tool calling which will let them access the latest data. I feel like we need to be a little more careful and the tools should cache in a smart way to avoid rework but clearly if we could have opus 4.8 level of work for like a one time payment of a system for local LLM it will have value for years into the future.
So I agree in a weird way that sol is good enough for certain tasks but really there is a long road ahead.
If SOTA models haven’t peaked, then the SOTA model companies would still be churning out better and better intelligence.
If companies working on ML-specific chips was evidence that large transformer models have fully saturated their potential, the field would have been done circa GPT-2.
Every time a company has spent resources doing this, a competitor innovated on the software and made the custom hardware irrelevant.
This often happened, though not always.
An important counterexample are 3D graphics cards, which basically put the OpenGL/DirectX fixed-function pipeline into silicon. It took a long time and many iterations to make the pipeline more programmable until the 3D graphics cards turned into modern GPUs.
Even today, GPUs live on as separate hardware in a computer instead of having become integrated into, say, the CPU. Intel's attempt to do something like this with the Larrabee project [1] was discontinued.
---
[1] https://en.wikipedia.org/w/index.php?title=Larrabee_(microar...
Only when they became more general with shaders, and then added support for GPGPU, did it truly take off.
I think this generality is the lesson here, not the fact that GPUs are not CPUs.
Taalas needed a giant chip (6nm) for an 8B model.
At best you could use a more advanced node to try to put a MoE model across several chips working together, but you can’t have GPT Sol size models on a single chip like that.
You're phrasing it like it was kind of an inherent technical limitation with this kind of burning weights into silicon. Which is also not new, it goes back to the 1980s with fixed function digital signal processors and little linear regressions or hardware classifiers for industrial control systems, all are the same basic principle.
It's just usually not worth it to go super small process node, because most models people thought to turn into silicon were pretty small parameter sizes. We're talking 10-100 weight regression or at most 2-4k weight neural net, used in some instrument or factory equipment. You can do a decent MNIST OCR with a 4k weight neural net. For this, 180/130nm is fine.
Or you might think it's required with their special 4-bit as transistor thing (plausible). It's more that when you're experimenting and iterating, TSMC 6nm is their advertised path for rapid prototyping at cost for proof of concepts. And that's already in hot demand, while good luck if you're a startup trying to break in with 3/4nm as your first run.
It is.
As I said, they could have shrunk it with a smaller process node, but that's not at all close to what would be required for a GPT Sol size model.
The other thing is, a lot of the time, model performance is improved with more 'thinking' time.
The thinking time is just more tokens... but instead of say 1000 tokens or 10,000 tokens worth of thinking its 1,000,000... how does that improve model performance? Could a 128B model hit levels of GPT Sol?
The more problem like these they solve the more they will look like GPU.
Let's say a magic set of chips comes along to host this. Maybe it's 2-3x more efficient in size and power. You're still talking a form factor that's a good chunk of a rack, draws tens of kilowatts, and could actually be sold at a similar if not higher price point because the OPEX is so much lower.
It may be useful but it's certainly uneconomic to spend >$1m to self host the model, plus ongoing power and maintenance costs, plus the cost to adapt whatever building you're in to be able to power it.
Cooling might be an issue though...
I think people underestimate how much of a revolution having an always-on, privacy-preserving personal notetaker / secretary would be.
Like he is optimizing to keep providing a vanilla token factory when weighted chips are coming and local models will supplement.
My head canon is savvy chip execs will be etching architecture his OpenAI pioneered into their flagship products while trying to minimize how much foothold he can get in hardware. Murica done offshored it. Not ours to control.
If you believe that, then you should expect to get Sol-level performance out of a Luna-cost model within six months or a year. If you have a system with the weights baked in, that means you're going to end up serving that Sol-class model several times more expensively than it will take someone who comes along a few months later. (such as what recently happened with DeepSeek's update.)
And under that assumption of continuing advancement, baking things in doesn't make sense in general - it's a play you'd make if you think things are slowing down a lot. Which may be right but it's not OpenAI or anthropic's play.
Even if the balance was net positive, you would also not be able to train them against new tools/harnesses or knowledge. How many years do you expect to keep using them?
Beyond the model, when would you freeze processor performance, such that it was good enough? Because that's exactly what freezing on Talaas is premised around.
The semiconductor technology will also continue to improve. You lose twice. Talaas is one of the dumbest ideas I've seen in semiconductors in decades.
I’d like to think that most parents would be weary of handing their children what basically amounts to a tape recorder that siphons all the data off to a large corporation.
OTOH, a completely local one (LLM + VAD + Speech Rec) would be a fun little thing to build.
https://en.wikipedia.org/wiki/AG_Bear
but you trade updatability, which I don't think is worth it yet.
I suspect the answer to both of these questions is yes right now, but I agree it’s borderline.
That's quite a few parameters shy of today's trillion-weight behemoths, but it is fast.
It works like this:
1. Founder befriends undergrad interns/graduate student interns, buys them gifts, invite them to dinner/yacht/house/vc parties etc, or pays them to write articles 2. Founder extracts insider information out of these interns 3. Founder sells this information to companies paying "consulting" fees
One thing not on that comparison table: die size. If I'm understanding that correctly, it's about the same as the Rubin, but at 1/3 the number of NVFP4 PFLOPs. (The text disagrees with the table, I'm taking the table as truth, perhaps that's wrong...)
They were right about everything 50+ years ago, but they didn't have the budget for the right hardware, had to write conference papers and books instead.
I remember one soundblaster card I bought came with a Lara Croft demo, that exploited the incredible immersion of real time dynamic reverb.
Genuinely I think game audio took a few steps back from that heady era, the innovation in audio likely didn't sell as many cards as graphics innovations did.
First we had to have the audio processor. Good EAX was available on top of the line cards, and they were not always cheap. Lower end chips got less features.
Then we had to have the speaker setup to have the greatest sound, or needed to get a real 5.1 headphones, which were bulky and never provided the same fidelity.
Then Microsoft changed the Windows driver model, cutting the driver's direct access to the card. All of the timing sensitive effects were gone in an instant. I remember installing the new drivers and getting literally nothing. Sound Blaster was the only card with an hardware mixer, and Microsoft didn't feel like enabling them. Mixing at the DirectX layer killed the cards.
Soundblaster's very closed stance didn't help them either. None of the cards after Audigy2 worked with Linux when I had my desktop system.
After my Audigy2ZS, I moved to Asus Xonar D2X. Its positional audio capabilities were nice, but I mostly bought it for its Linux support and sound quality, and that was top notch in that regards.
Then sound cards became commodity. Everybody stopped making good cards. Musicians moved to audio interfaces, audiophiles moved to DACs.
Just looked to the SoundBlaster website. Internal cards are very limited. One DAC, one DTS enabled 7.1 sound card for PC cinema systems, three game oriented lower end cards, nothing else.
In early 2000s for example, friend with something like a Sound Blaster Live, but couldn't use it anymore as they lost the drivers, their website only had downloads for driver updates, requiring you to still have a driver CD, so no more CD meant no way to get the drivers.
They had not done very much meaningful stuff since the release of EAX and used patents to prevent anyone else from competing with them. Onboard sound cards were by and large indistinguishable from a quality perspective as Creative Labs ones, but cheaper which Creative Labs combated mostly with lawyers as opposed to upping their game.
Some guy after being frustrated with a long-standing bug in drivers for their Creative Labs sound card, dug into the binaries and made a fix for it. During which they also discovered that you could simply flip a switch in the driver to unlock features only meant to be available on more expensive hardware. Creative Labs of course went straight to lawyers to shut them down.
My brother bought the Creative Labs WoW headset which would have its mic get progressively softer until he would leave and re-join the call/voice chat room. They never released a driver update to fix this.
By the time that Microsoft announced no more "hardware acceleration" for sound cards, I had zero sympathy for Creative Labs, I was already convinced that they made pretty shoddy hardware/software and were mostly riding on their reputation from the 90s and some patents they managed to get.
It wasn't really the cool reverb effects or wave tables, though those were a nice bonus. It was just "I can tell my computer to make sound and it actually makes sound without days of troubleshooting."
Granted, similar things could be said about 3dfx. It's was a 3D card with drivers that actually worked.
And then there's the obvious "sound blasters and voodoos go in my computer, jalapeno goes in someone else's computer" thing.
In fairness on board (depending on the board but on the whole) is pretty good.
Apple is in the software first, the hardware second. Everyone at Apple has been trained to understand this for decades, and Jobs pointed it out endlessly. Apple's real moat is software (services, iOS, experience, MacOS).
Windows, Office, Azure, et al. Microsoft accumulated approximately one zillion dollars in profit on the back of software. It's a vastly superior business to anything hardware has traditionally seen. Nvidia is the first true juggernaut hardware profit machine, and the AI boom in extended hardware (RAM, storage) will prove temporary (even if there is a feast during that time). Microsoft's advantage and moat was Windows-Office for decades. It was a far better business than Intel's chip biz.
Google is a software company first. Every aspect of what made them and maintains them is software first, hardware second. They're a $400 billion software company. Their ad machine is software. Search is software.
Facebook is software. Instagram is software. WhatsApp is software. A $200 billion software company. They're not selling hardware, they're selling ads via software, they're monetizing users that use their software.
AWS is at least half software as an entity in terms of complexity, competitive advantage, et al. That's a two trillion dollar business.
LLMs can run successfully with various hardware approaches. The software is the value at the end of this, regardless of the hardware under it. The sole exception so far that may be sustainable is Nvidia, and we'll see if the bottom falls out from under that margin monster (China, specialized AI chips, whatever it happens to be that cuts under them massively).
Hardware always gets its margin squeezed eventually because it's a manufactured good (with inventory, fabs, etc). Software is hyper margin by default, you have to layer a lot of garbage on top of it to kill the margin. Nvidia is 33 years old, they have had a rich business for three years, that's it.
The AI boom is the sole reason anything in hardware has looked great in the past 20 years. Check the margins & op income for the top 20 hardware companies, from TI to AMD to Intel to Nvidia to Micron to Sandisk to Samsung to TSMC to ASML, prior to the AI boom of the past couple years. It won't last indefinitely. And after the return to a more normal environment happens, the hyper margins in software will persist.
To me, the efficiency gains of inference chips are so significant that they are certainly here to stay — barring a revolution of sorts that leads to a world devoid of AI as we know it.
This couldn't have been easy. The team at OpenAI has worked a miracle.
The only report is a smartnic fpga from Alibaba where we take an onnx design and write our own. https://essenceia.github.io/projects/alibaba_cloud_fpga/
On my M2 Pro Mac Mini the ANE only allows 2 gigabytes compared to the Metal GPU which can use the system ram.
Currently playing with https://www.asus.com/motherboards-components/ai-accelerator/... which is a 4bit, 8bit and 16 bit ai inference chip with 8 gigabytes of ram.
The UGen300 has the Hailo-10H chipset.
The ASUS Store price for the ugen300-usb-8g costs $365.00 Canadian dollars.
Take that, Jalapeno!
This is about the same rate you get out of Sol Ultraspeed.
Why do you think extra tool calls like that would be so unthinkable? It'd run circles around this, especially if the problem can be split up among a live-collaborating agent swarm, so that it's not a single user thing anymore, which is exactly what they have in the cooker with Astra.
I don’t see how tokens can’t produce speech or track metabolic needs. You can talk to chatgpt can’t you? Or do you mean literally talking? Because that’s not a brain function, that’s the mouth, vocal chords, and lungs.
It probably could, but the point is this would require additional tokens, blowing up the comparison. The token output of LLMs and "token output" of speech are simply at different abstraction levels. Hence my comparison to the LLM brain driving the robot harness to produce speech etc. This would be more comparable, and also look significantly worse than "only" the 22x less efficient number.
But the true number is IMO far bigger: orders of magnitude greater if we think in terms of equivalent performance.
Based on a human output rate of 3.3 tok/s, which seems questionable as a means of comparison
Also brain produces quality tokens @ 3.3 tps instead of fast generating hallucinated tokens by certain models. Thus MTP can produce low quality tokens at 2x speed.
Patience pays.
I am relatively certain we have already squarely been beaten in net efficiency at scale.
Productivity is not the only reason to let these meatbags burn oxygen.
Yeah, those guys aren't biased at all.
The article you link says not a lot of criticisms with very many words, and the AI prose gets much worse towards the end, seemingly when the author also gave up on reading it. I am disappointing in the plagiarism though, especially of Ryan Smith.
I am much more interested in what you think of the site though vs your own experiences running a GPU cloud. I've seen your comments on it for a long time, it's super interesting. So if you think their takes are mostly bunk I'd consider it way more than this hot aisle guy.
lol. lmao even.
Have you seen the quality of their output? I'd take Claude or ChatGPT Free Tier over advice from McKinsey these days.
I mean, previously you could have said something much the same except substitute "frat boys".
In short, better hardware will drive down token cost in the near-term, but will drive up the demand for tokens as it gets cheap enough for other sectors to start to use it heavily.
It comes from steam engines where economists originally thought that coal demand would plummet with more efficient engines, but it actually just meant that we found more uses for steam engines.
Codex keeps doing these usage resets. What do I do? Burn even more tokens than ever before. I know I'm not the only one.
(I switched to using local models as usage limits, api instability and the concept of paying per token stresses me out)
This doesn't mean the concept is BS, but one single concept cannot explain away everything in such a system.
Continue what? The cost per output token has kept going up for the past three years across the board, as thinking models keep leaning more on test-time scaling.
The quality of the said output tokens obviously increased, and arguably increased more than their price, but the price still went up. Or, on the flip side, the price of combined tokens went down (a bit, it did not "plummet" at all though) but so did the average token quality if you count thinking tokens.
Although given we have marginal pricing we need to push through to those lower prices in the face of increasing demand, so timing of this is uncertain and the key to the AI financial markets
Token prices plummeting is probably a good thing, but not without the regulatory backstops that prevent these effectively industrial facilities from being operated with no regard for the externalities they impose on people who live near them.
If we can’t even build data centers, the least disruptive industrial use possible, there’s no hope to reindustrialize the US or anywhere outside of China.
how much pollution do you believe modern gas-turbine engines to produce?
>Not to mention the water use controversy.
what percentage of US water usage do you believe is by AI data centers?
you really don't get it?
So as much as i agree “more profits to stakeholders screw the customer”, i think its more of an emergency to get to profitability before the music stops.
what makes you think this?
I was early at efabless.com well now chipfoundry.io - they've done about 800 chip tape outs.
They've been doing open source silicon tape outs for a decade plus.
Founder recently built this: https://nativechips.ai --- not involved but I'm inclined to believe it's the future of where the market is going. I'm skeptical of many of the AI chip design startups and whether they've actually taped out chips and how many and at what scale.
ASICs are deployed when the application area is economically large enough to so there is return on the investment in developing one. Bitcoin mining few years ago or today inference or more mundane things like video decoding/encoding.
General purpose chips on the other hand have to be general purpose first to be useful, i.e. support as many application domains and instruction sets as possible . It can be long tail of support which both slow your chip down and also slow development down. Apple's took a long time to develop M series to be general purpose enough and still need even now software tooling like Rosetta to make say virtualization work for a good reason.
New tooling would always help and there is already lot of software emulation for developing chips today but you still need physical iterations to tap-out and have high enough yield, no LLM can help with that.
If you make an inference only chip, you better be damn sure that it's significantly better than Nvidia's GPUs at it.
Otherwise, it's better to buy Nvidia' GPUs because they're more flexible. You can do a big training run, then use them for inference right after.
Picking the best open source models is really the best they can do.
If the chips weren't this compelling they would have something different to announce.
These are paperclip maximizers who just happen to wear human skin - there is no underlying premise nor ideological goal.
Or at least Nvidia GPUs will become slightly cheaper for regular consumers again
There is probably a ~50% chance that the next Dem candidate for presidency runs on a national datacenter moratorium or something equally as crippling.
The current datacenter owners love a compute-bound world anyhow. A moratorium on new datacenters would increase their valuation, encourage efficiency and make computers cheap again. If Chinese labs can ship frontier models under 1T parameters, why not American labs too?
It's fine if you're one of the people selling shovels to gold miners for a while, but sucks to be building houses in the boom town?
> However, as previously mentioned, Jalapeño’s results are obtained without speculative decoding and Vera Rubin’s results use speculative decoding. Speculative decoding leads to a ~3-5x reduction in cost per token. When speculative decoding is implemented on Jalapeño, this will enable Jalapeño to serve tokens even more cost effectively.
How much speculative decoding improves throughput is workload-dependent. Yes, it can improve performance by 5x, but it can also slow down performance by 2x. It also requires a vastly more complex decode loop than the standard token-by-token decode. The tech journos didn't ask themselves if speculative decoding improves perf so much why wasn't it on by default? Perhaps, because speculative decoding is not optimal for OpenAI's ASIC? Perhaps that is also why they were only able to benchmark the not-very-representative single-user-decode case?
> it states that OpenAI choose not to disaggregate prefill and decode
They disaggregate INSIDE the chip, not by having separate machines for the 3 phases. the slides:
https://x.com/beffjezos/status/2092416851737518190
Maybe the money will still flow into this industry after all
I went down a rabbit hole after watching Dylan Patel on Dwarkesh today: https://www.youtube.com/watch?v=aV26V1UvkJw
I was initially just surprised by how bullish Dylan is on OpenAI/Anthropic and how bearish he is on China, despite Chinese labs getting closer to US SOTA while offering inference at dramatically lower prices.
So, I started digging while waiting for various day-job inference calls to return, ha.
Dylan says he spent years obsessively posting on hardware forums, moderating hardware subreddits, and running anonymous hardware blogs/videos before SemiAnalysis. But he also says most of that history is now gone, including from the Internet Archive, because he asked for it to be removed.[1]
In a 2024 interview he described his post-college job as “data science” around hurricane/earthquake/wildfire simulations for a financial company.[1] In a 2026 Sequoia interview he described himself as having been a “quant at a small quant risk firm” who generated $10M+ of “risk-free revenue.”[2] The Information reports that he declined to identify the employer and doesn’t list it on LinkedIn.[3]
Even harmless/silly stuff seems to drift. In February he said he kept bees for ~1.5 years. Today it was “few months, few months.”[4][5] I know, sort of silly and doesn't matter.
The Information reports that Patel owns stakes in ~20 startups in the same ecosystem SemiAnalysis covers, organized a $50M Fluidstack SPV, and is now targeting a $400M venture fund.[3][6]
And, in a 2022 HN discussion about SemiAnalysis disclosures, after saying his reports had moved smaller stocks by 20% in a day, Patel wrote: “If I thought I could move the stock, I'd make the position in the morning alongside my clients, and publish shortly after.”[7]
I don’t know that any of this is false or that anything improper happened (I’m definitely not claiming that). More that 1-2 of these things would just be odd. Taken together, though, they made me question how much trust I was putting in the broader story.
The dynamic of reminds me of crypto, WeWork, Theranos, Citron, etc. Once enough important people validate someone, things that would normally invite basic diligence somehow stop getting questioned.
[1] https://www.dwarkesh.com/p/dylan-jon
[2] https://sequoiacap.com/podcast/dylan-patel-of-semianalysis-w...
[3] https://www.theinformation.com/articles/dylan-patel-semianal...
[4] https://www.latent.space/p/dylanpatel-cooking
[5] https://www.dwarkesh.com/p/dylan-patel-3
[6] https://www.theinformation.com/briefings/exclusive-semianaly...
[7] https://news.ycombinator.com/item?id=31065646
Once models are more or less interchangeable, the price of LLMs will drop to essentially the price of energy required to run them, and the big labs will be able to run them cheaper than anyone else.
With competition we will actually have the fair split, whatever that is, and thus much lower prices.
At the moment, to have a big AI firm, or really AI firm at all, you need to be blessed by NVIDIA, in the form of receiving circular financing for your compute. They know that their prices aren't fair, or competitive.
Commoditization of inference is the end of that. The end of the mega-premium on inference hardware, and it's good not only for people who like running their LLMs, but it's the first step towards commoditization of training.
If what you said were true, you would hardly see people complaining about the quality of Opus 5 or good writing from Sol. But people do.
They are all converging.
I’m not an expert in metallurgy by any means, but this seems really off. There are many recipes for steel and varied processes that also impact the final product.
Each of those categories then has different grades of quality.
Tokens are a lot more like steel than oil, especially since a lot of tokens are used as structural material in the form of code.
They also fawn over the chip’s TDP when all other chips have to support 16 bit floating point and thus must run much hotter.
They make the classic mistake of equating max TDP with in-use-watts, and praise this magnificent (fictitious) performance per watt at FP8 with other chips’ max-TDP at FP16, which draw twice the power.
Evidence that the IPO can’t be far away.
Yeah, stopped reading there, this is obviously some deranged sam altman paid blog post, I can't wait for the bubble to pop just so his newly launched chip falls flat on his face.
The other side of the conversation is the idea that large models in DCs on custom silicon is the future. Maybe for enterprise? But consumers will eventually (10 yrs) have affordable hardware designed to run crazy-good local models (more RAM + higher bandwidth). That will take pressure off of datacenters, but also reduce AI profits, and move that money to consumer chip/device makers. Apple is once again the biggest winner. Nvidia consumer chips might get cheaper, but nerfed, to encourage datacenter use where they make more money. I'm hoping AMD can stop being terrible at software so that when we finally have their better hardware we can actually use it.
- If they use either evaporative cooling or a liquid-cooled heat exchanger, that uses tons of water consistently. This requires less energy (it's mostly passive) so you use more water.
- If they use closed-loop water cooling and/or heat pumps/electric chillers, that uses much less water - at the DC. But it does require more energy to circulate the water, run fans, etc. If you are using more energy, where is the energy coming from? It's coming from power plants, which require... you guessed it... more water (e.g. thermoelectric, hydroelectric, geothermal, concentrated solar). They need water in order to generate the power, and lots of it. Coal, natural gas, nuclear, and concentrated solar, all use steam to generate energy. Nuclear also uses water to cool the reactor. And water is used extensively to extract coal, oil, and natural gas. Geothermal uses water in the ground.
You can't not use a ton of water in one fashion or another. It just depends what method, and on what end the water is used. And the crazy thing is, most new datacenters are being built in places with extremely little water. Guess how that's gonna work out as the planet gets hotter?
I don't know why I got downvoted to hell for stating facts every datacenter architect knows. HN be HN'in.
One objective of the project might be simply to provide credible negotiating leverage when dealing with existing suppliers like NVidia. You don't have to deploy at scale for that to work, but you do have to look like you could if pushed hard enough.
But then that means you have no actual moat against the behemot, right? Your competitor can move into the market as soon as they want to, at much better cost (so at slightly better price)... and Nvidia certainly can adapt much faster around hard hardware specs innovation than a new entrant ever could.
One answer is they're quite good at poaching talent.
They should GTFO with this crap.
Create barriers to computing for ordinary people while milking businesses for tokens.
There's a reason why Micron and Nvidia are the crown jewels of American technology right now and for the foreseeable future.
To give some context, Intel started making DRAM, I think they were actually the company that came up with modern memory techniques. They exited the market and pursued a more lucrative moat with CPUs.
If somehow you get around the patent issues, you're now faced with huge research and development costs, fabs to build, processes to sort out and all of that has very high failure rates.
Last time I checked Micron was the largest patent holder in the world and even for them this is a hard area where they are number 3 in the market.
If the rich decided to buy all drinking water, you would probably be saying that's okay, making water is difficult, shortly before dying.
I remember when Tesla was building its own inference chips, and after about 2 years and billions spent, the whole effort was scuttled b/c they simply could not keep up with the iteration and R&D cycles of dedicated chip companies. I suspect the same will be the case with OpenAI vs Cerebras + Nvidia/Groq
Hes got a press release.
The issue is, baking something to silicon requires discipline and about 2 years.
This isn't something you can just change your mind on halfway through. Trust me, I know. You need a clear vision of what you want to support, why and what bits of a chip you need to achieve that.
Man, if only someone made like, chips that could lots of different calculations all at the same time!
Even in nvidia land rubin + LPU does a similar thing.
It has its downsides of course - if your traffic swings prefill heavy to decode heavy, you can't suddenly use your lpu for prefill. With GPUs they're totally interchangeable. Tradeoffs.
That sounds quite like...nonsense?
Chip companies work on years-long cycles. They know today what are they launching 4-5 years from now.