most people don't wanna do that. there are plenty of people who would infect people with crypto botnets

fantastic token savings and performance... but unlike grep it's probabilistic search on search terms.

is that an issue? the tiny model might not surface something important

It's not probabilistic, and exact matches will always be preferred over non-exact. So if you search for a function name this will surface it.

It was. The internet really has been filled with abject shit and social media is bots talking to bots.

give it a skill that runs a timer in the background and every 4.5 minutes says "ping? pong!"

Interesting idea. I suppose one could also have response settings (e.g. max response tokens) to ensure the model doesn't waffle on and run up costs. In a best-case scenario "ping" would be one or two input tokens and a "pong" response would be one or two output tokens, so the cost of the operation would be the preserved context size times the cache read cost (one could avoid doing a cache write since I believe the cache read would reset the platforms cache timer).

It would be interesting to graph the cost/savings of this approach based on context length, percent cached, etc.

The UI for this is a bit tricky, I could mark conversations as "active" and then do the ping/pong dance on only active conversations and up to some determined max cached (e.g. 1 hour).

This kind of thing is harder for regular end-users to understand following the change removing reasoning details.

you're going to save money by having your own physical servers?

Yes, a lot of money in fact.

Agree. Also because of the way AI writes, it takes SO LONG to read through it (they're trained on blogspam where the page tells you the author's life story as well as the bloody history of bread before telling you how to bake it)

That's why in this case I usually ask to another AI to make me a short summary with the main points. I wish the human behind the looong article idea chooses to publish a short summary directly instead.

I mean, I personally just saw some stuff inside dollar signs and went "huh, weird choice of delimiter"

I enjoy driving (basically) wherever I want to though.

I don't want to have the freedom to go places determined by some faceless multinational, according to my subscription. Or via some "safety" regime.

Self driving doesn't mean forced self driving, there will always be a manual driving override. But your insurance may be more expensive.

that's a lot of energy consumption.

For a more ecologically conscious alternative, I recommend carrying a handful of sparkplugs.