Nice set of experiments! I appreciate that you're running benchmarks on real object storage setups to validate rapid design variations. (Meta-note: I love how agents have recently made this kind of experimental-research work possible with much less human time investment.)
I've been doing some experiments of my own in a relatively similar space, also focusing on S3/Tigris-backed SQLite on ephemeral compute, also with B-tree aware prefetching (see https://github.com/wjordan/sqlite-prefetch).
I think the idea of storing grouped pages together to optimize read-locality is interesting. Note that it steers in the opposite direction of the temporal locality that a format like LTX/Litestream uses to provide transaction-aware features like point-in time restore. The tradeoff also involves significantly greater write amplification (re-upload the entire page group every time a single page dirties), heavily favoring cold-read-heavy workloads over mixed-write or warm-read workloads.
The query-plan frontrunning is a very novel experiment as well, discovering in advance that SQLite is about to run a full-table scan seems like a very useful optimization hint to work with. I'd love to see experiments validating how much of an improvement that offers compared to simple reactive prefetch (which takes at least a couple page faults to get up to speed).
Fantastic comment, thanks for jumping in. I have well-tuned replies here lol given we're working on exactly the same problems!
First: your prefetch solution is precisely what I have in the roadmap for the next level of optimization: B-tree introspection at the page-child level not the table level. I haven't launched it yet as I only realized the potential in the past couple days and I wanted to focus on stability before showing this to folks. I am 100% going to try to use sqlite-prefetch approach in that experiment. I'm curious what kind of results you're seeing.
On write amplification: you're right, re-uploading an entire page group when one page is dirty feels inefficient. This tradeoff is intentional because turbolite believes all PUTs are equal, and it is aimed at optimizing for queries on cold databases with bursty reads, not write-heavy workloads (though writes work too). Checkpoints are ideally infrequent due to upload latency, and the page group sizes are tunable (default 256 64KB pages = 16MB uncompressed, a few MB compressed to S3). For the use case I'm targeting, the read locality wins dominate. That being said, turbolite does let the user choose when to checkpoint locally (durable on disk) vs to S3 (atomic commit in case of disk crash or new compute).
On LTX temporal locality vs page-group spatial locality: agreed, they're different design goals. LTX optimizes for transaction-aware features like PITR. turbolite optimizes for cold query speed from object storage. You could imagine a system that does both (WAL shipping for durability + grouped pages for reads), which is roughly where my roadmap goes. In fact, WAL + immutable page groups on checkpoints gives a much faster restore time than Litestream's snapshots+WAL if only because uploading the entire snapshot on each commit is slow. I'm starting to explore embedded WAL shipping in my walrust project https://github.com/russellromney/walrust.
On frontrun vs reactive: I have very specific benchmarks for this I think you will be intrested in. The tiered-bench binary has a --plan-aware flag that toggles between prefetch schedule (reactive, similar in spirit to your sibling detection) and frontrun (EQP-based). Look at the benchmark/README.md for detail. On 100K rows, local to Tigris, EQP is:
- simple queries (mutual friends): 1.3x, less to gain
The wins are largest on cold index lookups and SCANs where reactive prefetch has to discover the scan through sequential misses. Frontrun knows the full plan upfront and fires everything in parallel. For single-table scans, reactive catches up quickly after 1-2 cache misses, so the gap is smaller.
Finally - I included a query tuner CLI in the repo that lets you compare different prefetch aggression levels for a given query on given data. You may be interested in using this.
Thanks for the extra details, the frontrun benchmark numbers seem compelling for various cold-read use cases.
A system that combines both WAL frames with cold-read-optimized grouped pages is another interesting point in the design-space. Tuning the intervals separately could make it work well- frequent WAL-checkpoint uploads, and grouped pages only on higher-level compaction cycles for cold-read optimization on longer-lived objects.
Looking forward to seeing where you head with this!
Anthropic uses it across all their websites, here's a typical example where the effect is obvious as you scroll down: https://claude.com/solutions/agents
I could be wrong, but my simple guess is that it's become widespread in LLM-generated websites partly because of Anthropic's own style guides getting adopted through Claude-bundled skills and such.
It is partly to blame, yes. This is from Claude’s official frontend skill:
“Motion: Use animations for effects and micro-interactions. Prioritize CSS-only solutions for HTML. Use Motion library for React when available. Focus on high-impact moments: one well-orchestrated page load with staggered reveals (animation-delay) creates more delight than scattered micro-interactions. Use scroll-triggering and hover states that surprise.”
Who thinks like this? The last thing I want on a website is surprise. I want to do what I came there to do, the same way it worked last time, and then get on with my day.
On a Web site or anywhere else. Apple, Microsoft, "flat design," and peek-a-boo UI all insult the user and waste his time by turning an important tool into an Advent calendar.
That's a bit different since those are separate chunks of content rather than running prose (and they're mainly meaningless marketing fluff anyway). I don't find it all that annoying compared to the original article.
It's still incredibly insulting to waste the user's time trying to force them to read the page sequentially instead of being able to immediately scroll to the chunk of content they are actually trying to find. Especially if it is not the user's first visit to the page.
I used to inject a prefers-reduced-motion: true to most websites a few years back when this trend really picked up; and generally have all animations turned off at the OS level and at least on mac that injects reduced motion preference to the browser.
So for example on claude's website I get no animations, pretty good QoL improvement. Now webdevs HONORING prefers-reduced motion, that can vary.
This is a great example of LLM feedback loops. Anthropic's site uses scroll fade, Claude's training data includes Anthropic's site, Claude recommends scroll fade to users, those sites become future training data. The web converges on one aesthetic and nobody remembers choosing it.
Are the little hand animation graphics meant to flicker like they're an epilepsy test? That was so awful I didn't have brain power left to notice the fade scroll.
Sounds like a promising idea, but a blog post with bits of implementation code is not a "Show HN", there's nothing to try out here. See the guideline [1]:
> Off topic: blog posts, sign-up pages, newsletters, lists, and other reading material. Those can't be tried out, so can't be Show HNs. Make a regular submission instead.
> If your work isn't ready for users to try out, please don't do a Show HN. Once it's ready, come back and do it then. Don't post landing pages or fundraisers.
It's well known that an oatmeal diet lowers cholesterol (the article itself cites a 1907 'oat cure' in its intro). The new finding here is insight into the exact mechanism- a short-term, high-dose oatmeal diet (300g/day for two days) had significantly greater LDL-lowering effect than a medium-term, moderate-dose oatmeal diet (80g/day for six weeks), and they associated the difference with increases in several plasma phenolic compounds triggered by specific changes in the gut microbiome.
I'm left wondering what happens if you feed people 300g/day of barley, shredded wheat, brown rice, or any other wholegrain. For that matter, what happens if you do the same thing with legumes?
The experiment halved energy intake at minimum and still provided 30+ grams of fibre then kept doing it until the gut emptied, which I reckon most people would expect to nuke and replace the gut microbiome, but did oatmeal have any specific advantage?
Their hypothesis for the mechanism is "gut bacteria" but these people in the study all had a trifecta of "high" body weight (overweight? obese? not specified in this article), high blood pressure, and hyperlipidemia.
So we've got some unhealthy people, we cut their calories to less than half, we jack their fiber way up (most likely - we don't know their baseline diet but with those biomarkers we can make some educated guesses), we restrict the timing of when they eat and remove all junk food.
So is this oatmeal specifically? Fiber? Calorie deficit? Meal timing effects? Removal of processed food for two days?
The idea that you can "shock" your body to better biomarkers like this and have it last over a month is extremely cool, but I wonder how they can be certain that this is some oatmeal thing versus a general "eat way less and limit yourself to a food that is high in fiber" thing.
The low protein here is a problem when in a calorie deficit, for example, because if you don't have enough protein you're likely to lose weight as muscle mass rather than fat. If you could do the same technique with legumes your protein would be way better.
I did a similar unscientific experiment in the past where I did a juice fast for two weeks and had bloodwork done before and after (similar bodytype, unhealthy, overweight, high blood sugars, high cholesterol etc). Basically the doctor was shocked how all my numbers became the same as someone really healthy during the fast. So I think the lack of junk and calorie restriction is doing more than the fiber.
The paper does a better job explaining why oats were chosen:
> Oats offer an interesting and promising approach for treating MetS due to their unique composition characterized by a high fiber content, especially β-glucan, essential minerals and vitamins, and various bioactive substances, including phenols which exert antioxidant and anti-inflammatory effects that may improve metabolic function. Furthermore, oats are an accessible and sustainable food item.
I eat Bob's Red Mill steel cut oats for breakfast every day; 1/2c dry is about 88g. That's a pretty decent meal. 3.5x that is probably most of what you eat that day.
Yeah, the article showed that the high-dose intervention (modeled after von Noorden's famous century-old 'oat cure') is most effective. A large bowl of oatmeal (100g) all 3 meals for 2 days, 6 large bowls total.
6 weeks of 'oatmeal for breakfast every day' was less effective than 2 days of 'stuff yourself with oatmeal'.
It's quite a bit of volume, but it's "only" about 1000 Calories if it doesn't have any oils/sugar added.
I'd guess the easiest way to get it down would be to just blend the oats into water without cooking so you have something that you can just drink like water.
They had a control group who also went on a reduced calorie deficit but without oats and found that the oats eating group had a much higher decrease of cholesterol, it's in the article.
3/4 cup is about 315 ml, or another 315 g, for just over 400g cooked weight. The linked article does not make clear whether or not the 300g is dry or cooked weight.
The original paper describes two protocols, short-term and long-term (2-day and 6-week respectively) of 100g dry oats and 80g dry oats prepared for each meal. That's a generous but not outrageous serving size, and the former comes out to 300g oats per day over three meals.
> It's like a dozen paragraphs, forming one complete argument. Is this too much material to take in all at once, in this brave new TLDR tomorrow?
The issue is not that you cited a dozen-paragraph argument, it's that you inlined all the text directly into a series of comments instead of a link to the text on a separate page. It visually overwhelms the discussion thread and is disruptive to the broader discussion, which is not strictly against guidelines but generally seen as non-normative behavior.
The actual code being benchmarked is trivial and open-source, but I don't see the actual JuiceFS setup anywhere in the ZeroFS repository. This means the self-published results don't seem to be reproducible by anyone looking to externally validate the stated claims in more detail. Given the very large performance differences, I have a hard time believing it's an actual apples-to-apples production-quality setup. It seems much more likely that some simple tuning is needed to make them more comparable, in which case the takeaway may be that JuiceFS may have more fiddly configuration without well-rounded defaults, not that it's actually hundreds of times slower when properly tuned for the workload.
(That said, I'd love to be wrong and confidently discover that ZeroFS is indeed that much faster!)
For a proper comparison, also significant to note that JuiceFS is Apache-2.0 licensed while ZeroFS is dual AGPL-3.0/commercial licensed, significantly limiting the latter's ability to be easily adopted outside of open source projects.
This clarification is helpful, thanks! The README currently implies a slightly different take, perhaps it could be made more clear that it's suitable for use unmodified in closed source products:
> The AGPL license is suitable for open source projects, while commercial licenses are available for organizations requiring different terms.
I was a bit unclear on where the AGPL's network-interaction clause draws its boundaries- so the commercial license would only be needed for closed-source modifications/forks, or if statically linking ZeroFS crate into a larger proprietary Rust program, is that roughly it?
Also worth noting (as a sibling comment pointed out) that despite these assurances the untested legal risks of AGPL-licensed code may still cause difficulties for larger, risk-averse companies. Google notably has a blanket policy [1] banning all AGPL code entirely as "the risks outweigh the benefits", so large organizations are probably another area where the commercial license comes into play.
Related, "The Design & Implementation of Sprites" [1] (also currently on the front page) mentioned JuiceFS in its stack:
> The Sprite storage stack is organized around the JuiceFS model (in fact, we currently use a very hacked-up JuiceFS, with a rewritten SQLite metadata backend). It works by splitting storage into data (“chunks”) and metadata (a map of where the “chunks” are). Data chunks live on object stores; metadata lives in fast local storage. In our case, that metadata store is kept durable with Litestream. Nothing depends on local storage.
reply