Even trivial things like an ETL pipeline for processing some data at my work fall into this category. It seemed trivial on its surface, but when I spoke to everyone about what we were doing with it and why (and a huge amount of context regarding the past and future of the project), the reason the pipeline wasn’t working properly was both technically and contextually very complex.
I worked with LLMs on solving the problems (I always do, I guess I need to “stay sharp”), and they utterly failed. I tried working from state machine definitions, diagrams, plain English, etc. They couldn’t pick up the nuances at all.
Initially I thought I must be over complicating the pipeline, and there must be some way to step it back and approach it more thoughtfully. This utterly failed as well. LLMs tried to simplify it by pruning entire branches of critical logic, hallucinating bizarre solutions, or ignoring potential issues like race conditions, parallel access to locked resources, etc. entirely.
It has been a bit of an eye opener. Try as I might, I can’t get LLMs to use streams to conditionally parse, group, transform, and then write data efficiently and safely in a concurrent and parallel manner.
Had I done this with an LLM I think the result eventually could have worked, but the code would have been as bad as what we started with at best.
Most of my time on this project was spent planning and not programming. Most of my time spent programming was spent goofing around with LLM slop. It was fun, regardless.
those llms all learned to code by devouring github. How much good code is on github and how much terrible code is on github.
My favorite thing is writing golang with co-pilot on. It make suggestions that use various libraries and methods that were somewhat idomatic several years ago but are now deprecated.
