The real best case scenario is using LLMs to help build deterministic systems. Instead of asking an LLM to do some task that you know will be repeated, instead ask the LLM to build a program (Python script or whatever) to do the task.
If it's a one-off script/program that doesn't require additional "domain knowledge", sure. But what if you need to give as context your whole backend repository because you need to take into account a few business rules? Why give anthropic/openai access to my "secret sauce" (e.g., company private repos)?
In that case, it's way better to simply write the code yourself.
From all possible concerns, "giving access to anthropic/openai" to your "secret sauce" is the least important one for 99% of the companies out there.
No, is not way better to simply write the code yourself. Most of the code is written faster and better with Claude Code or equivalent. Very niche code is better written by hand. Even then, you're probably better off nudging something like Claude Code in the direction you need it to go. There's nothing interesting about writing it yourself unless you're still learning to code (in which case is a learning exercise for you, not only about the outcome).
Your sauce is not worth stealing to OpenAI. They have bigger fish to fry
In fact - had there ever been a single case in the history of the internet where a service provider of any kind - hosting, storage, email, search, anything - has ever stolen intellectual property or trade secrets from a user and benefitted from it? Has that literally ever happened?
Making systems fully deterministic ignores the entire purpose of having agents involved.
IMHO the best of both worlds option is agents working with deterministic CLIs. Where the agent does the reasoning (and text generation) but uses CLIs to carry out all of the actions (issuing refunds, unblocking accounts, or whatever).
It's possible to get very reliable and consistent work out of agents when they're using well written prompts with well designed CLIs.
I think it just depends on the workflow. A lot of times (probably the majority of the time) a well crafted deterministic process is what people want.
Sometimes you can have a multistep deterministic workflow with a decision that needs made somewhere in the middle, which is where an LLM that can call tools is useful.
I think there are much less repeated tasks where you would want a fully agentic process (but there are many ad hoc tasks where that is exactly what you want).
At some level everything an agent does is through a "programmatic interface" (tool calls).
Some people might use skill-based scripts, MCPs, or some kind of raw access to a database. My point is that well designed CLIs are the optimal programmatic interface, for many reasons.
At Big Tech Company I Work At the LLM is quite happy to make raw API calls. If it thinks the data is big, then it'll write a Python tool to do it.
The reason crafted backing CLIs are useful is you can guide the LLM towards stuff that is immediately useful rather than hoping the nondetermism can separate the wheat from the chaff.
Take CI: is it interesting to know which tests passed? Maybe, but probably not. What is really interesting is what failed. Instead of having the LLM go out and talk directly to the CI system, write an intermediate CLI that filters out less actionable stuff by default, and have a flag that'll deliver the full dump if necessary.
It's a skill to do this stuff, and it's a lot of hard won experience than something I think is easily teachable. You kind of have to feel out your model and how it "thinks" about solving problems.
And then a new model version comes out and you have to learn it all again!
>Making systems fully deterministic ignores the entire purpose of having agents involved.
That sounds backwards to me. I hope that most places don't see "having agents involved" as the ultimate purpose, but will use agents where it makes sense, i.e. when deterministic systems fall short.
The best case scenario of LLM is transforming input into output where both are languages and accuracy doesn't matter, e.g. "rewrite this poem in pirate speech."
I've already commented on other posts that having LLMs build deterministic and testable tools is the real unlock.
Even for things like customer service, a LLM that analyzes customer support transcripts and then updates your call tree to better route people is a huge win.
Because typing “code” takes time and significant amounts of it.
We are slowly waking up to the fact, which was always true, that “coding” is just a fanciful preparatory task in order to appease the spirits properly so that we may invoke the spirit of what we are actually after: a live, running process that does useful things. Code is completely useless when separated from that fact.
Typing it is a complete waste of time unless getting up close and personal with it will result in some kind of useful and actionable improvement in you or your understanding. Knowing when it does and when it does not have this property is a skill of its own.
> Typing it is a complete waste of time unless getting up close and personal with it will result in some kind of useful and actionable improvement in you or your understanding.
I believe this is the general belief about basically every human skill, that if you stop doing the technical fundamentals you get worse at understanding the activity. The question is whether coding is like sailing a square-rigged wooden ship, which became completely useless knowledge after the invention of the steam engine, or if it's like playing an instrument, which while technically unnecessary after the advent of MIDI and other tools, absolutely hurts your ability to arrange, compose and perform if the skill is neglected.
For my money: I think the AI scenario is more like the latter, but "humans are worse at coding" isn't the consequence I see coming. I worry that in ten years we will be awash in software that's impossible to understand. I don't think that's happened in any human industry ever. Someone has always understood how the machines are built, even if they're very remote from the users of the machine.
The sci-fi novel A Fire in the Deep starts with describing a Software Archeologist, who digs through millennia of strata of layers of indirection and I think we could end up needing that one day.
No serious programmer is regularly bottlenecked by typing speed. Even the ones who type slowly.
If you find yourself writing repetitive code you should consider adding a layer of abstraction. If your language isn't powerful enough you can write a code generator.
> Typing it is a complete waste of time unless getting up close and personal with it will result in some kind of useful and actionable improvement in you or your understanding.
Like, perhaps, understanding that it is free of security and functionality bugs.
That is one of the things code does. It also communicates the developer's thoughts about how that process should work to others. If the latter is neglected, the code becomes very difficult to collaborate on. Very few lines of code that are written are "write once". Mostly they're changed, repeatedly, over time by many people. The live, running process is a very temporary entity by comparison. Yes, it needs to exist and do useful work. No, it is absolutely not the only thing that matters.
It depends on where you're using AI. If you're working on a project for yourself or in a tiny company. Then sure, writing the code probably was your bottleneck. But at mid to large companies writing code is maybe 50% of the job, and the other 50% is the process around it. All those processes are the bottle neck, no matter how fast you can write the code. And this was a bottleneck I was hitting well before AI.
I'd put it even lower than that, since there's also the "understand the problem space" portion outside of the external processes and before writing the code.
Yeah, so again the point of text editors is you shouldn't be typing all that much with them. That's why we use them. Macros. Shortcuts. Metaprogramming. Snippets. Identifier completion.
Were people actually typing out the full text of source code before LLMs? Why?
> Can you type a hundred lines a second? If not, then it is.
No one has ever needed to do that for something that is new. And if it’s not new, you want to do it repeatedly with some guarantee of reliability. Not just in an uncontrolled manner.
That is why we have snippet systems, macros and code generators. And the best with code is to solve problem once and reuse the solution. Which we have done with libraries, frameworks and supporting software.
This is such a delusional take it's borderline trolling. Code is an expression tool to precisely describe a process that does useful thing. Typing prompts is not too different from writing some very vague code, which is arguably a waste of time by itself.
> Typing it is a complete waste of time unless getting up close and personal with it will result in some kind of useful and actionable improvement in you or your understanding.
I would argue that this is nearly always the case. I don't think people really understand programs that they've only read at more than a very superficial level. This is why I tend to make (temporary) small changes, printlns, etc. when exploring a new code base: it aids greatly in understanding how a program actually works.
And it's even worse (in my experience) with LLM generated code, as it tends not to result in particularly understandable code. It is a lot like LLM generated prose: it often looks entirely reasonable at a surface level, but has a of weirdness/incorrectness hidden beneath the surface. But that surface level makes it very hard to avoid glossing over the details when reviewing the code. For this reason, I personally find it's much more effort to carefully review code than it is to write it.
Humans make mistakes all the time, but their code tends to naturally be structured for human understanding (to some degree based on skill/experience) because they themselves needed to understand it to write it.
I think LLMs are very useful tools, but after quite a lot of experience using them, I think it's generally better to use them as a sounding board, or to help you get unstuck or remove points of friction. Using them to write all of your code (at least for me) seems like a net negative.
I also think it's extremely easy to overestimate how much time they save. It feels like they're a productivity boost because it takes less intense focus to implement something. But I've experienced several instances where actually writing the code myself would have been both quicker and have resulted in better code.
All that being said, it can also be really hard to not write all of your code with agents once you get used to it. There's also a kind of slot-machine-like effect where you write a prompt, excited for the result, and when it doesn't quite come out right, you think "ah just one more prompt and it'll be good." It's hard to see when you're actually doing it though.
It's also weird to me how much people think typing is what the LLM is replacing. Typing was never the hard part. It's the translation of the high-level idea into an unambiguous process that's hard. That's also the valuable part, that requires thinking through the edge cases and consequences of decisions, and that just gets glossed over when using an LLM unless you rigorously review what the LLM has done.
At the end of the day there's a real tradeoff to be made, and it's worth being conscious of what's being given up.
If you already know what the inputs/outputs are, why should you spend days or weeks of your life typing it out rather than giving it in a well-specified and tested form to an LLM to get it done a hundred times faster?
Because it’s rarely so black and white. Knowing the inputs and outputs is merely the first steps, you need to think about the transitions too as they have their own costs.
Those costs don’t disappear and it’s truly naive to think they don’t matter. Take security issues, they may arise because what you thinks was the input is merely a subset of the true input range. And the extra possibilities lead to unforeseen behavior.
A lot of programming is about ensuring that the input and the output are the sets defined in the specs. And the rest is that the transition/relation is the right tradeoffs of performance, correctness, and costs.
Libraries like Guidance guarantee that the output of an LLM will be syntactically correct (i.e. it will be valid JSON or whatever output format you are wanting). They do not, and fundamentally cannot guarantee that the data contained in them is actually correct, and cannot make the actual behavior of the LLM "well-specified". Or, as you put it, trumps voice wrong.
Getting a PhD is almost never worth the foregone wages if your goal is to be in private industry. For sure, people should get one if they are interested in research, or specific jobs that are only available to people with PhDs. But otherwise it isn't something to get into half-hearted.
This is only true when you compare the PhD field degree to something like finance where you can make real money at 23 years old. If the alternative is a bs for a path where most people end up going PhD, you will be working for like $20/hr most your life. You will probably be breaking even with what the PhD stipend would have been anyhow and you aren't getting any healthcare benefits.
It's true for computer science and any STEM discipline. Even if the PhD eventually does account for increased wages (it often doesn't), you usually have to work decades at that higher rate to make up the foregone wages.
I would only ever get a PhD due to intrinsic interest, or if it unlocks a specific type of job I want that would be unavailable without it. It's a bad investment to get a PhD to get higher wages (in America, I don't know the labor market in other countries).
I agree, it seems like the current most popular languages and frameworks will become ossified, because they have the highest amount of training data. It's hard to see a future where Python and JavaScript aren't the most popular languages to use (assuming LLM-assisted development is the norm moving forward).
LLMs can be pretty conmpetent at languages that have zero training data, at least to the extent that those languages use features/ideas that are familiar. I wrote a toy language/compiler and AI can write code for it competently.
Isn't this just grading on a curve, which has been done probably as long as universities have existed? The key is the instructor making sure a high standard is met (which seems to be the crux of the issue).
Yes it has been in practice for a long time, but it's now being used to push clear fail cases into passing grades just to meet quotas. Prior it was used to adjust for particularly difficult assessments, and was closely monitored.
You should probably use software to do such large transformations (especially in dynamic languages). In Python LibCST is available, not sure what exists for Ruby.
To be fair PwC is a very well established initialism. It would be like saying HTTP but actually referring to something other than the well understood meaning of that thing.
Humans also make mistakes in ways that other humans can understand or expect. Sometimes LLMs make mistakes in a way that makes you say “no human would have ever done that”.
The original post said “in college”. It might be true for PhD candidates halfway through their program, but that’s like 0.5% of college students. The vast majority of students are leagues behind their instructors in domain knowledge.
I wouldn't say leagues behind, but otherwise I think we are on the same page, though I guess I worded it wrong. It is common for a couple students in any class to know more than the instructor in some niche part of the field even though the instructor has much more knowledge overall.
People game benchmarks for fake internet points to get their favorite web framework to the top of the list. I'm pretty sure they will do it for billions of dollars.
reply