Meetings are too easy to game. I worked with a bunch of new managers from LEGACY_CORP and learned the extremes of how to BS.
As an example, if you think there might be any sort of pushback, just never stop talking. Once a manager talked for 35 straight minutes to answer a question on an unpopular decision. By the end there were no follow-ups because everyone was too confused and checked out to care.
1. Everything is specified, written and tested by me, then cleaned up by AI. This is for the core of the application.
2. AI writes the functions, then sets up stub tests for me to write. Here I’ll often rewrite the functions as they often don’t do what I want, or do too much. I just find it gets rid of a lot of boilerplate to do things this way.
3. AI does everything. This is for experiments or parts of an application that I am perfectly willing to delete. About 70% of the time I do end up deleting these parts. I don’t allow it to touch 1 or 2.
Of course this requires that the architecture is setup in a way where this is possible. But I find it pretty nice.
> A messy codebase is still cheaper to send ten agents through than to staff a team around. And even if the agents need ten days to reason through an unfamiliar system, that is still faster and cheaper than most development teams operating today.
I’ve been on 2 failed projects that have been entirely AI generated and it’s not that agents slow down and you can just send more agents to work on projects for longer, it’s that they becoming completely unable to make any progress whatsoever, and whatever progress they do make is wrong.
>I’ve been on 2 failed projects that have been entirely AI generated and it’s not that agents slow down and you can just send more agents to work on projects for longer, it’s that they becoming completely unable to make any progress whatsoever, and whatever progress they do make is wrong.
This rhymes a lot with the Mythical Man Month. There's some corollary Mythical Machine Month thing going on with agent developed code at the moment.
Same here. I have now deleted 43k and counting lines of my codebase. There is no point in putting any AI code into production anymore as it almost always uses none or the wrong abstractions.
When you try to throw more agents at the problem or even more verification layer, you just kill your agility even if they would still be able to work
Case in point, just this morning I contributed a one-line change to an open source repo and the CI started failing.
I asked Claude (Opus High Effort) and pasted in all the logs. I went back and forth and it very confidently made over 20 separate changes in the repo, none of which fixed the issue. Eventually I stepped in and figured out it was a versioning issue.
I fear what would happen if I ran “10 agents for 10 days” on this simple issue.
The more I work with AIs (I build AI harnessing tools), the more I see similarities between the common attention failures that humans make. I forgot this one thing and it fucks everything up, or you just told me but I have too much in my mind as context that I forget that piece, or even in the case of Claude last night attesting to me while I am ordering it around that it cannot SSH into another server but I find it SSHing into said server about the 5th time I come back with traceback and it just fixes it!
All of these things human do, and i don't think we can attribute it directly to language itself, its attention and context and we both have the same issues.
Right, but when humans are writing the code, they have learned to focus on putting downward pressure on the complexity of the system to help mitigate this effect. I don't get the sense that agents have gotten there yet.
>...I see similarities between the common attention failures that humans make. I forgot this one thing and it fucks everything up, or you just told me but I have too much in my mind as context that I forget that piece
Or you're working in a trendy, modern open-plan office and between the noise from the salespeople nearby talking loudly to customers on their speakerphones, some coworkers talking about their medical issues, and the guy right next to you talking loudly to himself in a different language, you're unable to concentrate at all on your programming task.
> Potentially, yes, but as with other software, you need to know AND have (automated) verifications on what it does, exactly.
Yes, but even here one needs some oversight.
My experiments with Codex (on Extra High, even) was that a non-zero percentage of the "tests" involved opening the source code (not running it, opening it) and regexing for a bunch of substrings.
I'm wondering how much value there is in a rewrite once you factor in that no one understands the new implementation as well as the old one.
Not only is it difficult to verify, but also the knowledge your team had of your messy codebase is now mostly gone. I would argue there is value in knowing your codebase and that you can't have the same level of understanding with AI generated code vs yours.
The point of a rewrite is to safely delete most of that arcane knowledge required to operate the old system, by reducing the operational complexity of it.
> “Now we know what we were trying to build - let’s do it properly this time!”
I wonder if AI will avoid the inevitable pitfalls their human predecessors make in thinking "if I could just rewrite from scratch I'd make a much better version" (only to make a new set of poorly understood trade offs until the real world highlights them aggressively)
I was involved in a big re-write years ago. The boss finally put the old product on his desk with a sign "[boss's name]'s product owner" - that is when people asked how should this work the most common answer was exactly like the old version. 10 years latter the rewrite is a success, but it cost over a billion dollars. I have long suspected that billion dollars could have been better spend by just fixing technical debt.
That's correct, the more I work with AI the more it's obvious that all the good practice for humans is also beneficial for AI.
More modular code, strong typing, good documentation... Humans are bad at keeping too much in the short-term memory, and AI is even worse with their limited context window.
I don’t think it’s necessarily bad to be unprofitable but definitely weird to be sending 100% of your revenue to what is essentially your main competitor
What are you talking about? We are in this position precisely because of government regulation prioritizing cars. Most of American cities are still only zoned for SFH within the city limits. It’s nearly impossible to build in any other way because of regulation. Not to mention the many many subsidies for the auto industry.
The thing that frustrates me most about libertarians is how everything they don’t want is regulation or government spending, but everything they do want is provided by the grace of god or something.
The comment that you cite doesn’t actually have any supporting citations…
> Rules are rules, unless someone thinks that bikers are special and aren't subject to the same laws the rest of us have to follow.
I would love for enforcement of cyclists who break the law to increase. But that would require actually providing working infrastructure for bikes.
You can’t write someone a ticket for not riding in a bike lane when literally every single block someone has parked in the bike lane. If there was a place where every single day on every block, someone parked in the only road for cars in a major city and on a major road artery, this would be a national news story. But this is what happens on weekdays at ~8:45am where I am in LA.
Every city also has completely different rules about cycling, which very few people bother to know about. In many cities it’s fine to ride on the sidewalk.
Absolutely idiotic article and title. Obviously it's possible to reidentify a person if you have 15 demographic attributes if you don't specify which attributes you use. I can do even better, I can reidentify 100% of people, with only their name, DOB, fingerprints and SSN. The fact that DOB and zip code are in the dataset make this result completely trivial.
A couple years ago, I got into an argument on reddit where someone claimed that any mapping could be recovered "using deep learning techniques" (e.g. if you take 3*0 = 0, you can get back that the original value was 3 with no other information except for the value "0"), and that obviously I was just too stupid to understand deep learning if I couldn't see that.
I mean, yes, some people are factually incorrect. But I think the general idea is more like, if you have a massively over-determined system of linear equations, you can omit many of the values and still be able to recover them all from the remaining values and knowledge of the equations.
And it's not intuitively obvious which combinations of values allow you to recover which other ones.
For context, this was when ISPs were planning on selling data, and someone was collecting donations saying they'd reidentify senators internet history. I said that people shouldn't donate to them, because it wasn't even clear what the ISPs would release. Their point was it doesn't matter what the ISPs release, they could reidentify anyone with deep learning.
> And it's not intuitively obvious which combinations of values allow you to recover which other ones.
I think it's pretty intuitive that Zip Code and DOB are identifiers. That's why they count as such in HIPAA, and are used to demonstrate identity by governments, credit cards, etc.
Personally I think this stuff just poisons the well when it comes to discussions of privacy. I think the goal is to remove the expectation of anonymity by claiming that it's never possible.
> I think it's pretty intuitive that Zip Code and DOB are identifiers.
It's great that you think that, but basically no company uses that definition. Most company privacy policies don't consider combinations of information when making this determination. E.g. your billing address might be personal information, but your zip code by itself might not. Similarly, IP address (with or without last octet), wifi SSID, location data, browsing history (or attributes derived from browsing history), and so on. Each individual piece of data isn't enough to personally identify you, so the privacy policy often doesn't have to be applied to it.
E.g. after reading the Google privacy policy[0], can you tell what protections your zip code and DOB have? Will Google treat them as personal information or personal identifiers or not?
> I think it's pretty intuitive that Zip Code and DOB are identifiers.
Sure, but what about job title? What about job title when someone's job title is "mayor" or "fire chief" and it's possible to deduce from other information what city they're in? Or someone's job title is just "Governor of the State of California"?
Any collection of random independent characteristics become uniquely identifying once you have enough of them. Then all the attackers need is another database with the same characteristics that also includes names or other identifiers, and you can associate the missing fields from one database with the other.
> Personally I think this stuff just poisons the well when it comes to discussions of privacy. I think the goal is to remove the expectation of anonymity by claiming that it's never possible.
It's not that it's never possible, it's that it's only possible if we don't feed these centralized databases enough information to uniquely identify people. So we need to stop doing that.
It’s a serious paper by serious people — the headline may oversimplify, as headlines often do, but the fundamental point is right on: deidentifucation is very hard.
Great article, and fantastic to see a spotlight on an issue that I've thought a lot about.
The sad part is that to a lot of scientists and researchers, software/software engineers isn't something worth paying for. It's not uncommon to see "programmer" jobs that are looking for 3+ years of experience that offer <$15 dollars an hour in the US. Sometimes they're "volunteer intern" positions. Of course the people who end up filling these positions aren't usually actual developers, so the software gets built poorly, eventually gets scrapped, and the cycle continues.
Management also hasn't really evolved past the 90's. Non-technical scientists often want 100% of control and to make each decision, but don't want to spend any time on it. This means developers often have little to no specs to work with, but spend all of their time guessing about what the scientists want, and having to go back and fix everything after.
>“That’s really the tragedy of the funding agencies in general,” says Carpenter. “They’ll fund 50 different groups to make 50 different algorithms, but they won’t pay for one software engineer.”
This is the crux of my frustration. It's not even 50 different algorithms often. A lot of the time, 50 different research groups will be working on very similar programs, and none will be able to deliver a working version.
Though the article mentions that research funding does exist, clicking on one of those funding pages and looking through their examples reveals that only ~1/10 of their websites are actually still active, and they aren't old sites. Again this goes back to the whole "scientists don't value software thing". I've seen scientists happily sign off on spending $20,000+ on hardware components that would usually cost <$100 to make, but balk at contributing $50 yearly to support open source.
I got lucky that I managed to find a place where I get paid fairly, and my boss is actually technical and can manage tech projects well, but these places are few and far between.
A lot of the points you bring up re related to cost. Here's the thing about cost, let's say it costs $100k/year to hire a good software engineer capable of writing scientific code (able to program and test complex algorithms, write HPC code, turn whitepapers into code), which might even be an underestimate depending on how benefits are paid out and the area. You can also fund 3 more grad students for that kind of money. The grad students will directly convert a PI's money into authorships while the software engineer's contribution will be only indirect, and likely take years to pay off.
Plus, with only a single software engineer, there's a good chance you get unlucky and end up with someone clueless/lazy. You would probably need 3-4 software engineers to make a functioning team with best practices and hedge your bets against accidentally hiring someone who sucks. So now we're talking 10+ grad students.
Open source software is a bit different because many labs can band together to fund things they find useful. But again there are still issues with cost-effectiveness. I'm guessing most lab contributors to OSS would want some sort of quid-pro-quo which may not be realistic for all OSS projects. And by funding OSS you are also funding competing labs' abilities to use the same features you use, which is good for science in general but not good for people's careers sometimes
You're right, and more generally all of the issues are related to the fact that scientific incentives don't typically align with good development. At the end of the day, over a period of 3+ years, I'd rather have the results of 3-4 software engineers compared to 10-12 grad students. However, for <3 years, I'd choose the grad students. Pretty much every incentive in science (e.g. grants, awards) prioritizes being prolific over a short period of time.
Here in Oxbridge there's pretty generous funding for hiring software engineers to maintain some biotech software, and much lower pressure to publish. I know many open positions. And they tend to fund you for very long time. I know people working on said positions for more than a decade. Sometimes 20 years.
The problem is more pay gap with industry. Even though they tend to pay well by academic standards (e.g. no PhD required, yet pay is much higher than a senior postdoc), the salary is still way below what industry in London or Oxbridge offers.
Furthermore, you tend to be surrounded by non-technical people which may be tough in the long term. Nobody appreciates what you do. Not even your boss, who may know zero about computers.
The bottom line is that positions end up being vacant for long time and tend to be filled by biologists with a bit of coding experience, underqualified IT people or, rarely, really competent individuals that want a break from industry.
> The problem is more pay gap with industry. Even though they tend to pay well by academic standards (e.g. no PhD required, yet pay is much higher than a senior postdoc), the salary is still way below what industry in London or Oxbridge offers.
I know sufficiently many good programmers who would love to do scientific programming (because they love science) and would immediately accept less pay.
The problem, in my opinion, rather lies in the non-monetary work conditions. For example, in Germany it is nearly impossible to get permanent employment contract when working at a scientific institute and doing something remotely related to scientific work. Even worse: you are not even allowed to work more than 6+6 years (before and after doctorate) in a fixed-term employment position at a scientific institute If this time is over, you are not even allowed to take any non-permanent contract at a scientific institute (the infamous/insane Wissenschaftszeitvertragsgesetz (WissZeitVG)).
No programmer (even if (s)he has a great passion for science) will be willing to work under such extraordinarily bad conditions.
Here it is a bit better. Most positions I know of are de facto permanent. On paper they are not, as most labs go through 5-year funding cycles. So there's a tiny chance of loosing funding. It's quite rare for big labs.
Besides, places like MRC have created permanent research assistant positions. Which are actually permanent and put zero pressure on publications.
As you say, connecting with interested and talented programmers is another problem. I feel that Nature Jobs postings, which were already a big leap forward for rusty uni administrators, are not good enough.
Why hire an engineer? Why not walk over to the CS (might have a different name) and talk to a professor there. They can set you up with plenty of undergrads who need this experience, and they should be able to guide them into something that is maintainable long term.
Note that I said should there. How to write maintainable programs seems to be lacking in research area.
Because undergrads are on average very bad programmers. They're both enthusiastic and don't know what they don't know. Good programmers are above all very experienced and disciplined and clear thinkers. Undergrads are stressed by their remaining workload so they have split responsibilities, and they just can't put the necessary time in.
I was such a CS undergrad working in a lab once upon a time. I don't think you really want that because the undergrad will probably only be working for like 4-15hr/week potentially for only a single semester. For a summer position, sure it's 40hr/week but still only for about 10-15 weeks.
And still, you're getting a generic CS undergrad's caliber of work and responsibility which I would say on average is not great. They might not be as familiar with version control, best practices, etc. and could just end up writing code just as bad as the scientists.
I think if hiring a team you would need at least one somewhat experienced full-time software engineer to act as team lead/PM for the other developers, whether fulltime or students.
Yeah, picking up undergrads (or even grad students) from the CS department is not a surefire way to end up code that follows best practices, is well maintained/documented, etc. and I'd probably argue if your team leader doesn't have that experience, you're more likely to end up with something that doesn't follow great practices (especially with regards to any sort of test suite).
All the replys about undergrad quality are correct. I stand by my statement though: we need to figure out how to solve this problem and research is sorely lacking.
This would be a good solution, but from what I've seen with psychologists and statisticians, it's unlikely to happen for reasons I don't fully understand. Another thing is that undergraduates often learn by adopting the norms of the institution that they're in (e.g. using version control, linters, etc) but when they're brought in as the technical person, they don't have that opportunity to improve (This is my personal anecdote as that cs undergrad at one time).
I’ve been that CS undergrad in the past who aided another department’s research, and the quality of my work back then was every bit as bad as you might expect. Of course I, like many of my CS peers, arrogantly thought I was doing great work at the time...
For sure, talented undergrad CS developers exist, but in my experience there are far fewer of them than many people might think. Experience counts for a lot, especially when trying to deliver even average quality software work.
As for something made by undergrads that is “maintainable” to a standard broadly comparable with an experienced developer? Maybe if you get lucky...
The fact that they need the experience should already tell you that they're not yet competent at what they are needed for. They should be put on low-impact projects that don't matter, or low-impact projects that do matter with solid mentoring. They should not be made to write software for something that is high-impact for you with little to no mentoring.
The thing that terrifies me about the state of affairs is what happens when the software gives the wrong result because it's poorly written? If a scientists has input B and expects output X, and writes code that accepts B, and it happens to output X, chances are they'll write the paper and send it off for publication. Even if there's a bug in the code, and the real output should have been Y. I genuinely believe that this happens more often than most of us would find acceptable. Hell, there have probably been instances of the code outputting Y and the scientist scratches their head like, "Hmmm, that's not right, something's wrong," and will massage the code until it eventually outputs X, and then they'll say, "Fixed it!" and move on.
I know for a fact it's happened at least once, and led to a scientific controversy that lasted for decades. Unfortunately, I don't remember the specifics; if someone recognizes my vague description please step in with a citation. But one group of scientists published a paper saying a certain dynamic system behaved in a certain way, and a second group published a paper saying it behaved in a different way, and the two results were completely incompatible with each other. Significant public disagreement ensued. One group published their code, and the other group attacked it saying it was poorly written, etc. The second group did not release their code. Decades later, some other scientist at the second institution released the code, and after a code review, it was found that the data was incorrectly initialized. They needed to initialize the particles with "random" initial velocity vectors, but the scientist who wrote the code didn't know how to do it correctly, and wrote an ad-hoc algorithm that gave the initial velocity vectors significant bias along an axis. But the paper was already written, peer reviewed, published, and cited, so even though the paper was wrong, the result was still accepted by (half of) the scientific community. AFAIK the paper was never retracted.
I have seen and fixed code that was first written as a prototype, then used as is in a follow on project and so on, until it turned out to be used for deciding on whether or not to grand certain subsidies. Except whoops in one spot it interpreted kilometers as meters without dividing by 1000. Along with literally dozens of other outright bugs. But nobody cares, really.
I'm using technical in a broad sense. One effect of scientific research becoming bigger is that roles are becoming more specialized. Often the people leading the research have primarily spend time writing grants or papers, and less senior professors and post-docs will carry out the actual tasks. When I first started I was a bit shocked at how many of the big names knew basically nothing about the process of how their own research is carried out.
The effects are more pronounced depending on what field we're talking about. For example, in physics, I'd imagine most people have at least the fundamentals of programming down, even if software design may be lacking. In the field I work in (Neuroimaging), a lot of PI's are doctors or neuropsychologists, and might barely even know how to use a computer.
What an incredibly pointless thing to argue about. I think we all knew what he meant and, this being a specialist forum, we benefit from using shorthand in speech that is a lossy representation of ideas to those not initiated.
I hate to have to create this conversation fork but I really wish people wouldn't make comments like this. They're so low signal.
I disagree - I think reducing anyone who doesn’t happen to program to being ‘non-technical’, with the implication that they don't know what they're talking about, is insidious.
It's clear he means "Scientists who don't want too much input into the engineering decisions". You're reading an implication of competency where there is none.
Two other points to consider--equipment is a one-off expense and staffing is a continuous expense. The other is that the pots of money for equipment and staff may be different....
Completely right, I've been trying to hype up the "Github sponsorships" program as a way of changing the thinking around software (e.g. tacking OSS onto grants as required equipment), but haven't found much support.
When said scientists want to build some new equipment, they get involved and they use professionals. When said scientists need to build their new lab, they hire professionals. When they need code for their research projects they?
> The sad part is that to a lot of scientists and researchers, software/software engineers isn't something worth paying for.
I have to ask how you came to this conclusion. Did you have anecdotes from them supporting this or did you just conclude "they are not paying for it therefore software is not worth paying for to them"?
I think the original poster is applying the "cheap talk" heuristic. People may say they value things, but we see what's really valued by what they pay for (given the resources they have)...
The term "privacy" has become a hollow word that people use to criticise anything they don't like.
Generally, I see the concept of "you shouldn't be able to know anything about me, but I should be able to know everything about you" everywhere.
Notably in tech circles, this is prevalent in how we see hacking.
People who hack into other people's computers, IT people who snoop onto employee accounts, and people who support personal data breaches like the Fappening (saying stuff like "if they don't want to have their images become public, they shouldn't have taken them"), and publishing of personal emails (e.g. Wikileaks), all have had conversations with me complaining that Facebook is violating their privacy rights. None have recognized that what they are involved in is way, way worse than anything that Facebook has done.
As an example, if you think there might be any sort of pushback, just never stop talking. Once a manager talked for 35 straight minutes to answer a question on an unpopular decision. By the end there were no follow-ups because everyone was too confused and checked out to care.
reply