Ultimately, the problem is that we are not good at creating stable APIs.
If we did have stable APIs, then once something (a feature, or even entire software) is written agains this stable API it would be available forever.
This is IMO one of the greatest challenge the software engineering field needs to solve.
It isn't just stable APIs but part of that is consistency of signatures, methods and usage.
APIs used to be cleaner and more atomic, today they seem like very leaky abstractions and breaking changes seem to be all the rage.
Previously, breaking changes were seen as a serious thing as it meant the API signatures/functionality was missing and needed to be broken.
Today, breaking changes is just part of the "move fast and break things" fad that developers don't even try to make a stable/consistent API anymore.
Lots of the problems are that development/engineering is no longer in the hands of developers/engineering/product people or designers, it is pushed by project managers and bizdev/MBAs/marketing/finance which is backwards. Good products should be built, with lots of thought into long living APIs and limited breaking changes and maintenance, that is then marketed and sold because it is value, instead we get the exact opposite. Rarely does software today ever get to a stable phase before the platform is version two'd, and there is no time added for software quality or solid products, features and underlying systems are in constant harvest and replanting.
Stable consistent APIs don't help lock-in and they don't sell books, conferences, or allow large companies more control. For larger companies to keep control they need to break standards, and regularly break their own platforms to keep people updating and invested, it sucks and is why software is so bad today.
I do not think this is true, Windows has a lock-in on desktop explicitly because it has a stable API that allowed it to amass a huge library of software that keeps people relying on it to get that software working.
For the largest majority of people, Windows' worth is its stable API (even if they do not realize it).
True but in terms of development platforms for Microsoft, it is constant churn.
When they have a monopoly or near monopoly like Microsoft does with the desktop and Windows, less need for churn to keep people on it but lack of competitors. Azure has gone through lots of churn because of heavy competition.
Maybe competition, which is good, causes all this breaking software changes but change is good as well. Software change should just be a little more stable and developer/engineer/product/designer led not just the money/marketing/oversight pushed.
Unfortunately standards get wrecked when platforms are going for lock-in and control, Microsoft/Google etc are all to blame for that. It isn't the engineers pushing for that mostly. Standards at least promote some level of stable APIs, but they don't lock-in and so big fish and sharks eat them up and break them.
However Apple is greatly responsible for HTML5 and browsers today, as well as the handheld gaming revolution on mobile.
Webkit, Canvas, SVG, OpenGL ES + WebGL (through Khronos sponsorship) and more were open sourced or pushed by Apple.
Chrome/Safari basically every browser that is good came from Webkit and Apple opening that up. Granted that originally game from KHTML and KJS libraries from KDE but it exploded with Apple support.
I left them out because they still have some grace left.
Google and Microsoft squash standards and Microsoft is famous for "embrace, extend, extinguish" which Google now employs.
Microsoft has been better recently but only because they lost developers during the Ballmer era and they have a desktop monopoly and their new OS is cloud where they are competitive with Azure so they are pushing standards more and cross platform to win back developers.
Google is entering their Ballmer era currently.
Apple doesn't have the power for that yet with developers, though they are not engineer/open focused typically and especially now in the COO era and have their own Ballmer vibe currently, but they are still riding on some fuel from Webkit, Canvas, SVG, HTML5, OpenGL ES + WebGL support which really helped handheld gaming and web gaming.
ABI compatibility is also important. For example, the Linux kernel takes it very seriously but Linux user space generally doesn't. One of the reasons why application developers like to package dependencies together with the program in containers.
On the one hand it gives hardware vendors an incentive to contribute their drivers directly into their kernel but on the other hand most Android phones are now restricted to one old kernel version due to the lack of ongoing support from SoC vendors.
I assume that the majority of Linux kernels are running on Android phones and due to this lack of stable ABI for drivers these kernels are all outdated or even unmaintained.
> I assume that the majority of Linux kernels are running on Android phones and due to this lack of stable ABI for drivers these kernels are all outdated or even unmaintained.
No, it's because the hardware manufacturers won't upstream their drivers. All they have to do is send the code licensed under GPLv2 and it will be updated and maintained.
Stable ABI for drivers will result in the Windows situation: hardware vendors will just release one driver for one Windows version and one processor architecture. They don't seem to care very much so why should people optimize for their comfort? The least they could do is release free software so that the people who do care can make it work well.
My laptop has a backlit keyboard with RGB LEDs. The only driver for it is a single proprietary Windows driver coupled with a buggy and slow application. I doubt these things will ever be updated.
I actually emailed the manufacturer about this. I asked for technical information in order to make it work on Linux. They didn't or couldn't help me. So I reverse engineered the keyboard, made a Linux program to control the LEDs and the result was much better than the software they wrote for Windows. At least it doesn't take a full minute to start up.
There is no point in making these companies comfortable. People should do the opposite: it should be as expensive as humanly possible to not release the drivers as free software. If they insist on maintaining proprietary drivers out of tree, then they should absolutely pay the maintenance cost. Kernel hackers shouldn't have to spend one second thinking about how a change will impact some crappy proprietary driver that's not even in the kernel tree.
What if they are ok with this? What if the device vendors use this lack of stable ABI to put an expiration date on their products?
Android phones don't get more than 3 years of support and my fear/tinfoil theory is that the vendors abuse the good intention of the kernel devs to deprecate old devices faster.
They can release drivers for a specific operating system and processor architecture. When those change and the driver breaks, they aren't going to spend time and money updating the driver for an old product they aren't even selling anymore. They'll just tell people to buy their new product instead.
> Stable ABI for drivers will result in the Windows situation: hardware vendors will just release one driver for one Windows version and one processor architecture.
Uh, I dare say Windows hardware + driver compatibility is pretty damn good, so this doesn't really resonate with me... at all. Has not being able to install Windows XP on a modern machine left you bitter...? Or have I been living in a bubble?
> Has not being able to install Windows XP on a modern machine left you bitter...?
Not at all.
I have been burned by drivers that only worked in 32 bit Windows XP though. They simply couldn't be used in newer versions of Windows. The manufacturer couldn't care less, they think I should just trash the unsupported hardware and buy the newest product instead.
Linux doesn't have this problem. Once a driver makes it into the kernel, it can be maintained, updated, fixed, refactored, improved, backported to older kernels, recompiled for new architectures. It's just so much better.
> Has not being able to install Windows XP on a modern machine left you bitter...?
Yes. The topic at hand is how newer software can screw things after all, which causes people to want to stick to older versions. The same applies with Windows 7 BTW - or whatever your favorite Windows version is.
But it also goes in the other way: being bitter that i am not able to install drivers on Windows 10 for older hardware that should otherwise work.
Windows hardware + driver compatibility is a nightmare fueled by proprietary duct-taped one-off drivers supporting only one or two devices each.
One of the reasons for this is that you can get away with anything on Windows, because everything is proprietary. There are headsets targeting Windows users that require kernel-mode drivers to work. Which, of course, is super fragile and will break at some point, rendering your expensive headset useless.
It is indeed horrifying, and the particular products I'm thinking of are indeed from a series of gaming headsets. There are other categories of headsets, speakerphones and the likes that are equally awful, though, which is why I see the problem as universal to products targeting Windows:
There is no scrutiny to stop this idiocy, and the average consumer has no idea why this is bad, and no idea that those blue screens were because of a headset.
Having a connection to said company, I'd rather not name them though.
> There is no scrutiny to stop this idiocy, and the average consumer has no idea why this is bad, and no idea that those blue screens were because of a headset.
Yeah. Microsoft doesn't exercise any control and the result is hardware manufacturers force bogus software down the throats of Windows users. The Linux community can count on Linus Torvalds and the kernel hackers to provide that scrutiny and people trust Linux a lot more as a result.
I always cite capcom.sys as an example of why random companies shouldn't be allowed to write drivers:
It is, right until you want to emulate an audio device that vastly differs from what you have (say, presenting a headset as a surround sound device and do in-driver signal processing).
> No, it's because the hardware manufacturers won't upstream their drivers. All they have to do is send the code licensed under GPLv2 and it will be updated and maintained.
You probably know this, but that's most definitely not true. After they get their code correctly licensed, they still have to get the code up to the standards of the mainline kernel, get it through reviews, plus at least act like they intend to maintain it. Probably actually do that too, if you want any future changes to go in.
> All they have to do is send the code licensed under GPLv2 and it will be updated and maintained.
That seems like a dubious assertion. I've heard several cases of driver regressions in new kernel versions, and because of the model that means you can't just use the older one.
Even stable APIs may need security fixes. I'm not convinced a world where APIs continue forever and their exploits do too would be much better than what we have. And I say that as someone who has just accidentally lost an expensive, fully-working piece of audio hardware to an ill-advised software upgrade.
I remember Microsoft around 2001 fixing a security issue in Windows Media Player. This security issue was combined with a feature which added DRM support to Windows Media Player. I'm still puzzled by decisions like these, and why they are even legal to begin with.
That wasn't the point I was trying to make. I was trying to point out that ideally you could just leave the old API there for old software to use and build a new API for new software to use. If it wasn't for security bugs you could leave the old API untouched. Sadly, security issues prevent this and after a while it becomes uneconomical to supply even security fixes. This is why hardware that relies on software has an operational life shorter than the hardware itself has.
And that is exactly my point: API changes are rarely—if ever—caused by security fixes, and thus are not the cause of breakage, despite you attributing the impossibility of stable APIs to it.
A fixed piece of software can have a limited lifetime for only two reasons: Dependence on ephemeral infrastructure (e.g. cloud services that disappear or change), and assumptions that fail over time (e.g. "the device will never be used after 2020" + hardcoded calendar).
Only the first can be caused by API changes, and the API changes are usually always caused by voluntary product rewrite and infrastructure deprecation, not by security issues.
In other words, the problems spawn from entirely voluntary and technically unnecessary actions, and you cannot claim that is caused by some unavoidable law of nature, such proliferation of security issues.
I was not trying say that security fixes cause breakages. I was trying to suggest that security fixes cost money and therefore old APIs would get withdrawn for monetary reasons (if they weren't simply replaced by something new and shiny which is the usual reason).
If old APIs required no maintenance it would be fairly simple to leave them around.
However, APIs are built on an ever deeper stack of APIs that go all the way back to the Operating system and through internet protocols (APIs) right across the internet. Each API gets updates and fixes and things get deprecated. Bit-rot sets in and things stop working.
My original comment was suggesting that if everyone just left their old APIs around forever then the world would be filled with unpatched APIs with security flaws and I was not sure that would be a better place to be than the world of bit-rot that we have now.
Many APIs are poorly designed and by design easy to use in insecure and inefficient ways. APIs also can't anticipate the future. APIs need to have a way to be deprecated and eventually discontinued.
Software that actually works of course, but that only happens with stable APIs, ABIs and certainly not with removing them no matter how planned and preannounced that might be (both because most will ignore things since what they do already works and doing something different just to get the same results is most of the time a waste of time but also because in many cases the software is simply not possible to upgrade/change due to other issues, including the developer not existing anymore).
This is IMO one of the greatest challenge the software engineering field needs to solve.