|Ross Addis on Tue, 5 Oct 2010 22:26:18 +0200 (SAST)|
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
|[GLUG-chat] Re: Some thoughts on packaging and why desktop Linux is a failure|
Anyway, I feel that the desktop is a dying paradigm. Mobile and cloud(and the various iterations) is where it's headed.
--Ross On 2010/10/05 10:24 AM, Damjan Jovanovic wrote:
Hi Of all the operating systems I've used, Linux has always seemed to me to hold more potential than any other. Yet this potential is often misused, abused, overvalued, misunderstood, underused, or ignored, resulting in what is at best a second-rate system, and at worst a hackers-only club. The example I want to discuss this time is this: a few years back Ian Murdock blogged about how package management is "the single biggest advancement Linux has brought to the industry" (http://ianmurdock.com/solaris/how-package-management-changed-everything/). I am of the exact opposite opinion: 1. Binary incompatibility Binary compatibility on Linux is just sad. The original a.out binary format was bad, so Linux moved to ELF. But ELF is not much better. With ELF, symbol lookup in dynamic libraries is "process scoped", meaning the same symbol names in independent libraries that happen to be loaded into the same process will clash (eg. 2 versions of libpng get loaded into the same process, both define "png_info_init" -> memory corruption -> crash). Windows and MacOS use "library scoped" symbol lookup to avoid this. Software development is fundamentally distributed, and nobody controls what names people use in their library symbols. So when different libraries with the same symbol come together, or even different versions of the same library get loaded into the same process even indirectly, this causes unpredictable crashes. A few real world examples: * Mozilla.org official builds of Firefox written in C++ (libstdc++.so.5) uses GTK+ (written in C) loads the Simple Chinese Input Method written in C++ (libstdc++.so.6) -> memory corruption -> undebuggable crash. * libgnutls can replace the openssl APIs by redefining them with its own API, if the real openssl is loaded even indirectly -> same API and symbols but different ABI -> memory corruption -> undebuggable crash (http://www.linux-archive.org/fedora-development/148626-libgnutls-openssl-real-openssl-conflict.html). There still isn't any general solution for this. There's the GNU libc specific RTLD_DEEPBIND flag to dlopen(), but that's only for run-time dynamic linking which is not used generally. There's also symbol versioning as a possible solution, but that's an obscure ELF-only feature and only libc uses that. Michael Meeks sent a patch for the -Bdirect option to ld (http://sourceware.org/ml/binutils/2005-10/msg00437.html) which would have fixed this, but Ulrich Drepper rejected it for no good reason; Solaris already uses and benefits from that feature. Distros have had to work around this issue in common cases, eg. the Ubuntu package apparently manually symbol versioned the various libdb's. The only conclusion that can be drawn is that ELF dynamic linking, as it is today, is fundamentally broken and should not be relied on in a desktop environment where you want to run anything from anywhere. Of course binary incompatibility is not just a matter of how ELF works. Other problems are numerous: * many libraries don't preserve API/ABI compatibility between versions * few/no libraries support "compile on new, run on old". GTK for instance uses macros that invisibly lock you into>= the version you compiled against, so you have to compile against a very old version to be able to run against any later version. * libc has a number of different options it can be compiled with, each of which alters the API/ABI needed by the application (eg. thread-local locales). In practice Linux distributions don't agree on options to compile libc with, meaning you can and do get applications compiled on one distro that won't run on another. * ELF is very slow (Windows OpenOffice on Wine starts up faster than Linux OpenOffice) so various hacks like DT_GNU_HASH were added to speed it up; these also broke compatibility between distros (eg. a while back pcsx2 compiled "Linux binaries" on Fedora, and Ubuntu users couldn't run them). * It is near-impossible to make ELF-based "portable apps" that can be run off a CD or a USB stick and store all their files on it, or applications that can be installed anywhere after compile-time, because ELF has hardcoded library search paths, and there's no easy way for an executable to look up its own path and load resources relative to it. Windows has GetModuleFileName(), the closest we have is a symlink in /proc/self/exe that exists only on Linux and works only for executables - a library has to - this one is really good - take the memory address of something inside itself like a static variable, then search the memory address ranges in /proc/self/maps for that address, then take the path following that range - all exotic, highly obscure, not well known, Linux-only tricks, that work only if /proc is mounted. On Windows, you can usually choose where to install something, on Linux, since nobody knows/cares about this, most software isn't relocatable after compile-time, and thus distro packages virtually always need to be installed into /, as *root*! 2. Rigid dependency lock-in Loose coupling is a virtue in software engineering: we have exactly the opposite. Binary compatibility problems from the above point lead to the only logical solution: we can't (more like won't) solve the problem globally so let's solve it locally, ie. things can work but you have to compile against the known-good environment, which locks you into its ABI, and others can then in turn compile against you. This leads to tight coupling - software components that shouldn't really depend on each other are tightly bound by the compiled binary interfaces. A distribution is thus built top-down, like a pyramid, where things on top depend on exact versions of things beneath. Nudge just one stone at the bottom, eg. downgrade the libc version, and the entire distro needs to be recompiled. Newer versions of apps thus link to newer versions of libraries, lockstepping together. Great when you want the latest version of everything, terrible when you want to backport. Eg. try installing the Ubuntu OpenOffice 3.2 .deb on Ubuntu 6.06 from 2006 - you'd have to replace half the libraries, but try installing it on the 6 years older Windows 2000 - it just works. 3. Centralized repository We had problems with binary compatibility, so we solved the problem locally by compiling specific versions against each other, and now we have a bunch of interconnected packages that work with each other but probably won't quite work anywhere else: a "repository". Repositories only support a single version of a package. Users needing a different version are stuck. I recently played a game of "Scorched 3D" - I couldn't connect to the server, turned out my version was wrong. The repository didn't have the newer version, the website didn't provide .deb files, and I wasn't in the mood to compile from source. I installed and played the Windows version of the game on Wine. This is the sad state of the art of software installation on Linux today: if it's in the repository, it's quicker and easier to install than on any other operating system, but if it's not, you're in for it. And this hasn't changed from the early 90's. A good quote I read somewhere is how "It's easier to distribute our [open source] stuff on Windows than on Linux". As for there being 30000 packages for Ubuntu, a lot of software I use isn't packaged, and 30000 is still less than a third of the applications on Sourceforge alone. Packaging scripts are usually written and managed by people that didn't develop the software being packaged, and even if they're initially correct, they get outdated, and then cause bugs themselves, like this one I found in Ubuntu's OpenOffice package: https://bugs.launchpad.net/bugs/479973. This bug affects all Ubuntu and probably all Debian users and it's been completely ignored for years. Bugs in distro Wine packages are legendary: packages used to put wineserver in /etc/init.d because it sounds like a server (it's actually automatically started by wine and runs as the current user), and Fedora and OpenSuSE both recently shipped completely broken 64-bit Wine on x86-64 architectures. Many common packages are broken if you dig a little, eg. the help file in k3b on Ubuntu 10.04 doesn't open because it needs some KDE service which isn't a dependency to the package, but then manually installing that service gives the error that the help path doesn't exist. 4. Particular distribution Now before the package is in the repository proper, come the distro-specific hacks and patches to make the package conform to the distro's view of the world, or reject it outright: changes to artwork, translations, integration with the distro bug tracker, addition of files, removal of "badly behaving" files or tools (eg. odbc-config removed from Debian and Ubuntu unixodbc package because it outputs paths in a way that isn't "Debian enough"), or even complete rejection of the package from the repository based entirely on a political ideology, something even Microsoft never did in the peak of its reign (eg. autopackage was rejected, as was a build of Gaim just for using an autopackage tool that would "defeat Debian's automated dependency tracking", as was the OpenWatcom compiler for not being licensed freely enough, and I'm sure there are many more). By controlling access to the repository - the only easy way to install software - distros have an effective weapon against software they do not like. Who are they to dictate what end users should be able install and run? Distro control is everything that freedom - the whole point of open source - is not. You might say there are legitimate issues that distros can't include something in their repository - eg. OpenWatcom had licensing issues that would affect its hosting in a repository - but that's also an argument against the existence of the repository in the first place. Packages made by distros deviate from the official software in unpredictable ways. Recall that serious security issue only last year or so, caused by Debian excluding the OpenSSH blacklist from its package. Distros often take in all bug reports, even for software they didn't write (since they patch it), but never propagate these upstream. I was shocked when I went onto Fedora's bug tracker and found many valid bug reports for Wine that hadn't made it into Wine's bug tracker. I even found and fixed some bugs in Wine by looking through these distro-specific bug trackers for interesting bugs, but there's more than 300 distros, I can't visit them all... There's no predicting the API/ABI/semantics of a piece of software when a distro provides it; Eclipse SWT developers had an issue where they work around a bug in certain versions of GTK by checking the GTK version and then enabling the workaround for broken versions, but Debian backported the GTK fix to older GTK versions, so they were stuck with a "if (GTK_version< X || distro == debian)" situation - imagine if they had to test for all 300+ distros in their code. Of course even that is not enough - each distro could change the package any way they like at any time. Distro packaging thus works only works if distros package absolutely everything - which they don't, by a long way, and never will. 5. 3rd party software is foreign This is end result of the previous problems, and the biggest problem in and of itself. Applications all come from your distro's repository, right? Because there is virtually zero infrastructure to support third party, unpackaged applications. Even forgetting binary compatibility problems, there is no standard way to install an application (autopackage, klik, zeroinstall, binary tarball, source tarball, self-extracting shell script, binary file, package: is it compatible with your distro+distro version?), no standard install location (/opt, /usr, /usr/local, ~/.local, anywhere), no standard automatic update service, no standard way to check/specify/request dependencies, no standard init.d services or even standard file format for new services (thank you Upstart), no standard fonts, and generally a very shaky foundation to stand on. Distros apparently fantasize that all software is in their repository, where it generally works with the distro's own infrastructure. It doesn't help we have more than 1 desktop environment. Luckily freedesktop.org standardized some some basic desktop functions, but many things are not standardized yet: thumbnailing, URI handling, default file open associations, MIME type actions, shared settings, keyring, and file browser right click menu integration, just to pick a few. Sadly the difficulty of developing applications independently of the whole distro repository system means that it generally isn't done. People either don't develop a "Linux version", target only a single distro version (eg. RHEL 4), or add onto the great lumbering beast of the distro repository, thus keeping the fundamentally broken trash alive due to sheer momentum rather than competitiveness and technical excellence. Distro repositories thus act as a deterrent to improving these problems. ISVs have been complaining for years about the difficulty of developing for the desktop (http://www.kegel.com/osdl/da05.html). And every open source project is ultimately an ISV: some like Eclipse and Inkscape even want to be seen that way. So these problems bite them all, some harder than others. Eclipse published their problems with the Linux desktop at http://vektor.ca/osdl-meeting3.txt, and while some of these issues have been fixed since 2005, the fundamental problems haven't changed. http://linuxhaters.blogspot.com/2010/04/year-later.html hits the nail on the head: "The big players in the linux desktop are the distributions. They distribute stuff. They're somewhat ok at distributing their own stuff. But they're supremely awesome at making it hard for third parties to distribute their stuff. It's the distributions (and more precisely, the fact that there are so many of them) that make the actual distributing hard. Go figure." *** A Linux distribution is thus not a solid, stable platform - it is merely a snapshot of a bunch of packages in their current state, compiled against each other. A Linux desktop in a distribution is not a general application environment - it is a fixed-function appliance that works for a predefined set of functions, but if you change it even a little, it will break in unpredictable ways. The whole fixed rigid repository structure is perfect for servers - and totally wrong for desktops. Case in point: it's easier to make an entire Linux distro for a special purpose application (eg. Mythbuntu, Trixbox, Untangle, EBox) than to make a software package that will install and run on any version of any distro. It is easier to make software to target all Windows versions since 95 than to target all the versions of a single distro over the same time period. Software and frameworks that do work well on a wide range of new and old distros have to go to unbelievable lengths. Java for example still has code that emulates connect() on UDP sockets on Linux 2.2 kernels which didn't support it properly, its selector framework uses the blazingly fast epoll on 2.6 kernels and falls back to poll on older kernels, it has manually written workarounds for quirks in each window manager, it uses delicate dlopen() logic to defensively load external functionality, and its C code often contains copies of definitions from system header files rather than including the dodgy native headers. The image I/O libraries all read and write PNG, GIF, BMP, and WBMP using 100% Java code instead of the unreliable native libraries; only JPEG is loaded using a Java-private known-good libjpeg. Java emulates alpha blending on GTK which doesn't have it by drawing each widget on a black and then a white background and then mixing the results. And Java has to ship private copies of some libraries it uses instead of linking to the dodgy ones distros ship (or might lack). The insanity of the current system has even resulted in people dissing Java for not linking to the native libraries (http://fnords.wordpress.com/2010/09/24/the-real-problem-with-java-in-linux-distros/) instead of understanding why it can't. And over time the each distro for itself way has resulted in 64 bit being done differently by each distro (/lib32 vs /lib vs /lib64 is all different on different distros). Open source people look to innovation and improvement in *distributions* (eg. the next Ubuntu), *desktops* (eg. KDE 4, Gnome 3), and *frameworks* (eg. Mono) whereas it's always been about the *applications*. Microsoft is now reporting record profits (http://mybroadband.co.za/news/business/13978-Microsoft-reports-record-high-sales.html). Mac OS, not Linux, has become the alternative OS - all you hear in the media is hype about the latest Apple products. So, are you waiting for Gnome 3 to revolutionize Linux? Are you waiting for Ubuntu 10.10 to make everything work? At most, these are going to make minor, cosmetic changes (menus, themes, windicators, the "Ubuntu font"). Looking for the "right distro" for you? They're all the same, they use the same upstream software, just packaged, patched and themed in various ways for no purpose other than to be different on that distro. As predicted in point 1, portable apps on Linux are a disaster: http://www.linuxinsider.com/story/70921.html And then some wonder why the "year of desktop Linux" is still not here... Regards Damjan Jovanovic (ducks and hides ;-)
-- To unsubscribe: send the line "unsubscribe glug-chat" in the subject of a mail to "glug-chat-request@xxxxxxxxxxxx". Problems? Email "glug-chat-admins@xxxxxxxxxxxx". Archives are at http://www.linux.org.za/Lists-Archives/ RULES: http://www.linux.org.za/glugrules.html