What is an operating system?

April 13, 2018 by Lucian Mogosanu

The last time I visited the field of operating system design, I concluded that journey with questions. That line of questioning didn't solve much, but merely provided a sort of hierarchy that aims to reconcile current well-established designs, i.e. Unix, with proposed designs from current research, e.g. microkernels, unikernels. I now propose the idea that my previous proposal was utter, unmitigated bullshit. I further propose that current definitions of the term operating system are ill-conceived, and that the problem of how to properly design a general-purpose operating system is in principle unresolvable, first and foremost because there is no such thing as a "general-purpose operating system".

In defining the operating system, let us first defer to our priors. We begin with Tanenbaum1:

The machine language typically has between 50 and 300 instructions, mostly for moving data around the machine, doing arithmetic, and comparing values. [...] For example, a disk can be commanded to read by loading the values of the disk address, main memory address, byte count and direction [...] into its registers.

To hide this complexity, an operating system is provided. It consists of a layer of software that (partially) hides the hardware and gives the programmer a more convenient set of instructions to work with. [...]

On top of the operating system is the rest of the system software. Here we find the command interpreter (shell), window systems, compilers, editors and similar application-dependent programs. It is important to realize that these programs are definitely not part of the operating system, even though they are typically supplied by the computer manufacturer. [...] The operating system is (usually) that portion of the software that runs in kernel mode or supervisor mode. [...] Compilers and editors run in user mode. If a user does not like a particular compiler, he is free to write his own if he so chooses: he is not free to write his own clock interrupt handler, which is part of the operating system and is normally protected by hardware against attempts by users to modify it.

The Modern Operating Systems (MOS) definition goes bottom up from the hardware to the user, and delimits the operating system as the software component running in supervisor mode. I propose that this is at best a confusion in terms (and at worst bullshit): the operating system kernel is indeed the software component residing in supervisor mode, or ring 0, name it what you will. But on what grounds do we state that the kernel is equivalent to the operating system in its entirety? "Andrew Tanenbaum is a god in the field of distributed systems" is not an argument, or at least not one worth taking into account.

Then Tanenbaum sets to make politics instead of science. Despite being a proponent of the microkernel design, he decides that the user may modify their compiler, but not their kernel. To which we humbly ask, on what fucking basis? I can readily invalidate this argument on any GNU/Linux box that I own, and thus there it is, GNU/Linux -- not just Linux! -- is an operating system.

Then later (not quoted here) Tanenbaum mentions embedded OSes, but he does not mention, say, Sinclair BASIC, which is technically speaking an operating system, despite the fact that it does not reside in a special supervisor mode. How do we delimit the operating system from the application base in that case? Are ZX Spectrums (or say, Lisp machines) embedded systems and I didn't know?

Since MOS does not even get close to satisfying our intellectual curiosity, we move on to Silberschatz et al.'s2 Operating System Concepts, aka OSC:

We have looked at the operating system's role from the views of the user and of the system. How, though, can we define what an operating system is? In general, we have no completely adequate definition of an operating system. Operating systems exist because they offer a reasonable way to solve the computer problem of creating a usable computing system. [...]

In addition, we have no universally accepted definition of what is part of the operating system. A simple viewpoint is that it includes everything a vendor ships when you order "the operating system". The features included, however, vary greatly across systems. [...] A more common definition is that the operating system is the one program running at all times on the computer (usually called the kernel), with all else being systems programs and application programs. This last definition is the one that we generally follow.

Here the authors try to give a more rigorous definition, but they admit in failing to do so. Indeed, there is no clear delineation of where the operating system ends and the service or the application starts; but this is not a matter of acceptance, but one of properly defining our notions so as to not muddy our understanding of this universe. And yet, Silberschatz et al. end up confusing the kernel with the OS again, and then they go on writing an entire book on the subject3.

Given the lack of a proper definition, we, as educated people, are left with the unpleasant task of figuring one out ourselves. Fortunately though, the previous definitions are not entirely wrong, they are merely crooked. The operating system is, as Tanenbaum points out, a set of abstractions, given that e.g. most of the time the user doesn't have the time to deal with processor interrupts and task scheduling. These abstractions, as Silberschatz et al. point out, may be implemented as programs meant to assist a human operator, i.e. user4, in the operation of a computer system.

Fundamentally, the set of abstractions that make up an operating system comprises two classes of items: first, a language to be employed by the user; then, a set of operation tools, which tools may be either general-purpose, or on the contrary, very specific, depending on the system and on the task that the user wants to achieve. Does this description sound familiar? Well, it should, because we've just described the key elements of any general-purpose programming language out there. And that's fine, because formal languages and automata are isomorphic!

So, is the C programming language an operating system? That sounds like a stupid proposition. I don't know about C, but isn't Unix an operating system? All *nixes are specified by the UNIX standard, also known as IEEE Std 1003.1™, also known as POSIX.1. In fact, the UNIX standard specifies an operating system at two levels, both of which rely on a common set of definitions. The first level is the system interface, usually employed by C programmers to write e.g. system applications such as web servers; the language here is thus usually C, while the tools are the system API. The second level is the shell interface, which includes the shell command language, some built-in utilities such as echo and set, and a set of general-purpose utilities such as ls, awk, grep etc. This, all of this, represents the UNIX operating system -- notice how I haven't mentioned "kernel" or "supervisor mode" anywhere, because these are implementation details which may vary from one system to another.

What other operating systems do we have at our disposal, then? Let's enumerate a few more.

Is Java an operating system? Again, I don't know about the language itself, but Java, the JVM run-time and the libraries do form an operating system. The same goes for most Common Lisp implementations, which may in some cases require some extensions for e.g. threading, but are otherwise operating systems in their own right. What about Emacs then? Yes, Emacs is an operating system built on top of a text editor, and I'm not even joking: I am writing this text, using IRC, sending e-mails, organizing notes and setting up music playlists, all from Emacs, so that makes it very much a general-purpose OS.

Notice how all these systems make use of general-purpose tools, which is what makes them systems, as opposed to appliances, which use only specific tools. However, each operating system differs from another in some specificity or another. For example, TeX and its tools form an operating system, and while theoretically they can be used for any task, in practice they are used for typesetting. Similarly, web browsers are operating systems, generally exposing JavaScript as their language, but we don't use them for... in fact, people are using them for all sorts of applications nowadays, so who am I to judge. Similarly, Excel is an operating system for data-flow applications, etc.

This then kills the myth of the general-purpose operating system. If there were such a thing, then it is conceivable that we would now use GPOS instead of GNU/Linux, Chrome OS, MacOS or SLIME -- and when I say "if there were", I mean that there exists no such thing in the observable universe, and until someone proves the opposite, I reserve the right to remain very skeptical. And since there is no such thing, we are left with operating systems which are only somewhat general-purpose, but which in fact differ very much in the way they work and/or the tasks they are suited for. Their particular organization is of course interesting -- they are nested, or co-located on the same hardware, and they often communicate in various ways from which stem such and such complexities -- but all these are details that must be taken precisely as the details they represent.

Thus, this essay untangles some concepts, laying a (not in any way novel, mind you!) foundation for designing operating systems that do not suck. Please to properly cite these ideas in your paper, or else.


  1. Tanenbaum, Andrew S. Modern operating systems, 2nd Ed. Prentice Hall Press, 2007. 

  2. Silberschatz, Abraham, et al. Operating system concepts, 7th Ed. John Wiley & Sons, 2005. 

  3. The reader might be inclined to handwave this as empty sarcasm, while my observation is pointedly not this. This misunderstanding has led us to the mess in current systems; for example, did you know that the entirety of Android has at least two layers of memory allocation present? There's the buddy allocator in Linux, and then on top of it there is the kernel slab allocator, which serves kernel object allocation. But this buddy scaffolding is also used to expose mmap to user space. Then on top of brk and mmap the C library builds malloc, so three! Three levels! Then on top of this the Java Virtual Machine builds its own memory management system based on garbage collection, so four levels!

    Despite what some may say that "we need" "abstractions", this hell is mostly the consequence of politics and incompetence. Yes, each of these abstractions may be useful under some circumstances, but no, they're not (at) all needed

  4. Please to not confuse the current popular "user" with the proper meaning of the term. A user is one who makes use of a computing machine in order to achieve some particular task. Thus an accountant doing their Excel stuff is as much a user as a hacker hacking at the kernel, only in two entirely different contexts. The user versus programmer tribalism is... uncivilized, to say the least. 

Filed under: computing.
RSS 2.0 feed. Comment. Send trackback.

9 Responses to “What is an operating system?”

  1. #1:
    spyked says:

    Early comments on this article (somewhere else, since I didn't have a comment box back then) somehow managed to completely miss the point it was trying to make -- namely that the so-called "operating system", similarly to the so-called "social network" and many other terms corrupted by the plague of buzzwordism, is not fundamentally a technical term.

    When placed in a technical context, however, as it is here, the operating system becomes a tension point between a. the hardware on one side, and b. inasmuch as we're discussing personal computing, the human, on the other. The operating system is supposed to be a lube for interacting with machines -- which doesn't make much sense when the other side is not human, but for the sake of metaphor, sure, the operating system can lubricate "hardware-to-hardware interaction", for whatever purpose.

    The problem, as expressed here, is that there is no single "standard hardware" and no single "standard human" and thus some operating systems will be better suited for some combinations of hardware and humans. From this point of view, sure, Excel is an excellent operating system for the folks over at the corps. Not that I find any interest in this particular aspect, but nor do I seek to invalidate "the Excel viewpoint". In other words: fuck you and your dumb ideologies.

  2. #2:
    spyked says:

    Actually, let's formalize this.

    Given a computing system C (most commonly some piece of hardware, but not necessarily) with a prescribed set of properties P (the system specification), the so-called operating system, or the set of system software is that set of software S which implements P.

    Some P/S may include a thin layer of software running on top of the BIOS (which on IBM PCs is itself part of the system software whether you like it or not) such as a shell and an executable loader, in which case you get DOS. Another P/S may also include preemptive scheduling, dynamic memory allocation and fault isolation mechanisms for the tasks running on top, in which case you get Unix. Some other P/S may include an interface to the web, in which case you get whatever shitty web browser you're using. And so on and so forth, P is defined based on the whatever the system designer requires in order to operate C and S implements that. One important aspect entailed by this model is that it matters very little whether the so-called "OS" mediates access to hardware resources or not: if you install a GPU on your machine, then the application itself may as well include the driver, just as they did in the days of old, before the bureaucrats started deciding on what the OS "oughta" or "oughtn't" do.

  3. [...] under a platonic abstraction hell of artificial languages, "interfaces", "engines", "libraries", "operating systems" and other "virtual machines", all in all anything but soft-ware. Not that it's you who lacks [...]

  4. [...] echoes precisely my later rants about operating systems. Perhaps now's a good time to restate my point, starting from the observation that in a certain [...]

  5. [...] Outside of this definition, the so-called "social network" is as much nonsense as that fabled "operating" system. [...]

  6. [...] a device driver is a piece of software, usually (but not necessarily!) delivered as part of an operating system, providing automation in order to mediate interaction between a user and a so-called "device". Said [...]

  7. [...] Speaking of musical instruments, the greatest achievement in musical tech of the last century -- and perhaps for a few more centuries to come -- is without doubt the synthesizer. In a certain sense, synths are very similar to computing machines: in the ideal, they are able to generate arbitrary information given arbitrary inputs, but the shape of concrete outputs is very much influenced by the concrete implementation of our "synth", which turns it from "the" synth into "a" synth. Can you see how this maps to my previous observation on musical instruments? Or, say, to my older one about "operating systems"? [...]

  8. [...] the second layer, you're likely going to need a bunch of pre-existing software (e.g. an operating system) to support the very same whatever-it-is-you're-trying-to-achieve. If for example you're doing [...]

  9. [...] programming and the ELIZA experiment on one hand, while on the other you got all the wads of system software and "apps" to make your interaction with machines somewhat bearable. In other words: once we all [...]

Leave a Reply