Assorted notes on bootstrapping Cuntoo

November 29, 2019 by Lucian Mogosanu

Abstract: Cuntoo is a gargantuan beast (Gentoo), tamed by a few human-edible bash scripts aiming to whip it into a usable operating system for The Most Serene Republic. Following a very limited amount of experience with Gentoo and a somewhat decent understanding of Trinque's bootstrapper, I am jotting down various notes on my experiences so far in getting this up and running. Please read below for the gory details.

For the first part of this adventure, the environment: I deliberately chose to run my experiment on QEMU, an emulator and virtual machine monitor -- considering that I made use of Intel's modern-schmodern hardware acceleration features combined with Linux's KVM, all this to get some decent compilation times. Yes, QEMU is a piece of shit, but it's one that gives its users the illusion of complete access to the "hardware" interface and software infrastructure, the latter including a x86 BIOS. This comes as a boon for the software-inclined1 among systems folks, which is perhaps one of those sexy things that attracts "software developers" to virtualization like flies to horse dung. This long parenthesis aside, it's very useful when you have an overprovisioned box around and you'd rather use that one for some testing than buying new hardware.

By the way, LXC or some other container-based thingamajigg should work just as well for this purpose. In any case, what you want to do is make yourself a disk image, e.g.:

$ qemu-img create -f qcow2 test.qcow2 10G

which will serve as the image for the test OS, i.e. the environment where the Cuntoo bootstrapper will run. Moreover, Cuntoo needs a target disk, where the bootstrapped system will reside -- on a real machine, this would be a USB stick or a second hard-disk or whatever. This'd be created exactly the same way2:

$ qemu-img create -f qcow2 cuntoo.qcow2 20G

This is where I should add some intricate details about booting a QEMU VM instance with the two disks plus a heathen operating system ISO image. I won't; please study the QEMU manual if you intend to do this.

Since it's becoming apparent that I'm trying to shoot myself in the foot, I've chosen to install on test.qcow2 a Debian Wheezy3 image, because ideally one should be able to bootstrap Cuntoo off any Linux-based operating system that hasn't been completely taken over by wreckers. It's never as easy as that, of course, but let's see where this leads us, shall we?

Given our Debian on test.qcow2 (or /dev/sda, as the OS calls it), our goal is to bootstrap a fully working Cuntoo system on cuntoo.qcow2, which from the bootstrapper-OS view is /dev/sdb. The first step towards that is to get a kernel config going. Thus, knowing that Cuntoo runs a 4.9 kernel, we download the 4.9 Linux release off kernel.org or whatever, then we run

$ make menuconfig

and go through the lengthy tree of configuration options, carefully selecting whatever works best for our QEMU VM4, resulting in a Linux kernel config which I named 4.9-qemu. Then, using Trinque's document as a guide, we verify and unpack the cuntoo tarball, we copy 4.9-qemu to the config directory and we run:

$ ./bootstrap.sh -k ./config/4.9-qemu -d /dev/sdb

and after waiting for a few hours -- the thing went at snail's pace, because fuck, I forgot to give this more CPUs! note to self for next time I reboot the thing -- we notice that the bootstrapping process has stopped and emerge gives a weird error when trying to install Python, along the lines of:

configure has detected that the sem_open function is broken.
Please ensure that /dev/shm is mounted as a tmpfs with mode 1777.

So looking at /dev/shm, we see that it's a symbolic link to nowhere, i.e. /run/shm, a directory which exists on the Debian system, but not in cuntoo/build, i.e. the place where the bootstrapper prepares a chroot jail for the new root filesystem. So we have /run mounted as a temporary filesystem, which means that we should maybe add a

mount --rbind /run run

to cuntoo/scripts/prepare_chroot.sh, in prepare_chroot5.

After getting that out of the way and after adding more CPUs to our virtual machine, we're now ready to run the bootstrapper again... and then yet again, after, as discussed in the second footnote, adding more space for the bootstrapping process. This time, the script stops at the kernel build phase, waiting for me to provide a few more kernel configuration options, which by the way, results in a second kernel config, 4.9-qemu-gentoo.

So now we run the bootstrapper yet yet again with the new config, which finally results in a genesis.vpatch. However, the signature doesn't verify against Trinque's key, which led me to look at the comments6 -- fortunately, Lobbes provides his own genesis, which supposedly gave him a working Cuntoo system. So diffing my genesis.vpatch file against his, I notice some very weird paths being created in cuntoo/portage, containing some pieces from my Debian system! After digging through the scripts which create those paths, more specifically make_portage_tree.sh, I notice that the destination path is set by the following line:

    dest=$pdir/profiles/${src#/mnt/trove/src/cuntoo/build/usr/portage/profiles/}

I have no fucking idea what mnt slash trove is, so I change the line to:

    dest=$pdir/profiles/${src#$bdir/usr/portage/profiles/}

in collect_used_profiles, where bdir is set before said function is called.

Rerunning the bootstrapper now finally yields a genesis.vpatch that is identical to Lobbes' and which verifies against Trinque's signature of the Keccak V patch. Now that I have all this, I should at least expect it to run, shouldn't I? Well...!

Let's not get ahead of ourselves: for some reason the QEMU Linux kernel I built insists to label the newly bootable disk as /dev/sdb instead of /dev/sda7. So I had to chroot into the Cuntoo system and update my LILO to point to /dev/sdb3 et voila, I finally have a bootable Cuntoo in front of me. All in all, arriving here required about two weeks of bootstrapper runs, modifications and reruns, accompanied by time passed waiting for those 4-hour builds to finish -- the GCC 5.4.0 -> 4.9.4 fandango in particular is truly depressing, especially since building GCC 5.4.0 on systems with less than 3GB of RAM causes the build to crash. So what can I say, if you ever meet the Gentoo maintainer who committed this atrocity, or perhaps any Gentoo maintainer at all, grab the nearest blunt object and hit the fucker as hard as you can.

I'll leave the reader with a few comments:

Comment the first: I could verify my genesis against Trinque's signature in the end, which is all nice and well. However, I did require a reference genesis.vpatch to do any debugging, and I'm not sure that this is even sufficient. Don't get me wrong, I very much like the idea of using the bootstrapper's output as a basis for further ebuilds, but what am I going to do if I can't even debug the process of producing a verifiable basis?

Comment the second:

spyked@cuntooer ~ $ file /bin/ls
/bin/ls: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, stripped

I really don't know whether the Musl-GCC 4.9 toolchain should have spat statically linked binaries, or whether I need to do a world rebuild, or what.

Comment the third: I made changes to the Cuntoo scripts and I have no sane way of publishing them. Shouldn't the bootstrapper itself, or maybe just the text included therein, come in the form of a genesis?

Comment the fourth: Emacs doesn't compile on my new system; and from what I understand, it's not possible to compile Emacs using Musl. Does that mean that Cuntoo and Emacs are mutually exclusive? I suspect there's no easy way out of this, probably a hero will have to saw the glibc-isms apart from Emacs at some point... or not, in which case I'll be left without a SLIME and with a horrible development environment for my Lisp stuff.

Comment the fifth: if I understand correctly -- and please do let me know if I'm wrong, because I hope I am -- Cuntoo still relies on downloading pieces off Gentoo mirrors in order for the bootstrapping process to complete successfully. This means that when -- very pointedly when, not if -- GCC 4.9 disappears off the Internets, no one will be able to use this anymore. So what do? This IMHO is a very important problem.

Comment the sixth: tries to make two points, one minor and one major. The minor point is that the scripts could be reorganized/recompartmentalized a tad, so as to allow the user to do partial bootstrapping runs, followed by later reruns, e.g. if a kernel build has failed at some point, then one could simply remount the target partition and rerun it without going through all the intermediate steps. Which brings us to the major point: this is probably obvious to those who've already been through the bootstrapping experience, but the Cuntoo bootstrapper is very different from ye olde "OS installers", in that you can in general expect it to fail midway for some reason or another. That's fine from my point of view, as long as there's some means of efficiently finding out the root cause and some means of fixing the problem and restarting the process, preferrably without fear that it's gonna fail again at the same point. More importantly, I'm not sure whether it's the best idea to target this at newbies coming from Windows8, at least not before going through some work to make the process one order of magnitude smoother, even if this "only" means documenting the whole thing to exhaustion.

Comment the seventh: I might have had a seventh comment, but if I did I forgot what it was. Well, I guess this is it then.


  1. In all honesty, "software-inclined" seems to lie in the same class of retardations as "being an engineer". What, measuring electrical signals and soldering a bunch of wires ain't rocket science, and I'm one of the (I'm guessing many) living testaments to the fact that even people with two left hands can do it! 

  2. By the way, there's a reason for choosing 10GB for the first disk and 20GB for the latter: while a binary OS distribution can be minimally fit into that 10GB, it looks like Cuntoo requires more space -- yes, I found this out the hard way. So yes, make sure you allocate plenty of disk space for your OS, even if you use it just for testing purposes. 

  3. Debian, because it's the operating system that's delivered the most bits of productivity to me in the last twelve years or so. Wheezy, because as far as I know it's the last version that doesn't completely suck balls

  4. The relevant pieces of hardware in our virtual machine are, as follows: the NIC, an emulated Realtek 8139; the emulated Intel 64-bit "QEMU Virtual CPU"; the emulated Intel 8237-based DMA/IDE/USB/etc.; and the emulated Cirrus Logic 5446 VGA. The latter in particular is useful for VNC sessions in case shit goes south. 

  5. This is the wrong thing to do, by the way, because the chroot jail will share its /dev/shm with its parent Debian system. The correct approach would be to mount a new tmpfs in the chroot jail's /dev/shm, but well, this is one of those times where I'll fuck the clean solution in favour of the convenient one. The former can wait for production. 

  6. And other writings on the matter, such as Lobbes', Mod6's (archived) and Hanbot's Cuntoo series. 

  7. I don't know why, nor have I dug into the matter, I expect it's a QEMU configuration issue. There is a /dev/sda there, which seems to point to a 8MB disk backed by... I have no idea what. The risk for this experiment to branch out in a gazillion directions is very real, so that's why I stopped here. 

  8. Assuming for a moment that this will be the official operating system supported by the Eulora client, it's conceivable that many users will have to go through similar troubles as the ones described here, and some of the poor souls might give up when faced with much easier challenges. I don't know what to make of this, I just thought it's worth mentioning.

    And yes, I said "one order of magnitude smoother", because installing Debian back in the day was many orders of magnitude easier. And Debian isn't (or wasn't) targeted at noobs, is it? 

Filed under: computing.
RSS 2.0 feed. Comment. Send trackback.

14 Responses to “Assorted notes on bootstrapping Cuntoo”

  1. #1:
    Diana Coman says:

    Re 5th : it's only Trinque's reduced tarball that still requires possibly some mirrors. Otherwise, I specifically set and ran this successfully OFFLINE. The issue is basically that the genesis should have included all the shit too since that's what one needs in fact to start it, no matter how ugly it is.

    #t thread relevant to the above: http://logs.ossasepia.com/log/trilema/2019-07-15#1922470

  2. #2:
    spyked says:

    Many thanks for all the pointers! Also adding here your mention of the discussion about paths.

    So if I understand correctly, populating the Portage repository with all the existing sources should pretty much ensure that the installation will happen offline (since emerge won't have to grab everything off mirrors). Is this the case?

    If so, then IMHO there's nothing wrong with at least a version of the tarball with all batteries included. It's not like trusting whatever's grabbed off those mirrors (against some GPG key that's not in the WoT, even) is any saner than trusting a bunch of ??? that comes bundled with the rest of the stuff. The bootstrapper tarball is "infected" with the stage3 image as it is, so... might as well add whatever other needfuls there are in there.

  3. #3:
    Diana Coman says:

    Yes and yes. I did even have a preliminary look at making the whole genesis as per http://logs.ossasepia.com/log/trilema/2019-07-19#1923622 but didn't proceed mainly because a. supposedly Trinque was moving this forwards one way or another b. there wasn't any decision made as to where/how this goes and so no point in just sinking in the work required there (mainly to eliminate/clean up .pdf and whatnot because yes, there are such things).

  4. > Please ensure that /dev/shm is mounted as a tmpfs with mode 1777.

    This was exactly where I broke my teeth last time I tried a Cuntoo bring-up.

    > ... GCC 4.9 disappears off the Internets, no one will be able to use this anymore. So what do? This IMHO is a very important problem...

    This was one of the problems that prompted me to write the 'M' MIPS emulator (idea being, can get a cross-compiler going instantly on any machine to which can port 'M' or equiv.) but was dead end -- ran in "geological" time.

  5. #5:
    bvt says:

    2. musl does not imply static linking, cuntoo is musl-based but dynamic. And gentoo has 'static' USE-flag for only some packages.

    4. emacs support on musl (a hack for its dumper really) should be in, it is added through musl overlay (/var/lib/layman/musl/app-editors/emacs on my system, the overlay is added by default during cuntoo installation). I have never tried to install it there, though.

    6. The simples change would be to move kernel configuration and installation to the last step.

  6. #6:
    Jacob Welsh says:

    from what I understand, it's not possible to compile Emacs using Musl.

    My "solution" to this was... XEmacs.

    Though since the GNU one is supposedly ported to all kinds of systems, it might just be assuming "Linux == glibc" and have a switch accessible somewhere to turn off the "unexec" thing which I recall was the culprit, at the cost of slower startup.

  7. #7:
    spyked says:

    @Diana Coman: FWIW, I completely agree with MP re. portage. Now, as to how Portage can or will be made to obey, that's another question.

    @Stanislav Datskovskiy: Re. GCC, why not just package a signed binary along with the rest of the stage3? Ave1's already spent a lot of time working on GNAT, so IMHO that'd be a proper replacement for GCC 5.4.0.

    @bvt: Re. Musl, I got that, but it's not clear to me whether Cuntoo was supposed to do static linking and just mine didn't, or what. I guess I'll have to review the older Cuntoo log threads to get a better idea of the decisions made there.

    Thanks re. the emacs pointer, I'll look into it. From what I understand there should be a magical incantation to patch the ebuild and get it compiling, or is there another package in there, or what?

    Re. where kernel installation goes, that makes sense; on the other hand, maybe Trinque did it in that order for some particular reason? Let's see what he says.

    @Jacob Welsh: At some point there was some heated debate about making Emacs portable to other libcs (and as it happens, I can't readily find it), IIRC the GNU people didn't want to give glibc up at all. I don't know what are the main differences between XEmacs and GNU Emacs, but I'm not against completely dumping the latter, provided that I can run e.g. a SLIME/SBCL environment on the former.

  8. #8:
    Robinson Dorion says:

    Rerunning the bootstrapper now finally yields a genesis.vpatch that is identical to Lobbes' and which verifies against Trinque's signature of the Keccak V patch.

    I bet that was a nice to see!

    All in all, arriving here required about two weeks of bootstrapper runs, modifications and reruns,

    Yikes! How much of that 2 weeks do you attribute to the QEMU set up and the gcc 5.4 fandango ?

    However, I did require a reference genesis.vpatch to do any debugging, and I'm not sure that this is even sufficient. Don't get me wrong, I very much like the idea of using the bootstrapper's output as a basis for further ebuilds, but what am I going to do if I can't even debug the process of producing a verifiable basis?

    Yeah, being able to debug the process is needed and ties right in to comment the third concerning publishing your modifications.

  9. #9:
    spyked says:

    @Robinson Dorion: It was nice to see indeed, if frustrating at the beginning. I tried as much as possible to work from clean sources, so I had to manually copy the Keccak sig every time I untarred the archive. Forgot to do this once or twice and when I finally got it right, I had to rewrite portions of this article to get the new results in.

    Re. QEMU setup: getting the runs ready didn't take more than a couple of hours (mostly debugging), but the dead blocking hours waiting for the results were quite annoying. GCC 5.4 takes significantly more time and RAM to compile than the 4.9 build, which on a more minimal system (e.g. the lenovo S10 I've acquired from Stan) makes builds practically impossible.

  10. [...] his status. [↩]lobbes is taking up Gentoo portage/ebuild reporting; spyked has published his notes on bootstrapping Cuntoo; it's unclear at present if either wants to own this domain. [...]

  11. [...] in the way I looked at Trinque's bootstrapper. The two could then be compared along the lines that bothered me the most, at the very [...]

  12. [...] Trinque published Cuntoo November 2018 and between then and November 2019, diana_coman , bvt , spyked , hanbot , lobbes , mod6 , and shinohai had published installation [...]

  13. [...] tuned in to my recent Debian adventures. I personally find it relevant within the larger area of bootstrapping systems, and while it perhaps doesn't bring much news, this field report serves as a reference on how to [...]

  14. [...] and without. Thus at some point I stopped installing Debian on new systems, then I played with it just recently for bootstrapping processes, then I got pissed at the whole "computing" thing and thought [...]

Leave a Reply