The Tar Pit

I've been burning to get this off my head and onto this piece of virtual paper for more than a month now. Alas, I didn't have the time then, and I'm purposely putting other work on the side to get this done now and... what else can I say; iarna-i grea, covidu' mare, semne bune anul are. Anyway, let us proceed.

Debian is, as many of you may know, a Linux-based operating system. I've been using it for a while now, use born out of a mix between chance, the project's age, my own habits, setup time and, to a very small degree, the tools it provides. No, I don't trust all its utilities down to the binary level, hence the "absolutely filthy" in the title, but it's worked for me for a while now, and by "worked" I mean that it does what I want and doesn't what I don't want it to, and by "it" I mean... well.

The project has seen its share of rape throughout time, the raping coming from both within and without. Thus at some point I stopped installing Debian on new systems, then I played with it just recently for bootstrapping processes, then I got pissed at the whole "computing" thing and thought that hey, I might as well try to mirror this shit.

And by "this shit" I mean the last version of Debian I've used, 7.0 (Wheezy), though I expect the recipe here will work for any APT-based operating system, including Ubuntu et al. Speaking of which: APT stands for the very pompous "Advanced Package Tool", a wrapper over "dpkg" (Debian Package), which is a tool for managing such things as "software packages". APT is mostly useful because it can grab stuff from "repositories", regardless of whether said repositories are to be found on your USB stick or on a web server.

This lengthy introduction being laid out, my goal is to steal a Debian release in its entirety, so that I may later use it to set up robust (albeit not necessarily trustworthy) systems based on this rusty ol' piece of shit. I wasn't kidding when I said I'm maintaining Feedbot-Linux, and I don't particularly care about the variety of "truckload of unknown unknowns" I'm grabbing, so... here I am. Anyway, these are the necessary ingredients:

• a GNU/Linux system, preferably of Debian variety¹, used as a bootstrapper;
• a tool for downloading e.g. pieces of archive.debian.org on one's machine; and
• a minimal install Debian ISO image, e.g. the "netinst" image,

such that in the end both the "installer" and the resulting operating system will be able to grab packages off the "bootstrapper".

In order to make this possible, one should be able to mirror the Debian archive, which would be a curl away, except... I don't want to mirror 600GB for 150GB's worth; more precisely, it should be possible to mirror only the Wheezy repository, not those of other versions of Debian. To this end, I propose employing one abject Perl script going under the name of "apt-mirror", which I have mirrored here for my own pleasure².

Before using this script, I have found the need to make some modifications to some paths that are apparently hardcoded there, or at the very least attempting to change them via the usual command-line magic didn't do anything for me. So I first changed the value of the base_path config variable (line 95) to something more fitting, e.g. some place in my home path. Then I changed the value of the config_file variable (line 120) to a custom mirror.lst, because I don't want to have to edit a global system config file to download some stuff off the Internet.

Regardless of your configuration options, make sure that the mirror.lst file is in its proper place and that it contains some minimal config info. For example, I set the following:

set defaultarch amd64
set nthreads=2
set base_path=/my/attempt/to/override/base_path

followed by a bunch of APT source.list entries, e.g.:

deb-amd64 http://archive.debian.org/debian/ wheezy contrib main non-free
deb-amd64 http://archive.debian.org/debian/ wheezy-backports contrib main non-free
deb-i386 http://archive.debian.org/debian/ wheezy contrib main non-free
deb-i386 http://archive.debian.org/debian/ wheezy-backports contrib main non-free
deb-src http://archive.debian.org/debian/ wheezy contrib main non-free
deb-src http://archive.debian.org/debian/ wheezy-backports contrib main non-free

So I just grabbed everything related to x86, plus sources, adding up to about 130GB's worth of Wheezy.

Assuming you've done all this, now you just have to fire up apt-mirror and sit and wait for it to download everything. It's supposedly also doing some setup afterwards using some post-mirror scripts, but I didn't find the need for any such thing: I found everything I needed in:

$base_path/mirror/$url

The final step in this recipe involves verifying the thing, not that this is actually possible. Still, you should be able to run some minimal tests, e.g. fire up a web server to serve the files you've just downloaded, and then you can fire up the installer on a machine in the local network, have it pointed to the newly-installed repository and then grab a coffee until the install is done³.

That's about it. You may have noticed the "deb-src" in my list; my guess is that these people must have (had) some process to bootstrap the whole system from sources... and while in that field the alternatives are clearly superior, I guess it doesn't hurt to look.

Filed under: computing.
RSS 2.0 feed. Comment. Send trackback.