The Tar Pit

There's been a lot of debate on the interwebs on "how to AI properly", and on one hand I'm too lazy to review the existing literature at length¹, while on the other, I think that most discussions tend to get too bogged down in technical details, without even trying to settle some basic premises first. I don't know, perhaps I'm very basic myself and I don't find said premises obvious, but in any case I'll take a step back and discuss those before any other considerations. And yes, the title is a joke, in case you're wondering; this article isn't really a recipe for anything.

In order to lay out the premises for the aforementioned premises, I need to begin first with some very trite observations on the mathematics behind Large Language Models. Feel free to skip them if you're not into this kind of thing or if it otherwise makes no sense to you.

The first step in LLM processing is something called tokenization -- and for the purpose of this discussion, a token is a linguistic atom such as a word, a punctuation sign, or maybe even a so-called morpheme. Whether we're discussing training or inference, the first step in the process involves taking a string of bytes and converting them into a structured form, where every particular token is associated with a unique identifier, e.g. a number. Now, consider training: the data corpus is, through this first process of tokenization, reduced to a set of unique numbers, from 0 to some very large N. Together, these identifiers form, if you will, a dictionary, and also, in the algebraic sense they form a very large-dimensional vector space. And in this vector space, some combinations of tokens, e.g. "salad llama bitch", have a lower weight than others, e.g. "freedom slavery is"; and thus the whole purpose of the so-called "pre-training" is to reconstruct the vector space of relations between words starting from said data corpus; and conversely, the purpose of inference is to build lookups of valid relations within instances of this algebraic structure.

Thus, when viewed from this particular angle, the "language" in "large language model" is an algebraic structure, and the "model" is an instance thereof, which expresses valid relations between atomic symbols in a dictionary. This, at its core, is the whole mechanism, and the rest, e.g. the fact that the reconstruction method is statistical, or the fact that inference queries are probabilistic, all these are particular details of the items falling under the category of LLMs.

Thus, when I say that "the quality of the output depends on the quality of the input data and that of the training corpus", I mean something very specific, and this specific meaning is precisely what sets the premise for "how to AI properly". What I mean is that, say, if I, Lucian, am looking to extract maximally useful bits from the set of relations in the LLM's vector space, then I need to know that set of relations, that is, I need to have it internalized within my own set of relations². Stated like that, that is, of course, infeasible bordering on impossible; even if I were to, say, have access to the weights of Claude Fable 5.1, which I definitely do not, I would still lack the patience, if not the willpower to memorize those weights. But that is not really the point, is it?

Sure, the quality of the training corpus varies, so if, say, Claude Fable 5.1 was trained on linguistic cum-gargle from Reddit, then of course it will spew back Reddit-talk -- assuming that the human is also trained on the same. But if the human is trained on Shakespeare and Nietzsche, then just by the way the human expresses him or herself, the input will activate the Shakespeare and Nietzsche-parts of Fable's vector space, and not the Reddit ones, with the exception of maybe some small overlapping bits, such as, say, the deep, yet at the same time superficial English language substrate.

Thus the premise to the discussion, and, funnily enough, the conclusion to my article, is quite clear: if you want to AI properly, then you gotta language properly. And, oh boy, I never thought that I'd have to spell this out, but here it goes: if want to language properly, then you have to learn how to read and write properly in one or more languages of your choice, as this makes a world of difference. Whaddaya know, you'd never have thought that your working on the conviction that literature (to take but one example) was useless back in the day, would cut this way, did you? 'Tis a sad day when computers are able to utter sentences in your native language more coherently than you are, but... here it is.

So anyway, this is how it goes: once we get to the point where we can discuss mastering language in all its aspects, and only then, we may discuss domain-specific knowledge, jargon, slang and so on and so forth, as derivatives of this mastery, and only downstream of that we may discuss efficient usage of textual AIs within some particular field. Otherwise it's futile to attempt to skip steps -- so chop chop, how did that old joke go? grab ahold of some books, and don't forget to roll yourself some paper cones for the sunflower seeds!

Filed under: asphalt.
RSS 2.0 feed. Comment. Send trackback.