Revisiting "AI"

 

Picture from here.

 

I discussed a while back my opinion on the stampede to bring AI into the kitchen, living room, workplace, and bedroom that seems to be going on. 

 

My opinion isn’t all that different though I’m now a bit more informed. I believed I referred to ChatGPT (and others) as a stochastic parrot—that is, a device that is able to reliably generate convincing language but does not understand the meaning of the language it is processing. 

 

ChatGPT is not, exactly, such a device. It is a language prediction mechanism in the sense that given its broad training it can respond to a statement (or “prompt”) in a manner consistent with how its model of the language suggests is the desired response. This is quite a bit more sophisticated than a stochastic parrot. I am not saying large language models (LLMs) are conscious or have any understanding of the language they generate in the human sense of that word. I am saying they are extremely adept and sophisticated in mapping how language works.

 

This is not language at the sentence or word level—though ChatGPT maps that, too—but a much broader and deeper map.

 

Consider a regular visual map—like what we used to use on road trips when I was a child. It’s a piece of paper with coordinates on it and imagery. If you have the right x/y (east/west, north/south) coordinates, you can find a city. That’s a two-dimensional map. You can find a three-dimensional map of the solar system here. However, since you have to tell the map when you’re looking at the solar system, it is, in effect, a four-dimensional map. With those four coordinates, you can place a planet geographically.

 

However, to go forward with that metaphor, each of the planets influence the other and they’re all influenced by the sun. Light comes from the sun, for example. If we added that to the map, that might come in as a fifth dimension—a separate coordinate. You can see how this gets complex pretty quickly. While computers can work with multidimensional math easily, humans have more trouble. We want to actually understand such things—and we’re pretty much embedded in a four-dimensional brain. 

 

Physicists came up with the idea of phase space. This is a space where all physical quantities are represented as different dimensions and a given state is described as a coordinate set in those dimensions. It’s not hard to see the utility of such a description—and it’s not much of a leap that something like that level of complexity would be needed to describe all the intricacies of language. That’s not all of what goes on inside of ChatGPT but a complex multi-dimensional representation of language is part of it.

 

There’s a good description how LLMs work here. I’m not going to reproduce that. Timothy Lee and Sean Trott, the authors, did a much better job than I ever could. So, you should go read it. Wikipedia has a broad discussion of LLMs here.

 

In place of what I described as phase space, LLMs use a word vector. Word vectors mathematically represent a word in all its dimensions. Like the coordinate systems I was describing before, and like phase space, you can determine something approximating distance between words by referring to them within a complex coordinate system. If you look at Washington, DC, and Baltimore, MD, and get their x/y coordinates, it’s not a difficult mathematical operation to determine that their close to one another. Similarly, if you look at Baltimore and Boston, MA, you can tell Boston is farther away from Baltimore than Washington. Seattle is farther away yet. 

 

Using word vectors, you can develop maps of words that are close to one another in use. Or far away. Or build nets describing sets of words. Computers don’t balk at numbers of dimensions—LLMs use hundreds or thousands. GPT-3 uses over twelve thousand. I suspect the number of dimensions will only increase as more computing power is applied to training.

 

Ah, yes. Training.

 

One of the interesting things about LLMs is that they are not programmed per se. They analyze reams of material on their own—the training. Again, Lee and Trott discuss this in digestible detail. I’m not going to go into it here.

 

The next level of LLMs is the emergence of word predictions from the raw word vectors. Language has syntax and grammar—it is organized into meaningful packets. So is computer language code. When the LLM is trained on a language, that underlying structure is embedded in the dimensional relationships between the word vectors. LLM processing takes input and applies it to transformers. The input to a transformer results in a more meaningful result. That result can be further input into another transformer. The final result set is the predicted output from the input prompt. (Again, refer to Lee and Trott.)

 

It should come as no surprise to anyone that a powerful engine such as an LLM is trained on a great deal of language input that it would return sophisticated language as a result.

 

But there’s more to this—and this is the important bit.

 

GPT-3 recently (reported in July, 2023) that it had scored highly in tests of reasoning by analogy. Reasoning by analogy is something SAT and other college entry exams test. You’ve seen this. The test presents evidence (Given a letter sequence, abcde, eabcd, deabc, what is the rule? Or, what is the next entry in the sequence? That’s reasoning by analogy.)

 

Now, it’s interesting that GPT-3 failed in extracting the analogy from text but it did extract it from prompts. Is this evidence of analogic reasoning?

 

For my part—and the article I linked to above suggests the same—I think it indicates that reason by analogy is embedded in the language training of the LLM. By this, I’m suggesting that what we are seeing is not intelligence in the LLM but intelligence embedded in the training material as represented by the LLM.

 

I think this is borne out by the LLMs being used to determine protein folding. A team at Meta thought that though the “L” in LLMs was intended for language, it really just signifies a packet of data that can be analyzed. Reconfiguring the LLM to use protein chemistry as training input instead of language seemed a productive path.

 

It is, of course, far more complex than I have stated. My point is that the LLM produced plausible protein sequences similar to the way it produced plausible language constructs. Let’s remember that the capacity of the LLM to retain data is far greater than any human and its ability to analyze multi-dimensional space is far superior. By representing data as digestible material to the LLM, the inherent intelligence imbued in protein systems that are the product of a billion years of evolution is amenable to analysis. 

 

This does not mean the LLM understands protein chemistry anymore than it understands language—in the human sense. What it does analyze the underlying structure beneath the training material and, given a catalytic prompt, presents plausible combinations that conform to that underlying structure.

 

Given this point of view, LLMs become more interesting. They are not conscious. They do not comprehend what they are doing. But they are potentially powerful tools  to express the underlying sophistication and intelligence of the material they’re given.

 

That lack of comprehension shows up regularly—anyone using an LLM without oversight is just asking for trouble. AtomicBlender—a YouTube channel I follow—asked ChatGPT to design a nuclear reactor of the future and it gave surprising—if impractical—results.

 

I think that is going to be the main problem of using LLMs: the output must be verified. In the case of protein chemistry, a solution candidate would have to be verified experimentally and be assured it didn’t violate existing patents. LLM produced original material cannot be taken at face value.

 

LLMs are a powerful tool. The current situation reminds me of something from the Larry Niven, Protector: Intelligence is a tool that is not always used intelligently.

 

State of the Farm: August, 2023

 

This will not be a long post as I’m in Pennsylvania for a family reunion.

 

I had planned an interesting post but left without the requisite materials. So, we’ll work with what we have.

 

The end of July and beginning of August is the beginning of harvest season. The first incoming is blueberries—of which we got an enormous amount, dwarfing all previous years. This is due to two reasons: 1) our blueberry bushes have finally matured, and 2) we’re using bird netting effectively.

 

We tried to use bird netting last year but failed miserably. Putting up the netting ended up destroying our harvest. I just couldn’t get the netting over the bush without ripping off the ripening blueberries. This year, I built two wooden frames over the low bush and high bush blueberry in the main garden. In addition, we had planted a large space area on the west side. 

 

This year we used cattle fence panels (Remember those? They’re in the picture above.) to make a large space: about 16x18 feet. This is interesting since the cattle panels only come in 16ftx50in units. We needed four sections so cut two 16x50 panels in half and wired them to four 16x50 sections. We staked them into the ground and it gave us a very large area. This is wide enough and high enough that we can let the high bush blueberries grow quite large without ever having to trim them much. 

 

That said, we still put bird netting over all the frames and that was an incredible PITA. The bird netting we used was the normal ½ inch mesh that we’ve seen. It’s always difficult, catches everything, tangles on itself and any leaf/branch/rock/shoe it comes in contact with. We did it but only managed by swearing the air blue.

 

After this, I looked on line. Surely, somebody had to do this better than we did. It turns out we used the wrong netting. While the thin variety is used quite a bit, when you look at what professional growers use, it’s a braided material that appears much less difficult to handler. Several videos I saw had the growers drape them right over the bushes themselves—which we will not be doing. In addition, when we were driving up in Vermont recently, we saw that some blueberry farmers were hanging long streamers of Mylar. My guess is this was to repel the birds.

 

So far we’ve gotten a few gallons of berries—it’s easier to measure it in pounds. Say 10+ pounds. The last harvest isn’t in yet but after that we’ll (Retch. Groan. Yowl.) remove the netting.

After the blueberries, come the pears and the Aronias. We’re doing the pears a little early this year. There’s been so much rain that some of the fruit is splitting. We could leave it on the tree but it’s a risk. Taking them from the tree, we’ll wait a week or so and they’ll ripen on the table. Not as great as letting them ripen on the tree but still quite good. Plus, we don’t have to fight the wasps for them.

 

Aronias are an... acquired taste. They always taste dry in my mouth even though they can be quite juicy. It comes from the astringency. I planted two trees with the idea I’d plant more if I liked them. I don’t much but the tree is doing well and it’s a shame to waste the fruit. I think I’ll press the fruit and see if I can make the juice palatable.

 

One new tree that’s coming on line is the mountain ash. Its berries should be ripe when we get back. We planted the tree because the paw paw we planted near the same spot wasn’t doing well. Now, both the ash and the paw paw are doing well. We’re still undecided about what to do next. Lose the ash? Let them grow together? We really like the paw paws.

 

We have five apples we’re trying to harvest. (We have some crab apples but they hardly count.) Two of them are still too young. One we planted a while back and lost the label so we’re not sure what it is. It seems to have a cinnamon flavor. The remaining two are the Granny Smith and the Sops of Wine. 

 

The Sops is a terrific apple. Not too sweet and with a spicy bite. It is coming in like gangbusters. I had to put props under the branches before we left The Granny—well, I’ve been fighting that Granny for twenty-five years. It never gives much of a harvest. We’re talking to a tree service to see if they can prune it right and help us get it under control. If not... well, Applewood is nice to work with.

 

The Cornelian Cherries are just about ready to pick. I’m going to have to make room in the freezer or make wine of them instantly. We’ll see which. 

 

We’ve reached the point in the main garden where the day to day work is less. The plants are just so well entrenched the weeds are being out competed. The biggest job I have is to make sure the melons don’t eat the bok choy or the pole beans and the cucumbers don’t eat the pintos. Work waiting for me when I get back.

 

As I said last time, the beans haven’t been setting much in the way of fruit because of the heat. That’s lessened some so I’m hoping we’ll see more. We started pulling in eggplant. The lettuce is in a shady spot so we’ve had good salads. Other crops such as sugar beets, cabbages, bush beans, and the like, are doing well. We’ll be harvesting them, soon, too. 

 

I’m a little nervous about the potatoes. They’re starting to turn yellow. Wendy thinks this is when they die back. I think it’s a bit early. Still, it has been a hot summer. We pulled some the other day and the tubers look good. 

 

All said, the harvest looks promising.