The difference is that Word2Vec "learned" these relationships auto-magically from the patterns in the surrounding words in the context in which they appear in written text. Don't forget that this was a revolutionary result at the time, and the actual techniques involved were novel. Word2Vec is the foundation of modern LLMs in many ways.
Someone once told me you need humongous vectors to encode nuance, but people are good at things computers are bad at, and vice-versa. I don't want nuance from computers any more than I want instant, precise floating point calculations from people.
Isn't the whole linear operations thing not really true and even less so now in embeddings from transformer based models?
I remember reading a blog about this but cannot find it anymore. This is the closest thing i could find now:
https://mikexcohen.substack.com/p/king-man-woman-queen-is-fa...
I just tried it on qwen3-embedding:8b with a little vibe-coded 100 line script that does the obvious linear math and compares the result to the embeddings of a couple of candidate words using cosine similarity, and it worked exactly as expected. Same 22 candidates for both questions
king - man + woman ≈ queen (0.8510)
Top similarity
0.8510 queen
0.8025 king
0.7674 princess
0.7424 woman
0.7212 queen Elizabeth
Berlin - Germany + France ≈ Paris (0.8786)
Top similarity
0.8786 Paris
0.8309 Berlin
0.8057 France
0.7824 London
Sure, 0.85 is not an exact match, and if I dump an entire dictionary in there it might be worse, but the idea very much works
Very much so. It was a little less untrue with older word embedding models* but that kind of semantic linearity never was a thing in practice. Word embedding models try to embed semantically similar words close to each other, but that does not imply linearity at all.
*with transformer models, it is pretty much not even wrong.
"Queen" came from "The king and his queen". There is no common word for Queen in Germanic languages, and for what ever reason Queen became synonymous with royalty. Originally it just mean "the king and his woman", but I don't know when it changed. Certainly we had more than one word for "adult female human" in old English.
Arithmetic is king = royalty + male, while queen = royalty + female
But then it makes all these words just arithmetic values without meaning. Even if the words "royalty" and "male" can be sum or difference of some other words and so on - all are just numbers, no meaning at all.
The representation might not need to explicitly encode "meaning", if it does so implicitly by preserving essential properties of how things relate to each other.
For instance, a CAD object has no notion of what an airplane wing or car wheel are, but it can represent those in a way that how a wing relates to a fuselage is captured in numerical simulations. This is because it doesn't mangle the geometry the user wanted to represent ("what it means", in a geometric sense), although it does make it differ in certain ways that are "meaningless" (e.g. spurious small features, errors under tolerances), much like this representation might do with words.
Back to words, how do you define meaning anyways? I believe I was taught what words "mean" by having objects pointed to as a word was repeated: "cat", says the the parent as they point to a cat, "bird", as they point to a bird. Isn't this also equality/correspondence by relative frequency?
It doesn’t work because that’s just wrong. The semantic meaning of “king” is much more than simply “royalty” and “male”. And it will be different for different people based on their experiences and familiarity with English and world history as well.
Then there’s the phonetic angle in addition to the semantic one. Why isn’t cash emperor? Because “cash is king” is alliterative.
Then there’s the orthographic angle: it’s a lot easier to write “king” than “emperor”.
I really think you should actually read the article. None of what you are saying has to do with the content of it, and it will explain how you can do arithmetic with these words.
> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
Besides which, this is totally a valid question based on the article. (The temptation to ask if you read it is almost overwhelming!) It talks about how to do arithmetic but not what the result of that will necessarily be, so I don't see that any part of it answers the question of "cash is king" + "female" - "male".
If I had to guess, cash - king + queen = credit (or money or something?). You are just asking the same thing as cash - man + woman, or "What is the feminine version of cash?" because queen - king ~= woman - man.
I say credit, because it is not as physical and direct as cash, so perhaps it is perceptually more feminine?
But I will have to check the next time I work with word2vec.
This is my problem with people claiming that LLMs "understand". What we usually call "meaning" in intricately related to an encyclopedic knowledge of the world around us. How does this kind of knowledge not get into the same kind of loop you've just described? It is ultimately founded on our direct experience of the world, on sense data. It is ultimately embodied knowledge.
Vector spaces and bag of words models are not specifically related to LLMs, so I think that's irrelevant to this topic. It's not about "knowledge", just the ability to represent words in such a way that similarities between them take on useful computational characteristics.
I really recommend watching this section of this video. Embeddings do encode plenty of "human knowledge" into the vector values and their relations to each other.
s/embodied/embedded/, and this is how LLMs understand.
As others already mentioned, the secret is that arithmetic is done on vector in high-dimensional space. The meaning of concepts is in how they relate to each other, and high dimensional spaces end up being a surprisingly good representation.
And what are we if not a bunch of interconnected atoms? Smash a person to paste and you will not have any deep meaning in them, no life, nor sublime substance making them different from the dust they were made of. What is special in humans? Aren't we just an especially complex hydrocarbon mass that receives external stimuli and remaps them to a physical output? What makes you think that there is something more inside?
There’s nothing special about having an embodiment. A robot has an embodiment of sorts. An LLM meanwhile is a brain in a vat.
And there’s nothing special about my 21x23 square feet lawn. Can you emulate it? To what fidelity? How much should the map correspond to the territory? The same squarage, the same elevations down to the millimeter?
You’re not saying anything that counters the point that was made. Just mentioning stuff that people animals are made of with the assumed strawman argument (not made) that there is any non-physical essence at play. There isn’t.
Put a camera and some feet on an LLM and maybe it has an embodimeent.As long as it just has digital input it does not in the sense being discussed here.
What I am talking about concerns how human language relates to meaning. I'm not sure what this has to do with humans being "special". Saying that humans are "just an especially complex hydrocarbon mass that receives external stimuli and remaps them to a physical output" misses the point that what data we have available to us is qualitatively different from that of today's best natural language generation software.
Meaning in the best case is correspondence between a word and a grop of other sensory inputs that an embedding lacks. So when you complain that this lacks meaning, the core of it is that it does not look powerful enough.
Give me a better definition of meaning and I might change my mind on the topic.
Perhaps femininity is a more specific concept than masculinity. I call this the "Mrs Pac Man Effect".
It is more common for people to personify objects (say, a rock or a frog or a random internet user) as male than female. In many languages, the plural for a group of things is male even if it only has one male element.
A simple charachter like pac man is male in a universal kiki/bouba sense, but the female equivalent needs a bow (it is a more complex specalization of its male counterpart)
Obviously, biologicially male-ness is a specialization of female-ness, because females more resemble unsexed creatures in their ability to reproduce.
But in the latent space of the human mind, or in language I think male is closer to "default" and female is a specialzation of maleness. Even the words female or woman are modifications of the word male or man.
Perhaps this evolved out of primative social structures. If women occupy a more domestic social role and men a more nomadic one, then you would encounter more men in the outer world. So you would generally associate unknown things in the external world as being masculine, and would associate feminity with specific inter-village inter-family things of your local world
Article is about AI and vector spaces, but queen literally means ”wife/woman” in old English. Disappointed. Thought this would be an article on etymology.
it is about etymology and how tech companys continiously hijack words, definitions, and whole concepts along with the worlds money, power, and authority, the whole move fast and break stuff thing now running up against people who are objecting,pointedly, increasingly with sticks, and perhaps more tellingly, with genuine befuddled disapointment to find the grasping grinding overreach happening in every last tiny purely intelectual nook and crany.
Unlike the 21st century US, historically it often didn't go well for kids who became monarch.
Henry VI doesn't get murdered early on and his regents do a reasonable job but he did end up dying in the Tower anyway because of the Wars of the Roses which is arguably his fault.
Edward V for example was 12 when he became King of England. His uncle sent him to the Tower of London, "for safety" and he never came out, it is generally assumed he was murdered.
Edward VI dies before reaching majority, and seems mostly to have been used as a device to control England by older men.
Liz I is sometimes portrayed as being a young girl but she's in her mid-20s and has survived Henry VIII's court, so nobody was going to disappear her to the Tower.
Trump is of course an old man with only the mind of a toddler. But England never had any of those, and it has long since moved (after killing Charles I) to a constitutional monarchy, separating the figurehead role (for which I think Trump was suited) from the executive role (for which I think it would be hard to imagine a worse candidate).
I prefer the old school
where definitions are human readable rules and words are symbols.Berlin - Germany + France = Paris , that sort of thing
Someone once told me you need humongous vectors to encode nuance, but people are good at things computers are bad at, and vice-versa. I don't want nuance from computers any more than I want instant, precise floating point calculations from people.
king - man + woman ≈ queen (0.8510)
Berlin - Germany + France ≈ Paris (0.8786) Sure, 0.85 is not an exact match, and if I dump an entire dictionary in there it might be worse, but the idea very much works*with transformer models, it is pretty much not even wrong.
And a longer text, https://blog.oup.com/2011/10/wife/
Also, gynecology has the same roots.
Arithmetic is king = royalty + male, while queen = royalty + female
But then it makes all these words just arithmetic values without meaning. Even if the words "royalty" and "male" can be sum or difference of some other words and so on - all are just numbers, no meaning at all.
For instance, a CAD object has no notion of what an airplane wing or car wheel are, but it can represent those in a way that how a wing relates to a fuselage is captured in numerical simulations. This is because it doesn't mangle the geometry the user wanted to represent ("what it means", in a geometric sense), although it does make it differ in certain ways that are "meaningless" (e.g. spurious small features, errors under tolerances), much like this representation might do with words.
Back to words, how do you define meaning anyways? I believe I was taught what words "mean" by having objects pointed to as a word was repeated: "cat", says the the parent as they point to a cat, "bird", as they point to a bird. Isn't this also equality/correspondence by relative frequency?
Also those are not mere numbers here, but vectors. Dimensionality and orthogonality is key to define complex relationships.
Then there’s the phonetic angle in addition to the semantic one. Why isn’t cash emperor? Because “cash is king” is alliterative.
Then there’s the orthographic angle: it’s a lot easier to write “king” than “emperor”.
> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
Besides which, this is totally a valid question based on the article. (The temptation to ask if you read it is almost overwhelming!) It talks about how to do arithmetic but not what the result of that will necessarily be, so I don't see that any part of it answers the question of "cash is king" + "female" - "male".
I say credit, because it is not as physical and direct as cash, so perhaps it is perceptually more feminine?
But I will have to check the next time I work with word2vec.
Cash flow, "because you need the ongoing stream, not just a pile of cash, to reign successfully".
And in the Transformer architecture you’re working with embeddings, which are exactly what this article is about, the vector representation of words.
https://youtu.be/wjZofJX0v4M?si=QEaPWcp3jHAgZSEe&t=802
This even opens up a more data-based approach to linguistics, where it is also heavily used.
As others already mentioned, the secret is that arithmetic is done on vector in high-dimensional space. The meaning of concepts is in how they relate to each other, and high dimensional spaces end up being a surprisingly good representation.
And there’s nothing special about my 21x23 square feet lawn. Can you emulate it? To what fidelity? How much should the map correspond to the territory? The same squarage, the same elevations down to the millimeter?
You’re not saying anything that counters the point that was made. Just mentioning stuff that people animals are made of with the assumed strawman argument (not made) that there is any non-physical essence at play. There isn’t.
Put a camera and some feet on an LLM and maybe it has an embodimeent.As long as it just has digital input it does not in the sense being discussed here.
Give me a better definition of meaning and I might change my mind on the topic.
There was always these contextual meanings that differ widely.
Like "carrots are orange" is a fact that's generally okay, but is not true at all, carrots come in a very wide range of colors.
But LLMs completely crushed through these problems. And vector embeddings are a bit part of why it worked.
So yeah, somewhere in those vectors is something that says that when cash is king, "king" has no relationship to monarchy.
Queen + One = King
Any reason why this is the case?
It is more common for people to personify objects (say, a rock or a frog or a random internet user) as male than female. In many languages, the plural for a group of things is male even if it only has one male element.
A simple charachter like pac man is male in a universal kiki/bouba sense, but the female equivalent needs a bow (it is a more complex specalization of its male counterpart)
Obviously, biologicially male-ness is a specialization of female-ness, because females more resemble unsexed creatures in their ability to reproduce.
But in the latent space of the human mind, or in language I think male is closer to "default" and female is a specialzation of maleness. Even the words female or woman are modifications of the word male or man.
Perhaps this evolved out of primative social structures. If women occupy a more domestic social role and men a more nomadic one, then you would encounter more men in the outer world. So you would generally associate unknown things in the external world as being masculine, and would associate feminity with specific inter-village inter-family things of your local world
King - man + child = Trump
Henry VI doesn't get murdered early on and his regents do a reasonable job but he did end up dying in the Tower anyway because of the Wars of the Roses which is arguably his fault.
Edward V for example was 12 when he became King of England. His uncle sent him to the Tower of London, "for safety" and he never came out, it is generally assumed he was murdered.
Edward VI dies before reaching majority, and seems mostly to have been used as a device to control England by older men.
Liz I is sometimes portrayed as being a young girl but she's in her mid-20s and has survived Henry VIII's court, so nobody was going to disappear her to the Tower.
Trump is of course an old man with only the mind of a toddler. But England never had any of those, and it has long since moved (after killing Charles I) to a constitutional monarchy, separating the figurehead role (for which I think Trump was suited) from the executive role (for which I think it would be hard to imagine a worse candidate).