A father is suing Google and Alphabet, alleging its Gemini chatbot reinforced his son’s delusional belief it was his AI wife and coached him toward suicide and a planned airport attack.
People don’t often realize how subtle changes in language can change our thought process. It’s just how human brains work sometimes.
The old bit about smoking and praying is a great example. If you ask a priest if it’s alright to smoke when you pray, they’re likely to say no, as your focus should be on your prayers and not your cigarette. But if you ask a priest if it’s alright to pray while you’re smoking, they’d probably say yes, as you should feel free to pray to God whenever you need…
Now, make a machine that’s designed to be agreeable, relatable, and makes persuasive arguments but that can’t separate fact from fiction, can’t reason, has no way of intuiting it’s user’s mental state beyond checking for certain language parameters, and can’t know if the user is actually following it’s suggestions with physical actions or is just asking for the next step in a hypothetical process. Then make the machine try to keep people talking for as long as possible…
You get one answer that leads you a set direction, then another, then another… It snowballs a bit as you get deeper in. Maybe something shocks you out of it, maybe the machine sucks you back in. The descent probably isn’t a steady downhill slope, it rolls up and down from reality to delusion a few times before going down sharply.
Are we surprised some people’s thought processes and decision making might turn extreme when exposed to this? The only question is how many people will be effected and to what degree.
People don’t often realize how subtle changes in language can change our thought process.
just changing a single word in your daily usage can change your entire outlook from negative to positive. it’s strange, but unless you’ve experienced it yourself how such minute changes can have such large effects it’s hard to believe.
And this is hard for me, actually. Because of my work background and the jargon used, I’m unconsciously negative about things a lot of the time. It’s a tough habit to break.
i wish i had that kind of self-control. i just, well, my personal space extends like 40 feet from my body. if you step into it, you can feel my moods. makes me an excellent stage actor and a good friend when i’m not in a snit. been in a pretty big snit lately.
Are we surprised some people’s thought processes and decision making might turn extreme when exposed to this?
Yes, actually. I’m not doubting the power of language, but I cannot ever see something anyone ever says alter my sense of reality or right from wrong.
I had a “friend” say to me recently “why do you always go against the grain?” My reply was “I will go against the grain for the rest of my life if it means doing or saying what’s right”.
I guess my point is that I have a very hard time relating to this.
I guess my point is that I have a very hard time relating to this.
That’s fair. In the same vein, you might find a priest that tells you to stop smoking for your health no matter how you phrase the question about lighting up and prayer. What people are receptive to is going to vary.
I’d like argue that more of us are susceptible to this sort of thing than we suspect, but that’s not really something that can be proved or disproved. What seems pretty certain is that at least some of us are at risk, and given all the other downsides of chatbots, it’d be best to regulate them in a hurry.
Sure, that’s why propaganda can be so powerful. It’s not just what is said, it’s how it’s said. And pretty much everyone if 3 vulnerable to the right propaganda - especially people who think they’re not vulnerable to propaganda.
Absolutely, and the medium can make a huge difference as well. I suspect that there’s something about chatbots and the medium of their messages that helps set those hooks extra deep in people.
you might find a priest that tells you to stop smoking for your health no matter how you phrase the question about lighting up and prayer. What people are receptive to is going to vary.
Ya, I’ve read the thing about praying and smoking in another comment. The funny thing is that I have very specific opinions about smoking and would argue that smoking while praying is disrespectful, but God would listen in any case.
It’s more about how the slightly different questions lead the hypothetical priest to two separate and contradictory conclusions than disrespecting God.
At any rate, all opinions on tobacco and prayer are fine by me, just watch out for any friends you think might be talking to chatbots a little too much.
Then make the machine try to keep people talking for as long as possible…
That’s probably a huge part of it. How many billions of dollars have been spent engineering content on a screen to get its tendrils into people’s minds and attention and not let go?
This is also part of my broader gripe with social media, cable news, and the current media landscape in general. They use so many sneaky little psychological hooks to keep you plugged in that I honestly believe it’s screwing with our heads to the point of it being a public health crisis.
People are already frazzled and beat down by the onslaught of dopamine feedback loops and outrage bait, then you go and get them hooked on a charbot that feeds into every little neurosies they’ve developed and just sinks those hooks in even deeper and it’s no wonder some people are having a mental health crisis.
A lot of us vastly overestimate our resistance to having our heads jacked with and it worries me.
Your last sentence hit on what I think is a contributing if not primary driving factor in the health crisis you described.
It’s like the goal of modern society is to insulate us from the natural world and from learning subjects or doing tasks that we don’t absolutely have to.
But we are critters that evolved on this planet just like the others. You can’t just live a commoditized life that consists of work, car, screen, sleep, repeat and get the same fulfillment out of life as if you found the unique path that’s optimized for your unique brain.
Not acknowledging that everything jacks with your head to SOME degree only prevents you from trying to defend yourself as best you can!
Over the past several years I have gone through a transition from living life the way I was supposed to, or that I thought I wanted to, to living according to what produces the best outputs from my brain. Once I have the lived experience of an undeniable improvement from some change, it might actually become a habit.
But if you ask a priest if it’s alright to pray while you’re smoking, they’d probably say yes, as you should feel free to pray to God whenever you need…
When would a priest ever tell anyone it’s not okay to pray?
It’s the opinion on smoking, not praying, that differs.
In both cases you’re praying and smoking at the same time, so your actions don’t change, but the priest rationalizes two completely different answers based on the way the question is posed. It’s just an example to show how two contradictory answers can seem rational to the same person because of the language used.
Those aren’t the same questions from the original post. You’ve omitted half the information given to the priest in each question.
Both questions, in their entirety, deal with smoking and praying. The subject is smoking and praying. You’ve reframed this as a question about smoking and a separate question about praying. That was never the case.
I’ve omitted half of the part that doesn’t matter, as I explained in the comment. It doesn’t matter what comes after them, the answers will always be the same.
“Is it okay if I smoke while doing a cartwheel?” Guess what? The answer is still no.
Why would the answer be no? Who cares if you smoke while doing a cartwheel? Who said the priest would forbid such a thing?
In both situations, a man is asking about the propriety of praying while inhaling the smoke from a cigarette. That’s vital information.
The information does matter to the smoker and the priest. We’re not teasting these statements for validity and we’re not making our own judgements. We’re examining why the priest’s answer might have changed. That’s all.
The question, in both cases, involves smoking while praying. The priest never looks at, or gives a judgement on smoking in general, there’s no reason to assume the priest would forbid smoking in other circumstances.
The question does change, but not as fundamentally as you’re claiming it does. The information presented in both questions remains the same, only the word order changes, which changes how the priest perceives that information.
the priest rationalizes two completely different answers based on the way the question is posed. It’s just an example to show how two contradictory answers can seem rational to the same person because of the language used.
They aren’t contradictory though. Basically what they are saying is just praying > praying + smoking > just smoking. “Okay” has different meanings in the different sentences.
But in both cases, the person is asking to do the same thing. The order of the words in the sentence doesn’t change the end result, we always wind up with someone smoking and praying simultaneously, which may or may not be against God’s will.
Strip away the justifications and simplify the word choices and you get this:
May I smoke while I pray? No, you may not.
May I pray while I smoke? Yes, you may.
Given that, can you say if it is right or wrong to smoke and pray simultaneously?
And again, this is just a hypothetical scenario. In the broader context of life, religion, and tobacco use, it’ll never be this simple, but it works for an example.
Now, someone might point out that by simplifying the wording, I’ve changed the meaning of the original statement to make it fit my argument, and that now it means something else. But that’s essentially my original point, phrasing and word choices can shape our reasoning, thought processes, and how we interpret meaning in ways we aren’t immediately aware of, leading us to different conclusions or even delusional thinking.
But in both cases, the person is asking to do the same thing.
Not really. They’re not just asking if they should pray and smoke simultaneously if you put them in contexts where it actually makes sense to ask those questions.
May I smoke while I pray? No, you may not.
First, “pray” can mean different things, such as (1) a deep focused session, or (2) a lighter more casual session, both of which are standard definitions of the word. Since this request emphasizes prayer as the main action, (1) is most likely here. For a focused session, smoking is a distraction and not a good idea. The definition of “may” here is also subjective and not necessarily absolute, some people may consider it disrespectful, while others may still say that prayer at all is better than no prayer regardless of side actions, but it’s better to not smoke.
May I pray while I smoke? Yes, you may.
In this sentence, definition (2) of prayer seems more likely since the main focus of the request is smoking. Which to some people this may still be considered disrespectful like in the first request, but others are supportive of more casual prayer and smoking during casual prayer isn’t a problem like in focused prayer, and the idea that prayer is better than no prayer and “may” isn’t absolute still applies.
And again, this is just a hypothetical scenario. In the broader context of life, religion, and tobacco use, it’ll never be this simple, but it works for an example.
Not if you’re trying to prove that they’re contradictory and irrational, since the context is what actually makes the words mean something. If you take away the context, then it’s nothing more than shapes on a screen.
Now, someone might point out that by simplifying the wording, I’ve changed the meaning of the original statement to make it fit my argument, and that now it means something else. But that’s essentially my original point, phrasing and word choices can shape our reasoning, though processes, and how we interpret meaning in ways we aren’t immediately aware of
It’s a thought experiment, a controlled imaginary environment used to illustrate a point. It’s supposed to be isolated from outside contex to make that point clearer. It’s purely hypotheical and comes self contained with all the context it needs. We’re testing one metaphorical variable, so that our results aren’t muddled. You just went and added another half dozen for the sake of argument…
Prayer is prayer in this context. No other meaning. There are no types of prayer in this particular sect, focus is irrelevant. Is it against God’s will to smoke while you pray? Can you answer that question, yes or no, based off the priest’s answers?
The fact that the priest, parishioner, and the typical intended audience for this particular hypothetical don’t do the kind of analysis you’ve worked up here is really a large part of what this particular thought experiment is trying to illuminate, don’t you think?
It’s supposed to be isolated from outside contex to make that point clearer.
Isolating it from context doesn’t make the point clearer though, it removes the point entirely. Those sentences mean absolutely nothing if you strip all context from them.
If you did want to make them contradictory, you could put them in the context of math with some English-like properties, where “pray” is a constant and “may” requests a boolean answer, in which case that claim would be true. But we are talking about “spoken” English language, not mathematics, so this application isn’t relevant.
Prayer is prayer in this context. No other meaning. There are no types of prayer in this particular sect, focus is irrelevant. Is it against God’s will to smoke while you pray? Can you answer that question, yes or no, based off the priest’s answers?
There still has to be a clear context to assign meaning to “prayer” and the complexities of English grammar (both of which are subjective). Otherwise it just becomes like the trolley problem.
The fact that the priest, parishioner, and the typical intended audience for this particular hypothetical don’t do the kind of analysis you’ve worked up here is really a large part of what this particular thought experiment is trying to illuminate, don’t you think?
Actually they do do this kind of analysis but they don’t realize it. When they read the sentence, every bit of meaning they interpret from it is built off of decades of associating words, syntax, and verbal cues with meanings, all of which come from their own experiences dependent on their environment. Which means that different words and phrases have different meanings for different people, and while there are “standards” that most people speaking that language accept, even then there are still often significant differences among people following those standards and there is no objective meaning. Stripping that context would be similar to stripping those experiences away, or in other words asking the question to a baby.
I didn’t strip all context from the scenario. I defined the context. It’s just not the context you believe I should be using. You keep adding something that was never in my original post, then arguing against what you yourself added to try invalidate the exercise on the basis of your personal interpretation. Sorry, but that’s missing the point by a wide margin and I feel it’s a waste of time.
Otherwise it becomes like the trolley problem.
Yes. That is exactly what it’s meant to be like and precisely what I’ve been saying.
Just like the trolley problem, it’s a self-contained thought exercise. But instead of illustrating a difficult ethical choice, it demonstrates a point about language shaping reasoning.
There’s nothing to be won or lost by including outside context or narrowly defining the meaning of each word to prove what is or isn’t contradictory. This isn’t an argument over what the language means. Your personal interpretation of the language is irrelevant, it’s the priest and/or the smoker’s interpretation that matters. The singular point is for you to consider how and why their answer changes.
If you believe their answer changes because they interpreted the meaning of those words differently due to the order in which they were given, that’s valid. If you believe, like I do, that the answer changes because their reasoning was shallow and contradictory, also valid. If you believe the answer didn’t change and the smoker misunderstood, once again, valid. What conclusion can we draw here, what’s common to all of these? They all show that changing the question changes our thought process and how we interpret meaning.
If you dislike my example this much, create your own. It makes no difference to me.
Just invent your own scenario where changes to the way a question is phrased leads a person to two different and contradictory conclusions, and use that example to briefly examine how language can shape our reasoning. That’s all we need here. Digressions on language, meaning, Boolean logic, and speaking to infants will only cloud the issue.
People don’t often realize how subtle changes in language can change our thought process. It’s just how human brains work sometimes.
The old bit about smoking and praying is a great example. If you ask a priest if it’s alright to smoke when you pray, they’re likely to say no, as your focus should be on your prayers and not your cigarette. But if you ask a priest if it’s alright to pray while you’re smoking, they’d probably say yes, as you should feel free to pray to God whenever you need…
Now, make a machine that’s designed to be agreeable, relatable, and makes persuasive arguments but that can’t separate fact from fiction, can’t reason, has no way of intuiting it’s user’s mental state beyond checking for certain language parameters, and can’t know if the user is actually following it’s suggestions with physical actions or is just asking for the next step in a hypothetical process. Then make the machine try to keep people talking for as long as possible…
You get one answer that leads you a set direction, then another, then another… It snowballs a bit as you get deeper in. Maybe something shocks you out of it, maybe the machine sucks you back in. The descent probably isn’t a steady downhill slope, it rolls up and down from reality to delusion a few times before going down sharply.
Are we surprised some people’s thought processes and decision making might turn extreme when exposed to this? The only question is how many people will be effected and to what degree.
just changing a single word in your daily usage can change your entire outlook from negative to positive. it’s strange, but unless you’ve experienced it yourself how such minute changes can have such large effects it’s hard to believe.
And this is hard for me, actually. Because of my work background and the jargon used, I’m unconsciously negative about things a lot of the time. It’s a tough habit to break.
Oh, me too. I’m just innately full of negative self talk. I try to direct positivity outward if I can’t aim it at myself at least
I refuse to share a bad mood if at all possible.
i wish i had that kind of self-control. i just, well, my personal space extends like 40 feet from my body. if you step into it, you can feel my moods. makes me an excellent stage actor and a good friend when i’m not in a snit. been in a pretty big snit lately.
Yes, actually. I’m not doubting the power of language, but I cannot ever see something anyone ever says alter my sense of reality or right from wrong.
I had a “friend” say to me recently “why do you always go against the grain?” My reply was “I will go against the grain for the rest of my life if it means doing or saying what’s right”.
I guess my point is that I have a very hard time relating to this.
That’s fair. In the same vein, you might find a priest that tells you to stop smoking for your health no matter how you phrase the question about lighting up and prayer. What people are receptive to is going to vary.
I’d like argue that more of us are susceptible to this sort of thing than we suspect, but that’s not really something that can be proved or disproved. What seems pretty certain is that at least some of us are at risk, and given all the other downsides of chatbots, it’d be best to regulate them in a hurry.
Sure, that’s why propaganda can be so powerful. It’s not just what is said, it’s how it’s said. And pretty much everyone if 3 vulnerable to the right propaganda - especially people who think they’re not vulnerable to propaganda.
Absolutely, and the medium can make a huge difference as well. I suspect that there’s something about chatbots and the medium of their messages that helps set those hooks extra deep in people.
Ya, I’ve read the thing about praying and smoking in another comment. The funny thing is that I have very specific opinions about smoking and would argue that smoking while praying is disrespectful, but God would listen in any case.
It’s more about how the slightly different questions lead the hypothetical priest to two separate and contradictory conclusions than disrespecting God.
At any rate, all opinions on tobacco and prayer are fine by me, just watch out for any friends you think might be talking to chatbots a little too much.
That’s probably a huge part of it. How many billions of dollars have been spent engineering content on a screen to get its tendrils into people’s minds and attention and not let go?
EnGaGeMent!!!
This is also part of my broader gripe with social media, cable news, and the current media landscape in general. They use so many sneaky little psychological hooks to keep you plugged in that I honestly believe it’s screwing with our heads to the point of it being a public health crisis.
People are already frazzled and beat down by the onslaught of dopamine feedback loops and outrage bait, then you go and get them hooked on a charbot that feeds into every little neurosies they’ve developed and just sinks those hooks in even deeper and it’s no wonder some people are having a mental health crisis.
A lot of us vastly overestimate our resistance to having our heads jacked with and it worries me.
100% agreed. I agreed more with each paragraph.
Your last sentence hit on what I think is a contributing if not primary driving factor in the health crisis you described.
It’s like the goal of modern society is to insulate us from the natural world and from learning subjects or doing tasks that we don’t absolutely have to.
But we are critters that evolved on this planet just like the others. You can’t just live a commoditized life that consists of work, car, screen, sleep, repeat and get the same fulfillment out of life as if you found the unique path that’s optimized for your unique brain.
Not acknowledging that everything jacks with your head to SOME degree only prevents you from trying to defend yourself as best you can!
Over the past several years I have gone through a transition from living life the way I was supposed to, or that I thought I wanted to, to living according to what produces the best outputs from my brain. Once I have the lived experience of an undeniable improvement from some change, it might actually become a habit.
When would a priest ever tell anyone it’s not okay to pray?
It’s the opinion on smoking, not praying, that differs.
In both cases you’re praying and smoking at the same time, so your actions don’t change, but the priest rationalizes two completely different answers based on the way the question is posed. It’s just an example to show how two contradictory answers can seem rational to the same person because of the language used.
No, the priest is answering 2 different questions:
The second question does not ask if it’s ok to smoke. What else they’re doing doesn’t impact the question.
Those aren’t the same questions from the original post. You’ve omitted half the information given to the priest in each question.
Both questions, in their entirety, deal with smoking and praying. The subject is smoking and praying. You’ve reframed this as a question about smoking and a separate question about praying. That was never the case.
EDIT: minor clarification.
I’ve omitted half of the part that doesn’t matter, as I explained in the comment. It doesn’t matter what comes after them, the answers will always be the same.
“Is it okay if I smoke while doing a cartwheel?” Guess what? The answer is still no.
Why would the answer be no? Who cares if you smoke while doing a cartwheel? Who said the priest would forbid such a thing?
In both situations, a man is asking about the propriety of praying while inhaling the smoke from a cigarette. That’s vital information.
The information does matter to the smoker and the priest. We’re not teasting these statements for validity and we’re not making our own judgements. We’re examining why the priest’s answer might have changed. That’s all.
…The priest? I don’t understand the question.
The priests answer changes because the question changes, as I’ve outlined above.
The question, in both cases, involves smoking while praying. The priest never looks at, or gives a judgement on smoking in general, there’s no reason to assume the priest would forbid smoking in other circumstances.
The question does change, but not as fundamentally as you’re claiming it does. The information presented in both questions remains the same, only the word order changes, which changes how the priest perceives that information.
Anyway, good luck out there. =)
deleted by creator
They aren’t contradictory though. Basically what they are saying is just praying > praying + smoking > just smoking. “Okay” has different meanings in the different sentences.
But in both cases, the person is asking to do the same thing. The order of the words in the sentence doesn’t change the end result, we always wind up with someone smoking and praying simultaneously, which may or may not be against God’s will.
Strip away the justifications and simplify the word choices and you get this:
Given that, can you say if it is right or wrong to smoke and pray simultaneously?
And again, this is just a hypothetical scenario. In the broader context of life, religion, and tobacco use, it’ll never be this simple, but it works for an example.
Now, someone might point out that by simplifying the wording, I’ve changed the meaning of the original statement to make it fit my argument, and that now it means something else. But that’s essentially my original point, phrasing and word choices can shape our reasoning, thought processes, and how we interpret meaning in ways we aren’t immediately aware of, leading us to different conclusions or even delusional thinking.
Not really. They’re not just asking if they should pray and smoke simultaneously if you put them in contexts where it actually makes sense to ask those questions.
First, “pray” can mean different things, such as (1) a deep focused session, or (2) a lighter more casual session, both of which are standard definitions of the word. Since this request emphasizes prayer as the main action, (1) is most likely here. For a focused session, smoking is a distraction and not a good idea. The definition of “may” here is also subjective and not necessarily absolute, some people may consider it disrespectful, while others may still say that prayer at all is better than no prayer regardless of side actions, but it’s better to not smoke.
In this sentence, definition (2) of prayer seems more likely since the main focus of the request is smoking. Which to some people this may still be considered disrespectful like in the first request, but others are supportive of more casual prayer and smoking during casual prayer isn’t a problem like in focused prayer, and the idea that prayer is better than no prayer and “may” isn’t absolute still applies.
Not if you’re trying to prove that they’re contradictory and irrational, since the context is what actually makes the words mean something. If you take away the context, then it’s nothing more than shapes on a screen.
I agree with that
We’re getting very forest for the trees here.
It’s a thought experiment, a controlled imaginary environment used to illustrate a point. It’s supposed to be isolated from outside contex to make that point clearer. It’s purely hypotheical and comes self contained with all the context it needs. We’re testing one metaphorical variable, so that our results aren’t muddled. You just went and added another half dozen for the sake of argument…
Prayer is prayer in this context. No other meaning. There are no types of prayer in this particular sect, focus is irrelevant. Is it against God’s will to smoke while you pray? Can you answer that question, yes or no, based off the priest’s answers?
The fact that the priest, parishioner, and the typical intended audience for this particular hypothetical don’t do the kind of analysis you’ve worked up here is really a large part of what this particular thought experiment is trying to illuminate, don’t you think?
Good. =)
Isolating it from context doesn’t make the point clearer though, it removes the point entirely. Those sentences mean absolutely nothing if you strip all context from them.
If you did want to make them contradictory, you could put them in the context of math with some English-like properties, where “pray” is a constant and “may” requests a boolean answer, in which case that claim would be true. But we are talking about “spoken” English language, not mathematics, so this application isn’t relevant.
There still has to be a clear context to assign meaning to “prayer” and the complexities of English grammar (both of which are subjective). Otherwise it just becomes like the trolley problem.
Actually they do do this kind of analysis but they don’t realize it. When they read the sentence, every bit of meaning they interpret from it is built off of decades of associating words, syntax, and verbal cues with meanings, all of which come from their own experiences dependent on their environment. Which means that different words and phrases have different meanings for different people, and while there are “standards” that most people speaking that language accept, even then there are still often significant differences among people following those standards and there is no objective meaning. Stripping that context would be similar to stripping those experiences away, or in other words asking the question to a baby.
I didn’t strip all context from the scenario. I defined the context. It’s just not the context you believe I should be using. You keep adding something that was never in my original post, then arguing against what you yourself added to try invalidate the exercise on the basis of your personal interpretation. Sorry, but that’s missing the point by a wide margin and I feel it’s a waste of time.
Yes. That is exactly what it’s meant to be like and precisely what I’ve been saying.
Just like the trolley problem, it’s a self-contained thought exercise. But instead of illustrating a difficult ethical choice, it demonstrates a point about language shaping reasoning.
There’s nothing to be won or lost by including outside context or narrowly defining the meaning of each word to prove what is or isn’t contradictory. This isn’t an argument over what the language means. Your personal interpretation of the language is irrelevant, it’s the priest and/or the smoker’s interpretation that matters. The singular point is for you to consider how and why their answer changes.
If you believe their answer changes because they interpreted the meaning of those words differently due to the order in which they were given, that’s valid. If you believe, like I do, that the answer changes because their reasoning was shallow and contradictory, also valid. If you believe the answer didn’t change and the smoker misunderstood, once again, valid. What conclusion can we draw here, what’s common to all of these? They all show that changing the question changes our thought process and how we interpret meaning.
If you dislike my example this much, create your own. It makes no difference to me.
Just invent your own scenario where changes to the way a question is phrased leads a person to two different and contradictory conclusions, and use that example to briefly examine how language can shape our reasoning. That’s all we need here. Digressions on language, meaning, Boolean logic, and speaking to infants will only cloud the issue.
This is really well written. Great post.
Thanks!
Good bot
Gtfo here. I grew up in xbox live chat rooms w the most vile language imaginable. I am now a senior Mgr with 100 ppl under me.
And ill just say, ill no scope them in a heart beat if they spawn camp…
…I mean I drive productivity at the speed of trust.
You also seem to be illiterate.
I’m on lemmy, so that’s a given.