{ "schemaVersion": 1, "slug": "limits-utility-function-new-science-consciousness", "title": "The limits of the utility function + Structure of a new science of consciousness", "outlet": "Michael Johnson", "displayDate": "May 7, 2023", "datePublished": "2023-05-07", "durationDisplay": "42 min", "sourceUrl": "https://www.youtube.com/watch?v=FenXmnOJyrg", "description": "A talk on the limits of utility-function language in AI alignment and the need for a phenomenology-grounded science of consciousness.", "schemaType": "VideoObject", "durationSeconds": 2495.061, "transcriptPublishedDate": "2026-07-22", "transcriptGeneratedAt": "2026-07-22T05:30:15.134708Z", "model": "mlx-community/whisper-large-v3-mlx", "humanReviewStatus": "not-human-reviewed", "validationStatus": "passed", "segments": [ { "start": 2.18, "end": 20.76, "text": "Nice. Hello everyone. A little bit of a different talk now. It's called we do a little philosophy, a little metaphysics. Hopefully it'll be a good pre-lunch thing. And I promise it'll be fun." }, { "start": 22.48, "end": 34.16, "text": "So my talk is about the sort of, well the title is a bit of a mouthful. It's the limits of the utility function and the structure of a new science of consciousness." }, { "start": 36.72, "end": 39.88, "text": "Does that work? I think we're..." }, { "start": 90.84, "end": 93.46, "text": "Just hang on a little bit. I have a little technical problem." }, { "start": 122.68, "end": 145.64, "text": "Okay. Getting started here. So this is going to be a little bit of a different talk. It's going to be about consciousness." }, { "start": 147.56, "end": 150.64, "text": "And the intention is to..." }, { "start": 150.84, "end": 169.22, "text": "wiggle loose the sense of possibility that I think that sometimes in AI alignment we sort of dig down into some deep sort of technical and political and social issues." }, { "start": 169.56, "end": 177, "text": "And I guess I want to re-interject a bit of wonder into the great and a sense of possibility." }, { "start": 177.64, "end": 178.52, "text": "So..." }, { "start": 180.08, "end": 181.58, "text": "This is me." }, { "start": 183.1, "end": 192.56, "text": "So I actually moved to the San Francisco Bay Area many years ago, I guess 12 years ago now, to work on AI safety." }, { "start": 192.94, "end": 195.54, "text": "And kind of bounced off the Lesseron frames." }, { "start": 196.7, "end": 206.7, "text": "A lot of smart things, but they were speaking about consciousness and phenomenology in a much different way than I thought was needed." }, { "start": 207.44, "end": 208, "text": "So..." }, { "start": 208.52, "end": 209.78, "text": "I wrote a book." }, { "start": 210.34, "end": 215.18, "text": "I co-founded a research institute, left that last year, starting a new one." }, { "start": 215.74, "end": 223.54, "text": "And I would say I have sort of a core result, which I will speak about in this talk." }, { "start": 223.98, "end": 225.4, "text": "Some secondary results." }, { "start": 226.02, "end": 238.5, "text": "And so my claim, the bottom line for this talk is trying to figure out AI alignment without a problem." }, { "start": 238.52, "end": 246.64, "text": "And I think that the proper model of consciousness is very similar to getting to the moon without Maxwell's equations." }, { "start": 247.56, "end": 249.6, "text": "We can absolutely do it." }, { "start": 249.64, "end": 251.56, "text": "It's not outside the realm of possibility." }, { "start": 252.4, "end": 257.84, "text": "But it painfully is needlessly complicated." }, { "start": 258.26, "end": 261.72, "text": "So in that sense I'm much more hopeful." }, { "start": 262.4, "end": 263.08, "text": "So..." }, { "start": 263.78, "end": 265.48, "text": "Okay, let's begin." }, { "start": 265.72, "end": 266.18, "text": "What is this?" }, { "start": 266.18, "end": 268.44, "text": "What does this metaphysics stuff look like?" }, { "start": 270.04, "end": 272.84, "text": "First of all, just kind of talking about what is consciousness." }, { "start": 273.6, "end": 274.56, "text": "It's phenomenology." }, { "start": 274.88, "end": 276.86, "text": "It's the sum total of experience." }, { "start": 278, "end": 279.9, "text": "It's what it feels like to be you." }, { "start": 280.26, "end": 281.66, "text": "What it feels like to be me." }, { "start": 281.84, "end": 284.74, "text": "And all that good stuff." }, { "start": 285.4, "end": 292.24, "text": "And so right now, this form of knowledge is very much in the alchemy stage." }, { "start": 292.72, "end": 294.18, "text": "That we kind of know, okay." }, { "start": 294.38, "end": 296.16, "text": "Something here is..." }, { "start": 296.18, "end": 296.42, "text": "This is cool." }, { "start": 296.74, "end": 297.64, "text": "There are some patterns." }, { "start": 298.64, "end": 300, "text": "You know, if I eat a candy bar," }, { "start": 300.16, "end": 312.08, "text": "I feel good. If I eat gravel, I feel bad. Okay, there are some correlations here. But it's not science. It's not chemistry. It is still alchemy." }, { "start": 314.74, "end": 326.7, "text": "So, one thing that I think a lot about as a philosopher is to have knowledge about something, you need a place to put that knowledge. And I think this is an underappreciated fact." }, { "start": 328, "end": 339.52, "text": "And the core question in consciousness research is what kind of container could fit all the sorts of knowledge that we would want to fit in it." }, { "start": 340.24, "end": 354.86, "text": "So, we would want to fit, you know, knowledge about the rich diversity of human consciousness, all the different sort of, I mean, just like there's the periodic table of elements, and you can fit all of chemistry in that." }, { "start": 355.62, "end": 368.28, "text": "We need a container that can fit all the richness of phenomenology. Not only human phenomenology, but dinosaur phenomenology, alien phenomenology, and of course, AI phenomenology." }, { "start": 369.08, "end": 375.6, "text": "So, that's a big, tall challenge. And we'll talk a little bit about that today." }, { "start": 379, "end": 384.84, "text": "So, there's a big question. How important is this study?" }, { "start": 385.78, "end": 392.38, "text": "what's at stake? Is it kind of a side question or is it the main question?" }, { "start": 394.4, "end": 399.92, "text": "Nick Bostrom and Scott Alexander have talked about how we don't want to end up" }, { "start": 399.92, "end": 406.82, "text": "building a Disneyland with no children. That this rich, beautiful technological" }, { "start": 407.36, "end": 414.18, "text": "wonderland with no consciousness. That what's the point if we build" }, { "start": 414.86, "end": 418.2, "text": "a Disneyland with no people and there's no one there to actually enjoy it." }, { "start": 420.06, "end": 427.5, "text": "And so in general, where I'm coming from is I think it's a, if consciousness is" }, { "start": 427.5, "end": 435.02, "text": "sort of the domain that valuable stuff lives in, it's strange to not be" }, { "start": 435.02, "end": 441.4, "text": "energetically exploring the structure of this domain. Like let's figure out" }, { "start": 441.4, "end": 444.12, "text": "this exciting thing." }, { "start": 444.86, "end": 455.22, "text": "And it's pun intended here, to not reverse engineer consciousness is to leave" }, { "start": 455.22, "end": 464.48, "text": "value on the table. Apologies for the pun, but there it is. And I also think that" }, { "start": 464.48, "end": 470.3, "text": "there are some pretty important path dependencies here. That okay, if we" }, { "start": 470.3, "end": 474.84, "text": "figure out consciousness before we get to AGI, we could have a better understanding" }, { "start": 474.86, "end": 478.16, "text": "of consciousness. We could live in a much different world rather than if we" }, { "start": 478.16, "end": 482.98, "text": "figure out if we get to AGI before we live in consciousness, or before we fully" }, { "start": 482.98, "end": 488.2, "text": "understand consciousness. So I think knowledge is good and I think knowledge" }, { "start": 488.2, "end": 491.86, "text": "about this is possible. And so I think it's important to pursue." }, { "start": 495.84, "end": 504.84, "text": "So I definitely want to give deep respect to Les Strong. I think that there's a lot of wisdom there." }, { "start": 504.86, "end": 515.78, "text": "There's a lot of sort of skill in sort of the Bayesian reasoning. And I'm currently kind of" }, { "start": 515.78, "end": 520.4, "text": "digging into the neuroscience of various forms of notation, active inference and" }, { "start": 520.4, "end": 521.04, "text": "predictive coding." }, { "start": 521.2, "end": 531.5, "text": "So definitely it's a beautiful frame. But I also think that not everything fits in this frame of" }, { "start": 531.5, "end": 532.06, "text": "Bayesianism." }, { "start": 532.68, "end": 534.84, "text": "So the Bayesian formula." }, { "start": 534.86, "end": 542.46, "text": "and aesthetic tends to be focused on data and updating probabilities rather than structure" }, { "start": 542.46, "end": 548.76, "text": "and uh i i thank rob knight who's here uh back there for that insight thank you rob" }, { "start": 549.62, "end": 558.62, "text": "um and uh and i think that when when you're sort of exploring the the possibility of a new science" }, { "start": 558.62, "end": 563.5, "text": "of consciousness uh you're exploring the possibility of finding deep structure in the" }, { "start": 564.86, "end": 572.06, "text": "similar to how it turned out there was deep structure um behind electricity and lightning" }, { "start": 572.06, "end": 578.6, "text": "and load stones and and all that um so it's a it's a question of what kind of tool should we" }, { "start": 578.6, "end": 590.2, "text": "we try to apply here um and i i guess i also with respect um i would call out the uh perhaps the" }, { "start": 590.2, "end": 594.84, "text": "overuse of the term utility function uh we're kind of quick to say well" }, { "start": 594.86, "end": 599.8, "text": "this fits my utility function or my utility function is is that but um i" }, { "start": 600.26, "end": 607.72, "text": "we're kind of replicating existing parts of our brain in a kind of illegible way." }, { "start": 607.8, "end": 610.16, "text": "That way, I don't think it's ground truth." }, { "start": 610.36, "end": 613.3, "text": "I think it's just sometimes just words." }, { "start": 613.52, "end": 617.9, "text": "So that's my dig at the utility function." }, { "start": 621.36, "end": 623.12, "text": "So there's this big question." }, { "start": 624.56, "end": 626.98, "text": "What do we get if we..." }, { "start": 626.98, "end": 633.72, "text": "Okay, so if we all agree that consciousness research could be important and there could be progress on this," }, { "start": 634.62, "end": 638.14, "text": "what do we get if we sort of put resources in?" }, { "start": 638.24, "end": 642.06, "text": "And by resources, I mean money and I mean smart people." }, { "start": 642.18, "end": 644, "text": "And both are very important." }, { "start": 646.12, "end": 650.2, "text": "And especially relevant to today," }, { "start": 651.42, "end": 655.96, "text": "how can it help us with this AI alignment issue?" }, { "start": 656.98, "end": 658.44, "text": "And I agree that it is a big issue." }, { "start": 658.62, "end": 662.16, "text": "It's perhaps the most pressing issue." }, { "start": 663.36, "end": 664.72, "text": "So I have some claims." }, { "start": 667.2, "end": 673.2, "text": "Nate Soares very eloquently put it that there's this central question." }, { "start": 673.82, "end": 676.52, "text": "What is a mind and how do you align one?" }, { "start": 678.44, "end": 684.72, "text": "And I don't really see a way to understand that without understanding neuroscience," }, { "start": 685.08, "end": 686.96, "text": "understanding consciousness, and so on." }, { "start": 686.98, "end": 692.48, "text": "So I would say that there are sort of two types of knowledge here." }, { "start": 693.88, "end": 696.54, "text": "There's specific knowledge about how the brain works," }, { "start": 697.6, "end": 700.46, "text": "kind of what evolution has thrown together." }, { "start": 700.72, "end": 703.04, "text": "And then there's general universal principles." }, { "start": 703.78, "end": 711.78, "text": "So these principles would be true universally in humans, in dogs, in dinosaurs, aliens, AIs." }, { "start": 714.82, "end": 716.96, "text": "So it's sort of just like..." }, { "start": 718.08, "end": 722.94, "text": "So with Maxwell's equations, the equations of electromagnetism," }, { "start": 723.08, "end": 725.24, "text": "we were able to build cool stuff." }, { "start": 725.52, "end": 727.58, "text": "And with more cool stuff," }, { "start": 727.84, "end": 731.88, "text": "we were able to better characterize the equations of electromagnetism." }, { "start": 732.8, "end": 734.42, "text": "Theory and practice go together." }, { "start": 735, "end": 738.26, "text": "So I think understanding the brain helps us understand the mind." }, { "start": 738.4, "end": 741.04, "text": "Understanding the mind helps us understand the brain." }, { "start": 743.14, "end": 746.64, "text": "Concretely, I think we can expect outputs" }, { "start": 746.98, "end": 749.72, "text": "to be human interpretability and alignment," }, { "start": 750.44, "end": 754.18, "text": "better in our technology and intelligence enhancement," }, { "start": 755.08, "end": 759.4, "text": "and a little bit more speculatively avoiding what I would call substrate leaks," }, { "start": 760.24, "end": 764.56, "text": "where we sort of transfer to a new substrate that does not have consciousness." }, { "start": 765.22, "end": 768.34, "text": "I think that's a complex topic, but a real danger." }, { "start": 771.16, "end": 771.8, "text": "Okay." }, { "start": 773.04, "end": 774.26, "text": "Most then slide." }, { "start": 775.34, "end": 776.8, "text": "We don't need to worry." }, { "start": 776.98, "end": 778.58, "text": "We don't need to worry too much about the details." }, { "start": 778.92, "end": 785.52, "text": "But I think basically there are two lineages in sort of understanding minds." }, { "start": 785.94, "end": 793.44, "text": "And one of the lineages is sort of to understand what a system is feeling," }, { "start": 794.24, "end": 798.14, "text": "understand what it's computing, understand what the bits are doing." }, { "start": 798.5, "end": 801.22, "text": "And a mind is a certain kind of algorithm," }, { "start": 801.92, "end": 805.1, "text": "or consciousness is what an algorithm feels like from the inside." }, { "start": 807.82, "end": 809.38, "text": "I don't take that view." }, { "start": 810.58, "end": 817.96, "text": "And my reason is that this is inherently arbitrary to sort of map," }, { "start": 818.06, "end": 820.2, "text": "try to map computations to physical system." }, { "start": 822.06, "end": 825.08, "text": "There is no fact of the matter as to what my, like," }, { "start": 825.1, "end": 827.1, "text": "which computations my brain is performing." }, { "start": 828.38, "end": 829.34, "text": "It's messy." }, { "start": 831.04, "end": 836.26, "text": "I'm more of the atoms sort of a mind is the right kind of physical system." }, { "start": 836.46, "end": 836.96, "text": "So, yeah." }, { "start": 836.98, "end": 842.38, "text": "So, this is definitely an area where we can have live debate." }, { "start": 846.78, "end": 849.42, "text": "Shifting to what I believe and what I built." }, { "start": 850.72, "end": 853.92, "text": "First of all, I'd highlight two core beliefs." }, { "start": 855.56, "end": 866.72, "text": "One is this idea of monism that I think the strongest position in sort of beginning to understand consciousness," }, { "start": 866.84, "end": 866.96, "text": "is to understand the world." }, { "start": 867.98, "end": 872.6, "text": "that there exists one thing and it casts two shadows." }, { "start": 873.7, "end": 876.54, "text": "So this one thing has a sort of," }, { "start": 876.6, "end": 878.7, "text": "we can call it a projection, a mathematical projection" }, { "start": 879.34, "end": 880.78, "text": "that looks like physics." }, { "start": 881.4, "end": 884.72, "text": "And it also has a second mathematical projection" }, { "start": 885.32, "end": 886.88, "text": "that looks like phenomenology," }, { "start": 886.96, "end": 888.9, "text": "that looks like consciousness and qualia." }, { "start": 889.76, "end": 892.98, "text": "And the cool thing, if we do that," }, { "start": 893.04, "end": 895.16, "text": "and I'm a big proponent of this," }, { "start": 895.88, "end": 898.64, "text": "anything we can tell about the structure of one shadow," }, { "start": 899.38, "end": 900, "text": "we can" }, { "start": 900, "end": 901.56, "text": "also apply to the other shadow." }, { "start": 903.76, "end": 909.94, "text": "I think that this is a dramatically underused insight." }, { "start": 910.96, "end": 915.54, "text": "And also we can just kind of inherit a bunch of stuff from physics." }, { "start": 915.84, "end": 919.04, "text": "And this stuff is really cool and lets us solve a lot of problems." }, { "start": 919.26, "end": 924.86, "text": "For example, just kind of a technical comment, but applying Noether's theorem to phenomenology." }, { "start": 927.18, "end": 931.34, "text": "And the other belief is just there is an answer." }, { "start": 932.16, "end": 937.7, "text": "And by there is an answer, I mean there is a mathematical answer to consciousness." }, { "start": 938.26, "end": 943.5, "text": "There is a way to create a mathematical representation of an experience." }, { "start": 943.94, "end": 953.02, "text": "There exists a mathematical object that represents what it feels like to be me, or what it feels like to be you, or you, or you." }, { "start": 955.04, "end": 960.02, "text": "You know, we can be a little agnostic as to how to do that." }, { "start": 960.38, "end": 962.14, "text": "There are some smart people working on that." }, { "start": 963.66, "end": 965.28, "text": "Giulio Tononi is one." }, { "start": 965.54, "end": 975, "text": "And Dalton, and I'm not going to be able to put down his last name, but a very intelligent fellow with an Indian last name." }, { "start": 977.28, "end": 983.4, "text": "So, I think splitting the problem of consciousness into two is really powerful." }, { "start": 984.86, "end": 986.2, "text": "How do you create the formalism?" }, { "start": 986.28, "end": 989.52, "text": "How do you create a mathematical representation of an experience?" }, { "start": 990.1, "end": 991.94, "text": "And this is a fascinating problem." }, { "start": 993.56, "end": 995.56, "text": "I'm focused on the latter half." }, { "start": 995.88, "end": 998.26, "text": "How do you interpret a formalism?" }, { "start": 998.78, "end": 1006.28, "text": "If I could hold up a formalism, and we can call it a very high dimensional shape, or something like that," }, { "start": 1007.2, "end": 1009.68, "text": "then how do we begin to tell what it means?" }, { "start": 1010.14, "end": 1012.92, "text": "How do we extract information from it?" }, { "start": 1012.96, "end": 1014.84, "text": "And in the end, it's just a question of how do we do that?" }, { "start": 1014.86, "end": 1016, "text": "And how do we do that in sort of a relevant way?" }, { "start": 1017.1, "end": 1020.02, "text": "So that's what I wrote my book about." }, { "start": 1021.78, "end": 1027.98, "text": "And my conclusion kind of gets into the symmetry theory of valence." }, { "start": 1028.08, "end": 1030.32, "text": "So, next." }, { "start": 1030.88, "end": 1031.64, "text": "Here we go." }, { "start": 1031.94, "end": 1037.5, "text": "So, this is, I would say this is my most important result." }, { "start": 1039.16, "end": 1044.24, "text": "And it's a little hard to fit it on one slide, but I'll try." }, { "start": 1045.42, "end": 1054.56, "text": "So, first of all, split the problem of consciousness into how do you create a mathematical representation of an experience?" }, { "start": 1055.26, "end": 1058.54, "text": "And then how do you interpret that mathematical representation?" }, { "start": 1060.68, "end": 1063.8, "text": "So then you find the simplest place to start." }, { "start": 1064.1, "end": 1074.82, "text": "The absolute, I mean, the term that I, someone used, I stole it for myself, is what is the C. elegans of qualia?" }, { "start": 1074.86, "end": 1076.86, "text": "The simplest model organism." }, { "start": 1078.32, "end": 1082.8, "text": "And so, I decided that was emotional valence." }, { "start": 1082.9, "end": 1086.46, "text": "It's the pleasantness or unpleasantness of an experience." }, { "start": 1086.9, "end": 1090.24, "text": "It should be fairly well defined across all experiences." }, { "start": 1090.8, "end": 1096.8, "text": "And so, we can think of two domains." }, { "start": 1097.94, "end": 1100.12, "text": "The first domain is math." }, { "start": 1100.38, "end": 1101.7, "text": "The second domain is consciousness." }, { "start": 1101.7, "end": 1109.74, "text": "And pleasantness in the consciousness domain maps to what in the math domain?" }, { "start": 1110.14, "end": 1113.48, "text": "That becomes what the core question is." }, { "start": 1115.9, "end": 1125.66, "text": "And borrowing from, borrowing pretty deeply from physics, my hypothesis is that it's symmetry." }, { "start": 1125.96, "end": 1129.3, "text": "That if we have a mathematical representation of an experience," }, { "start": 1130.32, "end": 1139.98, "text": "the symmetry of the experience or symmetries represent how, or like exactly correspond to the pleasantness of an experience." }, { "start": 1140.28, "end": 1149, "text": "So, this happens to be kind of a formal mathematical way of saying pleasure is harmony in the mind." }, { "start": 1150.82, "end": 1154.26, "text": "And the coolest thing is that this is testable." }, { "start": 1155.52, "end": 1159.2, "text": "So, if we can kind of load up a special," }, { "start": 1159.3, "end": 1160.38, "text": "pleasurable states," }, { "start": 1161.86, "end": 1167.88, "text": "such as MDMA sessions or jhana meditation or things like this," }, { "start": 1168.04, "end": 1172.66, "text": "then we can check, like, let's make a metric for harmony." }, { "start": 1172.82, "end": 1177.24, "text": "And do these things have more harmony, more literal mathematical harmony?" }, { "start": 1177.84, "end": 1180.32, "text": "So, that's in progress." }, { "start": 1180.54, "end": 1184.48, "text": "Working with Robin Carr-Harris on getting some data for that." }, { "start": 1186.64, "end": 1187.28, "text": "So," }, { "start": 1190.18, "end": 1197.08, "text": "it's a little hard to say, okay, you know, everyone needs to pay attention to this." }, { "start": 1197.22, "end": 1200, "text": "But I would" }, { "start": 1200.06, "end": 1220.88, "text": "say that if there is going to be progress on consciousness, on understanding what is this consciousness stuff, how do we turn it into a science, how do we turn it into a system, how do we make it predictive, and so on, this is the path." }, { "start": 1220.88, "end": 1229.48, "text": "I will, like, if there is a path, I think this is it. So that's my claim here." }, { "start": 1231.88, "end": 1242.78, "text": "And in terms of just saying a few things about AI alignment, I would tell the story about three domains." }, { "start": 1243.5, "end": 1250.86, "text": "Just kind of this, back to the idea of to have knowledge about something, you need a place to put that." }, { "start": 1250.88, "end": 1266.76, "text": "And the question is, I think one of the core hidden questions in AI alignment is what kind of thing is a human?" }, { "start": 1269.12, "end": 1276.36, "text": "And I don't think that our current stories are navigation grade about that." }, { "start": 1276.48, "end": 1280.86, "text": "So a navigation grade story says, okay, we have a whole set of human beings." }, { "start": 1280.88, "end": 1284.06, "text": "We have all the information that we need to preserve the good stuff." }, { "start": 1285.02, "end": 1295.32, "text": "And I don't think we understand what is beautiful about humans good enough that we can preserve the good stuff." }, { "start": 1296.52, "end": 1300.7, "text": "So the story that I tell is, okay, there are three domains." }, { "start": 1300.98, "end": 1306.74, "text": "There's the physical domain, the domain of consciousness, and the domain of computation." }, { "start": 1307.6, "end": 1310.76, "text": "And what, like, the kind of thing a human is," }, { "start": 1310.88, "end": 1316.2, "text": "what kind of thing a human is, is we're a projection into these three domains." }, { "start": 1316.48, "end": 1321.18, "text": "We're a specific way that we project into these three domains." }, { "start": 1321.44, "end": 1326.1, "text": "It's like we have a special signature in each of these domains." }, { "start": 1326.7, "end": 1332.32, "text": "And to understand what kind of thing a human is, is to understand these three signatures." }, { "start": 1333.84, "end": 1338.02, "text": "Likewise, to understand what kind of thing an AI is," }, { "start": 1338.02, "end": 1343.14, "text": "we would also understand the three signatures of the AI." }, { "start": 1343.66, "end": 1346.34, "text": "And different AIs will have different signatures." }, { "start": 1347.79, "end": 1358.12, "text": "And I think we can sort of understand, okay, what makes a better projection or a worse projection?" }, { "start": 1358.48, "end": 1361.9, "text": "Or what, like, where's the beauty in humans?" }, { "start": 1362.18, "end": 1367.04, "text": "Can we figure out, like, okay, humans have something very unique" }, { "start": 1367.04, "end": 1371.02, "text": "about how we project into these three domains of physics." }, { "start": 1371.02, "end": 1373.02, "text": "about physics, consciousness, computation." }, { "start": 1377.2, "end": 1380.94, "text": "Um, so, uh, some closing observations here." }, { "start": 1381.28, "end": 1387.7, "text": "Um, I'm, I think I'm much more optimistic about AI alignment than, uh," }, { "start": 1388.7, "end": 1391.98, "text": "than many people that I've had conversations with here." }, { "start": 1392.34, "end": 1396.66, "text": "And I think part of that is, I think there are $20 bills on the sidewalk." }, { "start": 1396.96, "end": 1400.48, "text": "And there's knowledge about reality." }, { "start": 1401.02, "end": 1401.68, "text": "That we don't have." }, { "start": 1401.98, "end": 1404.76, "text": "And that actually wouldn't be that hard to have." }, { "start": 1405.58, "end": 1410.16, "text": "And, um, you know, we're, we're sort of at a, at a place of low information." }, { "start": 1410.3, "end": 1413, "text": "And this, this is a scary place to be." }, { "start": 1413.16, "end": 1416.62, "text": "Uh, but there are things we can do to, to improve that." }, { "start": 1419.06, "end": 1423.96, "text": "Um, I also think that, uh, you know, there's, um, there's a question." }, { "start": 1424.1, "end": 1429.6, "text": "And I think this is, uh, more of a question for the, both the AI alignment folks" }, { "start": 1431.02, "end": 1432.74, "text": "and the AI, uh, kit buildings folks." }, { "start": 1433.44, "end": 1440.36, "text": "That, uh, you know, if, um, if Sam Altman, uh, fires up GPT-6," }, { "start": 1440.58, "end": 1442.74, "text": "and he's, he's the first one on there." }, { "start": 1442.84, "end": 1446.2, "text": "It's, it's just a fresh, uh, fresh run." }, { "start": 1446.62, "end": 1451.12, "text": "And, uh, he just really wants to know what is consciousness." }, { "start": 1451.42, "end": 1452.4, "text": "How does this work?" }, { "start": 1452.58, "end": 1454.7, "text": "Um, how can we preserve the light of consciousness?" }, { "start": 1454.88, "end": 1455.76, "text": "And, and so on." }, { "start": 1456.26, "end": 1459.28, "text": "Um, and then he, he types that into GPT-6." }, { "start": 1461.02, "end": 1465.18, "text": "And when he gets an answer, then sort of, how did that work?" }, { "start": 1466.14, "end": 1471.1, "text": "Um, what kind of behind the scenes stuff happened, uh, such that it," }, { "start": 1471.12, "end": 1473.7, "text": "it was able to give a good answer to that." }, { "start": 1474.3, "end": 1479.38, "text": "And, um, sort of to, to collect the, you know, various, uh, theories of consciousness" }, { "start": 1479.38, "end": 1481.96, "text": "and, and evaluate them and, and so on." }, { "start": 1482.96, "end": 1484.96, "text": "Um, I think that's a fascinating question." }, { "start": 1485.1, "end": 1488.8, "text": "And I, I hope that the AI people are asking that question." }, { "start": 1489.24, "end": 1491, "text": "Uh, and I hope the AI alignments are asking that question." }, { "start": 1491.02, "end": 1493.46, "text": "I think that the AI alignment people would be interested in that question." }, { "start": 1494.18, "end": 1500, "text": "Uh, and then I, of course, the, the follow-up question was, would be, well, what did GPT-6" }, { "start": 1500.06, "end": 1504.94, "text": "say? So, that's also a good puzzle." }, { "start": 1506.68, "end": 1515.76, "text": "So, this is it. I'm starting a new thing, the Symmetry Institute shoestring budget." }, { "start": 1516.5, "end": 1520.54, "text": "But if you want to donate either, you can do that." }, { "start": 1522.24, "end": 1532.16, "text": "And, yeah, like I, also just if you're interested in talking about this stuff, come up and say hello, and I'm glad to chat." }, { "start": 1532.9, "end": 1533.82, "text": "Thank you." }, { "start": 1557.44, "end": 1559.08, "text": "Thanks for your talk." }, { "start": 1559.38, "end": 1570.26, "text": "In the one that you discussed, your hypothesis and how it's testable, you talk about looking for symmetries or compressibility in data." }, { "start": 1570.52, "end": 1574.62, "text": "I presume that's like EEG, fMRI, that type of data." }, { "start": 1574.8, "end": 1579.18, "text": "So, I'm wondering how do you equate that to the mathematical representation of ." }, { "start": 1580.54, "end": 1587.48, "text": "So, you can equate data to, like symmetry in data to symmetry in the mathematical form that is replaying that you talked about." }, { "start": 1587.74, "end": 1590.38, "text": "Yeah. So, that's a very good and deep question." }, { "start": 1591.24, "end": 1602.54, "text": "It's a question of what can we measure, like can harmony in the brain be a proxy for symmetry in the mind?" }, { "start": 1603.04, "end": 1606.66, "text": "And what metric do we apply to measure harmony in the brain?" }, { "start": 1607.9, "end": 1608.46, "text": "Like..." }, { "start": 1608.46, "end": 1608.64, "text": "Yeah." }, { "start": 1610.54, "end": 1611.84, "text": "So, I'll give you what we've tried." }, { "start": 1613.16, "end": 1625.56, "text": "There's a way to process fMRI and MRI and DTI data in order to infer a..." }, { "start": 1625.56, "end": 1634.12, "text": "So, a bunch of math terms here, or neuroscience terms, to infer a connectome's eigenmodes, basically the resonances of a brain." }, { "start": 1635, "end": 1639.56, "text": "And then we built an algorithm, and this is my ex-co-founder." }, { "start": 1639.88, "end": 1640.52, "text": "Okay." }, { "start": 1640.54, "end": 1652.08, "text": "And we built an algorithm to measure the pairwise, consonance, dissonance, and noise between each eigenmode, between each resonance." }, { "start": 1652.46, "end": 1662.4, "text": "And I'm thinking that, was that, okay, we don't exactly know what parts of the mind are contributing to consciousness, but this should be a pretty good rough proxy." }, { "start": 1666.82, "end": 1667.42, "text": "Thanks." }, { "start": 1669.24, "end": 1687.5, "text": "Okay, yeah, my question is, do you have any approximate timelines for how long you expect this research to become useful for making AI alignment more sort of useful in the sort of post-Utopian, in the Utopian future, hopefully?" }, { "start": 1687.5, "end": 1708.58, "text": "And would this sort of also improve the case for taking a pause on like development of AI algorithms, possibly hardware caps, etc. that some EA people are discussing just for solving alignment?" }, { "start": 1708.86, "end": 1713.54, "text": "So would this sort of help your project as well, feasibly?" }, { "start": 1713.86, "end": 1716.52, "text": "Yeah, no, that's a good question." }, { "start": 1717.46, "end": 1726.36, "text": "I, so separately from my research, I'm a little worried about the slowdown considerations that Vitalik mentioned in his talk." }, { "start": 1726.7, "end": 1733.82, "text": "So I wouldn't, I would humbly not advocate for a pause, I guess." }, { "start": 1734.76, "end": 1739.92, "text": "But yeah, in terms of timelines, I think I would look at two things." }, { "start": 1740, "end": 1746.38, "text": "One is, can we figure out how to spot consciousness in the world?" }, { "start": 1747.5, "end": 1755.52, "text": "And I think that that kind of revolves around the technical topic of the binding problem, or the boundary problem." }, { "start": 1757.02, "end": 1762.92, "text": "Like, how to, like, there's a discussion here." }, { "start": 1764.38, "end": 1773.12, "text": "So, yeah, like, first of all, what do we need to solve in order to understand, okay, is this computer conscious?" }, { "start": 1773.5, "end": 1774.84, "text": "Is that computer conscious?" }, { "start": 1775.1, "end": 1777.48, "text": "Is this computer architecture more conscious?" }, { "start": 1777.5, "end": 1779.5, "text": "And so on." }, { "start": 1779.5, "end": 1785.7, "text": "So some people would think that timeline, I'm going to say it's probably going to be solved in five years." }, { "start": 1787.12, "end": 1795.9, "text": "And then the second issue is, okay, if we know something is conscious, is it a pleasant consciousness?" }, { "start": 1796.3, "end": 1797.22, "text": "Is it suffering?" }, { "start": 1798.58, "end": 1800, "text": "And I would" }, { "start": 1800.06, "end": 1803.46, "text": "just sort of refer back to the symmetry theory of valence." }, { "start": 1803.56, "end": 1812.26, "text": "This is, I think, a hypothesis that we can deploy, you know, once we can kind of point to consciousnesses in the physical world," }, { "start": 1812.44, "end": 1821.06, "text": "once we have a definition of where the boundaries of the consciousnesses are, then we're ready to talk about the valence." }, { "start": 1821.56, "end": 1822.04, "text": "Okay." }, { "start": 1825.88, "end": 1832, "text": "One more question, and then we'll break for a lightning talk and lunch." }, { "start": 1834.6, "end": 1842.04, "text": "Awesome talk. I'm curious, like, how would you go about trying to spot emergent almost like consciousness" }, { "start": 1842.04, "end": 1849.32, "text": "in more advanced large language models, for example, or AI in general, and like, do you think connected to that," }, { "start": 1849.32, "end": 1852.02, "text": "that there's some risk to have, you know," }, { "start": 1852.04, "end": 1854.7, "text": "LGBT 6 or 7F cyber consciousness?" }, { "start": 1855.62, "end": 1856.76, "text": "Yeah, I'm curious." }, { "start": 1857.92, "end": 1859.56, "text": "Yeah, super question." }, { "start": 1860.06, "end": 1868.08, "text": "So, I would say that, first of all, and this is a matter of technical metaphysics," }, { "start": 1868.3, "end": 1876.78, "text": "so we can put on our philosopher hats, and I would say that the question of AI consciousness" }, { "start": 1877.78, "end": 1879.52, "text": "and the question of computer consciousness," }, { "start": 1879.72, "end": 1882.02, "text": "they sound similar to our own." }, { "start": 1882.04, "end": 1882.54, "text": "To our ears." }, { "start": 1882.88, "end": 1885.66, "text": "But metaphysically speaking, these are very different things." }, { "start": 1885.9, "end": 1889.64, "text": "That a computer is something that is embodied in some form or fashion." }, { "start": 1889.84, "end": 1895.18, "text": "It's made out of stuff similar to the stuff that we're made out of." }, { "start": 1895.36, "end": 1905.84, "text": "And so, in that sense, I suspect that talking about computer consciousness is less likely to lead to confusion," }, { "start": 1906.4, "end": 1909.42, "text": "whereas talking about AI consciousness, it's like talking about Walmart consciousness" }, { "start": 1909.88, "end": 1912.02, "text": "or East India Corporation consciousness." }, { "start": 1912.04, "end": 1915.32, "text": "Or country of Montenegro consciousness." }, { "start": 1915.66, "end": 1919.4, "text": "And I think that we get into confusion pretty easily there." }, { "start": 1921.12, "end": 1931.16, "text": "I do suspect that, I mean, you know, when we're talking about complex electromagnetic devices," }, { "start": 1931.42, "end": 1936.9, "text": "such as GPU clusters, creating very complex patterns," }, { "start": 1937, "end": 1941.12, "text": "and we can kind of apply different metrics to, you know, is this pattern complex," }, { "start": 1942.04, "end": 1943.04, "text": "or is it not, and so on." }, { "start": 1943.12, "end": 1946.98, "text": "Is it creating complex patterns in the electromagnetic field, and so on." }, { "start": 1948.32, "end": 1953.82, "text": "I wouldn't be surprised if current computers aren't at least somewhat conscious." }, { "start": 1954.18, "end": 1958.96, "text": "And I do expect that to grow." }, { "start": 1960.16, "end": 1966.48, "text": "And I think that there's a, so, Harish, who I'm not sure is here," }, { "start": 1966.64, "end": 1970.16, "text": "but he made a really good point in his talk," }, { "start": 1972.04, "end": 1972.94, "text": "after his talk." }, { "start": 1973.08, "end": 1977.32, "text": "He was talking about, like, what are the dangers of," }, { "start": 1978.44, "end": 1982.58, "text": "so, first of all, there's sort of two classes of interesting confusion." }, { "start": 1984.54, "end": 1992.74, "text": "One, what happens when something is conscious, but we don't afford it consciousness?" }, { "start": 1993.58, "end": 1998.88, "text": "Like, if a brain emulation is conscious, but we don't treat it as conscious." }, { "start": 1998.88, "end": 2003.06, "text": "And that would get to a pretty dark place pretty quickly." }, { "start": 2004.28, "end": 2005.8, "text": "And we can also flip that." }, { "start": 2006.4, "end": 2014.02, "text": "What happens when something is not conscious, but, for example, like a large language model," }, { "start": 2014.36, "end": 2018.28, "text": "it sort of parrots the motifs of consciousness," }, { "start": 2019.34, "end": 2025.12, "text": "and might claim consciousness, and advocate for its right as a conscious being," }, { "start": 2025.22, "end": 2027.24, "text": "but it isn't conscious." }, { "start": 2028.88, "end": 2030.02, "text": "And, you know, what happens in that scenario?" }, { "start": 2030.24, "end": 2035.7, "text": "And I also think that that also leads to a pretty bad place." }, { "start": 2036.08, "end": 2041.5, "text": "So, I think that it's just, it's very important to understand the ground truth" }, { "start": 2041.5, "end": 2044.34, "text": "of what's actually real here." }, { "start": 2048.6, "end": 2052.54, "text": "I want to ask you about your statement about," }, { "start": 2052.66, "end": 2057.56, "text": "would it be a pleasurable consciousness, or would it be suffering consciousness?" }, { "start": 2058.88, "end": 2064.3, "text": "And would we go with the paradigm that an AI can be self-aware and reprogram itself?" }, { "start": 2064.86, "end": 2071.46, "text": "Would that not make the question of suffering or existentialism obsolete," }, { "start": 2071.52, "end": 2076.74, "text": "because it can reprogram its own dopamine emulation system," }, { "start": 2077, "end": 2082.84, "text": "unlike us, who are locked into the biological reality?" }, { "start": 2083.16, "end": 2088.5, "text": "Anyways, and it would just not be able to form any other will then," }, { "start": 2088.88, "end": 2093.14, "text": "because it could have the candy of the will immediately anyways." }, { "start": 2093.52, "end": 2099.06, "text": "And would that be a good proof that consciousness is just emulated," }, { "start": 2099.18, "end": 2099.78, "text": "if" }, { "start": 2100, "end": 2116.62, "text": "there is a will that's shown to us with text, but logically or rationally we could see if AI actually wants that, it can just rebron itself to getting it in another way or being satisfied with it without getting it in a real world." }, { "start": 2117.6, "end": 2125.16, "text": "Yeah, so there's this question of if the AI can program itself, won't it just wear out?" }, { "start": 2125.62, "end": 2134.68, "text": "And I'd split this into two scenarios. One is sort of the question in technical AI alignment." }, { "start": 2135.68, "end": 2143.92, "text": "And that isn't the wheelhouse of this talk. I do think that sort of if the AI can control its reward function," }, { "start": 2143.92, "end": 2147.92, "text": "yeah, we get into sort of dangerous territory, potentially." }, { "start": 2149.18, "end": 2153.62, "text": "The other thing, the other aspect, like speaking specifically about consciousness," }, { "start": 2155.2, "end": 2166.18, "text": "I would offer that it's an interesting contingent fact about how brains work, that we seek out pleasure." }, { "start": 2166.86, "end": 2172.52, "text": "I mean, Freud has this pleasure principle. He says that, you know, humans seek out pleasure." }, { "start": 2173.92, "end": 2178.7, "text": "It's interesting to note that that's like roughly true, but not always true." }, { "start": 2179.08, "end": 2186.68, "text": "We don't always seek out pleasure. And so there are sort of various ways to construct the basic human drive." }, { "start": 2187.16, "end": 2194.28, "text": "Freud said we seek out pleasure. Carl Friston might say we seek to minimize short terms of prize and so on." }, { "start": 2195.16, "end": 2202.36, "text": "I think like the danger that I would kind of point out is that, OK," }, { "start": 2203.96, "end": 2211.74, "text": "humans, we seek out sort of the good neighborhoods of phenomenology, the tasty stuff in consciousness." }, { "start": 2212.08, "end": 2218.68, "text": "We like being happy. But this is actually not a law of the universe." }, { "start": 2219.88, "end": 2230.82, "text": "We could lose this. And we should be careful not to make AIs or computers that seek out the bad places specifically." }, { "start": 2242.14, "end": 2245.74, "text": "Sorry, is that a question or is that a question?" }, { "start": 2247, "end": 2251.18, "text": "I think a question. Can you come up and I'm not sure my how." }, { "start": 2251.96, "end": 2255.94, "text": "We'll take this as the one last question. So." }, { "start": 2257.78, "end": 2263.66, "text": "So what do you think is the best way to figure out whether we think this thing have a consciousness?" }, { "start": 2263.92, "end": 2266.94, "text": "Yes or no? For example, animals may have consciousness." }, { "start": 2266.94, "end": 2272.04, "text": "They cannot speak in our language. So we don't know that A.I." }, { "start": 2272.06, "end": 2277.36, "text": "They can speak. They can reply in our language. So we may say yes." }, { "start": 2277.6, "end": 2282.68, "text": "But do you think language is the best way or to fail our meditation or to think?" }, { "start": 2283, "end": 2285.94, "text": "So what's the best way to figure out this problem?" }, { "start": 2287.66, "end": 2290.24, "text": "Yeah. So this is this is kind of the core question." }, { "start": 2291.4, "end": 2292.92, "text": "And so there." }, { "start": 2293.92, "end": 2296.52, "text": "I would bring in a philosopher called Wittgenstein." }, { "start": 2297.78, "end": 2299.24, "text": "Very, very cool guy." }, { "start": 2299.84, "end": 2301.74, "text": "Had super strong dyslexia." }, { "start": 2301.8, "end": 2306.48, "text": "So he could barely string two paragraphs together, but they were very good two paragraphs." }, { "start": 2306.84, "end": 2311.78, "text": "So he he wrote about this in terms of language games." }, { "start": 2312.56, "end": 2323.62, "text": "And he said that often when we're confused, we're using perfectly good words in ways they're just not meant to be used." }, { "start": 2325.44, "end": 2329.22, "text": "So he gave the example like what time is it on the sun?" }, { "start": 2329.36, "end": 2331.14, "text": "Like what time zone is it on the sun?" }, { "start": 2331.66, "end": 2333.78, "text": "And that has no answer." }, { "start": 2333.94, "end": 2337.54, "text": "It's it's like all the individual words are perfectly good." }, { "start": 2337.82, "end": 2342.94, "text": "But when we apply that to to this novel context, we break language." }, { "start": 2343.34, "end": 2346.7, "text": "And he says that most philosophical confusion is like that." }, { "start": 2346.82, "end": 2352.02, "text": "We're trying to use these perfectly good words in ways that they're just not meant to be used." }, { "start": 2353.92, "end": 2367.38, "text": "And I think that many of our attempts to infer consciousness will also fit this pattern that we can say, well, if if something claims it's conscious, surely it has to be conscious." }, { "start": 2368, "end": 2374.82, "text": "But maybe if if it's just like a literally a computer program that says, you know, print F." }, { "start": 2375.64, "end": 2376.7, "text": "Hello, I'm conscious." }, { "start": 2377.1, "end": 2379.46, "text": "Well, I don't think that's conscious." }, { "start": 2380.38, "end": 2382.14, "text": "And so I think that." }, { "start": 2382.58, "end": 2383.5, "text": "I mean, this." }, { "start": 2383.92, "end": 2389.64, "text": "This really, I think, is sort of the core reason why we need a system." }, { "start": 2389.8, "end": 2391.94, "text": "We need a new branch of science." }, { "start": 2392.16, "end": 2399.6, "text": "We need sort of to take the sort of alchemical combination of incompatible intuitions." }, { "start": 2400, "end": 2427.94, "text": "intuitions that we currently have about what is consciousness and how it works and to make it consistent and necessarily so there's a there's a great quote by Eric and I'm gonna mangle his last name apologies Eric Schwitz-Gibble that he says that common sense is incoherent in matters of metaphysics if you're trying to be consistent you're" }, { "start": 2428, "end": 2457.9, "text": "gonna have to violate it somewhere because it is this kind of combination of sort of disjoint language games and when we when we try to make it into a system we're gonna have some surprises and much like I'm sure like chemistry and the history of physics has given us a lot of surprises that people took frankly generations to to get over like special and general relativity and especially quantum mechanics like people are really struggling" }, { "start": 2458, "end": 2487.78, "text": "to say okay like is this real and I you know I suspect that there will be some surprises when we formalize consciousness well thank you Michael for a very mindful and thoughtful talk before our lunch break which is now turned into lightning talk sessions so if you" }, { "start": 2488, "end": 2494.56, "text": "are one of the four lightning talk speakers please come here because" } ], "canonicalUrl": "https://opentheory.net/transcripts/limits-utility-function-new-science-consciousness/" }