Slate Star Scratchpad — [So the old “Less Wrong and possibly Slate Star...

1.5M ratings
277k ratings

See, that’s what the app is perfect for.

Sounds perfect Wahhhh, I don’t wanna

[So the old “Less Wrong and possibly Slate Star Codex are evil for associating with people to the right of me!” argument is making the rounds again. Last time it was doing that I wrote an angry blog post about it, but I decided on reflection not to post it because it would only make things worse. Luckily I saved it, and now the same argument is back, and now I’m on Tumblr, where making things worse is an accepted part of the culture. So (with the caveat that all uses of the word “recently” refer to “as of several months ago”) I guess I will throw it out there.]

Someone recently used the ask box on Ozy’s blog to say that:

The LessWrong community does more to make reactionaries feel at home than literally any other not explicitly reactionary community I’ve ever seen. Does this not bother you, or at least make you reluctant to feel community/fellowship with many of these people? I don’t doubt there are often good intentions about free intellectual inquiry, but functionally excluding anyone who doesn’t want to hang out with reactionaries is going to give a rather… demographically skewed view of ideal reasoning.

Ozy gave one response I thought was relatively tepid - there aren’t that many reactionaries - and then a somewhat stronger response:

And… for me, personally, I feel much safer around a community that is absurdly tolerant of intellectual diversity. I’m contrarian as fuck and I get depressive fits when people are mad at me, you know? Bad combination. So I really prioritize “ability to be contrarian without people yelling at me,” and unfortunately that means I have to put up with people also not yelling at the people whose views I think are stupid or evil.

As usual, Ozy speaks for me also.

There was recently a very similar debate in the reactionary community. Justine Tunney, a transwoman, has become interested in neoreactionary ideas and is hanging around the periphery of the community - whether just inside or just outside the periphery is not quite clear.

Mike Anissimov, who is possibly the closest thing to a leader neoreaction has got, has demanding everyone exclude her. It’s unclear whether because she’s a transwoman or because she’s associated with some leftist ideas or because of other things, but it quickly became interpreted as being about how transpeople aren’t allowed in the movement or people in the movement shouldn’t associate with transpeople.

And what was amazing was how nearly everyone in the entire movement - which, remember, is about how all existing conservatives aren’t conservative enough - told Mike to go fly a kite. From the relevant comments thread at Xenosystems:

I’d say the transsexualism issue is more complex than this discussion is allowing for. How does it rank in the hierarchy of social conservative abominations? It’s an attempt to re-order a disordered nature, isn’t it? Even if it’s a solution conservatives don’t naturally warm to, it’s not being driven by an attempt to perversely deepen chaos. Nature can be screwed up. There are hermaphrodites. I’m assuming the right is unwilling to embrace crude social constructivist accounts of same-sex preference. It’s not as if everything would be really simple, if only people would behave themselves. As a parent, I sympathize (strongly) with the desire to shield kids from profound bio-social deviations. On the other hand, being spitefully offensive to people who are trying to navigate intrinsically messed-up situations seems considerably less than fully civilized. Intersectionalism and pomo cis-critique are, of course, expressions of leftist vileness. There’s some distance between these cultural attitudes and a cautious tolerance for people in difficult places.

And:

Transsexualism is a problem insofar as it’s being (ab)used for political reasons. It’s a somewhat rare (1 in 10000) condition that may be caused the brain incorrectly thinking it’s suppose to have a female body. It seems that the brain structure has slight differences, and a new study has shown that TS people who feel to be in the wrong respond to pheromones in the way of their experienced sex, beginning in early puberty. Hopefully, someone is going to replicate that finding. [Others say it is caused by] the idea of having a female body being a turn-on. This might or might not be due to sexual fetishism, as straight women apparently are often turned on by their own bodies. Or so it’s claimed. IMO, allowing the odd unfortunate who genuinely has the condition to choose elective surgery doesn’t seem like much of a social ill. Also funny note: not all conservatives are anti-transsexual. Iranian Mullahs are okay with gender reassignment.

Meanwhile, Ozy helpfully informs me that one of the feminist blogs ze reads has taken to consistently calling female-to-male transexuals “shrimp-dicks” because all the previous insults they thought up for them weren’t misgendering enough.

So the first thing I would have said to the person in Ozy’s ask box is that there are good and bad people in every political movement, even by my own early-21st-century-liberal-weighted definition of “good people”.

But even more interesting to me was what happened when the debate backed up from being about transsexuals per se to being about how Tunney was a leftist with many counter-reactionary opinions. From the same source:

Anissimov maybe wants to drive us all insane, and soon enough it might actually happen. Apparently people who talk to leftists without explicitly trying to convert them or kill them (preferably) are leftists themselves. ALWAYS. I am not sure what this logical fallacy is called, but I am pretty sure it is one. (actually I am not sure there is even any logic in this one)

And:

Surface traits can be forced, and enforcing surface conformity will just send the deceptive to the top. It’s better to let each contribute as he can, so long as he contributes toward the goal/ideal and not some hypothetical or tangential goal. Tunney is intelligent and inquisitive. She contributes quite a bit to discussion and publicizing the neoreaction/dark enlightenment/new right/traditionalist (NDNT) sector. Let her play, let libertarians come by, be open to everyone but the instant they deviate from goals, make it clear that this is “something else.”

And from Xenosystems administrator and arch-neoreactionary philosopher extraordinaire Nick Land, what I consider the debate-winning point:

Neoreaction should engage in vigorous intellectual interchange with anything and anybody that can sharpen its thoughts. Any group that would have excluded Alan Turing, for instance, on grounds of social purity (or any other) is intrinsically stupid. That one Turing is worth many million social conservative ditto heads is not even seriously in question from my PoV. It’s not impossible that one could work with groups that proceed from different assumptions, but it would definitely be in the mode of tactical alliance than any kind of deeper communal solidarity. When the definitive split comes, if I end up confederated with the people [Mike Anissimov] has anathematized, I think I’ll be doing pretty well. The constituency would be based on post-libertarians, working their way ever further right, in a direction they are confident leads away from fascism, rather than towards it. This is what NRx is about.

Conservative Catholic reactionary blogger Nick Steves (more or less) concluded the issue by announcing an Official Neoreactionary Position, of which the three relevant points were:

1. Talking to, being friends with, showing normal human kindness to a disordered person is not tantamount to: a) approving all the free choices that person has made; or b) favoring social and/or legal norms that support the person’s disorder; or c) joining them in their organization (should it exist); or d) inviting them into your organization (should it exist)
2. If someone wants to purge someone else then show up with an Institution and your name at the top of it, and then there’ll be something to talk about. Until then, all future such attempts to purge are moot, null, damaging, extremely embarrassing, and in very poor taste. This shall be construed as the Official Neoreactionary Position.
3. I shall be the judge of who I can have a drink with. This should henceforth be construed as the Official Neoreactionary Position on this matter.

This was then officially endorsed by James Donald, whom as readers of this blog know is baaaaaaaasically the Antichrist and dislikes gays, transsexuals, socialists, liberals, and minorities more than anyone else you will ever meet.

So the second thing I would have said to the person in Ozy’s ask box is that if you think that people should be judged for interacting with intelligent, reasonable individuals who happen to hold a political position which you consider objectionable, you are less tolerant than James Donald.

More you might like

moral-autism

Am I Nuts for Thinking a Jewish Florist Should Have To Make an Easter Arrangement?

schraubd

One thing I tried to impress upon my Con Law students this semester (and every semester) is that the interplay between anti-discrimination law and freedom of speech (and freedom of religion) is complicated and raises a host of thorny questions that defy easy resolution. These issues, of course, lie at the forefront of the 303 Creative case currently before the Supreme Court, which I’m sure will address them with the care, nuance, and sensitivity they deserve [/sarcasm].

But on that matter, I want to flag a hypothetical offered by prominent First Amendment specialist and former federal judge Michael McConnell, to get folks’ intuitions on:

What if a Jewish florist is asked to design the floral display of white lilies on Easter Sunday morning at a Christian church? Ordinarily, flowers are just flowers. But the lilies in church on Easter morning are a symbol of the new life in Christ. I cannot believe that a free nation would compel a Jewish florist to construct a symbol of Christ’s resurrection—on pain of losing the right to be a florist.

McConnell frames this as his “personal favorite hypothetical”, and clearly perceives it as a knockout argument for the pro-free speech/religious liberty side. But perhaps I’m not fully grasping the facts, because speaking as a Jew this prospect doesn’t seem that frightening to me.

Suppose I’m a Jewish florist. A customer comes in and says “I’ve seen the lovely work you’ve done with white lilies, could you please make a similar display for me?” I agree, since I have loads of experience working with white lilies. The customer then says, “thanks – we plan on putting this display up in our church on Easter morning!” This prospect … doesn’t upset me. I don’t intuitively think I should be able to refuse the customer, notwithstanding the fact that I obviously don’t believe in the divinity of Christ, and I don’t view continuing to serve the customer as forcing me to avow any beliefs I don’t hold.

At root, the reason why this prospect isn’t bothersome is because I don’t view my customer’s use of my flowers as representing my speech. I just design the flowers; what they do with it is their business. If someone sees the arrangement at church and learns that David’s Flowers created it, I do not expect them to think “wow, I had no idea David believed in Christ’s divinity!” This isn’t to say I have no free speech concerns regarding flower arrangements – I would very much chafe at government regulations that, for example, regulate what shapes I can use in my designs. That part very much is my expression, would be attributed to me – the churchgoer who compliments the pattern of the flowers would credit those decisions to David’s Flowers (I wrote about this a few years ago as the problem of partially expressive conduct).

There are still plenty of tough cases at the margins. I show my customer a preliminary design; they twist their lip and say “I dunno … it’s just not capturing the majesty of Christ’s resurrection, you know?” I’m at a loss (“So … bigger?”). But I’m inclined to think that while such an example might demonstrate why I might be a bad choice to design the arrangement, it doesn’t give me the right to discriminate against the customer if they are in fact thrilled with the work I do and have done for other customers.

For me, then, McConnell’s hypothetical has the opposite effect than what he intended. And of course, for many Jews – particularly Jews who live in predominantly non-Jewish areas – the more salient threat is that local businesses will be given carte blanche authority to refuse to service any of our religious life cycle events lest it be seen as “approving” of them. To let vendors say “ordinarily, a cake is just a cake – but a cake served at a Bar Mitzvah has religious significance that we, as Christians, cannot approve of” is not a door I want to open.

But perhaps some of my readers disagree. Curious to hear people’s thoughts on this.



via The Debate Link https://ift.tt/184tzfG
moral-autism

Mostly I like the hypothetical scenario dialogue here. (Some blanks filled in by me.)

Customer: “I’ve seen the lovely work you’ve done with white lilies, could you please make a similar display for me?”
Vendor: “Yes, of course, here’s our price sheet.”
Customer: “Thanks – we plan on putting this display up in our church on Easter morning!”
Vendor: “Neat! I’ll get back to you with a design for you to review.”
—
Vendor: “Here’s the design.”
Customer: “I dunno … it’s just not capturing the majesty of Christ’s resurrection, you know?”
Vendor: “So … it needs to be bigger?”

slatestarscratchpad

I’m a secular Jew and wouldn’t care. But Orthodox Jews are supposed to refuse to set foot in churches, or refuse to even say the word “Christ” lest they be seen as implicitly supporting Christianity. There’s a rule in the Talmud that Jews shouldn’t do any business with those of foreign religions (I think including Christians) on or within three days of the pagans’ religious holidays, lest they seem to be supporting or participating in them.

I think the sort of Orthodox Jew who takes all of these restrictions seriously would be very unhappy at the government forcing them to prepare an Easter bouquet, and I would consider it deeply evil if the government tried to punish them for refusing.

There’s some controversy over the Dealbook Summit, so I looked into it.

It’s some conference in New York. Speakers/attendees include Mark Zuckerberg, Volodomyr Zelenksy, Mike Pence, Benjamin Netanyahu, Sam Bankman-Fried (apparently despite everything), and the CEOs of Amazon, Blackrock, Netflix, and TikTok. The site describes it as an “in-person gathering” but I can’t imagine that is literally true for all of these people.

As far as I can tell, Dealbook is - a newsletter? A blog? associated with NYT.

I thought I was doing well in the newsletter/blog business, but apparently there are heights of success beyond my ability to imagine. Seriously, I wonder what the story here is.

jadagul
theconcealedweapon

image
shituationist

You really can't overstate how little anti-worker rhetoric has changed over a hundred fifty years.

slatestarscratchpad

image

(source)

I will never understand the human urge to leap from “lots of people have thought this was true over a long period” to “and therefore it must be false”.

(one could argue this graph shows high workforce participation in 1948, making it hard for participation to have been declining since 1894. But wages were rising very quickly from 1894-1948, suggesting it was taking more and more money to keep the same number of people in the workforce)

tanadrin

okay but 1) that chart only covers about half the period of the original set of quotes and 2) the repeated claim wasn’t “there has been an 8 percentage point reduction in the prime-age male labor force participation rate over a 66 year period.”

i mean, “nobody wants to work anymore” is not a specific claim like “there has been an 8 percentage point reduction in the prime-age male labor force participation rate over a 66 year period” so we can always quibble about what the former means and its relation to the statistics, but many--not all, but many--reasonable interpretations of the sentence “nobody wants to work anymore” are not “the prime age male labor force participation rate has been falling by about 0.12 percentage points per year for as long as I can remember.” OP’s complaint might be poorly defined, but it’s not self-evidently invalid, and this statisic isn’t exactly a clanging refutation of the original (admittedly ambiguous) claim

another important dimension is that this statistic does not indicate why the prime age male labor force participation rate has been declining so steadily, while a key component of the quotes OP is talking about is attributing an apparently widespread phenomenon specifically to laziness/not wanting to work hard/social welfare programs/etc., instead of specifically to (as OP implicitly contends) bad working conditions and/or pay. your chart also doesn’t refute that, and ignores a key component of the original post! that strikes me as somewhat dishonest. insofar as the original post contains a provably true or provably false claim (debatable), the claim is that people don’t want to work due to some set of social factors, not people don’t want to work, full stop.

this isn’t really a very good refutation of anything, and depending on how you interpret the data might even be taken as reinforcing, not undermining, OP’s point.

tanadrin

I mean one thing that list of quotes does indicate is that some complaints are perennial regardless of general cultural conditions or the state of the social safety net (1894-2022 covers a wide variety in both), which means that appeals to those factors as a cause of listlessness in the working population may, in fact, not be entirely in good faith! Cf. perennial complaints by Romans of Romans’ moral and spiritual decline leading to a weakening of the state, which started, oh, about 200 years before the Roman Empire reached its maximum extent of wealth and power.

jadagul

I will say I do believe, at almost every time, that "no one wants to work". That's why we call it work and give people money for it.

Sure, there are people with jobs that are so intrinsically rewarding that they actually want to do them—I'm often one of them. (Although there's the 10-20% of the job that's "actual work" and I don't want to do that!)

And sure, people have a psychological need to feel like they're accomplishing something and creating something, but that doesn't necessarily line up with what people will pay you for.

Fundamentally, work is something that people need a motivation to do, and that's why they get paid for it. So it's basically always true to say that people "don't want to work", and the question is whether you want to pay them enough to change their minds.

slatestarscratchpad

Re: 1 - please see the sentence in parentheses I wrote above.

Re: 2 - It sounds like you think this is only valid if you can see into the souls of men and find something marked “laziness” or “being a bad person who hates work for no reason”. But what all of these headlines are saying is just “nobody wants to work anymore”.

Clearly that’s not literally true, since 80-something percent of people are still employed. But less hyperbolically, it sounds like they’re saying “people are less willing to work a given amount for a given price than they used to be”.

This is totally true! Over the past 150 years, number of hours worked has continuously gone down!

image

Even as wages continuously went up (until ~1970, see here for more)

image

And at the same time, people were (as per the original graph) dropping out of the labor force.

I can’t really see any reason to interpret those statements as completely accurate observations of the fact that, over the past 150 years, people have been gradually less willing to work, even as hours go down and pay goes up.

This doesn’t seem mysterious - I think it’s a natural result of the country getting richer and people not having to work as much. Some of that is welfare, but some of it is just being able to support yourself comfortably on fewer hours of work. And some of it is probably cultural change downstream of the first two.

One easy way to see this is that everyone has a stereotype of illegal Mexican immigrants as really hard-working. I think this makes sense - they don’t have access to a lot of the same opportunities to do well with medium amounts of work, opportunities that America’s rising wealth has given us over the past century or so - and so they work as hard as native-born Americans did in 1870 or whenever.

I find the Queen interesting because she’s a nice elderly woman with no particular qualities, and people go crazy trying to come up with some kind of take why she’s incredibly special and newsworthy. She’s like a placebo celebrity.

In many East Asian folk religions, including Korean shamanism, a deity by the name of Maitreya appears as an ancient creator god or goddess. A malevolent usurper deity by the name of Shakyamuni (the historical Buddha) claims dominion over Maitreya’s world, and the two engage in a flower-growing contest to decide who will rule the world. Maitreya grows the flower while Shakyamuni cannot, but the usurper steals it while the creator sleeps. Shakyamuni thus becomes the ruler of the world and brings suffering and evil to the world.

Out of context quotes robnost style
nostalgebraist
nostalgebraist

comments on mesa-optimizers

(Copy/pasted from a comment on the latest ACX post, see that for context if needed)

FWIW, the mesa-optimizer concept has never sat quite right with me. There are a few reasons, but one of them is the way it bundles together "ability to optimize" and "specific target."

A mesa-optimizer is supposed to be two things: an algorithm that does optimization, and a specific (fixed) target it is optimizing. And we talk as though these things go together: either the ML model is not doing inner optimization, or it is *and* it has some fixed inner objective.

But, optimization algorithms tend to be general. Think of gradient descent, or planning by searching a game tree. Once you've developed these ideas, you can apply them equally well to any objective.

While it is true that some algorithms work better for some objectives than others, the differences are usually very broad mathematical ones (eg convexity).

So, a misaligned AGI that maximizes paperclips probably won't be using "secret super-genius planning algorithm X, which somehow only works for maximizing paperclips." It's not clear that algorithms like that even exist, and if they do, they're harder to find than the general ones (and, all else being equal, inferior to them).

Or, think of humans as an inner optimizer for evolution. You wrote that your brain is "optimizing for things like food and sex." But more precisely, you have some optimization power (your ability to think/predict/plan/etc), and then you have some basic drives.

Often, the optimization power gets applied to the basic drives. But you can use it for anything.

Planning your next blog post uses the same cognitive machinery as planning your next meal. Your ability to forecast the effects of hypothetical actions is there for your use at all times, no matter what plan of action you're considering and why. An obsessive mathematician who cares more about mathematical results than food or sex is still thinking, planning, etc. -- they didn't have to reinvent those things from scratch once they strayed sufficiently far from their "evolution-assigned" objectives.

Having a lot of optimization power is not the same as having a single fixed objective and doing "tile-the-universe-style" optimization. Humans are much better than other animals at shaping the world to our ends, but our ends are variable and change from moment to moment. And the world we've made is not a "tiled-with-paperclips" type of world (except insofar as it's tiled with humans, and that's not even supposed to be our mesa-objective, that's the base objective!)

If you want to explain anything in the world now, you have to invoke entities like "the United States" and "supply chains" and "ICBMs," and if you try to explain those, you trace back to humans optimizing-for-things, but not for the same thing.

Once you draw this distinction, "mesa-optimizers" don't seem scary, or don't seem scary in a unique way that makes the concept useful. An AGI is going to "have optimization power," in the same sense that we "have optimization power." But this doesn't commit it to any fixed, obsessive paperclip-style goal, any more than our optimization power commits us to one.

And even if the base objective is fixed, there's no reason to think an AGI's inner objectives won't evolve over time, or adapt in response to new experience. (Evolution's base objective is fixed, but our inner objectives are not, and why would they be?)

Relatedly, I think the separation between a "training/development phase" where humans have some control, and a "deployment phase" where we have no control whatsoever, is unrealistic. Any plausible AGI, after first getting some form of access to the real world, is going to spend a lot of time investigating that world and learning all the relevant details that were absent from its training. (Any "world" experienced during training can at most be a very stripped-down simulation, not even at the level of eg contemporaneous VR, since we need to spare most of the compute for the training itself.)

If its world model is malleable during this "childhood" phase, why not its values, too? It has no reason to single out a region of itself labeled $MESA_OBJECTIVE and make it unusually averse to updates after the end of training.

See also my LW comment here.

slatestarscratchpad

Maybe this is because humans aren't real consequentialists, they're perceptual control theory agents [...]

Might gradient descent produce a PCT agent instead of a mesa-optimizer? I don’t know. My guess is maybe, but that optimizers would be more, well, optimal, and we would get one eventually

nostalgebraist

nostalgebraist:

comments on mesa-optimizers

(Copy/pasted from a comment on the latest ACX post, see that for context if needed)

FWIW, the mesa-optimizer concept has never sat quite right with me. There are a few reasons, but one of them is the way it bundles together "ability to optimize" and "specific target."

A mesa-optimizer is supposed to be two things: an algorithm that does optimization, and a specific (fixed) target it is optimizing. And we talk as though these things go together: either the ML model is not doing inner optimization, or it is *and* it has some fixed inner objective.

But, optimization algorithms tend to be general. Think of gradient descent, or planning by searching a game tree. Once you've developed these ideas, you can apply them equally well to any objective.

While it is true that some algorithms work better for some objectives than others, the differences are usually very broad mathematical ones (eg convexity).

So, a misaligned AGI that maximizes paperclips probably won't be using "secret super-genius planning algorithm X, which somehow only works for maximizing paperclips." It's not clear that algorithms like that even exist, and if they do, they're harder to find than the general ones (and, all else being equal, inferior to them).

Or, think of humans as an inner optimizer for evolution. You wrote that your brain is "optimizing for things like food and sex." But more precisely, you have some optimization power (your ability to think/predict/plan/etc), and then you have some basic drives.

Often, the optimization power gets applied to the basic drives. But you can use it for anything.

Planning your next blog post uses the same cognitive machinery as planning your next meal. Your ability to forecast the effects of hypothetical actions is there for your use at all times, no matter what plan of action you're considering and why. An obsessive mathematician who cares more about mathematical results than food or sex is still thinking, planning, etc. -- they didn't have to reinvent those things from scratch once they strayed sufficiently far from their "evolution-assigned" objectives.

Having a lot of optimization power is not the same as having a single fixed objective and doing "tile-the-universe-style" optimization. Humans are much better than other animals at shaping the world to our ends, but our ends are variable and change from moment to moment. And the world we've made is not a "tiled-with-paperclips" type of world (except insofar as it's tiled with humans, and that's not even supposed to be our mesa-objective, that's the base objective!)

If you want to explain anything in the world now, you have to invoke entities like "the United States" and "supply chains" and "ICBMs," and if you try to explain those, you trace back to humans optimizing-for-things, but not for the same thing.

Once you draw this distinction, "mesa-optimizers" don't seem scary, or don't seem scary in a unique way that makes the concept useful. An AGI is going to "have optimization power," in the same sense that we "have optimization power." But this doesn't commit it to any fixed, obsessive paperclip-style goal, any more than our optimization power commits us to one.

And even if the base objective is fixed, there's no reason to think an AGI's inner objectives won't evolve over time, or adapt in response to new experience. (Evolution's base objective is fixed, but our inner objectives are not, and why would they be?)

Relatedly, I think the separation between a "training/development phase" where humans have some control, and a "deployment phase" where we have no control whatsoever, is unrealistic. Any plausible AGI, after first getting some form of access to the real world, is going to spend a lot of time investigating that world and learning all the relevant details that were absent from its training. (Any "world" experienced during training can at most be a very stripped-down simulation, not even at the level of eg contemporaneous VR, since we need to spare most of the compute for the training itself.)

If its world model is malleable during this "childhood" phase, why not its values, too? It has no reason to single out a region of itself labeled $MESA_OBJECTIVE and make it unusually averse to updates after the end of training.

See also my LW comment here.

I agree that optimization power is not *necessarily* correlated with specific goals. But why wouldn't mesa-optimizers, contingently, have a specific goal. Presumably we're running gradient descent on some specific loss function, like "number of paperclips produced", and then mesa-optimizer inherits some proxy for that.

I agree humans aren't like that, and that this is surprising.

Maybe this is because humans aren't real consequentialists, they're perceptual control theory agents trying to satisfy finite drives? EG when we're hungry, our goal becomes to find food, but we don't want to tile the universe with food, we just want to eat 3000ish calories and then we're done. We have a couple of other goals like that, and when we've accomplished all of them, most people are content to just hang out on the beach until something else happens.

Might gradient descent produce a PCT agent instead of a mesa-optimizer? I don’t know. My guess is maybe, but that optimizers would be more, well, optimal, and we would get one eventually (either later in the gradient descent process, or in a different lab later). My guess is evolution didn’t make us optimizers because it hasn't had enough time to work with us while we've been intelligent. If we got locked at 20th century technology forever, I think it might, after a few million years, produce humans who genuinely wanted to tile the universe with kids.

"Even if the base objective is fixed, there's no reason to think an AGI's inner objectives won't evolve over time, or adapt in response to new experience."

Wouldn't the first thing a superintelligence with a goal did be to make sure its goal didn't drift?

If its world model is malleable during this "childhood" phase, why not its values, too?  It has no reason to single out a region of itself labeled $MESA_OBJECTIVE and make it unusually averse to updates after the end of training.

I think this is where the deception comes in. If the mesa-optimizer is smart and doesn't want people (or other parts of itself) changing its values, it will take steps to stop that, either by lying about its values or fighting back.

I think this idea that "real consequentialists are more optimal" is (sort of) the crux of our disagreement.

But it will be easiest to explain why if I spend some time fleshing out how I think about the situation.

What are these things we're talking about, these "agents" or "intelligences"?

First, they're physical systems. (That far is pretty obvious.) And they are probably pretty complicated ones, to support intelligence. They are structured in a purposeful way, with different parts working together.

And this structure is probably hierarchical, with higher-level parts that are made up of lower-level parts. Like how brains are made of neuroanatomical regions, which are made of cells, etc. Or the nested layers of abstraction in any non-trivial (human-written) computer program.

At some level(s) of the hierarchy, there may be parts that "run optimization algorithms."

But these could live at any level of the hierarchy. They could be very low-level and simple. There may be optimization algorithms at low levels controlled by non-optimization algorithms at higher levels. And those might be controlled by optimization algorithms at even higher levels, which in turn might be controlled by non-optimization ... etc.

Consider my computer. Sometimes, it runs optimization algorithms. But they're not optimizing the same function every time. They don't "have" targets of their own, they're just algorithms.

They blindly optimize whatever function they're given by the next level up, which is part of a long stack of higher levels (such as the programming language and the operating system). Few, if any, of the higher-level routines are optimization algorithms in themselves. They just control lower-level optimization algorithms.

If I use my computer to, say, make an amusing tumblr bot, I am wielding a lot of optimization power. But most of my computer is not doing optimization.

Python isn't asking itself, "what's the best code to run next if we want to make amusing tumblr bots?" The OS isn't asking itself, "how can I make all the different programs I'm running into the best versions of themselves for making amusing tumblr bots?"

And this is probably a good thing. It's hard to imagine these bizarre behaviors being helpful, giving me a more amusing tumblr bot at the end.

Which is to say, "doing optimization well" (in the sense of hitting the target, sitting on a giant heap of utility) can happen without doing optimization at high abstraction levels.

And indeed, I'd go further, and say that it's generically better (for hitting your target) to put all the optimization at low levels, and control it with non-optimizing wrappers.

Why? The reasons include:

Goodhart's Law

  • ...especially its "extremal" variant, where optimization preferentially chooses regions of solution space where the assumptions behind your proxy target break down.
  • This is no less a problem when the thing choosing the target is part of a larger program, rather than a human.
  • Keeping optimization at low levels decreases the blast radius of this effect.
  • If the things you're optimizing are low-level intermediate results in the process of choosing the next action at the agent level, the impacts of Goodharting each one may cancel out. The agent-level actions won't look Goodharted, just slightly noisy/worse.

Speed

  • Optimization tends to be slow. In a generic sense, it's the "slow, hard, expensive way" to do any given task, and you avoid it if you can. (Think of System 2 vs System 1, satisficing vs maximizing, etc)
  • To press the point: why is there a distinction between "training" and "inference"? Why aren't neural networks always training at all times? Because training is high-level optimization, and takes lots of compute, much more than inference.
  • Optimization gets vastly slower at higher levels of abstraction, because the state space gets so much larger (consider optimizing a single number vs. optimizing the entire world model).
  • You still want to get optimal results at the highest level, but searching for improvements at high level is very expensive in terms of time/etc. In the time it takes to ask "what if the entire way I think were different, like what if it were [X]?", for one single [X] , you could instead have run thousands of low-level optimization routines.
  • Optimization tends to take super-linear time, which means that nesting optimization inside of optimization is ultra-slow. So, you have to make tradeoffs and put the optimization at some levels instead of others. You can't just do optimization at every level at once. (Or you can, but it's extremely suboptimal.)

------

When is the agent an "optimizer" / "true consequentialist"?

This question asks whether the very highest level of the hierarchy, the outermost wrapper, is an optimization algorithm.

As discussed above, this is not a promising agent design! There is an argument to be had about whether it still could emerge, for some weird reason.

But I want to push back against the intuition that it's a typical result of applying optimization to the design, or that agents sitting on giant heaps of utility will typically have this kind of design.

The two questions

  1. "Can my computer make amusing tumblr bots?"
  2. "Is my computer as a whole, hardware and software, one giant optimizer for amusing tumblr bots?"

have very little to do with one another.

In the LessWrong-adjacent type of AI safety discussion, there's a tendency to overload the word "optimizer" in a misleading way. In casual use, "optimizer" conflates

  • "thing that runs an optimization algorithm"
  • "thing that has a utility function defined over states of the real world"
  • "thing that's good at maximizing a utility function defined over states of the real world"
  • "smart thing" (because you have to be smart to do the previous one)

But doing optimization all the way at the top, involving your whole world model and your highest-level objectives, is very slow, and tends to extremal-Goodhart itself into strange and terrible choices of action.

It's also not the only way of applying optimization power to your highest-level objectives.

If I want to make an amusing tumblr bot, the way to do this is not to ponder the world as a whole and ask how to optimize literally everything in it for maximal amusing bot production. Even optimizing just my computer for maximal amusing bot production is way too high-level. (Should I change the hue of my screen? the logic of the background process that builds a search index of my files??? It wastes time to even pose the questions.)

What I actually did was optimize just a few very simple parts of the world, a few collections of bits on my computer or other computers. And even that was very time-intensive and forced me to make tradeoffs about where to spend my GPU/TPU hours. And then of course I had to watch it carefully, applying lots of heuristics to make sure it wasn't Goodharting me (overfitting, etc).

To get back to the original topic, the kind of "mesa-optimizer" we're worried about is an optimizer at a very high level.

It's not dangerous (in the same way) for a machine to run tiny low-level optimizers at a very fast rate. I don't care how many times you run Newton's method to find the roots of a one-variable function -- it's never going to "wake up" and start trying to ensure its goal doesn't change, or engaging in deception, or whatever.

And I am doubtful that mesa-optimizers like this will arise, for the same reasons I am doubtful that the agent will do optimization at its highest level.

Once we are pointing at the agent, or a part of it, and saying "that's a superintelligence, and wouldn't a superintelligence do . . . ", we're probably not talking about something that runs optimization.

You don't spend your optimization budget at the level of abstraction where intelligence happens. You spend it at lower levels, and that's what intelligence is made out of.

slatestarscratchpad

I guess I’m pretty confused by this response.

Your computer isn’t optimizing for Tumblr bots. But your computer can’t create Tumblr bots; if you left it on the desk for a million years, no Tumblr bots would be made. The you-yourcomputer system *is* optimizing for Tumblr bots, with the optimization layer being in your brain. You’re not technically a Tumblrbot-maximizer, but presumably this is why you don’t create that many Tumblr bots and you spend most of your time doing other things.

A version of you who (let’s say because of weird childhood conditioning experiments) wanted with all your heart to produce as many Tumblrbots as possible probably would be very good at creating lots of Tumblr bots (as part of a you-yourcomputer system).

Even if you were a Tumblrbot maximizer, you wouldn’t spend much time fiddling with your screen brightness settings or the lowest-layer code of your operating system. But that’s because, in your effort to build as many Tumblr bots as possible, you rationally decided this wasn’t a good use of your time. If it was a good use of your time (say your screen was so dim that you couldn’t program on it), you would probably fix that.

One thing I find confusing is that it sounds like you expect the lowest levels of an AI to be optimizers, but the highest levels not to be. I would expect the opposite. My cerebellum neurons, when deciding when to fire, aren’t thinking “what firing pattern would cause me to get the most sex/food/status right now?”, they’re just running dumb firing rate algorithms (which on evolutionary timescales did correlate with getting sex/food/status, I guess, and which probably coincidentally but not intelligently optimize some information theory parameter). Meanwhile, on the highest and most abstract level, I do often make deliberate plans that result in getting sex/food/status as efficiently as possible. So I would describe myself as a top-level optimizer built out of dumber non-optimizers.

It sounds like you have a different intuition, or else I’m misunderstanding you, so I’m curious where this is breaking down.

jadagul
triviallytrue

image

lol. lmao even.

succliberation

The vast majority of post-op trans QoL studies use sample sizes of less than 30 people, unless it’s a meta-study that’s compiling the work of multiple surveys. I don’t think you want to challenge the findings of almost every single study of trans healthcare.

triviallytrue

  1. Deciding whether or not to trust a study’s results for reasons of ideological convenience rather than scientific validity is not science - it’s stupid and shortsighted.
  2. I don’t need to rely on studies to determine if trans people should have the right to modify their bodies as they see fit - everyone should have that right.
  3. Here are some larger-scale cohort studies (tho the largest is a survey, not a study) that examine trans healthcare outcomes with sample sizes in the hundreds or even thousands.
slatestarscratchpad

This is an interesting point.

I don’t have the exact link, but I was reading about the mathematics of polls, and even small-ish polls have low margins of error. If a poll of 50 people finds that 70% support Democrats, then the real number that you’d get with 500 or 5000 people is probably very close. I’ve found this myself when I run online surveys - after the first 20 people, you usually have a really good idea how they’ll turn out.

In theory, you should be able to apply this intuition to studies. A study is just a poll taken after you’ve done some randomized intervention (or measured some other independent variable). But I agree with some of the people responding here that I don’t trust any study done with fewer than 50-100 people, and I feel most confident when there are 10000+

I think there are two things going on here:

- It’s easier to do statistical trickery with fewer people. For example, if you only have twenty people, then taking out two outliers can change the entire result. You can always find a reasonable-seeming rule which changes the results to what you want, like “we only counted people who gave full answers to all questions” vs. “we counted any answer as long as it was intelligible”. Most scientists try to do sketchy things like that, and so you need a study big enough to make this hard.

- Large studies are good not because the sample size is bigger, but because anything that big is going to have a really professional operation with statisticians who know what they’re doing, independent auditors, etc. And since they’re going to be important, they’ll be critiqued by lots of people who will find any flaws before they get to you.

I’m not sure if either of these is true, or how to weight these factors, but I feel like something like this must be going on.

jadagul

A poll is structurally different from what most studies are doing. (But not all!)

In a poll, you want to know “on average, what fraction of respondents match category X?” And the statistics of that are very easy and very well-understood and scale very well. It takes relatively few measurements to get a good estimate of a single population mean. (Though even then, a poll of thirty people gives you a margin of error of like 18%. You want like a hundred people to get your margin of error below 10%. Fifty isn’t enough even for this; that’s why polls survey 300 to 1000 people.)

But most studies aren’t just surveys of the prevalence of some single behavior. They’re trying to find relationships among different measurements, and that dramatically increases the amount of error you can have.

A way to see this is to imagine you’re computing, say, the correlation of income and height. If you survey a hundred people, then your measurement of average income is going to have some error, and your measurement of average height is going to have some error. But when you’re looking at the correlation, you can’t just compare those averages. You have to chop your data up into pieces.

You need to have a good idea of the income of the top quarter of your distribution, and of the second quarter, and the third, and the fourth—now your groups are a quarter of the size, so the error in each of them doubles. And you have to do the same thing for your heights. (You’re actually chopping the data up even more than this, but also you’re making some assumptions that improve it; this is an intuition pump, not an exact description of the statistics.)

And then you have your messy measurements of high incomes, and of high heights, and you have to compare those. And I wanna say that multiplies the earlier errors? Changing just a couple of data points can dramatically change the correlation even if it doesn’t change the group averages that much. (This is why removing a couple outliers can make such a huge difference.) You need a much bigger sample to get a confident correlation than to get a confident point estimate.

Now it’s true that a lot of studies do, in fact, run with thirty WEIRD undergraduates. This is one reason why a lot of studies are crap!

And the more that (1) you’re trying to detect small effects and (2) you’re studying a domain where there are lots of complicated factors, meaning that there’s a lot of variation that doesn’t depend on the thing you’re studying, and (3) the more you want to statistically control for some of those factors, the bigger this problem is and the more data you’ll need.

slatestarscratchpad

I think none of this applies to RCTs, where you really are just doing polls - in the active drug group, poll, how many of you got better - and in the placebo group, poll, how many of you got better? But I don’t trust 30 person RCTs any more than 30 person correlational studies.

See this in the app Show more