Friendly Superintelligence

My assumptions

• Need to make friendliness work in general, not just for particular AI designs– we do not know which will succeed

• Hard takeoff unlikely – AI will develop over time in interaction with

society• The context systems are developed in must be taken into

account, we cannot use simple a priori arguments

Friendly AI is in the end a practical problem

• AI will be created for economic reasons, and will be involved in economic transactions with humans from the start.

• Whether AI, IA or something else will be developed will be determined only to a minor extent by deliberate global choices and more by what technologies provide payoffs during their development

Friendliness as a game

• Friendly AI as a game: we want an infinite game for humans

• It is not a game for a single player, but from the start consisting of many different players with slightly different goals.

Do we aim for no risk or acceptable risk?

• As risks become smaller the cost of removing them increases with no limit

• The hard take-off assumption assumes that there is going to be one gamble with a single large risk, while the soft take-off implies many interactions with medium risks.

Suggested approaches to friendly AI

• Internal constraints (Asimov’s laws)

• Built in values or goals (“Love humans”)

• Learned values (Brin, Lungfish)

• External (law, economics)

Problems with the approaches

• Asimov laws allow accidental unfriendly behaviour – the full consequences of a complex formal

system are unknowable, and being in contact with the messy real world makes things worse.

• Internal constraints and values are design solutions, but there are many designers and some might be malevolent, misguided or make mistakes.

• Designs compete with each other - a risky architecture may show greater economic potential

• If values are learned, then they can be mis-learned.

• External approaches can seldom be proven to work due to their complexity.

Law of comparative advantages

• Trade is mutually profitable even when one part is more productive than the other in every commodity that is being exchanged – specialisation enables the more productive agent to

produce more of the commodity most profitable to it.

• AI and humans can profit from specialisation, even when their capabilities are vastly different.

External Approaches

• Seek to reward friendliness and punish unfriendliness

• Relevant for the soft takeoff scenarios– AIs that have “grown up” within a human culture

are more likely to encompass its ethics and values, and have tight economical connections

• Defection is profitable only as long as there are no interactions that can make it unprofitable

A Combination Approach

– Guidelines for AI development

• will be useful for selling AI in any case

– Good rearing?

– Make sure we set up a legal and economical framework where friendly AIs prosper and unfriendly are inhibited

– This will not be a guarantee of friendliness, any more than current systems of upbringing, education and law guarantee it.