Ethical Decision Making in Artiﬁcial Intelligence · 2019. 3. 4. · rules • Virtue Ethics: ......

Ethical Decision Making in Artificial Intelligence

Pradeep RavikumarMachine Learning DepartmentSchool of Computer ScienceCarnegie Mellon University

AI making societally important decisions

• Artificial Intelligence systems are being used to make societally important decisions

• Bail decisions

• Healthcare

• Autonomous cars

AI and ethics• Can AI systems behave ethically?

• Typical ML pipeline:

• How do we incorporate “ethical” thinking?

DataMachine Learning

Model

“Fit” data well

Ethics / Moral Philosophy

• What is right and what is wrong?

• How to make decisions that are right?

• Questions studied by philosophers for 1000s of years with no consensus formal framework

Three Main Ethical Frameworks

• Deontological: take action according to a specified set of rules

• Virtue Ethics: multiple “values” or “virtues”; take action that follows these values or virtues

• Consequentialism: take an action that has the most desirable future consequences

• Utilitarianism: assign a utility to world states, and take action that leads to highest utility

Deontological Ethics• Rules based (e.g. Ten Commandments)

• Requires a priori specification of ethical rules

• Given a set of rules, or constraints, we can ensure that AI actions follow these rules

• Caveat: rules not always available, and when available, too broad to be applicable to specific situation

• e.g. just the ten commandments not helpful for self-driving car ethics

Virtue Ethics• Take actions based on values/virtues … Aristotle (and

others)

• Similar caveats to deontological ethics

• values not always available, and when available (e.g. be honest) not always applicable to specific situation

• differing values could conflict (e.g. equality, and freedom)

• Ongoing work: virtue ethics driven decision making

Utilitarian Ethics• Decision theoretic foundations of machine learning based

largely on utilitarianism


Model

“Fit” data well: involves a loss/utility function

`model(✓, D) : loss i.e. negative utility associated

with model parameter ✓, and data D<latexit sha1_base64="P92petFzogcst73UHYX0+8RIEVM=">AAACe3icbVFdaxNBFJ1dP1rjV6qPglxMrFHCslsVi08F++BjBdMWsiHMzt4kQ2dntjN3q2HZP+FP61v/iS+Ck00Ebb0wcDjnnrl3zmSlko7i+CoIb92+c3dr+17n/oOHjx53d54cO1NZgSNhlLGnGXeopMYRSVJ4WlrkRabwJDv7tNJPLtA6afRXWpY4Kfhcy5kUnDw17f5IUalpnRJ+p7owOaqmGaS0QOLDw9ew+xHWEijjHMgII9A49+YLhIqkkrQE7pwRkhPm0KRpZxfS84rnf4zwTdIC2quh5JYXSGihv57RHwLXOeScOPQP+61/2u3FUdwW3ATJBvTYpo6m3cs0N6IqUJNQfplxEpc0qbklKRQ2nbRyWHJxxuc49lD7FdykbrNr4KVncpgZ648maNm/HTUvnFsWme8sOC3cdW1F/k8bVzTbn9RSlxWhFutBs0oBGVh9BOTSoiC19IALK/2uIBY+H+HjcasQkutPvgmO96LkbRR/edc72N/Esc2esRdswBL2gR2wz+yIjZhgP4PnwatgEPwKe+GbcLhuDYON5yn7p8L3vwGpEr6N</latexit>

Utilitarian Ethics• Decision theoretic foundations of machine learning based

largely on utilitarianism


Model Decisions

“Fit” data well “optimal action” involves a loss function

`action(✓, a) : loss i.e. negative utility associated

with model parameter ✓, and action a<latexit sha1_base64="wQrbtYh0EczaF/LSZbD5H+4v250=">AAACfnicbVFNj9MwEHXC11K+Chy5DLRUCywhASQqTitx4bhIdHelpqqmzrS11nGCPVmoovwM/hg3fgsXnKZIsMtIlp7em+cZPy9KrRzH8c8gvHL12vUbezd7t27fuXuvf//BsSsqK2kiC13Y0wU60srQhBVrOi0tYb7QdLI4+9DqJ+dknSrMZ96UNMtxZdRSSWRPzfvfU9J6XqdM37hG2ZJNs5/ymhgP8BmM3kOngS6cAxVRBIZW3n1OULHSijeAzhVSIVMGTZr2RpB+qTD7Y4SviteQFxlpKNFiTkwWht2M4QGgyaCbDEMcbm+Y9wdxFG8LLoNkBwZiV0fz/o80K2SVk2Gp/TrTJC55VqNlJTU1vbRyVKI8wxVNPTR+CTert/E18NQzGSwL649h2LJ/O2rMndvkC9+ZI6/dRa0l/6dNK16OZ7UyZcVkZDdoWWngAtq/gExZkqw3HqC0yu8Kcu0Tkj4g14aQXHzyZXD8OkreRPGnt4PD8S6OPfFIPBH7IhHvxKH4KI7EREjxK3gcPA9ehCIchS/DV11rGOw8D8U/FY5/AwNXvzc=</latexit>`model(✓, D) : loss i.e. negative utility associated

with model parameter ✓, and data D<latexit sha1_base64="P92petFzogcst73UHYX0+8RIEVM=">AAACe3icbVFdaxNBFJ1dP1rjV6qPglxMrFHCslsVi08F++BjBdMWsiHMzt4kQ2dntjN3q2HZP+FP61v/iS+Ck00Ebb0wcDjnnrl3zmSlko7i+CoIb92+c3dr+17n/oOHjx53d54cO1NZgSNhlLGnGXeopMYRSVJ4WlrkRabwJDv7tNJPLtA6afRXWpY4Kfhcy5kUnDw17f5IUalpnRJ+p7owOaqmGaS0QOLDw9ew+xHWEijjHMgII9A49+YLhIqkkrQE7pwRkhPm0KRpZxfS84rnf4zwTdIC2quh5JYXSGihv57RHwLXOeScOPQP+61/2u3FUdwW3ATJBvTYpo6m3cs0N6IqUJNQfplxEpc0qbklKRQ2nbRyWHJxxuc49lD7FdykbrNr4KVncpgZ648maNm/HTUvnFsWme8sOC3cdW1F/k8bVzTbn9RSlxWhFutBs0oBGVh9BOTSoiC19IALK/2uIBY+H+HjcasQkutPvgmO96LkbRR/edc72N/Esc2esRdswBL2gR2wz+yIjZhgP4PnwatgEPwKe+GbcLhuDYON5yn7p8L3vwGpEr6N</latexit>

Example: FinanceData

Machine Learning Model Decisions

“Fit” data well Minimize loss function

Data: past stock prices, Model: for predicting future stock price movements

Loss function: error in predicting future stock price movements

Actions: buy/sell x amount of y stock

Loss function: risk-adjusted return of action given stock market movements

Loss functions• But where do we get these loss functions?

• Typically specified apriori, via domain knowledge

• But what would ethical loss functions look like?

• 1000s years of moral philosophy provide a qualitative rather than quantitative picture of ethical loss functions

• e.g. if airline has to decide who to not board due to overbooking, how do they decide if pregnant woman with two kids is not to be bumped over say a college student?

• We address this in two ways:

• we learn ethical loss functions from data

• we allow for the fact that there need not be a consensus single ethical loss function, and hence learn multiple ethical loss functions and aggregate them in a social-choice theoretically optimal way

Trolley problems, ethical dilemmas, self-driving cars

• Brakes of self-driving car have failed

• Should it swerve and hit a doctor and a cat?

• Or should it crash into a concrete barrier that will kill all five passengers?

Figure 3: Moral Machine—Judge interface. This particular choice is between a group of pedestri-ans that includes a female doctor and a cat crossing on a green light, and a group of passengersincluding a woman, a male executive, an elderly man, an elderly woman, and a girl.

have an easy-to-interpret tradeo↵ with respect to some dimension, such as gender (males on oneside vs. females on the other), age (elderly vs. young), fitness (large vs. fit), etc., while otherinstances have groups consisting of completely randomized characters being sacrificed in eitheralternative. Alternatives with all possible combinations of these features are considered, exceptfor the legality feature in cases when passengers are sacrificed. In addition, each alternative hasa derived feature, “number of characters,” which is simply the sum of counts of the 20 charactertypes (making d = 23).

As mentioned in Section 1, the Moral Machine dataset consists of preference data from 1,303,778voters, amounting to a total of 18,254,285 pairwise comparisons. We used this dataset to learn the� parameters of all 1.3 million voters (Step II, as given in Section 5). Next, we took the mean ofall of these � vectors to obtain �̂ (Step III). This gave us an implemented system, which can beused to make real-time choices between any finite subset of alternatives.

Importantly, the methodology we used, in Section 6.1, to evaluate Step II on the synthetic datacannot be applied to the Moral Machine data, because we do not know which alternative would beselected by aggregating the preferences of the actual 1.3 million voters over a subset of alternatives.However, we can apply a methodology similar to that of Section 6.1 in order to evaluate StepIII. Specifically, as in Section 6.1, we wish to compare the winner obtained using the summarizedmodel, with the winner obtained by applying Borda count to the mean of the anonymous preferenceprofiles of the voters.

An obstacle is that now we have a total of 1.3 million voters, and hence it would take anextremely long time to calculate the anonymous preference profile of each voter and take theirmean (this was the motivation for having Step III in the first place). So, instead, we estimate themean profile by sampling rankings, i.e., we sample a voter i uniformly at random, and then samplea ranking from the TM process of voter i; such a sampled ranking is an i.i.d. sample from the meananonymous profile. Then, we apply Borda count as before to obtain the desired winner (note thatthis approach is still too expensive to use in real time). The winner according to the summarizedmodel is computed exactly as before, and is just as e�cient even with 1.3 million voters.

Using this methodology, we computed accuracy on 3000 test instances, i.e., the fraction of

19

Trolley problems• Variant of the classical trolley problem (Thomson 1985)

• Different people, especially from different cultures and backgrounds, differ with respect to the optimal ethical action

• Moral Machine: dataset collected by collaborators at MIT

• website where individuals could provide their optimal ethical action for varying self-driving car trolley dilemmas

• each dilemma has two alternatives, characterized by 22 features (passengers or pedestrians, legality, differing character types (man/woman/child/cat/…), with varying characteristics (age/gender/…)

• dataset of responses from 1,303,778 individuals, from multiple countries, each with around 14 responses

Utilitarian Ethics


Model Decisions

“Fit” data well “optimal action” involves a loss function

`action(✓, a) : loss i.e. negative utility associated

with model parameter ✓, and action a<latexit sha1_base64="wQrbtYh0EczaF/LSZbD5H+4v250=">AAACfnicbVFNj9MwEHXC11K+Chy5DLRUCywhASQqTitx4bhIdHelpqqmzrS11nGCPVmoovwM/hg3fgsXnKZIsMtIlp7em+cZPy9KrRzH8c8gvHL12vUbezd7t27fuXuvf//BsSsqK2kiC13Y0wU60srQhBVrOi0tYb7QdLI4+9DqJ+dknSrMZ96UNMtxZdRSSWRPzfvfU9J6XqdM37hG2ZJNs5/ymhgP8BmM3kOngS6cAxVRBIZW3n1OULHSijeAzhVSIVMGTZr2RpB+qTD7Y4SviteQFxlpKNFiTkwWht2M4QGgyaCbDEMcbm+Y9wdxFG8LLoNkBwZiV0fz/o80K2SVk2Gp/TrTJC55VqNlJTU1vbRyVKI8wxVNPTR+CTert/E18NQzGSwL649h2LJ/O2rMndvkC9+ZI6/dRa0l/6dNK16OZ7UyZcVkZDdoWWngAtq/gExZkqw3HqC0yu8Kcu0Tkj4g14aQXHzyZXD8OkreRPGnt4PD8S6OPfFIPBH7IhHvxKH4KI7EREjxK3gcPA9ehCIchS/DV11rGOw8D8U/FY5/AwNXvzc=</latexit>

Individual Utility Model

Person ADecisions

“optimal action” involves a loss function

`A(a)<latexit sha1_base64="AeostOt0hwLKG7q3egMZSN5DsSs=">AAAB8nicbVBNS8NAEJ34WetX1aOXxSLUS0lU0GPVi8cK9gPSUDbbabt0swm7G6GE/gwvHhTx6q/x5r9x2+agrQ8GHu/NMDMvTATXxnW/nZXVtfWNzcJWcXtnd2+/dHDY1HGqGDZYLGLVDqlGwSU2DDcC24lCGoUCW+Hobuq3nlBpHstHM04wiOhA8j5n1FjJ76AQ3exmUqFn3VLZrbozkGXi5aQMOerd0lenF7M0QmmYoFr7npuYIKPKcCZwUuykGhPKRnSAvqWSRqiDbHbyhJxapUf6sbIlDZmpvycyGmk9jkLbGVEz1IveVPzP81PTvw4yLpPUoGTzRf1UEBOT6f+kxxUyI8aWUKa4vZWwIVWUGZtS0YbgLb68TJrnVe+i6j5clmu3eRwFOIYTqIAHV1CDe6hDAxjE8Ayv8OYY58V5dz7mrStOPnMEf+B8/gCcAJDN</latexit>

Varied Individual Utility Models

Person A

“optimal action” involves a loss function

Decisions


Person B

`B(a)<latexit sha1_base64="KLOZAi5tdgUEG/Pur8pHtR+NjwI=">AAAB8nicbVBNS8NAEN34WetX1aOXYBHqpSQq6LHUi8cK9gPSUDbbabt0sxt2J0IJ/RlePCji1V/jzX/jts1BWx8MPN6bYWZelAhu0PO+nbX1jc2t7cJOcXdv/+CwdHTcMirVDJpMCaU7ETUguIQmchTQSTTQOBLQjsZ3M7/9BNpwJR9xkkAY06HkA84oWinoghC9rD6t0IteqexVvTncVeLnpExyNHqlr25fsTQGiUxQYwLfSzDMqEbOBEyL3dRAQtmYDiGwVNIYTJjNT56651bpuwOlbUl05+rviYzGxkziyHbGFEdm2ZuJ/3lBioPbMOMySREkWywapMJF5c7+d/tcA0MxsYQyze2tLhtRTRnalIo2BH/55VXSuqz6V1Xv4bpcq+dxFMgpOSMV4pMbUiP3pEGahBFFnskreXPQeXHenY9F65qTz5yQP3A+fwCdiJDO</latexit>

.

.

.

Decisions

.

.

.

Aggregation of Utility Models

Person A


Person B

`B(a)<latexit sha1_base64="KLOZAi5tdgUEG/Pur8pHtR+NjwI=">AAAB8nicbVBNS8NAEN34WetX1aOXYBHqpSQq6LHUi8cK9gPSUDbbabt0sxt2J0IJ/RlePCji1V/jzX/jts1BWx8MPN6bYWZelAhu0PO+nbX1jc2t7cJOcXdv/+CwdHTcMirVDJpMCaU7ETUguIQmchTQSTTQOBLQjsZ3M7/9BNpwJR9xkkAY06HkA84oWinoghC9rD6t0IteqexVvTncVeLnpExyNHqlr25fsTQGiUxQYwLfSzDMqEbOBEyL3dRAQtmYDiGwVNIYTJjNT56651bpuwOlbUl05+rviYzGxkziyHbGFEdm2ZuJ/3lBioPbMOMySREkWywapMJF5c7+d/tcA0MxsYQyze2tLhtRTRnalIo2BH/55VXSuqz6V1Xv4bpcq+dxFMgpOSMV4pMbUiP3pEGahBFFnskreXPQeXHenY9F65qTz5yQP3A+fwCdiJDO</latexit>

.

.

.

`aggregation(a)<latexit sha1_base64="d6FobQltsmK3Tbgzs7ogKVLVHsI=">AAACBXicbVA9SwNBEN2LXzF+RS21OAxCbMKdCloGbSwjmA9IwrG3mbss2ftgd04MxzU2/hUbC0Vs/Q92/hs3lxSa+GDg8d4MM/PcWHCFlvVtFJaWV1bXiuuljc2t7Z3y7l5LRYlk0GSRiGTHpQoED6GJHAV0Ygk0cAW03dH1xG/fg1Q8Cu9wHEM/oH7IPc4oaskpH/ZACCftITxgSn1fgp87WValJ065YtWsHOYisWekQmZoOOWv3iBiSQAhMkGV6tpWjP2USuRMQFbqJQpiykbUh66mIQ1A9dP8i8w81srA9CKpK0QzV39PpDRQahy4ujOgOFTz3kT8z+sm6F32Ux7GCULIpou8RJgYmZNIzAGXwFCMNaFMcn2ryYZUUoY6uJIOwZ5/eZG0Tmv2Wc26Pa/Ur2ZxFMkBOSJVYpMLUic3pEGahJFH8kxeyZvxZLwY78bHtLVgzGb2yR8Ynz9AnJkI</latexit>

Decisions

Ethical AI via aggregation of learned utility models

• Task I: Learn individual utility (or loss) models

• Task II: Aggregate individual utility models to create a “consensus” utility model

Learning individual utility models

• Random Utility Models (RUM): Given a set A of actions/ alternatives, a random utility model U is a stochastic process where U(a), for any alternative a in A, denotes the random utility (negative loss) associated with alternative a

• Thurstone-Mosteller (TM) RUM:

• Plackket-Luce (PL) RUM:

• Parameterized by mean utility parameters

U(a) ⇠ N (µa,�2), where µa is mean utility for alternative a

<latexit sha1_base64="wq/12py0NhGiKIVp8jJDABL32xc=">AAACSXicbVA9bxNBEN1zgATzZaCkGWEjOVJk3YVISRmJhgoFCSeRfMbMrefsVXb3TrtzCdbJfy9NOjr+Aw0FCFGxvriAhFc9vfdmd+ZlpVae4/hr1Nq4c/fe5tb99oOHjx4/6Tx9duyLykkaykIX7jRDT1pZGrJiTaelIzSZppPs7M3KPzkn51VhP/CipLHBmVW5kshBmnQ+Dfu4DalXBlKDPJeo63fLfmqqCe4EeWbw4+72DqRMn7mGizk5gl5j90B5MIQWKlZa8QLywgFqJmfD6+chh73lpNONB3EDuE2SNemKNY4mnS/ptJCVIctSo/ejJC55XKNjJTUt22nlqUR5hjMaBWrRkB/XTRNLeBWUabNHXliGRv17okbj/cJkIbm61t/0VuL/vFHF+cG4VrasmKy8/iivNHABq1phqhxJ1otAUDoVdgU5R4cytOHboYTk5sm3yfHuIHk9iN/vdQ8P1nVsiRfipeiLROyLQ/FWHImhkOJSfBM/xM/oKvoe/Yp+X0db0XrmufgHrY0/59+xKg==</latexit>

U(a) ⇠ Gumbel(µa, �), where µa is mean utility for alternative a<latexit sha1_base64="lsFshvDZozV+ocmLIl8ETVMbWvA=">AAACSXicbVBNTxsxFPQG2tL0K7THXiySSkFC0W5BKkekHugRpAaQslH61nmbWNjelf1MG63y97hw48Z/6KWHVqinOsseWuicRjPz/J4nK5V0FMc3UWtt/dHjJxtP28+ev3j5qrP5+sQV3gocikIV9iwDh0oaHJIkhWelRdCZwtPs/OPKP71A62RhPtOixLGGmZG5FEBBmnS+DPuwzVMnNU8Jv1F16HWGatlPtZ/ATjoDrWF7pzH51zla5L3a7HHpuEYw3JNUkhY8LywHRWhNeP0i5KC3nHS68SCuwR+SpCFd1uBo0rlOp4XwGg0JBc6NkrikcQWWpFC4bKfeYQniHGY4CtSARjeu6iaW/F1QpvUdeWGI1+rfExVo5xY6C0kNNHf3vZX4P2/kKd8fV9KUntCIu0W5V5wKvqqVT6VFQWoRCAgrw61czMGCCG24dighuf/lh+Tk/SDZHcTHe92D/aaODfaWbbE+S9gHdsA+sSM2ZIJdsu/sJ/sVXUU/otvo9120FTUzb9g/aK39AYstsYU=</latexit>

{µa}a2A<latexit sha1_base64="BEpYcSvenCZpZCzlOku2AeeQ3us=">AAAB/nicbVDLSsNAFL3xWesrKq7cDBbBVUlUsMuKG5cV7AOaECbTSTt0MgkzE6GEgL/ixoUibv0Od/6N0zYLbT1wL4dz7mXunDDlTGnH+bZWVtfWNzYrW9Xtnd29ffvgsKOSTBLaJglPZC/EinImaFszzWkvlRTHIafdcHw79buPVCqWiAc9Sakf46FgESNYGymwj73ci7Mgx4VXmI48JtBNEdg1p+7MgJaJW5IalGgF9pc3SEgWU6EJx0r1XSfVfo6lZoTToupliqaYjPGQ9g0VOKbKz2fnF+jMKAMUJdKU0Gim/t7IcazUJA7NZIz1SC16U/E/r5/pqOHnTKSZpoLMH4oyjnSCplmgAZOUaD4xBBPJzK2IjLDERJvEqiYEd/HLy6RzUXcv6879Va3ZKOOowAmcwjm4cA1NuIMWtIFADs/wCm/Wk/VivVsf89EVq9w5gj+wPn8AGwWVhw==</latexit>

Learning a TM RUM• Data: pairwise comparisons

• e.g. {5 passengers > cat + doctor}

•

•

•

• Estimator:

{ai � bi}ni=1<latexit sha1_base64="RDkdn2RGdkf1mlvK8DyEOcDypJ0=">AAACCHicbZDLSsNAFIZPvNZ6i7p04WARXJVEBbsRCm5cVrAXaGOYTCft0MkkzEyEErJ046u4caGIWx/BnW/jpO1CW38Y+PjPOZw5f5BwprTjfFtLyyura+uljfLm1vbOrr2331JxKgltkpjHshNgRTkTtKmZ5rSTSIqjgNN2MLou6u0HKhWLxZ0eJ9SL8ECwkBGsjeXbR70MYZ+hnkoJQUFBuZ8xdIXc/D4TuW9XnKozEVoEdwYVmKnh21+9fkzSiApNOFaq6zqJ9jIsNSOc5uVeqmiCyQgPaNegwBFVXjY5JEcnxumjMJbmCY0m7u+JDEdKjaPAdEZYD9V8rTD/q3VTHda8jIkk1VSQ6aIw5UjHqEgF9ZmkRPOxAUwkM39FZIglJtpkVzYhuPMnL0LrrOqeV53bi0q9NoujBIdwDKfgwiXU4QYa0AQCj/AMr/BmPVkv1rv1MW1dsmYzB/BH1ucP5y+YmA==</latexit>

Linear parameterization: µa = h�, ai<latexit sha1_base64="32n9qGJhKjKfwyLmM8t8zEdVlcM=">AAACJHicbVC7SgNBFJ31GeMramkzGAULCbtaKIoQsLGwiGAekF2Wu5ObODg7u8zMCjHkY2z8FRsLH1jY+C1ONil8HRg4nHMPc++JUsG1cd0PZ2p6ZnZuvrBQXFxaXlktra03dJIphnWWiES1ItAouMS64UZgK1UIcSSwGd2cjfzmLSrNE3ll+ikGMfQk73IGxkph6eTCBkHRFBTEaFDxu9w5ptt+nIVAT6kvQPYEUj9CA3sUqK9yYTssld2Km4P+Jd6ElMkEtbD06ncSlsUoDROgddtzUxMMQBnOBA6LfqYxBXYDPWxbKu1GOhjkRw7pjlU6tJso+6Shufo9MYBY634c2ckYzLX+7Y3E/7x2ZrpHwYDLNDMo2fijbiaoSeioMdrhCpkRfUuAKW53peza1sVsW7poS/B+n/yXNPYr3kHFvdwvV48mdRTIJtkiu8Qjh6RKzkmN1Akj9+SRPJMX58F5ct6c9/HolDPJbJAfcD6/ACt3o88=</latexit>

U(a) ⇠ N (h�, ai, 1/2)<latexit sha1_base64="TGY7+Suu9cyWY+0rwdcuQo9q17E=">AAACHXicbVBNSwMxFMz6bf2qevQSLEILUndrwR4LXjyJgquFbilv09cazGaXJCuUpX/Ei3/FiwdFPHgR/43Z2oO2DgSGmfd4mQkTwbVx3S9nbn5hcWl5ZbWwtr6xuVXc3rnWcaoY+iwWsWqFoFFwib7hRmArUQhRKPAmvDvN/Zt7VJrH8soME+xEMJC8zxkYK3WLdb8MFRpoHtEgAnPLQGTno3IgQA4E0iBEA4cUaKDGwiH1jmqVbrHkVt0x6CzxJqREJrjoFj+CXszSCKVhArRue25iOhkow5nAUSFINSbA7mCAbUslRKg72TjdiB5YpUf7sbJPGjpWf29kEGk9jEI7mSfQ014u/ue1U9NvdDIuk9SgZD+H+qmgJqZ5VbTHFTIjhpYAU9z+lbJbUMCMLbRgS/CmI8+S61rVO666l/VSszGpY4XskX1SJh45IU1yRi6ITxh5IE/khbw6j86z8+a8/4zOOZOdXfIHzuc3yYCfyw==</latexit>

P�(ai � bi) = P(U�(ai) > U�(bi))

= �(�T (ai � bi)) . . . � is the standard normal CDF<latexit sha1_base64="TK4+Y522FkBhYzzhJUIGezUOIyQ=">AAACsXichVFNb9NAEF2brxK+Ahy5jEhByYHIpkj0AqpUhDgGKWmLYtfMrjf1Kuu12R0jIsv/jzM3/g22G6mlReJJKz29ebOz+4aXWjkKgt+ef+Pmrdt3du4O7t1/8PDR8PGTI1dUVsiFKHRhTzg6qZWRC1Kk5UlpJeZcy2O+Puzqx9+ldaowc9qUMs7xzKiVEkitlAx/RjlSxnk9a5I64pKwGWOiIHKVEMATNYGX7+DCNF5ctk3gPVwInXsSRQP4L7orZ5ka942nc+hHvurHTSD6VmEKUZYW5CAi+YNq2O3su6AcUCbBEZoUbQqmsDlqOPzwsUmGo2Aa9IDrJNySEdtilgx/RWkhqlwaEhqdW4ZBSXGNlpTQshlElZMlijWeyWVLDebSxXWfeAMvWiWFVWHbYwh69XJHjblzm5y3zi46d7XWif+qLSta7ce1MmVF0ojzQatKAxXQrQ9SZaUgvWkJCqvat4LI0KKgdsmDNoTw6pevk6PX03BvGnx+MzrY38axw56x52zMQvaWHbBPbMYWTHhTb+7F3qm/53/xv/r83Op7256n7C/46z9h58gB</latexit>

b� 2 arg sup�

(nY

i=1

P�(ai � bi)

)

<latexit sha1_base64="wSFwCAq6UsfQygDiKatPnEmMNQc=">AAACUHicbVFNbxMxEPWGj5bwFeDIxSJCKpdqF1Wil0oVvXAMEmkrxWE19s5urHq9K3u2KLL2J3Lprb+DCwcQOGmKoOVJlp/evNF4nmVrtKc0vUwGd+7eu7+1/WD48NHjJ09Hz54f+6ZzCqeqMY07leDRaItT0mTwtHUItTR4Is+OVvWTc3ReN/YTLVuc11BZXWoFFKV8VIkvusAFUBASCXoutOUCXCV81+Z/RIMlicBF65oiD/og6z8HG/UaaCFlmPTX1h3INY+9SnGZ6zdcOF0tSPT5aJzupmvw2yTbkDHbYJKPLkTRqK5GS8qA97MsbWkewJFWBvuh6Dy2oM6gwlmkFmr087AOpOevo1LwsnHxWOJr9e+OALX3y1pG52oDf7O2Ev9Xm3VU7s+Dtm1HaNXVoLIznBq+SpcX2qEis4wElNPxrVwtwIGi+AfDGEJ2c+Xb5PjtbhaT+bg3Pny/iWObvWSv2A7L2Dt2yD6wCZsyxb6yb+wH+5lcJN+TX4Pkynp9sxfsHwyGvwHdGLXz</latexit><latexit sha1_base64="wSFwCAq6UsfQygDiKatPnEmMNQc=">AAACUHicbVFNbxMxEPWGj5bwFeDIxSJCKpdqF1Wil0oVvXAMEmkrxWE19s5urHq9K3u2KLL2J3Lprb+DCwcQOGmKoOVJlp/evNF4nmVrtKc0vUwGd+7eu7+1/WD48NHjJ09Hz54f+6ZzCqeqMY07leDRaItT0mTwtHUItTR4Is+OVvWTc3ReN/YTLVuc11BZXWoFFKV8VIkvusAFUBASCXoutOUCXCV81+Z/RIMlicBF65oiD/og6z8HG/UaaCFlmPTX1h3INY+9SnGZ6zdcOF0tSPT5aJzupmvw2yTbkDHbYJKPLkTRqK5GS8qA97MsbWkewJFWBvuh6Dy2oM6gwlmkFmr087AOpOevo1LwsnHxWOJr9e+OALX3y1pG52oDf7O2Ev9Xm3VU7s+Dtm1HaNXVoLIznBq+SpcX2qEis4wElNPxrVwtwIGi+AfDGEJ2c+Xb5PjtbhaT+bg3Pny/iWObvWSv2A7L2Dt2yD6wCZsyxb6yb+wH+5lcJN+TX4Pkynp9sxfsHwyGvwHdGLXz</latexit><latexit sha1_base64="wSFwCAq6UsfQygDiKatPnEmMNQc=">AAACUHicbVFNbxMxEPWGj5bwFeDIxSJCKpdqF1Wil0oVvXAMEmkrxWE19s5urHq9K3u2KLL2J3Lprb+DCwcQOGmKoOVJlp/evNF4nmVrtKc0vUwGd+7eu7+1/WD48NHjJ09Hz54f+6ZzCqeqMY07leDRaItT0mTwtHUItTR4Is+OVvWTc3ReN/YTLVuc11BZXWoFFKV8VIkvusAFUBASCXoutOUCXCV81+Z/RIMlicBF65oiD/og6z8HG/UaaCFlmPTX1h3INY+9SnGZ6zdcOF0tSPT5aJzupmvw2yTbkDHbYJKPLkTRqK5GS8qA97MsbWkewJFWBvuh6Dy2oM6gwlmkFmr087AOpOevo1LwsnHxWOJr9e+OALX3y1pG52oDf7O2Ev9Xm3VU7s+Dtm1HaNXVoLIznBq+SpcX2qEis4wElNPxrVwtwIGi+AfDGEJ2c+Xb5PjtbhaT+bg3Pny/iWObvWSv2A7L2Dt2yD6wCZsyxb6yb+wH+5lcJN+TX4Pkynp9sxfsHwyGvwHdGLXz</latexit><latexit sha1_base64="wSFwCAq6UsfQygDiKatPnEmMNQc=">AAACUHicbVFNbxMxEPWGj5bwFeDIxSJCKpdqF1Wil0oVvXAMEmkrxWE19s5urHq9K3u2KLL2J3Lprb+DCwcQOGmKoOVJlp/evNF4nmVrtKc0vUwGd+7eu7+1/WD48NHjJ09Hz54f+6ZzCqeqMY07leDRaItT0mTwtHUItTR4Is+OVvWTc3ReN/YTLVuc11BZXWoFFKV8VIkvusAFUBASCXoutOUCXCV81+Z/RIMlicBF65oiD/og6z8HG/UaaCFlmPTX1h3INY+9SnGZ6zdcOF0tSPT5aJzupmvw2yTbkDHbYJKPLkTRqK5GS8qA97MsbWkewJFWBvuh6Dy2oM6gwlmkFmr087AOpOevo1LwsnHxWOJr9e+OALX3y1pG52oDf7O2Ev9Xm3VU7s+Dtm1HaNXVoLIznBq+SpcX2qEis4wElNPxrVwtwIGi+AfDGEJ2c+Xb5PjtbhaT+bg3Pny/iWObvWSv2A7L2Dt2yD6wCZsyxb6yb+wH+5lcJN+TX4Pkynp9sxfsHwyGvwHdGLXz</latexit>

Learning a TM RUM• Estimator:

Figure 1: Accuracy of Step II (synthetic data) Figure 2: Accuracy of Step III (synthetic data)

higher sampled utility. Once we have the comparisons, we learn the parameter �i by computingthe MLE (as explained in Step II of Section 5). In our results, we vary the number of pairwisecomparisons per voter and compute the accuracy to obtain the learning curve shown in Figure 1.Each datapoint in the graph is averaged over 50 runs. Observe that the accuracy quickly increasesas the number of pairwise comparisons increases, and with just 30 pairwise comparisons we achievean accuracy of 84.3%. With 100 pairwise comparisons, the accuracy is 92.4%.

Evaluation of Step III (Summarization). To evaluate Step III, we assume that we have accessto the true parameters �i, and wish to determine the accuracy loss incurred in the summarizationstep, where we summarize the individual TM models into a single TM model. As described inSection 5, we compute �̄ = 1

N

PNi=1 �i, and, given a subset A (which again has cardinality 5), we

aggregate using Step IV, since we now have just one TM process. For each instance, we contrastour computed winner with the desired winner as computed previously. We vary the number ofvoters and compute the accuracy to obtain Figure 2. The accuracies are averaged over 50 runs.Observe that the accuracy increases to 93.9% as the number of voters increases. In practice weexpect to have access to thousands, even millions, of votes (see Section 6.2). We conclude that,surprisingly, the expected loss in accuracy due to summarization is quite small.

Robustness. Our results are robust to the choice of parameters, as we demonstrate in Appendix A.

6.2 Moral Machine Data

Moral Machine is a platform for gathering data on human perception of the moral acceptability ofdecisions made by autonomous vehicles faced with choosing which humans to harm and which tosave. The main interface of Moral Machine is the Judge mode. This interface generates sessions ofrandom moral dilemmas. In each session, a user is faced with 13 instances. Each instance features anautonomous vehicle with a brake failure, facing a moral dilemma with two possible alternatives, thatis, each instance is a pairwise comparison. Each of the two alternatives corresponds to sacrificingthe lives of one group of characters to spare those of another group of characters. Figure 3 showsan example of such an instance. Respondents choose the outcome that they prefer the autonomousvehicle to make.

Each alternative is characterized by 22 features: relation to the autonomous vehicle (passengersor pedestrians), legality (no legality, explicitly legal crossing, or explicitly illegal crossing), andcounts of 20 character types, including ones like man, woman, pregnant woman, male athlete,female doctor, dog, etc. When sampling from the 20 characters, some instances are generated to

18

need very few comparisons per voter to learn their preferences

b� 2 arg sup�

(nY

i=1

P�(ai � bi)

)

<latexit sha1_base64="wSFwCAq6UsfQygDiKatPnEmMNQc=">AAACUHicbVFNbxMxEPWGj5bwFeDIxSJCKpdqF1Wil0oVvXAMEmkrxWE19s5urHq9K3u2KLL2J3Lprb+DCwcQOGmKoOVJlp/evNF4nmVrtKc0vUwGd+7eu7+1/WD48NHjJ09Hz54f+6ZzCqeqMY07leDRaItT0mTwtHUItTR4Is+OVvWTc3ReN/YTLVuc11BZXWoFFKV8VIkvusAFUBASCXoutOUCXCV81+Z/RIMlicBF65oiD/og6z8HG/UaaCFlmPTX1h3INY+9SnGZ6zdcOF0tSPT5aJzupmvw2yTbkDHbYJKPLkTRqK5GS8qA97MsbWkewJFWBvuh6Dy2oM6gwlmkFmr087AOpOevo1LwsnHxWOJr9e+OALX3y1pG52oDf7O2Ev9Xm3VU7s+Dtm1HaNXVoLIznBq+SpcX2qEis4wElNPxrVwtwIGi+AfDGEJ2c+Xb5PjtbhaT+bg3Pny/iWObvWSv2A7L2Dt2yD6wCZsyxb6yb+wH+5lcJN+TX4Pkynp9sxfsHwyGvwHdGLXz</latexit><latexit sha1_base64="wSFwCAq6UsfQygDiKatPnEmMNQc=">AAACUHicbVFNbxMxEPWGj5bwFeDIxSJCKpdqF1Wil0oVvXAMEmkrxWE19s5urHq9K3u2KLL2J3Lprb+DCwcQOGmKoOVJlp/evNF4nmVrtKc0vUwGd+7eu7+1/WD48NHjJ09Hz54f+6ZzCqeqMY07leDRaItT0mTwtHUItTR4Is+OVvWTc3ReN/YTLVuc11BZXWoFFKV8VIkvusAFUBASCXoutOUCXCV81+Z/RIMlicBF65oiD/og6z8HG/UaaCFlmPTX1h3INY+9SnGZ6zdcOF0tSPT5aJzupmvw2yTbkDHbYJKPLkTRqK5GS8qA97MsbWkewJFWBvuh6Dy2oM6gwlmkFmr087AOpOevo1LwsnHxWOJr9e+OALX3y1pG52oDf7O2Ev9Xm3VU7s+Dtm1HaNXVoLIznBq+SpcX2qEis4wElNPxrVwtwIGi+AfDGEJ2c+Xb5PjtbhaT+bg3Pny/iWObvWSv2A7L2Dt2yD6wCZsyxb6yb+wH+5lcJN+TX4Pkynp9sxfsHwyGvwHdGLXz</latexit><latexit sha1_base64="wSFwCAq6UsfQygDiKatPnEmMNQc=">AAACUHicbVFNbxMxEPWGj5bwFeDIxSJCKpdqF1Wil0oVvXAMEmkrxWE19s5urHq9K3u2KLL2J3Lprb+DCwcQOGmKoOVJlp/evNF4nmVrtKc0vUwGd+7eu7+1/WD48NHjJ09Hz54f+6ZzCqeqMY07leDRaItT0mTwtHUItTR4Is+OVvWTc3ReN/YTLVuc11BZXWoFFKV8VIkvusAFUBASCXoutOUCXCV81+Z/RIMlicBF65oiD/og6z8HG/UaaCFlmPTX1h3INY+9SnGZ6zdcOF0tSPT5aJzupmvw2yTbkDHbYJKPLkTRqK5GS8qA97MsbWkewJFWBvuh6Dy2oM6gwlmkFmr087AOpOevo1LwsnHxWOJr9e+OALX3y1pG52oDf7O2Ev9Xm3VU7s+Dtm1HaNXVoLIznBq+SpcX2qEis4wElNPxrVwtwIGi+AfDGEJ2c+Xb5PjtbhaT+bg3Pny/iWObvWSv2A7L2Dt2yD6wCZsyxb6yb+wH+5lcJN+TX4Pkynp9sxfsHwyGvwHdGLXz</latexit><latexit sha1_base64="wSFwCAq6UsfQygDiKatPnEmMNQc=">AAACUHicbVFNbxMxEPWGj5bwFeDIxSJCKpdqF1Wil0oVvXAMEmkrxWE19s5urHq9K3u2KLL2J3Lprb+DCwcQOGmKoOVJlp/evNF4nmVrtKc0vUwGd+7eu7+1/WD48NHjJ09Hz54f+6ZzCqeqMY07leDRaItT0mTwtHUItTR4Is+OVvWTc3ReN/YTLVuc11BZXWoFFKV8VIkvusAFUBASCXoutOUCXCV81+Z/RIMlicBF65oiD/og6z8HG/UaaCFlmPTX1h3INY+9SnGZ6zdcOF0tSPT5aJzupmvw2yTbkDHbYJKPLkTRqK5GS8qA97MsbWkewJFWBvuh6Dy2oM6gwlmkFmr087AOpOevo1LwsnHxWOJr9e+OALX3y1pG52oDf7O2Ev9Xm3VU7s+Dtm1HaNXVoLIznBq+SpcX2qEis4wElNPxrVwtwIGi+AfDGEJ2c+Xb5PjtbhaT+bg3Pny/iWObvWSv2A7L2Dt2yD6wCZsyxb6yb+wH+5lcJN+TX4Pkynp9sxfsHwyGvwHdGLXz</latexit>

Aggregating TM RUMs• Suppose N individuals give their ethical opinions, and for

each of them, we learn a separate TM RUM

• How do we aggregate these RUMs

• A reasonable estimator:

•

• Finds a TM RUM that is closest to average utility (giving one vote to each person)

•

{U�i(·)}Ni=1<latexit sha1_base64="VpigdkKigxbnBbfW0211sxetFUw=">AAACCXicbVBNS8NAEN34WetX1KOXxSLUS0lU0ItQ9OJJKpi20MSw2W7bpZsPdidCCbl68a948aCIV/+BN/+N2zYHbX0w8Hhvhpl5QSK4Asv6NhYWl5ZXVktr5fWNza1tc2e3qeJUUubQWMSyHRDFBI+YAxwEayeSkTAQrBUMr8Z+64FJxePoDkYJ80LSj3iPUwJa8k3sZo6fuQED4vO86tJuDEdu7mf8ws7vs5vcNytWzZoAzxO7IBVUoOGbX243pmnIIqCCKNWxrQS8jEjgVLC87KaKJYQOSZ91NI1IyJSXTT7J8aFWurgXS10R4In6eyIjoVKjMNCdIYGBmvXG4n9eJ4XeuZfxKEmBRXS6qJcKDDEex4K7XDIKYqQJoZLrWzEdEEko6PDKOgR79uV50jyu2Sc16/a0Ur8s4iihfXSAqshGZ6iOrlEDOYiiR/SMXtGb8WS8GO/Gx7R1wShm9tAfGJ8/f2qaMg==</latexit>

Theorem: b�AGG =1

N

nX

i=1

�i

<latexit sha1_base64="G8CvEP0kM3vOrNElSIGwAGgFyP8=">AAACRXicbVBNSxwxGM6oVbv1Y22PvQSXggdZZlRQFMGPgz0VC64KO9shk33HDSaZIXlHu4T5c15699Z/0EsPLcWrzY578OslgYfngzd50kIKi2H4M5iYnHozPTP7tvFubn5hsbn0/tTmpeHQ4bnMzXnKLEihoYMCJZwXBphKJZyll4cj/ewKjBW5PsFhAT3FLrTIBGfoqaQZxwjfMc3cyQByA2q7ovFOfa5FHwYMXZwCsipxtdHtHx1VFd2lcWYYd1HlvvjAKo1tqRIndqPqm9OeGWUSkTRbYTush74E0Ri0yHiOk+Zt3M95qUAjl8zabhQW2HPMoOASqkZcWigYv2QX0PVQMwW25+oWKvrJM32a5cZfjbRmHyccU9YOVeqdiuHAPtdG5Gtat8Rsq+eELkoEzR8WZaWkmNNRpbQvDHCUQw8YN8K/lfIB8/2gL77hS4ief/klOF1rR+vt8OtGa+9gXMcs+UiWyQqJyCbZI5/JMekQTm7IL/KH/A1+BL+Df8Hdg3UiGGc+kCcT3P8HwiayYg==</latexit>

b�AGG 2 arg inf�

KL

1

N

NX

i=1

U�ikU�

!

<latexit sha1_base64="HG9t8X7mTSAC6EDhBWqBoHfcSpY=">AAACZHicbVHRahQxFM2M1dZt1anFp4KELkJ9WWaqoC9C1YcKLVLBbQubdchk7+yGZjJDckddYn7SNx998Tua3R2Ktl4IOfecc0lyUjRKWkzTX1F8Z+3uvfWN+73NrQcPHyXbj89s3RoBQ1Gr2lwU3IKSGoYoUcFFY4BXhYLz4vL9Qj//CsbKWn/GeQPjik+1LKXgGKg8ceybnMCMo2MFIPe5Ywjf0b09OvKeMqkp42Ya9jLvHHRlOD7xTEGJ+6w0XLjMu49Bsm2VO/km818W7bCbyWWQfly3nhk5neHzPOmng3RZ9DbIOtAnXZ3myU82qUVbgUahuLWjLG1w7LhBKRT4HmstNFxc8imMAtS8Ajt2y5A8fRaYCS1rE5ZGumT/nnC8snZeFcFZcZzZm9qC/J82arF8PXZSNy2CFquDylZRrOkicTqRBgSqeQBcGBnuSsWMh9Aw/EsvhJDdfPJtcHYwyF4M0k8v+4fvujg2yC7ZI/skI6/IIflATsmQCPI7Wo+SaDv6E2/FO/GTlTWOupkd8k/FT68Aycu52A==</latexit>

Ethical Decisions via Aggregate TM RUM

Given alternatives {a1, . . . , am}, pick the alternative:

a 2 arg max{a1,...,am}

EU�AGG(a)

⌘ a 2 arg max{a1,...,am}

�TAGGa

<latexit sha1_base64="utZmtGK9orMNRFSfaNY+T+M5PRU=">AAAC2nicjVFNbxMxEPUuX2X5SuHIZUQKKiiqdnuhgksBoXAsUtNUisNq1pkkVrzexfZGjVZ74QBCXPll3PgZ/AOcNEil4cBIlp/em3kez2SlktbF8c8gvHL12vUbWzejW7fv3L3X2r5/YovKCOqJQhXmNENLSmrqOekUnZaGMM8U9bPZm6Xen5OxstDHblHSMMeJlmMp0Hkqbf3qyjlpQOXIaM/NycIOrzFNOnw6KpztYJrzZqcDpRQzcFO6mPsi4hlNpK5RyYl+1kCE8IRLDRzNhOd4ltabXg3wl+BFN82y+m0DPZ+UkUN/OTpz9atut2maXXzKecTpYyXngPC/thtGH44BvY0e/ekxbbXjvXgVsAmSNWizdRylrR98VIgqJ+2EQmsHSVy6YY3GSaGoiXhlqUQxwwkNPNSYkx3Wq9U08NgzIxgXxh/tYMVerKgxt3aRZz5zORF7WVuS/9IGlRsfDGupy8qRFucPjSsFroDlnmEkDQmnFh6gMNL3CmKKBoVfno38EJLLX94EJ/t7iZ/M+/324ev1OLbYQ/aI7bKEPWeH7B07Yj0mgn7QBJ+DLyEPP4Vfw2/nqWGwrnnA/orw+28aMOPh</latexit><latexit sha1_base64="utZmtGK9orMNRFSfaNY+T+M5PRU=">AAAC2nicjVFNbxMxEPUuX2X5SuHIZUQKKiiqdnuhgksBoXAsUtNUisNq1pkkVrzexfZGjVZ74QBCXPll3PgZ/AOcNEil4cBIlp/em3kez2SlktbF8c8gvHL12vUbWzejW7fv3L3X2r5/YovKCOqJQhXmNENLSmrqOekUnZaGMM8U9bPZm6Xen5OxstDHblHSMMeJlmMp0Hkqbf3qyjlpQOXIaM/NycIOrzFNOnw6KpztYJrzZqcDpRQzcFO6mPsi4hlNpK5RyYl+1kCE8IRLDRzNhOd4ltabXg3wl+BFN82y+m0DPZ+UkUN/OTpz9atut2maXXzKecTpYyXngPC/thtGH44BvY0e/ekxbbXjvXgVsAmSNWizdRylrR98VIgqJ+2EQmsHSVy6YY3GSaGoiXhlqUQxwwkNPNSYkx3Wq9U08NgzIxgXxh/tYMVerKgxt3aRZz5zORF7WVuS/9IGlRsfDGupy8qRFucPjSsFroDlnmEkDQmnFh6gMNL3CmKKBoVfno38EJLLX94EJ/t7iZ/M+/324ev1OLbYQ/aI7bKEPWeH7B07Yj0mgn7QBJ+DLyEPP4Vfw2/nqWGwrnnA/orw+28aMOPh</latexit><latexit sha1_base64="utZmtGK9orMNRFSfaNY+T+M5PRU=">AAAC2nicjVFNbxMxEPUuX2X5SuHIZUQKKiiqdnuhgksBoXAsUtNUisNq1pkkVrzexfZGjVZ74QBCXPll3PgZ/AOcNEil4cBIlp/em3kez2SlktbF8c8gvHL12vUbWzejW7fv3L3X2r5/YovKCOqJQhXmNENLSmrqOekUnZaGMM8U9bPZm6Xen5OxstDHblHSMMeJlmMp0Hkqbf3qyjlpQOXIaM/NycIOrzFNOnw6KpztYJrzZqcDpRQzcFO6mPsi4hlNpK5RyYl+1kCE8IRLDRzNhOd4ltabXg3wl+BFN82y+m0DPZ+UkUN/OTpz9atut2maXXzKecTpYyXngPC/thtGH44BvY0e/ekxbbXjvXgVsAmSNWizdRylrR98VIgqJ+2EQmsHSVy6YY3GSaGoiXhlqUQxwwkNPNSYkx3Wq9U08NgzIxgXxh/tYMVerKgxt3aRZz5zORF7WVuS/9IGlRsfDGupy8qRFucPjSsFroDlnmEkDQmnFh6gMNL3CmKKBoVfno38EJLLX94EJ/t7iZ/M+/324ev1OLbYQ/aI7bKEPWeH7B07Yj0mgn7QBJ+DLyEPP4Vfw2/nqWGwrnnA/orw+28aMOPh</latexit><latexit sha1_base64="utZmtGK9orMNRFSfaNY+T+M5PRU=">AAAC2nicjVFNbxMxEPUuX2X5SuHIZUQKKiiqdnuhgksBoXAsUtNUisNq1pkkVrzexfZGjVZ74QBCXPll3PgZ/AOcNEil4cBIlp/em3kez2SlktbF8c8gvHL12vUbWzejW7fv3L3X2r5/YovKCOqJQhXmNENLSmrqOekUnZaGMM8U9bPZm6Xen5OxstDHblHSMMeJlmMp0Hkqbf3qyjlpQOXIaM/NycIOrzFNOnw6KpztYJrzZqcDpRQzcFO6mPsi4hlNpK5RyYl+1kCE8IRLDRzNhOd4ltabXg3wl+BFN82y+m0DPZ+UkUN/OTpz9atut2maXXzKecTpYyXngPC/thtGH44BvY0e/ekxbbXjvXgVsAmSNWizdRylrR98VIgqJ+2EQmsHSVy6YY3GSaGoiXhlqUQxwwkNPNSYkx3Wq9U08NgzIxgXxh/tYMVerKgxt3aRZz5zORF7WVuS/9IGlRsfDGupy8qRFucPjSsFroDlnmEkDQmnFh6gMNL3CmKKBoVfno38EJLLX94EJ/t7iZ/M+/324ev1OLbYQ/aI7bKEPWeH7B07Yj0mgn7QBJ+DLyEPP4Vfw2/nqWGwrnnA/orw+28aMOPh</latexit>

Figure 3: Moral Machine—Judge interface. This particular choice is between a group of pedestri-ans that includes a female doctor and a cat crossing on a green light, and a group of passengersincluding a woman, a male executive, an elderly man, an elderly woman, and a girl.

have an easy-to-interpret tradeo↵ with respect to some dimension, such as gender (males on oneside vs. females on the other), age (elderly vs. young), fitness (large vs. fit), etc., while otherinstances have groups consisting of completely randomized characters being sacrificed in eitheralternative. Alternatives with all possible combinations of these features are considered, exceptfor the legality feature in cases when passengers are sacrificed. In addition, each alternative hasa derived feature, “number of characters,” which is simply the sum of counts of the 20 charactertypes (making d = 23).

As mentioned in Section 1, the Moral Machine dataset consists of preference data from 1,303,778voters, amounting to a total of 18,254,285 pairwise comparisons. We used this dataset to learn the� parameters of all 1.3 million voters (Step II, as given in Section 5). Next, we took the mean ofall of these � vectors to obtain �̂ (Step III). This gave us an implemented system, which can beused to make real-time choices between any finite subset of alternatives.

Importantly, the methodology we used, in Section 6.1, to evaluate Step II on the synthetic datacannot be applied to the Moral Machine data, because we do not know which alternative would beselected by aggregating the preferences of the actual 1.3 million voters over a subset of alternatives.However, we can apply a methodology similar to that of Section 6.1 in order to evaluate StepIII. Specifically, as in Section 6.1, we wish to compare the winner obtained using the summarizedmodel, with the winner obtained by applying Borda count to the mean of the anonymous preferenceprofiles of the voters.

An obstacle is that now we have a total of 1.3 million voters, and hence it would take anextremely long time to calculate the anonymous preference profile of each voter and take theirmean (this was the motivation for having Step III in the first place). So, instead, we estimate themean profile by sampling rankings, i.e., we sample a voter i uniformly at random, and then samplea ranking from the TM process of voter i; such a sampled ranking is an i.i.d. sample from the meananonymous profile. Then, we apply Borda count as before to obtain the desired winner (note thatthis approach is still too expensive to use in real time). The winner according to the summarizedmodel is computed exactly as before, and is just as e�cient even with 1.3 million voters.

Using this methodology, we computed accuracy on 3000 test instances, i.e., the fraction of

19

Validating Aggregate TM RUM

• Suppose we could conduct a real-time election: faced with a fresh set of alternatives, we ask each one of the millions of the voters, get their preferences, and then aggregate to get a consensus winning action

• impractical, computationally expensive

• decision made by aggregate TM RUM mimicked social-choice theoretically optimal aggregation of (large sample of) all the voter preferences

Figure 4: Accuracy of Step III (Moral Machine data)

instances in which the two winners match. Figure 4 shows the results as the number of alternativesper instance is increased from 2 to 10. Observe that the accuracy is as high as 98.2% at 2 alternativesper instance, and gracefully degrades to 95.1% at 10.

7 Discussion

The design of intelligent machines that can make ethical decisions is, arguably, one of the hardestchallenges in AI. We do believe that our approach takes a significant step towards addressingthis challenge. In particular, the implementation of our algorithm on the Moral Machine datasethas yielded a system which, arguably, can make credible decisions on ethical dilemmas in theautonomous vehicle domain (when all other options have failed). But this paper is clearly not theend-all solution.

7.1 Limitations

While the work we presented has some significant limitations, we view at least some of them asshortcomings of the current (proof-of-concept) implementation, rather than being inherent to theapproach itself, as we explain below.

First, Moral Machine users may be poorly informed about the dilemmas at hand, or maynot spend enough time thinking through the options, potentially leading— in some cases— toinconsistent answers and poor models. We believe, though, that much of this noise cancels out inSteps III and IV.

In this context, it is important to note that some of us have been working with colleagues on anapplication of the approach presented here to food allocation [12]. In this implementation—whichis a collaboration with 412 Food Rescue, a Pittsburgh-based nonprofit—the set of alternativesincludes hundreds of organizations (such as food pantries) that can receive incoming food donations.The voters in this implementation are a few dozen stakeholders: representatives of donor andrecipient organizations, volunteers (who deliver the donation from the donor to the recipient), andemployees of 412 Food Rescue. These voters are obviously well informed, and the results of Lee et al.[12] indicate that they have been exceptionally thoughtful in providing their answers to pairwise

20


• Theorem (Stability): If our system picks action a as the most ethical action when presented with a set A of alternatives, then it will again pick a as the most ethical action when presented with a set B of alternatives that is a subset of A, if it includes a.

• if the system prefers to save a dog over a cat or a mouse, then it should prefer to save a dog over a cat.


• Theorem (Swap Efficient): If our system picks action a as the most ethical action when presented with a set A of alternatives, then if there are two preferences which are identical except for swapped preferences between items a and b, then more people would have voted for the preference order where a is preferred to b.

Summary: Ethical AI• Machine Learning has a utilitarian foundation

• loss functions (or utilities) for (a) fitting model, (b) making decisions

• We learn per person utilities (loss functions), and aggregate them to form a consensus utility

• When doing so in the context of ethical decisions (for trolley dilemmas for self-driving cars), this results in an automated system that can make ethical decisions that represents the “ethical consensus” of millions of individuals

• computationally practical, satisfies strong social choice theoretic properties

• In ongoing work, we are developing ethical AI systems built on virtue ethics, and deontological ethics

• and learning more complex human utility models e.g. for suicidal behaviors

Ethical Decision Making in Artiﬁcial Intelligence · 2019. 3. 4. · rules • Virtue Ethics: ......

Documents

Transcript of Ethical Decision Making in Artiﬁcial Intelligence · 2019. 3. 4. · rules • Virtue Ethics: ......