www.cloudsec.com | #cloudsec

Robustness in AI Technology

Prof Rajesh Vasa | Applied AI Institute @ Deakin University

Image generated by an AI

www.thispersondoesnotexist.com

This is state-of-art in pattern

generation (machines are moving

beyond pattern detection &

forecasting)

Image Generation

Power of AI (~2019)

Do you see the AI here?

So, what is AI?

AI is “A system that has the

appearance of intelligence which

aids a human in the completion of a

task”

Effective AI systems are very heavily

reliant on a human in the loop.

AI is “machines doing tasks

that until recently only

humans were able to do well”

AI is … Machines that mimic

"cognitive" functions that we

associate with human minds,

such as reasoning, learning and

problem solving

AI techniques scale human intelligence - mostly augment

Be honest about why you want AI (Learning, marketing, provable business value)

Data alone is not enough. Need subject matter experts and domain partner

Need different methodology for AI

Failure of AI is a first-class citizen - must deal with it in all parts are workflow

Utility and value of AI systems can only be properly determined in the real-world

Always be aware if AI is function or feature (in your context)

Evaluation protocol is hard (not easy to know if it works)

Total cost of ownership is high for AI

Key Points

Humans are involved – socio-technical problem

● Political

● Economic (business case)

● Engineering/Technology

If a human cannot do it – can a machine do it?

Even if a machine does a task, how can we

know ‘it works?’

AI systems serve humans

Evaluation is a hard problem

Not easy to know if it works & definition of “it

works” is hard

If we cannot define “it works” – how we do know

when it stops working or if performance has

degraded?

● How do you support and fix it?

● Evaluation in dev, trials and production are different

● Evaluation is expensive with significant overhead

Example - The elephant in the room (~ 2018)

How can we evaluate below?

Text generated from an initial lead line (~ 2019 state of art)

“Cooking rice and beans by steaming a roast in a wok is easy! Just follow these 40 simple

steps to update your XBox firmware, and you'll end up with a nice fried soup”

“Comedians fear the looming resolution of a long-running comedian feud. Also, Soviet

spectators at the Munich Olympics cheer Yuri Gagarin, who, although escorted by Russian

soldiers, uses rockets and airplanes in his Olympic performances to win multiple medals.

The crowd of Soviet spectators, "[l]argely composed of high school students in tight-fitting

vacant uniforms [...] walked away believing that Gagarin was the next North America's

greatest athlete”

• Machines learn from past (curated/clean) data & the expectation is that

future data will be like past data

• Data has no meaning by itself - humans are required to provide context.

• Machine do not have a model of reality and cannot make assumptions -

humans are required to provide context.

• Machine reasoning has limits - machines cannot do anything viable without

human support.

Data (or compute) alone is not enough

AI systems work (mostly) on probabilities

By their very nature – it can guess wrong

When they guess wrong….

- Is cost of failure acceptable (stakeholders)?

- Does the user know it can guess wrong?

- Do you hide the failure (from stakeholders)?

- Is machine aware of its competence?

Failure is a first-class citizen

Function = Core purpose of the system.

Features = Complement

(Needed competitiveness or marketing)

Function of a car is to move people safely.

Cup holders are a feature (helpful for

marketing, but not core functionally).

Is AI a Feature or function?

The true value best known in production.

Reality provides feedback needed (no short-cuts)

Value to the end-user takes time to determine

(politics of managing expectations)

Utility/Value determined only after building and using AI

Moving beyond prototype is time consuming

Prototype Phase (Months)

Excitement | Feasibility | Excitement

Production (Year+)

Refinement evaluation loop | Scale | Market

Evaluation (Months)

Does it work for real?

Requirement Discovery

Software Development

Data Processing

Model Development

Strict,structured testing and evaluation Data ProcessingData ProcessingData ProcessingData Processing

Model DevelopmentModel DevelopmentModel DevelopmentModel Development & testing

Software DevelopmentSoftware DevelopmentSoftware Development

RequirementsRequirementsRequirementsRequirements

• Need evaluation protocols to measure

operational effectiveness not technical accuracy

• Must be in production(-ish) environment early

• Issues & bugs may mean a full life-cycle of work

• Need much more training and support than

traditional systems

We need a different methodology for AI

AI systems need data, subject matter experts, machine learning experts,

specialized testers, and domain partners to evaluate.

They take longer to get right - high dev cost

They take longer to test and tune - high QA cost

They need more complex infrastructure - expensive to run

They need better trained resources to operate - expensive HR

They degrade in unpredictable ways - risk return ratio is uneven

So, only put in AI ... If the business case really stacks up!

TCO for A.I. is high

Focus on robustness and business

case early..

else, your AI project will deliver a

cute prototype

Key Point

Would you climb Mt Everest in a t-shirt?

Cyber-Security Perspective

• Vendors are offering ML/AI based features – evaluate in your context (i.e. they are not perfect & will fail in unexpected ways)

• The data and optimization choices in machine learning matter a LOT (may not work universally)

• AI/ML technology offers an asymmetric advantage to hackers (but it is hard for them too & may back-fire on them)

• If you are looking to invest in your own AI/ML – be patient

#cloudsec www.cloudsec.com

Thank you