Dark Reading is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Vulnerabilities / Threats

2/13/2020
04:35 PM
Connect Directly
Twitter
LinkedIn
RSS
E-Mail
50%
50%

Architectural Analysis IDs 78 Specific Risks in Machine-Learning Systems

The new threat model hones in on ML security at the design state.

Researchers at the Berryville Institute of Machine Learning (BIML) have developed a formal risk framework to guide development of secure machine-language (ML) systems.

BIML's architectural risk analysis of ML systems is different from previous work in this area in that it focuses on issues that engineers and developers need to be paying attention to at the outset when designing and building ML systems. Most of the previous work on securing ML systems has focused on how to best protect operational systems and data against particular attacks and not on how to design them securely in the first place.

"This work provides a very solid technical foundation for taking a look at the risks associated with adopting and using ML," says Gary McGraw, noted security researcher, author, and co-founder of BMIL. The need for this kind of a risk analysis is critical because very few are really paying any attention to ML security at the design state, even as ML use is growing rapidly, he says.

For the architectural risk analysis, BIML researchers considered nine separate components that they identified as common to setting up, training, and deploying a typical ML system: raw data; dataset assembly; datasets; learning algorithms; evaluation; inputs; trained model; inference algorithm; and outputs. They then identified and ranked multiple data security risks associated with each of those components so engineers and developers can implement controls for mitigating those risks where possible.

For instance, they identified data confidentiality, the trustworthiness of data sources, and data storage as key security considerations around the raw data used in ML systems, such as training data, test inputs, and operational data. Similarly, for the datasets used in ML systems, the researchers identified data poisoning — where an attacker manipulates data to cause ML systems to go awry — as a major risk. For training algorithms, BIML researchers identified the potential for attackers to subtly nudge an online learning system in a direction not intended by its developers as a major concern.

In total, BIML's architectural analysis showed that typical ML systems are exposed to as many as 78 specific security risks across all individual components. They categorized the risks under multiple categories including input manipulation, data manipulation, model manipulation, and extraction attacks where threat actors try and extract sensitive data from an ML system dataset.

McGraw says the BIML analysis is about identifying and discussing ML risks and discussing them, and not so much about what to do about them. "Identifying the risks is more than half the battle," he says. "Once you know what the risks are, it's a lot easier to design around them."

The BMIL report listed the top 10 risks impacting ML systems. According to the think tank, the biggest — and most commonly discussed risks — to ML systems are so-called "adversarial examples" involving the use of malicious inputs to cause the system to make false predictions or categorizations. Data poisoning, online system manipulation, and attacks impacting data confidentiality, data integrity, and data output were all identified as other top ML security risks.

The Importance of Data Security
"One of the remarkable differences in ML security and, say, normal operational security is that data and data security play a huge role," says McGraw. "When you are training up a system, you can train it up to be racist and xenophobic and horrible if your data are set up that way," he says.

As one example, he points to Microsoft's very short-lived experiment with Tay, an AI-enabled chatbot that learned from interactions on Twitter and quickly began spewing out venomous tweets of its own. "Tay was learning about Twitter by being on it, and what happened was it became a racist, bigoted troll," he says. "Tay learned what it was like to be on Twitter, and it wasn't pretty."  

Such incidents highlight why organizations need to think carefully about the data they are using for machine training, how the data gets sourced, and whether the sources are reliable, he says.

Contrary to what some might assume, attacking a machine-learning system is not all that complicated, McGraw notes. "Imagine the input data for Google Translate is anything that you type in," he says. "If you are using public data sources to train your machine learning model, you have to think about what happens when an attacker starts screwing around with.

"The good news is if you are an engineer or a designer, you can make it harder for someone to attack your system. That's the purpose of this work."

Related Content:

Check out The Edge, Dark Reading's new section for features, threat data, and in-depth perspectives. Today's featured story: "Chaos & Order: The Keys to Quantum-Proof Encryption"

Jai Vijayan is a seasoned technology reporter with over 20 years of experience in IT trade journalism. He was most recently a Senior Editor at Computerworld, where he covered information security and data privacy issues for the publication. Over the course of his 20-year ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Threaded  |  Newest First  |  Oldest First
gem@garymcgraw.com
50%
50%
[email protected],
User Rank: Apprentice
2/13/2020 | 11:22:44 PM
BIML Risk Analysis
Thanks for this thoughtful writeup of our work at BIML.  Though there is a link to our work in the body of the article, it may be hard to spot.

You can download the article here: berryvilleiml.com/results (cut and paste)

There is also an interactive risk model you can play with: berryvilleiml.com/interactive (cut and paste)

gem
Register for Dark Reading Newsletters
White Papers
Video
Cartoon Contest
Current Issue
6 Emerging Cyber Threats That Enterprises Face in 2020
This Tech Digest gives an in-depth look at six emerging cyber threats that enterprises could face in 2020. Download your copy today!
Flash Poll
State of Cybersecurity Incident Response
State of Cybersecurity Incident Response
Data breaches and regulations have forced organizations to pay closer attention to the security incident response function. However, security leaders may be overestimating their ability to detect and respond to security incidents. Read this report to find out more.
Twitter Feed
Dark Reading - Bug Report
Bug Report
Enterprise Vulnerabilities
From DHS/US-CERT's National Vulnerability Database
CVE-2020-8144
PUBLISHED: 2020-04-01
The UniFi Video Server v3.9.3 and prior (for Windows 7/8/10 x64) web interface Firmware Update functionality, under certain circumstances, does not validate firmware download destinations to ensure they are within the intended destination directory tree. It accepts a request with a URL to firmware u...
CVE-2020-8145
PUBLISHED: 2020-04-01
The UniFi Video Server (Windows) web interface configuration restore functionality at the “backup� and “wizard� endpoints does not implement sufficient privilege checks. Low privileged users, belonging to the PUBLIC_GROUP ...
CVE-2020-8146
PUBLISHED: 2020-04-01
In UniFi Video v3.10.1 (for Windows 7/8/10 x64) there is a Local Privileges Escalation to SYSTEM from arbitrary file deletion and DLL hijack vulnerabilities. The issue was fixed by adjusting the .tsExport folder when the controller is running on Windows and adjusting the SafeDllSearchMode in the win...
CVE-2020-6009
PUBLISHED: 2020-04-01
LearnDash Wordpress plugin version below 3.1.6 is vulnerable to Unauthenticated SQL Injection.
CVE-2020-6096
PUBLISHED: 2020-04-01
An exploitable signed comparison vulnerability exists in the ARMv7 memcpy() implementation of GNU glibc 2.30.9000. Calling memcpy() (on ARMv7 targets that utilize the GNU glibc implementation) with a negative value for the 'num' parameter results in a signed comparison vulnerability. If an attacker ...