What Would Privacy-Preserving ML Look Like?

Artificial intelligence (AI) and machine learning (ML) have made huge gains over the last few years, thanks to the vast amount of data available and to the gains made in raw hardware processing power e.g., powerful CPUs, GPUs and more recently TPUs (tensor processing units).

With the success of AI and ML also come certain risks, in particular the risks around privacy of the collected data. Although privacy concerns with any type of data collection have always existed, ML systems amplify these concerns due to the sheer amount of data and the processing power available to train the models on such data. Information about the training data could leak at any point from how and where the data is collected and stored to any information that can be inferred from access to the trained machine learning model.

As far as ML models are concerned, training data may not be the only asset of importance that need protection. Depending on the how the ML model is accessed, the parameters and type/architecture of the trained models and the model itself, are also of importance and are considered intellectual property. Access control of the ML models is one way to protect the models and their parameters. For example, in ML as a Service (MLaaS) cloud platforms, the models are only accessible through a public query interface.

The naive approach to ensuring privacy has traditionally been to remove (or not collect) any identifying information about the data used for training. The Netflix Prize example is often cited to show this. Back in 2006, Netflix announced a contest to improve their movie rating predictions and published their dataset of 10 million anonymous movie ratings by 500,000 customers. Arvind Narayanan and Vitaly Shmatikov, from the University of Texas at Austin came up with a way to de-anonymize some of the data by comparing the anonymous movie rankings with publicly available information from IMDb. The researchers were able to link individuals with public ratings on IMDb with their privately rated films on Netflix, giving them the ability to figure out sensitive information about the individuals such as their political affiliation or other personal details.

The Netflix example wholly depended on access to their published dataset and also access to IMDb. What about cases where systems are more closed off and isolated from all other networks?

Let’s look at industrial control systems (ICS) for example. Such systems used to be a lot more isolated, but the growing demands of the industry for increased connectivity between the factory floor and the corporate network have altered connectivity landscape [3]. However, with this increased connectivity and utility, also comes various cybersecurity risks and as such confidentiality and privacy of data is even more important.

 The Triton Trojan[4] which targeted core industrial systems in the Middle East in 2018 exploited a zero-day flaw in Triconex controllers (used in ICS) to carry out its attack. It turns out the targeted Triconex SIS controllers were configured with the physical key switch in ‘program mode’ during operation, allowing a payload to be downloaded and deployed. The controllers should have been in ‘run mode’ to prevent arbitrary changes to the controller logic.

One can imagine the knowledge of which controllers were in ‘run mode’ could have helped attackers to target specific controllers instead of a more brute force approach. Knowing there is a correlation between firmware versions/patch levels and the key switch state, attackers would only need controller identifying information such as hostname, open ports, etc. Such identifying data is typically collected for ML and data mining mechanisms for intrusion detection for example, making this scenario highly plausible.

Privacy Model

To keep the training datasets private, the ML community has settled on a useful model for defining and evaluating privacy. Differential privacy (DP) is the main privacy model. Intuitively, it says that adding/removing an individual’s data to a training dataset should not affect the result of prediction queries by “much.”

Example of violating DP: Knowing the average grade of a group of people, can an individual’s grade be inferred? Ask the individual to leave the room, ask the remaining people to give you their new average. Use the difference between two sums, the individual’s grade (X1) is found.

How do we keep the individual’s grade differentially private? How about we add some random noise ε?

Now the individual’s grade can still be found but with a large noise added to it (e.g., nε):

Membership Inference Attacks[1]

The privacy of an ML system may be boiled down to the privacy of its most important asset – the training dataset. A training dataset may be compromised if membership in the training dataset can be inferred. This attack scenario is an example of a membership inference attack. The adversary in this setting is assumed to have black-box access to the ML model and can query the model on certain data points and see responses. The main insight is that ML models behave differently with regard to the training dataset vs. data they have not seen, i.e., some test data. This often leads to overfitting and can be exploited to identify members of the training set.

Federated Learning

Privacy concerns are especially prevalent in mobile devices or IoT device networks where device and/or client level data is collected. Typically, data is collected on the device and then shared with a central server (or cloud infrastructure). The data may be encrypted while in transit, but gets decrypted once in the cloud. The collection of data from all the devices in the network is then used to train an ML model in the cloud. If differential privacy is to be adhered to, this model of distribution of data has many pitfalls.

Researchers at Google[2] presented federated learning (FL) in 2017. FL is distributed ML and, as such, uses a decentralized approach for training the model using the user’s sensitive data. In this scenario, data never leaves the devices. Only the minimal information necessary for model training (weight deltas) is transmitted via short bursts of model updates and incorporated into the global model in the cloud. As privacy protection becomes a must-have for ML systems specially in mobile/IoT device networks, FL has the potential to become even more popular due to its many benefits. However, FL also makes new attacks possible.

Stay tuned for more on this topic from Irdeto. Our team of researchers is getting actively involved in the AI space, and in particular the security of AI and ML systems. In part 2 of this blog post, we will explore some of the security drawbacks for FL and explore a specific model stealing attack in detail.


[1] R. Shokri, M. Stronati, and V. Shmatikov, “Membership inference attacks against machine learning models,” arXiv preprint arXiv:1610.05820, 2016.

[2] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Agüera y Arcas: Communication-Efficient Learning of Deep Networks from Decentralized Data. AISTATS 2017: 1273-1282

[3] Vinay M. Igure, Sean A. Laughter, Ronald D. Williams: Security issues in SCADA networks. Computers & Security 25(7): 498-506 (2006)

[4]  https://www.zdnet.com/article/triton-exploited-zero-day-flaw-to-target-industrial-systems/

About Bahman Sistany

Bahman Sistany, Senior Security Analyst and Researcher

Bahman joined Irdeto in 2003 as a Compiler Designer, Bahman since then worked on several advanced application-level security features such as various control flow obfuscations. He has held several roles, including manager of large development, infrastructure and QA teams and later as a product evangelist and owner. More recently, Bahman has been leading research into obfuscation and tamper resistance technology, high assurance software protection and functional safety, application of AI/ML systems to security, and confidentiality/privacy of ML systems. Bahman holds a PhD in computer science (formal verification) from University of Ottawa.