Crypto-mining hackers hit Kubernetes clusters

New campaign abused Kubeflow dashboards to install malicious containers

Bitcoin cryptocurrency mining

Security researchers have warned of hackers' continued attacks against Kubernetes clusters running Kubeflow machine learning (ML) instances by installing malicious containers that mine cryptocurrencies, such as Monero and Ethereum.

According to Microsoft Senior Security Researcher Yossi Weizman, the attacks began at the end of last month as he and his team discovered a spike in TensorFlow machine learning pod deployments. An investigation of the entry point of the pods revealed deployment aimed to mine cryptocurrency.

"The burst of deployments on the various clusters was simultaneous. This indicates that the attackers scanned those clusters in advance and maintained a list of potential targets, which were later attacked at the same time," said Weizman.

The hackers used two images in the attack. The first was the latest version of TensorFlow (tensorflow/tensorflow:latest), and the second was the latest version with GPU support (tensorflow/tensorflow:latest-gpu).

The images were legitimate but ran malicious crypto-mining code. The attackers abused the access to the Kubeflow centralized dashboard to create a new pipeline. Kubeflow Pipelines is a platform for deploying ML pipelines based on Argo Workflow. These dashboards were exposed to the internet instead of being only open to local access.

Hackers deployed at least two pods on each cluster: one for CPU mining, and the other for GPU mining. Both containers used open-source miners from GitHub: Ethminer in the case of the GPU container and XMRIG in the CPU one.

Related Resource

A guide to enterprise detection and response providers

The 12 providers that matter most and how they stack up

Forrester enterprise detection WPDownload now

The malicious pods all had the same pattern of name; "sequential-pipeline-{random pattern}." 

Weizman said that as part of the attack, hackers deployed a reconnaissance container that queries information about the environment, such as GPU and CPU information, as preparation for the mining activity. This also ran from a TensorFlow container.

"The attack is still active, and new Kubernetes clusters that run Kubeflow get compromised," Weizman added.

The campaign is similar to one staged in April last year. This also abused Kubernetes clusters in a crypto-mining campaign. However, instead of using Kubeflow Pipelines to deploy ML pipelines, it used a Jupyter notebook server. This campaign was the first that Microsoft observed targeting Kubeflow environments.

Weizman said that organizations running Kubeflow should ensure that the centralized dashboard isn't insecurely exposed to the internet. If Kubeflow should be exposed to the internet, make sure you use authentication. Administrators should also search containers that run TensorFlow images and inspect the entry point of those containers.

Featured Resources

How to choose an AI vendor

Five key things to look for in an AI vendor

Download now

The UK 2020 Databerg report

Cloud adoption trends in the UK and recommendations for cloud migration

Download now

2021 state of email security report: Ransomware on the rise

Securing the enterprise in the COVID world

Download now

The impact of AWS in the UK

How AWS is powering Britain's fastest-growing companies

Download now

Recommended

ProtectedBy.AI’s CodeLock blocks malware at source code level
software as a service (SaaS)

ProtectedBy.AI’s CodeLock blocks malware at source code level

9 Jun 2021
Nigerian cyber criminals target Texas unemployment system
cyber security

Nigerian cyber criminals target Texas unemployment system

27 May 2021
CISOs aren’t leading by example when it comes to cyber security
cyber security

CISOs aren’t leading by example when it comes to cyber security

24 May 2021
Hackers use open source Microsoft dev platform to deliver trojans
Security

Hackers use open source Microsoft dev platform to deliver trojans

14 May 2021

Most Popular

Ten-year-old iOS 4 recreated as an iPhone app
iOS

Ten-year-old iOS 4 recreated as an iPhone app

10 Jun 2021
Fastly blames software bug for major outage
public cloud

Fastly blames software bug for major outage

9 Jun 2021
GitHub to prohibit code that’s used in active attacks
cyber security

GitHub to prohibit code that’s used in active attacks

7 Jun 2021