IT Pro is supported by its audience. When you purchase through links on our site, we may earn an affiliate commission. Learn more

Crypto-mining hackers hit Kubernetes clusters

New campaign abused Kubeflow dashboards to install malicious containers

Bitcoin cryptocurrency mining

Security researchers have warned of hackers' continued attacks against Kubernetes clusters running Kubeflow machine learning (ML) instances by installing malicious containers that mine cryptocurrencies, such as Monero and Ethereum.

According to Microsoft Senior Security Researcher Yossi Weizman, the attacks began at the end of last month as he and his team discovered a spike in TensorFlow machine learning pod deployments. An investigation of the entry point of the pods revealed deployment aimed to mine cryptocurrency.

"The burst of deployments on the various clusters was simultaneous. This indicates that the attackers scanned those clusters in advance and maintained a list of potential targets, which were later attacked at the same time," said Weizman.

The hackers used two images in the attack. The first was the latest version of TensorFlow (tensorflow/tensorflow:latest), and the second was the latest version with GPU support (tensorflow/tensorflow:latest-gpu).

The images were legitimate but ran malicious crypto-mining code. The attackers abused the access to the Kubeflow centralized dashboard to create a new pipeline. Kubeflow Pipelines is a platform for deploying ML pipelines based on Argo Workflow. These dashboards were exposed to the internet instead of being only open to local access.

Hackers deployed at least two pods on each cluster: one for CPU mining, and the other for GPU mining. Both containers used open-source miners from GitHub: Ethminer in the case of the GPU container and XMRIG in the CPU one.

Related Resource

A guide to enterprise detection and response providers

The 12 providers that matter most and how they stack up

Forrester enterprise detection WPDownload now

The malicious pods all had the same pattern of name; "sequential-pipeline-{random pattern}." 

Weizman said that as part of the attack, hackers deployed a reconnaissance container that queries information about the environment, such as GPU and CPU information, as preparation for the mining activity. This also ran from a TensorFlow container.

"The attack is still active, and new Kubernetes clusters that run Kubeflow get compromised," Weizman added.

The campaign is similar to one staged in April last year. This also abused Kubernetes clusters in a crypto-mining campaign. However, instead of using Kubeflow Pipelines to deploy ML pipelines, it used a Jupyter notebook server. This campaign was the first that Microsoft observed targeting Kubeflow environments.

Weizman said that organizations running Kubeflow should ensure that the centralized dashboard isn't insecurely exposed to the internet. If Kubeflow should be exposed to the internet, make sure you use authentication. Administrators should also search containers that run TensorFlow images and inspect the entry point of those containers.

Featured Resources

Four strategies for building a hybrid workplace that works

All indications are that the future of work is hybrid, if it's not here already

Free webinar

The digital marketer’s guide to contextual insights and trends

How to use contextual intelligence to uncover new insights and inform strategies

Free Download

Ransomware and Microsoft 365 for business

What you need to know about reducing ransomware risk

Free Download

Building a modern strategy for analytics and machine learning success

Turning into business value

Free Download

Recommended

What is cryptocurrency mining?
cryptocurrencies

What is cryptocurrency mining?

23 Feb 2022
IMF urges El Salvador to remove Bitcoin as legal tender
cryptocurrencies

IMF urges El Salvador to remove Bitcoin as legal tender

26 Jan 2022
El Salvador announces plans to build a 'Bitcoin city' at the foot of a volcano
cryptocurrencies

El Salvador announces plans to build a 'Bitcoin city' at the foot of a volcano

22 Nov 2021
Hackers use Linux backdoor on compromised e-commerce sites with software skimmer
malware

Hackers use Linux backdoor on compromised e-commerce sites with software skimmer

19 Nov 2021

Most Popular

Russian hackers declare war on 10 countries after failed Eurovision DDoS attack
hacking

Russian hackers declare war on 10 countries after failed Eurovision DDoS attack

16 May 2022
Windows Server admins say latest Patch Tuesday broke authentication policies
Server & storage

Windows Server admins say latest Patch Tuesday broke authentication policies

12 May 2022
IT admin deletes company’s databases and is jailed for seven years
Policy & legislation

IT admin deletes company’s databases and is jailed for seven years

16 May 2022