Crypto-mining hackers hit Kubernetes clusters

New campaign abused Kubeflow dashboards to install malicious containers

Bitcoin cryptocurrency mining

Security researchers have warned of hackers' continued attacks against Kubernetes clusters running Kubeflow machine learning (ML) instances by installing malicious containers that mine cryptocurrencies, such as Monero and Ethereum.

According to Microsoft Senior Security Researcher Yossi Weizman, the attacks began at the end of last month as he and his team discovered a spike in TensorFlow machine learning pod deployments. An investigation of the entry point of the pods revealed deployment aimed to mine cryptocurrency.

"The burst of deployments on the various clusters was simultaneous. This indicates that the attackers scanned those clusters in advance and maintained a list of potential targets, which were later attacked at the same time," said Weizman.

The hackers used two images in the attack. The first was the latest version of TensorFlow (tensorflow/tensorflow:latest), and the second was the latest version with GPU support (tensorflow/tensorflow:latest-gpu).

The images were legitimate but ran malicious crypto-mining code. The attackers abused the access to the Kubeflow centralized dashboard to create a new pipeline. Kubeflow Pipelines is a platform for deploying ML pipelines based on Argo Workflow. These dashboards were exposed to the internet instead of being only open to local access.

Hackers deployed at least two pods on each cluster: one for CPU mining, and the other for GPU mining. Both containers used open-source miners from GitHub: Ethminer in the case of the GPU container and XMRIG in the CPU one.

Related Resource

A guide to enterprise detection and response providers

The 12 providers that matter most and how they stack up

Forrester enterprise detection WPDownload now

The malicious pods all had the same pattern of name; "sequential-pipeline-{random pattern}." 

Weizman said that as part of the attack, hackers deployed a reconnaissance container that queries information about the environment, such as GPU and CPU information, as preparation for the mining activity. This also ran from a TensorFlow container.

"The attack is still active, and new Kubernetes clusters that run Kubeflow get compromised," Weizman added.

The campaign is similar to one staged in April last year. This also abused Kubernetes clusters in a crypto-mining campaign. However, instead of using Kubeflow Pipelines to deploy ML pipelines, it used a Jupyter notebook server. This campaign was the first that Microsoft observed targeting Kubeflow environments.

Weizman said that organizations running Kubeflow should ensure that the centralized dashboard isn't insecurely exposed to the internet. If Kubeflow should be exposed to the internet, make sure you use authentication. Administrators should also search containers that run TensorFlow images and inspect the entry point of those containers.

Featured Resources

Shining light on new 'cool' cloud technologies and their drawbacks

IONOS Cloud Up! Summit, Cloud Technology Session with Russell Barley

Watch now

Build mobile and web apps faster

Three proven tips to accelerate modern app development

Free download

Reduce the carbon footprint of IT operations up to 88%

A carbon reduction opportunity

Free Download

Comparing serverless and server-based technologies

Determining the total cost of ownership

Free download

Recommended

El Salvador announces plans to build a 'Bitcoin city' at the foot of a volcano
cryptocurrencies

El Salvador announces plans to build a 'Bitcoin city' at the foot of a volcano

22 Nov 2021
Hackers use Linux backdoor on compromised e-commerce sites with software skimmer
malware

Hackers use Linux backdoor on compromised e-commerce sites with software skimmer

19 Nov 2021
Iranian hackers ramp up attacks against IT services sector
hacking

Iranian hackers ramp up attacks against IT services sector

19 Nov 2021
TikTok phishing campaign tried to scam over 125 influencer accounts
social media

TikTok phishing campaign tried to scam over 125 influencer accounts

18 Nov 2021

Most Popular

What should you really be asking about your remote access software?
Sponsored

What should you really be asking about your remote access software?

17 Nov 2021
What are the pros and cons of AI?
machine learning

What are the pros and cons of AI?

30 Nov 2021
What is single sign-on (SSO)?
single sign-on (SSO)

What is single sign-on (SSO)?

2 Dec 2021