Select Page

Introduction

Apache Spark is a distributed open-source system for large-scale data processing and fast general-purpose calculation. It also supports a wide range of advanced tools, including Spark SQL for SQL and DataFrames, pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing. Its powerful features combined with free access make it widely used worldwide. As a result, the shell command injection vulnerability has a broad impact.

Vulnerability

The shell command injection vulnerability revolves around manipulating application ACLs, which allows users/groups – excluding the owner who starts the application – access to an application. The Apache Spark UI offers the possibility to enable ACLs via the configuration option spark.acls.enable. If ACLs are enabled, Apache spark will go through the HttpSecurityFilter filter to verify the user’s permissions. In the code of HttpSecurityFilter, the username can be entered arbitrarily and the fields will be concatenated later. Therefore, a malicious user might then be able to inject a piece of shell command by separating the characters of the command to several username fields. Then after the fields are concatenated into a piece of the shell command the executeAndGetOutput function will cause the command to execute.

Affected Version

Spark Core – Apache<=3.0.3

3.1.1 <= Spark Core – Apache <=3.1.2

3.2.0 <= Spark Core – Apache <=3.2.1

Remediation

1. Official fix: The vendor has released the patches, and we recommend that users update Apache Spark to 3.1.3, 3.2.2, 3.3.0, or a higher version. Please visit:

https://spark.apache.org/downloads.html

2. Reference methods on GitHub: We can fix the vulnerability by removing the call to bash in the ShellBasedGroupsMappingProvider function. Please visit:

https://github.com/apache/spark/pull/36315/files

Implementing the Fix

The Hillstone IPS can support the detection and protection of this vulnerability.

Figure 1. Apache Spark Shell Command Injection Vulnerability detected by Hillstone IPS

The hotspot intelligence of this vulnerability is available on Hillstone iSource as well.

Figure 2. Hotspot intelligence on Hillstone iSource

TOP 5 Reasons for SD-WAN — Unifying Networking and Security into One Platform

ADC V3.0 Delivers Traffic and Balances Links at an Unmatched Level

The Past, Present, and Future of the XDR

Simplifying SIEM, EDR, XDR & SOAR

Establish Broader Secure Network Edge with Hillstone A6800 and A7600

Increase Your Resilience Through Layered Security with NIPS V4.5

Hillstone sBDS Included in 2022 Gartner® Emerging Trends: Adoption Growth Insights for Network Detection and Response

DEFCON 30: Skytalks, Villages and Infosec Research

Hillstone Networks Included on the MSRC 2022 Global Most Valuable Security Researcher List