Patching Perforce perforations: Essential RCE vulnerability found in Perforce Helix Core Server

Perforce Server listens on TCP port 1666 by default, although server directors will usually change this port quantity to cover from scanners or to host Perforce Server by way of TLS. Microsoft scanned the web in November 2023 for TCP port 1666 with a {custom} Perforce Server community signature and located over 1,000 uncovered Perforce Server cases.

On this weblog, we element how we found every of the vulnerabilities and spotlight the potential impression if exploited. Alongside making use of Perforce’s patches, we additionally embody further mitigation and safety steerage for patrons to attenuate the danger of exploitation. Lastly, we’re sharing this data with the broader neighborhood to drive consciousness to additional enhance protections throughout the safety ecosystem, and to emphasise the significance of accountable disclosure and collaboration to safe platforms and gadgets.

Discovering the vulnerabilities

To maintain Microsoft’s sport growth studios and their clients secure, we just lately performed an utility safety evaluate of Perforce Server, the supply code administration platform relied on by most of our studios. For our safety evaluate, we analyzed Perforce Server model 2023.1.244.2900 and put in it on Home windows 11 22H2. We used Perforce Server’s default set up choices, which resulted by-design within the Perforce Server service operating as LocalSystem:

Recovering debug symbols

In 2014, Perforce open-sourced the code for his or her CLI Perforce Shopper, and knowledgeable customers we will obtain the code from the bin.instruments subdirectory of any given launch. Whereas having any supply code is invaluable for utility safety vulnerability analysis functions, this supply code is particular to the shopper, not the server. The latter is just distributed in compiled binary type.

The binaries which might be put in by Perforce Server’s installer have their debug symbols stripped (faraway from the distributed executable pictures), which makes it tougher to grasp the disassembled code throughout static evaluation. To assist our evaluate, we tried to get better these debug symbols.

Discovering debug symbols

Generally purposes provide software program growth kits (SDKs) that may be mined for debug image information. Within the case of Perforce Server, Perforce affords a “C/C++ API” package deal for the Home windows (x64) platform that comes within the type of a .zip file containing three directories: embody, lib, and pattern. The lib listing is very attention-grabbing for us, because it accommodates about 400 MB of .lib recordsdata:

Like .exe recordsdata, .lib recordsdata are COFF recordsdata that may comprise debug symbols. Through the use of dumpbin.exe /symbols to examine every .lib file, we discovered that the 9 .lib recordsdata within the package deal comprise a complete of 1,251,756 debug image entries.

To know why that is helpful to us, allow us to think about an approximation of how .exe and .lib recordsdata are constructed:

Within the diagram above, we will see that the SDK .obj recordsdata have been linked together with server-specific .obj recordsdata to create Perforce Server’s p4s.exe (“Perforce Service”) file. Throughout that linking course of, the debug symbols have been stripped. Nonetheless, the identical SDK .obj recordsdata had their debug symbols retained when linked into the SDK .lib recordsdata. Because the .lib recordsdata comprise debug symbols, we will match every compiled SDK operate in every .lib file to its SDK operate identify. If we will then discover those self same compiled SDK capabilities in Perforce Server’s .exe and .dll recordsdata, we will map the SDK operate names to these capabilities as effectively, thus simplifying our evaluation of the p4s.exe file.

To start, we should first decide which SDK package deal to make use of for our evaluation. If we take a look at the containing listing for the .zip file downloaded from the “C/C++ API” package deal, we see it accommodates 144 p4api SDK packages:

The explanation we see 144 packages listed is that there’s each mixture of the next:

That’s 9 potential values for compiler, two potential values for linking, two potential values for construct kind, and 4 potential values for OpenSSL model. In different phrases, multiplying these 4 values collectively leads us to 144 potential combos. To map named capabilities from the SDK’s .lib recordsdata to Perforce Server’s p4s.exe file, we’ll want to decide on the proper SDK package deal, since, for instance, a operate compiled with Visible Studio 2005 might look very completely different from the identical operate compiled with Visible Studio 2022.

So how do we all know which compiler, linker choice, construct kind, and OpenSSL model have been used for our put in distribution of Perforce Server? We don’t. We might make some educated guesses and study artifacts such because the binaries’ Wealthy Headers to find out the best mixture, however as an alternative we selected to make use of automation to check all potential combos. (Observe that “Wealthy Headers” is a colloquial time period used within the business, not a Microsoft-official identify for this construction.)

Discovering the best set of debug symbols

After downloading the entire statically linked p4api archives from Perforce’s web site, we used IDA Professional’s F.L.I.R.T. know-how to create signatures for every Perforce Server SDK package deal. To take action, we automated the next steps:

1. Use pcf.exe (“parsecoff”) from IDA Professional’s Quick Library Acquisition for Identification and Recognition (FLAIR) SDK to create .pat (“sample”) recordsdata for every Perforce Server SDK package deal’s .lib file.
2. Use sigmake.exe from the FLAIR SDK to create a .sig (“signature”) file for all of the .pat recordsdata from every given Perforce Server SDK package deal.
3. Use zipsig.exe from the FLAIR SDK to compress every .sig file.
4. Disassemble Perforce Server’s p4s.exe file with IDA Professional and save the ensuing .idb (“IDA database”) file.
5. For every .sig file, open the .idb file, apply the .sig file, rely the variety of .sig file operate matches, and shut the .idb file with out saving the modifications.
6. Rank the variety of operate matches for every .sig file.

After following the method above, we discovered the debug symbols from p4api_vs2017_static_openssl1.1.1.zip had probably the most operate matches in p4s.exe:

The rest of this weblog put up leverages these signatures for p4s.exe’s operate names and kind data.

Investigating the RPC header

Provided that Perforce Server runs as LocalSystem, native elevation of privilege assaults will surely be worthwhile to discover. Nonetheless, distant assaults by way of a community are way more intriguing from a vulnerability analysis perspective. Our subsequent step is to research how Perforce Server handles information it receives from distant customers, or in our case, attackers.

Utilizing TCPView, we will see that p4s.exe is listening for incoming connections on TCP port 1666:

Packages constructed for Home windows that hear on TCP ports for incoming connections nearly all the time use Winsock’s recv() operate to obtain incoming community information from purchasers. Utilizing IDA Professional’s cross-references (“CODE XREF”s under), we will see that recv() is named by a number of capabilities:

We’re seeking to assess how acquired community information is parsed and dealt with, and to avoid wasting time in figuring out which of the capabilities above really receives the related shopper information by way of recv(), we used a debugger to set a breakpoint on recv() and reviewed its thread’s call-stack to disclose the next chain of operate calls:

Within the call-stack above, “Rpc” is brief for “Distant Process Name”, a frequent time period used for remotely executing capabilities.

Though we’re assessing the Perforce Server, the operate RpcTransport::Obtain() (in Determine 8) can also be included in the shopper supply code mentioned above (notice that the feedback are from Perforce’s builders, not from Microsoft):

The code above does the next:

1. On line 69, calls NetBuffer::Obtain() to obtain 5 bytes of information from the related TCP shopper. We are going to refer to those 5 bytes because the RPC header.
2. On line 72, verifies that the primary byte’s worth equals the worth of the next 4 bytes utilizing the XOR operation to compute a parity byte checksum.
3. On line 78, interprets these following 4 bytes as a 32-bit little-endian worth named size.
4. On line 85, verifies that size >= 12 and that size < 0x1FFFFFFF.
5. On line 93, allocates reminiscence of dimension size and receives size bytes from the related TCP shopper.

Nonetheless, there’s a design threat within the code above, in that there’s not ample safety towards uneven useful resource consumption assaults from distant unauthenticated attackers. An attacker might hook up with the Perforce Server, ship a five-byte RPC header specifying a size worth of 0x1FFFFFFE, and trigger the server to allocate 0x1FFFFFFE bytes (about 537 MB) of reminiscence. An attacker might exploit this vulnerability by establishing quite a few connections and requesting these massive reminiscence allocations by way of every connection, shortly consuming all of the server’s accessible reminiscence. As soon as accessible reminiscence is exhausted, the following name to Alloc() (step 5 above) will lead Perforce Server’s reminiscence allocator (which occurs to be mimalloc) to throw an unhandled std::bad_alloc() exception from mi_try_new_handler(), inflicting the Perforce Server course of to crash and never restart. This denial-of-service (DoS) assault is exploitable by distant unauthenticated attackers.

This vulnerability is now recognized as CVE-2023-5759 and it has a CVSS rating of seven.5.

Investigating RPC handler capabilities

We confirmed within the call-stack above that RpcTransport::Obtain() is named by Rpc::DispatchOne(). This latter operate takes the allotted buffer acquired by RpcTransport::Obtain(), parses it as an RPC command with non-compulsory arguments, seems up the handler for the given RPC command, and calls the handler with the acquired arguments. Many of those RPC instructions are mapped to the p4 instructions listed right here. Particularly, there are 202 formally documented p4 instructions, and about 450 outlined RPC instructions, although not all RPC instructions have their handlers registered by default at runtime.

Since we’re most interested by the potential for distant unprivileged assaults towards Perforce Server in its default configuration, we created our personal Perforce shopper from scratch that makes an attempt to name (with none authentication) every of the roughly 450 RPC instructions outlined in p4s.exe. Of these, we discovered that about 360 RPC instructions have their handlers registered by default at runtime. That is too excessive of a rely to manually assess in an inexpensive period of time, so we needed to discover different means to prioritize our RPC command evaluation.

We discovered that p4s.exe statically imports 382 API capabilities. Of these, we recognized probably the most attention-grabbing capabilities that would probably obtain distant code execution, assuming an unauthenticated distant attacker might each execute an RPC operate that calls considered one of these API capabilities and management the arguments to that API operate. These capabilities are:

Assessing this brief listing of API capabilities and analyzing code-flow paths from RPC handlers to those capabilities was a way more tractable drawback than manually reviewing every of the roughly 360 registered RPC handlers.

The bgtask command

By reviewing cross-references with IDA Professional, we have been in a position to determine the next call-chain from an RPC command handler to CreateProcess():

Based on Perforce’s documentation, the p4 bgtask command “allows a Helix Core superuser on the p4 command-line shopper to run instructions or packages remotely on the server within the background.” It’s thus not stunning that this kind of command would find yourself calling CreateProcess(), however for the reason that documentation states that this command can solely be run by a superuser, our solely hope of discovering a safety vulnerability right here was if there was a bug within the authentication element or in how the RPC arguments have been getting parsed.

To start our evaluation of p4 bgtask, we used the {custom} Perforce shopper that we wrote to see how the server would reply if we tried remotely calling bgtask with none authentication. To our shock, the server didn’t return any errors. In reality, the server ran the command line that we despatched to it, and this little one course of ran as LocalSystem.

Upon additional investigation, that is by design, with the handbook noting to “Run p4 defend instantly after putting in Helix Server for the primary time. Earlier than the primary name to p4 defend, each Helix Server consumer is a superuser and thus can entry and alter something within the depot”. On this context, “each Helix Server consumer” additionally contains unauthenticated nameless distant customers.

If an administrator doesn’t manually carry out these post-installation steps, this lacking authentication for a vital operate permits unauthenticated distant attackers to run arbitrary command traces (together with PowerShell command traces with script blocks) as LocalSystem when Perforce Server is put in with its default configuration.

This vulnerability is now recognized as CVE-2023-45849 and it has a CVSS rating of 10.0.

The rmt-Shutdown RPC handler

When a consumer (or attacker) makes use of the p4 bgtask mentioned above with the usual Perforce Shopper, the shopper sends the RPC command identify user-bgtask to the server to execute that command. Nonetheless, some RPC command names which might be accepted by the server don’t have a corresponding Perforce Shopper command; a type of RPC command names is rmt-Shutdown.

Though it’s not potential to ship the rmt-Shutdown RPC command with the usual Perforce Shopper (nor the Perforce Admin Instrument), and it doesn’t seem like documented on Perforce’s web site, we have been in a position to ship the command with our {custom} Perforce shopper. We discovered that the handler for rmt-Shutdown requires a username of distant however doesn’t require any authentication credentials for that username. When the Perforce Server receives this command, it terminates the Perforce Server course of, thereby permitting unauthenticated distant attackers to carry out DoS assaults towards the server.

This vulnerability is now recognized as CVE-2023-35767 and it has a CVSS rating of seven.5.

The rmt-UpdtFovrCommit RPC handler

Much like the rmt-Shutdown RPC command identify, the RPC command identify rmt-UpdtFovrCommit (which is probably going brief for “distant replace failover commit”) can’t be despatched by way of the usual Perforce Shopper nor Admin Instrument and doesn’t seem like documented on Perforce’s web site however may be despatched with a {custom} Perforce shopper. This RPC operate piqued our curiosity after we first examined for registered RPC handlers, since when our custom-built scanner despatched a rmt-UpdtFovrCommit RPC command as an nameless consumer and with none command arguments to the Perforce Server, the Perforce Server course of crashed.

We will see the explanation for the crash within the decompiled rmt-UpdtFovrCommit handler code under:

As beforehand mentioned, RPC messages despatched from the shopper to the server comprise the RPC operate identify and may optionally comprise RPC operate arguments. Within the code above, StrDict::GetVar() is used to get the shopper’s RPC operate arguments from the pRpc object. If the given argument identify was not supplied within the shopper’s RPC message, then StrDict::GetVar() returns zero. Within the first line above, StrDict::GetVar() is used to get the worth of the consumer RPC operate argument. Nonetheless, if the consumer (or attacker) doesn’t specify a price for consumer of their RPC message then pStrPtrUser will get set to zero. Within the final line above, we see pStrPtrUser handed because the second argument to StrOps::PackOctet(), (the supply code for which is accessible in the shopper supply code mentioned above):

As may be seen within the code above, when StrOps::PackOctet() is named with zero as the worth for s, StrBuf::Append() will get referred to as with zero as the worth for t. This leads to StrBuf::Append() making an attempt to dereference the size subject of t, the place the buffer subject of t is at offset 0 relative to the start of the t object and the size subject of t is at offset 8 relative to the start of the t object (since char *buffer is 64-bits). When the worth of t is 0, dereferencing size results in studying from digital reminiscence handle 0x0000000000000008, which ends up in a read-access violation or segmentation fault. We discovered that most of these exceptions should not dealt with gracefully by the server and that such read-exceptions trigger your complete server course of to crash and never restart. This DoS assault is exploitable by distant unauthenticated attackers.

This vulnerability is now recognized as CVE-2023-45319 and it has a CVSS rating of seven.5.

Coordinated disclosure

Microsoft reported these 4 safety vulnerabilities to the seller Perforce on the finish of August 2023. Instantly afterwards, on September 1, Perforce acknowledged these 4 vulnerabilities and commenced work to research and remediate them. All through September and October, Perforce communicated standing updates to Microsoft on implementing fixes and placing these fixes by their QA processes. Perforce reserved CVE IDs on October 24, 2023, shared these IDs with Microsoft on October 25, 2023, and knowledgeable Microsoft at the moment that the patches can be revealed by mid-November 2023. On November 7, 2023, Perforce revealed Perforce Server model 2023.1/2513900, which mitigates these 4 vulnerabilities.

Microsoft want to thank Perforce for his or her professionalism and for his or her speedy response in addressing these safety vulnerabilities. Microsoft is grateful for this partnership and for Perforce’s dedication to safety.

Mitigation and safety steerage

Microsoft just isn’t conscious of any adversaries exploiting these vulnerabilities, however mitigations ought to be utilized by all Perforce Server clients as quickly as potential.

Threat detection

Lengthen vulnerability and threat detection past the firewall with platforms like Microsoft Defender Exterior Assault Floor Administration. Clients can determine internet-exposed infrastructure operating Perforce Server of their stock and use the insights tile below the Assault Floor Abstract dashboard to floor property susceptible to CVE-2023-5759, CVE-2023-45849, CVE-2023-35767, and CVE-2023-45319.

What to do now when you’re affected

Replace to model 2023.1/2513900 instantly, accessible right here: https://www.perforce.com/downloads/helix-core-p4d.

Protection-in-depth

Along with following Perforce’s steerage on “Securing the server”, Microsoft recommends adhering to the next defense-in-depth ways to attenuate the danger of exploitation of those or different Perforce Server vulnerabilities.

• Usually monitor for and apply patches for third-party software program.
• Use a VPN and/or an IP allow-list to restrict who can talk along with your Perforce Server.
• Difficulty TLS certificates to legit Perforce customers and use a TLS termination proxy in entrance of Perforce Server to validate shopper’s TLS certificates earlier than permitting them to hook up with Perforce Server.
• Log all entry to your Perforce Server, each by way of your community home equipment and by way of Perforce Server itself.
• Configure alerting to inform IT directors and your safety staff if the Perforce Server course of crashes.
• Use community segmentation to make sure that in case your Perforce Server is compromised, an attacker’s capability to pivot in your community is proscribed.

Appendix

Risk intelligence reviews

Microsoft clients can use the next reviews in Microsoft merchandise to get probably the most up-to-date details about the risk actor, malicious exercise, and strategies mentioned on this weblog. These reviews present intelligence, safety data, and advisable actions to forestall, mitigate, or reply to related threats present in buyer environments.

Microsoft Defender Risk Intelligence

Microsoft 365 Defender Risk analytics 

Jason Geffner

Microsoft Risk Intelligence Neighborhood

References

Acknowledgments

Microsoft want to acknowledge https://www.keysight.com/blogs/tech/nwvs/2022/06/08/a-sneak-peek-into-the-protocol-behind-perforce for earlier work achieved in analyzing Perforce’s RPC protocol.

Be taught extra

For the newest safety analysis from the Microsoft Risk Intelligence neighborhood, try the Microsoft Risk Intelligence Weblog: https://aka.ms/threatintelblog.

To get notified about new publications and to affix discussions on social media, comply with us on Twitter at https://twitter.com/MsftSecIntel.

To listen to tales and insights from the Microsoft Risk Intelligence neighborhood in regards to the ever-evolving risk panorama, take heed to the Microsoft Risk Intelligence podcast: https://thecyberwire.com/podcasts/microsoft-threat-intelligence.