JumpCloud Security Update: Okta, LastPass and CircleCI breaches

[alexaemerson]

We have been actively evaluating the impact of recent breaches at Okta, LastPass, and CircleCI on our operations and want to give our community an update on what we are doing to assess any potential impact to JumpCloud or our customers.

LastPass

We do not use LastPass and introduced our own Password Manager last fall, which employs a decentralized architecture, protecting users from the kind of breach that LastPass experienced. You can learn more about the differences here. We encourage anyone using LastPass to assess the security architecture of cloud stored password vaults and understand the risks associated with that approach.

Okta

We use JumpCloud SSO internally and do not use Okta; we do offer an integration with Okta via the Okta Integration Network. The integration does not present any immediate risk to JumpCloud, however many of our customers leverage this integration and are exposed to risks from the Okta compromise and should take appropriate action. Okta has recently announced three breaches this past year and customers should continue to look for updates on the immediate, and yet unknown, risks and impacts from these breaches.

CircleCI

We are a CircleCI customer and this breach has potential to impact any CircleCI customer, including JumpCloud and our end users. Within minutes of being notified of the CircleCI breach on January 4, our security team initiated cyber incident response protocols to investigate any potential impact. This included an immediate effort to rotate credentials directly supplied to CircleCI. We are treating this as an ongoing, active incident until we have sufficient data to provide an all-clear. As we get more information from CircleCI on its own investigation, that will help us evaluate whether there is any impact to JumpCloud or our customers. If we do find evidence of activity that puts our customers at risk, we will be promptly communicating that.

While we are not aware of any direct impact to JumpCloud today, we strongly encourage and advocate that all customers enable multi-factor authentication to all IT resources including devices, email, applications, and VPNs. This is one of the most impactful security steps your organization can take. Please let our support team know if we can help you implement this immediately.

The pace of breaches is continually accelerating as criminals, nation state adversaries and other threat actors look to disrupt business and profit from their nefarious activities. This requires more diligence and investment from the software vendor community, as well as every organization. As a JumpCloud customer, you have access to our Open Directory Platform, which provides a fundamentally different approach to securing access and authentication. We see identities as the new perimeter, and believe that organizations will benefit from a single platform that provides many layers of protection for their users - including SSO, Password Management, conditional access, enforced disk encryption, automated patch management, MFA, and more. We encourage everyone to be sure they are taking full advantage of the tools available to you to improve your security stance.

As we enter a New Year, our continued goal is to operate JumpCloud using modern security tools and following best in class security practices. As many of you know, this is a never ending journey with new types of attacks and vulnerabilities discovered almost daily. We recommend following industry guidance and tracking vulnerable applications and third parties in your environment. To help you learn from our journey, we will be hosting a webinar in the coming weeks on the steps we have taken and our plans to secure our environment and protect our customers in 2023. Please watch for that invite in the coming weeks.

We will continue to provide additional updates as we have new information. If you have any questions, please contact security@jumpcloud.com or your customer success manager.

Sincerely,

Robert Phan, CISO

[alexaemerson]

Hello JumpCloud Community – 

We are going to update this thread regularly to ensure you know where we are with the CircleCI incident investigation.

In addition, Bob Phan, our CISO will be joining the IT Hour tomorrow at 11.30 AM ET to shed additional light on the situation.

Details of how to join this call can be found here.

[alexaemerson]

As of today, we completed mitigating the primary attack vectors we have identified that could have been opened by CircleCI’s breach, and we continue to see no indicators of compromise.  

CircleCI posted additional information about the breach this afternoon. In this update they said:

On January 7, 2023, at 18:30 UTC, we began working with our partners at AWS to notify customers of potentially affected AWS tokens. We understand that those notifications were complete as of January 12, 2023, at 00:00 UTC.

It is encouraging that we did not receive this notification, and to date we have seen no evidence of malicious activity in our AWS environments. We will continue to actively monitor our AWS environments.  

CircleCI also provided specifics in its post on the IP addresses, VPNs, data centers, domains, files and signatures used by the threat actor. We have updated our endpoint detection and response (EDR) with this information.

As previously stated, we will treat CircleCI’s breach as an ongoing, active incident until we have sufficient data to provide an all-clear. The additional information from CircleCI today has significantly helped us search for – and prevent — impacts to JumpCloud or our customers. 

If we do find evidence of activity that puts our customers at risk, we will promptly communicate with you.

We will provide our next progress update on Tuesday, January 17th. If there is relevant information to share prior to that, we will post it here.

Robert Phan

JumpCloud CISO

[alexaemerson]

We continued our investigation over the weekend and are happy to report that to date, we have found no indicators of compromise. As reported on Friday, we have mitigated all the primary attack vectors, and will continue to treat CircleCI’s breach as an ongoing active incident until we have sufficient data to provide an all-clear. 

If we do find evidence of activity that puts our customers at risk, we will promptly communicate with you.

We will provide our next progress update tomorrow. If there is relevant information to share prior to that, we will post it here.

Robert Phan

JumpCloud CISO

[alexaemerson]

We continue our investigation and are happy to report that to date, we have found no indicators of compromise. As previously reported, we have mitigated all the primary attack vectors, and will continue to treat CircleCI’s breach as an ongoing active incident until we have sufficient data to provide an all-clear. 

If we do find evidence of activity that puts our customers at risk, we will promptly communicate with you.

We will provide our next progress update tomorrow. If there is relevant information to share prior to that, we will post it here.

Robert Phan

JumpCloud CISO

[alexaemerson]

Status quo today: we continue our investigation and are happy to report that to date, we have found no indicators of compromise. As previously reported, we have mitigated all the primary attack vectors, and will continue to treat CircleCI’s breach as an ongoing active incident until we have sufficient data to provide an all-clear. 

If we do find evidence of activity that puts our customers at risk, we will promptly communicate with you.

We will provide our next progress update tomorrow. If there is relevant information to share prior to that, we will post it here.

Robert Phan

JumpCloud CISO

[alexaemerson]

Happy Friday to you all. Another consistent update in that we still have found no indicators of compromise. As previously reported, we have mitigated all the primary attack vectors, and will continue to treat CircleCI’s breach as an ongoing active incident until we have sufficient data to conclude that this incident is contained. 

Folks who have been following the CircleCI breach have asked us how they might use JumpCloud to reduce the risk of session hijacking.  

While all SAML providers are susceptible to session hijacking, there are settings within the JumpCloud platform, as well as other security tools to help mitigate this risk, including Antivirus and EDR tools like CrowdStrike. Within JumpCloud we recommend the following steps to help reduce the risk of session hijacking:

1. Enable MFA for all users and all connections. You can learn more about JumpCloud’s MFA here.
2. Leverage session timeouts. Within JumpCloud the default timeout for the user portal is 8 hours, and can be manually adjusted to as short as 1 minute. The shorter that window is, the less time a bad actor has to hijack and execute an attack via the session.  You can learn more about configuring your session timeouts here.

As always our team is here to help and if you’d like to meet with one of our support team to help with these settings, we are happy to assist.

[alexaemerson]

Happy Tuesday all. No news to report today: we still have found no indicators of compromise. As previously reported, we have mitigated all the primary attack vectors, and will continue to treat CircleCI’s breach as an ongoing active incident until we have sufficient data to conclude that this incident is contained. 

In the interest of helping you all improve your own security defenses, I’ll restate what I shared on Friday in response to customers asking how JumpCloud can help reduce the risk of session hijacking.

While all SAML providers are susceptible to session hijacking, there are settings within the JumpCloud platform, as well as other security tools to help mitigate this risk, including Antivirus and EDR tools like CrowdStrike. Within JumpCloud we recommend the following steps to help reduce the risk of session hijacking:

1. Enable MFA for all users and all connections. You can learn more about JumpCloud’s MFA here.
2. Leverage session timeouts. Within JumpCloud the default timeout for the user portal is 8 hours, and can be manually adjusted to as short as 1 minute. The shorter that window is, the less time a bad actor has to hijack and execute an attack via the session.  You can learn more about configuring your session timeouts here.

As always our team is here to help and if you’d like to meet with one of our support team to help with these settings, we are happy to assist.

[alexaemerson]

CircleCI Breach Update

From: Robert Phan, JumpCloud CISO

Hello everyone. I am happy to report that as of today, we are de-escalating CircleCI’s breach from active incident status. I’d like to take this opportunity to provide more detail on what we have concluded from our investigation.

As previously reported, we were notified of CircleCI’s breach on January 4, 2023. Within minutes of our notification, we initiated cyber incident response protocols to investigate. We are a CircleCI customer and this breach has potential to impact any CircleCI customer, including JumpCloud and our end users. 

Like any cautious CISO, I am hesitant to share specifics of how we run an investigation, or tools and tactics we use to combat cyber threats. I do, however, want to balance that with providing our community visibility of how we think about CircleCI’s breach.

Our immediate actions, starting within minutes of notification, were focused on securing any openings that the hackers could access. Next it was to remediate any customer impact. In parallel we have been looking for evidence of threat actor activity in our systems. 

From the information provided by CircleCI, we are confident that all of its customers' tokens were exposed. We are concerned by the amount of time that the attackers had access to these tokens before CircleCI understood and communicated to its customers that there had been a significant breach. That resulted in a lengthy amount of exposure: per CircleCI’s communication, the attacker(s) exfiltrated credentials on December 22nd, and CircleCI informed customers of the breach on January 4th. That gave the attacker(s) a lengthy period of time to operate undetected with valid credentials.

The amount of time that attackers had is driving our conservative approach to this incident. We feel that JumpCloud—and every other CircleCI customer – needs to operate under the assumption that their code was accessed during the window that the attackers were able to operate undetected.

That will leave our customers with a few common questions. Let me address those:

Has any JumpCloud data been accessed?

Based on the type of attack, from the start we have executed security protocols that assume our code has been accessed. Although we have no conclusive evidence that allows us to definitively say our code was accessed, due to the nature of the attack and a short period of unusual behavior in GitHub during the impact window, we have executed security protocols that assume that is the case.

In our research, we have found no evidence of modifications and we have rotated all credentials, ensuring that any that might have been exposed are no longer usable. We have seen no evidence of misuse of credentials that could have been exposed this way.

Has any customer data been stolen?

During our investigation we found potential risk associated with 12 API keys. We immediately contacted potentially exposed customers and worked with them to rotate those keys. We saw no evidence that the keys were used from any origin other than the customer.

We have no other evidence of any customer data being exposed.

What does it mean to JumpCloud – and other CircleCI customers, if the attacker(s) have downloaded code?

While it is extremely concerning to us to think that the attacker(s) could have downloaded our code, we don't rely on security by obscurity. Our security posture is not reliant on the secrecy of our source code, and we are confident that we have been, and continue to operate effective product security and SDLC practices.

We have regularly scheduled 3rd party pentests of our software and will continue our 3rd party pentesting program in 2023. We continue to perform internal threat modeling, pentesting, vulnerability management, and SDLC.

Why are you confident that you can downgrade the incident now?

We have conducted a thorough incident response. This includes rotation of every credential provided to CircleCI, and every credential that could have been indirectly accessed by leveraging a credential provided to CircleCI.

We have thoroughly investigated all activity in our cloud production infrastructure and our internal systems and have found no indicators of compromise, unexplained changes, or anomalous behavior. As an output of that work, the additional enhancements to our security monitoring to perform this specific investigation will operate in perpetuity. Additionally we have asserted the authenticity of the binaries we distribute to our customers.

Lastly, to proceed with the utmost caution, we are pursuing 3rd party evaluations to discover anything that may have gone unnoticed by our security team. As always, if we learn of anything concerning, we will share with our community.

What does downgrading the incident mean?

We have closed the attack vectors we believe were opened as a result of CircleCI’s breach. After thorough investigation of all activity that occurred during the impact window, up through and including the time the last vector was closed, we have found nothing that indicates we have an attacker active in our systems. 

It does not mean that our work is over. One of our core values at JumpCloud is 1% better everyday and our security team strives to deliver that. Our Post Incident Review (PIR) process has identified areas of investment that will further reduce our risk from vendor supply chain breaches, as well as opportunities to mature our response processes. As a side note, we hope that this ongoing communication effort has been useful to our community, as it is one of the many areas we are striving to improve.

The de-escalation does mean that we will no longer be providing daily updates. Our security team will continue monitoring for any threat actor activity and will operate the new detection enhancements we created for this incident indefinitely.