On-Prem and Kubernetes: A delicate relationship

In cloud security, context is everything.

In the previous two installments of our Customers Care Chronicles, we wrote about how a security vendor needs to be a true business partner and the potential headaches when migrating tools in the cloud. In this installment, we tackle another non-security concept that happens to be crucial for security: environment

The pace and speed of innovation in the cloud is unprecedented – and businesses are embracing it as fast as possible. But all transitions have their challenges, especially at the enterprise level. Switching over from on-premises (on-prem) to a more flexible infrastructure can be complex, lengthy, and sometimes even undesirable. This is why many businesses opt for hybrid environments, retaining some on-prem services, while also enjoying the benefits of Kubernetes. This is good for business, but definitely adds security challenges.

From base architecture to deployment

Our customer was a government organization with a robust infrastructure and (as most government organizations) a number of individual security requirements. We knew from the beginning that this would mean a lot of out-of-the box thinking and customization.

The agreement we were a part of included provisioning of physical hardware (servers), installation of the Kubernetes cluster over those servers and then installation of the Sysdig backend on the newly created Kubernetes cluster.

We knew that the environment we had to create was going to take a lot of effort and would be complicated. This was something which had never been done before.

With a project of this magnitude, we knew we had to focus on the:

  • Deployment design: Creating the base level architecture for an infrastructure of this complexity is its own project.
  • Sysdig backend: We had to make sure that our product backend could be safely installed over the top of the customer’s environment.
  • Infrastructure complexity: A government institution needs to be functional and secure — this meant that opening even a single port took days.
  • Air-gapped environment: We had to get all our images into an internal registry and make sure that it was accessible from the customer’s clusters.

Once we understood the assignment, we immediately formed a Sysdig task force — including our infra and support teams — to make sure we had all our ducks in a row.

A Customer Success Engineer on the road

The project needed a Managed Customer Success Engineer to be based at the customer’s site. I have been there since we started the deployment, personally overseeing everything from design to implementation, and iteration.

We agreed to provision the customer with five servers. We had to install them across two data centers. In addition, the customer asked us to make the Kubernetes cluster into a stretch cluster spanning across both data centers. This meant we had to create the base level architecture and design for this deployment. Our primary goals were high availability and disaster recovery. 

Once the Kubernetes cluster was ready, we installed the Sysdig On-Premises backend over it and connected the customer’s clusters by installing the Sysdig agents on them.

The whole deployment process took nearly three months and it was done remotely with me facilitating onsite. It involved multiple teams working in coordination, continuous communication with the customer’s security and executive team, and lots of iterating.

The customer has different vendors for security, vulnerability management, and application development. This also included their identity teams who managed the SSO/PAM access to the applications, and the SOC team who are responsible for managing the security incidents which Sysdig would forward to their SIEM. Getting our solution onboarded meant collaborating with all these teams and stakeholders.


In this story, we provided Sysdig’s On-Premises services. On-premises users install and manage the Sysdig backend components as they see fit. This could be in a data center, or in an enterprise’s cloud-provider space, such as Azure, AWS or GKE.

Conclusion

It’s clear that successfully navigating the complexities of cloud security requires more than just technical expertise—it demands meticulous planning and context awareness. 

From designing a robust and flexible infrastructure to overcoming unique challenges, the deployment process involved designing architecture, ensuring Sysdig on-premises services seamlessly integrated with the customer’s complex infrastructure, and a lot of DevSecOps collaboration. The three-month project underscored the value of ongoing communication and teamwork.

The effectiveness of any solution is deeply intertwined with the environment it’s designed to protect. For organizations with complex and high-stakes requirements, having a tailored and secure setup is crucial. Staying aware of the technical, environmental and business context is pivotal when ensuring that your infrastructure is ready to protect your data — and ultimately your customers.

Sulav is a Sr. Customer Solutions Engineer at Sysdig. He manages the India region and is responsible for the customers’ journey throughout their contract with Sysdig — which includes onboarding, solution designing, implementation, technology adoption and upsell.