Founded in 1960, Holder Construction Company is a leading provider of services for the entire planning, preconstruction, and construction process. Priding itself in a collaborative relationship with clients and design partners, the company boasts a repeat client business rate of more than 80% and a position as one of the nation’s top 100 contractors.
Holder had a core, mission-critical customer-facing web application and related systems that needed to be migrated to Azure. This application relied on a set of Kubernetes APIs which were hosted with a third-party provider. The company needed to ensure that the system would be highly available, minimizing the chance of downtime in the event of failures in an Azure region or data center. It was important that the latency of the solution remain as low as possible for users that were evenly split between the East and West coasts. Wintellect was requested to help the Holder team modernize the application and create an architecture and solution that could meet these requirements.
To ensure that Holder was well-positioned to understand the changes and be able to take ownership of the completed system, Wintellect began with training on Azure fundamentals. This positioned the team to understand the design decisions being made and allowed Holder to actively participate and make informed decisions throughout the process. Following that, the overall system and application was assessed to identify any potential risks or challenges which might limit its ability to support automatic failover. Finally, the deployment architecture was enhanced to support multiple Azure regions, using Azure App Services in a across two paired regions, East US and West US. Azure Front Door was deployed to provide automatically failover and low-latency routing of traffic from users to the nearest App Service instance. Additional supporting virtual machines were re-deployed using availability zones to create a high-availability solution. The Microsoft Cloud Adoption Framework (CAF) was used for the overall methodology.
Azure Fundamentals Guidance
As part of the educational portion of the solution, an overview of Azure Fundamentals was provided. The one-day guided tour of Azure focuses on ensuring that technical stakeholders involved in the development, operations, or infrastructure practices have an understanding of the basics of Azure and building solutions which are cloud-native. This includes coverage of Azure infrastructure and platform-as-a-service solutions, resource and resource group management, monitoring and sizing, cost management, and maximizing uptime and resiliency.
Azure Web Application Assessment
Determining the current state of the existing system was a fundamental step in the process of providing a highly available solution. By identifying areas of concern and areas that required improvement, a point of reference was established for maximizing the availability of the application. The assessment involved reviewing the application design, deployment strategy, approach to configuration, and interactions with other resources. From the assessment, several areas of the application were identified to be optimized or improved, including:
- Adding health check endpoints to assist with uptime monitoring and failover
- Properly sizing the services based on the monitored performance
- Utilizing retry policies and the circuit breaker pattern to mitigate the risks created by changes to the remotely hosted services
- Configuring an appropriate auto-scaling strategy
- Deploying the application into App Services in paired regions to protect against regional outages
- Setup and configure Azure Front Door to ensure continuous uptime and minimize geographic latency
- Reviewing the deployment automation strategy to eliminate the human factor
- Utilizing Azure Application Insights to improve the ability to monitor and debug the application at scale
- Resizing virtual machines and deploying sets of machines to availability zones to increase the resiliency to datacenter failures
These design changes improved the ability of the application and its related services to remain highly available and scalable on Azure.
Implementing Highly Available and Resilient Optimizations
The initial assessment identified several areas to optimized, which set the stage for the implementation phase. Wintellect not only implemented optimizations to the system, but also provided guidance and hands-on assistance for the Holder team through the process to ensure that they were able to understand and manage the final environment.
Wintellect configured two App Service Plans in paired regions, East US and West US (see Figure 1). Paired regions are two Azure regions separated by at least 300 miles, located within the same geography, and used for data replication and as a maintenance boundary. These pairs guarantee that planned maintenance will only affect one member at a time; additionally, they guarantee that in the event of a massive outage, one member will always have priority during a recovery. For georeplicated storage solutions, the data is guaranteed to be durable between the regions, with three copies preserved in each region automatically. From the end-user perspective, the application performance is improved by enabling the data to be served from a location which is closer to the end-user.
In order to take advantage of this design, Azure Front Door was implemented. This allows the two regional deployments to be treated as a single logical entry. When a request arrives, it is sent to the web application in the region with the lowest latency relative to the incoming request. If that region is unresponsive or unhealthy, traffic is automatically redirected to the next-closest available instance to ensure availability. In addition, Front Door includes a CDN that supports dynamic content compression at the edge, creating smaller, faster responses to connected clients. Because Front Door supports TLS termination at the edge and supports connection reuse to the backend, the overall load on the underlying App Services is reduced, enabling more efficient scaling. As an added feature, Front Door can automatically create and manage certificates, eliminating the need to purchase or configure certificates to support HTTPS connections. Azure Front Door natively supports HTTP, HTTPS, and the newer HTTP/2 protocol (which is enabled by default).
Holder also relied on multiple virtual machines responsible for hosting supporting services. In addition to working with holder to right-size the virtual machines, Wintellect helped Holder to define an architecture for the virtual machines which would support high availability using availability zones. Availability zones are unique physical locations within an Azure region with independent power, cooling, and networking and with very low-latency connections. Within a single region, availability zones provides high availability against a datacenter outage. Wintellect assisted Holder to review the services on the virtual machines, providing a design for a highly available, cross-region architecture (see Figure 2). For stateless servers that required support for automatically scaling out, virtual machine scale sets were utilized to simplify the deployments and enable scale on demand.
To properly route traffic to the virtual machines over HTTPS across availability zones, Wintellect recommended Azure Application Gateway v2. Azure Application Gateway provides a layer 7 load balancer with zone redundancy and makes routing decisions based on attributes of the HTTP request, such as the URL path or host headers. Combined with Azure Front Door, this enables support for connection training, cluster-level session affinity, and 100% TLS/SSL offload. Additionally, the Azure Application Gateway supports a Web Application Firewall (WAF v2), providing centralized protection for web applications from common exploits and vulnerabilities.
Figure 2 - Azure Virtual Machine Architecture
Holder needed a partner with proven experience at implementing highly available and resilient systems. Expertise in implementation was only half of the equation, as Holder also required experience and training to gain the essential skills required to maintain and continue development of a highly available application environment.
- Wintellect has deep experience in developing highly available systems, utilizing patterns that promote resiliency and stability.
- Wintellect is a Microsoft Gold Cloud Platform, Application Development, Data Platform, and Data Analytics partner.
The system was redeployed as part of a new highly available architecture, with a detailed understanding of the deployed environment. Testing and validation were performed during the ongoing development phase as well as prior to system sign-off to validate that the system met the specified performance and high availability requirements. A combined implementation and training engagement ensured that the team was prepared for adopting a resilient cloud strategy, with hands-on understanding of Azure fundamentals. With implementation assistance from Wintellect, Holder was able to create services which were more resilient to failure, supported users across the country at lower latency, and were highly available both within Azure data centers and across regions.