Hosting ASP.NET Core behind https in Google Kubernetes Engine

Side-note: this may be one of the clumsiest titles I've ever written for a blog post. But it does what it says on the tin. Oh, and the space after "ASP" in "ASP .NET Core" everywhere it to avoid auto-linking. While I could use a different dot or a zero-width non-breaking space to avoid it, I'm not sure I trust WordPress to do the right thing with those"

Background

Over the past few weeks, I've moved nodatime.org, csharpindepth.com and jonskeet.uk over to Google Kubernetes Engine. (They all used to be hosted on Azure.)

I've done this for a few reasons:

• As my job is primarily making .NET developers more productive on Google Cloud Platform, it feels natural to run my own code there. I want to see where there are friction points, so I can help fix them.
• I wanted more redundancy, particularly for nodatime.org; Kubernetes felt a simple way of managing that at a reasonable cost
• HTTPS certificate management (via Let's Encrypt) has been a bit painful for me on Azure; I could have automated more, but that would have taken extra time I don't have. (It may also have improved since I last looked.)

The first of these is the most important, by a long way. But the HTTPS management part - and then the knock-on effects - is what I'm looking at in this blog post.

Basic hosting

Hosting an ASP .NET Core application in Google Kubernetes Engine (GKE from now on) is really simple, at least once you've understood the Kubernetes concepts. I have:

• A single cluster
• A single ingress backed by a Google Cloud Load Balancer for all my domains, with a reserved static IP address (set up in DNS)
• A service per application (nodatime.org example)
• A deployment per application (nodatime.org example)

In each case, the ASP .NET Core application is built with a vanilla Dockerfile which would not look unusual to anyone who's hosted ASP .NET Core in Docker anywhere else.

I happen to use Google Cloud Build to build the Docker images, and Google Container Registry to host the images, but neither of those are required. (For csharpindepth.com and jonskeet.uk there are simple triggers in Google Cloud Build to build and deploy on GitHub pushes; for nodatime.org it's a bit more complicated as the documentation build currently has some Windows dependencies. I have a machine at home that polls GitHub every half hour, and pushes the built application to Google Cloud Build for packaging when necessary.)

So, that gets HTTP hosting sorted. I dare say there are some aspects I've not configured as well as I could have done, but it was reasonably straightforward to get going.

HTTPS with Google-managed certificates

With HTTP working, it's time to shoot for HTTPS. It's important to note that the apps I'm talking about are all hobby projects, not commercial ones - I'm already paying for hosting, so I don't want to have to pay for SSL certificates as well. Enter Let's Encrypt, of course.

A while ago I used Let's Encrypt to set up HTTPS on Azure, and while it was free and I didn't have to write any code, it wasn't exactly painless. I followed two guides at the same time, because neither of them exactly matched the Azure portal I was looking at. There were lots of bits of information to grab from various different bits of the portal, and it took a couple of attempts to get right" but I got there. I also set up a web job to renew the certificates, but didn't go through the hoops required to run those web jobs periodically. (There were instructions, but it looked like they'd take a while to work through compared with just manually running the web job every couple of months or so. I decided to take the pragmatic approach, knowing that I was expecting to move to GKE anyway. If Azure had been the expected permanent home for the apps, I'd have gone through the steps and I'm sure they'd have worked fine.) I don't know which guide I worked through at the time, but if I were starting today I'd probably try Scott Hanselman's guide.

So, what can I do on Google Cloud Platform instead? I decided to terminate the SSL connection at the load balancer, using Google-managed certificates. To be really clear, these are currently in beta, but have worked well for me so far. Terminating the SSL connection at the load balancer means that the load balancer forwards the request to the Kubernetes service as an HTTP request, not HTTPS. The ASP .NET Core app itself only exposes an HTTP port, so it doesn't need to know any details of certificates.

The steps to achieve this are simple, assuming you have the Google Cloud SDK (gcloud) installed already:

• Create the certificate, e.g.
  gcloud beta compute ssl-certificates create nodatime-org --domains nodatime.org
• Attach the certificate to the load balancer, via the Kubernetes ingress in my case, with an annotation in the ingress metadata:
  ingress.gcp.kubernetes.io/pre-shared-cert: "nodatime-org"
• Apply the modifications to the ingress:
  kubectl apply -f ingress.yaml
• Wait for the certificate to become valid (the provisioning procedure takes a little while, and I've seen some interesting errors while that's taking place)
• Enjoy HTTPS support, with auto-renewing certificates!

There are only two downsides to this that I've experienced so far:

• Currently each certificate can only be associated with a single domain. For example, I have different certificates for nodatime.org, http://www.nodatime.org and test.nodatime.org. (More about the last of these later.) This is a minor annoyance, but the ingress supports multiple pre-shared certificates, so it has no practical implications for me.
• I had to accept some downtime on HTTPS when transferring from Azure to GKE, while the certificate was provisioning after I'd transferred the DNS entry. This was a one-time issue of course, and one that wouldn't affect most users.

Beyond the basics

At this point I had working HTTPS URLs - but any visitor using HTTP would stay that way. (At the time of writing this is still true for csharpindepth.com and jonskeet.uk.) Obviously I'd like to encourage secure browsing, so I'd like to use the two pieces of functionality provided by ASP .NET Core:

• Redirection of HTTP requests via app.UseHttpsRedirection()
• HSTS support via app.UseHsts()

I should note here that the Microsoft documentation was fabulously useful throughout. It didn't quite get me all the way, but it was really close.

Now, I could have just added those calls into the code and deployed straight to production. Local testing would have worked - it would have redirected from localhost:5000 on HTTP to localhost:5001 on HTTPS with no problems. It would also have failed massively for reasons we'll look at in a minute. Just for a change, I happened to do the right thing"

For hosting changes, always use a test deployment first

In Azure, I had a separate AppService I could deploy to, called nodatimetest. It didn't have a fancy URL, but it worked okay. That's where I tested Azure-specific changes before deploying to the real AppService. Unfortunately, it wouldn't have helped in this situation, as it didn't have a certificate.

Fortunately, creating a new service in Kubernetes, adding it to the ingress, and creating a managed certificate is so easy that I did do this for the new hosting - and I'm so glad I did so. I use a small script to publish the local ASP .NET Core build to Google Cloud Build which does the Docker packaging, pushes it to Google Container Registry and updates the Kubernetes deployment. As part of that script, I add a small text file containing the current timestamp so I can check that I'm really looking at the deployment I expect. It takes just under two minutes to build, push, package, deploy - not a tight loop you'd want for every day development, but pretty good for the kind of change that can't be tested locally.

So, I made the changes to use HTTPS redirection and HSTS, deployed, and" there was no obvious change.

Issue 1: No HTTPS port to redirect to

Remember how the ASP .NET Core app in Kubernetes is only listening on HTTP? That means it doesn't know which port to redirect users to for HTTPS. Oops. While I guess it would be reasonable to guess 443 if it didn't know any better, the default of "don't redirect if you haven't been told a port" means that your application doesn't stop working if you get things wrong - it just doesn't redirect.

This is easily fixed in ConfigureServices:

services.AddHttpsRedirection(options => options.HttpsPort = 443);

" but I've added conditional code so it doesn't do that in development environment, as otherwise it would try to redirect from localhost:5000 to localhost:443, which wouldn't work. This is a bit hacky at the moment, which is a common theme - I want to clean up all the configuration at some point quite soon (moving things into appsettings.json as far as possible) but it's just about hanging together for now.

So, make the change, redeploy to test, and" observed infinite redirection in the browser. What?

Issue 2: Forwarding proxied headers

Remember again how the ASP .NET Core app is only listening on HTTP? We want it to behave differently depending on whether the end user made a request to the load balancer on HTTP or HTTPS. That means using headers forwarded from the proxy (in our case the load balancer) to determine the original request scheme. Fortunately, again there's documentation on hand for this.

There are two parts to configuring this:

• Configuring the ForwardedHeadersOptions in ConfigureServices
• Calling app.UseForwardedHeaders() in Configure

(At least, that's the way that's documented. I'm sure there are myriad alternatives, but my experience level of ASP .NET Core is such that I'm still in "follow the examples verbatim, changing as little as possible" at the moment.)

I won't go into the gory details of exactly how many times I messed up the forwarded headers options, but I will say:

• The example which just changes options.ForwardedHeaders is probably fine if your proxy server is local to the application, but otherwise you will need do to extra work
• The troubleshooting part of the documentation is spectacularly useful
• There are warnings logged if you get things wrong, and those logs will help you - but they're at a debug log level, so you may need to update your logging settings. (I only realized this after I'd fixed the problem, partly thanks to Twitter.)

Lesson to learn: when debugging a problem, turn on debugging logs. Who'd have thought?

Configuring this properly is an area where you really need to understand your deployment and how a request reaches you. In my case, the steps are:

• The user's HTTPS request is terminated by the load balancer
• The load balancer makes a request to the Kubernetes service
• The Kubernetes service makes a request to the application running on one of the suitable nodes

This leads to a relatively complex configuration, as there are two networks to trust (Google Cloud load balancers, and my internal Kubernetes network) and we need to allow two "hops" of proxying. So my configuration code looks like this:

services.Configure<ForwardedHeadersOptions>(options =>{ options.KnownNetworks.Clear(); // Google Cloud Platform load balancers options.KnownNetworks.Add(new IPNetwork(IPAddress.Parse("130.211.0.0"), 22)); options.KnownNetworks.Add(new IPNetwork(IPAddress.Parse("35.191.0.0"), 16)); // GKE service which proxies the request as well. options.KnownNetworks.Add(new IPNetwork(IPAddress.Parse("10.0.0.0"), 8)); options.ForwardedHeaders = ForwardedHeaders.XForwardedFor | ForwardedHeaders.XForwardedProto; options.ForwardLimit = 2;});

(The call to KnownNetworks.Clear() probably isn't necessary. The default is to include the loopback, which is safe enough to leave in the list.)

Okay, deploy that to the test environment. Everything will work now, right? Well, sort of"

Issue 3: make sure health checks are healthy!

As it happens, when I'd finished fixing issue 2, I needed to help at a birthday party for a family we're friends with. Still, I went to the party happily knowing everything was fine.

I then came home and found the test deployment was broken. Really broken. "502 Bad Gateway" broken. For both HTTP and HTTPS. This is not good.

I tried adding more logging, but it looked like none of my requests were getting through to the application. I could see in the logs (thank you, Stackdriver!) that requests were being made, always to just "/" on HTTP. They were all being redirected to HTTPS via a 307, as I'd expect.

This confused me for a while. I honestly can't remember what gave me the lightbulb moment of "Ah, these are load balancer health checks, and it thinks they're failing!" but I checked with the load balancer in the Google Cloud Console and sure enough, I had multiple working backends, and one broken one - my test backend. The reason I hadn't seen this before was that I'd only checked the test deployment for a few minutes - not long enough for the load balancer to deem the backend unhealthy.

I was stuck at this point for a little while. I considered reconfiguring the load balancer to make the health check over HTTPS, but I don't think that could work as the app isn't serving HTTPS itself - I'd need to persuade it to make the request as if it were a user-generated HTTPS request, with appropriate X-Forwarded-Proto etc headers. However, I saw that I could change which URL the load balancer would check. So how about we add a /healthz URL that would be served directly without being redirected? (The "z" at the end is a bit of Googler heritage. Just /health would be fine too, of course.)

I started thinking about adding custom inline middleware to do this, but fortunately didn't get too far before realizing that ASP .NET Core provides health checking already" so all I needed to do was add the health check middleware before the HTTPS redirection middleware, and all would be well.

So in ConfigureServices, I added a no-op health check service:

services.AddHealthChecks();

And in Configure I added the middleware at an appropriate spot:

app.UseHealthChecks("/healthz");

After reconfiguring the health check on the load balancer, I could see /healthz requests coming in and receiving 200 (OK) responses" and the load balancer was then happy to use the backend again. Hooray!

After giving the test service long enough to fail, I deployed to production, changed the load balancer health check URL, and all was well. I did the two parts of this quickly enough so that it never failed - a safer approach would have been to add the health check handler but without the HTTPS redirection first, deploy that, change the health check URL, then turn on HTTPS.

But the end result is, all is working! Hooray!

Conclusion

Moving the service in the first place has been a long process, mostly due to a lack of time to spend on it, but the HTTPS redirection has been its own interesting bit of simultaneous pleasure and frustration. I've learned a number of lessons along the way:

• The combination of Google Kubernetes Engine, Google Cloud Load Balancing, Google Cloud Build and Google Container registry is pretty sweet.
• Managed SSL certificates are wonderfully easy to use, even if there is a bit of a worrying delay while provisioning
• It's really, really important to be able to test deployment changes (such as HTTPS redirection) in an environment which is very similar to production, but which no-one is depending on. (Obviously if you have a site which few people care about anyway, there's less risk. But as it's easy to set up a test deployment on GKE, why not do it anyway?)
• HTTPS redirection caused me three headaches, all predictable:
  • ASP .NET Core needs to know the HTTPS port to redirect to.
  • You need to configure forwarded headers really carefully, and know your deployment model thoroughly .
  • Be aware of health checks! Make sure you leave a test deployment "live" for long enough for the health checks to mark it as unhealthy if you've done something wrong, before you deploy to production.
• When debugging, turn on debug logging. Sounds obvious in retrospect, doesn't it? (Don't start trying to copy middleware source code into your own application so you can add logging, rather than using the logging already there")

I also have some future work to do:

• There's a particular URL (http://nodatime.org/tzdb/latest.txt) which is polled by applications in order to spot time zone information changes. That's the bulk of the traffic to the site. It currently redirects to HTTPS along with everything else, which leads to the total traffic being nearly double what it was before, for no real benefit. I've encouraged app authors to use HTTPS instead, but I've also filed a feature request against myself to consider serving that particular URL without the redirect. It looks like that's non-trivial though.
• I have a bunch of hard-coded information which should really be in appsettings.json. I want to move all of that, but I need to learn more about the best way of doing it first.

All in all, this has been a very positive experience - I hope the details above are useful to anyone else hosting ASP .NET Core apps in Google Kubernetes Engine.