We use Nginx in our hosting cluster where we have many tenants/vhosts. Although I'm not sure it was necessary to choose Nginx over Apache, we have been able to squeeze a lot of performance from our machines with it. The learning curve associated with the switch has caused us to make some rookie configuration mistakes.
Years back we experienced an issue where the content from the wrong vhost was being served up to the wrong domain. This was due to a misconfiguration resulting from our lack of understanding of the Nginx listen parameter in the server directives.
When you configure your server with multiple tenants, you create one or more new Nginx server blocks in the nginx.conf file for each endpoint or domain you'll be responding to. Inside that server block you define things like the hostname you're expecting for that server, the IP address and port to listen on, SSL certificates, the root directory, and much more. When an HTTP request comes in, Nginx will find the best server block match for the request and use its configuration to create the response.
For example, if I make an HTTP request over port 80 to www.exmaple.com and in my nginx.conf I have a server block that looks like the following:
The match on the port and server name will result in Nginx using this server block for the request and the content from the root path will be served up, as expected.
If you have many virtual hosts on your server, you'll have many of these server blocks. The problem arises when a request comes into your server that doesn't match a server block, for example if beta.example.com is also pointed at this server. When the request comes in, Nginx will try and find a server block match. When it can't find one, it will resort the the first server block in the list, commonly in alphabetical order. That's right - instead of just aborting the request, Nginx will just serve up whatever it finds first, meaning you'll get a response from some other vhost on the server. It is so eager to complete the request it will serve up anything!
There are two solutions to this problem:
- Put a server block at the top of the list that returns a 404 page or something, or simply return an HTTP status code of 403 (forbidden) or 444 (Nginx specific no response / abort).
- Specify one of your server block listeners as the default listener for when no match can be found. This is done by appending default_server to the listen directive.
We patched the issue on our server using option #1 but recently it cropped up again in a different form.
The next, more critical version of this problem, is with HTTPS traffic. When you have the following conditions:
- Your site is on a shared IP (possible thanks to SNI)
- Your site is configured to listen on HTTPS
- Your site has no SSL certificate
Nginx again, refusing to admit defeat, takes up this challenge by first trying to negotiate the SSL handshake even though you don't have a certificate. It does this by finding the first SSL certificate it can on your server, which probably belongs to another domain! You'll then get a warning that "the certificate for xyz.com does not match the domain example.com" and your client will be confused / angry. This issue can compound with the first issue resulting in the security alert followed by the serving of some other site. In short, it's a mess.
The solution is the same as mentioned above, only you should also include a second listen directive on the secure port you're using, commonly 443. Returning status 444 is probably the right thing to do in that case as well, otherwise you'll need to specify a default certificate to use to negotiate that SSL handshake.
It sounds kind of messed up but really it's just a difference in HTTP server methodologies. I've struggled a bit with the problem, mostly to do with the fact that the default_server flag never seems to work for me...I still can't figure that out. If you run into this problem, what you'll be looking to do it get a catch all server block in place and then do whatever you want with that block.