Nginx, AWS ELB, DNS resolution, Nginx resolver directive and Black magic

We use Nginx to reverse proxy traffic to our app servers in a few places (on our Varnish servers we have a Nginx proxy that reverse proxy traffic to our backend servers and the backend servers them selves reverse proxy traffic to other app servers). And all our servers are hosted on AWS and they all sit behind ELBs.

So just like a lot of people on the inter webs (look at the references section bellow) we found out that Nginx is only resolving the ELB DNS once on startup and then not refreshing it. This caused some of our servers to break because AWS changes IPs for ELBs on the fly.

The usually suggested work around is to use the Nginx resolver directive. To define a DNS server for Nginx to do name lookup on + more importantly a time of validity for the DNS lookups!.

So you’d have:

http {  
    ...

    resolver 10.xxx.xxx.2 valid=30s;

    ...
}

server {

    ...

    set $elb "{{ lp_app_elb }}";

    location / { 
        ...

        proxy_pass http://$elb/;

        ...
    }
...    
}

In your Nginx config. And Nginx will resolve domain names every 30 seconds. And it will pass all the traffic through to the defined ELB (or to whatever server).

The alternative is to use Nginx+. In which you don’t have to set a variable.

However, things were not so ‘simple’ for us. We found out that the above configuration would strip out all the following parts of the URL and any parameters. Like:

/path1/path2/path3?p1=v1

will be passed on as

/path1

This is obviously not what we want. And strangely this seems to happen only when we set a variable and then pass that in to proxy_pass. After banging our heads on the provable wall for a while we settled on the following workaround.

rewrite ^/(.*) $1 break;  
proxy_pass http://$elb/$1?$args;  

So now our Nginx config looks something like

http {  
    ...

    resolver 10.xxx.xxx.2 valid=30s;

    ...
}

server {

    ...

    set $elb "{{ lp_app_elb }}";

    location / { 
        ...

        rewrite ^/(.*) $1 break;
        proxy_pass http://$elb/$1?$args; 

        ...
    }
...    
}

It would be ideal if we didn’t have to mess around with rewrite to get this to work. But for the time being we are stuck with it. Let me know if you know better!

-

Refferences

Primary

Others