• nginx + apache

    http://devkardia.com/easyblog/virtualmin-apache-and-nginx-reverse-proxy.html

    I wanted to be able to setup a reverse proxy with nginx and apache but continue using Virtualmin GPL to manage my domains. The only reason I wanted a reverse proxy is that several of the domains I host utilize a very large amount of images. I wanted a efficient way to handle these images since apache uses the same process to handle everything for the domain user and thus its memory footprint continued to grow till the process was killed. Thus I had processes using 150+ MB of memory just to serve a picture.

    Anyway, Virtualmin does not support this config out of box so I had to do a bit of rigging 🙂 I figure I would share how I went about doing this.

    A bit of background. Normally with this setup, you would have nginx accessible by the world and apache only accessible by the local host. But if you plan on continue allowing Virtualmin to manage your domains, you can’t do this. Unless you want to manually edit your config files each time Virtualmin creates a domain. So instead, I had to leave both open to the world.

    Installing/configuring nginx

    Ok, so assuming that you have a fully functioning Virtualmin and Apache2 servers, let’s install nginx. I use Ubuntu 10.04 LTS under Linode and the nginx server available in it’s ppa is a bit old. So let’s get the latest version from nginx’s ppa:

    sudo add-apt-repository ppa:nginx/stable 
    sudo apt-get update 
    sudo apt-get install nginx

    Now create the file /etc/nginx/proxy.conf and add the following to it:

    proxy_redirect          off;
    proxy_set_header        Host            $host;
    proxy_set_header        X-Real-IP       $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    client_max_body_size    10m;
    client_body_buffer_size 128k;
    proxy_connect_timeout   90;
    proxy_send_timeout      90;
    proxy_read_timeout      90;
    proxy_buffers           32 4k;

    Those are the standard settings. You may have to adjust the numbers to best fit your site’s needs.

    Open /etc/nginx/nginx.conf and edit it to match the following (adjust to your server’s needs)

    user www-data www-data;
    worker_processes  2;
    
    error_log  /var/log/nginx/error.log;
    pid        /var/run/nginx.pid;
    
    events {
        worker_connections  1024;
        # multi_accept on;
    }
    
    http {
        include       /etc/nginx/mime.types;
    
        access_log  /var/log/nginx/access.log;
    
        sendfile        on;
        #tcp_nopush     on;
    
        #keepalive_timeout  0;
        keepalive_timeout  65;
        tcp_nodelay        on;
    
        gzip  on;
        gzip_disable "MSIE [1-6]\.(?!.*SV1)";
    
        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
    }

    I created/edited the default host file to handle any domains that do not have a specific config file:

    /etc/nginx/sites-available/default

    server {
        listen 12.34.56.78:80 default;
        server_name  _;
        access_log /var/log/nginx/default.access.log;
        error_log /var/log/nginx/default.error.log;
    
        location / {
            proxy_pass http://12.34.56.78:9091;
            include /etc/nginx/proxy.conf;
       }
    }

    Now create a virtual host file. I created one for each of my domains and named them like domain.com.conf so that my Virtualmin script can automatically handle creating them (see below).

    So for example, create /var/nginx/sites-available/mydomain.com.conf:

    server {
        listen 12.34.56.78:80;
        server_name www.mydomain.com mydomain.com foo.mydomain.com bar.mydomain.com;
        access_log /var/log/virtualmin/mydomain.com_nginx_access_log;
        error_log  /var/log/virtualmin/mydomain.com_nginx_error_log;
        location / {
            proxy_pass http://12.34.56.78:9091;
            include /etc/nginx/proxy.conf;
        }
    
        location ~* ^.+\.(jpe?g|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mp3)$ {
            expires 30d;
            root /home/mydomainuser/public_html;
        }
    }
    

    Enable your site:

    sudo ln -s /etc/nginx/sites-available/mydomain.com.conf /etc/nginx/sites-enabled/mydomain.com.conf

    Of course you will replace 12.34.56.78 with your server’s IP address (if you are using a host with multiple IPs, be sure to use the correct one for this host!). Also change mydomain.com and add any other server names to the server_name directive. Now for the proxy_pass, normally this would 127.0.0.1:9091 (9091 being the port we are going to configure apache to run on). But like I said earlier, I still want Virtualmin to be fully functional without me having to manually edit the apache config files after Virtualmin created them to force apache to listen to 127.0.0.1. So I am leaving both open to the world and thus apache will listen to the host’s IP address.

    Test your configs with:

    nginx -t -c /etc/nginx/nginx.conf

    You should get a success message. If you get errors fix it before starting nginx.

    Start/restart nginx:

    sudo /etc/init.d/nginx restart

    Configuring Apache

    We need to now configure Apache to listen to port 9091 and to tell it to use the correct IP address in the logs and in the $_SERVER var for PHP.

    Open /etc/apache2/apache2.conf and change NameVirtualHost 12.34.56.78:80 to 12.34.56.78:9091

    Open /etc/apache2/ports.conf and change Listen 80 to Listen 9091

    Open all the config files in /etc/apache2/sites-available and change to

    Install mod_rpaf so that apache knows the true IP address of the user accessing the site:

    sudo apt-get install libapache2-mod-rpaf
    sudo a2enmod rpaf

    Open /etc/apache2/mods-available/rpaf.conf and add your IP address(es) to the RPAFproxy_ips directive so that it looks something like this:

    RPAFenable On
    RPAFsethostname On
    RPAFproxy_ips 127.0.0.1 12.34.56.78

    Its important to add your server’s IP address(es) to the RPAFproxy_ips directive as if you do not, $_SERVER[‘REMOTE_ADDR’] will always be your server’s IP address which can be bad for scripts that rely on this (like php based firewalls).

    Reload apache:

    sudo /etc/init.d/apache2 restart

    Configuring Virtualmin

    Now we have to make some changes to Virtualmin and its config files.

    First we need to edit all the existing servers to use the new apache port. So open each file in /etc/webmin/virtual-server/domains (each file represents a server) and change web_port=80 to web_port=9091.

    Now, login to Virtualmin and go to Server Settings -> Server Templates -> click on your default template -> choose Apache website from the template section dropdown -> change Port number for virtual hosts to 9091 then click Save and Next.

    Restart webmin:

    sudo /etc/init.d/webmin restart

    Choose Log file rotation from the template section dropdown, choose Log files below for the Additional files to rotate field. Add the following to the field’s textbox:

    /var/log/virtualmin/${DOM}_nginx_access_log
    /var/log/virtualmin/${DOM}_nginx_error_log

    Note, that you will need to manually add the nginx log files to logrotate for existing domains. You can use Webmin -> System -> Log File Rotation to do so.

    Automating Virtualmin for nginx

    Now I didn’t want to manually create new virtual host files for nginx each time Virtualmin created a server. So, I created a little script to do it for me.

    First, I created a template file /etc/nginx/sites-available/template.conf which has the following in it:

    server {
        listen {SITE_IP}:80;
        server_name www.{DOM} {DOM};
        access_log /var/log/virtualmin/{DOM}_nginx_access_log;
        error_log  /var/log/virtualmin/{DOM}_nginx_error_log;
    
        location / {
            proxy_pass http://{SITE_IP}:9091;
            include /etc/nginx/proxy.conf;
        }
    
        location ~* ^.+\.(jpe?g|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mp3)$ {
            expires 30d;
            root {HOME}/public_html;
        }
    }
    

    Now, I created a script that Virtualmin will run after it creates/deletes/modifies a server.

    /usr/local/bin/virtualmin.sh

    #!/bin/sh
    NGINX_CONF_FILE="/etc/nginx/sites-available/${VIRTUALSERVER_DOM}.conf "
    
    if [ "$VIRTUALSERVER_ACTION" = "CREATE_DOMAIN" ]; then
    	if [ "${VIRTUALSERVER_WEB}" = "1" ];
    	then
    		cp /etc/nginx/sites-available/template.conf $NGINX_CONF_FILE
    
    		perl -pi -e "s#{DOM}#$VIRTUALSERVER_DOM#g" $NGINX_CONF_FILE
    		perl -pi -e "s#{SITE_IP}#$VIRTUALSERVER_IP#g" $NGINX_CONF_FILE
    		perl -pi -e "s#{HOME}#$VIRTUALSERVER_HOME#g" $NGINX_CONF_FILE
    		ln -s $NGINX_CONF_FILE /etc/nginx/sites-enabled/${VIRTUALSERVER_DOM}.conf
    		/etc/init.d/nginx reload
    	fi
    
    
    elif [ "$VIRTUALSERVER_ACTION" = "DELETE_DOMAIN" ]; then
            if [ "${VIRTUALSERVER_WEB}" = "1" ];
            then
    		    rm /etc/nginx/sites-enabled/${VIRTUALSERVER_DOM}.conf
        		rm /etc/nginx/sites-available/${VIRTUALSERVER_DOM}.conf
    	    	rm /var/log/virtualmin/${VIRTUALSERVER_DOM}_nginx_*
                /etc/init.d/nginx reload
        	fi
    
    
    elif [ "$VIRTUALSERVER_ACTION" = "MODIFY_DOMAIN" ]; then
            if [ "${VIRTUALSERVER_WEB}" = "1" ];
            then
        		if [ ! -f $NGINX_CONF_FILE ]; then
     		        cp /etc/nginx/sites-available/template.conf $NGINX_CONF_FILE
    	        	perl -pi -e "s#{DOM}#$VIRTUALSERVER_DOM#g" $NGINX_CONF_FILE
    		        perl -pi -e "s#{SITE_IP}#$VIRTUALSERVER_IP#g" $NGINX_CONF_FILE
            		perl -pi -e "s#{HOME}#$VIRTUALSERVER_HOME#g" $NGINX_CONF_FILE
    		        ln -s $NGINX_CONF_FILE /etc/nginx/sites-enabled/${VIRTUALSERVER_DOM}.conf
    	    	fi
        	fi
    
    	    if [ "$VIRTUALSERVER_DOM" != "$VIRTUALSERVER_OLDSERVER_DOM" ]; then
    	        if [ "${VIRTUALSERVER_WEB}" = "1" ];
            	then
    			    OLD_NGINX_CONF_FILE=/etc/nginx/sites-available/${VIRTUALSERVER_OLDSERVER_DOM}.conf
    			    mv $OLD_NGINX_CONF_FILE $NGINX_CONF_FILE
    			    rm /etc/nginx/sites-enabled/${VIRTUALSERVER_OLDSERVER_DOM}.conf
            		perl -pi -e "s#$VIRTUALSERVER_OLDSERVER_DOM#$VIRTUALSERVER_DOM#g" $NGINX_CONF_FILE
    	        	perl -pi -e "s#$VIRTUALSERVER_OLDSERVER_IP#$VIRTUALSERVER_IP#g" $NGINX_CONF_FILE
    		        perl -pi -e "s#$VIRTUALSERVER_OLDSERVER_HOME#$VIRTUALSERVER_HOME#g" $NGINX_CONF_FILE
    			    ln -s /etc/nginx/sites-available/${VIRTUALSERVER_DOM}.conf /etc/nginx/sites-enabled/${VIRTUALSERVER_DOM}.conf
    	    	fi
    	    fi
    
            if [ "${VIRTUALSERVER_WEB}" = "1" ];
            then
    	        /etc/init.d/nginx reload
    	    fi
    fi
    

    Now go to Virtualmin -> System Settings -> Virtualmin Configuration -> choose Actions upon server and user creation from the category dropdown and add /usr/local/bin/virtualmin-postaction.sh to “Command to run after making changes to a server.” Now Virtualmin will automatically manage the nginx host file for you.
    That should do it!


1111111