Subscribe to Postshttp://devkardia.com/easyblog/virtualmin-apache-and-nginx-reverse-proxy.html
I wanted to be able to setup a reverse proxy with nginx and apache but continue using Virtualmin GPL to manage my domains. The only reason I wanted a reverse proxy is that several of the domains I host utilize a very large amount of images. I wanted a efficient way to handle these images since apache uses the same process to handle everything for the domain user and thus its memory footprint continued to grow till the process was killed. Thus I had processes using 150+ MB of memory just to serve a picture.
Anyway, Virtualmin does not support this config out of box so I had to do a bit of rigging 🙂 I figure I would share how I went about doing this.
A bit of background. Normally with this setup, you would have nginx accessible by the world and apache only accessible by the local host. But if you plan on continue allowing Virtualmin to manage your domains, you can’t do this. Unless you want to manually edit your config files each time Virtualmin creates a domain. So instead, I had to leave both open to the world.
Installing/configuring nginx
Ok, so assuming that you have a fully functioning Virtualmin and Apache2 servers, let’s install nginx. I use Ubuntu 10.04 LTS under Linode and the nginx server available in it’s ppa is a bit old. So let’s get the latest version from nginx’s ppa:
sudo add-apt-repository ppa:nginx/stable sudo apt-get update sudo apt-get install nginx
Now create the file /etc/nginx/proxy.conf and add the following to it:
proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; client_max_body_size 10m; client_body_buffer_size 128k; proxy_connect_timeout 90; proxy_send_timeout 90; proxy_read_timeout 90; proxy_buffers 32 4k;
Those are the standard settings. You may have to adjust the numbers to best fit your site’s needs.
Open /etc/nginx/nginx.conf and edit it to match the following (adjust to your server’s needs)
user www-data www-data;
worker_processes 2;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
# multi_accept on;
}
http {
include /etc/nginx/mime.types;
access_log /var/log/nginx/access.log;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
tcp_nodelay on;
gzip on;
gzip_disable "MSIE [1-6]\.(?!.*SV1)";
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
I created/edited the default host file to handle any domains that do not have a specific config file:
/etc/nginx/sites-available/default
server {
listen 12.34.56.78:80 default;
server_name _;
access_log /var/log/nginx/default.access.log;
error_log /var/log/nginx/default.error.log;
location / {
proxy_pass http://12.34.56.78:9091;
include /etc/nginx/proxy.conf;
}
}
Now create a virtual host file. I created one for each of my domains and named them like domain.com.conf so that my Virtualmin script can automatically handle creating them (see below).
So for example, create /var/nginx/sites-available/mydomain.com.conf:
server {
listen 12.34.56.78:80;
server_name www.mydomain.com mydomain.com foo.mydomain.com bar.mydomain.com;
access_log /var/log/virtualmin/mydomain.com_nginx_access_log;
error_log /var/log/virtualmin/mydomain.com_nginx_error_log;
location / {
proxy_pass http://12.34.56.78:9091;
include /etc/nginx/proxy.conf;
}
location ~* ^.+\.(jpe?g|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mp3)$ {
expires 30d;
root /home/mydomainuser/public_html;
}
}
Enable your site:
sudo ln -s /etc/nginx/sites-available/mydomain.com.conf /etc/nginx/sites-enabled/mydomain.com.conf
Of course you will replace 12.34.56.78 with your server’s IP address (if you are using a host with multiple IPs, be sure to use the correct one for this host!). Also change mydomain.com and add any other server names to the server_name directive. Now for the proxy_pass, normally this would 127.0.0.1:9091 (9091 being the port we are going to configure apache to run on). But like I said earlier, I still want Virtualmin to be fully functional without me having to manually edit the apache config files after Virtualmin created them to force apache to listen to 127.0.0.1. So I am leaving both open to the world and thus apache will listen to the host’s IP address.
Test your configs with:
nginx -t -c /etc/nginx/nginx.conf
You should get a success message. If you get errors fix it before starting nginx.
Start/restart nginx:
sudo /etc/init.d/nginx restart
Configuring Apache
We need to now configure Apache to listen to port 9091 and to tell it to use the correct IP address in the logs and in the $_SERVER var for PHP.
Open /etc/apache2/apache2.conf and change NameVirtualHost 12.34.56.78:80 to 12.34.56.78:9091
Open /etc/apache2/ports.conf and change Listen 80 to Listen 9091
Open all the config files in /etc/apache2/sites-available and change to
Install mod_rpaf so that apache knows the true IP address of the user accessing the site:
sudo apt-get install libapache2-mod-rpaf sudo a2enmod rpaf
Open /etc/apache2/mods-available/rpaf.conf and add your IP address(es) to the RPAFproxy_ips directive so that it looks something like this:
RPAFenable On RPAFsethostname On RPAFproxy_ips 127.0.0.1 12.34.56.78
Its important to add your server’s IP address(es) to the RPAFproxy_ips directive as if you do not, $_SERVER[‘REMOTE_ADDR’] will always be your server’s IP address which can be bad for scripts that rely on this (like php based firewalls).
Reload apache:
sudo /etc/init.d/apache2 restart
Configuring Virtualmin
Now we have to make some changes to Virtualmin and its config files.
First we need to edit all the existing servers to use the new apache port. So open each file in /etc/webmin/virtual-server/domains (each file represents a server) and change web_port=80 to web_port=9091.
Now, login to Virtualmin and go to Server Settings -> Server Templates -> click on your default template -> choose Apache website from the template section dropdown -> change Port number for virtual hosts to 9091 then click Save and Next.
Restart webmin:
sudo /etc/init.d/webmin restart
Choose Log file rotation from the template section dropdown, choose Log files below for the Additional files to rotate field. Add the following to the field’s textbox:
/var/log/virtualmin/${DOM}_nginx_access_log
/var/log/virtualmin/${DOM}_nginx_error_log
Note, that you will need to manually add the nginx log files to logrotate for existing domains. You can use Webmin -> System -> Log File Rotation to do so.
Automating Virtualmin for nginx
Now I didn’t want to manually create new virtual host files for nginx each time Virtualmin created a server. So, I created a little script to do it for me.
First, I created a template file /etc/nginx/sites-available/template.conf which has the following in it:
server {
listen {SITE_IP}:80;
server_name www.{DOM} {DOM};
access_log /var/log/virtualmin/{DOM}_nginx_access_log;
error_log /var/log/virtualmin/{DOM}_nginx_error_log;
location / {
proxy_pass http://{SITE_IP}:9091;
include /etc/nginx/proxy.conf;
}
location ~* ^.+\.(jpe?g|gif|png|ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mp3)$ {
expires 30d;
root {HOME}/public_html;
}
}
Now, I created a script that Virtualmin will run after it creates/deletes/modifies a server.
/usr/local/bin/virtualmin.sh
#!/bin/sh
NGINX_CONF_FILE="/etc/nginx/sites-available/${VIRTUALSERVER_DOM}.conf "
if [ "$VIRTUALSERVER_ACTION" = "CREATE_DOMAIN" ]; then
if [ "${VIRTUALSERVER_WEB}" = "1" ];
then
cp /etc/nginx/sites-available/template.conf $NGINX_CONF_FILE
perl -pi -e "s#{DOM}#$VIRTUALSERVER_DOM#g" $NGINX_CONF_FILE
perl -pi -e "s#{SITE_IP}#$VIRTUALSERVER_IP#g" $NGINX_CONF_FILE
perl -pi -e "s#{HOME}#$VIRTUALSERVER_HOME#g" $NGINX_CONF_FILE
ln -s $NGINX_CONF_FILE /etc/nginx/sites-enabled/${VIRTUALSERVER_DOM}.conf
/etc/init.d/nginx reload
fi
elif [ "$VIRTUALSERVER_ACTION" = "DELETE_DOMAIN" ]; then
if [ "${VIRTUALSERVER_WEB}" = "1" ];
then
rm /etc/nginx/sites-enabled/${VIRTUALSERVER_DOM}.conf
rm /etc/nginx/sites-available/${VIRTUALSERVER_DOM}.conf
rm /var/log/virtualmin/${VIRTUALSERVER_DOM}_nginx_*
/etc/init.d/nginx reload
fi
elif [ "$VIRTUALSERVER_ACTION" = "MODIFY_DOMAIN" ]; then
if [ "${VIRTUALSERVER_WEB}" = "1" ];
then
if [ ! -f $NGINX_CONF_FILE ]; then
cp /etc/nginx/sites-available/template.conf $NGINX_CONF_FILE
perl -pi -e "s#{DOM}#$VIRTUALSERVER_DOM#g" $NGINX_CONF_FILE
perl -pi -e "s#{SITE_IP}#$VIRTUALSERVER_IP#g" $NGINX_CONF_FILE
perl -pi -e "s#{HOME}#$VIRTUALSERVER_HOME#g" $NGINX_CONF_FILE
ln -s $NGINX_CONF_FILE /etc/nginx/sites-enabled/${VIRTUALSERVER_DOM}.conf
fi
fi
if [ "$VIRTUALSERVER_DOM" != "$VIRTUALSERVER_OLDSERVER_DOM" ]; then
if [ "${VIRTUALSERVER_WEB}" = "1" ];
then
OLD_NGINX_CONF_FILE=/etc/nginx/sites-available/${VIRTUALSERVER_OLDSERVER_DOM}.conf
mv $OLD_NGINX_CONF_FILE $NGINX_CONF_FILE
rm /etc/nginx/sites-enabled/${VIRTUALSERVER_OLDSERVER_DOM}.conf
perl -pi -e "s#$VIRTUALSERVER_OLDSERVER_DOM#$VIRTUALSERVER_DOM#g" $NGINX_CONF_FILE
perl -pi -e "s#$VIRTUALSERVER_OLDSERVER_IP#$VIRTUALSERVER_IP#g" $NGINX_CONF_FILE
perl -pi -e "s#$VIRTUALSERVER_OLDSERVER_HOME#$VIRTUALSERVER_HOME#g" $NGINX_CONF_FILE
ln -s /etc/nginx/sites-available/${VIRTUALSERVER_DOM}.conf /etc/nginx/sites-enabled/${VIRTUALSERVER_DOM}.conf
fi
fi
if [ "${VIRTUALSERVER_WEB}" = "1" ];
then
/etc/init.d/nginx reload
fi
fi
Now go to Virtualmin -> System Settings -> Virtualmin Configuration -> choose Actions upon server and user creation from the category dropdown and add /usr/local/bin/virtualmin-postaction.sh to “Command to run after making changes to a server.” Now Virtualmin will automatically manage the nginx host file for you.
That should do it!
