VPS Experience (Part 2): Configuring Fedora for VPS

Since I rented my current VPS web last year, I had been using CentOS 7.4 for its operating system. It is derived from Red Hat Enterprise Linux (RHEL) project which offers similar stability to an enterprise-grade server. Unfortunately, a few days ago I discovered that some packages in CentOS were really outdated, even on the latest version (7.6). Those packages are related to software development, such as GCC, Clang toolchains, as well as Boost library. The available GCC on the repository cannot even compile C++14! At that time, I was thinking to use my VPS to compile some codes I wrote, which depends on several libraries that I have to also compile manually. Thus, I could not manage to do my experiment and promised myself to destroy the VPS and rebuild from scratch.

My VPS provider provides several operating systems, which are OpenSUSE, Fedora, Ubuntu, CentOS, Scientific, and Debian. I am not really a fan of Debian and its derivatives, such as Ubuntu, despite that now I am heavily using Ubuntu on my Windows Subsystem for Linux (WSL) platform. I thought that Ubuntu is too mainstream and configured mostly for workstation system. I might be wrong and this is purely a taste, so don’t start any distro war here :P. On the other hand, I used OpenSUSE also quite extensively as my desktop workstation. But my VPS providers only provide OpenSUSE Leap rather than rolling-release, which I expect (and experienced) it also has slow-moving repositories. So, I picked Fedora as my new distro for my VPS.

My previous choice on CentOS was also influenced by my experience in using Fedora as my main office workstation. I am quite accustomed to its commands and file system structures, and I have rarely got a big issue when using it as my daily driver (except for installing Tizen Studio, of course, :P). And it has a quite frequent update cycle, which therefore I will not miss any new software update or utilizing the latest feature. But apparently, my VPS only provides an installer for Fedora 27 (currently is 29), and it is not working. The working version that is provided by my provider is Fedora 21, which is way too old. Fortunately, my provider offers a way to upload my own ISO, and I can install directly. My provider also supports web-based display console, which allows me to install the OS easily without the need for an automated script to perform the installation. I will describe some server configurations in this post.

Partitioning and Mount Point

Partition Scheme

I usually tried to separate between system files and user content into separate partitions. This allows me to perform an operating system reinstallation without backing up my personal data. In my regular Linux configuration, I put my user data in my /home folder. But for server configuration, /home folder is not adequate for storing other user content, such as web sites. Web sites usually are stored under /var/www folder and have its own permission model. Therefore, we need to also consider files under /var folder on our partition design.

I own 100GB of space from my VPS provider and want to allocate the storage into two part: system and data. I am thinking to reserve 70% for data, and the remaining for the system. 30GB for the system is fairly adequate considering that this is a headless VPS that does not require workstation application and window manager. The data space will be shared for both home and var. I don’t want to split the data space into two different partitions and prefer to combine both folders to reside in the same space.

Mount Point and fstab

Putting two root-folders (/home and /var) into one partition is not a trivial task. Basically, Linux only allows a partition to be mounted to one specific point. For example, partition /dev/sda1 is mounted to / (root), while /dev/sda2 is mounted to /home. It is not possible to mount /dev/sda2 to two different points such as /home and /var altogether. However, there is a workaround for this which therefore allows us to put /home and /var folder into one partition.

We can utilize Bind mount to remount an existing folder to a new mount point so the new mount point will act similar to a symbolic link to an existing folder. To apply this method, we need to mount the partition to a specific folder first as a temporary mount point, then remount using bind to /var and /home folder. The partition should consist of home and var folder only, which will be accessible from the temporary mount point. Below is the fstab configuration:

/dev/sda2		/mnt/data	xfs	defaults        0 0
/mnt/data/home		/home		none	bind
/mnt/data/var 		/var		none	bind

In fstab configuration above, we mount our second partition temporarily at /mnt/data point, then remount the /mnt/data/home and /mnt/data/var to /home and /var respectively. The remounted folder will behave like a regular mount point and it will work with SELinux policy system which will be discussed at later part.

Software Packages

Nginx

I used Nginx (spelled eNGINe X) for HTTP server. It is very easy to configure and it works with many different scenarios, such as reverse proxy, static pages, and HTTPS. I configured Nginx to serve two domains: a static page and a WordPress site. Both sites are configured to use HTTPS using Let’s Encrypt certificates. Certbot tool is used to configure the certificate automatically; practically it is a one-type installer and pretty easy to use.

Configuring Nginx with PHP-FPM to serve WordPress site is also quite straightforward. Nginx act as a reverse proxy, delegating request processing for PHP pages to PFP-FPM service. Additionally, my WordPress installation is configured with Polylang, where I handled different language for the site under different subdomain. All these settings are simply need to be put on the Nginx configuration script without additional hassle. Below is the example of my Nginx configuration script for my installation:

server {
    server_name  www.heliosky.com heliosky.com id.heliosky.com en.heliosky.com;
    root         /var/www/wordpress;
    index        index.php index.html index.htm;

    # Load configuration files for the default server block.
    include /etc/nginx/default.d/*.conf;

    location / {
        try_files $uri $uri/ /index.php?$args;
    }

    location ~ \.php$ {
        try_files $uri =404;
        fastcgi_pass unix:/var/run/php-fpm.sock;
        fastcgi_index index.php;
        fastcgi_param PHP_VALUE "upload_max_filesize=8M \n post_max_size=8M";
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
    }

    listen [::]:443 ssl; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    # Some ssl config by Certbot...
}

MariaDB

MariaDB is a fork of MySQL by the creators of MySQL themselves. It is intended to be a direct replacement of MySQL, that will remain free and open source, in case of Oracle run amok with their stuff :P. So, MariaDB is completely compatible with MySQL and applications that are built to work with MySQL, including WordPress. Moreover, all necessary tools and commands are still backward compatible with MySQL, including the command line client and additional installation tools. The decision of using MariaDB is because WordPress requires it and WordPress cannot work with other RDBMS. If you are thinking to use other RDBMS, such as PostgreSQL, no you can’t.

Not many configuration for MariaDB as you are provided with a tool called mysql_secure_installation. Run this tool to preconfigure your MariaDB (or MySQL) installation with a quite optimal secure settings. You will be prompted to remove test database, disallow anonymous user, and so on. After it is done, you can login to MySQL using root user and grant a new user to manipulate database as necessary. It is basically a good practice to avoid using root user at all times, so creating a specific user for manipulating the database is better. Please note that this will be different with your WordPress user as we are going to cover that at later section.

PHP-FPM

The server is going to use PHP7 for a more up-to-date and secure platform. As the server is going to use Nginx to serve HTTP requests, PHP-FPM is the only choice to serve PHP content as there is no PHP module similar to mod_php as in Apache stack. PHP-FPM, or FastCGI Process Manager, is a process manager that pools PHP handler to serve PHP using FastCGI mechanism. In old era, CGI is outperformed by module-based handlers like mod_php or ISAPI because the cost of interprocess communication (IPC) between the web server and CGI handler, which both of them run on separate processes, is too high. But nowadays, FastCGI is an improvement on CGI-based stack which provides greater speed comparable with module-based handlers.

PHP-FPM hosts a pool of handlers which are ready to be dispatched when a PHP content is requested by the web server. Web server communicates with it via UNIX socket and FastCGI protocol, and wait until the handler completes the processing. The handler runs on a separate process and can also be configured to run as a different user. Unlike traditional CGI, which shuts down after a request is processed, FastCGI allows the process to keep running, eliminating the overhead of restarting the process on each request. Moreover, it also enables the handler to cache the execution thus making the processing faster.

Various discussions have compared Apache + mod_php and Nginx + PHP-FPM stacks, and it is concluded that PHP-FPM stack can perform with similar performance as mod_php in real life application. The overhead that is caused by the interprocess communication between Nginx and PHP-FPM does not significantly affect the performance. It rather improves security and reliability altogether, for example, the failure in PHP system does not propagated to the web server, avoiding domino effect of server crash.

Configuring Security

SSH Configuration

SSH is the main access control to the server. Although many VPS providers provide VNC remote desktop, SSH remains the most important command control as it is lightweight and secure, if configured properly. Therefore, it is compulsory to perform specific configuration for SSH server and not using default settings provided by the distro installation. The settings include: using SSH key, disable password-based login, disable root login and use non-default port.

First, we need to create SSH key and install it on server. You can generate the key in your local computer, or directly on the server. To generate the key, we can use ssh-keygen tool: ssh-keygen -t rsa -b 4096. This command generates a 4096-bit RSA keypair, which is considered quite secure at this moment. The key will be stored at .ssh folder as id_rsa and id_rsa.pub. Don’t forget to use passphrase to ensure the key can only be accessed by you.

To install the key, run ssh-copy-id command: ssh-copy-id -i <keyfile> <username>@<host>. This will install the key into the server. The command will request for your password prior to the key installation. After you install it, make sure to secure your key and remove it from the server if you are generating it there.

The next one is to configure the sshd on several settings: disable password-based login, disable root login, and change the default port. Here’s the configuration

Port <PORTNUMBER>
PermitRootLogin no
PasswordAuthentication no

The reason for changing the port number is to avoid brute force attack from botnet which continuously tries to authenticate on normal SSH port 22. When I installed my VPS for the first time, I noticed a large amount of entry in my /var/log/secure log. Apparently, many botnet tries to attack the SSH connection by doing brute force attack. I tried to trace back the origin IP address, and most of them came from China. I think it is fairly common that botnet continuously tries to attack insecure server to gain access, so you have to be careful with that.

Changing the port does not necessarily makes everything secure, where an attacker practically can perform port scanning against your server. However, changing the port number can eliminate passive attack like that and interested attackers need to analyze your server manually if they want to gain access. Make sure you also use a port that is not used in your system, and avoid popular port number altogether. Also, it is a good idea if the number is something that you can memorize yourself, so it will similarly acts like a passphrase to access your server. Finally, changing the port number will also require you to change firewall and SELinux configuration, which will be discussed later on.

Firewall Configuration

Next important thing is configuring firewall on your system. Never ever think to disable firewall just because you are lazy to configure it. Fedora provides firewall-cmd to handle firewall configuration easier than using iptables. Yes, I agree that iptables is very hard to configure and can be painful just to make it work correctly. With firewall-cmd, configuring the firewall is easier and more straightforward as it is organized logically using zones and services. For simple hosting server, this is more than adequate.

Some firewall settings are automatically configured by installation script, such as when installing web server. So, no action needed to configure those server. Manual configuration will be required if you make nonstandard changes, such as the previous SSH port change. To configure that, use this example command: sudo firewall-cmd --permanent --service=ssh --add-port=<PORT>/tcp.

DAC Configuration for PHP-FPM

PHP-FPM can be configured to run under a specific user, which can benefit from Discretionary Access Control (DAC) configuration and privilege separation. To configure that, we need to create a new user account first using useradd command. The user should has no login access, and no home directory. Here’s the example command: sudo useradd -M -s /sbin/nologin php-user. It is a good idea to make multiple users if you have multiple PHP application in your server.

After creating the user, we need to configure PHP-FPM pool. The pool configuration is stored in /etc/php-fpm.d/. There is a default configuration www.conf which can be used as template. Each configuration file stored in this folder will spawn a separate pool. So, you can create multiple pool for different PHP application as you need, and each of them will not interfere with each other. I suggest you copy www.conf file and creates your own new file, and rename the default www.conf so an unused PHP-FPM pool will not be spawned. Below is some changes to the default configuration:

[php-poolname]
user = php-user
group = php-user
listen = /var/run/php-myapp.sock
listen.owner = nginx
listen.group = nginx
listen.mode = 0660
listen.allowed_clients = 127.0.0.1
chdir = /var/www/myphpapp

The Nginx is also have to be configured to access the correct socket. Change the fastcgi_pass entry as shown in the example Nginx configuration above with the correct socket file path.

DAC for MariaDB

MariaDB can also be configured to benefit Linux DAC system for its user permission. Normally, MariaDBuses its own user database to perform authentication. By configuring the DAC, MariaDB can authenticate database user using Linux user instead, eliminating the requirement to specify a password in configuration file. This is similar concept with Windows style DAC where the authentication is performed using Windows User Accounts or Active Directory, although less powerful in Linux.

Since the PHP-FPM now has its own user account to run its worker process, we can set MariaDB to authenticate this user to the database using Unix socket, when the user is trying to establish the connection using the socket file. The required library has to be installed first, by issuing this command on MariaDB console: INSTALL SONAME 'auth_socket';. Then, we create a user in MariaDB with exact same name and authenticate it using auth_socket: CREATE USER 'php-user'@localhost IDENTIFIED VIA unix_socket;. Finally, you can grant database access to this user as required.

In this way, the PHP-FPM worker, which runs as php-user, will be able to authenticate itself to MariaDB without supplying a password. And other process which runs as different user cannot access or impersonate php-user when accessing the database. According to the PHP documentation, mysqli_connect function still requires a username to be passed to the function, therefore, you are still required to fill the username field in WordPress config and set password to null.

SELinux Policies

The final part is configuring SELinux policies. This is an important security feature which imposes a Mandatory Access Control (MAC) on Linux system, which governs more strictly than the DAC system. SELinux is quite complex and a little bit tricky to configure, hence many users disable this feature. THIS IS DEFINITELY NOT RECOMMENDED. I repeat DO NOT DISABLE SELINUX AND ALWAYS SET YOUR SELINUX ENFORCING. SELinux is modular and organized, especially when combined with the SELinux command line scripts scripts that is available in the repository which can aid a lot when configuring SELinux policies.

You need to ensure that policycoreutils-python package is installed in your system. It provides semanage command which is very important for configuring SELinux policies. Other important command is restorecon, which is to apply and restore the policy of an object to its default value.

First, we need to add the custom port for SSH to SELinux policy. Although we have opened the port on firewall for SSH, it is not enough as SELinux also impose redundant security measure which disallow unauthorized application to open certain port. If we don’t specify it on the policy, sshd will fail to start as it is prevented to open a socket to listen to that custom port. To add the port for ssh, use this command: semanage port -a -t ssh_port_t -p tcp <your-new-port>.

The next part is configuring our web application root folder. SELinux restrict web server from writing the web application folder by default. Therefore, if your application requires a write access to the folder, such as for uploading images, you will have to explicitly add the rule to the SELinux file context policy, or your web application will fail to perform the writing. If you are using WordPress and put your WordPress installation under /var/www/html/* folder, SELinux has a preconfigured file context for that kind of installation, and you can skip this step. Otherwise, you will also have to modify the file context to ensure your WordPress installation works correctly.

To modify the file context policy, you can either modify the file (/etc/selinux/targeted/contexts/files/file_contexts) yourself using text editor, or using semanage command. Basically, file_contexts file format is very straightforward and you can modify it yourself. Each line in the file represents one policy rule which is applied to an object (file or folder) based on regular expression pattern matching. You can add the example policy below to add rules for your WordPress installation:

/var/www/bloggue/wp-content(/.*)?	system_u:object_r:httpd_sys_rw_content_t:s0
/var/www/bloggue/wp_backups(/.*)?	system_u:object_r:httpd_sys_rw_content_t:s0

The policy describes that var/www/bloggue/wp-content and /var/www/bloggue/wp_backups folders and all theircontents to be labelled as httpd_sys_rw_content_t type. This type label describes a file or folder that can be read and written by httpd (web server) application. You can adjust the policy to suit your needs. And if in some cases your application resides outside of /var/www folder, you also have to label the folder with httpd_sys_content_t so that the web server can read the application files. Failing to do so will make your web returns 403 (forbidden) response.

Then finally, to apply the file context rules, you have to use restorecon command under the folder that you want to be relabeled. If you are applying custom mount point and partition as I did, you will have to relabel the entire /home and /var directory as those folders may have different labels applied (usually it is still as mnt_t since it was mounted at mnt folder after the OS installation). The command is: restorecon -R <folder-to-relabel>.

If you have issues with your application, for example, your web application is not properly loaded, it is worth to read the log file at /var/log/secure to understand what is going on. In many cases, SELinux is actively blocking some access to some functionality. One common cases is that your web application cannot open a connection to remote site using CUrl. This is because httpd_can_network_connectsetting is disabled. To enable that, use setsebool httpd_can_network_connect true command. To list all configurable items in SELinux, use getsebool -a

Conclusion

This explanation roughly covers everything I do to set up my simple VPS for my blog. You can try to apply it on your own setup especially if you want to have fully functional VPS server with basic security in mind. This far from perfect settings, but at least it is sufficient for hosting a small web application in the internet.

Should you also have any idea, discussion, or challenge the explanation, feel free to drop a comment :).

References

  1. https://www.chriswiegman.com/2011/10/fastcgi-vs-suphp-vs-cgi-vs-mod_php-dso/
  2. https://www.ssh.com/ssh/copy-id
  3. https://www.tecmint.com/add-users-in-linux/
  4. https://mariadb.com/kb/en/library/authentication-plugin-unix-socket/
  5. https://wiki.centos.org/HowTos/SELinux#head-ad837f60830442ae77a81aedd10c20305a811388
  6. Image by Victorgrigas – Creative Common Attribution-Share Alike 3.0 Unported

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *