Fixing permissions using inotify and ACLs

I was working on a shared hosting project and I noticed the permissions were getting a bit complicated.

For each website:

• Each user must have read/write permission
• Each user's group must have read/write permission (reseller access)
• Apache must have read access
• Would be nice if admins had read/write as well

New files can appear at any time, created by all those users, and it would be nice to keep trace of who created them. This is where ACLs kick in.

For the case above, you need this command:

setfacl -m 'u:www-data:r-X,g:example-group:rwX,g:sudo:rwX,o::---' "/srv/www/example.org/htdocs"

ACLs quirks

The problem with ACLs is that they are very fragile and a lot of programs don't propagate them, even if you specify default rules. For example, all programs that move instead of creating will honour the ACL of the directory in which the file was created and the destination. Hence, if you untar the whole website in the directory, the default permissions won't be applied.

Running a script

You could run a script that will loop through all your websites and force the permissions through a cron, but this is a slow process (around 2 minutes on my current setup with ~10 websites), there is a delay before it will run (15-30 minutes, depending on how you set it up), and it is very I/O intensive.

inotifywait

inotify is a library that can warn you if some event occurs on a file or in a folder (recursively). It can be very useful for various tasks like syncing files, doing backups, recording edits progress, etc. inotifywait it simply a program that will wait until the event occurs, so you can execute what you want after.

Watching specifically on CREATE and MOVE events, we can fix the permissions of only the files we need.

The event ATTRIB (chown, chmod, setfacl) was intentionally left out not to cause loops, but it could probably be worked around.

Included below is an example of watching two folders and the corresponding init script. Tested on Ubuntu. 

Run a script with lowest priority

When doing low-priority tasks like backups are fixing permissions on a cronjob, it is a good idea to modify the niceness of the script. By using ionice and renice, you can ensure it won't get in the way of your important programs.

Simple templating system using Bash

I often have to deploy several config files that are very similar. Things like Apache VirtualHost and PHP FPM pools. The solution to this kind of problem is to use something like Puppet or Chef that will apply a real template engine and much more like creating folders and stuff. However, this kind of solution is often lengthy to implement and prevents you from doing some quick editing on-the-fly.

Hence, for very simple needs, I started using simple scripts that would only replace variables and give me a basic template to start with. This is however not very flexible and needs to be adapted for each case. And so I did a templater that replaces variables with the value in the environment. It also supports defining default values and variable interpolation.

Example with Apache + FPM

{{LOG_DIR=/var/log/apache2}}
{{RUN_DIR=/var/run/php-fpm}}
{{FCGI=$RUN_DIR/$DOMAIN.fcgi}}
{{SOCKET=$RUN_DIR/$DOMAIN.sock}}
{{EMAIL=$USER@$DOMAIN}}
{{DOC_ROOT=/home/$USER/sites/$DOMAIN/htdocs}}
<VirtualHost *:80>
  ServerAdmin {{EMAIL}}
  ServerName {{DOMAIN}}
  ServerAlias www.{{DOMAIN}}

  DocumentRoot "{{DOC_ROOT}}"

  <Directory "{{DOC_ROOT}}">
    AllowOverride All
    Order allow,deny
    Allow From All
  </Directory>

  AddHandler php-script .php
  Action php-script /php5.fastcgi virtual
  Alias /php5.fastcgi {{FCGI}}
  FastCGIExternalServer {{FCGI}} -socket {{SOCKET}}

  LogLevel warn
  CustomLog {{LOG_DIR}}/{{DOMAIN}}.access.log combined
  ErrorLog {{LOG_DIR}}/{{DOMAIN}}.error.log
</VirtualHost>

Invocation

DOMAIN=cslavoie.com ./templater.sh examples/vhost-php.conf

Help

If you add the -h switch to the invocation, it will print all the variables and their current values

And the code is available on GitHub.

Batch update modules or themes of Drupal 6/7 in command line

To update Drupal modules, you need to download manually all modules and this can quickly become tedious. Especially if you have multiple modules and Drupal installations.

I wrote a simple script that parse the module's page and install or update the most up-to-date version.

This works for themes as well. To use it, simply go into a modules or themes folder and run one of the tool.

Valid folders are:

• /modules
• /themes
• /sites/all/modules
• /sites/all/themes
• /sites/*/modules
• /sites/*/themes

However, not that you should not try to update root folders, they are reserved for core modules and are updated with Drupal.

drupal-install-module.sh will install or update one module, drupal-update-modules.sh will batch update all modules in current folder.

Introducing Dotfiles Builder

Managing bashrc sucks

We all have our nice little bashrc that we are proud of. It tests for files, programs and terminal features, detect your OS version, builds a PATH, etc. For all of our OS and different setups, various solutions exist.

Keeping several versions

Pros:

• Ultimate fine-tuning
• Easy to understand
• Usually optimized for every setup

Cons:

• Very time consuming to manage
• Hard to “backport” new ideas

Keep a single unusable file with everything and edit accordingly

Pros:

• Easy to backport, you just need to rembember to do it
• Good performance
• Since you edit at each deployment, nice fine-tuning capability

Cons:

• The single file can become unbearably cluttered.
• You eventually end up managing several version.
• Tedious to edit at each deployment

Include several subfiles

Pros:

• Still have a lot fine-tuning capabilities
• If well constructed, can be easy to understand
• Easy to deploy new features

Cons:

• Hard to detect which file to include
• Multiplicates the number of files to manage
• Slow performance
• Until recently, this was my prefered method.

Wanted features

So, what does a good bashrc have?

Should have:

• Good performance. On a busy server, you really don't want to wait 5 seconds for your new terminal because your IO is sky rocketing.
• Reduce number of included files
• Reduce tests for environment features
• Reduce tests for program and files
• High flexibility
• Cross-OS compatible
• A lot of feature detection
• Ideally, configuration files
• Ease and speed of configuration
• It should not take more than a minute to setup a new bashrc
• If you need to specify your developer email, it would be nice to do it only once.

Yes, you read right, reduce tests AND do a lot of feature detection. You don't want to do Java specific configuration or set an empty variable if Java is not even installed, but you do want Java to be automatically detected.

Generating a bashrc

Let's face it, you will install or remove Java way less often then you will start a new shell. Why then test for Java at each new shell?

This is where I introduce the Dotfiles Builder. The script runs in Bash and outputs the wanted bashrc.

This way, instead of doing:

if [ -d "$HOME/bin" ]; then
  PATH="$HOME/bin:$PATH"
fi

You would do:

if [ -d "$HOME/bin" ]; then
  echo "PATH=\"$HOME/bin:$PATH\""
fi

And the result would simply be

PATH="$HOME/bin:$PATH"

But constructing PATH is a rather common task and you want to make sure the folder is not already on your PATH. Why not wrap it up ?

Take a look at the alpha version: https://github.com/lavoiesl/dotfiles-builder/
As well as the example output.

This is a very alpha version of the intended program, but I still want to share what I have and maybe get some feedback and collaborators along the way. Currently, it only generates a bashrc, but expect more to come.

Use credentials in /etc/mysql/debian.cnf to export MySQL database

Quite a usual task is to dump a database to do backups. You may even want to do this in a cronjob to snapshots, etc.

A very bad solution

A very bad solution is to hardcode the root password in the cronjob or in your backup script; doing so have a very high chance of exposing your password.

• It may appear in the cron.log
• It may be sent by email if you have an error
• It may appear in your history
• It is a bad idea to your backups using the root account

A better solution

You could create an account with read-only access to all your databases and use it to to your backups. This is indeed better but can lead to the same issues mentioned above

Putting the password in a file

The safest way to use passwords on the command line is to store them in a file and have a script load them when needed. You then just need to make sure those files have the correct permissions

An “already done for me” solution

As it turns out, installations of dbconfig on Debian/Ubuntu creates a user called debian-sys-maintainer. It is used to do MySQL management, mainly through the package manager. Well, this user has all the needed privileges to backup your database and you are sure it will always work. Unless, of course, you manually change the password without updating the file.

This script uses sudo so it will ask your password even if you forgot to prepend sudo.

Typical usage

$ export-database.sh my_database [mysqldump options] | gzip > /tmp/my_database.sql.gz

PHP script to replace site url in Wordpress database dump, even with WPML

Wordpress has the nice habit of storing every URL with its full path in the database. You sure can hardcode the HOME_URL and SITE_URL in the wp-config.php but it won't change the references to your medias, serialized strings, encoded HTML, etc.

The only solution is really just to edit the database. At least, I haven't found a better solution.

Usage

wordpress-change-url.php http://old-domain.com https://new-domain.com < database.orig.sql > database.new.sql

Or

wordpress-change-url.php database.orig.sql https://new-domain.com > database.new.sql




Will output all remaining mentions of http://old-domain.com to stderr

Apache VirtualHost Template with variable replacement

This is in no way a robust solution for deploying a real web hosting infrastructure, but sometimes, you just need basic templates. I use this simple template on my dev server.

Remove temporary and junk files from Windows and OS X

One of the most annoying things with being able to see all files in the terminal is that… you see all the files. That includes backups, swap and temp files.

Well, it’s a rather good thing, it remembers you to remove them once in a while.