Introducing Dotfiles Builder

Managing bashrc sucks

We all have our nice little bashrc that we are proud of. It tests for files, programs and terminal features, detect your OS version, builds a PATH, etc. For all of our OS and different setups, various solutions exist.

Keeping several versions

Pros:

• Ultimate fine-tuning
• Easy to understand
• Usually optimized for every setup

Cons:

• Very time consuming to manage
• Hard to “backport” new ideas

Keep a single unusable file with everything and edit accordingly

Pros:

• Easy to backport, you just need to rembember to do it
• Good performance
• Since you edit at each deployment, nice fine-tuning capability

Cons:

• The single file can become unbearably cluttered.
• You eventually end up managing several version.
• Tedious to edit at each deployment

Include several subfiles

Pros:

• Still have a lot fine-tuning capabilities
• If well constructed, can be easy to understand
• Easy to deploy new features

Cons:

• Hard to detect which file to include
• Multiplicates the number of files to manage
• Slow performance
• Until recently, this was my prefered method.

Wanted features

So, what does a good bashrc have?

Should have:

• Good performance. On a busy server, you really don't want to wait 5 seconds for your new terminal because your IO is sky rocketing.
• Reduce number of included files
• Reduce tests for environment features
• Reduce tests for program and files
• High flexibility
• Cross-OS compatible
• A lot of feature detection
• Ideally, configuration files
• Ease and speed of configuration
• It should not take more than a minute to setup a new bashrc
• If you need to specify your developer email, it would be nice to do it only once.

Yes, you read right, reduce tests AND do a lot of feature detection. You don't want to do Java specific configuration or set an empty variable if Java is not even installed, but you do want Java to be automatically detected.

Generating a bashrc

Let's face it, you will install or remove Java way less often then you will start a new shell. Why then test for Java at each new shell?

This is where I introduce the Dotfiles Builder. The script runs in Bash and outputs the wanted bashrc.

This way, instead of doing:

if [ -d "$HOME/bin" ]; then
  PATH="$HOME/bin:$PATH"
fi

You would do:

if [ -d "$HOME/bin" ]; then
  echo "PATH=\"$HOME/bin:$PATH\""
fi

And the result would simply be

PATH="$HOME/bin:$PATH"

But constructing PATH is a rather common task and you want to make sure the folder is not already on your PATH. Why not wrap it up ?

Take a look at the alpha version: https://github.com/lavoiesl/dotfiles-builder/
As well as the example output.

This is a very alpha version of the intended program, but I still want to share what I have and maybe get some feedback and collaborators along the way. Currently, it only generates a bashrc, but expect more to come.

Use credentials in /etc/mysql/debian.cnf to export MySQL database

Quite a usual task is to dump a database to do backups. You may even want to do this in a cronjob to snapshots, etc.

A very bad solution

A very bad solution is to hardcode the root password in the cronjob or in your backup script; doing so have a very high chance of exposing your password.

• It may appear in the cron.log
• It may be sent by email if you have an error
• It may appear in your history
• It is a bad idea to your backups using the root account

A better solution

You could create an account with read-only access to all your databases and use it to to your backups. This is indeed better but can lead to the same issues mentioned above

Putting the password in a file

The safest way to use passwords on the command line is to store them in a file and have a script load them when needed. You then just need to make sure those files have the correct permissions

An “already done for me” solution

As it turns out, installations of dbconfig on Debian/Ubuntu creates a user called debian-sys-maintainer. It is used to do MySQL management, mainly through the package manager. Well, this user has all the needed privileges to backup your database and you are sure it will always work. Unless, of course, you manually change the password without updating the file.

This script uses sudo so it will ask your password even if you forgot to prepend sudo.

Typical usage

$ export-database.sh my_database [mysqldump options] | gzip > /tmp/my_database.sql.gz

PHP script to replace site url in Wordpress database dump, even with WPML

Wordpress has the nice habit of storing every URL with its full path in the database. You sure can hardcode the HOME_URL and SITE_URL in the wp-config.php but it won't change the references to your medias, serialized strings, encoded HTML, etc.

The only solution is really just to edit the database. At least, I haven't found a better solution.

Usage

wordpress-change-url.php http://old-domain.com https://new-domain.com < database.orig.sql > database.new.sql

Or

wordpress-change-url.php database.orig.sql https://new-domain.com > database.new.sql




Will output all remaining mentions of http://old-domain.com to stderr

Apache VirtualHost Template with variable replacement

This is in no way a robust solution for deploying a real web hosting infrastructure, but sometimes, you just need basic templates. I use this simple template on my dev server.

Remove temporary and junk files from Windows and OS X

One of the most annoying things with being able to see all files in the terminal is that… you see all the files. That includes backups, swap and temp files.

Well, it’s a rather good thing, it remembers you to remove them once in a while.

Document Root fix in .htaccess when using VirtualDocumentRoot

CMS often come with a .htaccess that has a RewriteRule like this:

# RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ app.php [QSA,L]

The RewriteBase has to be enabled if you are using a VirtualDocumentRoot but when you are sharing code, developers may all have different setups.

By checking the content of DOCUMENT_ROOT, we can guess which setup we are using and prepend a / when necessary.

Note however that this method is heavy on string comparison which is slow and should not go on production.

Writing a PHPUnit test for Symfony2 to test email sending

The idea is to test that an email is sent by a contact form by inspecting the SwiftMailler Collector.

We generate a fake unique content and loop through all sent emails to verify that is was sent.

Note that this does not actually test if the email is sent to the server but merely tells you if Symfony is trying to send it.


Inspiration: http://docs.behat.org/cookbook/using_the_profiler_with_minkbundle.html

Validating emails using SMTP queries

Email validation are traditionally done in three flavors. They have pros and cons but they are widely spread; let us go through them.

Regex validation

This is perhaps the most simple validation validation. May you do it via <input type="email">, Javascript or a server-side language, it's basically the same technique: checking for the format of the email. The problem is, about any format is valid [Wikipedia examples] . Most regex are oinly validating a subset of the RFC even though I doubt someone uses ""@example.org as his email.

Further reading: http://www.regular-expressions.info/email.html

Upside: Very early validation, can be done client-side to prevent common errors.

Downside: Only checks for format and if you want to support the full RFC, it is a nice but rather "light" validation.

MX record check

A MX record basically is at which IP address the email will end-up. For example, gmail.com points to 173.194.77.27. Doing a MX record means checking if the domain of the email address is valid and if it points somewhere. If yes, assume the server knows how to handle the email. This is done server-side.

Upside: You do a real test on the domain name, it will prevent errors like bob@gmai.lcom.
Downside: You do not test if there is actually a mail server accepting emails at this domain and the full address is not validated.

Real validation using a confirmation email.

The most common are secure way to validate an email is to actually send one and wait for the user to interact with it. Confirmation code, link to click, etc. You really are sure the email exists, but this is not really error-friendly: if the email is invalid, how will you or the user know ? For sure you can setup a daemon that will receive and parse bounce-backs, but how will you let the user know and do you seriously want to do that ? It also won't let you know if you hit a default email which sends all unknown emails to another account.

Furthermore, it usually annoys the user.

Introducing SMTP query

I wrapped up some code I found about a asking the SMTP server if the email account exists. Here it is on GitHub: https://github.com/lavoiesl/smtp-email-validator. 

It starts by a MX record check and then, using the SMTP protocol, open a socket to the server and starts writing an email saying HELO, MAIL FROM and RCPT TO. The last one is the user, the email you want to test. Now three things may happen:

1. This email is known, server replies 250 => Email is valid.
2. Email is greylisted or some minor occurs: 450 or 451 => Email should be valid.
3. Other error => Email is invalid.

This is a rather quick test to do and you are now pretty sure the email is valid.

Conclusion

The complete way to handle email validation would be:

1. Handle common format issues using client-side and/or server-side regex validation.
2. MX record check
3. SMTP query
4. Send a confirmation email, not asking for any validation but explaining to the user that he should contact you if he doesn't receive an email.