Auditing Downed Hosts on a Network

by Rainer on March 4, 2015, no comments

To conduct an audit of downed hosts (or unused IPs) within a network:

$ nmap -v -sP <network IP range>

For instance:

$ nmap -v -sP | grep down

Once you identify all the unreachable hosts, you can verify their up/down status by pinging the host, then trying to connect to any well-known hosts:

$ ping -c 4 <IP>
$ nc -vz <IP> <port>
$ nslookup <IP>

Responsive Media Queries

by Rainer on October 30, 2014, no comments

Until recent years, CSS media queries have been most commonly used to specify the printed-page parameters of a given site page (e.g., removing background colors, header/footer metadata, etc.), but there is, in fact, a multitude of other reasons to define media query types in relation to the means and devices by which people are viewing your work. Of course, you’ve heard of their role in responsive rendering for browsers and mobile devices, but group/type-specific media queries can also be used to accomodate TV screen dimensions, audio devices, text-to-speech emulators, and Braille printer rendering, to name a few things.

(For the most exhaustive rundown imaginable of media query standards for CSS3, check out the W3C’s docs here)

For a basic example, here’s a standard query for a ‘screen’ (what’s seen on the device) and a ‘print’ (what’s seen coming out of the printer) CSS stylesheet:

<link href="styles.css" type="text/css" media="screen" rel="stylesheet"/>
<link href="print.css" type="text/css" media="print" rel="stylesheet"/>

Alternately, you can write media queries into your CSS files like so:

@media screen and (min-width:900px) {
     .class {
     	background: #f8f8f8;
     	color: #222;

With more complex attributes, we can target specific physical properties of a viewing device, like screen dimensions, resolution, and orientation.

For example, let’s say we want to target media devices having a max. horizontal screen resolution of 480px (like, I dunno, mobile OSes no one’s ever heard of – Android, iOS, etc.) – in this case, we’d set the media query type to ‘screen’ and then inspect the maximum device width by means of a parenthetical attribute:

<link href="mobile-devices.css" type="text/css" media="screen and (max-device-width: 480px)" rel="stylesheet"/>

So, our query will now apply the mobile-devices.css stylesheet to all viewing devices with a horizontal resolution of 480px or less; any device with a horizontal screen resolution exceeding 480px (and/or devices without screens) will ignore this line entirely.

An especially intriguing approach to responsive media queries can be found in the Goldilocks framework – basically, they’ve crafted a device-agnostic, “universal” design template that isn’t tied down to specific dimensions or parameters. Read more about it at http://goldilocksapproach.com

Git Troubleshooting

by Rainer on October 3, 2014, no comments

Just a quick note – if ever you’re attempting to push a project to GitHub (new or existing) and run into a “return code 22 fatal: git-http-push failed” error, try this:

## Edit your .git/config and change this...

[remote 'origin']
fetch = +refs/heads/*:refs/remotes/origin/*
url = http://git.repository.url/repo.git

## ... to this:

[remote 'origin']
fetch = +refs/heads/*:refs/remotes/origin/*
url = http://username:password@git.repository.url/repo.git

## Then try 'git push origin master' again:
$ git push origin master

How to create a new GitHub repository (or add an existing project to GitHub)

by Rainer on September 11, 2014, no comments

* If you’re creating a GitHub repository for a Chef cookbook, see this post instead.

1. First, make sure git is installed:

# For those of you running Debian/Ubuntu/Mint:
$ sudo aptitude install git

# Or, for RHEL/CentOS users:
$ sudo yum install git

2. Create a new repository on GitHub (and don’t initialize the new repository with a README file!)

3. In a terminal, change the current working directory to that of your local project, then initialize that directory as a Git repository:

$ cd /path/to/project/dir/
$ git init

4. Add all of the project files in your local repository (this prepares the files for the first commit/push to your remote repo on GitHub):

$ git add .

5. Now, commit the files that you’ve just added:

$ git commit -m 'First commit'

Essentially, ‘commits’ track each revision to each of the added/committed files, line-by-line, and prepares them to be pushed to a remote repository.

6. From your GitHub account, open your newly-created repository, copy the contents of the “HTTPS clone URL” in the right-hand sidebar, and then flip back to your local terminal. Enter the following to ensure a connection between your local project and your remote repository:

$ git remote add origin <URL of your remote repository>

For instance, if you’ve already created an empty “project-something” repo via GitHub:

$ git remote add origin https://github.com/<username>/project-something.git

If you haven’t created a new repo via GitHub, and you just want to push an existing project to a brand new repo:

$ git remote add origin https://github.com/<username>/make-something-up-here.git

# You can verify the new remote URL with this:
$ git remote -v

7. Now, push the changes in your local repository to GitHub.

$ git push origin master

Now that you’ve set a remote repo location and pushed at least once using the command above, this gets much easier. From now on, you can push changes in your local project to your remote repo by doing only the following:

$ cd /path/to/project/dir/
$ git add

# OR, to add ALL recently-changed files at once:
$ git add .

$ git commit -m 'Added/removed/updated whatever in somefile.ext'
$ git push

Creating a new Chef cookbook

by Rainer on September 11, 2014, no comments

Assuming that you’re working from a knife workstation*, first create the cookbook file structure:

$ knife cookbook create whatever
** Creating cookbook whatever
** Creating README for cookbook: whatever
** Creating metadata for cookbook: whatever

This will create the following directory structure:

$ tree cookbooks/whatever
tree cookbooks/whatever
+-- README.md
+-- attributes
+-- definitions
+-- files
|   +-- default
+-- libraries
+-- metadata.rb
+-- providers
+-- recipes
|   +-- default.rb
+-- resources
+-- templates
    +— default.rb
10 directories, 3 files

The cookbook create command builds a skeleton directory structure for the cookbook. It contains directories that may not be used by every cookbook, but just so you don’t forget something you might want, it creates them all. You can delete the ones you don’t want. For now, we’re going to leave this directory structure as is.

* If you’re not working from a knife workstation, though, you can (sloppily) create a new (exhaustive) cookbook structure with the following command:

$ mkdir -p cookbooks/whatever/attributes && mkdir -p cookbooks/whatever/definitions && mkdir -p cookbooks/whatever/files
 && mkdir -p cookbooks/whatever/libraries && mkdir -p cookbooks/whatever/providers && mkdir -p cookbooks/whatever/recipes && mkdir -p cookbooks/whatever/resources && mkdir -p cookbooks/whatever/templates && touch cookbooks/whatever/README.md && touch cookbooks/whatever/attributes/default.rb && touch cookbooks/whatever/recipes/default.rb && touch cookbooks/whatever/templates/default.rb

(More detailed info to come…)

Extending filesystems in Linux

by Rainer on September 8, 2014, no comments

How to extend a filesystem

To display all logical volumes currently mounted on your system, you can use the lvdisplay command.

Before attempting to add more space to your logical volume, make sure there’s enough remaining space on the hosting volume group. To find the volume group mapped to the filesystem you want to extend:

$ df –h

Look for the volume group hosting your filesystem and see how much free space it contains:

$ vgdisplay
# vgdisplay testvg | grep "Total PE"

# Or:
$ pvs

Then, provided there’s allocatable space, extend your logical volume:

$ lvextend -L +&lt;#M/G&gt; /dev/mapper/(volgroup)-lvname
$ resize2fs /dev/mapper/(volgroup)-lvname

## For example, to extend the somelv01 logical volume (mounted on the somevg volume group) BY 5 GB:
$ lvextend -L +5G /dev/mapper/somevg-somelv01
$ resize2fs /dev/mapper/somevg-somelv01

## Likewise, to extend the somelv01 logical volume (mounted on the somevg volume group) TO 5 GB:
$ lvextend -L 5G /dev/mapper/somevg-somelv01
$ resize2fs /dev/mapper/somevg-somelv01

Migrating a MySQL Instance to a Dedicated Filesystem without Downtime

by Rainer on August 25, 2014, no comments

Registering new disk storage

(*Particularly if you have added new storage to a running virtual machine, you probably won’t see it just yet – the SCSI bus to which the storage devices are connected needs to be rescanned in order to detect the new hardware.)

If you have expanded storage on an existing physical disk and know the device name (e.g., /dev/sda), you can simply enter the following to rescan the bus:

$ echo 1 > /sys/class/scsi_device/<device_name>/rescan

In the event that you are working with a new physical disk, you’ll first need to find the host bus number:

$ grep mpt /sys/class/scsi_host/host?/proc_name

This should return a line like


, which you’ll need to plug into the following command for rescanning the SCSI bus. Then, check to see that your new disk appears:

$ echo "- - -" > /sys/class/scsi_host/host*/scan

## Or to rescan all host buses:
$ for i in {0..4}; do echo "- - -" > /sys/class/scsi_host/host$i/scan; done

## Check to see that new disk appears:
$ fdisk -l | grep Disk

Creating a new volume group and logical volume to host /var/lib/mysql

1. Create a physical volume using the name of the physical disk on which it will be hosted (e.g., /dev/sda3)
2. Create the volume group and the logical volume
3. Format the logical volume
4. Add it to the list of filesystems mounted upon boot:

## Create a physical volume
$ pvcreate /dev/<sd*> /
$ pvcreate /dev/mapper/*
## E.g., '$ pvcreate /dev/mapper/mpath3p1'</pre>
<pre>## Create the volume group
$ vgcreate vgname /path/to/volume/group
Ex: $ vgcreate mysqlvg00 /dev/mapper/mpath3p1

## Create the logical volumes to host the datadir and backups:
$ lvcreate -L -n
$ lvcreate -L 750G mysqlvg00 -n mysqldatalv00
$ lvcreate -L 150G mysqlvg00 -n mysqlbackuplv00

## Create the LV filesystem on each LV created in the previous step:
$ mkfs.ext4 /dev/*/
Ex: $ mkfs.ext4 /dev/mysqlvg00/mysqldatalv00

## Add the LVs to the list of filesystem mount points in /etc/fstab:
$ echo "/dev/*/ /data ext3 defaults 0 0 " >> /etc/fstab
$ echo "/dev/mysqlvg00/mysqldatalv00 /var/lib/mysql ext4 defaults 0 0 " >> /etc/fstab
$ echo "/dev/mysqlvg00/mysqlbackuplv00 /data/backups/mysql/ ext4 defaults 0 0 " >> /etc/fstab

Migrating MySQL instance to new filesystem

1. Stop the MySQL service.

$ service mysql stop

Then, mount the new mysqldatalv00 LV in a location other than /var/lib/mysql so that the existing files in /var/lib/mysql can be transferred. Then, apply the correct file permissions, move the existing files, unmount the LV, remount in the correct location, and start MySQL:

## Temporarily mount the new filesystem somewhere like /tmp or /data
$ mkdir /data/tmp
$ mount /dev/mysqlvg00/mysqldatalv00 /dat/tmp

## Apply correct file permissions
$ chown -R mysql:mysql /data/tmp
$ cd /data/tmp && rmdir lost+found/

## Migrate existing files from /var/lib/mysql to the temporary mount point of the data LV
$ mv /var/lib/mysql/* /data/tmp/

## Unmount the data LV and then mount both the backup and data LVs using the mount points specified in /etc/fstab
$ umount /data/tmp
$ mount -a

## Check /var/lib/mysql for the correct files and permissions, then start MySQL
$ service mysql start

(Troubleshooting section coming soon…)

MySQL Replication Repair

by Rainer on August 25, 2014, no comments

Note: The functioning master node will be referred to as “node A,” while the broken/out of sync node will be “node B”; also, this guide also assumes master-master replication for the time being.

On node A:


1) Do not close this MySQL session – it will need to stay open in order to retain the read lock on the tables while the backup is performed (haven’t found a great way to do this via screen or nohup yet, either)
2) Make note of the output of ‘SHOW MASTER STATUS’ in a text editor, Evernote, etc. Save this information in a format that can be read later.

Both of these steps are vital to the success of this operation, especially since it is usually very time-consuming.

2. Take a mysqldump backup of all databases on node A (use screen or nohup to keep the process going even if your session terminates)

$ nohup mysqldump -u root -p --skip-lock-tables --single-transaction --master-data=1 --all-databases > /path/to/all.sql

3. From node A, copy the sqldump over to node B.

$ scp /data/backups/mysql/path/to/all.sql @:/path/to/dir/with/enough/extra/space

4. From node B, each database that you plan to restore from backup must first be dropped and recreated:

mysql> drop database <db_name>;
mysql> create database <db_name>;

For example:

mysql> drop database hive;
mysql> create database hive DEFAULT CHARACTER SET utf8;
mysql> grant all on hive.* TO 'hive'@'%' IDENTIFIED BY '<hive_password>';

5. On node B, restore the backup (use screen or nohup to keep the process going even if your session terminates; if the instance exceeds 20GB in size, the restoration process could take two hours or more)

$ nohup mysql -u root -p < /data/backups/mysql/dumpfiles/dumpfile.sql

6. Set up master-slave replication (one-way replication from node A to node B; fills in the gaps since node B was last up to date, but won’t write anything to node A):

## On node B:
mysql> STOP SLAVE;

## Fill in the CHANGE MASTER commands using the binary log file and position information obtained from the
‘SHOW MASTER STATUS’ performed on node A in step 1:
mysql> CHANGE MASTER TO master_host=’node A IP]’,
-> master_user=’repl’,
-> master_password=’[repl_user_pw]‘,
-> master_log_file=’[node A mysql-bin.000XXX]‘,
-> master_log_pos=[node A position ###];


## On the same node as the steps performed above, run this periodically to monitor the process as the node reads updated binary logs from the current master node:
$ echo ‘SHOW SLAVE STATUS \G’ | mysql -u root -p | grep ‘Master_Log_File\|Read_Master_Log_Pos’

## On the master node, keep track of the current binary log and file position:
$ echo ‘SHOW MASTER STATUS;’ | mysql -u root -p

Continue until the Master_Log_File and Read_Master_Log_Pos values on the slave node match the File and Position values from the master node, then proceed to the next step.

7. Unlock the tables on node A:


8. Restore master-master replication:

## On node B:

## On node A:
mysql> STOP SLAVE;

## Fill in the CHANGE MASTER commands using the binary log file and position information obtained from
the ‘SHOW MASTER STATUS’ performed on node B just now:
mysql> CHANGE MASTER TO master_host=’node B IP]’,
-> master_user=’repl’,
-> master_password=’[repl_user_pw]‘,
-> master_log_file=’[node B mysql-bin.000XXX]‘,
-> master_log_pos=[node B position ###];


## On both nodes A and B, run the following to make sure that replication is in sync:

If both nodes are in sync, then replication has been restored.

MySQL Replication Setup

by Rainer on August 25, 2014, no comments

1. Create a replication user if one does not exist:
If there is not an existing replication user on each MySQL instance, create a new user:

## On each MySQL node:
$ mysql -u root
mysql> CREATE USER 'repl'@'localhost' IDENTIFIED BY '<repl_password>';
mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'localhost';

In the event that there is an existing password-less replication user (or you create the replication user but forget to set a password), run the following on each MySQL node:

mysql> SET PASSWORD FOR 'repl'@'localhost' = PASSWORD('<repl_password>');

2. On each node, stop the ‘slave’ (if it isn’t already stopped), check the current ‘master’ status, and make note of the current binary log (“File”) and current line position (“Position”):

On each node:

mysql> STOP SLAVE;

3. Make sure each node’s replication user can log in to the MySQL instance on its partner node, test connectivity between the nodes, then run the appropriate ‘change master’ commands on each node (plug in the IP address, current binary log, and current log line position from its peer node):

## On node A:
mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'[node B IP]' IDENTIFIED BY '[node B repl_user_pw]';

## On node B:
mysql> GRANT REPLICATION SLAVE ON *.* TO ‘repl’@’[node A IP]‘ IDENTIFIED BY ‘[node A repl_user_pw]‘;

## From the terminal, test each node’s repl user login in from the other node; from node A:
$ mysql -u repl -p -h

## … And from node B:
$ mysql -u repl -p -h

If this step is unsuccessful from either node, refer again to the replication grants in Step 3

4. Point each node to the other as its master node (master-master replication), using the current binary log and line position information obtained from each node in step 2. On node A:

mysql> CHANGE MASTER TO master_host='[node B IP]',
-> master_user='repl',
-> master_password='[repl_user_pw]',
-> master_log_file='[node B mysql-bin.000XXX]',
-> master_log_pos=[node B position ###];

On node B:

mysql> CHANGE MASTER TO master_host='[node A IP]',
-> master_user='repl',
-> master_password='[repl_user_pw]',
-> master_log_file='[node A mysql-bin.000XXX]',
-> master_log_pos=[node A position ###];

5. Check each node to make sure replication is running without errors: