Valcora Blog - Scripts

Monitoring Disk Space

nospam@example.com (Valcora) — Mon, 04 May 2009 12:15:56 -0600

Some time back, when a client wanted us to setup MySQL Enterprise Monitor, we were surprised to find out that disk monitoring was not available! We worked hard to come up with a solution. Eventually, we decided to setup a custom agent to monitor the disk. Below is the result of that.

While this script may not work as-is for everyone, it should at least provide a basis for such a script. This script has been modified to send an email instead of plug directly into the MySQL Enterprise Monitor. But, it hopefully will get our creative juices flowing...

#!/bin/bash
#
# This script does a very simple test for checking disk space.
# 
# Valcora: http://www.valcora.com
#

CHECKDISK=`df -h | awk '{print $5}' | grep % | grep -v Use | sort -n | tail -1 | cut -d "%" -f1 -`
ALERT_VALUE="80"
MAIL_USER="root@localhost.com"
MAIL_SUBJECT="Daily Disk Check"

if [ "$CHECKDISK" -ge "$ALERT_VALUE" ]; then
  echo "At least one of my disks is nearly full!" | mail -s "$MAIL_SUBJECT" $MAIL_USER
else
  echo "Disk space normal" | mail -s "$MAIL_SUBJECT" $MAIL_USER
fi

This would be a simple process to cron. Let us know if this has been of any help to you!

Export All To CSV

nospam@example.com (Valcora) — Mon, 04 May 2009 12:00:27 -0600

Last night I was asked by a client to export all of the tables into CSV format. I thought no problem. Then I saw the long list of table names! They wanted every one of them in CSV format! At that point, I started thinking about making a backup and then importing them all as CSV storage engine and then I realized CSV was disabled on the sandbox server! So, I thought well, I could try MySQL Query Browser, but then I figured I would still have to do it on a table-by-table basis. That is when I decided to go the Bash script route!

This lead to the formation of the brief script you see below. It simply connects to a given database, shows the list of tables, and then dumps out the contents in CSV format to individual files named tablename.csv in the current directory. After just a few minutes of scripting, it was all over and I had a handy little script for the next time they or another client ask for the same thing!

Hopefully this script will be of some value to everyone else as well. Just be sure to change the configuration section to meet your needs.

Name the file something like: export_csv.sh. Then be sure to make it executable. In Linux, do something like "chmod a+x ./export_csv.sh" and you should be able to rock and roll.

If you want to have all of the scripts in a certain directory, you could either modify the script or just make the cirectory, "cd" into it, and then run the script. It assumes you want to create the files in the current working directory. To change that behavior, you could easily modify the "OUTFILE" variable to something like:

OUTFILE="/my_path/$TABLE.csv"

Anyway, here is the script for your enjoyment and use!

#!/bin/bash
#
# Simple script to crawl through all tables in a database and export to CSV format.
#
# Valcora: http://www.valcora.com
#

#### Begin Configuration ####
DB="mydb"
MYSQL_USER="root"
MYSQL_PASSWD="mypass"
MYSQL_HOST="127.0.0.1"
MYSQL_PORT="3306"
MYSQL="/usr/bin/mysql"
#### End Configuration ####

MYSQL_CMD="$MYSQL -u $MYSQL_USER -p$MYSQL_PASSWD -P $MYSQL_PORT -h $MYSQL_HOST"

TABLES=`$MYSQL --batch -N -D $DB -e "show tables"`
for TABLE in $TABLES
do
  SQL="SELECT * FROM $TABLE;"
  OUTFILE=$TABLE.csv
  $MYSQL --database=$DB --execute="$SQL" | sed 's/\t/","/g;s/^/"/;s/$/"/;s/\n//g' > $OUTFILE
done

As always, be sure to test this script on a non-critical server first to ensure it is working to your satisfaction. There is no error trapping, so be sure you don't accidentally over-write important files!

Converting Storage Engine

nospam@example.com (Valcora) — Mon, 04 May 2009 11:46:21 -0600

Have you ever been asked to convert all of the tables in a given database or on a server to a new storage engine? Well, we have! Sometimes, if it is only a few tables, firing off a few "ALTER TABLE ... ENGINE=InnoDB;" is fine. Other times, it is a nightmare if there are lots of tables and/or databases.

To make our life easier in such times, we created the following Bash script. Save the file as something like "convert_tables.sh" and make sure to make it executable. You can do this in Linux by executing "chmod a+x ./convert_tables.sh." Take a look at the code and we will give you an example of how to run it!

#!/bin/bash
# Crawl through all of the tables in a database or databases and convert all tables to a given storage engine.
# 
# Valcora: http://www.valcora.com
#

#### Begin Configuration ####
DBNAMES="mydb1 mydb2"		# Separate list of Databases by spaces
NEW_ENGINE_TYPE="InnoDB"	# Storage engine to convert all tables to
DBEXCLUDE="information_schema mysql horde bacula vpopmail" # Databases to ignore
MYSQL_HOST="127.0.0.1"
MYSQL_PORT="3306"
MYSQL_USER="root"
MYSQL_PASSWD="mypass"
MYSQL="/usr/bin/mysql"
#### End Configuration ####

# Setup authentication info for mysql client
AUTH="-u $MYSQL_USER -p$MYSQL_PASSWD -P $MYSQL_PORT -h $MYSQL_HOST"

# Get our list of databases to change...
# NOTE: the DBEXCLUDE feature seemed to only work with Linux regex, GNU sed
if [ "$DBNAMES" = "all" ]; then
  DBNAMES="`$MYSQL $AUTH --batch -N -e "SHOW DATABASES"`"
  for i in $DBEXCLUDE
  do
    DBNAMES=`echo $DBNAMES | sed "s/\b$i\b//g"`
  done
fi

# Run through each database and execute our ALTER TABLE command for each table...
for i in $DBNAMES
do
  # to fancy up our log file
  echo ""
  echo "Database: $i"
  echo "---------------------------------------------------------"

  DBTABLES="`$MYSQL $AUTH --batch -N -e "SHOW TABLES" -D $i`"

  for j in $DBTABLES
  do
    echo "ALTER TABLE $j TYPE=$NEW_ENGINE_TYPE;" | $MYSQL $AUTH -D $i
  done
  echo ""
done

If you want to convert all tables in all databases, you could set the "DBNAMES" variable in the configuration section to "all". To convert certain databases, simply set it to a list of database names separated by spaces. And finally to only convert tables in a given database, you could simply list one name there.

We have foudn this useful over the years when implementing some piece of software that we downloaded. We love Open Source too mind you! But, we don't always like their choice of Storage Engines since a lot of developers use MyISAM. When appropriate, we like to run InnoDB. Using this script, it is easy to run through all of the tables and make the conversion in a matter of no time.

Anyway, good luck and as always be sure to test this on a machine that is not critical before implementing. And of yeah, be sure to make a backup first. Just in case...

Finding All MyISAM Tables

nospam@example.com (Valcora) — Mon, 04 May 2009 11:31:05 -0600

Below is a small Bash script to crawl through all the tables in a MySQL database and look for any and all MyISAM tables. We developed this script when we realized that a team of developers were ignoring our requests to create all tables as InnoDB. We set the default storage engine to be InnoDB, however, the team would declare the Storage Enginge to be MyISAM which caused MySQL to ignore our default declaration.

Yes, we could run a "SHOW TABLE STATUS," but that would require staring at a lot of table entries in some databases. Naturally, we could also query the Information Schema database as well. All of those methods are fine. For our purposes, we wanted a fairly simple script that would do the scanning for us lazy types and yet work on pretty well all versions of MySQL.

To solve the problem, we developed this simple script to find any and all tables that were MyISAM format. With that in mind, all of the values are hard-coded into the script to make cronning it simple! You could simply define your cron entry to send the output to an email address.

Save the below code as something like "find_myisam_tables.sh" and make sure it is executable by executing a "chmod a+x find_myisam_tables.sh." Just be sure to chage the configuration values to match your needs and the locations of the binaries.

#!/bin/bash
#
# This is a small bash script that checks all tables in a given database and looks for MyISAM tables.
#
# Written by Valcora
# http://www.valcora.com

DBNAMES=$1

MYSQL="/usr/bin/mysql"
MYSQL_HOST="localhost"
MYSQL_USER="root"
MYSQL_PASSWD="password"
MYSQL_SOCKET="/tmp/mysql.sock"
MYSQLCONNECT="$MYSQL -u$MYSQL_USER -p$MYSQL_PASSWD -h $MYSQL_HOST -S $MYSQL_SOCKET"

echo "=========================================="
echo "=             MyISAM Tables              ="
echo "=========================================="

# Use MySQL 'SHOW DATABASES'
DATABASES="`$MYSQLCONNECT --batch -N -e "show databases"`"

# Loop through each instance of MySQL and check all databases in that instance
for DATABASE in $DATABASES
do
  TABLES="`$MYSQLCONNECT --batch -D $DATABASE -N -e "show tables"`"
  for TABLE in $TABLES 
  do
    STORAGE_ENGINE=`$MYSQLCONNECT -D $DATABASE --batch -N -e "show table status like '$TABLE'" | awk '{print $2}'`
    if [ "$STORAGE_ENGINE" = "MyISAM" ]
    then
      echo "$DATABASE.$TABLE"
    fi
  done
done

To call the script just execute something like:

./find_myisam_tables.sh

Well, feel free to modify the script in any manner you want. You may want to search for other storage engine types also.

Let us know if this script has been of any help. We would love to hear from you!

DBA Scripts

nospam@example.com (Valcora) — Mon, 04 May 2009 11:23:49 -0600

Over the years the DBA's have created simple Bash and PHP scripts to make their life easier. We feel that sharing these with the MySQL community may help our fellow database administrators and make their life a little easier from time to time!

With that in mind, we will periodically post a new script that we hope will be of some value. Some of the scripts are meant to perform regular maintenance activity, routine tasks, and/or query the database schema for certain table types, and such.

Some of the scripts may work as-is, while others may need to be modified to suit your needs. Either way, it should provide a starting point for accomplishing your work.

Naturally, we can make no guarantees as to the effectiveness of these scripts. It should go without saying that you need to always test them on a non-critical server before you attempt to run them in a critical production environment. While, we believe our scripts are safe for most applications, you must be the final judge whether they fit your purpose or not as well as whether they would be safe to run on your systems!

Now that we have our disclaimer out of the way, we hope that you will find these to be as valuable as we have. Enjoy!