Many of us at Valcora were "fortunate" to start our tech careers in a small web development shop where we were expected to be "jack-of-all-trades." We say "fortunate" because at the time, it doesn't seem so great. Looking back, however, it is clear that we got a lot of experience that was invaluable that way.
We were expected to be Linux System Administrators, handle user accounts, Linux installs, manage DNS, install/configure/manage Apache, do custom PHP programming, MySQL Database Administrator, search engine optimization, and on and on...
As we have worked with various companies over the years, we all have realized that not everyone else grew up in that kind of environment. The normal that we have encountered is highly specialized individuals who know one thing and know it fairly well. While there is absolutely nothing wrong with that in many cases, it can be a crutch and a short-coming for anyone wanting to be a senior-level MySQL DBA in our opinion. Why? Well, frankly, because a senior-level DBA should know a lot more than just MySQL.
After chatting with the guys here at Valcora, we all realized that we have been called upon in our careers to do pull some fairly heavy Linux tricks out of our "bag of tricks." Below are just a few of the things we have had to do and honestly, we are not sure a specialized junior would know what to do in most of these cases.
We were reminiscing and thought of the time a client wanted to see the queries that are passing through their production MySQL server. At this point, a junior might be tempted to say, well let's turn on the general query log, but that is not an option when you are running MySQL 4.1 and it would require a server restart! The DBA who had setup the server did not symlink the general query log to /dev/null so that it could be un-linked and a "flush logs" performed to start capturing logs. (Some senior DBAs might have missed that trick.)
The only option was to try and capture statements with some other method. After confirming with the client that they needed SELECT statements as well as writes (DML), we came to the conclusion that we were going to have to "sniff" for those. Doing a poll of the "SHOW PROCESSLIST" every second was not a n option since so many would be missed. So, we decided to use tcpdump and capture the queries with some serious grep statements. Thinking that was the job of the Linux System Administrator? Guess again. It was the DBA's job in this company.
Right about now, if you are thinking that is too heavy duty for a MySQL DBA, remember we are talking about real senior-level DBA's.
We were able to capture the data for a fixed time period and then massage it with more Linux commands into a report of the queries. For those of you who know about it, this was before the days of mysqlsniffer, which is a great little utility to do basically the same thing! We wish we had it at that time and it would have made our job easier!
On other occassions, we all seen at one time or another a problem where we try to start MySQL and Linux reports that the port is already in use. Now, a reboot, again, is a simple solution to this problem. But, in some cases, a reboot is not permitted without management approval and if they happen to be running anything else on that server, good luck getting downtime to perform one. Since there is a process that has that port, we need to find out what is holding onto it. (That is a rather straight forward process which we will detail in another blog coming soon.) Again, Linux skills to the rescue. Find the process and more than likely it will be one that you can kill since it probably shouldn't be using a port used for MySQL such as 3306 and then start MySQL up with no problems. No waiting for approvals or having to explain to some manager, why you need to reboot the server.
Custom scripting is another area that deserves mention. We all recalled how many Bash, Perl, and PHP scripts we have written over the years to perform some maintenance task in MySQL. In the early days, MySQL did not always have scripts/utilities like mysqlcheck for example. And speaking of that, one of us commented how we decided to use mysqlcheck to analyze a production server that had been rebooted in a not-so-graceful manner by a System Administrator without shutting down MySQL properly. This was like a power down by pulling the plug! Anyway, a few of the InnoDB tables were crashed and mysqlcheck just skipped them and never even reported a problem. Looking at its "report" made us think that everything was ok, but we knew better.
Bash scripting to the rescue. We crawled through every database and table and performed "CHECK TABLE" on each with a simple Bash script. We submitted a bug report to MySQL and it is supposed to have been resolved by now. While it probably is safe to use the newer version of mysqlcheck, some of us old-timers are still cautious enough to think our simple Bash script found the problem for sure and is likely to always find the problem, so why change to something that has proven to have problems?
One of the recurring things, we all joked about was how many times we have battled library issues when installing MySQL. We have fought with Perl DBI and DBD libs as well as Linux libraries such as glibc and such when compiling MySQL from source. Yes, some of us still like to compile from source for a variety of reasons, though we also love binaries.
In some companies, it is also helpful to be able to diagnose firewall issues. We have seen mis-configured firewall rules before where someone inadvertantly missed a rule and the application server could not talk with the database server. Since in most companies, the general assumption is that "it must be a database problem," (The senior DBAs are nodding their heads now!) it is critical to be able to diagnose networking issues since we are often guilty until proven innocent! If you as a DBA have to wait on a networking person to clear you, good luck! The sooner you can troubleshoot and discover latency or connectivity issues in the network, the sooner you get off the "hot seat" and look like a hero in finding the problem. Most juniors in this case, are probably thinking "But, that is not my job! I am the DBA!" The old-timers realize that it takes more than MySQL skills!
We have also seen development teams establish connections to the server that seem to terminate unexpectedly. After having MySQL be blamed by the development team, it took some special skills to start examining the JBoss and Tomcat configurations. Naturally, the individuals responsible for those applications also afrgued it was a MySQL "problem." But we knew better! As you can expect, the problem was with the configuration of connection pooling and timeouts in their configurations. When we notified them of what needed to be changed and why it was a problem, they finally conceded that this was the real problem. We tested it and voila, problem solved! Again, not a MySQL problem, but it was critical for us to diagnose it and clear our name.
Anyway, we could go on and on, but surely you get our point! To be a real DBA, we think you have to be well-rounded and be able to perform a lot of Linux Administrator skills to survive in a LAMP environment...
If you are not a senior DBA, don't worry. Find yourself a good senior and partner with them. Soak up all you can from them like a sponge. Then, one day, perhaps you too can call yourself a senior...
If you are a senior-level DBA, find yourself a willing junior to take under your wing and return the favor...
We were expected to be Linux System Administrators, handle user accounts, Linux installs, manage DNS, install/configure/manage Apache, do custom PHP programming, MySQL Database Administrator, search engine optimization, and on and on...
As we have worked with various companies over the years, we all have realized that not everyone else grew up in that kind of environment. The normal that we have encountered is highly specialized individuals who know one thing and know it fairly well. While there is absolutely nothing wrong with that in many cases, it can be a crutch and a short-coming for anyone wanting to be a senior-level MySQL DBA in our opinion. Why? Well, frankly, because a senior-level DBA should know a lot more than just MySQL.
After chatting with the guys here at Valcora, we all realized that we have been called upon in our careers to do pull some fairly heavy Linux tricks out of our "bag of tricks." Below are just a few of the things we have had to do and honestly, we are not sure a specialized junior would know what to do in most of these cases.
We were reminiscing and thought of the time a client wanted to see the queries that are passing through their production MySQL server. At this point, a junior might be tempted to say, well let's turn on the general query log, but that is not an option when you are running MySQL 4.1 and it would require a server restart! The DBA who had setup the server did not symlink the general query log to /dev/null so that it could be un-linked and a "flush logs" performed to start capturing logs. (Some senior DBAs might have missed that trick.
The only option was to try and capture statements with some other method. After confirming with the client that they needed SELECT statements as well as writes (DML), we came to the conclusion that we were going to have to "sniff" for those. Doing a poll of the "SHOW PROCESSLIST" every second was not a n option since so many would be missed. So, we decided to use tcpdump and capture the queries with some serious grep statements. Thinking that was the job of the Linux System Administrator? Guess again. It was the DBA's job in this company.
Right about now, if you are thinking that is too heavy duty for a MySQL DBA, remember we are talking about real senior-level DBA's.
We were able to capture the data for a fixed time period and then massage it with more Linux commands into a report of the queries. For those of you who know about it, this was before the days of mysqlsniffer, which is a great little utility to do basically the same thing! We wish we had it at that time and it would have made our job easier!
On other occassions, we all seen at one time or another a problem where we try to start MySQL and Linux reports that the port is already in use. Now, a reboot, again, is a simple solution to this problem. But, in some cases, a reboot is not permitted without management approval and if they happen to be running anything else on that server, good luck getting downtime to perform one. Since there is a process that has that port, we need to find out what is holding onto it. (That is a rather straight forward process which we will detail in another blog coming soon.) Again, Linux skills to the rescue. Find the process and more than likely it will be one that you can kill since it probably shouldn't be using a port used for MySQL such as 3306 and then start MySQL up with no problems. No waiting for approvals or having to explain to some manager, why you need to reboot the server.
Custom scripting is another area that deserves mention. We all recalled how many Bash, Perl, and PHP scripts we have written over the years to perform some maintenance task in MySQL. In the early days, MySQL did not always have scripts/utilities like mysqlcheck for example. And speaking of that, one of us commented how we decided to use mysqlcheck to analyze a production server that had been rebooted in a not-so-graceful manner by a System Administrator without shutting down MySQL properly. This was like a power down by pulling the plug! Anyway, a few of the InnoDB tables were crashed and mysqlcheck just skipped them and never even reported a problem. Looking at its "report" made us think that everything was ok, but we knew better.
Bash scripting to the rescue. We crawled through every database and table and performed "CHECK TABLE" on each with a simple Bash script. We submitted a bug report to MySQL and it is supposed to have been resolved by now. While it probably is safe to use the newer version of mysqlcheck, some of us old-timers are still cautious enough to think our simple Bash script found the problem for sure and is likely to always find the problem, so why change to something that has proven to have problems?
One of the recurring things, we all joked about was how many times we have battled library issues when installing MySQL. We have fought with Perl DBI and DBD libs as well as Linux libraries such as glibc and such when compiling MySQL from source. Yes, some of us still like to compile from source for a variety of reasons, though we also love binaries.
In some companies, it is also helpful to be able to diagnose firewall issues. We have seen mis-configured firewall rules before where someone inadvertantly missed a rule and the application server could not talk with the database server. Since in most companies, the general assumption is that "it must be a database problem," (The senior DBAs are nodding their heads now!) it is critical to be able to diagnose networking issues since we are often guilty until proven innocent! If you as a DBA have to wait on a networking person to clear you, good luck! The sooner you can troubleshoot and discover latency or connectivity issues in the network, the sooner you get off the "hot seat" and look like a hero in finding the problem. Most juniors in this case, are probably thinking "But, that is not my job! I am the DBA!" The old-timers realize that it takes more than MySQL skills!
We have also seen development teams establish connections to the server that seem to terminate unexpectedly. After having MySQL be blamed by the development team, it took some special skills to start examining the JBoss and Tomcat configurations. Naturally, the individuals responsible for those applications also afrgued it was a MySQL "problem." But we knew better! As you can expect, the problem was with the configuration of connection pooling and timeouts in their configurations. When we notified them of what needed to be changed and why it was a problem, they finally conceded that this was the real problem. We tested it and voila, problem solved! Again, not a MySQL problem, but it was critical for us to diagnose it and clear our name.
Anyway, we could go on and on, but surely you get our point! To be a real DBA, we think you have to be well-rounded and be able to perform a lot of Linux Administrator skills to survive in a LAMP environment...
If you are not a senior DBA, don't worry. Find yourself a good senior and partner with them. Soak up all you can from them like a sponge. Then, one day, perhaps you too can call yourself a senior...
If you are a senior-level DBA, find yourself a willing junior to take under your wing and return the favor...