Category Archives: System Administration

Troubleshooting SSH keys

SSH is one of those workhorse utilities that make lots of things work. Whole applications are built on top of ssh (for instance, Ansible).  Being able to set up ssh keys is, from a functional perspective, what makes this really work well. Sometimes when setting up SSH keys things fail to work and it is not at all clear why not. A lot of these are fairly simple things that amount to user error. The problem with user error is that it can be hard to detect when you are falling into that.

First..just for completeness.. what is the procedure for setting up ssh keys?  Fortunately this process is well documented on the Interwebz. See this page for instance, which is pretty readable.  Where these pages stop is in troubleshooting (which is why I have this post here.)

So… here’s a list of things to check….

  1. When you copied your public key to the remote host, the permissions on the directory and/or file are too open. SSH requires your .ssh directory to only be readable by the user you are logging in as (well, and root..)
  2. When you try to ssh into your remote server, it fails because you didn’t specify the right key name, or made a typo.
  3. You copied your public key to the server with the name ‘authorized_keys2’ and your ssh daemon was expecting ‘authorized_keys’. (Or vice-versa.)  It seems that the preferred thing now is just to use ‘authorized_keys’.  The exact name is set in the /etc/ssh/sshd_config file with the parameter name ‘AuthorizedKeysFile’.
  4. Often you will have more than one set of ssh keys on your laptop for different servers. Maybe you have one set for work related machines and another for a side project. When generating ssh keys you are prompted to give the filename, and so you do that. So… then you try connecting to your server … and it isn’t working?  “Why am I still being prompted for my password?” Well, you didn’t send the right key.  Again, your local .ssh/config is your friend here (instead of having to specify the key all the time with -i).  Add an entry IdentityFile=~/.ssh/my_server.rsa and on your laptop the appropriate ssh key will be sent to the server.
  5. Bonus points if you set up a different port for your ssh daemon to listen on. An easy mistake to make though is to neglect to use that non-standard port when you connect.  In your ssh config you will want to add this by just specifying the port with simply ‘Port’.

Running sshd manually

If you are still having problems, and if you are able to become root and manually run the sshd daemon in debug mode then you may find that helpful. Then you can watch both the client side and server side interact and see if you get any helpful messages from the daemon.

To run the daemon manually:

/usr/sbin/sshd -p 8888 -D -d -e

Update

So with OpenSSH 7 DSA keys have been deprecated by default.  If you suddenly find that you are prompted for a password or you just plain can’t get in at all then this may be the problem.  If you run ssh with the -v verbose flag then you may see this message:

Skipping ssh-dss key .ssh/whatever_id_dsa - not in PubkeyAcceptedKeyTypes

(I happened to notice this after I upgraded to Mac OS X Sierra recently.)

So, you can override this behavior by adding a line in your ssh config on your own machine  by adding a ‘PubkeyAcceptedKeyTypes‘ line…

Host myserver.org
     User me
     PubkeyAcceptedKeyTypes +ssh-dss
     IdentitiesOnly yes
     IdentityFile=~/.ssh/myserver_id_dsa
     Hostname=myserver.org
     TCPKeepAlive yes
     ServerAliveInterval 15

But probably the better thing to do is to regenerate your ssh keys and use RSA instead.  Of course you might temporarily need to add this override in order to get back into your account to update the key 😉

 

Failing to write out a file from MySQL — a tale of systems behavior

Sometimes you get into a situation where two separate issues conspire to drive you to question your sanity. For me this recently happened when I was trying to do a pretty simple operation: I just wanted to write out a result set to a .csv file from MySQL into the local file system. The problem (initially, anyway) was that I was logged into MySQL, ran the command, which worked without complain.. but when I went to look in the /tmp directory for my file… nada.

So, tried the command again just to see what would happen. Now MySQL complained (quite rightly) that the file already existed, yet I didn’t see it.

At this point the problem was that this particular machine I was on had a newer Fedora Linux install on it and one of the changes made was to add private tmp directories. It looks like that feature was added in Fedora 16 and they expanded the list of services that by default use private tmp directories in Fedora 17. Basically a subdirectory under /tmp will get created with some random string, and that’s where, in this case, my file was getting written to. There’s definitely a logic to that, but if you weren’t aware of that and/or didn’t have that in mind it is an easy thing to miss. And truth be told other folks have written about this in various places (such as this discussion post and this blog post I found later.) At least it wasn’t just me…

Initially I didn’t realize the problem of the private tmp directories though. My next step was just to try to write out my .csv file to a couple of other directories on the system. I set up a /mytmp directory and /usr/local/mytmp with appropriate permissions, but MySQL complained that it couldn’t write to the system. Huh? At this point I was confused… what was going on?

So, problem number two was that I didn’t realize SE Linux was enforcing limitations on where services, such as the MySQL daemon, could write to. This message gets logged to /var/log/messages as well, which is how I identified what was going on here. (This too has been commented on in this discussion post and elsewhere.)

In this particular case, the solution for me was to put SELinux into permissive mode, which then allowed MySQL to write out the my file somewhere I could have another script pick it up.

At the end it was clear that these two problems, although separate, did converge in my particular use case. This is an example of Apparently pathological behavior in systems is the result of different components doing exactly what they are supposed to do. Once you understand each component and its behaviors the system makes more sense.

Setting up and using Cron

Cron jobs are a mainstay of Unix and Linux. Cron basically lets you run a shell script or other program according to some schedule. The system has been around forever and it just plain works.. even if it is just a little on the cryptic side. The system is ubiquitous and so widely used. A common use of this is to automate backups, for instance. When administering a server I would say it is of the things to think about when setting up a system — what kinds of tasks will you need to run when? (This will, of course, evolve over time. But still….)

Pretty much every implementation of Linux/Unix does the same thing. There is a cron daemon that will go through all the cron files on the system and check in every minute to see what it needs to run. You can’t really do cron jobs any more precisely than at the minute level — there is way to say ‘run a script at 5:33:45 on Monday’ for instance (and why would you ever want to do that anyway?)

Basic Usage

Cron operates on crontab files. You don’t access these files directly to view or edit them. The system maintains these in some location. Whenever you want to do anything with a crontab file, you use the crontab command.

At the command line you can view your crontab with crontab -l. This simply prints out your crontab to stdout. If you haven’t set one up yet then you’ll most likely see a message saying something like ‘crontab: no crontab for you’ (where ‘you’ is your username on the system, of course.)

In general every user with an account has access to setting up cron jobs. To do this, run ‘crontab’ with the -e flag to edit your crontab file. That will kick you into your editor (whichever it is set to, or vi by default.) Initially this will be empty but here we’ll show you how to set that up.

[me@myserver ~]$ crontab -e

However, if you don’t want to edit the crontab file interactively, you can also edit the file in whatever editor you choose and set it up with the crontab command by giving the filename. Usually when doing that I check that I loaded in what I thought by running crontab -l right after.

[me@myserver ~]$ crontab my_crontab.txt

In a crontab file lines that start with a hash mark (‘#’) are comments. The cron program will ignore those lines.

One entry in a crontab file is set up per line. This is a space-delimited list that has the information about what program or script to run and how often. The format is:

mm hh dd tt ww command

Where each field is defined as such:

mm Number of minutes past the hour to run the cron job.

(having the minutes field first is usually the first thing that throws you off. We’re so used to using the hour figure first whenever we talk about time.)
hh Hour of day to run the cron job.
dd day of month (00 to 31)
tt month (1 to 12)
ww day of week (0 to 6, with 0=Sunday, 1=Monday, etc.)
command the absolute path to the script/program to run, along with any redirection to send output wherever.

The basic usage is pretty straight-forward. For example, we might want to run a script at 10pm every night…

0 22 * * * /backup/bar.sh > /dev/null 2>&1

Where cron can get a bit cryptic is when you want to run things multiple times a day or every few days. To run a command at two different times, say 10am and 10pm, you could put each hour value separated by a comma:

# Run foo.sh at 10:15am and 10:15pm every night and send the output to
* /dev/null (ie. throw it away)
15 10,22 * * * /home/me/foo.sh > /dev/null 2>&1

Or to run a command every four hours you would use an asterisk, but then specify the multiple of ‘every four hours’ with ‘/4’.

# Run foo.sh every 4 hours
0 */4 * * * /home/me/foo.sh > /dev/null 2>&1

You can also specify a range of values with a hyphen. In practice this probably only every makes sense in the column indicating which days of the week to run a script.

# Run bar.sh at 5:30pm every workday (mon-fri), and save the output to a logs directory…
30 17 * * 1-5 /home/me/bar.sh >> /home/me/logs/bar.log 2>&1

Version control of your crontab files

Since you can both load a crontab from a text file as well as list it out easily, it is not too much work to set up crontab files under a version control system (ie. git or subversion). Then as you add more things to your crontab you’ll have a log of these changes over time.

Considerations for scripts run via cron

There’s nothing to say you can’t run a compiled program with cron — we do this often enough. But probably the main use of cron is to run a script of some kind — BASH, Python, Perl, PHP scripts can all be run via cron.

1. At the risk of stating something kinda obvious, your script/program should start up, do it’s thing, and exit. It shouldn’t hang around running forever. If you want a daemon, set up a daemon (that’s a future blog post.)

2. Your script should use absolute paths — you can’t assume the script is run from your home directory, for instance.

3. Some jobs might take a long time to run. You should take this into account so that one invocation of your script does not overlap the next invocation by cron. Putting in a check of some kind — a sentinel or lock file — to see if your script is still running can be effective. Or, if they do overlap, that each instantiation of your script can coexist happily with the others.

*** Update: (2013-03-15) Just came across a link to Chronos, which is a replacement for Cron developed by the folks at AirBnB. I haven’t looked at it much yet but wanted to throw in a link to that here.