This is a post by guest blogger Wesley David. You can find more information about Wesley at the end of the article.
I have in two articles introduced you to SysRq and shared with you various ways it can be used. Now, in the last part of the series, I turn to troubleshooting SysRq with topics including what do you do if you don’t have a SysRq key, and limiting the capabilities of SysRq.
Do some keystroke hokey-pokey
Sometimes, SysRq just doesn’t seem to work. If it doesn’t, you’ll want to first check which architecture you’re running. Perhaps you’re on a PowerPC based machine when you thought it was Intel. Check with ‘uname’ to see if you’re on what you think you’re on. Hey, sometimes you get to pick low hanging fruit and eat it too!
Alas, not all SysRq failures are that easy to figure out. You might have to do some keystroke hokey-pokey to see if your hardware can only handle a limited amount of keystrokes. Yes, in some cases three simultaneous key presses cannot be handled by the hardware itself! If you’ve tried using Magic SysRq keystrokes without success, attempt various combinations of pressing and releasing the keys before giving up. For example, press Alt and SysRq (or Break, or F13, or whatever your architecture requires in lieu of the SysRq key) and then let go before pressing the single and final key to complete the SysRq keystroke. For example, press Alt and SysRq before letting go and tapping ‘b’ to forcibly reboot. Or perhaps try pressing and releasing Alt before pressing and releasing SysRq and then finally pressing and releasing ‘b’.
Involve a function key
Furthering the possible combinations, you may also have to press a function key before using the SysRq key itself. Some keyboards have SysRq as an alternate function on a shared key so you’ll need to experiment with pressing Function+alt+SysRq-[key] in any number of combinations and press/release patterns to see what will work.
Often I suggest that people test the Magic SysRq feature by using the help function. That is, Alt+SysRq+h. It’s possible that you won’t see any output in an open console, however. In those cases, SysRq may be working, but output isn’t visible to you. In some distributions of Linux, the kernel loglevel is too low to display SysRq output.
To see if that’s the case, check to see what SysRq messages are in the kernel buffer with the command ‘dmseg | grep –i sysrq’. If you see a message there but not one immediately printed to your screen after pressing the Magic SysRq sequence, then your log level is set to not show kernel messages to the console.
To change that, you can echo a number from 0 to 9 to the sysrq-trigger node in procfs. The number 0 means the least amount of messages make it to the kernel. A deeper discussion of kernel logging levels is outside of the scope of this article. However, if you see SysRq messages in dmesg’s output, just know that SysRq is likely working, just not immediately visible in your console and that you’ll need to set the loglevel to something higher. Check with the documentation for your Linux distribution.
What if you don’t have a SysRq key?
Sometimes a keyboard doesn’t have a SysRq key or for whatever reason your kernel isn’t responding with the keycode that has been traditionally used for the SysRq key. That traditional keycode is ‘99’. To see what value a key is triggering, use the command ‘showkey -k’ and start hammering on the keyboard to find a key that sends keycode 99.
For example, I have the Logitech Illuminated Keyboard, which has no SysRq key. Or is there? There’s a Print Screen key, and sure enough, showkey tells me that it sends the keycode ‘99’ so that’s my effective SysRq key.
In case no key on your keyboard seems to send the proper keycode, you can map a scancode to keycode 99 using the ‘setkeycodes’ command. To find a key’s scancode, run `showkey –s` and press a key. You will get the hexadecimal code for the key’s downstroke as well as upstroke. You’ll want to note the scancode for the downstroke only.
Once you know the scancode for the key you want to turn into an unofficial SysRq key, you can run the `setkeycodes` command to map the scancode to a kernel keycode (remember, we want to trigger keycode 99). As a result, the following command would map my ‘home’ key to keycode 99 and thus make it an alternate SysRq key: `setkeycodes e047 99`
Of course, remapping keycodes can cause some unintended consequences so you might want to run `dumpkeys` to find unused keycodes or at least keep track of which keycodes you’re overwriting. In fact, if you’re having trouble finding a key mapped to keycode 99, grep though dumpkeys’s output for clues.
And finally, every good SysAdmin documents each and every process so when you figure out what works for which systems that you are responsible for, do yourself and any other admins a favor and write it down in the centralized documentation system of your team’s choice.
Limiting SysRq’s Capabilities
Before I go any further with the topic of “limiting” SysRq, let’s be clear on one thing. If your kernel is compiled with SysRq support, then anyone with root access can trigger it manually if you have procfs mounted. Certainly anyone with root access can also do a litany of other bad things, but I digress too far. What little ability one has to limit Magic SysRq keystrokes is virtually useless, and I can only think of a few reasons to use it. Nevertheless, you can decide for yourself if the following feature is useful to you.
To check if SysRq is enabled, you can inspect /proc/sys/kernel/sysrq. However, the sysrq node can hold 8 bits of information. It’s not just an on or off proposition. The full list of SysRq operations that can be allowed or disallowed is in the SysRq documentation, but I’ll reproduce them here for ease. Keep in mind that the following are simple bitmasks:
- 2 – enable control of console logging level
- 4 – enable control of keyboard (SAK, unraw)
- 8 – enable debugging dumps of processes etc.
- 16 – enable sync command
- 32 – enable remount read-only
- 64 – enable signalling of processes (term, kill, oom-kill)
- 128 – allow reboot/poweroff
- 256 – allow nicing of all RT tasks
If you’d like to enable debugging dumps and allow the signaling of processes but disallow everything else, then enable SysRq with the value of 72. You can do that through the temporary means of echoing the number to /proc/sys/kernel/sysrq or the persistent means of sysctl and sysctl.conf mentioned in the previous article.
Now any Magic SysRq keys that do not fit within that scope will fail with the error “SysRq: This sysrq operation is disabled.” (of course, you’ll only see that if console logging is set at an acceptable number so you might have to grep through dmesg’s output to check)
The only somewhat useful scenario that I’ve been able to come up with is if you’re worried that someone might trigger a Magic SysRq keystroke at a physical machine (say a server racked up in a datacenter being accessed via a KVM switch by a local technician). With this feature, you can enable only the hardware-triggered commands that you’re willing to suffer the consequences of. Remember that Magic SysRq keystrokes will work at the login screen so even if a user doesn’t have account information, if they’ve got physical access they can potentially SIGKILL everything or cause a nasty crash with Alt+SysRq+c. Of course, physical access means there are a million and one other means of causing destruction as well, so limiting SysRq seems a tad bit silly at that point.
Jane, Stop This Crazy Thing!
So are you just fed up with everything SysRq? Want to make it all go away? It’s possible! Just not trivial.
Most modern Linux distributions include support for the Magic SysRq keystrokes compiled into the base kernel. If you don’t want the option to use SysRq in any form, whether by keystrokes or procfs, you’re going to need to recompile your kernel and answer “no” to CONFIG_MAGIC_SYSRQ. Details on how to perform that fall well outside the scope of this post.
echo “k” > /pingdomblog/sysrq-article
The limit of my experience with and knowledge of SysRq has finally been reached. SysRq is very powerful and can be quite helpful in tough situations. If you learn about its quirks, experiment with its capabilities and respect its consequences it has the potential to rescue you from some bad situations. It has even more potential to get you into a bad situation.
Do you have any experiences with SysRq? Has it saved your hide? Do you have anything else to add? Talk to me in the comments below!
About Wesley David
Wesley David is a systems administrator that works on anything that plugs into a wall. You can follow his misadventures on his blog www.TheNubbyAdmin.com or if you have a short attention span, his Twitter account @Nonapeptide.
