$ killall mysqld $ while pgrep mysqld; do sleep 5; done # waiting for mysql to s...

Nursie · on July 26, 2013

This would reboot it but might not achieve much-

  $ killall mysqld
  /usr/bin/killall - file not found
  $ while pgrep mysqld; do sleep 5; done
  /usr/bin/pgrep - file not found
  $ echo s >/proc/sysrq-trigger
  $ watch dmesg
  /usr/bin/watch - file not found
  $ echo b >/proc/sysrq-trigger

Unfortunately in this situation only the bash builtins seem to work. I even had an old system drive mounted under /altroot, but it wouldn't execute any of the commands in /altroot/bin or /altroot/usr/bin either (can't find something it needs to execute stuff. ld? libc? unknown).

So really only the 'b' seems to help. (edit: 's' is probably useful, even if you can't kill everything)

Of course the long term solution is to stop using a USB stick on an unreliable/badly supported USB3 add-on card for the root drive :)

Hello71 · on July 26, 2013

Oh, I see. Okay then.

  $ pgrep() { for proc in /proc/[0-9]*; do read CMDLINE < $proc/cmdline; if [[ ${CMDLINE/*$1*/x} == x ]]; then echo ${proc#/proc/}; return 0; fi; done; return 1; }
  $ kill $(pgrep mysqld)
  $ while pgrep mysqld; do sleep 5; done
  $ echo s >/proc/sysrq-trigger
  $ while sleep 2; do dmesg | tail; done
  $ echo b >/proc/sysrq-trigger

Nursie · on July 26, 2013

I admire your tenacity, but I think sleep, dmesg and tail are all out as well!

Though all could b mitigated by leaving it for a minute or two at the relevant point, and manually checking for the mysql process every so often. I'll keep this on file for next time it happens (though I'm hoping to replace the dodgy parts at the weekend).

Hello71 · on July 26, 2013

grumble grumble

  $ pgrep() { for proc in /proc/[0-9]*; do read CMDLINE < $proc/cmdline; if [[ ${CMDLINE/*$1*/x} == x ]]; then echo ${proc#/proc/}; return 0; fi; done; return 1; }
  $ kill $(pgrep mysqld)
  $ pgrep mysqld; echo $?
  $ echo s >/proc/sysrq-trigger
  $ while read LAST; do; done < /proc/kmsg; echo $LAST
  $ echo b >/proc/sysrq-trigger

Nursie · on July 26, 2013

You, sir or madam, are awesome and very resourceful. I'll use this one :)

cnvogel · on July 27, 2013

If /lib/ld.so is hosed and you cannot run any dynamically linked program, then you might be able to start programs in /altroot by running something similar to...

  LD_LIBRARY_PATH=/altroot/lib:/altroot/usr/lib  /altroot/lib/ld.so /altroot/bin/sh

...so that you run the dynamic linker directly, as a wrapper program.

Nursie · on July 29, 2013

That's another good tip, thanks for that.

I'm hoping not to have this situation happen again, mind!