I’ve tried and debug kernel crashes happening on one system, by sending its console messages to a remote machine.
I hope it will enable me to see what’s happening as it crashes, since most of the oops are lost when I look at the physical console.
Anyway, in order to do that, I configured the crashing machine’s kernel to use the netconsole
module to output console messages on the network.
Here’s the setup in /etc/modules on the crashing machine (whose IP is 192.168.0.2) :
netconsole netconsole=6665@192.168.0.2/eth0,6666@192.168.0.1/00:12:34:56:78:90
It will instruct the netconsole module to dump the logs on port 6666 on system 192.168.0.1 whose MAC address is 00:12:34:56:78:90.
On the logging machine (192.168.0.1), I installed syslog-ng to be able to dump the logs coming from netconsole into a log file. Here’s the setup :
source netlog {
udp(ip(0.0.0.0) port(6666));
};
destination netlog
{
file("/var/log/netconsole.log");
};
log
{
source(netlog);
destination("netlog");
flags(final);
};
I had tried previously to use cancd, but i prefer to keep all logs in one single file.
Hope this helps
FYI, it worked, and helped me see some details on today morning’s crash on the server… but unfortunately not everything can be seen…
Anyway, at least it gives me an idea of the time at which it crashes 😉