Another issue that popped up tonight. The time on a payroll server seem to be slower then usual. Futher troubleshooting on the box revealed that it took 4 seconds to move 1 second on the server. This caused the payroll servers to stop communicating between each other as time sync was part of a security measure built into the payroll software we run here.
Some googling later it seem to be a BUG with the kernel. The fix, suggested to either update the kernel, recompile the kernel or add some kernel parameters in GRUB to fix the issue. I decided to go with the kernel parameters because this was a production server and the downtime window was very slim.
The fix,
- vi /etc/boot/grub.conf
- Add to the end of the kernel line. “clock=pit noapic nolapic”
- reboot and check time. “watch date”
grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/md7
# initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=10
splashimage=(hd0,0)/grub/splash.xpm.gz
title Red Hat Linux (2.4.9-e.3smp)
root (hd0,0)
kernel /vmlinuz-2.4.9-e.3smp ro root=/dev/md7 clock=pit noapic nolapic
initrd /initrd-2.4.9-e.3smp.img
title Red Hat Linux Advanced Server (2.4.9-e.3)
root (hd0,0)
kernel /vmlinuz-2.4.9-e.3 ro root=/dev/md7 clock=pit noapic nolapic
initrd /initrd-2.4.9-e.3.img
Source: Rexiology::Work
Source: Linux Kernel Parameters