namad crash, but not sure :-)

[GNEEOT]

Hi guys,

Recently had problem with my Sarge firewall box. It just stopped answering to DNS requests from other machines on LAN. When I noticed, I tried to restart named daemon using /etc/init.d/named restart, but it just couldn't stop it. Then I killed this daemon using kill command and started it again. Everything started working again, but on next day I received mail from LogWatch and found there:

1 Time(s): 00000000 3bf00020 00000000 00000000 00000020 00000145 3bf00020 cbb20000
1 Time(s): c0db5414 c0db540c c0db5404 c0db5428 c0db5420 c0db5418 7fffffff 00000001
1 Time(s): [__pollwait+0/208] __pollwait+0x0/0xd0
1 Time(s): [do_select+607/720] do_select+0x25f/0x2d0
1 Time(s): [sys_select+731/1264] sys_select+0x2db/0x4f0
1 Time(s): [syscall_call+7/11] syscall_call+0x7/0xb
1 Time(s): printing eip:
1 Time(s): *pde = 00000000
1 Time(s): CPU: 0
1 Time(s): Call Trace:
1 Time(s): Code: 08 8b 44 24 18 83 ea 24 8b 5a 08 89 44 24 08 89 54 24 04 89
1 Time(s): EFLAGS: 00010a47 (2.6.8-3-686)
1 Time(s): EIP is at sock_poll+0x10/0x40
1 Time(s): EIP: 0060:[sock_poll+16/64] Not tainted
1 Time(s): Modules linked in: usbhid bridge ipt_TOS i830 nfsd exportfs lockd sunrpc lp ipv6 ipt_MASQUERADE ipt_REDIRECT ipt_REJECT ipt_LOG ipt_state ipt_pkttype ipt_recent ipt_iprange ipt_physdev ipt_multiport ipt_conntrack iptable_mangle ip_nat_irc ip_nat_tftp ip_nat_ftp iptable_nat ip_conntrack_irc ip_conntrack_tftp ip_conntrack_ftp ip_conntrack iptable_filter ip_tables af_packet analog parport_pc parport floppy pcspkr rtc 8139cp snd_intel8x0 snd_ac97_codec snd_pcm snd_timer snd_page_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd shpchp pciehp pci_hotplug intel_agp agpgart ehci_hcd uhci_hcd usbcore i810_audio ac97_codec soundcore 8139too mii quota_v2 tsdev mousedev capability commoncap evdev tun psmouse ide_cd cdrom ext3 jbd mbcache dm_mod ide_generic piix ide_disk ide_core unix font vesafb cfbcopyarea cfbimgblt cfbfillrect
1 Time(s): Oops: 0002 [#1]
1 Time(s): PREEMPT
1 Time(s): Process named (pid: 5158, threadinfo=cbb20000 task=cafff7f0)
1 Time(s): Stack: ce05c280 ca8e7080 ce0c9980 ce0c9900 c0168d2f ce9bb280 00000000 00000000
1 Time(s): Unable to handle kernel paging request at virtual address 51b3d6c4
1 Time(s): c0217b70
1 Time(s): device eth0 entered promiscuous mode
1 Time(s): device eth0 left promiscuous mode
1 Time(s): ds: 007b es: 007b ss: 0068
1 Time(s): eax: cb8bec58 ebx: ce9bb280 ecx: ce9bb280 edx: c03010a0
1 Time(s): esi: c03010a0 edi: 00000014 ebp: 00000014 esp: cbb21ee0
1 Time(s): eth0: Promiscuous mode enabled.

Don't really understand what is it all about. Can anyone help me please? Don't want it happen again, it stopped work at office for half a day.

Thank you in advance!

[touhidmustakhim]

sorry my friend i am just now started with linux so if you get any options or any answers for this type of questions please send it for me also thank you for your information

[pandora]

It looks like named couldn't be paged out to the swap partition. Is your server under high load? What is the load avaerage given with the "uptime" and "top" commands?

1 Time(s): Process named (pid: 5158, threadinfo=cbb20000 task=cafff7f0)
1 Time(s): Stack: ce05c280 ca8e7080 ce0c9980 ce0c9900 c0168d2f ce9bb280 00000000 00000000 
1 Time(s): Unable to handle kernel paging request at virtual address 51b3d6c4

Could be bad memory too, but that's less likely. Do you see this in the /var/log/messages file too? There could be other important messages there that you are missing.

[GNEEOT]

# uptime

 14:23:55 up 4 days,  5:23,  3 users,  load average: 0.00, 0.02, 0.02

top - 14:24:19 up 4 days,  5:24,  3 users,  load average: 0.00, 0.02, 0.02

How can I check memory?

/var/log/messages:

Jan 22 11:17:47 tenpostadsl nagios: Auto-save of retention data completed successfully.
Jan 22 11:24:08 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=41.243.98.82 DST=62.221.52.196 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=26970 DF PROTO=TC
P SPT=3033 DPT=25 WINDOW=65535 RES=0x00 SYN URGP=0
Jan 22 11:24:11 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=41.243.98.82 DST=62.221.52.196 LEN=48 TOS=0x00 PREC=0x00 TTL=116 ID=27007 DF PROTO=TC
P SPT=3033 DPT=25 WINDOW=65535 RES=0x00 SYN URGP=0
Jan 22 11:24:44 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=124.38.246.5 DST=62.221.52.196 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=51612 DF PROTO=TCP
 SPT=51260 DPT=25 WINDOW=5840 RES=0x00 SYN URGP=0
Jan 22 11:24:47 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=124.38.246.5 DST=62.221.52.196 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=51613 DF PROTO=TCP
 SPT=51260 DPT=25 WINDOW=5840 RES=0x00 SYN URGP=0
Jan 22 11:24:53 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=124.38.246.5 DST=62.221.52.196 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=51614 DF PROTO=TCP
 SPT=51260 DPT=25 WINDOW=5840 RES=0x00 SYN URGP=0
Jan 22 11:25:07 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=124.38.246.5 DST=62.221.52.196 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=51615 DF PROTO=TCP
 SPT=51260 DPT=25 WINDOW=5840 RES=0x00 SYN URGP=0
Jan 22 11:25:29 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=124.38.246.5 DST=62.221.52.196 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=51616 DF PROTO=TCP
 SPT=51260 DPT=25 WINDOW=5840 RES=0x00 SYN URGP=0
Jan 22 11:26:01 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=124.125.89.164 DST=62.221.52.196 LEN=48 TOS=0x00 PREC=0x00 TTL=113 ID=23392 DF PROTO=
TCP SPT=63192 DPT=25 WINDOW=65535 RES=0x00 SYN URGP=0
Jan 22 11:26:04 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=124.125.89.164 DST=62.221.52.196 LEN=48 TOS=0x00 PREC=0x00 TTL=113 ID=23431 DF PROTO=
TCP SPT=63192 DPT=25 WINDOW=65535 RES=0x00 SYN URGP=0
Jan 22 11:26:10 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=124.125.89.164 DST=62.221.52.196 LEN=48 TOS=0x00 PREC=0x00 TTL=113 ID=24504 DF PROTO=
TCP SPT=63192 DPT=25 WINDOW=65535 RES=0x00 SYN URGP=0
Jan 22 11:48:07 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=125.22.79.100 DST=62.221.52.196 LEN=48 TOS=0x00 PREC=0x00 TTL=106 ID=18441 DF PROTO=T
CP SPT=21064 DPT=25 WINDOW=65535 RES=0x00 SYN URGP=0
Jan 22 11:48:09 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=125.22.79.100 DST=62.221.52.196 LEN=48 TOS=0x00 PREC=0x00 TTL=106 ID=18487 DF PROTO=T
CP SPT=21064 DPT=25 WINDOW=65535 RES=0x00 SYN URGP=0
Jan 22 11:55:42 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=41.244.67.76 DST=62.221.52.196 LEN=48 TOS=0x00 PREC=0x00 TTL=113 ID=3125 DF PROTO=TCP
 SPT=3936 DPT=25 WINDOW=64240 RES=0x00 SYN URGP=0
Jan 22 11:55:45 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=41.244.67.76 DST=62.221.52.196 LEN=48 TOS=0x00 PREC=0x00 TTL=113 ID=3165 DF PROTO=TCP
 SPT=3936 DPT=25 WINDOW=64240 RES=0x00 SYN URGP=0
Jan 22 12:08:13 tenpostadsl clamd[5212]: SelfCheck: Database status OK.
Jan 22 12:14:18 tenpostadsl nagios: SERVICE ALERT: gw;SMTP;CRITICAL;SOFT;1;CRITICAL - Socket timeout af
ter 10 seconds
Jan 22 12:15:07 tenpostadsl nagios: SERVICE ALERT: gw;SMTP;OK;SOFT;2;SMTP OK - 0.008 sec. response time

Jan 22 12:17:47 tenpostadsl nagios: Auto-save of retention data completed successfully.
Jan 22 12:21:56 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=125.20.49.34 DST=62.221.52.196 LEN=48 TOS=0x00 PREC=0x00 TTL=106 ID=1884 DF PROTO=TCP
 SPT=3101 DPT=25 WINDOW=64240 RES=0x00 SYN URGP=0
Jan 22 12:21:59 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=125.20.49.34 DST=62.221.52.196 LEN=48 TOS=0x00 PREC=0x00 TTL=106 ID=1923 DF PROTO=TCP
 SPT=3101 DPT=25 WINDOW=64240 RES=0x00 SYN URGP=0
Jan 22 12:37:57 tenpostadsl kernel: c0217b70
Jan 22 12:37:57 tenpostadsl kernel: PREEMPT
Jan 22 12:37:57 tenpostadsl kernel: Modules linked in: usbhid bridge ipt_TOS i830 nfsd exportfs lockd s
unrpc lp ipv6 ipt_MASQUERADE ipt_REDIRECT ipt_REJECT ipt_LOG ipt_state ipt_pkttype ipt_recent ipt_ipran
ge ipt_physdev ipt_multiport ipt_conntrack iptable_mangle ip_nat_irc ip_nat_tftp ip_nat_ftp iptable_nat
 ip_conntrack_irc ip_conntrack_tftp ip_conntrack_ftp ip_conntrack iptable_filter ip_tables af_packet an
alog parport_pc parport floppy pcspkr rtc 8139cp snd_intel8x0 snd_ac97_codec snd_pcm snd_timer snd_page
_alloc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd shpchp pciehp pci_hotplug intel_agp agpg
art ehci_hcd uhci_hcd usbcore i810_audio ac97_codec soundcore 8139too mii quota_v2 tsdev mousedev capab
ility commoncap evdev tun psmouse ide_cd cdrom ext3 jbd mbcache dm_mod ide_generic piix ide_disk ide_co
re unix font vesafb cfbcopyarea cfbimgblt cfbfillrect
Jan 22 12:37:57 tenpostadsl kernel: CPU:    0
Jan 22 12:37:57 tenpostadsl kernel: EIP:    0060:[sock_poll+16/64]    Not tainted
Jan 22 12:37:57 tenpostadsl kernel: EFLAGS: 00010a47   (2.6.8-3-686)
Jan 22 12:37:57 tenpostadsl kernel: EIP is at sock_poll+0x10/0x40
Jan 22 12:37:57 tenpostadsl kernel: eax: cb8bec58   ebx: ce9bb280   ecx: ce9bb280   edx: c03010a0
Jan 22 12:37:57 tenpostadsl kernel: esi: c03010a0   edi: 00000014   ebp: 00000014   esp: cbb21ee0
Jan 22 12:37:57 tenpostadsl kernel: ds: 007b   es: 007b   ss: 0068
Jan 22 12:37:57 tenpostadsl kernel: Process named (pid: 5158, threadinfo=cbb20000 task=cafff7f0)
Jan 22 12:37:57 tenpostadsl kernel: Stack: ce05c280 ca8e7080 ce0c9980 ce0c9900 c0168d2f ce9bb280 000000
00 00000000
Jan 22 12:37:57 tenpostadsl kernel:        00000000 3bf00020 00000000 00000000 00000020 00000145 3bf000
20 cbb20000
Jan 22 12:37:57 tenpostadsl kernel:        c0db5414 c0db540c c0db5404 c0db5428 c0db5420 c0db5418 7fffff
ff 00000001
Jan 22 12:37:57 tenpostadsl kernel: Call Trace:
Jan 22 12:37:57 tenpostadsl kernel:  [do_select+607/720] do_select+0x25f/0x2d0
Jan 22 12:37:57 tenpostadsl kernel:  [__pollwait+0/208] __pollwait+0x0/0xd0
Jan 22 12:37:57 tenpostadsl kernel:  [sys_select+731/1264] sys_select+0x2db/0x4f0
Jan 22 12:37:57 tenpostadsl kernel:  [syscall_call+7/11] syscall_call+0x7/0xb
Jan 22 12:37:57 tenpostadsl kernel: Code: 08 8b 44 24 18 83 ea 24 8b 5a 08 89 44 24 08 89 54 24 04 89
Jan 22 12:39:17 tenpostadsl nagios: SERVICE ALERT: gw;SMTP;CRITICAL;SOFT;1;CRITICAL - Socket timeout af
ter 10 seconds
Jan 22 12:40:19 tenpostadsl nagios: SERVICE ALERT: gw;SMTP;CRITICAL;SOFT;2;CRITICAL - Socket timeout af
ter 10 seconds
Jan 22 12:41:17 tenpostadsl nagios: SERVICE ALERT: gw;SMTP;CRITICAL;HARD;3;CRITICAL - Socket timeout af
ter 10 seconds
Jan 22 12:41:17 tenpostadsl nagios: SERVICE NOTIFICATION: nagios;gw;SMTP;CRITICAL;notify-by-epager;CRIT
ICAL - Socket timeout after 10 seconds
Jan 22 12:41:33 tenpostadsl nagios: SERVICE NOTIFICATION: nagios;gw;SMTP;CRITICAL;notify-by-email;CRITI
CAL - Socket timeout after 10 seconds
Jan 22 12:44:20 tenpostadsl kernel:  <6>Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:
db:13:36:53:08:00 SRC=91.163.77.232 DST=62.221.52.196 LEN=52 TOS=0x00 PREC=0x00 TTL=113 ID=16373 DF PRO
TO=TCP SPT=18512 DPT=25 WINDOW=8576 RES=0x00 SYN URGP=0
Jan 22 12:44:23 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=91.163.77.232 DST=62.221.52.196 LEN=52 TOS=0x00 PREC=0x00 TTL=113 ID=16900 DF PROTO=T
CP SPT=18512 DPT=25 WINDOW=8576 RES=0x00 SYN URGP=0
Jan 22 12:44:29 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=91.163.77.232 DST=62.221.52.196 LEN=52 TOS=0x00 PREC=0x00 TTL=113 ID=18074 DF PROTO=T
CP SPT=18512 DPT=25 WINDOW=8576 RES=0x00 SYN URGP=0
Jan 22 12:45:24 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=91.163.77.232 DST=62.221.52.196 LEN=52 TOS=0x00 PREC=0x00 TTL=113 ID=25305 DF PROTO=T
CP SPT=20539 DPT=25 WINDOW=8576 RES=0x00 SYN URGP=0
Jan 22 12:45:27 tenpostadsl kernel: Shorewall:bogons:DROP:IN=eth0 OUT= MAC=00:02:44:ab:3b:f3:00:0f:db:1
3:36:53:08:00 SRC=91.163.77.232 DST=62.221.52.196 LEN=52 TOS=0x00 PREC=0x00 TTL=113 ID=25986 DF PROTO=T
CP SPT=20539 DPT=25 WINDOW=8576 RES=0x00 SYN URGP=0

A lot of bogus packets on 25 port. Then some problems with SMTP reported by nagios:

Jan 22 12:14:18 tenpostadsl nagios: SERVICE ALERT: gw;SMTP;CRITICAL;SOFT;1;CRITICAL - Socket timeout af
ter 10 seconds
Jan 22 12:15:07 tenpostadsl nagios: SERVICE ALERT: gw;SMTP;OK;SOFT;2;SMTP OK - 0.008 sec. response time

[pandora]

Memory usage can be optained from the "top", "vmstat" and "free" commands.

Bogons are IP addresses that shouldn't br present on the Internet. These include RFC1918 (10.0.0.0/8, 172.16.0.0/20 and 192.168.0.0/16) space forinternal home and corporate usage, and anything not yet assigned to be routed over the Internet.

High numbers of bogons are usually caused by some type of attack such as viruses randomly scanning the Internet for systems to attck.

The bogons listed all have a TCP destination port of 25, which is used for SMTP, and Nagios is alerting that the mail server isn't responding correctly.

3:36:53:08:00 SRC=124.38.246.5 DST=62.221.52.196 LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=51615 DF PROTO=TCP
 SPT=51260 DPT=25 WINDOW=5840 RES=0x00 SYN URGP=0
Jan 22 12:14:18 tenpostadsl nagios: SERVICE ALERT: gw;SMTP;CRITICAL;SOFT;1;CRITICAL - Socket timeout af
ter 10 seconds

It appears you have some infected machines on your network. The source IP address keeps changing, so the virus seems fairly sophisticated. With a changing source address it doesn't need to get a reply, or it is using the IP address of some other known infected host controlled by an IRC channel which can then figure out the next step of the attack. It could be an attack spread by SMTP mail SPAM, or it could be one in which the infected hosts send V1@gra SPAM after being infected by SPAM.

You'll need to isolate the system or systems quickly. Infected hosts ofter scan of new hosts on the network to be infected, which means they do a lot of ARP requests. try attaching a Linux laptop on each segment of your network and run TCP dump for ARP requests. Use the MAC address to isolate the NIC or switch port.

Some switches allow you to duplicate all traffic passing through it on a single port, you could try running TCP dump on that port looking for excessive SMTP traffic, and the MAC addresses of the hosts its coming from.

Scan for viruses on all your Windows boxes. If they have them, repair or rebuild them.

Is the named process running on the same server? If it is, it could be crashing because the SPAM is either doing execessive DNS lookups, or because the sheer volume of SPAM traffic is using up other resources such as RAM and swap space.

GNEEOT, it looks like you have a more serious problem than you first thought.