NFS & tcpdump error in CentOS

[nishith]

Hello,

I have configured NFS Server on CentOS 5.2 with an IBM Web Server(AIX).

The IBM Web Server can upload all data onto NFS Server. Now, today i was having slow response on IBM Web Server & by measuring the NFS, i found below error while running "tcpdump" command.

I have ran "tcpdump" command on NFS Server.

tcpdump -n -i eth1 | grep 2049

18:36:37.237451 IP 10.100.19.241.2049 > 10.100.19.88.1758143293: reply ok 1448 read [|nfs]
18:36:37.237476 IP 10.100.19.241.2049 > 10.100.19.88.539981409: reply ERR 1448

18:36:37.237481 IP 10.100.19.241.2049 > 10.100.19.88.796287348: reply ERR 1448
18:36:37.237488 IP 10.100.19.241.2049 > 10.100.19.88.1986098295: reply ERR 1448
18:36:37.237566 IP 10.100.19.241.2049 > 10.100.19.88.539762736: reply ERR 1448

18:36:37.237595 IP 10.100.19.241.2049 > 10.100.19.88.740307248: reply ERR 1448
18:36:37.237599 IP 10.100.19.241.2049 > 10.100.19.88.808333616: reply ERR 208
18:36:37.237792 IP 10.100.19.241.2049 > 10.100.19.88.1758143294: reply ok 1448 read [|nfs]

18:36:37.237810 IP 10.100.19.241.2049 > 10.100.19.88.1700950889: reply ERR 1448
18:36:37.237827 IP 10.100.19.241.2049 > 10.100.19.88.539762735: reply ERR 1332
18:36:37.237866 IP 10.100.19.241.2049 > 10.100.19.88.825830968: reply ERR 1448

18:36:37.237947 IP 10.100.19.241.2049 > 10.100.19.88.808333625: reply ERR 1200
18:36:37.237967 IP 10.100.19.241.2049 > 10.100.19.88.775435808: reply ERR 1448
18:36:37.237973 IP 10.100.19.241.2049 > 10.100.19.88.959330358: reply ERR 1448

18:36:37.237980 IP 10.100.19.241.2049 > 10.100.19.88.908078112: reply ERR 1448
18:36:37.237987 IP 10.100.19.241.2049 > 10.100.19.88.540030496: reply ERR 1448
18:36:37.238041 IP 10.100.19.241.2049 > 10.100.19.88.825897271: reply ERR 1448

18:36:37.238048 IP 10.100.19.241.2049 > 10.100.19.88.540094778: reply ERR 1448
18:36:37.238055 IP 10.100.19.241.2049 > 10.100.19.88.825237553: reply ERR 1448
18:36:37.238109 IP 10.100.19.241.2049 > 10.100.19.88.808529952: reply ERR 1448

18:36:37.238120 IP 10.100.19.88.1758143297 > 10.100.19.241.2049: 116 read [|nfs]
18:36:37.238129 IP 10.100.19.241.2049 > 10.100.19.88.842019120: reply ERR 704
18:36:37.238142 IP 10.100.19.241.2049 > 10.100.19.88.740306806: reply ERR 1448

18:36:37.238158 IP 10.100.19.241.2049 > 10.100.19.88.892805164: reply ERR 1200
18:36:37.238175 IP 10.100.19.241.2049 > 10.100.19.88.1758143288: reply ok 148 setattr [|nfs]
18:36:37.238256 IP 10.100.19.241.2049 > 10.100.19.88.1758143296: reply ok 1448

18:36:37.238263 IP 10.100.19.241.2049 > 10.100.19.88.1869440302: reply ERR 1448
16133 packets captured
23339 packets received by filter
7100 packets dropped by kernel

10.100.18.241 is the IP of NFS Server & 10.100.19.88 IP belongs to IBM Web Server.


Kindly Help




—
Regards,
Nishith N. Vyas

[nishith]

Dear All,

Still the NFS error continues on my LAN & I have found below error while running "wireshark" on NFS Server.

10.100.19.87 10.100.19.240 TCP {TCP ACKed lost segment] [TCP Previous segment lost] [TCP segment of a reassembled PDU] shilp > ex

Source Port : shilp (2049)
Destination Port :exp2 (1022)
Sequence Number:63813901 (relative sequence number)
Acknowledgement Number : 229369 (relative ack number)
Transmission Control Protocol, Src Port: shilp (2049) ,Dst Port: exp2 (1022) , Seq: 63813901, Ack: 229369,Len: 1448

Header Length : 32 bytes
[SEQ / ACK Analysis]
[TCP Analysis Flags]
A Segment before this frame was lost
This frame ACKs a segment we have not seen (lost ?)


This is the only error which is showing in Red/Yellow color in "wireshark" tool.

I think, this error is related to Network Interface. But, i have already changed my network interface. Already tried more then 3 NIC(gigabit) Network Cards on the same server.

Any Help Expected Now?

[peter]

Are you using a hub instead of a switch? The fact that frames are lost implies a bad connection? How are the cabling and routing on your network? Is it in good condition?

[nishith]

Hello,

My entire LAN environment is on Gigabit network. Here, I am managing 3 IBM Power6 Web Servers, Linux Servers & Oracle Application Servers with all security devices.

Due to some technical reason & unavailability of a Server Class hardware,I have to manage all my data on a single point of backup.That’s why NFS is there.

My LAN is totally secured with Juniper Firewall & CISCO Switch with RADWARE Load balancer. You can imagine that how big environment i have.

What i feel is problem lies on Network Part Only, because i worked on NFS(UDP & TCP) for almost 3 days & found nothing serious. Also I tried to replace my 3 IBM Workstations (as NFS Server),but the result is ZERO.

Also i tried to install 3 different network cards i.e. Broadcom,Intel & RealTek. Also marvell semiconductor card i tried,which is not giving any error after the installation. But, the problem with this card is it worked after upgrading my centOS kernel,so i can not rely on it for a long period of time.

Any Help?

[peter]

Is there a firewall or router ACLs in the path between the client and server? The dropped packets imply something in the way.

What are the error statistics on the NIC? Is anything showing up in the /var/log/messages file?

[gregw]

i know this is an older thread but should someone have ended up here searching for info on the tcpdump ERR 1448 message, here is the scoop… i think.

i believe this is more of an issue with tcpdump than it is your NFS server or clients. I believe tcpdump can not/does not parse the packets correctly. below is both tcpdump and snort parsing the same dump file, same packet. in the tcpdump we see an error reported, snort does not.

tcpdump:
12:53:23.327584 IP storage4.stg.corp.on.ca.nfs > ma2.stg.corp.on.ca.2210654877: reply ERR 1448


snort:
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ =+=+=+=+=+=+=+=+=+=+=+=+

08/10-12:53:23.327584 0:30:17:3:66 -> 0:1B:21:12:B8:3B type:0x800 len:0x5EA
10.1.2.14:2049 -> 10.1.2.152:991 TCP TTL:128 TOS:0x0 ID:17221 IpLen:20 DgmLen:1500
***A**** Seq: 0xE31E7A4B Ack: 0x5533124B Win: 0xFAF0 TcpLen: 32
TCP Options (3) => NOP NOP TS: 2590676461 334890879
.V……cK..]3..e……s….B.Ha.JV..t.ZN2uAM,k…G….0p…..n .
**SNIP***
…….._^./…*….R…Z..|..&..J.0.wc.a..)..).1."..O..+E…. ..
..2….jW(…9.U..U*….U……#>Gd.-..m
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ =+=+=+=+=+=+=+=+=+=+=+=+


by the way, doing "tcpdump | grep 2049" is just wrong and dirty.. a pipe and two processes? "tcpdump -p 2049" next time. (man it).

take care.