We see lot's packet drops on our cisco ip sla probes

Question

We see lot's packet drops on our cisco ip sla probes

cishoejnielsen opened this issue 8 years ago · 16 comments

cishoejnielsen commented 8 years ago

Hi,
We have app. 5000 probes pointing to the responder. We are seeing lots of packet loss.
It's a centos 5.6

ifconfig on eth0 is 195.41.248.165.

the responder config is very simple:
vlansupport=0
iface=eth0
debuglevel=0
ciscoport=64000

We don't see any drops on eth0 , and cpu and memory load is very little

Can you please help

Regards
Carsten

Answer 1 · 2016-11-28T12:57:55.000Z

Where are you seeing the packet loss? Can you describe your setup?

Answer 2 · 2016-11-28T13:35:06.000Z

Hi, Well, of course we se the drops on the router ip sla statistics. We have done a lot of tracing. When we ping the eth0 ip address on the responder from from the default router everything works fine, but when we ping the responder address , we get packet loss. So we expect the packets ar lost between the eth0 ip and the responder ip Med venlig hilsen / Regards Carsten Ishøj Nielsen System Engineer TDC Operations, OTVA , Bredbånd Teglholmsgade 1-3 , Bygning B1 2450 København SV Danmark T: +45 66 67 44 66 M: +45 20 10 13 34 E: cin@tdc.dk<mailto:cin@tdc.dk> [cid:image001.png@01D24984.9324D120] Fra: Aki Tuomi [mailto:notifications@github.com] Sendt: 28. november 2016 13:58 Til: cmouse/ip-sla-responder <ip-sla-responder@noreply.github.com> Cc: Carsten Ishøj Nielsen <cin@tdc.dk>; Author <author@noreply.github.com> Emne: Re: [cmouse/ip-sla-responder] We see lot's packet drops on our cisco ip sla probes (#10) Where are you seeing the packet loss? Can you describe your setup? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#10 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALWMKe373CY4mxx4p3OBPN_hp8B6Jw5bks5rCs_UgaJpZM4K9uqz>.

Answer 3 · 2016-11-28T13:37:56.000Z

Can you verify that the responder address is not mapped to any interface on the responder machine and that you are routing the packages to the responder machine's interface IP. This might sound bit illogical, but the responder software is supposed to care of it's own IP package handling.

Answer 4 · 2016-11-28T13:39:41.000Z

That is correct. Ifconfig only shows ip on eth0 Med venlig hilsen / Regards Carsten Ishøj Nielsen System Engineer TDC Operations, OTVA , Bredbånd Teglholmsgade 1-3 , Bygning B1 2450 København SV Danmark T: +45 66 67 44 66 M: +45 20 10 13 34 E: cin@tdc.dk<mailto:cin@tdc.dk> [cid:image001.png@01D24985.38B304E0] Fra: Aki Tuomi [mailto:notifications@github.com] Sendt: 28. november 2016 14:38 Til: cmouse/ip-sla-responder <ip-sla-responder@noreply.github.com> Cc: Carsten Ishøj Nielsen <cin@tdc.dk>; Author <author@noreply.github.com> Emne: Re: [cmouse/ip-sla-responder] We see lot's packet drops on our cisco ip sla probes (#10) Can you verify that the responder address is not mapped to any interface on the responder machine and that you are routing the packages to the responder machine's interface IP. This might sound bit illogical, but the responder software is supposed to care of it's own IP package handling. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#10 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALWMKWZISsXxC9m5f_8L9s9kFlZKrwnLks5rCtk0gaJpZM4K9uqz>.

Answer 5 · 2016-11-28T13:44:26.000Z

the IP address must not be configured on any interface on the responder host. the responder software will handle it's own IP stack. you must route the responder service IP towards the responder host IP.

Answer 6 · 2016-11-28T13:53:30.000Z

Hi, Here is the ip setup : [cid:image001.png@01D24987.24A0B450] Med venlig hilsen / Regards Carsten Ishøj Nielsen System Engineer TDC Operations, OTVA , Bredbånd Teglholmsgade 1-3 , Bygning B1 2450 København SV Danmark T: +45 66 67 44 66 M: +45 20 10 13 34 E: cin@tdc.dk<mailto:cin@tdc.dk> [cid:image002.png@01D24987.24A0B450] Fra: Aki Tuomi [mailto:notifications@github.com] Sendt: 28. november 2016 14:44 Til: cmouse/ip-sla-responder <ip-sla-responder@noreply.github.com> Cc: Carsten Ishøj Nielsen <cin@tdc.dk>; Author <author@noreply.github.com> Emne: Re: [cmouse/ip-sla-responder] We see lot's packet drops on our cisco ip sla probes (#10) the IP address must not be configured on any interface on the responder host. the responder software will handle it's own IP stack. you must route the responder service IP towards the responder host IP. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#10 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALWMKesS577kdaCOma1hfDbsWZStS-p2ks5rCtq6gaJpZM4K9uqz>.

Answer 7 · 2016-11-28T13:55:43.000Z

Github seems to ignore attachments over email.

Answer 8 · 2016-11-28T14:05:52.000Z

Also, have you checked that your CoPP filter does not ratelimit packets?

Answer 9 · 2016-11-28T14:10:01.000Z

cishoejnielsen commented 8 years ago

Answer 10 · 2016-11-28T14:13:48.000Z

thank you. can you confirm with packet sniffing on egress point that the server is indeed receiving more packets than it's not responding to? it also seems you are using virtual environment for running the software, and it might be the cause of your trouble. Last time I tested this with virtual machine it had trouble working correctly due to limitations in virtual platform.

Answer 11 · 2016-11-29T06:46:14.000Z

Hi, Yes, a sniffer trace taken over several hours , shows a packet loss of one percent. We have prepared a physical server , installed with responder. I will ask our server people if the can try replacing with the virtual server. Med venlig hilsen / Regards Carsten Ishøj Nielsen System Engineer TDC Operations, OTVA , Bredbånd Teglholmsgade 1-3 , Bygning B1 2450 København SV Danmark T: +45 66 67 44 66 M: +45 20 10 13 34 E: cin@tdc.dk<mailto:cin@tdc.dk> [cid:image001.png@01D24A14.A054D080] Fra: Aki Tuomi [mailto:notifications@github.com] Sendt: 28. november 2016 15:14 Til: cmouse/ip-sla-responder <ip-sla-responder@noreply.github.com> Cc: Carsten Ishøj Nielsen <cin@tdc.dk>; Author <author@noreply.github.com> Emne: Re: [cmouse/ip-sla-responder] We see lot's packet drops on our cisco ip sla probes (#10) thank you. can you confirm with packet sniffing on egress point that the server is indeed receiving more packets than it's not responding to? it also seems you are using virtual environment for running the software, and it might be the cause of your trouble. Last time I tested this with virtual machine it had trouble working correctly due to limitations in virtual platform. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#10 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALWMKR8mkV0UeAa9JiqaLJBzch6vQttyks5rCuGdgaJpZM4K9uqz>.

Answer 12 · 2016-12-13T07:19:09.000Z

Hi, We replaced the virtual with physical server. It worked. So virtual is no good. Thanks for your help Med venlig hilsen / Regards Carsten Ishøj Nielsen System Engineer IP tjenester, OBNT Teglholmsgade 1-3 , Bygning B1 2450 København SV Danmark T: +45 66 67 44 66 M: +45 20 10 13 34 E: cin@tdc.dk<mailto:cin@tdc.dk> [cid:image001.png@01D25519.8FA1E1A0] Fra: Aki Tuomi [mailto:notifications@github.com] Sendt: 28. november 2016 15:14 Til: cmouse/ip-sla-responder <ip-sla-responder@noreply.github.com> Cc: Carsten Ishøj Nielsen <cin@tdc.dk>; Author <author@noreply.github.com> Emne: Re: [cmouse/ip-sla-responder] We see lot's packet drops on our cisco ip sla probes (#10) thank you. can you confirm with packet sniffing on egress point that the server is indeed receiving more packets than it's not responding to? it also seems you are using virtual environment for running the software, and it might be the cause of your trouble. Last time I tested this with virtual machine it had trouble working correctly due to limitations in virtual platform. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#10 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALWMKR8mkV0UeAa9JiqaLJBzch6vQttyks5rCuGdgaJpZM4K9uqz>.

Answer 13 · 2018-11-02T01:21:37.000Z

sorry to jump on this after two years, but could i know what was the VM configuration and the physical server configuration you used?. i'm planning to test this, but most probably will not have option of physical server. I don't think there will be more than 2000 probes per vm.

Answer 14 · 2018-11-02T08:36:30.000Z

Hi, I wrote to Aki Toumi about running on vm. As I remember he recalled that there were issues running on VM. Our VM people tuned everything they could on the VM server , but we still saw pckt loss. Then we decided to install phys server instead, and then everything worked. We have app 5000 cisco sla probes and server has no issues. The server has very low cpu and mem load. Med venlig hilsen / Regards Carsten Ishøj Nielsen cin@tdc.dk<mailto:cin@tdc.dk> Fra: Allan Pinto <notifications@github.com> Sendt: 2. november 2018 02:22 Til: cmouse/ip-sla-responder <ip-sla-responder@noreply.github.com> Cc: Carsten Ishøj Nielsen <cin@tdc.dk>; Author <author@noreply.github.com> Emne: [EXT] Re: [cmouse/ip-sla-responder] We see lot's packet drops on our cisco ip sla probes (#10) UK/DK: Be aware, this is an external email and may not be genuine / Vær opmærksom på, at dette er en ekstern e-mail og muligvis ikke ægte. sorry to jump on this after two years, but could i know what was the VM configuration and the physical server configuration you used?. i'm planning to test this, but most probably will not have option of physical server. I don't think there will be more than 2000 probes per vm. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcmouse%2Fip-sla-responder%2Fissues%2F10%23issuecomment-435240689&data=02%7C01%7Ccin%40tdc.dk%7Ccf0bd07b3a90448e38c308d640618c2d%7Ce8dcf6e63acc4af99cb277f688cb688b%7C0%7C0%7C636767185056585678&sdata=%2F8gAgZqr0px1Ou4Bcunh1ZMjz1NyqbdXemBOlHTaB6c%3D&reserved=0>, or mute the thread<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALWMKXl0NuCo32nWzYc4RpYdJScip2Vmks5uq54jgaJpZM4K9uqz&data=02%7C01%7Ccin%40tdc.dk%7Ccf0bd07b3a90448e38c308d640618c2d%7Ce8dcf6e63acc4af99cb277f688cb688b%7C0%7C0%7C636767185056585678&sdata=TnzOWjJQZtSJdoNXxfl%2Fhw5Lt21soIMlMaHvPVZtS8M%3D&reserved=0>.

Answer 15 · 2018-11-02T09:27:47.000Z

Thanks a lot. Could I know the specs of the server so I can plan similarly.

…

On Fri, 2 Nov 2018, 19:36 cishoejnielsen ***@***.*** wrote: Hi, I wrote to Aki Toumi about running on vm. As I remember he recalled that there were issues running on VM. Our VM people tuned everything they could on the VM server , but we still saw pckt loss. Then we decided to install phys server instead, and then everything worked. We have app 5000 cisco sla probes and server has no issues. The server has very low cpu and mem load. Med venlig hilsen / Regards Carsten Ishøj Nielsen ***@***.******@***.***> Fra: Allan Pinto ***@***.***> Sendt: 2. november 2018 02:22 Til: cmouse/ip-sla-responder ***@***.***> Cc: Carsten Ishøj Nielsen ***@***.***>; Author ***@***.***> Emne: [EXT] Re: [cmouse/ip-sla-responder] We see lot's packet drops on our cisco ip sla probes (#10) UK/DK: Be aware, this is an external email and may not be genuine / Vær opmærksom på, at dette er en ekstern e-mail og muligvis ikke ægte. sorry to jump on this after two years, but could i know what was the VM configuration and the physical server configuration you used?. i'm planning to test this, but most probably will not have option of physical server. I don't think there will be more than 2000 probes per vm. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub< https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcmouse%2Fip-sla-responder%2Fissues%2F10%23issuecomment-435240689&data=02%7C01%7Ccin%40tdc.dk%7Ccf0bd07b3a90448e38c308d640618c2d%7Ce8dcf6e63acc4af99cb277f688cb688b%7C0%7C0%7C636767185056585678&sdata=%2F8gAgZqr0px1Ou4Bcunh1ZMjz1NyqbdXemBOlHTaB6c%3D&reserved=0>, or mute the thread< https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALWMKXl0NuCo32nWzYc4RpYdJScip2Vmks5uq54jgaJpZM4K9uqz&data=02%7C01%7Ccin%40tdc.dk%7Ccf0bd07b3a90448e38c308d640618c2d%7Ce8dcf6e63acc4af99cb277f688cb688b%7C0%7C0%7C636767185056585678&sdata=TnzOWjJQZtSJdoNXxfl%2Fhw5Lt21soIMlMaHvPVZtS8M%3D&reserved=0>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA7PbtTa2e8_MZWpLOdM_VSkvt2uKCX1ks5urAQPgaJpZM4K9uqz> .

Answer 16 · 2018-11-02T09:37:53.000Z

Hi, System Information Manufacturer: HP Product Name: ProLiant DL320e Gen8 CentOS release 6.7 (Final) Med venlig hilsen / Regards Carsten Ishøj Nielsen cin@tdc.dk<mailto:cin@tdc.dk> Fra: Allan Pinto <notifications@github.com> Sendt: 2. november 2018 10:28 Til: cmouse/ip-sla-responder <ip-sla-responder@noreply.github.com> Cc: Carsten Ishøj Nielsen <cin@tdc.dk>; Author <author@noreply.github.com> Emne: [EXT] Re: [cmouse/ip-sla-responder] We see lot's packet drops on our cisco ip sla probes (#10) UK/DK: Be aware, this is an external email and may not be genuine / Vær opmærksom på, at dette er en ekstern e-mail og muligvis ikke ægte. Thanks a lot. Could I know the specs of the server so I can plan similarly. On Fri, 2 Nov 2018, 19:36 cishoejnielsen <notifications@github.com wrote: <mailto:notifications@github.com%20wrote:%0b%0b>> Hi,

I wrote to Aki Toumi about running on vm. As I remember he recalled that there were issues running on VM. Our VM people tuned everything they could on the VM server , but we still saw pckt loss. Then we decided to install phys server instead, and then everything worked. We have app 5000 cisco sla probes and server has no issues. The server has very low cpu and mem load. Med venlig hilsen / Regards Carsten Ishøj Nielsen ***@***.******@***.******@***.******@***.***>> Fra: Allan Pinto ***@***.******@***.***>> Sendt: 2. november 2018 02:22 Til: cmouse/ip-sla-responder ***@***.******@***.***>> Cc: Carsten Ishøj Nielsen ***@***.******@***.***>>; Author ***@***.******@***.***>> Emne: [EXT] Re: [cmouse/ip-sla-responder] We see lot's packet drops on our cisco ip sla probes (#10) UK/DK: Be aware, this is an external email and may not be genuine / Vær opmærksom på, at dette er en ekstern e-mail og muligvis ikke ægte. sorry to jump on this after two years, but could i know what was the VM configuration and the physical server configuration you used?. i'm planning to test this, but most probably will not have option of physical server. I don't think there will be more than 2000 probes per vm. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub< https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcmouse%2Fip-sla-responder%2Fissues%2F10%23issuecomment-435240689&data=02%7C01%7Ccin%40tdc.dk%7Ccf0bd07b3a90448e38c308d640618c2d%7Ce8dcf6e63acc4af99cb277f688cb688b%7C0%7C0%7C636767185056585678&sdata=%2F8gAgZqr0px1Ou4Bcunh1ZMjz1NyqbdXemBOlHTaB6c%3D&reserved=0>, or mute the thread< https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALWMKXl0NuCo32nWzYc4RpYdJScip2Vmks5uq54jgaJpZM4K9uqz&data=02%7C01%7Ccin%40tdc.dk%7Ccf0bd07b3a90448e38c308d640618c2d%7Ce8dcf6e63acc4af99cb277f688cb688b%7C0%7C0%7C636767185056585678&sdata=TnzOWjJQZtSJdoNXxfl%2Fhw5Lt21soIMlMaHvPVZtS8M%3D&reserved=0>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#10 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA7PbtTa2e8_MZWpLOdM_VSkvt2uKCX1ks5urAQPgaJpZM4K9uqz> .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcmouse%2Fip-sla-responder%2Fissues%2F10%23issuecomment-435320505&data=02%7C01%7Ccin%40tdc.dk%7C94edab7c0a62430ab95808d640a57f2f%7Ce8dcf6e63acc4af99cb277f688cb688b%7C0%7C0%7C636767476883138627&sdata=4XZsdyMqFVUfHmYTTnXpfJrQMi9HhuaH7AoGoLK2GrE%3D&reserved=0>, or mute the thread<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALWMKcBTnS8hTFRh6BN_TliNZoZpVqRXks5urBAUgaJpZM4K9uqz&data=02%7C01%7Ccin%40tdc.dk%7C94edab7c0a62430ab95808d640a57f2f%7Ce8dcf6e63acc4af99cb277f688cb688b%7C0%7C0%7C636767476883138627&sdata=gOGfHLqHqtjjoC0eDv5yt90Etp7fgc%2FkhnsA4656PxA%3D&reserved=0>.