Keyboard Shortcuts
Thread View
j: Next unread messagek: Previous unread messagej a: Jump to all threadsj l: Jump to MailingList overview
Hello,
Is there any issues going on since yesterday that affect latencies and packet loss for some Indian ISP's connectivity from/to India/Europe ?
We noticed a +100ms increase and packet loss randomly during the day. (for multiple isps: TATA, Airtel, Vodafone India).
any clues ?
cheers,
Nuno Vieira
Hi Nuno
I just checked and did not saw any jumps in a many of the active latency checks I keep towards speed test endpoints for EU ISPs as well as routers of Indian ISPs connected to AMS-IX, DE-CIX etc. Did not saw any latency jumps. I have two source endpoints here: One is in downstream of Airtel and other in downstream of Jio.
Maybe what you faced is limited to specific upstream ISP/port which Indian networks use? Unlikely that impact was at submarine cable level.
Thanks.
On Sun, Nov 8, 2020 at 7:48 PM Nuno Vieira nuno@edgoo.io wrote:
Hello,
Is there any issues going on since yesterday that affect latencies and packet loss for some Indian ISP's connectivity from/to India/Europe ?
We noticed a +100ms increase and packet loss randomly during the day. (for multiple isps: TATA, Airtel, Vodafone India).
any clues ?
cheers,
Nuno Vieira _______________________________________________ INNOG mailing list -- innog@innog.net To unsubscribe send an email to innog-leave@innog.net
Hi Anurag, After pulling hair i dont have and doing proper troubleshooting it all comes out to be AS55410 problem... ticket is open... They shouldn't use the name "Vodafone"... This is all too bad to be true. They might have links and stuff, but zero clue. Also their helpdesk is awful and arrogant. Here some cool mtr's :) # zoom.usStart: 2020-11-09T11:42:07+0000HOST: mumbai Loss% Snt Last Avg Best Wrst StDev 1. AS141193 103.156.51.254 0.0% 25 0.8 0.7 0. 6 0.9 0.1 2. AS55410 122.15.103.94 0.0% 25 0.9 2.3 0.6 23.4 4.7 3. AS??? 182.19.108.202 0.0% 25 31.3 32.0 31.1 40.2 2.1 4. AS1273 ae5-100-xcr2.sgs.cw.net (213.185.219.17) 0.0% 25 79.9 56.2 54.2 79.9 5.7 5. AS1273 ae9-xcr1.sng.cw.net (195.2.24.165) 16.0% 25 328.2 328.0 327.7 328.8 0.3 6. AS1273 ae11-xcr1.att.cw.net (195.2.2.57) 4.0% 25 329.4 327.9 327.6 329.4 0.4 7. AS1273 ae38-xcr1.mar.cw.net (195.2.3.54) 8.0% 25 327.4 327.7 327.1 330.3 0.9 8. AS1273 et-7-1-0-xcr1.nyh.cw.net (195.2.24.241) 8.0% 25 327.5 327.4 326.9 329.6 0.6 9. AS1273 ae13-xcr2.nyk.cw.net (195.2.25.69) 8.0% 25 327.2 327.2 326.8 328.6 0.4 10. AS??? 52.95.216.78 24.0% 25 327.6 327.9 326.9 332.8 1.5 11. AS??? 52.93.4.85 20.0% 25 327.9 331.6 327.6 353.6 7.3 12. AS??? 52.93.4.46 8.0% 25 327.2 327.3 327.0 328.5 0.4 13. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 14. AS??? 150.222.242.116 4.0% 25 333.3 333.6 332.9 336.9 0.8 15. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 16. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 17. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 18. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 19. AS??? 150.222.243.219 0.0% 25 334.2 334.7 333.4 348.7 3.0 20. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 # ted.comStart: 2020-11-09T11:37:42+0000HOST: mumbai Loss% Snt Last Avg Best Wrst StDev 1. AS141193 103.156.51.254 0.0% 25 0.8 0.7 0. 5 0.9 0.1 2. AS55410 122.15.103.94 0.0% 25 0.8 1.2 0.6 8.4 1.5 3. AS??? 182.19.108.202 4.0% 25 33.2 34.1 33.1 45.8 2.6 4. AS1273 ae5-100-xcr2.sgs.cw.net (213.185.219.17) 0.0% 25 58.4 58.6 57.8 70.0 2.4 5. AS1273 ae9-xcr1.sng.cw.net (195.2.24.165) 8.0% 25 331.5 332.2 331.4 337.7 1.4 6. AS1273 ae11-xcr1.att.cw.net (195.2.2.57) 12.0% 25 331.4 332.3 331.2 348.4 3.6 7. AS1273 ae38-xcr1.mar.cw.net (195.2.3.54) 0.0% 25 331.0 334.0 330.5 350.7 6.0 8. AS1273 et-7-1-0-xcr1.nyh.cw.net (195.2.24.241) 12.0% 25 332.7 331.0 330.3 333.0 0.7 9. AS1273 ae13-xcr2.nyk.cw.net (195.2.25.69) 8.0% 25 330.7 330.9 330.3 333.5 1.0 10. AS??? 99.83.64.206 12.0% 25 331.4 332.9 331.0 344.4 3.5 11. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 12. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 13. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 14. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 15. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 16. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 17. AS??? 150.222.241.183 4.0% 25 337.8 337.9 337.0 342.1 1.3 18. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 4shared.com::::::::::::::Start: 2020-11-09T11:24:21+0000HOST: mumbai Loss% Snt Last Avg Best Wrst StDev 1. AS141193 103.156.51.254 0.0% 25 0 .8 0.8 0.7 0.9 0.1 2. AS55410 122.15.103.94 0.0% 25 1.0 1.0 0.7 3.2 0.5 3. AS??? 182.19.106.103 0.0% 25 1.0 1.1 0.9 1.4 0.1 4. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 5. AS55410 123.63.158.92 48.0% 25 1.0 2.8 0.9 15.8 4.4 6. AS1273 ae28-100- xcr1.mar.cw.net (195.89.115.197) 12.0% 25 198.7 198.0 196.8 200.4 1.1 7. AS1273 ae51.xcr1.mlu.cw.net (195.2.21.186) 16.0% 25 204.5 204.5 204.1 206.7 0.6 8. AS1273 ae0-xcr1.mlb.cw.net (195.2.25.98) 8.0% 25 205.2 207.0 205.0 247.4 8.8 9. AS174 be1274.rcr21.mil01.atlas.cogentco.com (130.117.14.25) 0.0% 25 203.9 204.2 203.9 204.6 0.2 10. AS174 be2194.ccr22.mrs01.atlas.cogentco.com (154.54.61.29) 4.0% 25 214.2 214.3 214.1 214.5 0.1 11. AS174 be3223.ccr31.vlc02.atlas.cogentco.com (154.54.57.210) 16.0% 25 221.4 221.3 221.1 221.7 0.2 12. AS174 be3355.ccr31.mad05.atlas.cogentco.com (154.54.57.229) 12.0% 25 221.2 221.4 220.9 221.7 0.2 13. AS174 be2324.ccr31.bio02.atlas.cogentco.com (154.54.61.129) 16.0% 25 226.2 230.6 225.9 249.6 7.0 14. AS174 be2331.ccr41.dca01.atlas.cogentco.com (154.54.85.241) 8.0% 25 295.4 295.4 295.0 295.6 0.2 15. AS174 be2112.ccr41.atl01.atlas.cogentco.com (154.54.7.158) 8.0% 25 307.0 309.2 306.8 356.3 10.3 16. AS174 be2687.ccr41.iah01.atlas.cogentco.com (154.54.28.70) 8.0% 25 334.0 333.8 333.4 334.2 0.2 17. AS174 be2441.ccr31.dfw01.atlas.cogentco.com (154.54.41.66) 12.0% 25 339.2 339.4 339.1 339.7 0.2 18. AS174 be2763.ccr41.dfw03.atlas.cogentco.com (154.54.28.74) 8.0% 25 340.0 339.9 339.7 340.4 0.1 19. AS174 38.142.87.106 0.0% 25 353.2 353.9 353.2 360.6 1.5 20. AS40824 204.155.149.43 0.0% 25 353.4 353.3 353.0 353.6 0.1 :-) cheers, Nuno Vieira
On Sun, 2020-11-08 at 23:39 +0530, Anurag Bhatia wrote:
Hi Nuno
I just checked and did not saw any jumps in a many of the active latency checks I keep towards speed test endpoints for EU ISPs as well as routers of Indian ISPs connected to AMS-IX, DE-CIX etc. Did not saw any latency jumps. I have two source endpoints here: One is in downstream of Airtel and other in downstream of Jio.
Maybe what you faced is limited to specific upstream ISP/port which Indian networks use? Unlikely that impact was at submarine cable level.
Thanks.
On Sun, Nov 8, 2020 at 7:48 PM Nuno Vieira nuno@edgoo.io wrote:
Hello,
Is there any issues going on since yesterday that affect latencies and packet loss for some Indian ISP's connectivity from/to India/Europe ?
We noticed a +100ms increase and packet loss randomly during the day. (for multiple isps: TATA, Airtel, Vodafone India).
any clues ?
cheers,
Nuno Vieira _______________________________________________ INNOG mailing list -- innog@innog.net To unsubscribe send an email to innog-leave@innog.net
INNOG mailing list -- innog@innog.net
To unsubscribe send an email to innog-leave@innog.net
Hi Nuno
There's a known issue going in Vodafone AS55410 and IDEA AS55644 (*both companies have merged as you might be aware but ASNs and backbone ASNs do have their own routing/upstreams + do use each other as well*). The issue is that both of them are leaking their upstream route to another upstream. Thus they are sitting between two upstream and routing traffic. In some case, it's both AS55410 + AS55644 sitting between two transit of each other entity. Since that's unexpected traffic and they probably haven't planned for it, it would be causing an impact on their circuit usage. It might be related to packet loss issues you are facing. I have tried best to communicate it to them through known network contacts and some of their customers. And I guess they are probably aware though the issue hasn't been fixed yet. I just scanned RIPE RIS RRC01 for routes with specific leak patterns and I get following:
Some of such leaks where Vodafone AS55410 is leaking its transit Tata Comm AS4755 routes to it's other upstream: 103.206.49.0/24|1299 1273 55410 55644 4755 135208 135208 135208 135208 103.206.49.0/24|51185 1273 55410 55644 4755 135208 135208 135208 135208 103.206.49.0/24|8218 1273 55410 55644 4755 135208 135208 135208 135208 103.248.120.0/24|13030 1273 55410 4755 58906 103.248.120.0/24|1299 1273 55410 4755 58906 103.248.120.0/24|51185 1273 55410 4755 58906 103.248.120.0/24|8218 1273 55410 4755 58906
and more (see the attached file as55410-leaking-AS4755.txt for the complete list)
And similarly for IDEA AS55644 (same organisation but ASN of other merged entity): 103.131.60.0/24|8218 1273 55410 55644 4755 132116 135208 135208 135208 135208 103.206.48.0/24|13030 1273 55410 55644 4755 132116 135208 135208 135208 135208 103.206.48.0/24|1299 1273 55410 55644 4755 132116 135208 135208 135208 135208 203.91.212.0/24|328145 3491 55644 55410 4755 23664 203.91.212.0/24|1299 3491 55644 55410 4755 23664 203.91.213.0/24|328145 3491 55644 55410 4755 23664
and few more (see the attached file as55644-leaking-AS4755.txt for complete list)
These leaks have a direct impact on those end networks who are originating them as in many cases routing is getting really odd. Here's an example of a trace from my server in Munich (a server in downstream of Telia AS1299) to Pune, India for Indian Software Giant WIPRO's prefix:
traceroute to 203.91.212.1 (203.91.212.1), 30 hops max, 60 byte packets 1 10.20.70.1 (10.20.70.1) [*] 0.061 ms 0.012 ms * 2 gw.giga-dns.com (91.194.90.1) [AS51167] 0.444 ms 0.390 ms * 3 gw02.giga-hosting.biz (213.248.101.77) [AS1299] 0.495 ms 0.460 ms 0.407 ms 4 mcn-b3-link.telia.net (62.115.120.79) [AS1299] 0.848 ms 0.851 ms 0.802 ms 5 win-bb3-link.telia.net (62.115.123.222) [AS1299] 16.340 ms 16.310 ms 16.339 ms 6 ffm-bb1-link.telia.net (62.115.137.202) [AS1299] 16.730 ms 16.448 ms 16.370 ms 7 ffm-b5-link.telia.net (62.115.114.89) [AS1299] 16.500 ms 16.551 ms 16.268 ms 8 ffm-b4-link.telia.net (62.115.116.17) [AS1299] 20.079 ms 24.380 ms 16.284 ms 9 pccw-ic-319976-ffm-b4.c.telia.net (213.248.68.71) [AS1299] 16.764 ms 16.807 ms 16.709 ms 10 HundredGE0-5-0-0.br03.sin02.pccwbtn.net (63.223.34.142) [AS3491] 253.066 ms HundredGE0-3-0-0.br03.sin02.pccwbtn.net (63.223.34.134) [AS3491] 288.743 ms HundredGE0-5-0-0.br03.sin02.pccwbtn.net (63.223.34.142) [AS3491] 253.335 ms 11 HundredGE0-5-0-0.br03.sin02.pccwbtn.net (63.223.34.142) [AS3491] 255.027 ms 252.918 ms 253.227 ms 12 vodafone-idea.be23.br03.sin02.pccwbtn.net (63.218.213.58) [AS3491] 292.545 ms vodafone-idea.be19.br03.sin02.pccwbtn.net (63.218.213.34) [AS3491] 291.531 ms 293.435 ms 13 223.196.24.42 (223.196.24.42) [AS55644] 288.549 ms 223.196.24.34 (223.196.24.34) [AS55644] 295.959 ms 223.196.23.193 (223.196.23.193) [AS55644] 291.529 ms 14 223.196.1.201 (223.196.1.201) [*] 204.809 ms 203.932 ms 202.876 ms 15 14.142.185.133.static-chennai.vsnl.net.in (14.142.185.133) [AS4755] 195.857 ms 195.745 ms 195.706 ms 16 * * * 17 * 14.143.171.154.static-pune.vsnl.net.in (14.143.171.154) [AS4755] 188.526 ms 188.429 ms 18 * * * 19 * * * 20 * * *
So instead of traffic coming from Europe -> India, it's going via Singapore and possibly from congested paths. Vodafone IDEA is picking traffic here from their transit 1 (PCCW Global AS3491) and handing off to transit 2 (Tata Comm AS4755). And because AS55644 connects to PCCW in Singapore, they are pulling traffic via Singapore. If this leak wasn't happening, Telia AS1299 would have handed off traffic to Tata Comm AS6453 in Germany itself (possibly or near Frankfurt) and it would be direct to India.
*Anyone here from Vodafone IDEA? Please consider contacting offlist for a discussion.* You are missing AS_PATH based filters and likely have a blunt prefix list of customers on transit BGP sessions. Consider matching customer ASN with ^ASN_ so that you announce directly learnt downstream routes only. A better way all together here can be to just tag downstream customer routes with BGP communities and match & export only those routes to upstreams.
Thanks.
On Mon, Nov 9, 2020 at 5:57 PM Nuno Vieira nuno@edgoo.io wrote:
Hi Anurag,
After pulling hair i dont have and doing proper troubleshooting it all comes out to be AS55410 problem... ticket is open...
They shouldn't use the name "Vodafone"... This is all too bad to be true.
They might have links and stuff, but zero clue. Also their helpdesk is awful and arrogant.
Here some cool mtr's :)
# zoom.us Start: 2020-11-09T11:42:07+0000 HOST: mumbai Loss% Snt Last Avg Best Wrst StDev
- AS141193
103.156.51.254 0.0% 25 0.8 0.7 0.6 0.9 0.1 2. AS55410 122.15.103.94 0.0% 25 0.9 2.3 0.6 23.4 4.7 3. AS??? 182.19.108.202 0.0% 25 31.3 32.0 31.1 40.2 2.1 4. AS1273 ae5-100-xcr2.sgs.cw.net (213.185.219.17) 0.0% 25 79.9 56.2 54.2 79.9 5.7 5. AS1273 ae9-xcr1.sng.cw.net (195.2.24.165) 16.0% 25 328.2 328.0 327.7 328.8 0.3 6. AS1273 ae11-xcr1.att.cw.net (195.2.2.57) 4.0% 25 329.4 327.9 327.6 329.4 0.4 7. AS1273 ae38-xcr1.mar.cw.net (195.2.3.54) 8.0% 25 327.4 327.7 327.1 330.3 0.9 8. AS1273 et-7-1-0-xcr1.nyh.cw.net (195.2.24.241) 8.0% 25 327.5 327.4 326.9 329.6 0.6 9. AS1273 ae13-xcr2.nyk.cw.net (195.2.25.69) 8.0% 25 327.2 327.2 326.8 328.6 0.4 10. AS??? 52.95.216.78 24.0% 25 327.6 327.9 326.9 332.8 1.5 11. AS??? 52.93.4.85 20.0% 25 327.9 331.6 327.6 353.6 7.3 12. AS??? 52.93.4.46 8.0% 25 327.2 327.3 327.0 328.5 0.4 13. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 14. AS??? 150.222.242.116 4.0% 25 333.3 333.6 332.9 336.9 0.8 15. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 16. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 17. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 18. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 19. AS??? 150.222.243.219 0.0% 25 334.2 334.7 333.4 348.7 3.0 20. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0
# ted.com Start: 2020-11-09T11:37:42+0000 HOST: mumbai Loss% Snt Last Avg Best Wrst StDev
- AS141193
103.156.51.254 0.0% 25 0.8 0.7 0.5 0.9 0.1 2. AS55410 122.15.103.94 0.0% 25 0.8 1.2 0.6 8.4 1.5 3. AS??? 182.19.108.202 4.0% 25 33.2 34.1 33.1 45.8 2.6 4. AS1273 ae5-100-xcr2.sgs.cw.net (213.185.219.17) 0.0% 25 58.4 58.6 57.8 70.0 2.4 5. AS1273 ae9-xcr1.sng.cw.net (195.2.24.165) 8.0% 25 331.5 332.2 331.4 337.7 1.4 6. AS1273 ae11-xcr1.att.cw.net (195.2.2.57) 12.0% 25 331.4 332.3 331.2 348.4 3.6 7. AS1273 ae38-xcr1.mar.cw.net (195.2.3.54) 0.0% 25 331.0 334.0 330.5 350.7 6.0 8. AS1273 et-7-1-0-xcr1.nyh.cw.net (195.2.24.241) 12.0% 25 332.7 331.0 330.3 333.0 0.7 9. AS1273 ae13-xcr2.nyk.cw.net (195.2.25.69) 8.0% 25 330.7 330.9 330.3 333.5 1.0 10. AS??? 99.83.64.206 12.0% 25 331.4 332.9 331.0 344.4 3.5 11. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 12. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 13. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 14. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 15. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 16. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 17. AS??? 150.222.241.183 4.0% 25 337.8 337.9 337.0 342.1 1.3 18. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0
4shared.com :::::::::::::: Start: 2020-11-09T11:24:21+0000 HOST: mumbai Loss% Snt Last Avg Best Wrst StDev
- AS141193
103.156.51.254 0.0% 25 0.8 0.8 0.7 0.9 0.1 2. AS55410 122.15.103.94 0.0% 25 1.0 1.0 0.7 3.2 0.5 3. AS??? 182.19.106.103 0.0% 25 1.0 1.1 0.9 1.4 0.1 4. AS??? ??? 100.0 25 0.0 0.0 0.0 0.0 0.0 5. AS55410 123.63.158.92 48.0% 25 1.0 2.8 0.9 15.8 4.4 6. AS1273 ae28-100-xcr1.mar.cw.net (195.89.115.197) 12.0% 25 198.7 198.0 196.8 200.4 1.1 7. AS1273 ae51.xcr1.mlu.cw.net (195.2.21.186) 16.0% 25 204.5 204.5 204.1 206.7 0.6 8. AS1273 ae0-xcr1.mlb.cw.net (195.2.25.98) 8.0% 25 205.2 207.0 205.0 247.4 8.8 9. AS174 be1274.rcr21.mil01.atlas.cogentco.com (130.117.14.25) 0.0% 25 203.9 204.2 203.9 204.6 0.2 10. AS174 be2194.ccr22.mrs01.atlas.cogentco.com (154.54.61.29) 4.0% 25 214.2 214.3 214.1 214.5 0.1 11. AS174 be3223.ccr31.vlc02.atlas.cogentco.com (154.54.57.210) 16.0% 25 221.4 221.3 221.1 221.7 0.2 12. AS174 be3355.ccr31.mad05.atlas.cogentco.com (154.54.57.229) 12.0% 25 221.2 221.4 220.9 221.7 0.2 13. AS174 be2324.ccr31.bio02.atlas.cogentco.com (154.54.61.129) 16.0% 25 226.2 230.6 225.9 249.6 7.0 14. AS174 be2331.ccr41.dca01.atlas.cogentco.com (154.54.85.241) 8.0% 25 295.4 295.4 295.0 295.6 0.2 15. AS174 be2112.ccr41.atl01.atlas.cogentco.com (154.54.7.158) 8.0% 25 307.0 309.2 306.8 356.3 10.3 16. AS174 be2687.ccr41.iah01.atlas.cogentco.com (154.54.28.70) 8.0% 25 334.0 333.8 333.4 334.2 0.2 17. AS174 be2441.ccr31.dfw01.atlas.cogentco.com (154.54.41.66) 12.0% 25 339.2 339.4 339.1 339.7 0.2 18. AS174 be2763.ccr41.dfw03.atlas.cogentco.com (154.54.28.74) 8.0% 25 340.0 339.9 339.7 340.4 0.1 19. AS174 38.142.87.106 0.0% 25 353.2 353.9 353.2 360.6 1.5 20. AS40824 204.155.149.43 0.0% 25 353.4 353.3 353.0 353.6 0.1
:-)
cheers,
Nuno Vieira
On Sun, 2020-11-08 at 23:39 +0530, Anurag Bhatia wrote:
Hi Nuno
I just checked and did not saw any jumps in a many of the active latency checks I keep towards speed test endpoints for EU ISPs as well as routers of Indian ISPs connected to AMS-IX, DE-CIX etc. Did not saw any latency jumps. I have two source endpoints here: One is in downstream of Airtel and other in downstream of Jio.
Maybe what you faced is limited to specific upstream ISP/port which Indian networks use? Unlikely that impact was at submarine cable level.
Thanks.
On Sun, Nov 8, 2020 at 7:48 PM Nuno Vieira nuno@edgoo.io wrote:
Hello,
Is there any issues going on since yesterday that affect latencies and packet loss for some Indian ISP's connectivity from/to India/Europe ?
We noticed a +100ms increase and packet loss randomly during the day. (for multiple isps: TATA, Airtel, Vodafone India).
any clues ?
cheers,
Nuno Vieira _______________________________________________ INNOG mailing list -- innog@innog.net To unsubscribe send an email to innog-leave@innog.net
INNOG mailing list -- innog@innog.net
To unsubscribe send an email to innog-leave@innog.net
--
Kind regards,
Nuno Vieira
edgoo networks datacentre and it services
m: +351911728100 | p: +351211224705 e: nuno@hashpower.ptnuno@edgoo.io | nuno@hashpower.pt w: https://edgoo.io
Hi @Anurag First i wouldn't call VI as Vodafone, as they are simply defaming that name... Also AS55410 != AS1273 (i wish it were, at least there in IN) but yes, indeed it is that... we have extensive ticket open with them... what i can say so far is that they seem to be totally clueless (i provbutided them already a lot of ripe atlas links and mtr's etc)... along others you mentioned they also have airtel on the mix aswell.. a lot of routes from EU>IN take airtel. at this hour it starts to be acceptable, even so its high for what was supposed to be... (example from one host in Lisbon) $ traceroute gw.bom1.as47787.nettraceroute to gw.bom1.as47787.net (202.94.174.251), 30 hops max, 60 byte packets 1 185.166.144.206 (185.166.144.206) 0.833 ms 0.964 ms 1.129 ms 2 er1.lis1.edgoo.net (100.64.60.2) 0.056 ms 0.055 ms 0.069 ms 3 te0-7-0- 9.rcr21.lis01.atlas.cogentco.com (149.6.145.201) 0.686 ms 0.691 ms 0.699 ms 4 be2447.rcr21.svq01.atlas.cogentco.com (130.117.1.21) 6.369 ms 6.368 ms 6.360 ms 5 be3240.ccr31.vlc02.atlas.cogentco.com (154.54.59.13) 20.193 ms 19.808 ms 20.085 ms 6 be3222.ccr21.mrs01.atlas.cogentco.com (154.54.57.205) 31.706 ms 31.897 ms 31.753 ms 7 bharti.demarc.cogentco.com (149.14.126.170) 31.317 ms 31.580 ms 31.578 ms 8 116.119.50.142 (116.119.50.142) 154.801 ms 151.997 ms 152.360 ms 9 125.18.238.202 (125.18.238.202) 186.860 ms 182.535 ms 182.159 ms10 * * *11 122.15.103.93 (122.15.103.93) 183.913 ms 183.646 ms 183.656 ms12 202.94.174.251 (202.94.174.251) 183.799 ms 191.078 ms 183.121 ms and from madrid $ traceroute gw.bom1.as47787.nettraceroute to gw.bom1.as47787.net (202.94.174.251), 30 hops max, 60 byte packets 1 192.168.1.1 (192.168.1.1) 0.932 ms 0.920 ms 0.922 ms 2 100.64.60.12 (100.64.60.12) 1.057 ms 1.077 ms 0.672 ms 3 ip4.gtt.net (154.14.130.57) 1.439 ms 1.407 ms 1.324 ms 4 ae25.cr5- mrs1.ip4.gtt.net (141.136.111.174) 19.122 ms 19.082 ms 19.032 ms 5 ip4.gtt.net (154.14.74.22) 46.036 ms 45.971 ms 45.858 ms 6 116.119.50.142 (116.119.50.142) 135.809 ms 137.058 ms 136.996 ms 7 125.18.238.202 (125.18.238.202) 186.472 ms 183.285 ms 182.920 ms 8 182.19.106.113 (182.19.106.113) 183.797 ms 182.908 ms 182.645 ms 9 122.15.103.93 (122.15.103.93) 190.194 ms 182.674 ms 183.353 ms10 202.94.174.251 (202.94.174.251) 186.333 ms 184.509 ms 185.128 ms reverse path now goes via London. (yes assymetric routing on VI is a must)... $ traceroute noc.as47787.nettraceroute to noc.as47787.net (185.166.144.193), 30 hops max, 60 byte packets 1 103.156.51.254 (103.156.51.254) 0.492 ms 0.464 ms 0.439 ms 2 122.15.103.94 (122.15.103.94) 0.542 ms 0.502 ms 0.458 ms 3 182.19.106.204 (182.19.106.204) 1.579 ms 1.558 ms 1.511 ms 4 195.89.116.245 (195.89.116.245) 136.426 ms 136.417 ms 136.372 ms 5 as174-gw- xcr1.lns.cw.net (195.2.22.22) 158.424 ms 158.363 ms as174-gw- xcr1.lns.cw.net (195.2.19.122) 158.748 ms 6 be2870.ccr41.lon13.atlas.cogentco.com (154.54.58.173) 165.414 ms be2869.ccr42.lon13.atlas.cogentco.com (154.54.57.161) 158.602 ms be2870.ccr41.lon13.atlas.cogentco.com (154.54.58.173) 165.597 ms 7 be12489.ccr42.par01.atlas.cogentco.com (154.54.57.70) 160.341 ms 160.419 ms 160.266 ms 8 be2315.ccr31.bio02.atlas.cogentco.com (154.54.61.113) 169.982 ms 170.040 ms be2318.ccr32.bio02.atlas.cogentco.com (154.54.61.117) 170.013 ms 9 be2313.rcr21.opo01.atlas.cogentco.com (154.54.61.102) 183.497 ms be2314.rcr21.opo01.atlas.cogentco.com (154.54.61.106) 184.283 ms 184.230 ms10 be2444.rcr21.lis01.atlas.cogentco.com (154.54.61.213) 183.526 ms 184.063 ms 183.979 ms11 149.6.145.205 (149.6.145.205) 182.927 ms 182.893 ms 183.154 ms12 * * *13 noc.as47787.net (185.166.144.193) 183.775 ms 183.769 ms 183.766 ms return paths are always scenic, or over sgn or via some kinky upstream... I believe they are just playing around with their own network.. Zero consistency.. at least i was able to find 4 endpoints on AS1273 from them (Marseilles, Milan, London and Singapure)... IMO this doesnt make much sense... instead of them playing nx 10GE's and do nasty TE, simply should do nx 100GE's and two decent endpoints instead of wanting a full buffet they cant really handle.. I am monitoring some endpoints from them into AS1273 and they are all congested (at least during mostly of the day)... Starting to consider to simply do a direct 10GE on EIG end-to-end for our usage, to avoid these messups... this is way too bad to be true... let see what comes out of here :) cheers Nuno Vieira On Mon, 2020-11-09 at 23:55 +0530, Anurag Bhatia wrote:
Hi Nuno
There's a known issue going in Vodafone AS55410 and IDEA AS55644 (both companies have merged as you might be aware but ASNs and backbone ASNs do have their own routing/upstreams + do use each other as well). The issue is that both of them are leaking their upstream route to another upstream. Thus they are sitting between two upstream and routing traffic. In some case, it's both AS55410 + AS55644 sitting between two transit of each other entity. Since that's unexpected traffic and they probably haven't planned for it, it would be causing an impact on their circuit usage. It might be related to packet loss issues you are facing. I have tried best to communicate it to them through known network contacts and some of their customers. And I guess they are probably aware though the issue hasn't been fixed yet. I just scanned RIPE RIS RRC01 for routes with specific leak patterns and I get following:
Some of such leaks where Vodafone AS55410 is leaking its transit Tata Comm AS4755 routes to it's other upstream: 103.206.49.0/24|1299 1273 55410 55644 4755 135208 135208 135208 135208 103.206.49.0/24|51185 1273 55410 55644 4755 135208 135208 135208 135208 103.206.49.0/24|8218 1273 55410 55644 4755 135208 135208 135208 135208 103.248.120.0/24|13030 1273 55410 4755 58906 103.248.120.0/24|1299 1273 55410 4755 58906 103.248.120.0/24|51185 1273 55410 4755 58906 103.248.120.0/24|8218 1273 55410 4755 58906
and more (see the attached file as55410-leaking-AS4755.txt for the complete list)
And similarly for IDEA AS55644 (same organisation but ASN of other merged entity): 103.131.60.0/24|8218 1273 55410 55644 4755 132116 135208 135208 135208 135208 103.206.48.0/24|13030 1273 55410 55644 4755 132116 135208 135208 135208 135208 103.206.48.0/24|1299 1273 55410 55644 4755 132116 135208 135208 135208 135208
203.91.212.0/24|328145 3491 55644 55410 4755 23664 203.91.212.0/24|1299 3491 55644 55410 4755 23664 203.91.213.0/24|328145 3491 55644 55410 4755 23664
and few more (see the attached file as55644-leaking-AS4755.txt for complete list)
These leaks have a direct impact on those end networks who are originating them as in many cases routing is getting really odd. Here's an example of a trace from my server in Munich (a server in downstream of Telia AS1299) to Pune, India for Indian Software Giant WIPRO's prefix:
traceroute to 203.91.212.1 (203.91.212.1), 30 hops max, 60 byte packets 1 10.20.70.1 (10.20.70.1) [*] 0.061 ms 0.012 ms * 2 gw.giga-dns.com (91.194.90.1) [AS51167] 0.444 ms 0.390 ms * 3 gw02.giga-hosting.biz (213.248.101.77) [AS1299] 0.495 ms 0.460 ms 0.407 ms 4 mcn-b3-link.telia.net (62.115.120.79) [AS1299] 0.848 ms 0.851 ms 0.802 ms 5 win-bb3-link.telia.net (62.115.123.222) [AS1299] 16.340 ms 16.310 ms 16.339 ms 6 ffm-bb1-link.telia.net (62.115.137.202) [AS1299] 16.730 ms 16.448 ms 16.370 ms 7 ffm-b5-link.telia.net (62.115.114.89) [AS1299] 16.500 ms 16.551 ms 16.268 ms 8 ffm-b4-link.telia.net (62.115.116.17) [AS1299] 20.079 ms 24.380 ms 16.284 ms 9 pccw-ic-319976-ffm-b4.c.telia.net (213.248.68.71) [AS1299] 16.764 ms 16.807 ms 16.709 ms 10 HundredGE0-5-0-0.br03.sin02.pccwbtn.net (63.223.34.142) [AS3491] 253.066 ms HundredGE0-3-0-0.br03.sin02.pccwbtn.net (63.223.34.134) [AS3491] 288.743 ms HundredGE0-5-0-0.br03.sin02.pccwbtn.net (63.223.34.142) [AS3491] 253.335 ms 11 HundredGE0-5-0-0.br03.sin02.pccwbtn.net (63.223.34.142) [AS3491] 255.027 ms 252.918 ms 253.227 ms 12 vodafone-idea.be23.br03.sin02.pccwbtn.net (63.218.213.58) [AS3491] 292.545 ms vodafone-idea.be19.br03.sin02.pccwbtn.net (63.218.213.34) [AS3491] 291.531 ms 293.435 ms 13 223.196.24.42 (223.196.24.42) [AS55644] 288.549 ms 223.196.24.34 (223.196.24.34) [AS55644] 295.959 ms 223.196.23.193 (223.196.23.193) [AS55644] 291.529 ms 14 223.196.1.201 (223.196.1.201) [*] 204.809 ms 203.932 ms 202.876 ms 15 14.142.185.133.static-chennai.vsnl.net.in (14.142.185.133) [AS4755] 195.857 ms 195.745 ms 195.706 ms 16 * * * 17 * 14.143.171.154.static-pune.vsnl.net.in (14.143.171.154) [AS4755] 188.526 ms 188.429 ms 18 * * * 19 * * * 20 * * *
So instead of traffic coming from Europe -> India, it's going via Singapore and possibly from congested paths. Vodafone IDEA is picking traffic here from their transit 1 (PCCW Global AS3491) and handing off to transit 2 (Tata Comm AS4755). And because AS55644 connects to PCCW in Singapore, they are pulling traffic via Singapore. If this leak wasn't happening, Telia AS1299 would have handed off traffic to Tata Comm AS6453 in Germany itself (possibly or near Frankfurt) and it would be direct to India.
Anyone here from Vodafone IDEA? Please consider contacting offlist for a discussion. You are missing AS_PATH based filters and likely have a blunt prefix list of customers on transit BGP sessions. Consider matching customer ASN with ^ASN_ so that you announce directly learnt downstream routes only. A better way all together here can be to just tag downstream customer routes with BGP communities and match & export only those routes to upstreams.
Thanks.
On Mon, Nov 9, 2020 at 5:57 PM Nuno Vieira nuno@edgoo.io wrote:
Hi Anurag,
After pulling hair i dont have and doing proper troubleshooting it all comes out to be AS55410 problem... ticket is open...
They shouldn't use the name "Vodafone"... This is all too bad to be true.
They might have links and stuff, but zero clue. Also their helpdesk is awful and arrogant.
Here some cool mtr's :)
# zoom.us Start: 2020-11-09T11:42:07+0000 HOST: mumbai Loss% Snt Las t Avg Best Wrst StDev
- AS141193
103.156.51.254 0.0% 25 0.8 0.7 0.6 0.9 0.1 2. AS55410 122.15.103.94 0.0% 25 0 .9 2.3 0.6 23.4 4.7 3. AS??? 182.19.108.202 0.0% 25 31 .3 32.0 31.1 40.2 2.1 4. AS1273 ae5-100-xcr2.sgs.cw.net (213.185.219.17) 0.0% 25 79.9 56.2 54.2 79.9 5.7 5. AS1273 ae9-xcr1.sng.cw.net (195.2.24.165) 16.0% 25 328.2 328.0 327.7 328.8 0.3 6. AS1273 ae11-xcr1.att.cw.net (195.2.2.57) 4.0% 25 329.4 327.9 327.6 329.4 0.4 7. AS1273 ae38-xcr1.mar.cw.net (195.2.3.54) 8.0% 25 327.4 327.7 327.1 330.3 0.9 8. AS1273 et-7-1-0-xcr1.nyh.cw.net (195.2.24.241) 8.0% 25 327.5 327.4 326.9 329.6 0.6 9. AS1273 ae13-xcr2.nyk.cw.net (195.2.25.69) 8.0% 25 327.2 327.2 326.8 328.6 0.4 10. AS??? 52.95.216.78 24.0% 25 327 .6 327.9 326.9 332.8 1.5 11. AS??? 52.93.4.85 20.0% 25 327 .9 331.6 327.6 353.6 7.3 12. AS??? 52.93.4.46 8.0% 25 327 .2 327.3 327.0 328.5 0.4 13. AS??? ??? 100.0 25 0 .0 0.0 0.0 0.0 0.0 14. AS??? 150.222.242.116 4.0% 25 333 .3 333.6 332.9 336.9 0.8 15. AS??? ??? 100.0 25 0 .0 0.0 0.0 0.0 0.0 16. AS??? ??? 100.0 25 0 .0 0.0 0.0 0.0 0.0 17. AS??? ??? 100.0 25 0 .0 0.0 0.0 0.0 0.0 18. AS??? ??? 100.0 25 0 .0 0.0 0.0 0.0 0.0 19. AS??? 150.222.243.219 0.0% 25 334 .2 334.7 333.4 348.7 3.0 20. AS??? ??? 100.0 25 0 .0 0.0 0.0 0.0 0.0
# ted.com Start: 2020-11-09T11:37:42+0000 HOST: mumbai Loss% Snt Las t Avg Best Wrst StDev
- AS141193
103.156.51.254 0.0% 25 0.8 0.7 0.5 0.9 0.1 2. AS55410 122.15.103.94 0.0% 25 0 .8 1.2 0.6 8.4 1.5 3. AS??? 182.19.108.202 4.0% 25 33 .2 34.1 33.1 45.8 2.6 4. AS1273 ae5-100-xcr2.sgs.cw.net (213.185.219.17) 0.0% 25 58.4 58.6 57.8 70.0 2.4 5. AS1273 ae9-xcr1.sng.cw.net (195.2.24.165) 8.0% 25 331.5 332.2 331.4 337.7 1.4 6. AS1273 ae11-xcr1.att.cw.net (195.2.2.57) 12.0% 25 331.4 332.3 331.2 348.4 3.6 7. AS1273 ae38-xcr1.mar.cw.net (195.2.3.54) 0.0% 25 331.0 334.0 330.5 350.7 6.0 8. AS1273 et-7-1-0-xcr1.nyh.cw.net (195.2.24.241) 12.0% 25 332.7 331.0 330.3 333.0 0.7 9. AS1273 ae13-xcr2.nyk.cw.net (195.2.25.69) 8.0% 25 330.7 330.9 330.3 333.5 1.0 10. AS??? 99.83.64.206 12.0% 25 331 .4 332.9 331.0 344.4 3.5 11. AS??? ??? 100.0 25 0 .0 0.0 0.0 0.0 0.0 12. AS??? ??? 100.0 25 0 .0 0.0 0.0 0.0 0.0 13. AS??? ??? 100.0 25 0 .0 0.0 0.0 0.0 0.0 14. AS??? ??? 100.0 25 0 .0 0.0 0.0 0.0 0.0 15. AS??? ??? 100.0 25 0 .0 0.0 0.0 0.0 0.0 16. AS??? ??? 100.0 25 0 .0 0.0 0.0 0.0 0.0 17. AS??? 150.222.241.183 4.0% 25 337 .8 337.9 337.0 342.1 1.3 18. AS??? ??? 100.0 25 0 .0 0.0 0.0 0.0 0.0
4shared.com :::::::::::::: Start: 2020-11-09T11:24:21+0000 HOST: mumbai Loss % Snt Last Avg Best Wrst StDev
- AS141193
103.156.51.254 0.0% 25 0.8 0.8 0.7 0.9 0.1 2. AS55410 122.15.103.94 0. 0% 25 1.0 1.0 0.7 3.2 0.5 3. AS??? 182.19.106.103 0. 0% 25 1.0 1.1 0.9 1.4 0.1 4. AS??? ??? 100 .0 25 0.0 0.0 0.0 0.0 0.0 5. AS55410 123.63.158.92 48. 0% 25 1.0 2.8 0.9 15.8 4.4 6. AS1273 ae28-100-xcr1.mar.cw.net (195.89.115.197) 12.0% 25 198.7 198.0 196.8 200.4 1.1 7. AS1273 ae51.xcr1.mlu.cw.net (195.2.21.186) 16.0% 25 204.5 204.5 204.1 206.7 0.6 8. AS1273 ae0-xcr1.mlb.cw.net (195.2.25.98) 8.0% 25 205.2 207.0 205.0 247.4 8.8 9. AS174 be1274.rcr21.mil01.atlas.cogentco.com (130.117.14.25) 0.0% 25 203.9 204.2 203.9 204.6 0.2 10. AS174 be2194.ccr22.mrs01.atlas.cogentco.com (154.54.61.29) 4.0% 25 214.2 214.3 214.1 214.5 0.1 11. AS174 be3223.ccr31.vlc02.atlas.cogentco.com (154.54.57.210) 16.0% 25 221.4 221.3 221.1 221.7 0.2 12. AS174 be3355.ccr31.mad05.atlas.cogentco.com (154.54.57.229) 12.0% 25 221.2 221.4 220.9 221.7 0.2 13. AS174 be2324.ccr31.bio02.atlas.cogentco.com (154.54.61.129) 16.0% 25 226.2 230.6 225.9 249.6 7.0 14. AS174 be2331.ccr41.dca01.atlas.cogentco.com (154.54.85.241) 8.0% 25 295.4 295.4 295.0 295.6 0.2 15. AS174 be2112.ccr41.atl01.atlas.cogentco.com (154.54.7.158) 8.0% 25 307.0 309.2 306.8 356.3 10.3 16. AS174 be2687.ccr41.iah01.atlas.cogentco.com (154.54.28.70) 8.0% 25 334.0 333.8 333.4 334.2 0.2 17. AS174 be2441.ccr31.dfw01.atlas.cogentco.com (154.54.41.66) 12.0% 25 339.2 339.4 339.1 339.7 0.2 18. AS174 be2763.ccr41.dfw03.atlas.cogentco.com (154.54.28.74) 8.0% 25 340.0 339.9 339.7 340.4 0.1 19. AS174 38.142.87.106 0. 0% 25 353.2 353.9 353.2 360.6 1.5 20. AS40824 204.155.149.43 0. 0% 25 353.4 353.3 353.0 353.6 0.1
:-)
cheers,
Nuno Vieira
On Sun, 2020-11-08 at 23:39 +0530, Anurag Bhatia wrote:
Hi Nuno
I just checked and did not saw any jumps in a many of the active latency checks I keep towards speed test endpoints for EU ISPs as well as routers of Indian ISPs connected to AMS-IX, DE-CIX etc. Did not saw any latency jumps. I have two source endpoints here: One is in downstream of Airtel and other in downstream of Jio.
Maybe what you faced is limited to specific upstream ISP/port which Indian networks use? Unlikely that impact was at submarine cable level.
Thanks.
On Sun, Nov 8, 2020 at 7:48 PM Nuno Vieira nuno@edgoo.io wrote:
Hello,
Is there any issues going on since yesterday that affect latencies and packet loss for some Indian ISP's connectivity from/to India/Europe ?
We noticed a +100ms increase and packet loss randomly during the day. (for multiple isps: TATA, Airtel, Vodafone India).
any clues ?
cheers,
Nuno Vieira _______________________________________________ INNOG mailing list -- innog@innog.net To unsubscribe send an email to innog-leave@innog.net
INNOG mailing list -- innog@innog.net
To unsubscribe send an email to innog-leave@innog.net
-- Kind regards, Nuno Vieira edgoo networksdatacentre and it services m: +351911728100 | p: +351211224705e: nuno@edgoo.io | nuno@hashpower.ptw: https://edgoo.io
_______________________________________________INNOG mailing list -- innog@innog.net To unsubscribe send an email to innog-leave@innog.net
Activity Summary
- 1057 days inactive
- 1057 days old
- innog@innog.net
- 2 participants
- 4 comments