My home ISP for a few years now has been Bell Canada. More specifically their FTTH (Fibre To The Home) service. For the most part this has been quite good and reliable.
However there has been one thing that has bothered me over the years about Bell Canada’s network. It is their tendency to not peer or have a presence at local large Canadian IXes (Internet eXchanges) like TorIX. I am picking TorIX here as an example because its the one most relevant to me personally. As far as I can tell from the publicly available information, Bell Canada does not peer at any Canadian IXes.
According to data from bgp.he.net and PeeringDB, Bell Canada is only at the following IXes:
- Equinix Ashburn
- Equinix Chicago
- Equinix New York
- NYIIX
- SIX
As you can see, all of these are IXes in the US. As far as I can tell Bell Canada is the only major Canadian ISP to have this specific type of peering behaviour. Using TorIX as an example again, the other two major Canadian ISPs, Rogers and TELUS are both present at TorIX. Take a gander at TorIX’s peer list to see a bunch of other networks both big and small.
Why does this matter? Personally, all of the inefficiencies of this particular peering design bother me. Let me demonstrate with an example.
Say you are a Bell Canada customer in Toronto and want to use Cloudflare’s public DNS resolver service 1.1.1.1. Cloudflare is on TorIX and 1.1.1.0/24 is also being announced from that PoP (Point of Presence).
This is what your path to 1.1.1.1 looks like:
traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 52 byte packets
1 10.0.0.1 (10.0.0.1) 5.214 ms 1.153 ms 1.153 ms
2 10.11.1.233 (10.11.1.233) 6.313 ms 7.468 ms 10.554 ms
3 * * *
4 tcore4-toronto63_hu2/3/0/3_38.net.bell.ca (64.230.59.184) 9.664 ms 10.215 ms 15.936 ms
5 dis68-toronto63_10-0-0.net.bell.ca (64.230.106.107) 3.078 ms 2.361 ms 2.446 ms
6 bx6-chicagodt_0-6-0-0.net.bell.ca (64.230.79.85) 19.509 ms
bx6-chicagodt_0-7-0-0.net.bell.ca (64.230.79.87) 17.273 ms
bx6-chicagodt_0-6-0-0.net.bell.ca (64.230.79.85) 22.198 ms
7 141.101.73.14 (141.101.73.14) 18.019 ms 31.454 ms 29.588 ms
8 one.one.one.one (1.1.1.1) 29.296 ms 18.112 ms 18.128 ms
Notice how instead of hitting Cloudflare’s Toronto PoP, our traffic takes the scenic route to their Chicago PoP instead.
Now if your ISP is on TorIX, this is what the path to 1.1.1.1 looks like from Toronto. This example is from a server in Linode’s Toronto location:
traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets 1 gw-li1215.linode.com (45.79.116.1) 0.159 ms 0.118 ms 0.102 ms 2 10.215.64.4 (10.215.64.4) 0.331 ms 0.320 ms 1.359 ms 3 10.215.32.8 (10.215.32.8) 0.782 ms * * 4 10.215.32.49 (10.215.32.49) 1.483 ms 10.215.32.51 (10.215.32.51) 1.523 ms 10.215.32.55 (10.215.32.55) 0.400 ms 5 172.105.31.9 (172.105.31.9) 1.508 ms 1.467 ms cloudflare.ip4.torontointernetxchange.net (206.108.34.208) 3.067 ms 6 cloudflare.ip4.torontointernetxchange.net (206.108.34.208) 3.030 ms 2.345 ms 2.321 ms 7 one.one.one.one (1.1.1.1) 2.710 ms 2.033 ms 2.038 ms
20ms vs 2ms. The difference in latency here is significant. Now imagine just how much traffic going from Bell Canada’s customers in Toronto to Chicago instead of going through TorIX.
Why does Bell Canada do this? As far as I can tell this is a business decision rather than a technical one. Bell Canada likely prefers paid peering arrangements in the form of PNIs (Private Network Interconnect) over using public peering at IXes. Any reasonably competent network engineer would tell you that this situation is not ideal from a network design perspective and let’s go with the assumption that Bell Canada’s network engineering team is reasonably competent.
So in the end, this situation exists because Bell Canada wants to make money from paid peering instead of improving their network for their customers. If you are a smaller network in Toronto who wants to have a optimal path to Bell Canada customers, you have to pay for a PNI in addition to your port fees at TorIX and your other transit costs or you hope your transit provider also provides transit to Bell Canada so your traffic can use that path. That really fucking sucks.
So Bell Canada, please publicly peer at the available major Canadian IXes and improve your network for your customers.