This morning it looks like a majority of ISP’s using BT’s Wholesale Broadband Connect went offline due to a failure in one single exchange - link here to my ISP status page.    Exact scope of outage is still uncertain.   What is known is this affects ISP’s who interlink to WBC in Docklands, which will be a lot of ISP’s as most link in Docklands due to the cheap comms in the Redbus/Telehouse buildings there.     Wholesale Broadband Connect is used to link ISP’s to the 21CN and to exchanges providing the BT ADSL2 products.   Ie it connects ISP’s to the exchanges that have been upgraded to 21CN.    BT have you not heard of Single points of failure, and how you should design them out of your network?  The ISP I use, Andrews and Arnold are particularly affected with a total loss of service on customers who have been migrated to the 21CN this week.   It thankfully isn’t affecting all their customers as all have not been moved yet as it depends on your local exchange - I personally am unaffected thankfully.     BT will apparently be releasing a press release on this outage today to explain the reasons, which I do await with anticipation.

This problem really makes ISPs connecting to BT from 2 exchanges from 2 POPs look quite silly - as what’s the point in the ISP having a resilient link to BT when BT’s network itself is not resilient?

Further to this BT has been annoying a lot of smaller ISP’s, especially mine by changing their plans  with the WBMC product - which to explain is a product that allows ISP’s to remove the legacy old style BT ATM links into the ADSL network, and instead use one link to cover users on ADSL2, and those on exchanges which have not been connected to the 21CN yet.     By the looks of the new rollout announced, a lot of exchanges will be 21CN before the WBMC can even be used, which questions the need for the product (Q2 09).    For small ISP’s this is a big deal as the BT central links used to provide service to users present one of their biggest cost bases - price per MBit into the BT cloud is way higher than cost of transit to to the internet as a whole…

My ISP are one of I believe 2 in the UK who are currently offering native IPv6 provision.   Its useful for those of us who want to test real world Ipv6 from home prior to general adoption on the Internet.  All  ISP’s will have to use Ipv6 in the future due to address space depletion in the Ipv4 world..

Over the past few months a few of the users of A&A noticed that when using native IpV6 over L2TP to A&A small Ipv6 packets do not get through in some circumstances due to what looks to be a firmware issue in some Cisco equipment used in the BT 21CN and general ADSL platform - with BT’s equipment silent dropping the packets L2TP as they transit from the ISP back to the customer.     L2TP is used to pass a PPP session from a customers ADSL line directly to an ISP’s termination equipment, from which it goes onwards to the Internet.    BT should not be interfering with L2TP traffic.   BT have claimed to my ISP  this will not happen with the 21CN, but it shold not be happening even on the old service in my opinion at least.

A&A raised a ticket with BT to get this fixed a few months ago but have got no timeframe to fix this problem as it stands currently.

Users of AAISP doing research found the bug looks to be a IOS firmware issue within the BT network affecting certain pieces of Cisco kit used within their network.   Full (very technical) details can be found on this forum thread.    A workaround for now is to tunnel IPv6 in IPv4, but this does have drawbacks, namely a smaller Maximum transmission unit (MTU) - meaning you get less data per packet due to the encapsulation overheads.

To directly quote one the owner of A&A (Adrian Kennard) this morning in a online chat with myself:

“AAISP feel that it is incorrect for BT to be interfering with any PPP packets and have been trying to get BT to address the issue for months. We believe the root cause is an old (2 year) cisco IOS bug for which there is a fix”

“We are still trying to get a clear response from BT as to whether they even consider it a fault or not”

“We appreciate that if they agree it is a fault it may take time to roll out IOS upgrades, but would like a clear statement from BT as to what they propose doing”