Monday, March 20, 2017

Current Network Status - No Known Issues


Network Backbone - No Known Issues

CCS Datacentre DC14 - No Known Issues

CCS Datacentre DC1 - No Known Issues

Customer Networks - No Known Issues

Leased Lines - No Known Issues

VoIP - No Known Issues

xDSL Platform - No Known Issues

SharedBand - No Known Issues

Mail Services - No Known Issues

DNS Services - No Known Issues

Web Hosting - No Known Issues

Scheduled Maintenance - None


Www.CcsLeeds.co.uk

*** RESOLVED *** VoIP Inbound Call Issues

Hello.

At 16:11 we were alerted to the loss of connectivity to our largest telephony provider which is currently affecting inbound calls.

We also lost outbound capability for around two minutes but then hit a capacity limit which further prevented outbound calls. This was addressed by 16:20 and outbound calls are now flowing fine.

Update: 16:35 - Inbound calls are still affected and we are awaiting a carrier update.

Our office phones although not operating on the customer facing telephony system are also affected, so please email support@ccsleeds.co.uk at the present moment.

Update: 16:52 - It has been established that around 40% of our anticipated inbound calls are flowing as normal, although connectivity is still patchy at present.

Update: 16:58 - All calls have been restored to normal. Traffic levels are also returning to normal.
Further detail of the cause and remedy will be posted as soon as we have this to hand.

Update: 17:05 - We note that inbound calls are failing again, and are investigating as a matter of priority once again..

Update: 17:15 - We are thankfully in a position to confirm that all call routing is performing as normal and inbound and outbound calls are 100% completion rate and traffic is returning to normal.

I can only apologise for the disruption and an incident report will be dispatched to those affected tomorrow.



Www.CcsLeeds.co.uk

Thursday, August 13, 2015

***RESOLVED*** VoIP - Major Incident Ongoing

We have been made aware of an incident of calls failing on one of our VoIP call routers.

This is being handled as top priority although we do not presently have an ETA for service restoration.

Update - The supplier has addressed the issue and normal service has been returned.

Apologies for the inconvenience.

Www.CcsLeeds.co.uk

Sunday, July 05, 2015

***RESOLVED*** Major BT Outage affecting Northern Leased Line Services

Good morning.

At 11:10 we became aware of a number of Internet Leased Line services out of service.
Following this at 11:49 this number rose substantially.

Presently we are investigating the cause, although it is clear this is a national BT network issue.

Update 1:


BT (who provide these circuits) are indeed aware and have sent a very short message as follows:

We are currently seeing an major outage in the network, possibly any circuits that are based in the north of the UK that handover in the south of the UK will be down at the minute. “
Unfortunately this is neither detailed nor explanatory, and of course presently provides no indication of an ETA for restoration.

Update 2:


Good afternoon.

We now have a more comprehensive update from BT as to the cause of the outage which reads as follows:

Outage Description
Summary Our NOC team have identified that at approximately 11:10 a number of 10Gig Links were lost followed by a further 18 x 10Gig network links at approximately 11:45. This is impacting multiple services in the Midlands are and specific details regarding the impact will be provided as son as available. Numerous B2B customer will experience a total loss of service and Residential customer may experience 1 way audio or congestions issues.

Updates
A conference call is currently ongoing with our NOC, Network support, MIM, Transmission and management team to discuss the issue. Further updates will be provided as soon as available

Resolution
Root cause has not currently been identified. The next update will follow on Sunday 5 July at 14:00 or upon receipt of further information


Please bear in mind the Midlands is a core node for BT and although you may not be located near or in the Midlands, many Internet services rely on connectivity to London for onward connections to other parts of the internet, and hence the effect on your service.

Update 3 (14:39)

Good afternoon.

The latest just in from BT reads as follows:

Updates
Our NOC team currently have a transmission engineer working on site in Irlam to restore service to a monitoring server which will allow out NOC to further investigate the associated circuits and provide more specific impact details. Our Network support teams are currently rerouting traffic for impacted B2B customers to restore service however this could potentially cause congestion issues resulting in slow speeds and intermittent connectivity. There may be a further brief loss of service for B2B customers when service to the transmissions links are restored and traffic is rebalanced. We will continue to liaise with all involved parties and provide further updates accordingly.   Summary Our NOC team have identified that at approximately 11:10 a number of 10Gig Links were lost followed by a further 18 10Gig network links at approximately 11:45. This is impacting multiple services in the Midlands are and specific details regarding the impact will be provided as son as available. Numerous B2B customer will experience a total loss of service and Residential customer may experience 1 way audio or congestions issues


Resolution
Root cause has not currently been identified.   The next update will follow on Sunday 5 July at 15:30 or upon receipt of further information

Update 4: 15:10

We have recorded all customer connections back in service, but have not been advised if this is a permanent restoration or a temporary re-route and as such service is still "at risk"

Update : 16:10
Just in from BT:


Updates
Zaio have confirmed a full power outage at Sherrington, their co-lo site (which is where our fibre passes into from Leicester (Shankton) exchange. A Zaio engineer powered up the generator but this wasn’t able to restore power and an electrical engineer was called out. We have had confirmation that power is now up, the local electricity board said there was no fault – power came back up of its own accord and Zaio are looking into this. Our NOC are now checking through the 28 links to ensure they have all come back and are taking traffic. NOC do not believe there will be any congestion in the network as the network has automatically bounced traffic away from the exchange but now it is back the traffic will re-route automatically. The LSP servers are also currently being bounced to re-route traffic B2B which has already been resorted for the majority of customer.   Summary Our NOC team have identified that at approximately 11:10 a number of 10Gig Links were lost followed by a further 18 10Gig network links at approximately 11:45. This is impacting multiple services in the Midlands are and specific details regarding the impact will be provided as son as available. Numerous B2B customer will experience a total loss of service and Residential customer may experience 1 way audio or congestions issues.
 

Although the issue was formally resolved at 15:10, the following has just arrived from BT so I shall pass it on out of interest:


Updates
We are now able to confirm full service restoration and investigation into the RCA details is ongoing with our NOC and Zayo. Details of this will provided on closure after a observation period has been held to ensure no fuhrer issue are experienced.   Summary Our NOC team have identified that at approximately 11:10 a number of 10Gig Links were lost followed by a further 18 10Gig network links at approximately 11:45. This is impacting multiple services in the Midlands are and specific details regarding the impact will be provided as son as available. Numerous B2B customer will experience a total loss of service and Residential customer may experience 1 way audio or congestions issues.  



Apologies for the inconvenience today.


Www.CcsLeeds.co.uk