C.C.S. (Leeds) Ltd Status Reports

Current Network Status - No Known Issues

2015-01-01T00:00:00.172Z

Network Backbone - No Known Issues

CCS Datacentre DC7 - No Known Issues

CCS Datacentre DC3 - No Known Issues

Scheduled Maintenance - None Planned

Customer Networks - No Known Issues

Leased Lines - No Known Issues

VoIP - No Known Issues

xDSL Platform - No Known Issues

MLPPP Bonded xDSL - No Known Issues

Mail Services - No Known Issues

DNS Services - No Known Issues

Web Hosting - No Known Issues

http://www.ccsleeds.co.uk/

RESOLVED - Virgin Provided Leased Line Issues.

2012-01-17T19:22:00.005Z

At 17:23 we noted a loss of service to around 90% of our VirginMedia provided leased line services affecting many of our customers.

Service returned at 17:37 to almost all customers and then promptly failed at 17:57 once again.

Most customers remained without connectivity until 19:20 when we saw a 100% return to service.

During this interval Virgin have confirmed a national network outage and have noted that senior engineering staff and manufacturer representatives are both on hand to address the issue.

Sadly while typing this update, I note a loss of service to a small handful of customers once again.

We can only apologise for the inconvenience and please rest assured we are keeping a close eye on the situation.

Further updates will be posted as provided by Virgin.

*** RESOLVED 19:20 ***

I am pleased to advise that all services have returned and traffic is flowing as expected. Virgin have noted that a full incident report will be posted tomorrow and this will of course be dispatched to all affected customers.

.http://www.ccsleeds.co.uk/

Leeds DC7 - Planned Maintenance

2012-01-04T14:53:00.003Z

Planned Maintenance Window Announcement

Window Start: 10th Jan 2012 - 22:00
Window End: 11th Jan 2012 - 06:00

In the above window, we will be performing a range of upgrade tasks on the London network, bringing on line further capacity and patching a vendor security bug at the same time.

We expect traffic to re-route on more than one occasion to Leeds DC7 and no other locations or services. There will also be two brief outages while cables are re-patched into new line cards and a router re-booted.

We would anticipate only a few very short minutes of outage in total, however the window of maintenance has been listed as 8 hours due to any unforeseen circumstances.

We apologise for the inconvenience in advance.

http://www.ccsleeds.co.uk/

RESOLVED - Leeds DC7 - Connectivity Issues

2011-10-31T16:22:00.002Z

We have recently become aware of connectivity issues in Leeds DC7.

At present we are investigating the cause.

Apologies for the inconvenience.

*** RESOLVED 16:42 ***

The issue has now been resolved and a full incident report will be promptly dispatched to those affected. Once again, apologies for the inconvenience this afternoon.

.http://www.ccsleeds.co.uk/

RESOLVED - VoIP - Intermittent Call Loss

2011-05-23T16:03:00.003+01:00

Good afternoon.

We have become aware of a loss of calls of approximately one in three on our Voip service at present.

This is being investigated as a Major Incident with the relevant provider and updates will follow as soon as possible.

*** RESOLVED 16:03 ***

We have been advised our PSTN providers' had an issue with some call routers in the London area where we break from VoIP to PSTN and that the issue has now been resolved.

Apologies for the inconvenience all.

.http://www.ccsleeds.co.uk/

Add to Technorati Favorites

RESOLVED - Leeds DC7 - Intermittent Packet Loss

2011-04-27T12:35:00.004+01:00

Good afternoon.

Over the past few minutes, we have become aware of a number of affected customers seeing packet loss to Leeds DC7.

Our network engineers presently investigating and we shall update as soon as we have further information.

*** UPDATE 12:41 ***

Telehouse have advised there is a power outage in one suite in the East wing which is affecting our carrier service connections into Leeds. Men are attending the suite as I type to address the issue and service should be restored within a very few minutes.

*** RESOLVED 12:45 ****

Service is fully returned to all. Telehouse have advised that faulty equipment in another rack on the same power feeds as one of our carrier connections caused the outage and that the faulty rack has been isolated pending further investigations by the owners. We do not anticipate further isseus, but apologies to all for the inconvenience caused.

.http://www.ccsleeds.co.uk/

Add to Technorati Favorites

RESOLVED - xDSL Intermittent Connectivity

2011-04-18T10:37:00.011+01:00

We have become aware of a loss of service to a large number of ADSL/SDSL lines around 10:10am today.

A fair quantity of lines are re-authenticating and dropping quite regularly and although continually returning to service, we have noted various lines where this is a recurring issue.

We have raised issue with the supplier who is investigating the issue as I type.

*** UPDATE 12:36 ***

We have been advised that due to a core network issue, the provider is routing all traffic away from Telehouse to Telecity while they address the underlying issue.

This will return stability and not affect end user service.

We await further information on the fuller resolution.

*** RESOLVED ***

We have been advised that the underlying issue has been addressed and the Telehouse handoffs will be re-enabled in the dead of the night.

Apologies for the inconvenience this issue has caused.

http://www.ccsleeds.co.uk/

Add to Technorati Favorites

RESOLVED - xDSL Packet Loss, various areas

2011-03-25T10:06:00.006Z

Good morning.

At 09:22 we became aware of packet loss in certain areas of our LLU xDSL network affecting ADSL and SDSL and also some bonded services.

These seem to be constrained to the west and south west of the country on various exchanges.

We have raised the issue with the carrier who is actively addressing this as I type.

*** UPDATE 10:08 ***

The carrier has advised this is being treated as a wide scale priority one incident and are acting as a matter of urgency.

*** UPDATE 11:12 ***

The carrier has just called back to advise that although this is still work in progress, there is sadly no ETA for resolution at this point in time and the matter is still being treated as top priority.

*** UPDATE 11:29 ***

We have been advised that this issue is considerably less widespread than we had first thought and is only affecting thirteen exchanges which seems to correlate with our affected userbase at present.

A list is available to any interested parties, please drop an email or comment on our post and I will provide this.

*** UPDATE 13:35 ***

The underlying provider Telefonica have located a faulty device on the network which they have disabled. This caused a ring flip which dropped pretty much all sessions as far as we can see although they immediately re-authenticated. At present the packet loss seems to have stopped and stable sessions are in evidence.

*** RESOLVED ***

An issue had been identified with network configuration within the carrier causing volumes of un-desired traffic at sites with smaller backhauls. This has now been addressed and all customers appear satisfactorily back in service.

We can only apologise for any inconvenience caused.

.http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - xDSL Authentication Issue

2011-02-28T11:12:00.002Z

Good morning.

It has just come to our attention that our Radius server is presently not responding to authentication attempts, causing a loss of service for anyone whose ADSL router is not presently in service, or attempts to re-authenticate.

We are addressing this as a matter of urgency.

*** RESOLVED 11:30 ***

The database service underlying the accounting data has been restarted and all is now well. All monitored users are logged into the service, but if your connection is still not authenticated, please restart your router.

Apologies for any inconvenienve caused.

http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - DNS Server Failure

2011-02-28T08:18:00.004Z

Good morning.

It has come to our attention that our primary resolver / caching DNS server has failed.

This is located on IP address 188.92.232.50

We are presently working in this issue, but in the mean time, please ensure you have our secondary 188.92.232.100 loaded in your equipment, or indeed swap them over. 188.92.232.100 is working correctly.

*** RESOLVED ***

DNS Lookup services are restored on the server. The server suffered a complete power supply failure and Telehouse were unable to restart it, so services have been transferred to another server. No IP address changes are required as these were moved with the services.

Apologies for any inconvenience caused.

http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - Core Network Instability

2011-02-27T15:57:00.002Z

Good afternoon.

At 15:55 this afternoon we were alerted to the loss of a Cisco switch on our core network in Telehoouse North.

The switch appears to have crashed and reloaded, and all customers are back in service.

An engineer has been dispatched to site to investigate in more detail if this is a physical issue or a software problem.

Further updates will follow.

*** RESOLVED *** - Following diagnosis of a faulty switch which was failing to recognise both its 10Gb modules and an interim replacement, we have now installed the maintenance replacement and full service is restored to all.

Major Incident reports will be produced in the morning and dispatched to all.

Apologies to all concerned, which are thankfully a limited subset of our customers.

http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - xDSL Issues, various areas

2011-02-14T09:23:00.005Z

Good morning.

Since around 8:50am we have been aware of issues in various exchange causing a loss of PPP session and a quick re-establishment for no apparent reason.

The wholesale suppliers are presently investigating urgently and will provide an update as soon as possible.

*** UPDATE 09:45 ****

We have seen a number of sessions stabilise, however we still have around 100 dsl lines that are extremely intermittent. Telefonica have affirmed they are aware and still working on the issue, but sadly with no ETR at this moment in time.

*** UPDATE 09:48 ***

We have been advised the cause of this issue is a fibre failure between London Hosting Centre and Telehouse North which is causing massive packet loss on the network and loss of PPP sessions. We understand that althought fault is logged with the fibre provider, further work around measures are being taken in the mean time.

*** UPDATE 10:51 ***

The interim measures have been implemented to ensure service is restored, albeit at risk due to a single fibre provision and all customers should be back in service. There may well be stale logon sessions on the network, and if so, please could you power down your router for a few moments and restart to return to service. Thank you

*** RESOLVED 11:24 ***

We have been advised that the broken fibre is now fixed, being a street side break between LHC and THN. The Telefonica fibre ring is now fully resilient and we are seeing traffic from both legs of the ring.

Apologies for the inconvenienve caused.

.http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - FTP Server Unavailability for some Web Sites

2011-01-02T12:56:00.004Z

Good afternoon.

We have just become aware of a none responsive FTP server, which follows on from last weeks' Windowsupdate patching/none restart issue.

Initial investigations would indicate this is more than likely hardware related as we can find no other cause, and an engineer is now on site testing the hardware. If this proves faulty, we will provision a new server and start the rebuild process once again.

*** Update ***

Please note the FTP hardware is reporting a diags fault, so new hardware is being assembled and configured.

It is also worthy of note a very few customers maybe seeing delays in email delivery as one of the anti-spam engines for long term customers was installed on the same server. All emails are being queued and will be delivered in due course.

*** RESOLVED ***

The issue is now resolved with a new FTP server in place and performing as expected.

Apologies for the inconvenience once again, but rest assured we are working on the issue right now.

.http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - Partial FTP Service Outage

2010-12-31T15:55:00.004Z

Good afternoon.

It has come to our attention that one of our FTP servers servicing approximately 30 web servers has failed to restart successfully after a Windows Update.

No amount of coaxing has brought this back to life and unfortunately the updates will not roll back.

We are at present building a replacement server from scratch, with hardware in place and Windows images presently being installed. Once this is complete, there is a substantial data restore to complete as this server also houses file data for certain customers.

At present we are unsure of a completion time, but a very rough ETA is 7pm today.

Apologies for the inconvenience, although this should only mean that you are unable to update your websites, and the sites themselves are functioning as expected.

*** RESOLVED 17:37 ***

The FTP Server and service is now fully restored and all files are back in place. We can only apologise for the inconvenience caused, and wish you all a pleasant New Year.

http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - xDSL Issues, Yorkshire and surrounding

2010-12-08T11:59:00.003Z

We have been advised our supplier of some xDSL services has an issue in Sheffield which is affecting xDSL services in the surrounding areas, including York and Lincoln.

They are presently investigating the cause and should revert with further details shortly.

*** Resolved 12:06 ***

All services are back up and running, and apologies for any inconvenience caused. BT have yet to revert with full reasoning for the outage, and I will update the post once we have further detail.

http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Scheduled Maintenance - xDSL Platform IPStream

2010-11-23T14:51:00.002Z

Work will be carried out on BT Wholesale's 21CN Broadband Network which will be carried out on the following dates and times;

Monday 29/11/2010 02:00 - 06:00
Tuesday 30/11/2010 02:00 - 06:00
Wednesday 01/12/2010 02:00 - 06:00
Thursday 02/12/2010 02:00 - 06:00

During these times line cards will be remotely reset to take new firmware.

There are 243 exchanges affected by this maintenance.

BT advise that customers will experience downtime of around 60 minutes, however we would ask customers to wait for the maintenance window to close and to reboot their router before contacting the support desk should connectivity issues persist.

.http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - Area Outage - xDSL / MLPPP Services

2010-11-22T14:05:00.004Z

Good afternoon.

A short while ago we became aware of an issue affecting xDSL services in the Yorks/Lincs/Lancs areas and have a good percentage of our services on the BT IPStream platform presently out of service.

At this moment in time, engineers are investigating the cause of the faults in conjunction with BT.

Unfortunately until the cause is established, we do not have an ETA for recovery.

**** Update 14:29 ****

We are aware all monitored services have successfully logged back into the LNSs, although no all clear has been provided by BT as yet, so services should be considered at risk at this stage. If your xDSL service has yet to reconnect, would you be so kind as to power off your router for a few seconds and power back up.

**** Resolved 15:02 ****

During the second core router crash, the cause of this issue, we were advised new firmware was installed to address the issue. We have been advised this will be the end of the issue, and all services appear stable.

Apologies to all for the inconvenience.

.http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - Partial xDSL Failure

2010-09-21T12:00:00.003+01:00

Dear customer.

We have become aware of an issue on the xDSL platform caused as we understand it by network flaps from our provider which have taken a number of circuits out of service.

They are aware of the issue and working towards a resolution at this present time.

This occured at 11:50 am.

Apologies for the inconvenience caused

*** Update 12:16 pm ***

All services have been restored and we have been advised the router and switching manufacturers are in communication to work towards a resolution of apparent compatibility. We are expecting emergency scheduled maintenance this evening to be advised shortly.

*** Resolved ***

The supplier has advised remedial measures have been taken and are confident the issue has been resolved.

Once again, apologies to all concerned.

.http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - PRTG Traffic Grapher

2010-08-17T17:38:00.001+01:00

It has come to out attention that traffic graphing stopped functioning at around 11am today. The server had run out of resources and was failing to commit log files to disk.

The services were re-started around 15 minutues ago, but as a consequence, there is no recorded traffic for this period, so please take that as a freebie on us!

Service monitoring was not affected as this is completed by out NightWatch platform on a separate server.

Apologies for the inconvenience caused.

http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - xDSL LLU Issues

2010-07-31T17:59:00.003+01:00

At 17:53 we became aware of a loss of service to xDSL customers as one LNS appears to have crashed.

We have alerted Telehouse to the issue and they are en-route to investigate.

*** RESOLVED *** 18:15

Telehouse have restarted the router and the crash dumps removed. These will be forwarded to Cisco for their consideration on Monday. However the router had an uptime of almost two years, so I guess that is not too bad!

Apologies for the inconvenience caused at this time.

.http://www.ccsleeds.co.uk

Add to Technorati Favorites

Resolved - Leased Lines Partial Loss of Service

2010-07-27T11:23:00.015+01:00

Good morning.

At 11:14 proactive monitoring is showing a loss of service on a substantial number customer leased lines.

These are all terminating on one VirginMedia piece of equipment in Telehouse and we are presently speaking with them to ascertain the issue.

*** UPDATE *** 11:32

VirginMedia have advised they have lost a large volume of equipment in Telehouse East and an engineer is en-route.

*** UPDATE *** 12:00

An engineer is in attendance and preliminary investigations have shown no cause. A senior network tech is en-route to assist. I am afraid there is presently no ETA for restoration of service.

Two MetNet nodes have been identified as being the root cause affecting a substantial number of both our and other ISPs and commercial customers.

*** UPDATE *** 12:30

Although we have not been advised as such, we have noted approximately 20 of our services are back up, although this still leaves a substantial number out of service at present. VirginMedia are still struggling to provide an ETA at present as feedback from the engineers on the ground is limited to the helpdesks while they are addressing the underlying cause. An ETA for recovery has unfortunately still not been provided.

*** UPDATE *** 13:00

We have just spoken with Virgin who are still investigating the none reload of two MetNet devices and are still not in a position to provide an ETA for recovery. We are pressing them for further information.

*** UPDATE *** 13:15

We have been advised that Virgin suffered a four device failure in Telehouse, the first two failing (crashing) caused excessive load to be put on the second pair. The first pair eventually crashed and recovered but the second pair remain down at present. We are awaiting an ETA for service recovery pending diagnosis of the second pair of failures upon which the remainder of our circuits are homed.

*** UPDATE *** 14:15

The fault has been attributed to a DC power failure to elements of the VirginMedia suite affecting more than one piece of equipment. Power engineers have restored power to the affected equipment, however some of the Alcatel hardware has not returned to service. The network engineers are now investigating the core site issues as a matter of urgency.

*** UPDATE *** 14:25

Virgin have just called to advise that the operating system on the two remaining affected devices, which are supplying a substantial number of our customer feeds has become corrupt during the abrupt outage and the senior techs are restoring the image as I type. Once this is complete the configs will be restored and service returned. Unfortunately this does not provide a timeline for restoration, but we are aware Virgins' network director is en-route to Telehouse now too to assist in the process.

*** UPDATE 15:17 ***

We note all customers are now back in service although note that the service remains at risk until Virgin provide the all-clear.

*** RESOLVED *** 15:47

VirginMedia have provided the "all clear" and confirmed all services are returned at full capacity.

Apologies for the inconvenience.

http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - Linx /LONAP Peering Issues.

2010-07-21T14:28:00.006+01:00

As short while ago, we became aware of instability on the Linx peering Lan.

Presently the Linx site is not accessible at all, and it would appear the "224" peering LAN is not functioning as expected.

We are attempting to re-route traffic over the "226" LAN while we get hold of Linx to understand what the issues are.

This may be causing network stability issues for many customers, as it is a primary cross-connect point for ISP's in the UK.

*** UPDATE 14:50 ***

All traffic has now been routed over alternate Linx Lans, but we have just been advised that Telehouse North, TFM1 Suite has lost power completely, which is affecting a large number of big name carriers and ISP's and there is congestion on many links at present to and from other ISPs and internationally.

We await an update from Telehouse reference the power situation.

*** UPDATE 14:59 ****

We understand that LONAP was also affected, and that power was not lost to the entire suite only part of it. Oddly the Linx website started working as LONAP returned showing zero traffic reporting for the duration. We await further
.

*** UPDATE 15:19 ***

Both Linx and Lonap ports are now back up and in service and traffic is flowing normally, with no further apparent packet loss. This remains under monitor for the present period until we receive the all clear from the carriers concerned..

*** Resolved ***

We have had confirmed that the issue was indeed a power failure to part of the suite in which Lonap are housed and that there was also an issue on the Linx network at the same time, effectively meaning the two largest public peering points in the UK were out of service at the same time. Unfortunately we are not at liberty to reveal the Linx problem as our Memorandum of Understanding prevents this. Never the less we appreciate they are working hard to prevent any possible similar issue in the future, and once again, apologies to all concerned.

Apologies for the inconvenience caused.

http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - xDSL Issues in Birmingham and surrounding areas

2010-07-21T10:52:00.005+01:00

We are aware of an issue that is affecting our xDSL customers that route via the Birmingham IPStream Connect nodes.

We are investigating the reasons for this problem and engineers will rectify the issue as soon as possible.

*** UPDATE 11:30 ***

Engineers have been dispatched to site and while en-route remote diagnostics are continuing. However we are no clearer as to the cause of the issue at present I'm afraid.

*** UPDATE 12:09 ****

Service has been restored to the affected users. We are still investigating the underlying cause at present.

We apologise for the inconvenience this may be causing.

*** Resolved 11:50 ***

We have been advised of the following:

The below report is a breakdown of the Birmingham IPStream Connect DSL problems experienced on Wednesday 21st of July between 09:25am and 11:45am.

Breakdown of the events:

At approximately 09:20am on Wednesday 21st our monitoring system detected a sudden drop in the number of online DSL connections running through the Birmingham IPSC node.

Initial remote diagnostics of the node and discussions with BT yielded no explanation for the sudden drop in sessions. Approximately 3.5% of our DSL handoff points were affected by this issue - in light of this, the decision was made to dispatch an engineer to the Birmingham node to carry out onsite diagnostics. The remaining 96.5% of the DSL nodes, including one in the same location remained operational and were unaffected.

Once on site our engineer discovered a micro-bend in the fibre link between the Birmingham node and the birmingham-te.core router. This fibre link has since been replaced and also rerouted within the Point of Presence to try and avoid further faults of this nature developing in the future. We are currently unaware of how this micro-bend occurred and do not have any recorded site visits that could have led to this failure.

We are continuing to investigate if any 3rd party access could be to blame.

At 11:45am the fibre replacement work was completed and sessions began reconnecting and normal service resumed thereafter.

We apologise for any inconvenience caused by this problem.

http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - Leeds DC7 Loss of Service

2010-05-27T18:55:00.004+01:00

We are aware of a loss of service for various customers at Leeds DC7.

We are presently investigating and will report further as soon as we know more.

*** Update 19:10 ****

We have suffered a loss of power to some racks in the site. All customer racks and core equipment are now back up and we are keyboard and monitor checking each server for stops at Bios and such.

*** Resolved/Closed 19:45 ****

All servers appear to be successfully backup and checked. There are a few remaining issues to deal with and we will move onto to these now.

Apologies for the inconvenience. A Major Incident Report has been dispatched detailing the circumstances and preventative measures to be put in place for the future.

.http://www.ccsleeds.co.uk/Add to Technorati Favorites

Resolved: Wireless Leeds Leased Line Issues

2010-05-25T18:57:00.002+01:00

Good afternoon.

At 14:53 we lost access to the East Leeds Wireless Leased Line controller covering areas of Leeds 7 in the most part.

Engineers were dispatched to site, where it quickly became apparent the drop cables from the radio transmitters had been damaged, we suspect by a hit from a ladder.

As a spare cable was in place, this was pressed into service while we re-ran replacement cables which are now complete and full resilience is restored.

As a further precautionaty measure, we will be installing protective steel capping to the cables to prevent a repeat of such an issue.

Service was restored at 15:52.

Apologies for the inconvenience caused, and an MSO notice has been dispatched detailing the incident to all affected customers.

.http://www.ccsleeds.co.uk/Add to Technorati Favorites

Resolved - Services Issues on LLU xDSL Platform

2010-05-22T20:28:00.005+01:00

Hi.

We became aware of issues in and around Bristol, extending to Bath and surrounding areas on our LLU platform occuring around 19:15.

We are informed the provider is working on the issue at this present moment.

Further detail will be provided as soon as we have it to hand.

*** Update ***

We have been advised that a substantial number of exchanges are affected as follows:

Chipping-Sodbury, Filton, Eastville, Bishopsworth, West, Winterbourne, Downend, Chepstow, Kingswood, Henbury, Portishead, Fishponds, Whitchurch, Easton, North, Trowbridge, Bristol-South, Melksham, Kingsmead

We are awaiting further details from Telefonica, but understand there is an issue with access to the location of the fault.

*** Update ***

Telefonica advise they have finally gained access to the trouble location and engineering should be able to provide an issue update within the next hour.

*** Resolved ***

Service was returned at 15:01 on the 23rd May 2010.

A notice of Major Service Outage will be dispatched to affected customers as soon as Telefonica provide it to us. Once again, apologies for the inconvenience caused.

.http://www.ccsleeds.co.uk/Add to Technorati Favorites

Resolved - xDSL Issues

2010-03-28T23:24:00.006+01:00

Hi.

At 22:45 we lost one of our interconnects with Telefonica, affecting a number of our ADSL/SDSL customers.

Presently we are awaiting a response from them as to the issue and will update in due course.

*** Update 23:53 ***

Service has been restored and apologies for the belated ticket update, but we have been playing headless chicken dealing with the affected customers in the last half an hour or so.

The wholesale provider of the connections has suffered two fibre breaks which would initially appear to be due to an accident in the datacentre. We have brought up interim connections in another site while the issue is address and service has been restored to all.

*** Resolved ****

All services are now delivered via optimal routes and service connectivity has been fully restored.

Apologies for the inconvenience caused.
.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - xDSL Platform Issues for certain customers

2010-03-10T08:59:00.007Z

Good morning.

We have become aware of issues with the xDSL platform affecting certain configurations of customers, mainly those running Cisco MLPPP, although a few single lines are affected too.

We are presently investigating the cause and will update as soon as I have furthe information.

Apologies for the inconvenience.

*** Update 09:05 ***

Service has been restored to all customers. Initial findings would indicate one of our LNS's crashed and automatically re-started. We will be investigating the cause shortly.

*** Update 16:25 ***

We have just suffered the second LNS crash today. We are presently re-starting the box and further escalating the issue. Apologies for any further inconvenience.

*** Resolved ***

Cisco replaced the Supervisor card, and now all seems well.
Thank you for your patience in this matter.

.
Www.CcsLeeds.co.uk

Add to Technorati Favorites

Resolved - Monitoring - Reporting Server

2010-02-27T01:48:00.001Z

Hi.

Good evening. I have just noticed that our monitoring and reporting server crashed a couple of hours ago. Embarrasingly I was sat watching a page refresh and realised the time and date wasn't changing.

The server auto rebooted during the backup at around 11pm and I've just restarted it now. We will investigate the full cause further tomorrow.

Anyone looking for statistics or availability figures will see a gap of just short of three hours. I am pleased to advise that a quick check of the carrier portals reveals no issues during the time our own Nightwatch system was down, but utilisation, latency and events will be missing for this period.

Apologies for any inconvenience this may have caused.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - Intermittent Routing Issues (Some Customers)

2010-02-22T08:45:00.006Z

Good morning.

We have become aware of intermittent and odd routing issues for some of our customers on older platforms affecting traffic to a variety of destinations, which are typically none Europe, although not exclusively.

We are presently speaking to the relevant provider to confirm the issue and address the problem and wil update as soon as we have more information.

*** Update 09:17 ***

We have been informed TiNet have a faulty line card and an engineer is now on site with a replacement. We are monitoring connectivity closely.

*** Resolved 10:50 ***

A solution has been implemented to address the issue while TiNet complete the resolution of the issue with their line card, which we have been advised is going to require a router reload this evening to bring the replacement card into service. As an interim measure, we have had some emergency cables run to link our legacy and new wide area networks together allowing us to inject the missing routes.

Apologies for the inconvenience caused.
.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - xDSL Intermittent Service Issues

2010-02-04T10:29:00.003Z

At 10:20 we noticed a drop in a large number of xDSL connections, most of which appear to be in the west of the country.

We are presently on the phone to BT to find out what is occuring.

*** Resolved 11:19 ***

Further to my earlier incorrect post, I am pleased to advise we have located the source of the problem. What initially looked like a BT issue was later found to be a core LNS repeatedly reloading due to the injection of EIGRP routes from a customers DSL connection. While these were blocked by an access list on the terminating LNS, unfortunately there appears to be an IOS bug that causes a crashdump and reload when the routes are injected.

We have modified the access rules to block these in the interim, and the issue is with Cisco for advise on potential IOS upgrades. This will be none service affecting should we proceed as we will move the connections to another LNS first.

Once again, apologies for the inconvenience.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - xDSL Service Issues, various areas

2010-01-22T13:00:00.005Z

All, at 12:56 we became aware of various customers loosing access to the xDSL platform, be that SDSL, ADSL and bonded ADSL.

We are presently investigating the issue, but initially it would appear to be a west side of the country service delivery issue in the BT network.

I will update further when we have fully established the cause.

*** Update 13:15 ****

All users are back in service and the cause appears to be a router crash in Telehouse North. We are presently investigating the issue with Cisco, but have no reason to expect a re-occurrence.

*** Update 15:45 ***

Our supplier has just advised they have cause to reload a core router around 15:55 which will cause a short outage. This is to change the operating system version on the hardware as they claim this is the cause of todays' instability.

*** Update 16:05 ***

Traffic was re-routed over alternate paths during this upgrade and little affect was experienced by users. All presently seems stable, so the issue is changed to "Under Observation"

*** Resolved 16:30 ***

All traffic is now flowing over resilient links and the affected router is performing well.

Apologies for the inconvenience.
.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - CCS xDSL Service Outage.

2010-01-20T12:40:00.003Z

All.

At 12:28 we noticed a volume of xDSL connections drop which were terminating on BT's Manchester 21CN nodes.

All connections re-authenticated at 12:39 and BT have admitted to a ERX crash/reload as the cause of the issue.

Apologies for any inconvenience caused.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - Emergency Planned Maintenance - Leased Line Services 6th November 2009 @ 1am

2009-11-03T21:29:00.005Z

Dear customer.

During routine maintenance in one of our racks in Telehouse earlier today, we have identified our fibre uplink to one of our Leased Line wholesale providers has suffered some damage.

Upon further investigation, it would appear the fibre tail has been trapped between some equipment during installations at some point in the past.

While the fibre itself is not fractured, and presently showing no errors, we are concerned that further disturbance may indeed break this tail.

As such we have ordered a pair of fibres to replace the in-service one and to leave a spare ready for service should such damage occur in the future.

We will be removing the damaged fibre from our switching infrastructure at approximately 1am and expect service to be restored within a few short minutes as soon as the replacement is attached at both the Metnet site, and our own infrastructure.

This will affect services to a number of customers, who have been advised individually of the situation. This outage does not affect all leased line customers, so if you have not received an individually addressed email, this change will not affect your service.

Apologies for the short notice of this maintenance, however we are keen to replace the failed fibre as soon as possible incase the damage worsens.

*** Update 01:52, 6th Nov ***

Apologies for the slightly late start folks, but the outage commenced at 01:41 and service was restored to all at 01:44.

All customers on this switch are Fully Managed Leased Line services and we have affirmed every one is back in service.

Following a day of monitoring, all services look perfectly stable, therefore this post is closed.

Thank you for your patience.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - Leeds DC7 Power Issue

2009-10-02T18:37:00.003+01:00

At 18:28 we lost access to some of our servers. Presently engineers are on their way to site to investigate.

Further updates will follow.

*** Resolved ***

Upon arrival we discovered power loss to a dual (A+B) feed rack which contains our Cisco ASA cluster servicing a number of customers within the site. Contained in the same sectional rack is a quarter dedicated to one customer, who had suffered a catastrophic power supply failure. Although only one of his pair of power supplies had failed, it had caused both circuit breakers feeding this rack to trip.

The customers' server has been isolated pending repair/bench testing and power returned.

The service affecting period was 18:28 to 18:59.

Apologies to all those affected, and next week we will be taking further steps to totally independently provision the Cisco ASA cluster in the same manner as our core routers and switches with physical diversity.

Regards,

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - Packet Loss, Leeds DC7

2009-07-04T14:34:00.005+01:00

Dear Customer.

We are presently aware of a considerable volume of packet loss at Leeds DC7.

The issue appears to be a BGP session related one, and we are presently working towards a resolution.

*** Update ***

We are still aware of intermittent routing to some other providers networks, and are still working on the issue.

*** Resolved 17:15 ***

The issue has been mitigated and stable service restored. Initial investigations showed multiple re-routing incidents, which we later discovered were due to a Denial of Service attack targetted at a customers' customer. Relevant ports have been filtered and we will look at further preventative measures early next week.

A major incident report will be dispatched to those affected early next week too.

Apologies for the inconvenience caused.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved: IP Transit / Connectivity Issues

2009-06-04T17:50:00.005+01:00

Dear Customer

At approximately 17:40 we lost connectivity to a number of our switches and our primary IP Transit provider.

We are presently investigating the issue and will respond as soon as we know further.

Please note in certain circumstances, dependent on your ISP and services you take from CCS, you may not be able to reach our core website or support systems at present.

*** Update 18:00 ***

We are still trying to pinpoint the exact cause of the failure, however at present we are aware dependent on where you are coming from (i.e. your ISP) you may not be able to reach your websites/email accounts and other services located on our London network.

If you are an ADSL / SDSL / Leased Line customer, you maybe noticing loss of service to certain external sites, however connectivity to most UK ISPs is still in place.

*** Update 18:50 ***

We have discovered the issue is related to the loss of one of the largest datacentres within London, not one we occupy, but one which elements of our upstream transit providers live within. At present we are awaiting a response from them and they have a call queue of 132... I guess they know there is an issue.

Please also be aware a good number of the larger hosts utilise this site and you will find a large number of websites off-line entirely. These are not our sites, but some large commercial brands and such like, due to the issues at this site.

*** Resolved 19:20 ***

After a switch re-load our elements of service are back at full strength, however there are still issues with some other datacentres in London at present. Unfortunately these are out of our control.

Please Note: This is NOT affecting any of our connectivity in the Leeds datacentres, but is specifically a London-centric issue.

Apologies in advance for the issue.
.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - ADSL Service Issues, Various Areas

2009-05-31T14:20:00.002+01:00

Dear Customer.

At approximately 14:20 a large number of ADSL sessions terminated for no apparent reason.

At present we are awaiting a response from our upstream provider as the to cause of the issue.

Please note, this is not affecting ALL customers and is not affecting ANY of the managed service customers. This issue is confined to the Family and Business ADSL services in a number of areas.
Apologies for the lateness of this post, however efforts have been concentrated on trying to address the issue thus far.

Further updates will be posted as we gain a greater understanding of the issue.

*** Update 15:20 ***

I have been advised by Engineering that the issue has been resolved, however I am not fully aware of the details at present.

*** Resolution ***

Upon investigation, our provider discovered that due to a mis-configuration error, those in certain IP address ranges were not routing correctly. Although there had been a platform failure, all connections failed over to one of the resilient platforms, however some routing rules were missing meaning the traffic didn't route. The rules have been added and the config double checked to ensure such an issue does not reoccur.

Apologies for the issue.
.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - MLPPP xDSL Packet Loss

2009-05-30T11:52:00.004+01:00

Dear Customer

At 11:24 we started experiencing packet loss on our terminating MLPPP Routers.

A quick investigation of the graphs shows we are suffering a couple or so Gig of a Denial of Service Attack.

Presently we are trying to isolate the cause and put filtering in place to mitigate this.

*** Update 12:00 ***

We have located the source of the attack, which appeared to be destined for a customer with a co-located server. Filters have been put in place to mitigate this and we are watching closely for further issues.

*** Resolved ***

No further reoccurrence has been noted, therefore case closed.

Apologies for the inconvenience and the lateness of this post.
.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - xDSL MLPPP Outage

2009-05-27T14:27:00.014+01:00

Dear customer. At 14:22 we suffered a service failure on our MLPPP Platform.

At present we are investigating the cause.

*** Update 15:15 ***

The issue appears to be one with our upstream provider with whom we are working to resolution.

*** Update 15:41 ***

We have established we are not seeing the authentication for services being passed over by our upstream provider and are presently working with them to establish the cause of the issue.

*** Update 16:43 ***

We are still awaiting resolution of the issue with our upstream and BT who are presently trying to diagnose the probelms.

*** Update 19:10 ***

After much to-ing and fro-ing with BT and our upstreams, it has been decided this is an interconnectivity issue between ourselves and them.

Our director and technical manager are presently on their way to London armed with spares to be on hand to assist the situation in any way we can. However it would appear that the issue is with intersite cabling, or so we are being advised. Cabling has been swapped and is presently being tested however we are not convinced that this is the issue and furthermore cannot understand how we can have had two failures on two cables terminating (at our end) on two switches, both of which are showing link status up, but over which no traffic will flow.

ETA for site for our tech manager is around 30 minutes. Our director will be around an hour away.

*** Update 21:40 ***

We have confirmed both of our cables supplying ATM sessions from xDSL have been damaged. They are both gig cables and one has one open pair and the other two open pairs.

Presently we are trying to locate the fault, but are being hampered by Telehouse's no out of hours cabling policy (hence the reason for two cables in the first instance).

*** Update 01:25 ***

We are still currently battling with Telehouse's archaic "no out of hours" cabling policy and are investigating potential options for utilising spare cabling already in place, borrowed from another provider and other options. Please rest assured all relevant members of the team are still working towards a resolution as soon as is possible given the constraints we are working within.

*** Update 02:20 ***

We have reached a deadlock with Telehouse who are point blank refusing to allow out-of-hours cabling in line with their policy and are presently booking in emergency cabling for first thing in the morning.

*** Update 07:00 / 28th May ***

One of Telehouse's approved cable run teams are booked in and on standby at site awaiting the arrival of Telehouse cabling manager to sign off their permits to work. As soon as we have this signed permit, we anticipate around two hours to run and test the cables plus a further short interval to connect and return to service. Until we get the permit to work signed it is quite hard to provide an ETR, however we are hoping for completion by perhaps ten am at present, all being well.

*** Update 09:40 ***

Replacement cabling has been installed and is tested and terminated at our rack. We are awaiting our upstream supplier to connect their end and service should resume very shortly.

*** Update 10:05 ***

We have been advised that a remote hands slot has been booked by our upstream supplier to get the cables connected.

*** Resolved 10:33 ***

The cables have been connected and all monitored/managed services are back in service.

A Major Incident Report will be published to those affected just as soon as the report is returned as to the cause of the mutiple simultaneous cable has been received.

Apologies for the inconvenience caused.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - xDSL Service Outage (2) in the Midlands and South West

2009-05-18T11:44:00.006+01:00

At 11:30 we experienced a second xDSL outage in the Bristol and Exeter areas affecting customers in Reading, Guilford, Evesham, North Wales and other surrounding areas.

We are awaiting an update from our supplier as to the issue.

*** Update 12:30 ***

Upon further investigation we have discovered this is not a regional issue, but an MLPPP (or Bonded xDSL) platform issue. We are working with the carrier concerned to a resolution at present.

*** Update 13:30 ***

Work is still on-going towards a resolution at present.

*** Update 14:05 ***

Our supplier tells us they have identified the cause of the fault and are rectifying it at present and we should see service and sessions starting to restore very shortly.

*** Update 14:16 ***

All Bonded / MLPPP Connections are re-authenticated and service is restored. We are still awaiting an explanation as to the cause of the issue and indeed what was done to resolve it, however we are just happy to have service back to all.

*** Resolved ***

BT have advised a management supervisor was reset and sessions re-established at the next re-try interval post this point.

Apologies for the inconvenience caused.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - xDSL Platform Failure, Sheffield and Nottingham

2009-05-18T04:30:00.002+01:00

Dear customer.

At 04:24am we became aware of an issue with the BT interconnect site in Sheffield which provides DSL services to the surrounding areas, covering from Stockport to Hull, and Lincoln to Harrogate.

At present the cause of the issue has not been identified and engineering are on their way to site to investigate in detail.

*** Update 07:23 ***

The circuit supplier has been called into attendance as there is no sign of equipment failure or otherwise.

*** Update 10:13 ***

We have been informed by the carrier that the fibre break has been located and is in the process of being rectified.

*** Resolved 11:30 ***

The fibres have been re-spliced and service is returned to affected regional customers.

However please be aware that if you are a Bonded or MLPPP customer, a further outage has occurred which may still be affecting service - please see above for more details.

Apologies for the inconvenience caused and a Major Incident Report will be sent in due course.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - xDSL Platform Stability - Sheffield and Surrounding

2009-05-09T12:49:00.004+01:00

Dear customer.

At 12:45pm we became aware of an issue with the BT interconnect site in Sheffield which provides DSL services to the surrounding areas, covering from Stockport to Hull, and Lincoln to Harrogate.

At present the cause of the issue has not been identified and engineering are on their way to site to investigate in detail.

As we have further, I will post updates, but in the mean time, apologies for the inconvenience.

*** Update 14:13 ***

Initial investigations at both the Sheffield and Nottingham PoP sites have not revealed the cause. The fibre OTDR team are on route to locate the fault with sectional testing.

*** Update 16:25 ***

The fault appears to be near Poplar, North London and a fibre crew have been dispatched to the area to pinpoint the fault and start to rectfy the issue.

*** Update 18:09 ***

We have been advised that the fault at Poplar is un-related to the fibre issue and that engineers are now walking the fibre route approximately 30km from Sheffield to locate the fault. They will then proceed with splicing and/or repair.

*** Update 22:05 ***

We are starting to see connections returning in some areas.

*** Update 23:00 ***

All customer connections have returned to service and are logged into the LNS's.

At present we have not officially been advised that the repair works are complete and therefore services must be considered "At Risk" until such time as we've been given the all clear from the maintenance team.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - xDSL Platform Stability - Kingston and surrounding

2009-04-01T10:27:00.004+01:00

At 10:26, we experienced and outage on the BT interconnect node handling traffic for Kingston and surrounding areas.

We are aware of failures in service in the Reading and Godalming areas plus surrounding.

BT and the network provider are currently investigating the issue and we will update as soon as we know further.

*** Update 11:30 ***

Carriers have advised there appears to be a facility issue with power to the NTE and associated equipment. Our engineers plus site facilities management are now on the way to the site to investigate further.

*** Update 13:00 ***

Due to the march in London, site access is impared and travel is taking far longer than anticipated.

*** Update 14:35 ***

Engineers are now on site assessing the situation - further will be posted as we know more.

*** Update 14:48 ***

Some facilities functions have been restored however there are still outstanding issues to some areas of the building which we are awaiting an update on.

*** Update 15:26 ***

We have made temporary facilities available to restore service to the majority of users. However a further brief interruption may be required this evening to remove the temporary measures and return full supplies to the equipment. Thank you for your patience in his matter.

*** Resolved 18:10 ***

The temporary supply has bee removed and permanent service restored. It was eventually discovered to be a utility company issue in a nearby sub-station which caused the outage, allowing the site UPS's to drain and therefore leaving us without power.
All connections seemed to have returned to stable service.

Apologies for the inconvenience caused.
.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - Loss of connectivity Leeds DC7

2009-03-27T13:57:00.002Z

At 13:57 we lost connectivity to Leeds DC7.

Although this has since recovered at 14:01 we are un-sure of the cause at this stage.

I will post further as soon as I know the cause of the issue.

*** Update ***
The root cause of the outage has been discovered to be the loss of power to an area in Telehouse, London which caused a routing change to propagate throughout the network, taking approximately 60 seconds per router to update the tables.

We are awaiting an explanation from Telehouse Europe as to the cause of the power failure. Power has since been restored, but the reason for the failure is not yet known.

Telehouse have admitted to a mis-labelling issue of the supplies to one of our racks and while they were performing works on a rack in the same position but in the previous row, accidentally removed the power to our rack. Unfortunately the affected rack is provisioned with only one power feed, although all the critical equipment, including the affected router are provisioned with twin power units.

As such we've raised a request for an additional power feed to be provisioned to try and mitigate such stupidity in future.

Once again, Apologies to those affected.
.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - Widescale DSL Outage

2009-03-12T15:10:00.007Z

At 15:08 we suffered a widescale DSL outage. At present the cause is not know but our engineers are investigating.

Apologies to all those concerned and as soon as I have further I will advise.

*** Update 15:29 ***

We have discovered this to be at the point of hand off of the network sessions from BT within London. We are awaiting an update from BT on the matter.

*** Update 15:43 ***

We have been advised that BT are trying to steer the sessions across another part of their network to be able to terminate them to us.

Work is still in progress at present.

*** Updte 16:10 ***

We have been in constant communication with BT over this matter and as yet have not established a resolution. Please rest assured we are still working on the matter.

*** Update 16:15 ***

BT are advising that they think our handoff port has failed and the secondary is not taking the over service. Telehouse have swapped the cable over into another port but we still are seeing no service at present. BT are sending an engineer to site to change the port at their end.

*** Update 16:25 ***

BT have identified a faulty blade in a router in Telehouse East. Engineers are on site installing a replacement blade and configuring this for service.

*** Update 16:52 ***

Sessions have started re-establishing as BT have moved out connection to another delivery card. Some circuits are taking a few moments to re-establish. Please bear with us, but service should be returned completely in the next few minutes.

*** Resolved 16:54 ***

All customer connections have re-established and service has resumed as normal.

The Major Incident Report will be sent out as soon as possible due to the late hour.

Thank you for your patience in this matter.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - xDSL Platform Stability

2009-02-24T18:08:00.005Z

We have become aware of an issue with connectivity in relation to the IPSC interconnect points in Sheffield, Nottingham and Peterborough causing a loss of service for xDSL customers in the surrounding areas.

At present we are awaiting an update from the carrier as to what has occured.

*** Update 20:00 ***

A fibre fault has been identified in the Trent Valley area and technicians are being mobilised to start the locate and repair process.

*** Update 22:00 ***

The fibre teams are in place to commence testing and resolution of the issue.

*** Update 06:00 ***

Service has been restored.

Once again, Apologies to those affected.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Restored - Leeds DC7 - Loss of Connectivity

2009-02-16T16:40:00.003Z

All.

Monitoring has recently alerted to a loss of connectivity into Leeds DC7. This is affecting various hosting and colocation customers plus leased lines in central and east Leeds.

Those related to Leeds DC11 are presently unaffected.

We have not yet established the cause, but engineers are on site investigating as I type.

Further update will be published as I know further.

*** Update 16:40 ***

We appear to be suffering a denial of service attack which is causing a core router to lock up repeatedly. At present we are investigating the device which is under attack in an attempt to isolate it and also block the traffic at source.

*** Update 17:05 ***

We have established the cause of the issue to be between Lonap and Cogent. As present we are addressing the issue with these suppliers. Connectivity is available to we estimate 80% of the internet with intermittence to other suppliers depending on the route taken to get into Leeds.

*** Update 17:19 ***

Service has been restored - We are still trying to establish the full facts at present, however routing seems to have returned to normal.

*** Resolved ***

The cause has been discovered and it appears to be a badly configured router injecting false routes into our and all other providers networks.

The Register has more on the matter : http://www.theregister.co.uk/2009/02/17/net_instability/

Apologies for any inconvenience caused.
.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Planned Maintenance, Leeds DC7 - 1400 to 1500 070209

2009-02-07T15:30:00.000Z

Following on from the issues last month it has been decided to press ahead with some upgrade works on a more urgent basis than previously planned.

This work will go ahead between 14:00 and 15:00 on the 7th Feb 2009

During this time we will be replacing some core routing equipment and an associated pair of switches.

All configuration works have been completed in advance and tested as far as is possible to do so in an off line nature.

We are not anticipating an outage of the Leeds DC7 Datacentre but sub-optimal routing is quite likely for a number of short periods during this window while routing tables are re-loaded and updated.

This will affect those hosting/co-located within Leeds DC7, those with Leased Lines terminating there (Leeds Central and East primarily), those utilising our wireless Leased Line services.

Email notifications will be sent to those who will be affected within the next few minutes.

We apologise for the short notice of this issue, but we have been awaiting an element of the hardware to arrive, which it did earlier today.

Updates will be posted to this site during and after the works.

Please rest assured we will address any issues with utmost urgency if they occur, including a planned backout procedure if necessary.

*** Update ***

This maintenance went without hitch and there was no customer service impact. Thank you for your patience and apologies once again for the short notice period.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

xDSL Platform Stability - Resolved

2009-02-07T07:35:00.000Z

We have just been made aware of an issue with the xDSL platform interconnects for customers served in and around Sheffield.

This is currently affecting customers from York to Derby including Leeds, Lincolnshire, Cheadle.

We do not as yet know the cause of the issue and are working on it as I type. As soon as we have further I will update the post.

*** Update 16:04 ***

Our carriers have confirmed they have suffered a "High Order" failure in the area. We await further updates as to the resolution.

*** Update 18:10 ***

We have been advised crews have been tasked to the fault and are currently travelling, but being hampered by traffic and bad weather. BT are declaring this fault under their MBORC notice (Matters Beyond our Reasonable Control) and as such cannot as yet provide an ETA for resolution.

*** Update 20:50 ***

The faulty fibre has been located near Leicester and jointing teams are on route to rectify this. Again weather is hampering their efforts, but work will continue until resolved.

*** Update 22:20 ***

While the stretch of faulty fibre has been identified, the exact location of the fault cannot be established as this appears to be a crush or fracture and not a total break. Microbend OTDR diagnosis is being performed joint by joint to try and pinpoint the issue but snow covered troughing and general conditions are not helping the situation.

At present, I am afraid there is still no estimated time for recovery.

*** Update 01:45 ***

We have been advised that the fault has been located and that a replacement run of fibre has been installed ready to commence splicing. Unfortunately there is a joint in the middle of this run to T off to another customer so three jointing teams have been tasked to the site to complete this work. One is in place, on is imminent, but the third team have to travel. ETA for arrival is 03:00 at present, conditions allowing.

*** Update 04:55 ***

The first fibre tubes have been re-spliced and service has returned to virtually all of our customers.

*** Update 06:00 ***

The remaining tubes are spliced and we now have a full service for all customers. However work is still on-going on the remaining tubes so the service is to be considered "at risk" at present.

*** Resolved 07:35 ***

All works have been completed and just remains for the fibre to be re-buried.

Apologies for the inconvenience caused.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Update: MBORC declaration due to recent heavy snowfall

2009-02-06T19:11:00.000Z

Further Openreach Update: MBORC declaration due to recent heavy snowfall

This communication follows recent MBORC declarations, issued as special alerts to all Communications Providers, following heavy snow across the UK.

Matter beyond our reasonable control (MBORC) / force majeure event has been declared in some parts of the UK. This communication provides an update on where we have been able to lift the MBORC declaration and on areas where MBORC has subsequently been declared.

The current impact on service, on a regional basis, is as follows:

South West - overnight snow and low temperatures have meant continuing bad road conditions. MBORC is currently in place in; Bristol, Bath, Swindon, Gloucester and Taunton. In addition, MBORC has now been declared, with effective from 08.00 today, in; Plymouth, Exeter, Salisbury and Bournemouth.
Wales & Midlands - with effective from 08.00 today, MBORC has been lifted in Cardiff, Swansea and Peterborough.

Scotland - continuing snow and low temperatures in the patches where we have open declarations. MBORC is currently in place in; Grampian, Highlands South (parts only), Highlands North (parts only), this MBORC has been extended to included a further 15 exchanges towards the west.

London & Home Counties - conditions remain poor in many areas. MBORC is currently in place in; Oxford, Aylesbury, Northampton, Luton, Maidenhead, Reading, Basingstoke and Guildford.

South East - the MBORC declaration was lifted at 08:00 today on all declared areas; Chelmsford, Crayford, Croydon, Edmonton, Esher, Greenwich, Lea Valley, Richmond, Romford, St Albans and Watford.

Our expectation is that if we have no more snow, and road conditions improve, we will be able to recover by very early next week. In the meantime, we are continuing to review this situation, and the weather, and will keep you updated as we make progress.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Potential Delay in resolution of xDSL and Leased Line Faults.

2009-02-03T17:43:00.002Z

BT Openreach MBORC declaration

BT Openreach have declared a force majeure event in several areas. The text of the declaration is below:

MBORC declaration due to recent heavy snowfall

Following heavy snow fall Openreach is now declaring a matter beyond our reasonable control (MBORC) / force majeure event in some areas.

The declaration is effective from 09:00 am on Monday 2nd February 2009. It applies to all provision and repair activities caused by the severe weather conditions in the specified areas.

The recent heavy snowfall has impacted Openreach in two main ways. Firstly, the reduced mobility of our engineering workforce and, secondly, the reduced numbers of Openreach people able to get to our control centres. This resulted in a number of missed jobs and appointments where, whilst every effort was made to contact those end users, this was not always successful.

We have increased our engineering resource capability for the rest of this week, especially in those areas worst affected, and put our control centre contingency plans into operation. The impact on service, on a regional basis, is as follows and is based upon the current seven day weather forecast.

Scotland - impact seen in East of Scotland. We are working normally and full recovery is expected by 9th February 2009.

Northern England - impact seen in the Pennines, Tees Valley, Yorkshire and Lincolnshire. Full recovery is expected by 9th February 2009.

Midlands & Wales - impact seen in East & Central Midlands. Full recovery is expected by 9th February 2009.

London Home Counties - impact seen in London and the Thames Valley areas. Full recovery is expected by 10th February 2009. We have declared MBORC in the following areas: Basingstoke, City, Docklands, Guildford, Heathrow, Southbank, Wembley, Westend and Westminster.

South West - impact seen in Solent. Full recovery is expected by 10th February 2009.

South East - impact seen in Sussex, Kent North and South London. Full recovery is expected by 10th February 2009. We have declared MBORC in the following areas: Chelmsford, Crayford, Croydon, Edmonton, Esher, Greenwich, Lea Valley, Richmond, Romford, St Albans and Watford.

We will review this situation on a day-by-day basis and will lift the MBORC in each area as soon as service returns to normal. In addition, we continue to watch the weather and its impact and will advise should we need to declare MBORC in any other areas, or if our recovery times are extended as a result.

Openreach Communications

Update: MBORC declaration due to recent heavy snowfall.

A matter beyond our reasonable control (MBORC) / force majeure event came into effect at 09.00 on Monday 2nd February in some areas in the South East and London & Home Counties.

Since then snow spread across the country and MBORC, effective from 15:00 on Monday 2nd February 2009, has subsequently been declared in some areas of the South West.

The areas affected by this declaration are as follows:

Plymouth: Beaworthy, Chagford, Holsworthy and Yelverton

Truro:Threewater, Devoran, Mawnan Smith, Penryn and Wadebridge

Taunton:Paracombe, Combe Martin, Ifracombe, Shirwell, Torrington, Clayhanger, Clovelly and Bideford

The situation in Plymouth and Truro is recovering and we expect to be able to end the declaration in these areas by the end of today. We will continue to monitor the situation in Taunton, where severe weather conditions remain and further heavy snow is expected.

In all areas of the UK, we are continuing to review this situation, along with the weather, and will keep you updated as we make progress.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Restored - Connectivity Leeds DC7

2009-02-02T12:25:00.002Z

At 12:25 we became aware of a loss of connectivity to Leeds DC7.

At present we are investigating the cause.

Further updates will be posted as we discover the cause.

**** Update 12:37 *****

Service has been restored. We are still at present trying to fully establish the cause and are parsing the routing logs. As we have further I will update the ticket.

**** Resolved *****

The issue with service outage has been isolated to one of our carriers kindly "donating" a multicast broadcast storm at one of our core routers causing all available memory to be consumed and forcing a crash dump and re-load. The carriers are currently on site investigating this issue, but in the mean time filtering has been put in place to prevent the multicast traffic entering the network in the first instance.

Apologies to those concerned.
.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Some Intermittent Connectivity Issues

2009-01-24T22:40:00.001Z

We have been advised that a shared network site at Poplar London is experiencing power issues.

This is affecting services provided by BT, NTL:TW and Global Crossing amongst others.

The service affect for our customers is intermittent ADSL/SDSL connectivity, intermittent leased line connectivity, intermittent web hosting, and intermittent loss of service due to multiple BGP re-routes to our Leeds datacentre.

We have been advised that this issue is caused by a power distribution issue in Poplar which is a site shared by all three carriers.

***** Update 12am *****

Engineers are on site and working on the issue. We have nothing further to add at this stage but will keep you informed as further information is published.

***** Update 00:20 *****

We have isolated this particular network segment into Leeds DC7 and 14 and no longer have issues with packet loss into the datacentres, however works are still on-going at Poplar to remedy the other affected services.

***** Update 00:40 *****

Our leased line traffic has been re-routed away from the affected site returning full service to those affected customers.

***** Update 03:35 *****

We are pleased to advise that the issue of Packet loss on the xDSL services has been resolved with assistance from BT and a traffic re-route to another handoff point.

Apologies to all concerned

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - xDSL Platform Stability

2009-01-23T23:23:00.001Z

All, we are aware of an issue with connectivity in the Manchester and surrounding areas on the ADSL / SDSL platform.

This is extending to Leeds, Sheffield, Rotherham, Lincoln, some of North Wales, the Peak District, Derby, York and other surrounding areas.

We are actively working with our network provider as I type to get to the bottom of the issue.

At the time of typing issues have been evident since 12:50pm

***** Update 13:23 *****

We have discovered this issue is one of authentication on the new BT IPstream Connect platform and are awaiting a response from BT.

***** Update 14:03 *****

We have been advised that BT are still working on the problem and have no further news to report at present I am afraid.

***** Update 15:00 *****

We have discovered this incident is a fibre break between Leeds and Sheffield which we suspect is related to vandalism near Rotherham where a cable has been badly burnt also. Currently BT can provide no ETA on the recovery of services for which we duly apologise.

***** Update 15:10 *****

We have been advised that the SLA for recovery of service is to be six hours, although we are pressing for an earlier recovery than this. Please accept our sincere apologies but our hands are tied by the carrier and the time taken to re-run a considerable length of damaged fibre and splice.

***** Update 20:30 *****

We have been advised to expect service restoration within the next hour or so as works are progressing.

***** Resolved 23:23 *****

We have been advised the replacement section of fibre is in place, spliced, tested and terminated and sessions are quicky re-establishing. If you still do not have service, please reboot your router to ensure it correctly re-authenticates.

We apologise for the extended outage and rest assured we will be taking this up with our account management first thing on Monday to gain some answers as to what they are doing about resilience at the location and why it isn't already in place.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - Loss of connectivity Leeds DC7

2009-01-18T18:11:00.003Z

At 18:10 we experienced a loss of connectivity, reported by monitoring to Leeds DC7. Network engineering are currently working on this issue and further updates will be posted as we discover the cause and work to a resolution.

***** Update - 18:37 *****

At present the cause of the fault is still unknown, but please rest assured we are working hard to pin point the cause.

***** Update - 19:19 *****

The cause of the fault has been identified and rectified. The reason for the outage is that we have suffered a complete carrier outage on one of out services and that failover to the newly installed third link (which was installed in September 2009) did not perform as expected. We believe there was a mis-configuration of the switching infrastructure which has never been tested in a full failure scenario as we have not suffered such an event and did not want to "cause" such a situation to test this.

We will be performing a full analysis of the cause and effect of the situation first thing in the morning.

After finally re-proving the second carriers' fibre pair, we are back in service with all three carriers. There has been something of a learning exercise gone on during this incident to the point of finding two carriers effectively have a pinch point on their network, by sharing adjacent buildings. We are looking at steps to diversify the traffic away from this estate and will be working on this in the coming weeks.

Apologies to all affected.
.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - Packet Loss in Leeds DC7

2009-01-06T13:00:00.007Z

We are aware at present that there is substantial packet loss on IP Transit circuits into our Leeds DC7 Datacentre. The cause has yet to be established but we are currently investigating the issue and will post more as we conclude our findings.

**** Update 13:15 ****

We have discovered we have lost a supervisor card in a router in Telehouse North which we are waiting to be rebooted.

**** Update 13:45 ****

After supervisor card restarts, we are seeing better routing, but still experiencing packet loss, some of which is being caused by less optimal routes for traffic, but there is still an underlying issue which we are still investigating.

**** Update 14:10 ****

With a second supervisor card re-start and some small routing changes we are back in full service. At present we have an outstanding issue with the way the secondary edge routers' didn't fully take over service and this is currently being raised to Cisco for consideration of the issue. Once we have formal response from them and remedial works can be planned, a full update will be provided to those affected by this outage.

*** Closed ***

Cisco have notified a number of config changes and also recommended an update to a later IOS on these pair of routers. Both these were put in place last night and testing carried out. We are happy the platform is stable and a fault of this nature cannot re-occur.

Once again apologies to all concerned, and hopefully it merely provided an excuse for a longer lunch than normal!

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

CCS London Datacentre Planned Maintenance 30/31st August 2008

2008-09-01T16:07:00.000+01:00

All.

As part of our on-going network expansion and improvement plan, we would like to advise you that a large element of our London network will be affected during the night of the 30/31st August 2008.

The work has become necessary to introduce additional capacity and resilience into our infrastructure and also to accommodate our expansion within Leeds into a new Datacentre due later this year.

Works will commence at 23:59 on Saturday (30th) night and will be completed by 8am Sunday (31st) morning. We expect service to be restored considerably earlier than 8am, but are allowing a good element of contingency in the works.

Services affected will include some ADSL / SDSL products, Bonded ADSL / SDSL products, Leased Lines, Web Hosting, DNS Services, Email, Anti-spam and Virus, MS SharePoint, MS Exchange, FTP Hosting, Voice over IP services and a number of other smaller products.

All affected customers have been advised via email and letters as appropriate, but if you have not received notification advising of the service affect, please do get in contact via the normal channels.

As works progress we will keep this post updated.

Apologies in advance for the inconvenience.

****** UPDATE ******

This work was completed without hitch two hours ahead of schedule. Thank you for you patience and understanding.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

CCS Leeds Cache DNS Servers

2008-07-15T20:10:00.003+01:00

All.

Due to an advisory notice from Cert entitled Multiple DNS implementations vulnerable to cache poisoning affecting our caching DNS servers it became important to patch all three in rather a hurry. This is for the protection of yourselves to ensure we do not issue invalid and spoofed DNS responses to any of our customers. Each DNS server will be down for around one minute while this is completed but we appreciate this should have no service affect as no one should be utilising a single server. However if an effect is seen on your networks, please do raise a support ticket at http://tickets.ccsleeds.co.uk/ or give us a call on the normal number as you may require assistance in re-configuration of your router(s), firewalls or web servers.

****** UPDATE ****** 20:18

All three cache servers have been patched and are back in service. No issues in the process were experienced and downtime was approximately 30 seconds per server. If there are any remaining issues, please do raise a ticket (URL above) or call us. We are always happy to help.

Apologies for the short notice, but network security is important to us all.
.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

CCS DC7 Datacentre IP Transit Issues

2008-07-11T09:30:00.008+01:00

All we have become aware of an issue with IP Transit in Leeds DC7. This is affecting (primarily) web hosting and co-lo customers but is having secondary effects on leased lines provisioned in the area and also secondary and tertiary mail services hosted there. As far as we can establish at this time, on of our upstream providers has de-announced our network from theirs meaning we are partially visible depending on which route you take to our site. We are currently in discussion with their technical department to gain a resolution.

****** UPDATE ****** 09:30

We are starting to see the successful re-announce of our AS across all providers and service is returning gradually as routing updates. I will keep you informed as work progresses. Thank you for your patience.

****** UPDATE ****** 15:00

While we were in an under monitor situation with DC7, we have started noticing events of packet loss again. This is only apparent from some inbound routes and is carrier and peering dependent again. Please rest assured we are back on the carriers' case and beginning the decision making process of whether to take this carrier out of the equation and disable their services until such time as they have rectified their issues. I will keep updates flowing as new information is gained. Once again, apologies to all those affected.

****** UPDATE ****** 12th July 15:40

All we have become aware that peering with one of our upstream carriers has failed again and this is affecting about 15% of UK users. We have the carrier on the phone as I type and we are looking for an immediate resolution otherwise we will be removing them as a carrier and de-peering to return service. Again apologies to those affected and rest assured we are working on the issue.

****** UPDATE ****** 17:15

We are continuing to experience packet loss on one leg of a single providers circuit, which has been raised to the highest level within the carrier and are promised we will see a resolution in the short term.

****** UPDATE ****** 21:20

All. Following extensive diagnosis with the carriers concerned, we have since learnt that the issue appears to be being caused by a sub feed we supply to a customer within the city. They have made some changes to their network configuration which has had a knock on effect on one of the routers at our transit providers site. Initially we thought it was a core switching issue in DC7 so we have replaced our core switch while investigating the issue. This has not cured the situation and we are now calling out one of the transit providers to get access to their pop site to replace the router we have at their location. Please rest assured we have all resources on the problem and a new router configured and ready to install. At the same time, we are disconnecting the sub-feed to this customer while they address their issues as they are not responding to our calls and we cannot get in contact with them. ETA for total clearance at the present time is around 11pm.

****** UPDATE ****** 11:54pm

Both our edge routers have been replaced and the service affecting customer isolated from the network. Service is restored packet loss free. As a knock on bonus our London to Leeds ping response has dropped from 8ms to 6ms. Our incident report will be distributed to those concerned on Monday as we need to clarify one issue with a carrier before presenting all the facts. Thank you for your patience and understanding in this matter.

****** CLOSED ****** 14th July 12:20

The service has been under monitor and close scrutiny since Saturday evening and we are confident of no further issues. Incident reports have been dispatched to those affected and I can only apologise once again for the issues.
FYI - The service availability measured London to Leeds Year to date 9th July was 99.997% of which we are proud. This has been a period I would prefer not to have to recall and rest assured we are taking further steps to prevent a recurrence.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - Major Service Outage - xDSL Platform

2008-06-12T20:45:00.000+01:00

All at 15:34 we have become aware of a xDSL service delivery failure. Other than seeing all the sessions clear and therefore service drop we do not know currently what has caused this and are working on the issue. Please bear with us and I will post more as I know something.

****** UPDATE ****** 15:54

We are currently experiencing an outage in Telehouse East that is affecting our xDSL platform and any permanent connections that terminate in this location. We are working to restore service as soon as possible. During this time we have an increased number of calls which is leading to increased wait times on the phones, we are endeavouring to answer all calls as quickly as possible. Apologies for the problems that this outage will cause.

****** UPDATE ****** 16:02

Telehouse have advised a piece of equipment failure in another occupants rack has caused a sub-main breaker to be taken out. They have engineering on site to isolate the other racks immediately and restore power to ours and those others affected. Service should be resumed in the next couple of minutes.

****** UPDATE ****** 16:10

Power has been restored and equipment is being brought back on sequentially as I update this post. Service will resume in just a few minutes.

****** UPDATE ****** 17:20

All our managed service customers have been returned to service. We can still see a number of stale sessions on the LNS's for other customers. If your service has not been restored as yet, please reboot your router, where it will log in and service will be restored.
We are still awaiting official confirmation as to what the exact cause of the issue was and will update this once we have further information.

****** UPDATE ****** 18:21

All services are now up and running, however due to a massive imbalance with our central pipes, it is necessary to rebalance them fairly aggressively. We will be doing this as soon as possible to ensure a reasonable balance is achieved for everyone.

****** UPDATE ****** 20:45

All Central Pipes are now rebalanced and traffic is flowing normally. May I once again apologise for the inconvenience experienced by all.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Planned Maintenance - xDSL Platform 06th June 2008

2008-05-29T19:01:00.001+01:00

Following the issues earlier this week with the crash of one of our core DSL termination routers, we have decided to bring forward the planned replacement program.

Therefore on the morning of the 6th June between the hours of 00:00 and 06:00 we will be replacing telehouse-east.dsl.ccsleeds.net which terminates users connected to Central Pipes 1 and 6. This work will impact users of our DSL network during the times specified above and may require us to rebalance the Central pipes following on from the work.

We are currently planning to minimise the effects of the work by moving the fibers in a controlled fashion and will look to ensure the minimum amount of downtime possible. This work will also involve the move of several LNS routers.

We apologise for the problems and inconvenience that this work will cause and wish to reassure you that we will do everything possible to complete the work in the speediest time with at little disruption as possible.

***** UPDATE ******

All works proceeded as planned with only momentary interruptions to service between 1 and 2am. All managed service customers were fully functioning again by 2.30am with a few others needing router re-boots in the morning where we have no out-of-band access to the devices and they had not come back of their own accord.
We apologise for the inconvenience, but have successfully re-provisioned a third of the DSL termination platform. I am pleased to advise as we proceed with the replacement of the rest of the routers, our newly installed router will handle all the traffic on its own and we will steer user sessions across to this before commencing with other replacement works, which should prevail as minimal (if any) downtime.

Thank you.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

At Risk - London Network

2008-05-27T19:00:00.001+01:00

All.

We have become aware in the past few minutes of a major routing issue on our London network.

This potentially affects web hosting, dns services and email services for those utilising a server which DOES NOT contain the letters DCx (x being a number) in the name.

Furthermore it is also affecting our leased line and other network service customers, again who DO NOT have DCx in their product title.

We are also aware of issues with cross-connects with other ISPs so those on xDSL platforms maybe experiencing service availability issues too.

We are working on the issues at this time and will post further as soon as we have information. At this stage we are unsure as to the cause, but suspect our cross-connects with Linx maybe related.

***** UPDATE ****** 19:00

We have confirmed the issue to be one with Linx switching between ISPs but they have not made a public statement to this effect as yet. Traffic is now stabilising and routing is re-converging but we are leaving the status as "At Risk" at this time until Linx confirm what the issue was and that it is resolved.

****** CLOSE ******

Please see the planned maintenance post as we have decided to bring forward our intended router replacement program to address this issue.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Resolved - Major Service Outage - xDSL Platform

2008-05-23T11:15:00.001+01:00

All.

We are aware of a major service outage on out xDSL platform, affecting ADSL, SDSL and Bonded SDSL customers at this time.

We are investigating and have all out engineers on the case, but have not yet established the cause of the failure. As soon as we have, resolutions will be put in place and I will post a service update to that effect.

Apologies to all concerned.

********** UPDATE ********* 10:07

We have discovered we have suffered from a router failure at our upstream provider which had caused stale sessions on the network once again. At this moment all traffic is being routed via a different switching house and all managed service customers are fully functional. For any remaining customers who still do not have service, please power off your router for thirty seconds and switch it back on. Then normal service should be resumed.
I will update the ticket once we know more.

********** UPDATE ********* 11:15

We have been advised a terminating router crashed and the dump files have been sent to Cisco for analysis. The router has been re-loaded and connections are now resilient across both sites again. Once we have an update from Cisco as to the cause, I will post further information.

********** UPDATE ********* CLOSED

Cisco have advised us to apply a none service affecting patch to the configuration which we completed last night to the router which had the service outage today. We will monitor this for a few days and if all is well apply it to the other terminating router late next week.

Once again, I can only apologise for the in-convenience and hope disruption was kept to a minimum for all.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

AT RISK Status - Telehouse North Power Failure

2008-05-09T19:25:00.001+01:00

Please be aware that we are currently in an AT RISK status. This is due to a Telehouse North core power failure where many providers' equipment was not sustained by the UPS's when the utility power failed and was re-booted by a power failure until the generators started.

While we are not directly affected, we are seeing a large number of customer routes flapping and experiencing general interconnect issues with Easynet, Entanet and the BBC to name just a few.

As situation updates occur, I will update the post.

Just to re-iterate, at the present time we are not experiencing any direct issues, but are seeing issues connecting to other providers.

********** UPDATE ********** 19:25

Telehouse have advised that the problem is only affecting those occupying Telehouse North. The issue was a UPS control module failure and all customers in North are running on dirty mains power, with generator backup, but no seamless failover. If utility power fails, the generators will start, but there will be an interruption in service for a number of minutes as all equipment will have lost power and have to reboot. They are unable to advice when the control board will be replaced, but Sunday or Monday are being touted as likely fixes, at which point there will be a further interruption while the UPS's are switched back into service.

Again I can re-iterate this is not affecting our service delivery at this stage, but many other ISPs are still showing signs of outages and much traffic is routing via Europe rather than directly in London.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

CCS DC7 Datacentre IP Transit Issues

2008-04-17T23:45:00.001+01:00

Morning,

We have become aware of routing and transit issues with our primary carrier, Kingston Communications displaying itself as intermittent routing from some networks to DC7 Datacentre in Leeds.

Rest assured we are on the case and chasing Kingston hard for an update.

Apologies to all concerned and I will post an update as soon as I have something to report.

***************** UPDATE ****************** - 08:00

Kingston have reported a core router failure which is poisioning the routes they are advertising. An engineer is on site now and working on the issue at both DC7 and Kingstons premises.

***************** UPDATE ****************** - 09:00

We have disabled BGP announcements across the Kingston network and routing is performing as expected, but we are still somewhat affected when connecting from some ISPs as the routing tables are not fully replicated as yet.

***************** UPDATE ****************** - 10:00

Kingston have resolved their routing issues, but we are leaving our primary feeds direct from London and via Thus until this evening to avoid further routing flips when we bring Kingston back on line.

***************** UPDATE ****************** - 23:00

We are now re-announcing via Kingston and all seems to be well. They have been stable all day and while we are not happy with the explanation they have given for the cause, we feel to ensure adequate resilience we need to re-announce on their network. However we have de-prioritised them and will leave them lower priority than our direct London connections picking up Peering, Tiscali International and NTT to ensure no future recurrence.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Critical Outage on xDSL Platform

2008-04-10T09:58:00.005+01:00

All.

We are aware many of our ADSL / SDSL and Bonded customers currently are without service. BT have experienced a major failure somewhere in the network, but we are unable to reach them and are sitting in a call queue awaiting an answer.

As soon as we have an update, we will post to the effect. Please bear with us, as we cannot currently provide any further news on the situation until BT answer out calls.

************* Update ********************** (10:15)

The initial cause of the fault was found to be a DSL Aggregate router crash which we will continue to investigate once we have restored customer service to all. There are a number of stale sessions on the BT network and if you are un-able to re-authenticate, please could you detach your router from the telephone line(s) and leave for 15 minutes for the sessions to clear before re-connecting.

Please bear with us, we are working hard on the issues and appreciate we have call queues at this present time.

***************** Update ******************* (11:48)

As all customer connections are now restored, we are working on investigating the router failure that caused the outage. Updates will be posted as further information is available.

***************** Update *******************

The issue is now resolved. Cisco have found an IOS issue, and while it is an un-corrected bug, it is a service that is not necessary to run on this router so we have disabled it.

Apologies to all concerned.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

xDSL Platform Stability

2008-03-18T16:00:00.000Z

At 13:42 we became aware of platform instability on our core xDSL termination router. Service is unavailable to a number of customers at this time.

This is affecting our ADSL, SDSL and MLPPP Bonded xDSL services.

We are investigating the cause.

**** Update ****

14:07 - We have discovered the cause of the issue which was our ADSL realms had been removed from the BT Central Pipes. They have been returned and all managed and monitored customers are now logged back in. We await the phone call from the BT account manager to have strong words.

**** Update 2 ****

BT have advised this was a mistake during "clean up" works on their Radius platform and we have a formal apology which I wholeheartedly pass on to our affected customers.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Increased Capacity on the xDSL Platform

2008-02-14T18:19:00.003Z

CCS are pleased to announce BT have enabled an additional 155 megabits of capacity on our centrals benefitting our ADSL, SDSL and MLPPP Bonded xDSL customers and should help to alleviate the slow downs of an evening and especially on Sundays.

Happy Days!

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Free Install and Setup on Business SDSL Packages

2007-12-31T23:59:00.000Z

Just in time for Christmas, we are pleased to announce FREE Setup on selected SDSL packages.

There is no catch, no fiddle, just a good quality 2Mb/s SDSL Service at standard IpStream 10:1 contention ratios with FREE Setup for the month of December. Static / Fixed IP addresses are included on all connections and blocks of IP addresses are available subject to normal RIPE justification.

They are available for standard Business SDSL use or for MLPPP Bonded SDSL in multiples.

The offer does not include the router but hey, you are getting a £350 discount on a standard 12 month contact and we are not a charity!

Full details can be found on out Christmas promotions page HERE > Christmas Promo

If you are interested, get in their fast as this is limited to 100 connections during December.

Give us a call on Tel: 0113 294 66 99 and register your interest before we run out of promotional connections!

Just a quick note to say we have 30 places still remaining on the FREE SETUP on SDSL remaining, so call in quick or fill in the webform on the site to register your interest.

http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Scheduled Maintenance, Software Updates and Patching

2007-12-30T22:00:00.000Z

On the 30th December 2007 we will performing routine maintenance on the following platforms:

1. Reporting service for Broadband covering Home ADSL, Business ADSL , SDSL, Bonded xDSL and Managed Broadband Services. This will not affect the services themselves, just the reporting capability as we upgrade the management software.

2. Web Hosting on all servers, including Discount Web Hosting (1-6), Company Web Hosting (1-6), Reseller Hosting and Domain Name Registration. This will not affect Co-location Hosting or Dedicated Server customers.

3. Reporting service for Leased Line customers. This will not affect service to the circuit(s), just the reporting where a brief outage will be shown upon return to service.

4. Reporting service for Managed Firewall customers. This will not affect service, merely the reporting and monitoring service. A brief outage will be showing in the reporting but service will not be affected.

5. Email Hosting, Anti-Spam, Webmail, Mobile Email and Exchange Hosting. Each service will experience an brief outage as patches are applied and the servers re-booted. No incoming email will be lost during this interval as it will queue on our Backup MX Servers.

6. Hosted Sharepoint, On-line File Hosting, FTP Web Hosting and Managed Small Business Servers. Each service will be unavailable for a few minutes while patches are applied. Again no email will be lost as it will be queued on the Backup MX Mail servers, but access will be lost briefly.

7. Access to our On-line Ticket System, Knowledgebase and Login to your Account portal. There will be a brief outage to each of these services for patching and server re-boots.

Before this extensive maintenance schedule takes place, we will run an additional snapshot backup to the on-site NAS server and replicate this to the off-site NAS in case of any issues.

The work will take place between the hours of 6pm and midnight on the 30th December but we expect to be completed within an hour if all goes well.

During this period we will have staff on-site to ensure as quick and smooth patching and re-boot however as we are committing all the staff resource to this operation to ensure it is as speedy as possible we will have limited telephone cover.
We would appreciate it if you can avoid calling during this period enabling as swift a process as possible.
Our on-line ticket system will be down for as short a period as is humanly possible and we would appreciate issues be raised this way, rather than by telephone if at all possible.

**********************Update*****************************

All work is now complete. Unfortunately we had a problem when the MS SQL cluster came back up regarding the quorum partitions vanishing but one that was resolved, all was completed painlessly. All services, bar SQL were returned to service within an hour, i.e. by 7pm. The cluster took around another hour to resolve the issues.

If anyone has any outstanding issues, please feel free to call or raise a ticket in the normal way.

Many thanks for your patience.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Downtime for Helm 4 Control Panel

2007-12-19T18:28:00.000Z

All.

Apologies for the late notification, but due to a security breach in MS FrontPage extensions we have had to take the Helm 4 Control server off line for a few minutes to perform an update.

This affected both the Discount Web Hosting / AKA Home and also the Shared Company Website Hosting packages' Helm 4 Control Panel only and not the actual Web Service.

We have patched the breach and cleaned up the defacation a kind team left on one of the servers from the snapshot backups and full service is restored.

One again apologies for the issue.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

ADSL / SDSL and MLPPP Bonded DSL Service Instability

2007-11-06T19:35:00.000Z

Due to wholesale carrier issues regarding a further damaged fibre our Residential DSL, Business Broadband, SDSL Services and MLPPP Bonded DSL are currently suffering instability and very short term outages.

Customers may be experiencing momentary disconnects of the services while fibre repair works are in progress and we are closely monitoring all our Managed Service customers closely and taking immediate remedial action as required.

Please be assured we are keeping the utmost pressure on our fibre providers to quickly and seamlessly resolve this issue to return to normal service as quickly as it possible.

* Update - We had had the all clear from the fibre providers who handily correctly labelled the fibres during testing to hopefully prevent further occurrences of such stupidity *

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Network Backbone - Reduced Resilience

2007-11-02T07:58:00.000Z

We have been advised by one of our wholesale providers that they have experienced fibre damage between London Telecity Harbour Exchange and Telehouse Metro.

While this is not service affecting as the traffic has re-routed the other way around the ring, it does mean we have a single point of failure as the ring is no longer a ring. Our fibre provider is working to a resolution as we speak.

*Update* - We have full service resumed with a diverse ring in place again and we have tested this by simulating a fault in the other segment and traffic flipped seamlessly. Good news for all.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

xDSL Issues in Bristol and Bedford

2007-10-08T12:09:00.000+01:00

For customers on our Bristol and Bedford LLU exchanges, we are aware on intermittent connectivity issues. We are working with out WAN supplier to isolate the cause and will update this post as soon as we have further information.

We are sorry for any inconvenience caused.

*Update* - 18:38

We have been advised by our carriers that their switching ring issues that caused these flaps has now been resolved. Once again apologies for any inconvenience.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Routing Flips at Linx.net

2007-10-04T09:03:00.000+01:00

At 09:03, 4th October 2007 we experienced a large drop in traffic travelling to partner ISP's via Linx. All the traffic has re-routed via other carriers and there was around a 30 second glitch in traffic during this re-route.

We are investigating the cause and will provide more details once we have established the fault, but traffic is currently travelling as normal via alternate carriers.

*Update*

LINX have had an issue with Spanning Tree on their two fibre rings for the past 2 hours. We have just been informed that they have shut down the automatic RSTP and are awaiting a response from hardware vendors.We will keep you informed when they implement a permanent fix.

*Update*

LINX have advised they have restored ring stability by implementing a long term work around and we have seen stable performance for the past few days indicating this is a success. They are working on a longer term plan with the switching manufacturers to develop an entirely new platform.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

DSL Speed Issues in Scotland and Northern Ireland

2007-10-01T19:13:00.000+01:00

Following an increased number of calls and email tickets in to support we can confirm that their are a number of users within parts of Scotland and Northern Ireland that may be experiencing speed issues. We have spoken to BT who have confirmed that there are transmission (cabling) faults between Edinburgh-Birmingham and Carlisle-Glasgow which are effecting these areas.
These faults have resulted in BT rerouting traffic over alternative sections of the network.

*Update*

We have been advised by BT that their network issues have been resolved and traffic is routing normally. Speed and performance have returned to expected levels.

.
http://www.ccsleeds.co.uk/

Add to Technorati Favorites

Network Maintenance - 02/10/07, 01:00 to 04:00

2007-09-28T21:16:00.000+01:00

We have received the following service notification from NTT:

Maintenance Window: From 02/10/2007 01:00 to 02/10/2007 04:00

Device: r01.londen03.uk.bb.gin.ntt.net

Customer Impact: We are directly connected to this router. To ensure things go as smoothly as possible we will be shutting our NTT sessions down overnight while the work is completed.

Traffic will re-route over other providers but some loss of performance may be evident during this period.

We apologise for any inconvenience caused in advance.

*Update*

This work was completeed successfully and having a look at our reporting graphs little if any user performance was evident and we didn't drop a single ping!

.
http://www.ccsleeds.co.uk/

New Bandwidth brought online

2007-09-28T11:42:00.000+01:00

Our DSL provider has advised that BT have brought an additional 155Mb/s on line on our tails.

This is excellent news for all.

.
http://www.ccsleeds.co.uk/

Internet Transit Flip

2007-09-28T03:40:00.000+01:00

At 00:34 on the 28/09 one of our upstream transit providers had a LAN switch fail. Traffic re-routed via another provider causing a 30 second or so outage while all the routing tables updated.

We apologise for any inconvenience caused.

* Update*

The faulty switch has now been replaced and traffic is routing as normal. There was no further outage as the new switch came back on line.
.
http://www.ccsleeds.co.uk/

Network stability September 7th, 2007 1:00am - 6:30am

2007-09-07T01:05:00.000+01:00

We are aware of problems with international connectivity last night, this looks the be down to a specific router fault. Engineers are currently investigating the cause.

*Update*

This has now been resolved. One of our international carriers had a fibre failure which has now been resolved.

Sorry for any inconvenience caused.
.

http://www.ccsleeds.co.uk/

EMERGENCY Software Update

2007-09-06T11:00:00.000+01:00

We are aware of problems with international connectivity last night, this looks the be down to a specific router fault. Engineers are currently investigating the cause.

*Update*

Due to last nights router problems we will be upgrading the software on core router 2. This will involve approx 10 mins down time while the router reboots. All customers connected directly to CR2 will lose service for the duration and there may be a period of routing instability while the routing protocols reconverge for all users.

This work will start at 11:00am on 7th Sept 2007.

We apologise for any inconvenience caused.

*Update2*

All works successfully completed and stable service has returned for all.

We apologise for the inconvenience during these works.

.

http://www.ccsleeds.co.uk/

VoIP service problems

2007-08-20T09:32:00.000+01:00

We are currently experiencing problems with our VoIP platform which is preventing inbound and outbound calls for anyone utilising the platform.

Our engineers are working to correct this and will resolve the issue as soon as possible.

* Update *

The issue has been resolved, our upstream provider had call handling issues which they have now resolved.

.

http://www.ccsleeds.co.uk/

Problem in the Lancashire and Cheshire area

2007-07-09T11:13:00.000+01:00

BT have informed us that there is an issue with one of the Manchester gateways which is affecting customers of all ISP's throughout the Lancashire and Cheshire area (mainly 0161 numbers). We are aware of this issue and will update this post once we hear further from BT.
.

http://www.ccsleeds.co.uk/

Severe Weather Conditions and BT Faults

2007-06-27T18:01:00.000+01:00

Due to the exceptionally severe weather BT have been experiencing in some parts of the country (Leeds, Bradford, Lincoln, Sheffield, Barnsley and York), BT Wholesale has declared this is as a matter beyond their reasonable control (’MBORC’) in these areas.

Flooding is causing network damage, leading to very high fault volumes. At the same time engineers are faced with road closures and other problems getting around their work areas. This means it is taking longer than usual to fix each fault.

This notification is with effect from 17.30 Wednesday 27 June. Other locations may be subject to declarations as a fuller picture of the effects on our operations and network emerges.

We currently expect the declarations to last for approximately 7 days from 27 June. However further rain is forecast for the affected areas, especially over the coming weekend, and this may delay the return to normal. We will keep the situation under review and provide updates as the situation develops.

.
http://www.ccsleeds.co.uk/

EMERGENCY Maintenance - ADSL Services

2007-05-23T20:15:00.000+01:00

Following on from today’s problems with L2TP sessions, it is necessary to perform an upgrade to the IOS software of our LTS pool. The reloads are scheduled for 03.00hrs and 04.00hrs on 24th May 2007 and are expected to take no longer than 25 minutes each.

Multiple reloads have been scheduled to reduce the Radius load on our customer’s servers. We apologise for any inconvenience this may cause.

We will ensure that all managed circuits are back in service after the all clear is given from the work.

*Update*

All work performed successfully and all customers reconnected without obvious issues.

Thank you for your patience in this matter.

.
http://www.ccsleeds.co.uk/