Lumen finally succeeded around 01:30 am on Saturday to reconnect their datacenter in Hamburg which hosts our colocation DC3.HAM.
We have been checkng and verifying all systems afterwards.
Systems including our ticket system are back and available.
We will follow up with Lumen on an incident report.
——————————————————————————————-
Colocation DC3 seems to be back online, we are checking related systems from MCON side
——————————————————————————————-
*** CASCADED EXTERNAL NOTES 2022-03-12 00:05:37 GMT From CASE: 23317190 – SM Parent
***************************************************
[CUSTOMER UPDATE] EMEA Service Desk ()
[SUMMARY OF WORK]
Good Morning
We are now seeing the services restored, we have asked the vendor to provide a full RFO.
We will keep you updated with all progress.
Kind Regards
Lumen
[PLAN OF ACTION]
Investigating RFO
[TIME – NOW] 2022-03-12 00:05 (UTC)
***************************************************
UPDATE 2022-03-11 23:56 AM CET
***************************************************
[CUSTOMER UPDATE] EMEA Service Desk (ScLo)[SUMMARY OF WORK]
Good AfternoonColt partner has confirmed the completion of splicing however Colt customers services are still down. Partner has been requested to recheck the splicing. We will keep you updated on our progress via this email address. Thank you
We will continue to push for an ETR.
[PLAN OF ACTION]
[TIME – NOW] 2022-03-11 20:58 (UTC)
[UPDATE ETA]
***************************************************
UPDATE 2022-03-11 19:06 AM CET
***************************************************
[CUSTOMER UPDATE] EMEA Service Desk (ScLo)[SUMMARY OF WORK]
Good AfternoonPlease be advised the local provider last mile field engineers continue in repair efforts for fix.
We continue to push for ETR.
[PLAN OF ACTION][TIME – NOW] 2022-03-11 17:58 (UTC)
[UPDATE ETA] 2022-03-11 19:15 (UTC)
***************************************************
UPDATE 2022-03-11 18:10 AM CET
***************************************************
[CUSTOMER UPDATE] EMEA Service Desk (ScLo)[SUMMARY OF WORK]
Good AfternoonPlease be advised we are pushing for an ETR from the local provider. They have stated cable repair preperation is ongoing and will update further.
[PLAN OF ACTION]
[TIME – NOW] 2022-03-11 16:40 (UTC)
[UPDATE ETA] 2022-03-11 17:40 (UTC)
***************************************************
UPDATE 2022-03-11 15:59 AM CET
***************************************************
[CUSTOMER UPDATE] EMEA Service Desk (ScLo)[SUMMARY OF WORK]
Good AfternoonPlease be advised our local providers last mile partner confirmed that situation is complex as damage location is occupied with heavy construction machinery which need to be cleared for digging work. Civil work is ongoing and expected time of restoration is awaited.
[PLAN OF ACTION]
We will follow up with further update in next 2 hours.[TIME – NOW] 2022-03-11 14:42 (UTC)
[UPDATE ETA] 2022-03-11 16:42 (UTC)
***************************************************Next update by: 2022-03-11 16:45 GMT
UPDATE 2022-03-11 14:54 AM CET
***************************************************
[CUSTOMER UPDATE] EMEA Service Desk (ScLo)[SUMMARY OF WORK]
Good AfternoonPlease be advised that we continue to work with the local provider for updates and progress in relation to this case, we have requested an urgent update and confirmation of when service will be restored, as original ETR provided has now passed, we will aim to provide a further update in the next 60 minutes
[PLAN OF ACTION]
await local provider update and update customer once feedbacj received[TIME – NOW] 2022-03-11 13:43 (UTC)
[UPDATE ETA] 2022-03-11 14:43 (UTC)
***************************************************Next update by: 2022-03-11 14:45 GMT
UPDATE 2022-03-11 14:20 AM CET
***************************************************
[CUSTOMER UPDATE] EMEA Service Desk (ScLo)[SUMMARY OF WORK]
Good AfternoonPlease be advised that we can confirm engineers continue to work to restore service, as advised previously, we have been given an ETR of 13:00 GMT, and this still stands at this time , however damage to fibre was extensive so this maybe pushed back ,we will aim to provide a further update in the next 60 minutes
[PLAN OF ACTION]
await local provider update and update customer once feedback received[TIME – NOW] 2022-03-11 12:20 (UTC)
[UPDATE ETA] 2022-03-11 13:20 (UTC)
***************************************************Next update by: 2022-03-11 13:20 GMT
UPDATE 2022-03-11 08:18AM CET
***************************************************
[CUSTOMER UPDATE] EMEA Service Desk (ScLo)[SUMMARY OF WORK]
Good MorningPlease be advised that we can confirm that engineers are onsite and working to restore service, the fibre break is located at Wuppertal City Germany, the local provider has confirmed that the ETR for completion of the work is 13:00 GMT
We will aim to provide a further update around 12:00-12:30 GMT to confirm if we are still on target for the ETR, once we have this confirmation, we will forward this over to you
[PLAN OF ACTION]
chase local provider around 12:30 to confirm we are still on target for ETR provided, once confirmed update customer[TIME – NOW] 2022-03-11 09:42 (UTC)
***************************************************Next update by: 2022-03-11 12:15 GMT
UPDATE 2022-03-11 08:18AM CET
***************************************************
[CUSTOMER UPDATE] EMEA Service Desk () [SUMMARY OF WORK]
Good MorningWe are still awaiting testing from the vendor. We will keep you updated in all progress.
Kind Regards
Lumen[PLAN OF ACTION]
Investigating[TIME – NOW] 2022-03-11 07:15 (UTC)
***************************************************
UPDATE 2022-03-11 07:19AM CET
***************************************************
[CUSTOMER UPDATE] EMEA Service Desk (AsHa)[SUMMARY OF WORK]
Good Afternoon,Field engineers are actively repairing the fault and we shall update you as soon as information is available.
Kind Regards,
Lumen
[PLAN OF ACTION]
Engage Local Carrier[TIME – NOW] 2022-03-11 06:17 (UTC)
***************************************************
UPDATE 2022-03-11 03:37AM CET
***************************************************
[CUSTOMER UPDATE] EMEA Service Desk (AsHa)[SUMMARY OF WORK]
Good Afternoon,Our local carrier have informed us there engineers are expected to arrive at the affected location at 04:30 GMT. We will update you at this time of there findings.
Kind Regards
Lumen
[PLAN OF ACTION]
Engage Local Carrier[TIME – NOW] 2022-03-11 02:23 (UTC)
***************************************************
UPDATE 2022-03-11 02:39AM CET
[CUSTOMER UPDATE] EMEA Service Desk (AsHa)[SUMMARY OF WORK]
Good Afternoon,There is an issue in our partners network on a link between Dortmund and Dusseldorf, Germany.We will inform you accordingly of any information as it becomes available.
Kind Regards
Lumen
[PLAN OF ACTION]
Engage Local Carrier[TIME – NOW] 2022-03-11 01:36 (UTC)
***************************************************Next update by: 2022-03-11 02:40 GMT
UPDATE 2022-03-11 02:22AM CET
no update from support of datacenter carrier LUMEN
escalation level has been raised
UPDATE 2022-03-11 00:56AM CET
ticket has been raised at support of datacenter carrier LUMEN
***************************************************
[CUSTOMER UPDATE] EMEA Service Desk (AsHa)
[SUMMARY OF WORK]
Good Afternoon,
Your services are affected by a major outage in our local carrier’s network. We are engaging them and will updat eyou accordingly.
Kind Regards,
Lumen
[PLAN OF ACTION]
Engage Local Carrier
[TIME – NOW] 2022-03-10 23:57 (UTC)
**********************************************
Next update by: 2022-03-11 01:00 GMT
UPDATE 2022-03-10 11:48PM CET
we just identified that uplinks of or datacenter carrier LUMEN are down right now
this being said we are currently tracking with their support to find a quick solution
UPDATE 2022-03-10 11:00PM CET
network is currently not working as expected
thus sites and services are not available right now
we are working with high pressure to resolve this issue as fast as possible