DC3HAM Network Outage

Lumen finally succeeded around 01:30 am on Saturday to reconnect their datacenter in Hamburg which hosts our colocation DC3.HAM.
We have been checkng and verifying all systems afterwards.
Systems including our ticket system are back and available.
We will follow up with Lumen on an incident report.

——————————————————————————————-

Colocation DC3 seems to be back online, we are checking related systems from MCON side

——————————————————————————————-

*** CASCADED EXTERNAL NOTES 2022-03-12 00:05:37 GMT From CASE: 23317190 – SM Parent
***************************************************
[CUSTOMER UPDATE] EMEA Service Desk ()

[SUMMARY OF WORK]
Good Morning

We are now seeing the services restored, we have asked the vendor to provide a full RFO.

We will keep you updated with all progress.

Kind Regards
Lumen

[PLAN OF ACTION]
Investigating RFO

[TIME – NOW] 2022-03-12 00:05 (UTC)
***************************************************

UPDATE 2022-03-11 23:56 AM CET

***************************************************

[CUSTOMER UPDATE] EMEA Service Desk (ScLo)

[SUMMARY OF WORK]

Good Afternoon

Colt partner has confirmed the completion of splicing however Colt customers services are still down. Partner has been requested to recheck the splicing. We will keep you updated on our progress via this email address. Thank you

We will continue to push for an ETR.

[PLAN OF ACTION]

[TIME – NOW] 2022-03-11 20:58 (UTC)

[UPDATE ETA]

***************************************************


UPDATE 2022-03-11 19:06 AM CET

***************************************************

[CUSTOMER UPDATE] EMEA Service Desk (ScLo)

[SUMMARY OF WORK]

Good Afternoon

Please be advised the local provider last mile field engineers continue in repair efforts for fix.

We continue to push for ETR.

[PLAN OF ACTION]

[TIME – NOW] 2022-03-11 17:58 (UTC)

[UPDATE ETA] 2022-03-11 19:15 (UTC)

***************************************************


UPDATE 2022-03-11 18:10 AM CET

***************************************************

[CUSTOMER UPDATE] EMEA Service Desk (ScLo)

[SUMMARY OF WORK]

Good Afternoon

Please be advised we are pushing for an ETR from the local provider. They have stated cable repair preperation is ongoing and will update further.

[PLAN OF ACTION]

[TIME – NOW] 2022-03-11 16:40 (UTC)

[UPDATE ETA] 2022-03-11 17:40 (UTC)

***************************************************


UPDATE 2022-03-11 15:59 AM CET

***************************************************

[CUSTOMER UPDATE] EMEA Service Desk (ScLo)

[SUMMARY OF WORK]

Good Afternoon

Please be advised our local providers last mile partner confirmed that situation is complex as damage location is occupied with heavy construction machinery which need to be cleared for digging work. Civil work is ongoing and expected time of restoration is awaited.

[PLAN OF ACTION]

We will follow up with further update in next 2 hours.

[TIME – NOW] 2022-03-11 14:42 (UTC)

[UPDATE ETA] 2022-03-11 16:42 (UTC)

***************************************************

Next update by: 2022-03-11 16:45 GMT


UPDATE 2022-03-11 14:54 AM CET

***************************************************

[CUSTOMER UPDATE] EMEA Service Desk (ScLo)

[SUMMARY OF WORK]

Good Afternoon

Please be advised that we continue to work with the local provider for updates and progress in relation to this case, we have requested an urgent update and confirmation of when service will be restored, as original ETR provided has now passed, we will aim to provide a further update in the next 60 minutes

[PLAN OF ACTION]

await local provider update and update customer once feedbacj received

[TIME – NOW] 2022-03-11 13:43 (UTC)

[UPDATE ETA] 2022-03-11 14:43 (UTC)

***************************************************

Next update by: 2022-03-11 14:45 GMT


UPDATE 2022-03-11 14:20 AM CET

***************************************************

[CUSTOMER UPDATE] EMEA Service Desk (ScLo)

[SUMMARY OF WORK]

Good Afternoon

Please be advised that we can confirm engineers continue to work to restore service, as advised previously, we have been given an ETR of 13:00 GMT, and this still stands at this time , however damage to fibre was extensive so this maybe pushed back ,we will aim to provide a further update in the next 60 minutes

[PLAN OF ACTION]

await local provider update and update customer once feedback received

[TIME – NOW] 2022-03-11 12:20 (UTC)

[UPDATE ETA] 2022-03-11 13:20 (UTC)

***************************************************

Next update by: 2022-03-11 13:20 GMT


UPDATE 2022-03-11 08:18AM CET

***************************************************

[CUSTOMER UPDATE] EMEA Service Desk (ScLo)

[SUMMARY OF WORK]

Good Morning

Please be advised that we can confirm that engineers are onsite and working to restore service, the fibre break is located at Wuppertal City Germany, the local provider has confirmed that the ETR for completion of the work is 13:00 GMT

We will aim to provide a further update around 12:00-12:30 GMT to confirm if we are still on target for the ETR, once we have this confirmation, we will forward this over to you

[PLAN OF ACTION]

chase local provider around 12:30 to confirm we are still on target for ETR provided, once confirmed update customer

[TIME – NOW] 2022-03-11 09:42 (UTC)

***************************************************

Next update by: 2022-03-11 12:15 GMT


UPDATE 2022-03-11 08:18AM CET

***************************************************

[CUSTOMER UPDATE] EMEA Service Desk ()

[SUMMARY OF WORK]

Good Morning

We are still awaiting testing from the vendor. We will keep you updated in all progress.

Kind Regards

Lumen

[PLAN OF ACTION]

Investigating

[TIME – NOW] 2022-03-11 07:15 (UTC)

***************************************************


UPDATE 2022-03-11 07:19AM CET

***************************************************

[CUSTOMER UPDATE] EMEA Service Desk (AsHa)

[SUMMARY OF WORK]

Good Afternoon,

Field engineers are actively repairing the fault and we shall update you as soon as information is available.

Kind Regards,

Lumen

[PLAN OF ACTION]

Engage Local Carrier

[TIME – NOW] 2022-03-11 06:17 (UTC)

***************************************************


UPDATE 2022-03-11 03:37AM CET

***************************************************

[CUSTOMER UPDATE] EMEA Service Desk (AsHa)

[SUMMARY OF WORK]

Good Afternoon,

Our local carrier have informed us there engineers are expected to arrive at the affected location at 04:30 GMT. We will update you at this time of there findings.

Kind Regards

Lumen

[PLAN OF ACTION]

Engage Local Carrier

[TIME – NOW] 2022-03-11 02:23 (UTC)

***************************************************


UPDATE 2022-03-11 02:39AM CET


[CUSTOMER UPDATE] EMEA Service Desk (AsHa)

[SUMMARY OF WORK]

Good Afternoon,

There is an issue in our partners network on a link between Dortmund and Dusseldorf, Germany.We will inform you accordingly of any information as it becomes available.

Kind Regards

Lumen

[PLAN OF ACTION]

Engage Local Carrier

[TIME – NOW] 2022-03-11 01:36 (UTC)

***************************************************

Next update by: 2022-03-11 02:40 GMT


UPDATE 2022-03-11 02:22AM CET
no update from support of datacenter carrier LUMEN
escalation level has been raised


UPDATE 2022-03-11 00:56AM CET
ticket has been raised at support of datacenter carrier LUMEN

***************************************************

[CUSTOMER UPDATE] EMEA Service Desk (AsHa)

[SUMMARY OF WORK]

Good Afternoon,

Your services are affected by a major outage in our local carrier’s network. We are engaging them and will updat eyou accordingly.

Kind Regards,

Lumen

[PLAN OF ACTION]

Engage Local Carrier

[TIME – NOW] 2022-03-10 23:57 (UTC)

**********************************************

Next update by: 2022-03-11 01:00 GMT


UPDATE 2022-03-10 11:48PM CET
we just identified that uplinks of or datacenter carrier LUMEN are down right now
this being said we are currently tracking with their support to find a quick solution


UPDATE 2022-03-10 11:00PM CET
network is currently not working as expected
thus sites and services are not available right now
we are working with high pressure to resolve this issue as fast as possible

Patch Day March 2022 – System Upgrades MCON Managed Services

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, March 10th 2022 : development starting at 08 am and QA starting at 12 noon*
  • Friday, March 11th 2022 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, March 15th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, March 16th 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, March 17th 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, March 18th 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service or when the operating system restarts.

Best regards

Your MCON Managed Services Team

The times are valid for the respective timezone

Status Patch Day February 2022 – System Upgrades MCON Managed Services

Dear Sir or Madam,

according to our announcement below the patches for Europe Region and America Region and APAC Region done.

All systems are up and running. However if you encounter any problem please contact service@mcon-group.support.

Best regards
Your MCON Managed Services Team

Announcement from 31st January 2022:
Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, February 10th 2022 : development starting at 08 am and QA starting at 12 noon*
  • Friday, February 11th 2022 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, February 15th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, February 16th 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, February 17th 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, February 18th 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service or when the operating system restarts.

Best regards

Your MCON Managed Services Team

The times are valid for the respective timezone

Patch Day February 2022 – System Upgrades MCON Managed Services

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, February 10th 2022 : development starting at 08 am and QA starting at 12 noon*
  • Friday, February 11th 2022 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, February 15th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, February 16th 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, February 17th 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, February 18th 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service or when the operating system restarts.

Best regards

Your MCON Managed Services Team

The times are valid for the respective timezone

Status Patch Day January 2022 – System Upgrades MCON Managed Services

Dear Sir or Madam,

according to our announcement below the patches for Europe Region and America Region and APAC Region were successfully done.

All systems are up and running. However if you encounter any problem please contact service@mcon-group.support.

Best regards
Your MCON Managed Services Team

Announcement from 4th January 2022:
Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, January 13th 2022 : development starting at 08 am and QA starting at 12 noon*
  • Friday, January 14th 2022 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, January 18th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, January 19th 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, January 20th 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, January 21st 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service or when the operating system restarts.

Best regards

Your MCON Managed Services Team

The times are valid for the respective timezone

Patch Day January 2022 – System Upgrades MCON Managed Services

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, January 13th 2022 : development starting at 08 am and QA starting at 12 noon*
  • Friday, January 14th 2022 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, January 18th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, January 19th 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, January 20th 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, January 21st 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service or when the operating system restarts.

Best regards

Your MCON Managed Services Team

The times are valid for the respective timezone

Status Patch Day December 2021 – System Upgrades MCON Managed Services

Dear Sir or Madam,

according to our announcement below the patches for Europe Region and Region APAC and Region America were successfully done.

All systems are up and running. However if you encounter any problem please contact service@mcon-group.support.

Best regards
Your MCON Managed Services Team

Announcement from 8th December2021:
Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, December 16th 2021 : development starting at 08 am and QA starting at 12 noon*
  • Friday, December 17th 2021 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, December 21st 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, December 22nd 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Monday, December 20th 2021, development starting at 08 am and QA starting at 12 noon*
  • Tuesday, December 21th 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service or when the operating system restarts.

Best regards

Your MCON Managed Services Team

The times are valid for the respective timezone

CVE-2021-44228

we are working to defeat https://nvd.nist.gov/vuln/detail/CVE-2021-44228

update monday 13th Dec 2021
vulnerable systems have been mitigated during weekend.
today (13th Dec) we are informing project stakeholders also of projects are not found vulnerable
if you feel you are lacking information from our side please create a ticket by mail to service@mcon-group.support

Patch Day December 2021 – System Upgrades MCON Managed Services

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, December 16th 2021 : development starting at 08 am and QA starting at 12 noon*
  • Friday, December 17th 2021 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, December 21st 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, December 22nd 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Monday, December 20th 2021, development starting at 08 am and QA starting at 12 noon*
  • Tuesday, December 21th 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service or when the operating system restarts.

Best regards

Your MCON Managed Services Team

The times are valid for the respective timezone