Status Patch Day November 2021 – System Upgrades MCON Managed Services

Dear Sir or Madam,

according to our announcement below the patches for Region Europe were successfully done.

All systems are up and running. However if you encounter any problem please contact service@mcon-group.support.

Best regards
Your MCON Managed Services Team

Announcement from 4th November 2021:

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, November 11th 2021 : development starting at 08 am and QA starting at 12 noon*
  • Friday, November 12th 2021 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, November 16th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, November 17th 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, November 18th 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, November 19th 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service or when the operating system restarts.

Best regards

Your MCON Managed Services Team

The times are valid for the respective timezone

Resolved: Outage DC3HAM

Dear Customer,

a trivial change on our external firewall with a (normal) subsequent sync to the secondary device caused both firewalls to go into “disabled”
state and not forward any packets anymore.

An deactivation and activation of the High Availability solved the problem.

The connection interruption lasted from 11:08 – 11:41 CEST, we apologize for any inconvenience.

We are in contact with the developers to find the rootcause of this issue.

We apologize for any inconvenience.

Patch Day September 2021 – System Upgrades MCON Managed Services

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, September 16th 2021 : development starting at 08 am and QA starting at 12 noon*
  • Friday, September 17th 2021 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, September 21st 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, September 22nd 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, September 23rd 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, September 24th 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service or when the operating system restarts.

Best regards

Your MCON Managed Services Team

The times are valid for the respective timezone

Resolved: DC3HAM Storage Outage

RESOLVED

UPDATE 2021-08-28 02:59PM CEST
On Friday 27th August 19:30 a redundant storage cluster in our colocation DC3.HAM was failing during normal operations.
After onsite ananlyis we found that the storage stopped all services due to a suspected split brain error.
As a result of the storage cluster virtual servers running on VMware could not run properly.

The repair of the storage cluster was started immediately after analysis and finished around 28th August 5a.m.
After storage recovery all running virtual servers have been restarted and checked, all production systems have been up and running after 28th August 09:45 a.m.

We are in further analysis of the root cause.

UPDATE 2021-08-28 10:25AM CEST
most systems are back, we are working to fix remaining problems mainly on QA system

UPDATE 2021-08-27 07:30PM CEST
VM storage cluster is currently not working as expected
thus sites and services are not available right now
we are working with high pressure to resolve this issue as fast as possible

Resolved: DC3HAM Network Outage

Update 04:26 PM CEST:

RESOLVED

Commercial power was restored, thus restoring services to a stable state.

We will provide details about exact root cause once we received from provider Lumen.

Update 04:12 PM CEST:

Transport NOC reports the main power breakers have been reset and commercial power is restored to the location. The team is working to turn individual breakers on one at a time to restore equipment. Services will begin to restore as each breaker is energized.

Update 02:59 PM CEST:

Field Operations have arrived on site and determined a commercial power failure to be the cause of impact to services. The local power provider has been engaged to assist with restoral efforts.

Update 02:05 PM CEST:

Lumen is still working on the issue

There is a major network event in Frankfurt, Germany that effects our services on a global scale, depending on the routing our services in our Colocation in Hamburg might not be reachable. The provider Lumen is working on that issue, we will update this as soon as we know more

Status Patch Day August 2021 – System Upgrades MCON Managed Services

Dear Sir or Madam,

according to our announcement below the patches for Region Europe, for Region America and for Region APAC were successfully done.

All systems are up and running. However if you encounter any problem please contact service@mcon-group.support.

Best regards
Your MCON Managed Services Team

Announcement from 5th August 2021:

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, August 12th 2021 : development starting at 08 am and QA starting at 12 noon*
  • Friday, August 13th 2021 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, August 17th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, August 18th 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, August 19th 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, August 20th 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service, or when the operating system restarts.

Best regards

Your MCON Managed Services Team

The times are valid for the respective timezone

Patch Day August 2021 – System Upgrades MCON Managed Services

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, August 12th 2021 : development starting at 08 am and QA starting at 12 noon*
  • Friday, August 13th 2021 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, August 17th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, August 18th 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, August 19th 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, August 20th 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service or when the operating system restarts.

Best regards

Your MCON Managed Services Team

The times are valid for the respective timezone

Status Patch Day July 2021 – System Upgrades MCON Managed Services

Dear Sir or Madam,

according to our announcement below the patches for Region Europe and Region America and Region APAC were successfully done.

All systems are up and running. However if you encounter any problem please contact service@mcon-group.support.

Best regards
Your MCON Managed Services Team

Announcement from 8th July 2021:

Dear customer,

In order to keep our server systems updated and secure, we need to apply patches on a regular basis.

The next patch days are scheduled :

Region Europe:

  • Thursday, July 15th 2021 : development starting at 08 am and QA starting at 12 noon*
  • Friday, July 16th 2021 between 0:00 am and 5:00 am: Production systems*

Region America:

  • Tuesday, July 20th 2021, development starting at 08 am and QA starting at 12 noon*
  • Wednesday, July 21st 2021 between 0:00 am and 5:00 am: Production systems*
  •  Region APAC:
  • Thursday, July 22nd 2021, development starting at 08 am and QA starting at 12 noon*
  • Friday, June 23rd 2021 between 0:00 am and 5:00 am: Production systems*

There may be short downtimes during service, or when the operating system restarts.

Best regards

Your MCON Managed Services Team

The times are valid for the respective timezone

Resolved: Connectivity issues in AWS AZ in Region Frankfurt

https://status.aws.amazon.com/

7:24 AM PDT Starting at 5:07 AM PDT we experienced increase connectivity issues for some instances, degraded performance for some EBS volumes and increased error rates and latencies for the EC2 APIs in a single Availability Zone (euc1-az3) in the EU-CENTRAL-1 Region. By 6:03 AM PDT, API error rates had returned to normal levels, but some Auto Scaling workflows continued to see delays until 6:35 AM PDT. By 6:10 AM PDT, the vast majority of EBS volumes with degraded performance had been resolved as well, and by 7:05 AM PDT, the vast majority of affected instances had been recovered, some of which may have experienced a power cycle. A small number of remaining instances and hosted on hardware which was adversely affected by this event and require additional attention. We continue to work to recover all affected instances and have opened notifications for the remaining impacted customers via the Personal Health Dashboard. For immediate recovery, we recommend replacing any remaining affected instances if possible.

6:29 AM PDT We continue to make progress in resolving the connectivity issues affecting some instances in a single Availability Zone (euc1-az3) in the EU-CENTRAL-1 Region. The increased error rates and latencies for the RunInstance and CreateSnapshot APIs have been resolved, as well as the degraded performance for some EC2 volumes within the affected Availability Zone. We continue to work on the remaining EC2 instances that are still impaired as a result of this event, some of which may have experienced a power cycle. While we do not expect any further impact at this stage, we would recommend continuing to utilize other Availability Zones in the EU-CENTRAL-1 region until this issue has been resolved.
6:05 AM PDT We are seeing increased error rates and latencies for the RunInstances and CreateSnapshot APIs, and increased connectivity issues for some instances in a single Availability Zone (euc1-az3) in the EU-CENTRAL-1 Region. We have resolved the networking issues that affected the majority of instances within the affected Availability Zone, but continue to work on some instances that are experiencing degraded performance for some EBS volumes. Other Availability Zones are not affected by this issue. We would recommend failing away from the affected Availability Zone until this issue has been resolved.
5:29 AM PDT We are investigating increased error rates and latencies for the EC2 APIs and connectivity issues for some instances in a single Availability Zone in the EU-CENTRAL-1 Region