Press Release

Strasbourg datacentre: latest information

2021
Press Release
Canada

Strasbourg datacentre: latest information

Roubaix, France | 23 April 2021 - 4:00 pm (CET)

Situation update for 23 April 2021:

  • For unaffected SBG1 services, as well as SBG3 and SBG4:
    • All SBG3 and SBG4 Bare Metal, VPS and Public Cloud services have been restored and made available to customers.
    • The physical migration of the SBG1 infrastructures to the other datacentres in our Strasbourg network is currently being completed. There are 150 servers to still be reconnected and this operation is expected to be completed by the middle of next week. 
  • For affected services in SBG1 and SBG2:
    • Our teams are continuing to support SBG2 customers with replacement solutions, as well as implementing their disaster recovery plans.
    • 14,472 Bare Metal servers were delivered as part of the implementation of alternative solutions in other datacentres.
    • 30,775 VPS have been restored, and the last 5,900 VPS services will be restored before the end of the month in SBG3 so that customers can keep the same IP addresses for their service(s). 
    • All Public Cloud services have been offered in our other datacentres.
  • In total, we estimate the impact as follows:
    • Currently, 113,000 of them have been fully recovered.
    • 120,000 services have been fully or partially affected.

OVHcloud took the decision to adopt free pricing measures for customers whose services were impacted. The practical arrangements for this will be communicated and offered to customers at the end of the month.

 

Follow-up on the incident 

When the fire broke out on 10 March at 00:47, our detection systems were instantly activated and our teams were able to respond according to the protocol in place for the energy rooms, where the smoke was detected.  Firefighters arrived on site very quickly, and their operations could only be carried out after the electrical power was switched off for the entire site, including all four datacentres. 

The OVHcloud teams are studying images from the 300 surveillance cameras installed as part of the Strasbourg site security measures, in order to draw key lessons that will result in an action plan to maximise the resilience of our infrastructures and cloud solutions. 

Investigations by the authorities and insurance companies are ongoing in order to determine the timeline, outbreak and spread of the fire, and thus identify its cause. This type of investigation takes up to several months to complete. The action plan to improve the resilience of our infrastructures will therefore be enhanced once conclusions have been drawn from the investigation.

 

Mobilising the OVHcloud ecosystem to offer solutions to our customers:

From the morning of 10 March, our teams were mobilised with one goal: to restore at least an equivalent service as soon as possible, for as many of our customers as possible. The following priorities were set to guide our actions: 

  • Secure the site to ensure a safe environment for teams.
  • Restore services or equivalent services to customers.
  • Provide alternative infrastructures in our other datacentres for affected customers.
  • Offer support for customers in implementing their Disaster Recovery Plan (DRP) mechanisms.

At the Strasbourg site: 

  • Within 48 hours, our teams prepared to be deployed in Strasbourg. More than 30 OVHcloud employees were put on a rotating schedule to work on the site, 24/7, to carry out the security measures and deploy the initial plan that would enable the SBG1, SBG3 and SBG4 datacentres to be gradually restarted from 17th March. The restart schedule was set out in strict compliance with a particularly stringent safety protocol.
  • A few days later, our teams were reinforced by our partners, who worked closely with us on: 
    • Electrical, climate and networks engineering: several partners, specialised in cabling and restoring datacentre pillars, worked to support the restoration of the power, the network distribution and the cooling systems.
    • Civil engineering: securing the SBG2 site and strengthening the structure.
    • Logistics: deploying stock and setting up logistics between various OVHcloud sites within 2 days. 
    • Environmental experts: conducted the smoke impact assessment on site. 
    • General management: deploying a campaign headquarters in the future SBG5 datacentre, with a canteen that could accommodate over 100 employees in compliance with Covid protocols.
    • Functional cleaning of servers impacted by soot: more than 80 specialists were deployed across the different floors at SBG3, cleaning up to 600-700 servers per day.
    • Data recovery specialists: worked to try and recover data from hard drives in the rooms affected by the fire. 
  • Our teams had to suspend and then fully resume this work after smoke was detected in a battery room from SBG1 (the smoke caused no further damage). This forced us to cut the power supply from SBG1 and SBG4 again on 19 March, in order to allow the firefighters to intervene at 7 pm. This delayed the resumption of services, which were then able to resume on 22 March. 
  • The teams followed the set reboot protocol (uninstall, clean, inspect, reinstall, start up) in compliance with security measures: room by room, aisle by aisle, rack by rack, server by server.
  • At the same time, our teams and partners cleaned the servers where traces of soot could be detected, then proceeded to physically migrate the servers from SBG1 to the other Strasbourg datacentres (SBG3 and SBG4).
  • Some servers in SBG1 underwent soot removal, where the process of cleaning and restoring them to service proved to be more delicate and therefore took longer than for the servers in SBG3. To speed up this complex and time-consuming process, a new cleaning chain was set up in our factory in Croix, and thanks to the support of our partners who trained our teams to carry out the tasks in parallel and increase the cleaning capacity.
  • A team of 20 people remain active in Strasbourg to proceed with the next steps of construction work.

 

At our production line in Croix: 

OVHcloud is based on an integrated model, which means we can manage our entire value chain using two production units: in France and in Canada. With this industry expertise, we were able to implement an emergency plan as early as 10 March to offer infrastructure capacity to our customers, and urgently produce nearly 18,000 servers in a matter of weeks: 

  • From the morning of 10 March, a stock of 2,000 servers was released and made available to our customers to support their migrations to the other 15 datacentres in our European network. 
  • By doubling the production lines, the production rate at our Croix site was increased from 1,500 servers per week to more than 3,000.
  • This pace has been maintained since then, in order to compensate for the destruction of the 14,046 servers that were in operation in the SBG2 datacentre.
  • On 17 March, after a comprehensive review, a dedicated workstation was set up to clean the Strasbourg servers exposed to the soot, in order to restore their required working condition. Our partners have trained our industry team employees in order to speed up the pace of this cleaning campaign.
  • In parallel with our production activities, on 11 March, two express containers dedicated to the network and high-voltage electricity were prepared in Croix and sent out for installation in Strasbourg on 12 March.
  • The teams remain mobilised on the physical migration site of the SBG1 infrastructures after detailed in-factory cleaning. We estimate that this work should be completed by the middle of next week.

We would like to thank our partners for their speed, commitment and flexibility. In just a few hours, they were able to design tailor-made joint solutions to carry out this huge industrial and logistical project. We would also like to thank our teams who worked hard and showed unwavering commitment during this ordeal.

 

Status of services as at 22 April 2021:

Cloud Universe 

Services

Progress in the provisioning of services*

Plan of action

Bare Metal Cloud

Bare Metal

99% of services have been re-delivered to customers across different sites

The last 400 servers from SBG1 are in the process of being restored in SBG3 by 25 April

Public Cloud

Public Cloud Instance

99% of services have been offered to customers from different sites 

 

 

Public Cloud Storage

99% of services have been offered to customers from different sites

Hosted Private Cloud

Hosted Private Cloud

88% of services have been restored on different sites

Production of compute and storage capacity in SBG and RBX to deliver all services within the next 10 days

Bare Metal Cloud

VPS 

98% of services have been offered to customers from different sites

8,000 services will be restored at the SBG site with the same IP address by the end of the month 

 

Beyond our teams, a whole ecosystem has been mobilised and sometimes doubled in workforce so that customers can retrieve all the features of their services. These teams continue to support affected customers one-on-one via their Control Panel or on social media. Despite the high number of requests on these platforms, we will keep working to restore all or equivalent services.

 

OVHcloud will keep working to restore its customers’ trust and remain the trusted partner for a cloud that respects European values. 

This incident should not make anyone doubt our digital, cloud or security interests. On the contrary, it teaches us the importance of best practices at all levels. From large companies to individuals, data is a valuable asset, and its security, both physical and cyber, requires strong solutions. 

The situation in Strasbourg has shown that OVHcloud teams have managed to rally together, and will continue to do so, for the company’s customers, without counting or measuring their efforts. These events, and the conclusions of the expert reports, will serve as important information from which we will draw all the lessons that we need to learn. We are committed to sharing the findings of these expert reviews, in full accordance with our company's core values: transparency and trust.

We are already working on an action plan to maximise the resilience of our infrastructures and services. We have therefore decided to rebuild the SBG1, SBG2 and SBG4 datacentres on the basis of these new standards. The Group’s other datacentres belonging to the same generation will also be included in this plan, which will span many months. Once again, progress will be regularly reported.

Through our 32 datacentres, 400,000 servers, 2 factories, 2,450 employees, and our network of over 3,000 partners in the 140 countries where our services are available, we will do everything we can to restore this unique bond that makes OVHcloud the market leader in providing a cloud service that respects European values. 

 

------
16 April 2021 - 17:00 (CET)

Customer information:

Following the action plans in progress and the regular visibility we want to give our customers, please find the current situation below:

Cloud Universe Services Progress in the provisioning of services* Location
Bare Metal Cloud Bare Metal 98% of servers have been delivered to customers The servers have been offered in Strasbourg and our other datacentres
Public Cloud Public Cloud Instance 95% of the customer services have been replaced The services offered are in our Gravelines (GRA) or Strasbourg (SBG) datacentres
  Public Cloud Storage 99% of the customer services have been replaced Services have been restored in our Roubaix (RBX) and Strasbourg (SBG) datacentres
Hosted Private Cloud Hosted Private Cloud 85% of services have been restored Services have been restored in our Strasbourg datacentres (SBG) and other datacentres
Bare Metal Cloud VPS 95% of the customer services have been replaced Services have been restored in our Strasbourg (SBG) datacentre and our other datacentres

 

The servers from SBG1 that have been cleaned and moved to SBG3 are in the process of being put into service, and the first servers are operational.
Our priority is to continue to restore services for all affected customers.

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. Find out more about our cleaning process via this link.
Today, the cleaning time for a rack is 7 hours, and our teams are improving this every day.
This schedule is also subject to Covid-19 and security protocols, defined for on-site interventions, with the goal of maintaining the health and ensuring the safety of our technical teams.
In addition, the current health restrictions have also affected logistics at our Strasbourg (SBG) site. We have had to source travel, accommodation and catering for all our employees who have come in to assist on site, while respecting government protocols, which of course made the operations more complex.

  • In the documentation & tutorials section, all tutorials useful for assisting all affected customers have been highlighted.
  • Your vouchers are currently being prepared. They will be visible in your Control Panel at the end of the month. In the meantime, please note that we have stopped billing for your services affected in SBG, as well as the services you ordered after 10 March as a replacement.
  • Real-time monitoring of the restart of SBG by room and rack is available at: vms.status-ovhcloud.com/sbg/

Communication:

Our mission remains to provide our customers with the highest quality of service to support their online business,  and we know how important this is to them. We sincerely apologise any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

---------
14 April 2021 - 20:00 (CET)

Customer information:

Following the action plans in progress and the regular visibility we want to give our customers, please find the current situation below:

Cloud Universe Services Progress in the provisioning of services* Location
Bare Metal Cloud Bare Metal 95% of servers have been delivered to customers The servers have been offered in Strasbourg and our other datacentres
Public Cloud Public Cloud Instance 86% of the customer services have been replaced The services offered are in our Gravelines (GRA) or Strasbourg (SBG) datacentres
  Public Cloud Storage 99% of the customer services have been replaced Services have been restored in our Roubaix (RBX) and Strasbourg (SBG) datacentres
Hosted Private Cloud Hosted Private Cloud 85% of services have been restored Services have been restored in our Strasbourg datacentres (SBG) and other datacentres
Bare Metal Cloud VPS 86% of the customer services have been replaced Services have been restored in our Strasbourg (SBG) datacentre and our other datacentres

 

The process of moving cleaned servers from SBG1 to SBG3 has started and the first servers are operational since this afternoon.

Our priority is to continue to restore services for all affected customers.

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. Find out more about our cleaning process via this link.

Today, the cleaning time for a rack is 7 hours, and our teams are improving this every day.

This schedule is also subject to Covid-19 and security protocols, defined for on-site interventions, with the goal of maintaining the health and ensuring the safety of our technical teams.

In addition, the current health restrictions have also affected logistics at our Strasbourg (SBG) site. We have had to source travel, accommodation and catering for all our employees who have come in to assist on site, while respecting government protocols, which of course made the operations more complex.

  • Your vouchers are currently being prepared. They will be visible in your Control Panel at the end of the month. In the meantime, please note that we have stopped billing for your services affected in SBG, as well as the services you ordered after 10 March as a replacement.
  • Real-time monitoring of the restart of SBG by room and rack is available at: vms.status-ovhcloud.com/sbg/.

Communication:

Our mission remains to provide our customers with the highest quality of service to support their online business and we know how important this is to them. We sincerely apologise any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

--------

12 April 2021 - 19:57 (CET)

Customer information:

Following the action plans in progress and the regular visibility we want to give our customers, please find the current situation below:

Cloud Universe Services Progress in the provisioning of services* Location
Bare Metal Cloud Bare Metal 92% of servers have been delivered to customers The servers have been offered in Strasbourg and our other datacentres
Public Cloud Public Cloud Instance 86% of the customer services have been replaced The services offered are in our Gravelines (GRA) or Strasbourg (SBG) datacentres
  Public Cloud Storage 99% of the customer services have been replaced Services have been restored in our Roubaix (RBX) and Strasbourg (SBG) datacentres
Hosted Private Cloud Hosted Private Cloud 79% of services have been restored Services have been restored in our Strasbourg datacentres (SBG) and other datacentres
Bare Metal Cloud VPS 84% of the customer services have been replaced Services have been restored in our Strasbourg (SBG) datacentre and our other datacentres

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. Find out more about our cleaning process via this link.

Today, the cleaning time for a rack is 7 hours, and our teams are improving this every day.

This schedule is also subject to Covid-19 and security protocols, defined for on-site interventions, with the goal of maintaining the health and ensuring the safety of our technical teams.

In addition, the current health restrictions have also affected logistics at our Strasbourg (SBG) site. We have had to source travel, accommodation and catering for all our employees who have come in to assist on site, while respecting government protocols, which of course made the operations more complex.

  • Your vouchers are currently being prepared. They will be visible in your Control Panel at the end of the month. In the meantime, please note that we have stopped billing for your services affected in SBG, as well as the services you ordered after 10 March as a replacement.
  • Real-time monitoring of the restart of SBG by room and rack is available at: vms.status-ovhcloud.com/sbg/.

Communication:

Our mission remains to provide our customers with the highest quality of service to support their online business,  and we know how important this is to them. We sincerely apologise any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

---------

 06 April 2021 - 19:57 (CET)

Customer information:

Following the action plans in progress and the regular visibility we want to give our customers, please find the current situation below:

  • SBG1 (rooms not destroyed):
    • 35% of Bare Metal Cloud servers are available to customers. Our ambition is to get 95% of our services back online by the end of this week.
  • SBG1 (destroyed rooms) and SBG2 (destroyed): substitute servers in other datacentres are offered to all our customers in order to restore the services.
    • The status of service availability for customers, as of:
      • VPS 34% of VMs are again offered to customers.
        •  The ambition is to reach 60% in the middle of the week and 95% at the end of this week.
      • Public Cloud Instance  80% of VMs are offered to customers.
      • Bare Metal Cloud 24% of services were restored on servers located in other sites.
      • The recovery schedule correlates with our server production capacity, which we have tripled to reduce time.
  • SBG3 is operational: SBG3 is operational: To date, 99% of VPS services and 99% of Bare Metal Cloud, Public Cloud Instance and Public Cloud Storage services have been made available to customers.
  • SBG4 is operational: 99% of Bare Metal Cloud servers are available to customers.

 

Below is the status of the services located in these datacentres:

Datacentre Cloud Universe Services Progress in the provisioning of services
SBG-1 (Not destroyed) Bare Metal Cloud Bare Metal 35%* with target 95% weekend of 05/04.
SBG1 and SBG2 (Destroyed) Bare Metal Cloud VPS 34%* of VMs offered to customers, with a target of 95% on the weekend of 05/04.
    Bare Metal 24%* of services restored.
  Public Cloud Public Cloud Instance 80%* of VMs offered to customers.
SBG-3 Public Cloud Public Cloud Instance 99%*
    Public Cloud Storage 95%* of the services available and returned to customers in read and write mode.
  Hosted Private Cloud Hosted Private Cloud Services are being gradually restarted, depending on their initial configuration,
on a 24/7 rotating schedule.
    Veeam Cloud Connect
Veeam Managed Backup		 The services are available to customers.
  Bare Metal Cloud VPS 99%*
    Bare Metal + vRack 99%*
SBG-4 Bare Metal Cloud Bare Metal + vRack 99%*

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. Find out more about our cleaning process via this link.

Today, the cleaning time for a rack is 7 hours, and our teams are improving this every day.

This schedule is also subject to Covid-19 and security protocols, defined for on-site interventions, with the goal of maintaining the health and ensuring the safety of our technical teams.

In addition, the current health restrictions have also affected logistics at our Strasbourg (SBG) site. We have had to source travel, accommodation and catering for all our employees who have come in to assist on site, while respecting government protocols, which of course made the operations more complex.

Communication:

Our mission remains to provide our customers with the highest quality of service to support their online business,  and we know how important this is to them. We sincerely apologise any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

--------------

02 April 2021 - 15:30 (CET)

Customer information:

Following the action plans in progress and the regular visibility we want to give our customers, please find the current situation below:

  • SBG1 (not destroyed): The servers in SBG1 are being inspected, cleaned and possibly repaired in Croix. Some have been returned to the Strasbourg site and are being reconnected. 
    • 25% of Bare Metal Cloud services are restored. Our ambition is to get 95% of our services back online by the end of next week.
  • SBG1 and SBG2 (destroyed): equivalent services in other datacentres are offered to all our customers, and we are supporting them with alternative solutions.
    • The status of service availability for customers, as of:
      • VPS 34% of VMs are again offered to customers.
        •  The ambition is to reach 60% early next week and 95% by next weekend.
      • Public Cloud Instance  80% of VMs are offered to customers.
      • Bare Metal Cloud 24% of services were restored on servers located in other sites.
      • The recovery schedule correlates with our server production capacity, which we have tripled to reduce time.
  • SBG3 is operational: To date, 99% of VPS services and 99% of Bare Metal Cloud, Public Cloud Instance and Public Cloud Storage services have been made available to customers.
  • SBG4 is operational: 99% of Bare Metal Cloud servers are available to customers.

Below is the status of the services located in these datacentres:

Datacentre Cloud Universe Services Progress in the provisioning of services
SBG-1 (Not destroyed) Bare Metal Cloud Bare Metal 25%* with target 100% weekend of 05/04.
SBG1 and SBG2 (Destroyed) Bare Metal Cloud VPS 34%* of VMs offered to customers, with a target of 95% on the weekend of 05/04.
    Bare Metal 24%* of services restored.
  Public Cloud Public Cloud Instance 80%* of VMs offered to customers
SBG-3 Public Cloud Public Cloud Instance 99%*
    Public Cloud Storage 95%* of the services available and returned to customers in read and write mode.
  Hosted Private Cloud Hosted Private Cloud Services are being gradually restarted,
depending on their initial configuration,
on a 24/7 rotating schedule.
    Veeam Cloud Connect
Veeam Managed Backup		 The services are available to customers.
  Bare Metal Cloud VPS 99%*
    Bare Metal + vRack 99%*
SBG-4 Bare Metal Cloud Bare Metal + vRack 99%*


*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. Find out more about our cleaning process via this link.

Today, the cleaning time for a rack is 7 hours, and our teams are improving this every day.

This schedule is also subject to Covid-19 and security protocols, defined for on-site interventions, with the goal of maintaining the health and ensuring the safety of our technical teams.

In addition, the current health restrictions have also affected logistics at our Strasbourg (SBG) site. We have had to source travel, accommodation and catering for all our employees who have come in to assist on site, while respecting government protocols, which of course made the operations more complex.

Communication:

  • An FAQ is available here.
  • To follow updates about this situation in real time:
  • The OVHcloud Community platform is available to interact with community members, and support you
  • The latest video from Octave Klaba, founder of OVHcloud, is available on his Twitter account.

Our mission remains to provide our customers with the highest quality of service to support their online business,  and we know how important this is to them. We sincerely apologise any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

The next publication of this status is scheduled for 06/04/2021.

-------
01 April 2021 - 15:30 (CET)

Customer information:

Following the action plans in progress and the regular visibility we want to give our customers, please find the current situation below:

  • SBG1: The servers in SBG1 are being inspected, cleaned and possibly repaired in Croix. Some have been returned to the Strasbourg site and are being reconnected. The first racks are already in service today. Operations will continue for one week. The SBG1 site is being dismantled.
  • SBG2: equivalent services in other datacentres are offered to all our customers, and we are supporting them with alternative solutions.
  • SBG3 is operational: To date, 95% of VPS services and 99% of Bare Metal Cloud, Public Cloud Instance and Public Cloud Storage services have been made available to customers.
    The vRack service has also been restored.
  • There was a malfunction on SBG3's energy equipment today around 6am, with no consequence on the service. Our teams put in place the necessary checks and are continuing operations on-site.
     
  • SBG4 is operational: 100% of Bare Metal Cloud servers are available to customers.

Below is the status of the services located in these datacentres:

Datacentre Cloud Universe Services Progress in the provisioning of services
SBG-3 Public Cloud Public Cloud Instance 95%*
    Public Cloud Storage 99% of services available to and returned to customers in read and write mode.
  Hosted Private Cloud Hosted Private Cloud Services are being gradually restarted,
depending on their initial configuration,
on a 24/7 rotating schedule.
    Veeam Cloud Connect
Veeam Managed Backup		 The services are available to customers.
  Bare Metal Cloud VPS 95%*
    Bare Metal + vRack 99%*
SBG-4   Bare Metal + vRack 100%*


*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. Find out more about our cleaning process via this link.
Today, the cleaning time for a rack is 7 hours, and our teams are improving this every day.
This schedule is also subject to Covid-19 and security protocols, defined for on-site interventions, with the goal of maintaining the health and ensuring the safety of our technical teams.
In addition, the current health restrictions have also affected logistics at our Strasbourg (SBG) site. We have had to source travel, accommodation and catering for all our employees who have come in to assist on site, while respecting government protocols, which of course made the operations more complex.

Communication:

  • An FAQ is available here.
  • To follow updates about this situation in real time:
  • The OVHcloud Community platform is available to interact with community members, and support you
  • The latest video from Octave Klaba, founder of OVHcloud, is available on his Twitter account.

Our mission remains to provide our customers with the highest quality of service to support their online business,  and we know how important this is to them. We sincerely apologise any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

------
31 March 2021 - 18:30 (CET)

Customer information:

Following the action plans in progress and the regular visibility we want to give our customers, please find the current situation below:

  • SBG1: The servers in SBG1 are being inspected, cleaned and possibly repaired in Croix. Some have been returned to the Strasbourg site and are being reconnected. The first racks will be in service between tonight and tomorrow morning. Operations will continue for one week. The SBG1 site is being dismantled.
  • SBG2: equivalent services in other datacentres are offered to all our customers, and we are supporting them with alternative solutions.
  • SBG3 is operational: To date, 95% of VPS services and 99% of Bare Metal Cloud, Public Cloud Instance and Public Cloud Storage services have been made available to customers.
    The vRack service has also been restored.
  • SBG4 is operational: 100% of Bare Metal Cloud servers are available to customers.

Below is the status of the services located in these datacentres:

Datacentre Cloud Universe Services Progress in the provisioning of services
SBG-3 Public Cloud Public Cloud Instance 95%*
    Public Cloud Storage 99% of services available to and returned to customers in read and write mode.
  Hosted Private Cloud Hosted Private Cloud Services are being gradually restarted,
depending on their initial configuration,
on a 24/7 rotating schedule.
    Veeam Cloud Connect
Veeam Managed Backup		 The services are available to customers.
  Bare Metal Cloud VPS 95%*
    Bare Metal + vRack 99%*
SBG-4   Bare Metal + vRack 100%*


*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. Find out more about our cleaning process via this link.
Today, the cleaning time for a rack is 7 hours, and our teams are improving this every day.
This schedule is also subject to Covid-19 and security protocols, defined for on-site interventions, with the goal of maintaining the health and ensuring the safety of our technical teams.
In addition, the current health restrictions have also affected logistics at our Strasbourg (SBG) site. We have had to source travel, accommodation and catering for all our employees who have come in to assist on site, while respecting government protocols, which of course made the operations more complex.

Communication:

  • An FAQ is available here.
  • To follow updates about this situation in real time:
  • The OVHcloud Community platform is available to interact with community members, and support you
  • The latest video from Octave Klaba, founder of OVHcloud, is available on his Twitter account.

Our mission remains to provide our customers with the highest quality of service to support their online business,  and we know how important this is to them. We sincerely apologise any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

------
30 March 2021 – 17:00 (CET)

Customer information:

  • SBG1: The recoverable Bare Metal Cloud servers are being cleaned for reinstallation in Strasbourg (SBG3 and SBG4).
  • SBG3 is operational: To date, 95% of VPS and Bare Metal Cloud services have been made available to customers.

The vRack service has also been restored for a large number of SBG3 clients.

  • SBG4 is operational: 100% of Bare Metal servers are available to customers.

Below is the status of the services located in these datacentres:

Datacentre Cloud Universe Services Progress in the provisioning of services
SBG-3 Public Cloud Public Cloud Instance 91%*
    Public Cloud Storage Services available and delivered to customers in read and write mode.
  Hosted Private Cloud Hosted Private Cloud Services are being gradually restarted,
depending on their initial configuration,
on a 24/7 rotating schedule.
    Veeam Cloud Connect
Veeam Managed Backup		 The services are available to customers.
  Bare Metal Cloud VPS 95%*
    Bare Metal 95%*
SBG-4   Bare Metal 100%*

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. Find out more about our cleaning process via this link.
Today, the cleaning time for a rack is 7 hours, and our teams are improving this every day.
This schedule is also subject to Covid-19 and security protocols, defined for on-site interventions, with the goal of maintaining the health and ensuring the safety of our technical teams.
In addition, the current health restrictions have also affected logistics at our Strasbourg (SBG) site. We have had to source travel, accommodation and catering for all our employees who have come in to assist on site, while respecting government protocols, which of course made the operations more complex.

**As backups are hosted on Public Cloud Storage services, restarting the PCS services will result in many backups being made available.

 

Priority 1: Restoring services to our customers

  • Status summary for each datacentre
     

Datacentre

Situation

Reboot

Electrical/Network

Server restart

SBG-1

Data centre not reactivated.

After inspection by teams with expertise, technical teams can access the rooms.

 

 

 

Moving recoverable servers after inspection and cleaning,

to our other datacentres: Gravelines (GRA), Roubaix (RBX), Croix (CRX) and Strasbourg (SBG4) to return to service.

Relaunch servers after cleaning from Thursday 01 April. (1).

 

SBG-2

Datacentre powered down and secured.

 

Recommendation: replacement of infrastructures in other datacentres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

SBG-3

Datacentre operational since 18 March.

 

 

Accomplished

 

VPS, Public Cloud, Bare Metal Cloud, Hosted Private Cloud, Veeam Cloud Connect, Veeam Managed Backup and vRack services are in the process of being restored. (2).

SBG-4

Datacentre operational

Accomplished

 

100% of Bare Metal customer servers are accessible.

 

 

(1) / (2) Additional information on the progress of operations:

  • SBG1: rooms A to D:
    • Servers to be transferred to racks in SBG4 with a scheduled return to service from Wednesday 31 March.
  • SBG1: rooms 61E and 62E:
    • The cleaning of all the servers located in these rooms continues in two eight-hour shifts with teams in rotation at our production site in Croix.
    • A total of 1,212 servers have been received in Croix:
      • 800 servers have been cleaned to date, 310 verified and 90 reconditioned to send to our Strasbourg (SBG) website.
      • The cleaned and tested servers will then be gradually returned, already in their rack.
      • Server transfers at our Strasbourg (SBG3) site: 48 hours between on-site transfer and reboot.

The services are scheduled to be gradually restarted from Thursday 01/04, with the last servers to be restarted between the 07/04 and 08/04.

  • SBG3:
  • the cleaning work on all Bare Metal servers is expected to be completed by the evening of Wednesday 31/03.
  • The cleaned and tested servers will then be returned to their rack and reconnected.
  • 347 servers and 59 switches from 5th floor of SBG3 are returning to our Croix site (CRX) on Wednesday for cleaning.
  • The racks will then be transferred to our Strasbourg (SBG) site.
     
  • SBG Service Status Pages - OVHcloud Control Panel

A view per datacentre is available per service in the Control Panel.

For Public Cloud customers who have services in Strasbourg in racks that are being cleaned, and who have not yet recovered their services, a notification has been added to their Control Panel:

Priority 2: Provisioning of infrastructure for our customers in other datacentres

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.
  • Due to the current demand, delivery times on our Bare Metal Cloud services may take longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.
  • On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres.
     

Update on quantities delivered since 10th March:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare Metal Cloud

Bare Metal

Servers

7,817

 

HA-NAS

TB

259

Public Cloud

VPS

VM

17,075

Hosted Private Cloud

Host

Servers

1,059

 

Datastore

Number of zpool

871

 

 

Volume of TB

4,763

 

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers and have a support plan in place.

Communications

  •  The latest video from Octave Klaba, founder of OVHcloud, is available on his Twitter account.
  • The OVHcloud Community platform is now available to interact with community members and support you.
  • An FAQ is available here.
  • To follow updates about this situation in real time:

Our mission remains to provide our customers with the highest quality of service to support their online business  and we know how important this is to them. We sincerely apologise any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

------
29 March 2021 – 19:00 (CET)

Customer information:

  • SBG1: The recoverable Bare Metal Cloud servers are being cleaned for reinstallation in Strasbourg (SBG3 and SBG4).

They will be gradually restarted at the beginning of this week.

  • SBG3 is operational: 90% of VPS services and 85% of Bare Metal Cloud services have been made available to customers again.
  • SBG4 is operational: 100% of Bare Metal servers are available to customers.

Below is the status of the services located in these datacentres:

 

Datacentre Cloud Universe Services Progress in the provisioning of services
SBG-3 Public Cloud Public Cloud Instance 90%*
    Public Cloud Storage Services will be returned to customers in read-only mode on the evening of Monday 29/03, with write access by Tuesday 30/03.**
  Hosted Private Cloud Hosted Private Cloud Services are being gradually restarted,
depending on their initial configuration,
on a round the clock, 24/7, rotation schedule.
    Veeam Cloud Connect
Veeam Managed Backup		 The services are available to customers.
  Bare Metal Cloud VPS 90%
    Bare Metal 85%
SBG-4 Bare Metal Cloud Bare Metal 100%*

 

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. Find out more about our cleaning process via this link.

Today, the cleaning time for a rack is 7 hours, and our teams are improving this every day.

This schedule is also subject to Covid-19 and security protocols, defined for on-site interventions, with the goal of maintaining the health and ensuring the safety of our technical teams.

In addition, the current health restrictions have also affected logistics at our Strasbourg (SBG) site. We have had to source travel, accommodation and catering for all our employees who have come in to assist on site, while respecting government protocols, which of course made the operations more complex.

 

**As backups are hosted on Public Cloud Storage services, restarting the PCS services will result in many backups being made available.

 

 

Priority 1: Restoring services to our customers

  • Status summary for each datacentre

 

Datacentre

Situation

Reboot

Electrical/Network

Server restart

SBG-1

Data centre not reactivated.

After inspection by teams with expertise, technical teams can access the rooms.

 

 

 

Moving recoverable servers after inspection and cleaning,

to our other datacentres: Gravelines (GRA), Roubaix (RBX), Croix (CRX) and Strasbourg (SBG4) to return to service.

After cleaning, the servers will be restarted by mid-week of 29 March. (1).

 

SBG-2

Datacentre powered down and secured.

 

Recommendation: replacement of infrastructures in other datacentres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

SBG-3

Datacentre operational since 18 March.

 

 

Accomplished

 

VPS, Public Cloud, Bare Metal Cloud, Hosted Private Cloud, Veeam Cloud Connect and Veeam Managed Backup services have started to be restored. (2).

SBG-4

Datacentre operational

Accomplished

 

100% of Bare Metal customer servers are accessible.

 

(1) / (2) Additional information on the progress of operations:
 

  • SBG1: rooms A to D:
    • Servers to be transferred to racks in SBG4 with a scheduled return to service from Wednesday 30/03.
  • SBG1: rooms 61E and 62E:
    • Teams are continuing to work in rotation in two eight-hour shifts to clean all servers located in these rooms at our production site in Croix, and will finish by Tuesday 30/03.
    • A total of 1,212 servers have been received in Croix:
      • As of Monday 29/03 at 15:00, 805 had been cleaned.
      • 216 have been retouched (power supply and casing changed):  87 have been tested and are operational, 107 are still being tested (the rest are out of order or waiting to be repaired).
      • The cleaned and tested servers will then be gradually returned, already in their rack.
      • Server transfers at our Strasbourg (SBG3) site: 48 hours between on-site transfer and reboot.

The services are scheduled to be gradually restarted from Thursday 01/04, with the last servers to be restarted between the 07/04 and 08/04.

  • SBG3:
  • the cleaning work on all Bare Metal servers is expected to be completed by the evening of Wednesday 31/03.
  • The cleaned and tested servers will then be returned to their rack and reconnected.
  • 347 servers and 59 switches from 5th floor of SBG3 are returning to our Croix site (CRX) on Wednesday for cleaning.
  • The racks will then be transferred to our Strasbourg (SBG) site.
  • The services are scheduled to be gradually restarted between Thursday 01/04 and Friday 02/04.

 

  • SBG Service Status Pages - OVHcloud Control Panel

A view per datacentre is available per service in the Control Panel.

 

For Public Cloud customers who have services in Strasbourg in racks that are being cleaned, and who have not yet recovered their services, a notification has been added to their Control Panel:

 

Priority 2: Provisioning of infrastructure for our customers in other datacentres

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.
  • Due to the current demand, delivery times on our Bare Metal Cloud services may take longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.
  • On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres.

Update on quantities delivered since 10th March:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare Metal Cloud

Bare Metal

Servers

7,597

 

NAS HA

TB

259

Public Cloud

VPS

VM

16,719

Hosted Private Cloud

Host

Servers

964

 

Datastore

Number of zpool

823

 

 

Volume of TB

4,718

 

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers and have a support plan in place.
     

Communications

  •  The latest video from Octave Klaba, founder of OVHcloud, is available on his Twitter account.
  • The OVHcloud Community platform is now available to interact with community members and support you.
  • An FAQ is available here.
  • To follow updates about this situation in real time:

Our mission remains to provide our customers with the highest quality of service to support their online business  and we know how important this is to them. We sincerely apologise any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

----
28 March 2021 – 17:00 (CET)

Customer information:

  • SBG1: The recoverable Bare Metal Cloud servers are being cleaned for reinstallation in Strasbourg (SBG3 and SBG4). Re-commissioning will be gradually started by the beginning of next week (after inspection and cleaning).
  • SBG3 is operational: 84% of Bare Metal Cloud (VPS) services have been made available to customers again, with a target of 90% by the evening of 28 March.
  • SBG4 is operational: 100% of Bare Metal servers are available to customers.

Below is the status of the services located in these datacentres:

Datacentre Cloud Universe Services Progress in the provisioning of services
SBG-3 Public Cloud Public Cloud Instance 90%*
    Public Cloud Storage Services will be gradually restarted from 27 March**
Services not yet returned to customers will be available again on Monday 29/03 and Tuesday 30/03.
  Hosted Private Cloud Hosted Private Cloud Services are being gradually restarted, depending on their initial configuration, on a round the clock, 24/7, rotation schedule.
    Veeam Cloud Connect Services will be gradually restarted from 25 March.
  Bare Metal Cloud VPS 84% with a target of 90% by the end of the day.
    Bare Metal 80% with a target of 85% by the end of the day.
SBG-4 Bare Metal Cloud Bare Metal 100%*

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. Find out more about our cleaning process via this link.

Today, the cleaning time for a rack is 7 hours, and our teams are improving this every day.

This schedule is also subject to Covid-19 and security protocols, defined for on-site interventions, with the goal of maintaining the health and ensuring the safety of our technical teams.

**As backups are hosted on Public Cloud Storage services, restarting the PCS services will result in many backups being made available.

 

Priority 1: Restoring services to our customers

  • Status summary for each datacentre
Datacentre Situation Reboot Electrical/Network Server restart
SBG-1 Datacentre not reactivated.
After inspection by teams with expertise, technical teams can access the rooms.		   Moving recoverable servers after inspection and cleaning, to our other datacentres: Gravelines (GRA), Roubaix (RBX), Croix (CRX) and Strasbourg (SBG4) to return to service.
Restart of the servers after cleaning by mid week of 29 March. (1).
SBG-2 Datacentre powered down and secured.   Recommendation: replacement of infrastructures in other datacentres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)
SBG-3 Datacentre operational since 18 March. Accomplished VPS, Public Cloud, Bare Metal Cloud, Hosted Private Cloud, Veeam Cloud Connect and Veeam Managed Backup services have started to be restored. (2).
SBG-4 Datacentre operational Accomplished 100% of Bare Metal customer servers are accessible.

 

(1) / (2) Additional information on the progress of operations:

  • SBG1: rooms A to D:
    • Servers to be transferred to racks in SBG4 with a scheduled return to service from Wednesday 30/03.
  • SBG1: rooms 61E and 62E:
    • Teams are working in rotation to clean all servers located in these rooms throughout the weekend, and will be finished by Tuesday 30/03 in Croix.
    • The cleaned and tested servers will then be returned to the different datacentres (RBX, GRA, SBG4) and set up in their new racks.
    • The services are scheduled to be gradually restarted between Thursday 01/04 and Friday 02/04.
  • SBG3: the cleaning work on all Bare Metal servers is expected to be completed by Wednesday 31/03.
  • SBG Service Status Pages - OVHcloud Control Panel

A view per datacentre is available per service in the Control Panel.

For Public Cloud customers based in Strasbourg (SBG) who have not yet recovered their services located in racks that are being cleaned, a notification has been added to their Control Panel:

Priority 2: Provisioning of infrastructure for our customers in other datacentres

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.
  • Due to the current demand, delivery times on our Bare Metal Cloud services may take longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.
  • On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres.

Update on quantities delivered since 10th March:

CLOUD UNIVERSE PRODUCTS   QUANTITY
Bare Metal Cloud Bare Metal Servers 7,484
  NAS HA TB 259
Public Cloud VPS VM 16557
Hosted Private Cloud Host Servers 887
  Datastore Number of zpool 813
    Volume of TB 4,698

 

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers and have a support plan in place.

Communications

  •  The latest video from Octave Klaba, founder of OVHcloud, is available on his Twitter account.
  • The OVHcloud Community platform is now available to interact with community members and support you.
  • An FAQ is available here.
  • To follow updates about this situation in real time:

Our mission remains to provide our customers with the highest quality of service to support their online business  and we know how important this is to them. We sincerely apologise any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

----------------

27 March 2021 – 16:00 (CET)

Customer information:

  • SBG1: The recoverable Bare Metal Cloud servers are being cleaned for relocation to Strasbourg or other datacentres. Re-commissioning will be gradually started by the beginning of next week (after inspection and cleaning).
  • SBG3 is operational: 83% of Bare Metal Cloud services have been made available to customers again, with a target of 90% by the evening of 28 March.
  • SBG4 is operational: 100% of Bare Metal servers are available to customers.

Below is the status of the services located in these datacentres:

Datacentre Cloud Universe Services Progress in the provisioning of services
SBG-3 Public Cloud Public Cloud Instance 89%*
    Public Cloud Storage Services will be gradually restarted from 27 March**
Services not yet returned to customers will be available again on Monday 29/03 and Tuesday 30/03.
  Hosted Private Cloud Hosted Private Cloud Services are being gradually restarted, depending on their initial configuration, on a round the clock, 24/7, rotation schedule.
    Veeam Cloud Connect Services will be gradually restarted from 25 March.
  Bare Metal Cloud VPS 83%* (90% by 28/03).
    Bare Metal 75%* (85% by 28/03). 
SBG-4 Bare Metal Cloud Bare Metal 100%*

 

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. Find out more about our cleaning process via this link.

Today, the cleaning time for a rack is 7 hours, and our teams are improving this every day.

This schedule is also subject to Covid-19 and security protocols, defined for on-site interventions, with the goal of maintaining the health and ensuring the safety of our technical teams.

**As backups are hosted on Public Cloud Storage services, restarting the PCS services will result in many backups being made available.

Priority 1: Restoring services to our customers

  • Status summary for each datacentre

Datacentre

Situation

Reboot

Electrical/Network

Server restart

SBG-1

Data centre not reactivated.

After inspection by teams with expertise, technical teams can access the rooms.

 

 

Moving recoverable servers after inspection and cleaning, to our other datacentres: Gravelines (GRA), Roubaix (RBX), Croix (CRX) and Strasbourg (SBG4) to return to service.

After cleaning, the servers will be restarted by mid-week of 29 March.(1)

SBG-2

Datacentre powered down and secured.

 

Recommendation: replacement of infrastructures in other datacentres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

SBG-3

Datacentre operational since 18 March.

Accomplished

VPS, Public Cloud, Bare Metal Cloud, Hosted Private Cloud, Veeam Cloud Connect and Veeam Managed Backup services are in the process of being restored.(2)

SBG-4

Datacentre operational

Accomplished

100% of Bare Metal customer servers are accessible.

 

(1) / (2) Additional information on the progress of operations:

  • SBG1: rooms A to D:
    • Servers to be transferred to racks in SBG4 with a scheduled return to service from Wednesday 30/03.
  • SBG1: rooms 61E and 62E:
    • Teams are working in rotation to clean all servers located in these rooms throughout the weekend, and will be finished by Tuesday 30/03 in Croix.
    • The cleaned and tested servers will then be returned to the different datacentres (RBX, GRA, SBG4) and set up in their new racks.
    • The services are scheduled to be gradually restarted between Thursday 01/04 and Friday 02/04.
       
  • SBG3: the cleaning work on all Bare Metal servers will probably be completed by Wednesday 31/03.
     
  • SBG Service Status Pages - OVHcloud Control Panel

A view per datacentre is available per service in the Control Panel.

For Public Cloud customers based in Strasbourg (SBG) who have not yet recovered their services located in racks that are being cleaned, a notification has been added to their Control Panel:

Priority 2: Provisioning of infrastructure for our customers in other datacentres

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.
     

Update on quantities delivered since 10th March:

CLOUD UNIVERSE PRODUCTS   QUANTITY
Bare Metal Cloud Bare Metal Servers 7,217
  NAS HA TB 259
Public Cloud VPS VM 16,337
Hosted Private Cloud Host Servers 870
  Datastore Number of zpool 796
    Volume of TB 4,532

 

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.

On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres.

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers and have a support plan in place.

Communications

  •  The latest video from Octave Klaba, founder of OVHcloud, is available on his Twitter account.
  • The OVHcloud Community platform is now available to interact with community members and support you.
  • An FAQ is available here.
  • To follow updates about this situation in real time:

Our mission remains to provide our customers with the highest quality of service to support their online business  and we know how important this is to them. We sincerely apologise any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

-----------

26 March 2021 - 17:00 (CET)

Customer information:

  • SBG1: The recoverable Bare Metal Cloud servers are being cleaned for relocation to Strasbourg or other datacentres. Re-commissioning will be gradually started by the beginning of next week (after inspection and cleaning).
  • SBG3 is operational.
  • SBG4 is operational: 100% of Bare Metal servers are available to customers.

Below is the status of the services located in these datacentres:

Datacentre Cloud Universe Services Progress in the provisioning of services
SBG-3 Public Cloud Public Cloud Instance 89%*
    Public Cloud Storage There will be a gradual restart for all services from 27 March**.
  Hosted Private Cloud Hosted Private Cloud Services are being gradually restarted,
depending on their initial configuration,
on a round the clock, 24/7, rotation schedule.
    Veeam Cloud Connect Services will be gradually restarted beginning this evening, 25 March.
  Bare Metal Cloud VPS 83%*
    Bare Metal 75%*
SBG-4 Bare Metal Cloud Bare Metal 100%*

 

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. Find out more about our cleaning process via this link.

Today, the cleaning time for a rack is 7 hours, and our teams are improving this every day.

This schedule is also subject to Covid-19 and security protocols, defined for on-site interventions, with the goal of maintaining the health and ensuring the safety of our technical teams.

**As backups are hosted on Public Cloud Storage services, restarting the PCS services will result in many backups being made available.

Priority 1: Restoring services to our customers

  • Status summary for each datacentre

Datacentre

Situation

Electrical restart

Network restart

Server restart

SBG-1

Data centre not reactivated.

After inspection by teams with expertise, technical teams can access the rooms.

 

 

 

 

Moving recoverable servers after inspection and cleaning, to our other datacentres: Gravelines (GRA), Roubaix (RBX), Croix (CRX) and Strasbourg (SBG4) to return to service.

Restart of the servers after cleaning by mid week of 29 March.

SBG-2

Datacentre powered down and secured.

 

 

Recommendation: replacement of infrastructures in other datacentres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

SBG-3

Datacentre operational since 18 March.

Accomplished

Accomplished

VPS, Public Cloud, Bare Metal Cloud, Hosted Private Cloud, Veeam Cloud Connect, and Veeam Managed Backup services have started to be restored.

SBG-4

Datacentre operational

Accomplished

Accomplished

100% of Bare Metal customer servers are accessible.

 

  • SBG Service Status Pages - OVHcloud Control Panel

A view per datacentre is available per service in the Control Panel.

 

 For Public Cloud customers based in Strasbourg (SBG) who have not yet recovered their services located in racks that are being cleaned, a notification has been added to their Control Panel:

 

Priority 2: Provisioning of infrastructure for our customers in other datacentres

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.

Update on quantities delivered since 10th March:

CLOUD UNIVERSE PRODUCTS   QUANTITY
Bare Metal Bare Metal Servers 6 987
  NAS HA TB 259
Public Cloud VPS VM 16 087
Hosted Private Cloud Host Servers 840
  Datastore Number of zpool 764
    Volume of TB 4 158

 

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.

On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres.

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers and have a support plan in place.

Communications

  •  The latest video from Octave Klaba, founder of OVHcloud, is available on his Twitter account.
  • The OVHcloud Community platform is now available to interact with community members and support you.
  • An FAQ is available here.
  • To follow updates about this situation in real time:

Our mission remains to provide our customers with the highest quality of service to support their online business  and we know how important this is to them. We sincerely apologise any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

-------------

25 March 2021 - 16:30 (CET)

Customer information:

  • SBG1: Bare Metal recoverable servers are being cleaned to be put back into service in Strasbourg or other datacentres by early next week (following inspection and cleaning).
  • SBG3 is operational.
  • SBG4 is operational: 95% of Bare Metal servers are available to customers.

Below is the status of the services located in these datacentres:

Datacentre Cloud Universe Services Progress
SBG-3 Public Cloud Public Cloud Instance 86%*
    Public Cloud Storage Restart on 27 March
  Hosted Private Cloud Hosted Private Cloud Services are being gradually restarted,
depending on their initial configuration,
on a round the clock, 24/7, rotation schedule.
    Veeam Cloud Connect Services will be gradually restarted beginning this evening, 25 March.
  Bare Metal Cloud VPS 83%*
    Bare Metal 71%*
SBG-4 Bare Metal Cloud Bare Metal 95%*

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. Find out more about our cleaning process via this link.

This schedule is also subject to Covid-19 and safety protocols, defined for on-site interventions, to maintain the health and ensure the safety of our technical teams.

 

Priority 1: Restoring services to our customers

  • Status summary for each datacentre

Datacentre

Situation

Electrical restart

Network restart

Server restart

SBG-1

Datacentre not reactivated.
After inspection by teams with expertise, technical teams can access the rooms.

 

 

 

 

Moving recoverable servers after inspection and cleaning, to our other datacentres: Gravelines (GRA), Roubaix (RBX), Croix (CRX) and Strasbourg (SBG4) to return to service.
The aim is to restore services from the beginning of the week of 29 March.

SBG-2

Datacentre switched off and secured.

 

 

Replacement of infrastructure in other datacentres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

SBG-3

Datacentre operational since 18 March.

Accomplished

Accomplished

VPS, Bare Metal, Public Cloud, and Veeam Cloud Connect services are currently being restored. Hosted Private Cloud services will be restored gradually.

SBG-4

Datacentre operational

Accomplished

Accomplished

95% of Bare Metal customer servers are accessible.
  • SBG Service Status Pages - OVHcloud Control Panel

A view per datacentre is available per service in the Control Panel.

Priority 2: Provisioning of infrastructure for our customers in other datacentres

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.

Update on quantities delivered since 10th March:

CLOUD UNIVERSE PRODUCTS   QUANTITY
Bare Metal Bare Metal Servers 6812
  NAS HA TB 259
Public Cloud VPS VM 15732
Hosted Private Cloud Host Servers 723
  Datastore Number of zpool 706
    Volume of TB 4079

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.

On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres.

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers and have a support plan in place.

Communications

  • The OVHcloud Community platform is now available to interact with community members and support you.
  • An FAQ is available here.
  • To follow updates about this situation in real time:

Our mission remains to provide our customers with the highest quality of service to support their online business and we know how important this is to them. We sincerely apologise any inconveniences related this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

-----------

24 March 2021 - 18:00 (CET)

Customer information:

  • In the Documentation & tutorials page, we’ve prepared useful guides to provide assistance to customers concerned.
  • SBG4 is operational.
  • SBG3 is fully operational, below is the status of the services located in this datacentre:
Datacentre
SBG-3		 Cloud Universe Services Progress
  Public Cloud Public Cloud Instance 86%*
    Public Cloud Storage Restart on 27 March
  Hosted Private Cloud Hosted Private Cloud Services are being gradually restarted,
depending on their initial configuration,
on a round the clock, 24/7, rotation schedule.
  Bare Metal Cloud VPS 83%*
    Bare Metal 71%*

 

*The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. However, server cleanup is required, and this will determine when certain racks are put back into to service. This schedule is also dependent on the safety protocol defined for on-site interventions, as the safety of our technical teams must be guaranteed.

  • At the beginning of the week, all customers concerned received a communication on the commercial measures that will be applied to their situation. You can find a dedicated section about this in the FAQ.

Priority 1: Restoring services to our customers

  • Status summary for each datacentre

Datacentre

Situation

Electrical restart

Network restart

Server restart

SBG-1

Data centre not reactivated. After inspection by teams with expertise, technical teams can access the rooms.

 

Switched off as a precaution

Relocation of SBG1 servers to our other datacentres: Gravelines (GRA), Roubaix (RBX), Croix (CRX) and Strasbourg (SBG4) to return to service.
ETA: Wednesday 24 March.

SBG-2

Datacentre switched off and secured.

 

 

Replacement of infrastructure in other datacentres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

SBG-3

Datacentre operational since 18 March

Accomplished

Accomplished

VPS, Bare Metal, and Public Cloud services are currently being restored. Hosted Private Cloud services will be restored gradually.

SBG-4

Datacentre operational.

Accomplished

Accomplished

ETA: 25 March.

 

  • SBG Service Status Pages - OVHcloud Control Panel

A view per datacentre is available per service in the Control Panel.

Priority 2: Provisioning of infrastructure for our customers in other datacentres

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.

Update on quantities delivered since 10th March:

CLOUD UNIVERSE PRODUCTS   QUANTITY
Bare metal Bare Metal Servers 6103
  NAS HA TB 259
Public Cloud VPS VM 14973
Hosted Private Cloud Host Servers 640
  Datastore Number of zpool 662
    Volume of TB 3764

 

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.

On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres. 

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers and have a support plan in place.

Communications

  • Octave Klaba, founder of OVHcloud, is available on his Twitter account.
  • The OVHcloud Community platform is now available to interact with community members and support you.
  • An FAQ is available here.
  • To follow updates about this situation in real time:

Our mission remains to provide our customers with the highest quality of service and to support their online business and we know how important this is to them. We sincerely apologise for the problems caused by this fire. We will continue to communicate with the utmost transparency about the cause of the fire and its consequences.

An investigation is ongoing. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers. All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

-----------

23 March 2021 - 16:30 (CET)

Customer information:

  • SBG3 is fully operational, below is the status of the services located in this datacentre:
Datacentre
SBG-3		 Cloud Universe Services Progress
  Public Cloud Public Cloud Instance 86%
  Hosted Private Cloud Hosted Private Cloud Services are being gradually restarted,
depending on their initial configuration,
on an around the clock ( 24/7) rotation schedule.
  Bare Metal Cloud VPS 79%
    Bare Metal 71%

 

The restoration of services is carried out according to a restart schedule room by room, aisle by aisle and rack by rack. As the safety of our technical teams must be guaranteed, this schedule is conditioned by the safety protocol defined for on-site interventions.

  • At the beginning of the week, all customers concerned received a communication on the commercial measures that will be applied to their situation. You can find a dedicated section about this in the FAQ.

Priority 1: Restoring services to our customers

  • Status summary for each datacentre

Datacentre

Situation

Electrical restart

Network restart

Server restart

SBG-1

Data centre not reactivated. After inspection by teams with expertise, technical teams can access the rooms.

 

Switched off as a precaution

Relocation of SBG1 servers to our other datacentres: Gravelines (GRA), Roubaix (RBX), Croix (CRX) and Strasbourg (SBG4) to return to service.
ETA: Wednesday 24 March.

SBG-2

Datacentre switched off and secured.

 

 

Replacement of infrastructure in other datacentres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

SBG-3

Datacentre operational since 18 March

Accomplished

Accomplished

VPS, Bare Metal, and Public Cloud services are currently being restored. Hosted Private Cloud services will be restored gradually.

SBG-4

The datacentre will be operational on Wednesday 24 March.

Power restored this evening 23 March

Network restart  24 March

ETA: 25 March, during the day.

 

  • SBG Service Status Pages - OVHcloud Control Panel

A view per datacentre is available per service in the Control Panel.

Priority 2: Provisioning of infrastructure for our customers in other datacentres

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.

Update on quantities delivered since 10th March:

CLOUD UNIVERSE PRODUCTS   QUANTITY
Bare metal Bare Metal Servers 6103
  NAS HA TB 259
Public Cloud VPS VM 14973
Hosted Private Cloud Host Servers 640
  Datastore Number of zpool 662
    Volume of TB 3764

 

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.

On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres. 

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers and have a support plan in place.

Communications

  • Octave Klaba, founder of OVHcloud, is available on his Twitter account.
  • The OVHcloud Community platform is now available to interact with community members and support you.
  • An FAQ is available here.
  • To follow updates about this situation in real time:

Our mission remains to provide our customers with the highest quality of service and to support their online business and we know how important this is to them. We sincerely apologise for the problems caused by this fire. We will continue to communicate with the utmost transparency about the cause of the fire and its consequences.

An investigation is ongoing. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers. All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

----------

22 March 2021 - 16:30 (CET)

Customer information:

  • SBG3 is fully operational, below is the status of the services located in this datacentre:
Datacentre
SBG-3		 Cloud Universe Services Progress
  Public Cloud Public Cloud Instance Progressive restart of services beginning 22 March
with a schedule spread throughout the entire week, including weekends.
  Hosted Private Cloud Hosted Private Cloud Progressive restart of services from 22 March,
depending on the initial configuration of the services,
schedule spread throughout the entire week, including weekends.
  Bare Metal Cloud VPS 79%
    Bare Metal 55%

 

  • Public Cloud customers have been informed by email of the gradual restart of their services on Monday 22 March.
  • Bare Metal customers (hosted in SBG4) have been informed by email that their services will be restarted from Wednesday 24 March.
  • Real-time monitoring of the restart of SBG by room and rack is available at: vms.status-ovhcloud.com/sbg/.
  • At the beginning of the week, all customers concerned received a communication on the commercial measures that will be applied to their situation. You can find a dedicated section about this in the FAQ.

Priority 1: Restoring services to our customers

  • Status summary for each datacentre

Datacentre

Situation

Electrical restart

Network restart

Server restart

SBG-1

Data centre not reactivated. After inspection by teams with expertise, technical teams can access the rooms.

 

Switched off as a precaution

Relocation of SBG1 servers to our other datacentres: Gravelines (GRA), Roubaix (RBX), Croix (CRX) and Strasbourg (SBG4) to return to service.
ETA: Wednesday 24 March.

SBG-2

Datacentre switched off and secured.

 

 

Replacement of infrastructure in other datacentres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

SBG-3

Datacentre operational since 18 March

Accomplished

Accomplished

VPS and Bare Metal services started to be restored on 21/03. Hosted Private Cloud and Public Cloud services will be gradually restored starting today.

SBG-4

The datacentre will be operational on Wednesday 24 March.

Power restoration planned for the evening of March 24.

Network restart
Wednesday 24 March

ETA: Wednesday 24 March.

 

  • SBG Service Status Pages - OVHcloud Control Panel

A view per datacentre is available per service in the Control Panel.

Priority 2: Provisioning of infrastructure for our customers in other datacentres

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), Londres (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.

Update on quantities delivered since 10th March:

CLOUD UNIVERSE PRODUCTS   QUANTITY
Bare metal Bare Metal Servers 5767
  NAS HA TB 259
Public Cloud VPS VM 14598
Hosted Private Cloud Host Servers 616
  Datastore Number of zpool 747
    Volume of TB 3706

 

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.

On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres. 

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers and have a support plan in place.

Communications

  • Octave Klaba, founder of OVHcloud, has uploaded a new video to his Twitter account today.
  • The OVHcloud Community platform is now available to interact with community members and support you.
  • An FAQ is available here.
  • To follow updates about this situation in real time:

Our mission remains to provide our customers with the highest quality of service and to support their online business and we know how important this is to them. We sincerely apologise for the problems caused by this fire. We will continue to communicate with the utmost transparency about the cause of the fire and its consequences.

An investigation is ongoing. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers. All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

---------

21 March 2021 - 19:30 (CET)

Customer information:

  • In the SBG3 datacentre, the teams are continuing to work around the clock to gradually reactivate the services.
  • Current status of services located in SBG3:
Datacentre
SBG-3		 Cloud Universe Services Progress
  Public Cloud Public Cloud Instance Reboot - 22 March
  Hosted Private Cloud Hosted Private Cloud Reboot - 22 March
  Bare Metal Cloud VPS 68%
    Bare Metal 21%

 

 

  • At the beginning of the week, all customers concerned received a communication on the commercial measures that will be applied to their situation. You can find a dedicated section about this in the FAQ.

Priority 1: restoring services to our customers

  • Status summary for each datacentre

Datacentre

Situation

Electrical restart

Network restart

Server restart

SBG-1

Inspection and expert assessments in progress.

 

 

Switched off as a precaution

Moving servers to other datacentres (Gravelines (GRA), Roubaix (RBX) and Strasbourg (SBG4)) before services are restarted

SBG-2

Datacentre switched off and secured.

 

 

Replacement of infrastructure in other datacentres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

SBG-3

Datacentre operational

Accomplished

Accomplished

Bare Metal VPS services are currently being restored. Hosted Private Cloud and Public Cloud services will be restored gradually from tomorrow (see status above).

SBG-4

After inspection and verification, this datacentre will be back in service on Wednesday 24 March.

Power supply work in progress

Switched off as a precaution

ETA: Wednesday, 24 March.

 

  • SBG Service Status Pages - OVHcloud Control Panel

A view per datacentre is available per service in the Control Panel.

 

 

Priority 2: Provisioning of infrastructure for our customers in other datacentres

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), Londres (LON), Warsaw (WAW), and Francfort (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.

Update on quantities delivered since 10th March:

CLOUD UNIVERSE PRODUCTS   QUANTITY
Bare metal Bare metal Servers 5558
  NAS HA TB 259
Public Cloud VPS VM 14,327
Hosted Private Cloud Host Servers 616
  Datastore Number of zpool 747
    Volume of TB 3706

 

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.

On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres. 

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers and have a support plan in place.

Communications

  • The OVHcloud Community platform is now available to interact with community members and support you.

 

  • An FAQ is available here.
  • To follow updates about this situation in real time:

Our mission remains to provide our customers with the highest quality of service, to support their online business.  and we know how important this is to them. We sincerely apologise any inconveniences related  this situation An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

----------

20 March 2021 - 20:00 (CET)

Customer information:

Yesterday evening, on Friday 19 March at 18:50 CET, our on-site teams detected smoke in an unconnected battery room in SBG1 (see situation report dated 19 March 22:55). The situation has been brought under control and we are adjusting the operations schedule for the weekend as shown below.

Priority 1: restoring services to SBG1, SBG3 and SBG4

1. Service status as of 20 March 2021 - 20:00

  • Real-time monitoring of the restart of SBG by room and rack is available at: vms.status-ovhcloud.com/sbg/
    -   As a precautionary measure, we decided to take the following actions:
    -   The SBG1 datacentre has been switched off and secured.
    -   The SBG2 datacentre has been switched off and secured.
    -   The SBG4 datacentre is operational, but has been switched off and also secured.
    -   The restart scenario is being reviewed this weekend for the SBG1 and SBG4 datacentres.
    -   SBG3 is partially restored.
  • The teams are continuing to work on the SBG3 datacentre from today, while following security measures.

 

  • Status summary for each datacentre

 

Datacentre

Situation

Electrical restart

Network restart

Server restart

SBG-1

Datacentre switched off and secured.

 

 

Switched off as a precaution

Replacement of infrastructure in other datacentres: Gravelines (GRA), Roubaix (RBX) and Strasbourg (SBG4).

SBG-2

Datacentre switched off and secured.

 

 

Replacement of infrastructure in other datacentres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

SBG-3

Datacentre operational

Accomplished

Accomplished

The teams are still working to restart the services.

SBG-4

Datacentre operational but currently switched off and secured as a precautionary measure.

Power supply work in progress

Switched off as a precaution

We are devising a restart scenario, with security as our primary concern.

 

  • SBG Service Status Pages - OVHcloud Control Panel

A view per datacentre is available per service in the Control Panel.

 

2. Commercial actions

  • At the beginning of the week, all customers concerned received a communication on the commercial measures that will be applied to their situation. For customers who have not received the email, a procedure has been indicated in the FAQ.

 

Priority 2: Provisioning of infrastructure for our customers in other datacentres

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), Londres (LON), Warsaw (WAW), and Francfort (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.

Update on quantities delivered since 10th March:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

5250

 

NAS HA

TB

259

Public Cloud

VPS

VM

13,985

Hosted Private Cloud

Host

Servers

612

 

Datastore

Number of zpool

747

 

 

Volume of TB

3702

 

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.

On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres. 

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers, and have set up a support plan for customers who have not yet activated their DRP.

Communications

  • The OVHcloud Community platform is now available to interact with community members and support you.
  • An FAQ is available here
  • To follow updates about this situation in real time:
    • Our main communication tool remains "Task Status "
    • Status videos are available here:

- FR: https://twitter.com/olesovhcom/status/1371908744775360518
- EN: https://twitter.com/olesovhcom/status/1371934994751500288

Our mission remains to provide our customers with the highest quality of service, to support their online business.  and we know how important this is to them. We sincerely apologise any inconveniences related  this situation An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

---------

Roubaix, France | 19 March 2021 - 22:55 (CET)

This evening at 6:50 pm, our teams on site identified smoke in an unconnected battery room at SBG1.

The fire department was immediately contacted. In regards to the situation, OVHcloud's management decided to apply the precautionary principle: the power supply of SBG1 and SBG4 was immediately cut off. Operations on site were temporarily interrupted, all OVHcloud personnel and all partner teams were asked to return home, only the security, fire safety and maintenance personnel remained on site.

Once on site, firefighters identified the source in the battery room, and the incident was contained within minutes. No injuries were reported among the OVHcloud teams or its partners. As a precautionary measure, two security personnel who were affected by the smoke were examined by medical professionals.

The situation is under control and we are adapting the operations schedule for the coming day.

---------

 19 March 2021 - 15:00 (CET)

Customer information:

Priority 1: restoring services to SBG1, SBG3 and SBG4

1. Service status

  • SBG4: the datacentre is fully operational.
  • SBG1: all racks of servers that have been cleaned are operational (60 racks).
  • SBG3:  the first racks in SBG3 have been put in service today.
  • Real-time monitoring of the restart of SBG by room and rack is available at: vms.status-ovhcloud.com/sbg/.

 

  • Status summary for each datacentre

 

Datacentre Situation Electrical restart Network restart Server restart
SBG-1 Datacentre operational Accomplished Accomplished All of the servers in the cleaned racks are operational (60 racks).
SBG-2 Building not operational.
Site secured		     Replacement of infrastructure in other datacentres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)
SBG-3 Datacentre operational Accomplished Accomplished First racks in service today, Friday March 19.
SBG-4 Datacentre operational Accomplished Accomplished 100% of the racks are operational (80 racks).

 

  • SBG Service Status Pages - OVHcloud Control Panel

A view per datacentre is available per service in the Control Panel.

2. Commercial actions

  • At the beginning of the week, all customers concerned received a communication on the commercial measures that will be applied to their situation. For customers who have not received the email, a procedure has been indicated in the FAQ.

 

Priority 2: Provisioning of infrastructure for our customers in other datacentres

 

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), Londres (LON), Warsaw (WAW), and Francfort (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.
  • 430 servers will be produced today.

 

Update on quantities delivered since 10th March:

 

CLOUD UNIVERSE PRODUCTS   QUANTITY
Bare metal Bare metal Servers 4844
  NAS HA TB 259
Public Cloud VPS VM 13519
Hosted Private Cloud Host Servers 538
  Datastore Number of zpool 583
    Volume of TB 3429

 

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.

On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres. 

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers and have set up a support plan for customers who have not yet activated their DRP.

Communications

  • The OVHcloud Community platform is now available to interact with community members and support you.

 

  • An FAQ is available here.
  • To follow updates about this situation in real time:
    • Our main communication tool remains "Task Status"
    • Status videos are available here:

- FR : https://twitter.com/olesovhcom/status/1371908744775360518

- EN : https://twitter.com/olesovhcom/status/1371934994751500288

 

Our mission remains to provide our customers with the highest quality of service, to support their online business and we know how important this is to them. We sincerely apologise any inconveniences related to this situation An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

    ---------

    18 March 2021 - 16:30 (CET)

    Customer information:

    1. Service Status

    • In the OVHcloud Control Panel, each affected customer can view the status of their services located in Strasbourg (SBG) - as detailed in the FAQ.
    • The OVHcloud Community platform is now available to interact with community members and support you.
    • By following the security protocol defined for on-premises operations: the restart will be carried out room by room, aisle by aisle and rack by rack.
    • Procedures for restarting services will be communicated to affected customers as details become available in the coming days.
    • Real-time monitoring of the restart of SBG by room and planned rack is available at: vms.status-ovhcloud.com/sbg/.

    2. Commercial actions

    • At the beginning of the week, all customers concerned received a communication on the commercial measures that will be applied to their situation. For customers who have not received the email, a procedure has been indicated in the FAQ.

    3. Plan of action

    Priority 1: restoring services to SBG1, SBG3 and SBG4

    • Status summary for each datacentre

    Datacentre

    Situation

    Electrical restart

    Network restart

    Server restart

    SBG-1

    Datacentre operational.

     

    Accomplished

    Accomplished

    First racks tested with success.

    Handover to the customer today with visibility of the status in their OVHcloud Customer Control Panel.

    SBG-2

    Building non-operational.

    Site secured.

     

     

    Replacement of infrastructure in other datacentres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

    SBG-3

    Datacentre operational.

     

    Servers undamaged.

     

     

     

    Accomplished

     

    Accomplished

    Test today on one rack and if conclusive provisional estimate for Friday 19 March for a progressive restart of the services.

    SBG-4

    Datacentre operational.

     

    All floors and servers undamaged.

    Accomplished

     

     

    Accomplished

     

    Provisional ETA: Monday, 22 March for gradual restart of services.

     

    • 136 people are mobilised on premises today
      • The team rotation schedule for the coming weekend has been established.
    • Cleaning of site:
      • Cleaning teams are active in SBG1 and SBG3 in all affected areas.
      • Some of the servers in SBG3 require specific cleaning prior to their return to service. We have started this process.

    Priority 2: Provisioning of infrastructure for our customers in other datacentres

    • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), Londres (LON), Warsaw (WAW), and Frankfurt (FRA).
    • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.
    • Update on quantities delivered since 10th March:

    CLOUD UNIVERSE

    PRODUCTS

     

    QUANTITY

    Bare metal

    Bare metal

    Servers

    4519

     

    NAS HA

    TB

    246

    Public Cloud

    VPS

    VM

    12353

    Hosted Private Cloud

    Host

    Servers

    485

     

    Datastore

    Number of zpool

    522

     

     

    Volume of TB

    3067
    • 400 servers will be produced today.

    Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.

    On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres. 

    Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

    • Our Support teams and Technical Account Managers are in contact with our customers and have set up a support plan for customers who have not yet activated their DRP.

    Communications

    • The OVHcloud Community platform is now available to interact with community members and support you.
    • An FAQ is available here.
    • To follow updates about this situation in real time:
      • Our main communication tool remains "Task Status"
      • Status videos are available here:

    - FR: https://twitter.com/olesovhcom/status/1370045226929299461

    - EN: https://twitter.com/olesovhcom/status/1370045891806175232

    Our mission remains to provide our customers with the highest quality of service, to support their online business.  and we know how important this is to them. We sincerely apologise any inconveniences related  this situation An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

    All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

    -------------

    17 March 2021 - 16:30 (CET)

    Customer information:

    1. Service Status

    • Octave Klaba, founder of OVHcloud, has released a new video on the situation in Strasbourg (SBG).
    • In the Control Panel, each affected customer can view the status of their services located in Strasbourg (SBG) - as detailed in the FAQ.
    • The OVHcloud Community platform is now available to interact with community members and support you.

    By following the security protocol defined for on-premises operations: restarting will be carried out room by room, aisle by aisle and rack by rack.

    • Procedures for restarting services will be communicated to affected customers as details become available in the coming days.
    • Real-time monitoring of SBG restart by room and planned rack via: vms.status-ovhcloud.com.
    • Progressive updates to the status page for services and backups in Strasbourg (SBG), from today onwards, accessible via the FAQ.
    • Updating and tracking the percentage of services restarted per datacentre scheduled in the “Task Status” interface in the coming days.
    • A new order form has been set up (for Bare Metal Cloud) to offer concerned customers services similar to those they had at the Strasbourg (SBG) site.

    2. Commercial actions

    • All concerned customers received, at the beginning of the week, a communication on the commercial measures that will be applied to their situation. For customers who have not received the email, a procedure has been indicated in the FAQ.
    • For each customer concerned:
      • A billing suspension notification for the month of March has been added in the Customer Control Panel.
      • Commercial actions have been indicated, the details of which will be communicated this week.

    3. Plan of action

    Priority 1: restoring services to SBG1, SBG3 and SBG4

    • Status summary for each datacentre

    Datacentre

    Situation

    Electrical restart

    Network restart

    Server restart

    SBG-1

    4 of 12 rooms were damaged

     

    - Rooms 61AD/62AD: servers not affected

    - Rooms 61E/62D: cleaning of servers underway

    - Restored power to datacentre

     

    - Backbone: reconnected

    - Redeployment of the local network (Cabling finalised this Wednesday) and configuration today

     

     

    A rack was tested today and if conclusive a provisional estimate of Friday, 19 March for a gradual restart of services after cleaning and drying servers.

     

    SBG-2

    Building out of use Site secured

     

     

    Replacement of infrastructure in other data centres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

    SBG-3

    Servers undamaged

     

     

     

    - Restored power to datacentre

     

     

    - New network power restored

    - Redeployment of the local network this Wednesday 17th March (Cabling finalised).

     

    Test tomorrow on a rack and if conclusive a provisional estimate of Friday, 19 March for a gradual restart of services.

    SBG-4

    All floors and servers were spared

    - Restored power to datacentre

     

    Redeployment of the local network carried out.

    Provisional ETA: Monday, 22 March for gradual restart of services.

     

    • 116 people are mobilised at the premises today
    • Network redeployment: 
      • The SBG1, SBG3 and SBG4 datacentres will be connected to the network tonight.
    • Watercooling:
      • The entire system is operational in the SBG1/SBG3/SBG4 datacentres.
    • Cleaning of site:
      • Intensive mobilisation (24/24).
      • SBG1: finalisation of cleaning Wednesday, 17th March.

    Priority 2: providing infrastructures in other datacentres for our affected customers

    • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), Londres (LON), Warsaw (WAW), and Frankfurt (FRA).
    • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks. Update on quantities delivered since 10th March:

    CLOUD UNIVERSE

    PRODUCTS

     

    QUANTITY

    Bare metal

    Bare metal

    Servers

    4200

     

    NAS HA

    TB

    276

    Public Cloud

    VPS

    VM

    11998

    Hosted Private Cloud

    Host

    Servers

    434

     

    Datastore

    Number of zpool

    466

     

     

    Volume of TB

    2950

     

    • 400 servers will be produced today.

    Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.

    On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres. 

    Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

    • Our Support teams and Technical Account Managers are in contact with our customers and have set up a support plan for customers who have not yet activated their DRP.

    Communications

    - FR: https://twitter.com/olesovhcom/status/1370045226929299461

    - EN: https://twitter.com/olesovhcom/status/1370045891806175232

    Our mission remains to provide our customers with the highest quality of service, to support their online business.  and we know how important this is to them. We sincerely apologise any inconveniences related  this situation An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

    All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

    -------

    16 March 2021 - 16:00 (CET)

    Customer information:

    1.  Service Status

    • Progressive updates to the status page for services and backups in Strasbourg (SBG), from today onwards, accessible via the FAQ.
    • Monitoring of the % of service restarts per datacentre to be displayed in the "Travaux" interface in the coming days. Restarts will be carried out room by room, aisle by aisle and rack by rack.
    • Finalized procedures for restarting services will soon be communicated to customers.

    2. Commercial actions

    • For each customer concerned:
      • A billing suspension notification for the month of March has been added in the Customer Control Panel.
      • Commercial actions have been indicated, the details of which will be communicated this week.

    3. Plan of action

    Priority 1: restoring services to SBG1, SBG3 and SBG4

    • Status summary for each datacentre

    Datacentre

    Situation

    Electrical restart

    Network restart

    Server restart

    SBG-1

    4 of 12 rooms were damaged

     

    - Rooms 61AD/62AD: servers not affected

    - Rooms 61E/62D: cleaning of servers underway

    - Power restored

    - Damaged UPS have been checked and are being replaced

    - Backbone: temporary connection to SBG on 15th March

    - LAN redeployment and configuration today

     

     

    Provisional ETA: Monday, 22 March for gradual restart

    SBG-2

    Building out of use. Structural expertise carried out. Site secured.

     

     

     

     

    Infrastructure replacement in other data centres Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

    SBG-3

    Servers undamaged

     

     

    - Power restored today

    - All inverters are operational

     

     

    - New network room powered up today

    - Redeployment of the local network on 17th March

    Provisional ETA: Monday, 22 March for gradual restart

     

     

    SBG-4

    All floors and servers were spared

    Power restored today

     

    Redeployment of the local network today

    Provisional ETA: Monday, 22 March for gradual restart

     

    • 101 people are mobilised and on premises
    • Network redeployment:
      • SBG connection to the backbone today (alternative routes to Paris and Frankfurt underway)
      • Reconnection of buildings and rooms:
        • Fibre optic redeployment in progress 
        • Configuration underway
    • Watercooling:
      • The entire system is operational.
    • Cleaning of site:
      • Intensive mobilisation (24/24).

    Priority 2: Provisioning of infrastructure for our customers in other datacentres

    • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres: Gravelines (GRA), Roubaix (RBX), Londres (LON), Warsaw (WAW), Francfort (FRA).
    • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks. Update on quantities delivered since 10th March:

    CLOUD UNIVERSE

    PRODUCTS

     

    QUANTITY

    Bare metal

    Bare metal

    Servers

    3939

     

    NAS HA

    TB

    242

    Public Cloud

    VPS

    VM

    10731

    Hosted Private Cloud

    Host

    Servers

    372

     

    Datastore

    Number of zpool

    393

     

     

    Volume of TB

    2704

     

    • 350 servers will be produced today.

    Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

    • Our Support teams and Technical Account Managers are in contact with our customers and have set up a support plan for customers who have not yet activated their DRP.

    Communications

    - FR: https://twitter.com/olesovhcom/status/1370045226929299461

    - EN: https://twitter.com/olesovhcom/status/1370045891806175232

    Our mission remains to provide our customers with the highest quality of service, to support their online business and we know how important this is to them. We sincerely apologise for any inconveniences related to this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

    All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

    -------

    15 March 2021 - 17:00 (CET)

    Customer information:

    1 - Server status

    • The status page for services and backups in Strasbourg (SBG) is now available in the FAQ.
    • For VPS service, a question about “zone vs datacentre (SBG)” has been added for clarification in the FAQ.

    2 - Commercial actions

    • For each customer concerned:
      • A billing suspension notification for the month of March has been added in the Customer Control Panel.
      • Commercial actions have been indicated, the details of which will be communicated this week.

    3 - Action plan

    Priority 1: restoring services to SBG1, SBG3 and SBG4

    • Status summary for each datacentre

     

    Datacentre

    Situation

    Electrical restart

    Network restart

    Server restart

    SBG-1

    4 of 12 rooms were damaged

    Temporarily repowered on 13th March and will be restored permanently on 16th March

    - Backbone: temporary connection to SBG on 15th March

    - Internal network to be redeployed on 16th March

    Provisional ETA: Monday, 22 March

    for gradual restart

    SBG-2

    Building out of use

    Structural expertise carried out.  Site secured

     

     

    Replacement of infrastructure in other datacentres

    SBG-3

    Servers undamaged

    Temporarily repowered on 12th March and will be restored permanently on 16th March

    - New network room to be deployed and powered on 16th March

    - Internal network to be redeployed on 17th March

    Provisional ETA: Monday, 22 March

    for gradual restart

    SBG-4

    Servers undamaged

    Power to be restored on 16th March

    - Internal network to be redeployed on 16th March

    Provisional ETA: Monday, 22 March

    for gradual restart

     

    • 111 people are dedicated on-site to connecting network and electricity units (in rotation, 24/24), for: 
    • Power redeployment:
      • The electrical installations required to restart SBG1/SBG4 have been verified.
      • The high-voltage power supply for SBG3 is deployed and operational.
      • Inverter re-commissioning protocol underway today (verification of all UPS).

     

    • Network redeployment:
      • The SBG1 network room is being tested.
      • The new network room is being completed and will be supplied with electricity as soon as the inverters are verified.
      • The installation of the network equipment will be finalised this evening.
      • The connection of the backbone is scheduled for tomorrow.
      • The local network is being redeployed.

     

    • The cleaning of any soot in the SBG3 server rooms will begin on Tuesday, 16 March. The clean-up teams have arrived on site and are beginning deployed.

     

    Priority 2: providing infrastructures in other datacentres for our affected customers

    • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our Roubaix (RBX) and Gravelines (GRA) datacentres.
    • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.
    • 500 servers will be produced today.

     

    • Update on quantities delivered since 10th March:

    CLOUD UNIVERSE

    PRODUCTS

     

    QUANTITY

    Bare metal

    Bare metal

    Servers

    3658

     

    HA-NAS

    TB

    220

    Public Cloud

    VPS

    VM

    8903

    Hosted Private Cloud

    Host

    Servers

    328

     

    Datastore

    Number of zpool

    361

     

     

    Volume of TB

    2598

     

    Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

    • Our Support teams and Technical Account Managers are in contact with our customers, and have set up a support plan for customers who have not yet activated their DRP.

    Communications

    - FR: https://twitter.com/olesovhcom/status/1370045226929299461

    - EN: https://twitter.com/olesovhcom/status/1370045891806175232

     

    Our mission remains to provide our customers with the highest quality of services to support their online activities, and we know how important this is to them. We sincerely apologise for the problems caused by this incident. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

    All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

    -----------

    14 March 2021 - 15:00 (CET)
     

    Customer information:

    1 - Server status

    • A page has been created listing the backups of services hosted in Strasbourg (SBG) managed by the customer or by OVHcloud along with their status, which will be available in the FAQ within the next few hours.
    • For Public Cloud and VPS customers, a question about “zone vs datacentre (SBG)” has been clarified in the FAQ today.

    2 - Commercial actions

    • All affected customers will be notified about their suspended March bill in their Control Panel.
    • Details on how these measures will be implemented will be sent next week, and will also apply to March bills for impacted services that were already issued before 10th March.

    3 - Action plan:

    Priority 1: restoring services to SBG1, SBG3 and SBG4

    • Status summary for each datacentre

     

    Datacentre

    Situation

    Electrical restart

    Network restart

    Server restart

    SBG-1

    4 of 12 rooms were damaged

    Temporarily repowered on 13th March and will be restored permanently on 17th March

    - Backbone: temporary connection to SBG on 15th March

    - Internal network to be redeployed on 17th March

    Provisional ETA: Monday, 22 March

    for gradual restart

    SBG-2

    Building out of use

     

     

    Replacing infrastructures in other datacentres

    SBG-3

    Servers undamaged

    Temporarily repowered on 12th March and will be restored permanently on 16th March

    - New network room to be deployed and powered on 16th March

    - Internal network to be redeployed on 17th March

    Provisional ETA: Monday, 22 March

    for gradual restart

    SBG-4

    Servers undamaged

    Power will return permanently on 17th March

    Internal network to be redeployed on 17th March

    Provisional ETA: Monday, 22 March

    for gradual restart

     

    • A team of 60 people is working on site to connect mobile network and electricity units (working in rotation, 24 hours a day): 
      • Cabling in progress.
      • Connection planned for tonight.
    • The drying out of the SBG3 buildings will be finished tonight.
    • The cleaning of any soot in the SBG3 server rooms will begin on Tuesday, 16 March.
    • The cooling system is operational :
      • Water cooling is protected from the cold weather (forecast locally on Thursday,  18 March and Friday, 19 March).
      • The pumps are operational.
    • Priority 2: providing infrastructures in other datacentres for our affected customers
       
    • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our Roubaix (RBX) and Gravelines (GRA) datacentres.
    • We are committed to providing around 15,000 new servers in the coming weeks.
    • Update on quantities delivered since 10th March:

     

     

    • Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers
      • Our Support teams and Technical Account Managers are in contact with our customers, and have set up a support plan for customers who have not yet activated their DRP.

    Communications

    - FR: https://twitter.com/olesovhcom/status/1370045226929299461

    - EN: https://twitter.com/olesovhcom/status/1370045891806175232

     

    Our mission remains to provide our customers with the highest quality of services to support their online activities, and we know how important this is to them. We sincerely apologise for the problems caused by this fire. We will continue to communicate with the utmost transparency about the cause of the fire and its consequences.

    An investigation is ongoing. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

    All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

    -------

    Roubaix, France | 13 March 2021 - 16:00 (CET)
     

    Customer information:

    1 - Server Status

    • We have sent emails to all our customers informing them of the status of their services.

    2 - Commercial actions

    • Billing has been suspended for Strasbourg (SBG) datacentre customers from the date of the incident.
    • Commercial measures have been set out to each affected customer.
    • Details on how these measures will be implemented will be sent next week, and will apply in particular to March bills for impacted services that were already issued before 10th March.
    • We have updated the FAQs.

     Our three priorities are as follows:

    • Priority 1: restoring services to SBG1, SBG3, and SBG4
    • Priority 2: providing infrastructures in other datacentres for our affected customers
    • Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

    Action plan:

    Priority #1:

    • Status summary for each datacentre

    Datacentre

    Situation

    Electrical restart

    Server restart

    SBG-1

    4 of 12 rooms were damaged

    From Monday, 15 March

    Progressive restoration of undamaged servers by Monday, 22 March

    SBG-2

    Building out of use

    Audit and inventory of premises

    Replacing infrastructures in other datacentres

    SBG-3

    Servers undamaged

    Tests for the high voltage supply will be carried out this weekend.  Provisional restart on 15th March

    A progressive restart of all services is estimated from Monday, 22 March.

     

    SBG-4

    Servers undamaged

    Scheduled for Monday, 15 March

    A progressive restart of all services is estimated from Monday, 22 March.

     

     

    • Mobile high-voltage network and electricity units arrived at the Strasbourg site today at 12pm.
    • 60 experts are currently working on site (in rotation over the weekend) in four teams, with the following missions:

    Team 1 (network) is tasked with connecting the mobile network unit and testing the network room in SBG1

    Team 2 (electricity) is working on high-voltage electricity connections

    Team 3 (electricity) is working on low-voltage electricity connections

    Team 4 is working on protecting and restoring the cooling systems

    The aim is to reconnect, repower and restart the backbone access network from Monday, 15 March (following the authorisation of the ESR (Électricité de Strasbourg Réseaux)).

    To repower the site:

    • Two generators have been rented since 11th March to power the temporary office.
    • One additional generator was rented to recharge the batteries of our generators following their activation during the incident.

     

    • Several service providers are currently working on site (30 people)
      • The water on the floor of the SBG3 building has been cleaned up
      • The SBG3 building is currently being dried out, which should be completed by Sunday evening
      • The cleaning of any soot in the server rooms of SBG3 will begin on Monday.
         
    • Repair of the damaged cooling system:
      • A crane has arrived on site.

    Priority #2:

    • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our Roubaix (RBX) and Gravelines (GRA) datacentres.
       
    • We are committed to providing around 15,000 new servers in the coming weeks.
       
    • Update on quantities delivered since 10th March:

    Priority #3:

    • Backup mechanisms:
      • An overview of all backup mechanisms will be published in the FAQ today.
      • For Public Cloud and VPS customers, questions about “zone vs datacentre (SBG)” will be clarified in the FAQ today.
      • An ad hoc mechanism is in place to help our customers set up their DRP system.

    Communications

    - FR: https://twitter.com/olesovhcom/status/1370045226929299461

    - EN: https://twitter.com/olesovhcom/status/1370045891806175232

    Our mission remains to provide our customers with the highest quality of service, to support their online business. And we know how important this is to them. We sincerely apologise for the problems caused by this fire. We will continue to communicate with the utmost transparency about the cause of the fire and its consequences.

    An investigation is ongoing. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

    All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

    -------------

    12/03/2021 - 14:00 (CET)

    Customer Information:

    • Octave Klaba, founder of OVHcloud, has uploaded videos providing information about the situation.

    FR: https://twitter.com/olesovhcom/status/1370045226929299461

    EN: https://twitter.com/olesovhcom/status/1370045891806175232

    • Today, we will inform our customers by email about the status of their services. We also have a 24/7 support team in place to support them.

    As a reminder, our three priorities are as follows:

    • Priority 1: restoring services to SBG1,SBG3, and SBG4
    • Priority 2: providing our customers with infrastructures in our non-affected datacentres
    • Priority 3:  Implementation of all DRP (Disaster Recovery Plan) mechanisms with our customers.

    Priority 1: restoring services to SBG1, SBG3 and SBG4

    • Diagnostics and inventory are ongoing.
    • To guarantee the safety of our teams, cleaning and repairs to the damaged buildings are in progress.
    • The tasks of cleaning the foam and the removal of water on the public road have been carried out. The supply tank of the hydraulic system has been drained.
    • All long-distance fibre connections are operational.
    • SBG1 was partially damaged (4 out of 12 rooms):
      • The network room was checked yesterday.  An isolated electrical test run, without connectivity, is planned for this weekend.  We estimate an orderly restart of connectivity early next week.
      • Reactivation of the electrical network is tentatively planned for  Monday, 15 March.
      • Servers will be audited and reactivated one by one.
    • SBG3 was spared from the fire
      • A network room will be set up by the end of this week, in order to be operational at the beginning of the week of 15th March.
      • Tests for the high voltage power supply will be carried out this weekend. Provisional estimate for the reestablishment of power: Monday, March 15.
      • A progressive restart of all services is estimated beginning Monday, 22 March.
    • SBG4 was spared from the fire
      • We will finalise our service reactivation plan by the end of this week.

    All affected Email Pro services began being restored on Thursday, 11 March, starting at 4:55 p.m. No email was lost.

    Priority 2: providing our customers with infrastructures in our non-affected datacentres.

    • For customers that have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our Roubaix (RBX) and Gravelines (GRA) datacentres.
    • We have already delivered to our impacted customers:
      • Wednesday, 10 March: 2100 new servers.
      • Thursday, 11 March: 630 new servers.
    • The new organisation of our assembly lines has been operational since the morning of Friday, 12 March and has tripled our production capacity (3,000 to 4,000 servers per week).
    • We are committed to producing nearly 15,000 new servers in the coming weeks.
    • Delivery times for the new services will be communicated as soon as possible.

    Priority 3:  Implementation of all DRP (Disaster Recovery Plan) mechanisms with our customers.

    • By the end of the week, our support teams and Technical Account Managers will define a support plan for our customers which have not yet activated their DRP.

    Communications

    - FR: https://twitter.com/olesovhcom/status/1370045226929299461

    - EN: https://twitter.com/olesovhcom/status/1370045891806175232

    Our mission remains to provide our customers with the highest quality of service and  to support their online business and we know how important this is to them. We sincerely apologise for the problems caused by this fire. We will continue to communicate with the utmost transparency about the cause of the fire and its consequences.

    An investigation is ongoing. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

    All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

    ---------

    11/03/2021 – 15:30 (CET)

    People :

    • HR teams arrived on site today to support the teams.
    • Additional industrial and technical teams arrived from other offices and datacenters to reinforce the on-site staff and provide specific expertise.
    • Logistic and food services have been provided to all on-site teams.
    • A temporary office has been established.
    • We’ve reinforced the staff on the French production site (Croix) with internal volunteers to address the current production plan and additional workload caused by the Strasbourg incident.
    • The production will be carried out in extended working hours and during the weekends.

    Production and Datacenters :

    • We’ve started to propose replacement infrastructure (Dedicated servers, Hosted Private Cloud, and Public Cloud) in Roubaix and Graveline datacenters to our customers.
    • To address the demand, additional assembly lines will be in place in the next 48 hours. This will triple our production capacity.

    On the Strasbourg site:

    • Diagnostic and inventory currently in progress.
    • We are in the process of clearing up and repairing the damaged buildings to guaranty the safe working environment for our teams.
    • We have finished removing the extinguishing foam and water from the street. We will remove the water from the water tank today.
    • SBG2 will need to be almost entirely reconstructed.
    • SBG1 was heavily damaged.
      • Following a preliminary audit, Network room SBG1 will be restored at the beginning of next week.
      • The provisional date for restoring the power supply is Monday, March 15th.
      • In the coming days, the servers will be activated room by room after the audit.
    • SBG4: preliminary audits did not reveal issues. Our ambition now is to restore power during week 12 and then gradually reactivate all services.
    • SBG3 was not impacted by the fire. We aim to restore the power and network during week 12 and then gradually reactivate all services.

    Our three priorities remain as follows:

    Reserve infrastructures at our other datacentres for our affected customers: we have a stock of new servers at the Roubaix and Gravelines sites, ready to be delivered to the majority of affected customers. We will further enhance availability in these datacentres, with the production of nearly 10,000 new servers in the coming weeks. Affected customers will be notified about this process as soon as possible.

    Secure the site now that we have regained access, clean it up, and reconnect the electricity and the network for the three affected datacentres.

    Continue to assess the impact on our customers’ servers at the affected datacentres, in order to find the best solutions and communicate to all customers
     

    Additional actions taken by OVHcloud:

    Network:

    • All fiber paths that enter the datacenter site are operational.
    • We have placed orders with different suppliers – due date for delivery on-site, March 11th.

    Communication:

    Our mission is to provide our customers with the highest quality of services to support their online activities and we know how important this is to them. We sincerely apologise for the issues caused by this fire. We will continue to communicate with the greatest transparency about the cause of the fire and its consequences.

    An investigation is ongoing. Regardless of the cause of the incident, OVHcloud is currently evaluating the potential technical and operational measures aimed to bring solutions to the impacted customers.

    An FAQ is already available here.

    ---------------

    10/03/2021 – 21:00 (CET)

    Summary:

    • At 00:47 on Wednesday 10 March 2021, a fire broke out in a room at one of our four OVHcloud datacentres in Strasbourg (SBG2). 
    • The fire was contained by the early hours of the morning.
    • There are no injuries.
    • The fire mostly destroyed the SBG2 datacentre and partially damaged the SBG1 datacentre (4 of the 12 rooms destroyed). The two other OVHcloud datacentres in Strasbourg were not affected by the fire; SBG3 and SBG4 servers are currently switched off but undamaged.
    • The site is not classified as a Seveso site. 
    • The cause of the fire has yet to be established and an investigation has been launched as mandated by the authorities.

    Actions taken by OVHcloud:

    • The technical and commercial teams have been working since this morning to inform our customers and handle the unavailability of our Strasbourg site.
    • The company’s founder, Octave Klaba, has been on site since this morning with the industrial and technical teams.
    • To follow updates to this situation in real time:
    • Incident tracking platform: https://network.status-ovhcloud.com/incidents/vlcqgm66ffnz
    • Twitter: https://twitter.com/ovhcloud
    • A customer announcement, along with an FAQ, will be available tomorrow morning.
    • To help us handle customer requests, we recommend using the ticket feature on our website or consulting our help centre.

    Our three priorities are as follows:

    Reserve infrastructures at our other datacentres for our affected customers: we have a stock of new servers at the Roubaix and Gravelines sites, ready to be delivered to the majority of affected customers. We will further enhance availability in these datacentres, with the production of nearly 10,000 new servers in the coming weeks. Affected customers will be notified about this process as soon as possible.

    Secure the site now that we have regained access, clean it up, and reconnect the electricity and the network for the three affected datacentres.

    Continue to assess the impact on our customers’ servers at the affected datacentres, in order to find the best solutions.

    We are doing everything we can to ensure a continuity of service to our customers:

    • We are working on a plan to relaunch the three unaffected datacentres (SBG3 and SBG4), the partially affected datacentre (SBG1), as well as our network, as quickly as possible.
    • We ask that our customers exercise caution around the emails they receive: in times of crisis, it is common for malicious activity (phishing, spam, etc.) to increase. It is more important than ever to stay alert. 

    Impact on our operation:

    • We are continuing to assess the impact of this incident, particularly for the customers whose data was located in the datacentre destroyed by the fire.
    • Our Web-based VOIP services in France were not affected.
    • All of our services in our other France-based datacentres and across the world (including 15 datacentres in Europe) are fully operational.

    Our mission is to provide our customers with the highest quality of services to support their online activities and we know how important this is to them. We sincerely apologise for the issues caused by this fire. We will continue to communicate with the greatest transparency about the cause of the fire and its consequences.

    We are assessing the environmental impact by working with the relevant authorities on a procedure to confirm that no pollution was caused.

    At this stage, we can confirm that the local residents are not at any risk.

    We are continuously assessing the impact of this incident and will communicate as transparently as possible on the progress of our analyses and the solutions to be implemented.

    All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

    (Incident tracking platform: https://network.status-ovhcloud.com/incidents/vlcqgm66ffnz)

    -------------

    10/03/2021 – 11:00 (CET)

    At 00:47 on Wednesday, March 10, 2021, a fire broke out in a room in one of our 4 datacenters in Strasbourg, SBG2. Please note that the site is not classified as a Seveso site.  Firefighters immediately intervened to protect our teams and prevent the spread of the fire.

    At 2:54 am they isolated the site and closed off its perimeter.  By 4:09 am, the fire had destroyed SBG2 and continued to present risks to the nearby datacenters until the fire brigade brought the fire under control. From 5:30 am, the site has been unavailable to our teams for obvious security reasons, under the direction of the prefecture. The fire is now contained. We are relieved that no one was injured, neither among our teams nor among the firefighters and the services of the prefecture, whom we thank for their exemplary mobilization at our side.

    With our 15 datacenters in Europe, our technical and commercial teams are fully invested to support our customers, implement solutions and mitigate the impact of the unavailability of our Strasbourg site. Our mission is to provide our customers with the highest quality of services to support their online activities and we know how important this is to them. We sincerely apologize for the difficulties and challenges caused by this fire. We are committed to continue to communicate with the greatest transparency about the cause of the fire and its consequences.

    We are currently assessing the impact of this incident and will communicate as transparently as possible on the progress of our analyses and the implementation of solutions. All of our communication channels, starting with our incident tracking platform (www.status-ovhcloud.com), are accessible so that you can stay informed of developments in real-time.