• Visitors can check out the Forum FAQ by clicking this link. You have to register before you can post: click the REGISTER link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. View our Forum Privacy Policy.
  • Want to receive the latest contracting news and advice straight to your inbox? Sign up to the ContractorUK newsletter here. Every sign up will also be entered into a draw to WIN £100 Amazon vouchers!

UniSuper had its entire account WIPED OUT at Google Cloud

Collapse
X
  •  
  • Filter
  • Time
  • Show
Clear All
new posts

    UniSuper had its entire account WIPED OUT at Google Cloud

    https://x.com/WallStreetSilv/status/1792193498600669470

    #2
    That's the problem with SES* services.

    *Someone Else's Servers - AKA 'cloud'

    Comment


      #3
      Originally posted by sadkingbilly View Post
      That's the problem with SES* services.

      *Someone Else's Servers - AKA 'cloud'
      More likely an issue with poor security and unprotected backups.
      Always forgive your enemies; nothing annoys them so much.

      Comment


        #4
        Originally posted by vetran View Post

        More likely an issue with poor security and unprotected backups.
        my guess would be that someone deleted their accounts by mistake, and that update rippled out to all the standby instances...

        Comment


          #5
          Originally posted by vetran View Post

          More likely an issue with poor security and unprotected backups.
          yeh, Someone Else's Security and backup

          Comment


            #6
            Originally posted by vetran View Post
            More likely an issue with poor security and unprotected backups.
            If you click the link from the original post, the first reply is a joint statement from UniSuper and Google, which is also posted on the UniSuper website:
            A joint statement from UniSuper and Google Cloud | UniSuper

            Google Cloud CEO, Thomas Kurian has confirmed that the disruption arose from an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription.

            This is an isolated, ‘one-of-a-kind occurrence’ that has never before occurred with any of Google Cloud’s clients globally. This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again.

            UniSuper had duplication in two geographies as a protection against outages and loss. However, when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies.
            There's some analysis/speculation here:
            What went wrong with UniSuper and Google Cloud? – Daniel Compton
            “Unprecedented” Google Cloud event wipes out customer account and its backups | Ars Technica

            Going back to the joint statement, this line is maybe the most important:
            UniSuper had backups in place with an additional service provider. These backups have minimised data loss, and significantly improved the ability of UniSuper and Google Cloud to complete the restoration.
            I.e. if you're using cloud services, don't rely on one company exclusively.

            Comment


              #7
              Following up on the joint statement, Google have now published their root cause analysis:
              Details of Google Cloud GCVE incident | Google Cloud Blog

              During the initial deployment of a Google Cloud VMware Engine (GCVE) Private Cloud for the customer using an internal tool, there was an inadvertent misconfiguration of the GCVE service by Google operators due to leaving a parameter blank. This had the unintended and then unknown consequence of defaulting the customer’s GCVE Private Cloud to a fixed term, with automatic deletion at the end of that period. The incident trigger and the downstream system behavior have both been corrected to ensure that this cannot happen again.
              (My emphasis.)

              Comment


                #8
                Maybe a benefit to contractors. The highest paid contract I ever had was to replace some code in a project I had worked on before that had somehow been deleted.

                PS No it wasn't me that did it.
                bloggoth

                If everything isn't black and white, I say, 'Why the hell not?'
                John Wayne (My guru, not to be confused with my beloved prophet Jeremy Clarkson)

                Comment

                Working...
                X