• Visitors can check out the Forum FAQ by clicking this link. You have to register before you can post: click the REGISTER link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. View our Forum Privacy Policy.
  • Want to receive the latest contracting news and advice straight to your inbox? Sign up to the ContractorUK newsletter here. Every sign up will also be entered into a draw to WIN £100 Amazon vouchers!

test please delete

Collapse
This is a sticky topic.
X
X
  •  
  • Filter
  • Time
  • Show
Clear All
new posts

    Right, back to this script to fix some old data that got messed up by Amazon's Database Migration Service for no good reason

    Comment


      Originally posted by NickFitz View Post
      Right, back to this script to fix some old data that got messed up by Amazon's Database Migration Service for no good reason
      So, DMS changed Postgres's JSON fields to character fields and truncated them, rendering them useless. This wasn't a major problem as the data had already been extracted from the JSON and stored in various tables as required; but it seemed like a shame.

      Then I remembered that the original JSON field in the old server I had running as localhost on an EC2 machine was still there.

      So I've written a script to fix the RDS database.

      Starting point, about ten minutes ago:

      Code:
          # Tweets: 1,861,797 on RDS, 1,585,739 on EC2
          # Placeholder JSON: 1,585,582 on RDS, 0 on EC2
          # NULL JSON: 0 on RDS, 10 on EC2
          # Tweets to update: 1,585,572
      Current status:

      Code:
      Placeholder JSON: 1,486,963 on RDS
      So it looks like it's managing to fix about 10,000 tweets a minute, which isn't too bad under the circumstances (small EC2 server updating a small RDS server somewhere else in AWS's massive data centres)

      Comment


        The hotpot should be ready before long. Looking forward to that

        Comment


          TFBSZ
          …Maybe we ain’t that young anymore

          Comment


            Originally posted by NickFitz View Post
            The hotpot should be ready before long. Looking forward to that
            And very nice it was

            Internet connection is playing up again. The bloke said it might be unreliable for a while yet, but it's annoying that it's been fine all day then struggles to cope in the evening. More people at home streaming video and that, I suppose

            Comment


              fine fish & chips procured by Miss V senior from work apparently no world leaders visited her work tonight.

              S&K pie & Cod + chips for me.
              Cod & chips for Mrs V (plus a bit of sausage)
              Fishcake & chips for middle V
              Battered Sausage & chips for micro V.


              Yum!
              Always forgive your enemies; nothing annoys them so much.

              Comment


                Originally posted by BR14 View Post
                not vivaldi then? - i used to like vivaldi, until i had to deal with service desks. i wouldn't mind if they played the whole thing, but they loop a random piece. - philistines!

                Mind, you, one lot in ireland, put me off van morrison for a while, too
                indeed we need a curated sample "the best of Viv" for on hold action.

                Now being put off Brown eyed girl is a sin!
                Always forgive your enemies; nothing annoys them so much.

                Comment


                  The script has done its thing. There were ten tweets in the database from before I added the field to save the raw JSON which had been deleted by their owners before I ran the script to fill in those blanks, so they were never going to be recovered; but the script ran to its conclusion, and now the "production" database has only those ten rows left with the placeholder JSON

                  Comment


                    Tonight's viewing opened with Shot Through the Heart (TV Movie 1998), set in the siege of Sarajevo. I've seen it before, late one night around the end of the last century, and had been keeping an eye out for it since then. It's quite hard to find these days: I got a DVD of the German version from a company over there (via Amazon) which has the original English soundtrack as an option on its menus. It deserves to be more widely available, because it's a brilliant, upsetting, and deeply moving film. It's disturbing to see how quickly normal life in a civilised, highly developed community can degenerate into siege warfare, which is what happened less than 24 hours drive away from where I live, according to Google Maps

                    To follow: V for Vendetta (2005), which is kind of the opposite extreme in terms of the political situation portrayed. I once recorded this when it was on the telly, a few years ago now, but I wasn't in the mood for it and turned it off after the first few minutes. I seem to have been in the mood for it this time, because I thought it was absolutely bloody phenomenal

                    Goodnight all

                    Comment


                      Morning all sunny day out, time to do some gardening.
                      …Maybe we ain’t that young anymore

                      Comment

                      Working...
                      X