• Visitors can check out the Forum FAQ by clicking this link. You have to register before you can post: click the REGISTER link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. View our Forum Privacy Policy.

test please delete

Collapse
This is a sticky topic.
X
X
  •  
  • Filter
  • Time
  • Show
Clear All
new posts

    Originally posted by NickFitz View Post
    FFS. Big barney with the other developer, because he thinks we should fetch the entire contents of a database table into memory, manipulate it in application code, and end up throwing away the 98% of it that we never needed in the first place. I think we should use (very simple) SQL queries that fetch precisely the data needed and no more, then not have to write all the code to do the manipulations in the first place

    Apparently this is my fault because I failed to raise it when reviewing some of his code several weeks ago. As I've pointed out, I assumed that the unnecessary complexity was an artefact of having nothing but flat files to work with then, and that we'd get rid of all that once we had access to the database and could offload the work. But he's convinced that his over-engineered, over-generalised code shouldn't be changed, and that avoiding placing unnecessary load on the database is old-fashioned and outdated

    Now the project manager is having to find somebody from another team to come in and adjudicate, even though I have working code (still using his thing, just bypassing most of it) that does exactly what's required

    I suggested somebody from the team that's responsible for the database, as I reckon they'll automatically come down on the side that minimises the load on their kit
    returning all the data will load the database. SQL server is designed to filter data down to a small subset. How is he dealing with security? if you want to secure it do it in sql once its out of there you have no control.
    "If you didn't do anything that wasn't good for you it would be a very dull life. What are you gonna do? Everything that is pleasant in life is dangerous."

    I want to see the hand of history on his collar.

    Comment


      Originally posted by NickFitz View Post
      FFS. Big barney with the other developer, because he thinks we should fetch the entire contents of a database table into memory, manipulate it in application code, and end up throwing away the 98% of it that we never needed in the first place.
      He's a moron. I don't know how things are in your technical environment, but the SAP mantra is: do everything you can on the database server, always minimise all data transrers to the application servers. Or, simply, don't be a moron.

      However, i can propose a solution that should suit you both. Switch to an in memory database!

      Down with racism. Long live miscegenation!

      Comment


        Originally posted by vetran View Post

        returning all the data will load the database. SQL server is designed to filter data down to a small subset. How is he dealing with security? if you want to secure it do it in sql once its out of there you have no control.
        Apparently that may have been the case fifteen years ago, but he informed us that nowadays it isn't a problem

        Originally posted by NotAllThere View Post
        He's a moron. I don't know how things are in your technical environment, but the SAP mantra is: do everything you can on the database server, always minimise all data transrers to the application servers. Or, simply, don't be a moron.

        However, i can propose a solution that should suit you both. Switch to an in memory database!
        Departmental requirements are that all data gathered from various sources and used for generating reports sits in this big PostgreSQL server, and all reports are generated therefrom. I've no doubt it could handle the excess load of fetching whole tables given that our data isn't even very big; but I also have no doubt that the DB developers (one of whom is being called in to decide) will see his "SELECT *" without any kind of WHERE or LIMIT and scream in horror

        It's all hosted on AWS but AFAIK the DB and application servers are both in eu-west-2 (London), so I can't throw Amazon's egregious prices for cross-region data transfer into the equation

        Comment


          Can I just say that I wish to find anyone and everyone who designed BACS and take them out the back and shoot the lot of them.

          Why does every bank have multiple different formats none of which are identical and most contradict one another to the extent the AI has given up...
          Last edited by eek; 5 August 2021, 11:03.
          merely at clientco for the entertainment

          Comment


            Originally posted by NickFitz View Post

            Apparently that may have been the case fifteen years ago, but he informed us that nowadays it isn't a problem



            Departmental requirements are that all data gathered from various sources and used for generating reports sits in this big PostgreSQL server, and all reports are generated therefrom. I've no doubt it could handle the excess load of fetching whole tables given that our data isn't even very big; but I also have no doubt that the DB developers (one of whom is being called in to decide) will see his "SELECT *" without any kind of WHERE or LIMIT and scream in horror

            It's all hosted on AWS but AFAIK the DB and application servers are both in eu-west-2 (London), so I can't throw Amazon's egregious prices for cross-region data transfer into the equation
            Do you have any indexes on those database tables at the moment as if the database is having to do full table scans it may as well chuck all the data at the other server (although the transfer times even over fibre will add to processing times).
            merely at clientco for the entertainment

            Comment


              Originally posted by eek View Post

              Do you have any indexes on those database tables at the moment as if the database is having to do full table scans it may as well chuck all the data at the other server (although the transfer times even over fibre will add to processing times).
              It's all heavily indexed - I believe they even go to some trouble to add indexes to things like date fields automatically, just in case whoever owns the data forgets

              The meeting was had, and the DB guy said he wasn't too fussed about the load given that we only have maybe a few thousand rows and they have tables that are literally billions of rows long. But he came down very much in favour of using SQL properly, as apart from loading issues, they also have a general policy of doing as much in SQL as possible, they're training people who need it in using SQL, and every single other tool and application they've created for this data warehouse is based around SQL

              There was then some kerfuffle about whether the other dev would have to rewrite two weeks' work to use SQL instead of Pandas, until I pointed out that there was no need - his existing code isn't going to stop working because we move to my preferred approach in future

              TBH it would take about ten minutes to make the change, as the SQL in question wouldn't be at all complex. But that made everybody happy.

              Comment


                Lunch has been a bowl of last night's Scotch broth with an end of sesame bloomer. The soup has developed very nicely in the fridge overnight

                So that's one less I have to try and fit into the freezer, which is full to bursting again

                Comment


                  Rain's here

                  Comment


                    Originally posted by NickFitz View Post
                    Rain's here
                    Been and gone here but expected to return.

                    Comment


                      Originally posted by NickFitz View Post

                      Apparently that may have been the case fifteen years ago, but he informed us that nowadays it isn't a problem



                      Departmental requirements are that all data gathered from various sources and used for generating reports sits in this big PostgreSQL server, and all reports are generated therefrom. I've no doubt it could handle the excess load of fetching whole tables given that our data isn't even very big; but I also have no doubt that the DB developers (one of whom is being called in to decide) will see his "SELECT *" without any kind of WHERE or LIMIT and scream in horror

                      It's all hosted on AWS but AFAIK the DB and application servers are both in eu-west-2 (London), so I can't throw Amazon's egregious prices for cross-region data transfer into the equation

                      The thing people seem to have a real problem with is that the cloud is someone else's computer connected via your probably limited internet pipe.

                      Key things are it may be a big computer but they charge per cycle. The price rises will be like boiling lobsters, raise the temperature slowly enough and the Lobster doesn't realise he is for lunch.

                      Easy way to avoid it is to use so few cycles they forget you are in the bucket.

                      Also security frequently stops at the server because once its out people export it to excel and send it out without a clue.

                      And of course you want to do it properly so no one blames you later.



                      "If you didn't do anything that wasn't good for you it would be a very dull life. What are you gonna do? Everything that is pleasant in life is dangerous."

                      I want to see the hand of history on his collar.

                      Comment

                      Working...
                      X