test please delete

[vetran]

FFS. Big barney with the other developer, because he thinks we should fetch the entire contents of a database table into memory, manipulate it in application code, and end up throwing away the 98% of it that we never needed in the first place. I think we should use (very simple) SQL queries that fetch precisely the data needed and no more, then not have to write all the code to do the manipulations in the first place

Apparently this is my fault because I failed to raise it when reviewing some of his code several weeks ago. As I've pointed out, I assumed that the unnecessary complexity was an artefact of having nothing but flat files to work with then, and that we'd get rid of all that once we had access to the database and could offload the work. But he's convinced that his over-engineered, over-generalised code shouldn't be changed, and that avoiding placing unnecessary load on the database is old-fashioned and outdated

Now the project manager is having to find somebody from another team to come in and adjudicate, even though I have working code (still using his thing, just bypassing most of it) that does exactly what's required

I suggested somebody from the team that's responsible for the database, as I reckon they'll automatically come down on the side that minimises the load on their kit

returning all the data will load the database. SQL server is designed to filter data down to a small subset. How is he dealing with security? if you want to secure it do it in sql once its out of there you have no control.

[NotAllThere]

FFS. Big barney with the other developer, because he thinks we should fetch the entire contents of a database table into memory, manipulate it in application code, and end up throwing away the 98% of it that we never needed in the first place.

He's a moron. I don't know how things are in your technical environment, but the SAP mantra is: do everything you can on the database server, always minimise all data transrers to the application servers. Or, simply, don't be a moron.

However, i can propose a solution that should suit you both. Switch to an in memory database!

[NickFitz]

returning all the data will load the database. SQL server is designed to filter data down to a small subset. How is he dealing with security? if you want to secure it do it in sql once its out of there you have no control.

Apparently that may have been the case fifteen years ago, but he informed us that nowadays it isn't a problem

He's a moron. I don't know how things are in your technical environment, but the SAP mantra is: do everything you can on the database server, always minimise all data transrers to the application servers. Or, simply, don't be a moron.

However, i can propose a solution that should suit you both. Switch to an in memory database!

Departmental requirements are that all data gathered from various sources and used for generating reports sits in this big PostgreSQL server, and all reports are generated therefrom. I've no doubt it could handle the excess load of fetching whole tables given that our data isn't even very big; but I also have no doubt that the DB developers (one of whom is being called in to decide) will see his "SELECT *" without any kind of WHERE or LIMIT and scream in horror

It's all hosted on AWS but AFAIK the DB and application servers are both in eu-west-2 (London), so I can't throw Amazon's egregious prices for cross-region data transfer into the equation

[eek]

Can I just say that I wish to find anyone and everyone who designed BACS and take them out the back and shoot the lot of them.

Why does every bank have multiple different formats none of which are identical and most contradict one another to the extent the AI has given up...

[eek]

Apparently that may have been the case fifteen years ago, but he informed us that nowadays it isn't a problem



Departmental requirements are that all data gathered from various sources and used for generating reports sits in this big PostgreSQL server, and all reports are generated therefrom. I've no doubt it could handle the excess load of fetching whole tables given that our data isn't even very big; but I also have no doubt that the DB developers (one of whom is being called in to decide) will see his "SELECT *" without any kind of WHERE or LIMIT and scream in horror

It's all hosted on AWS but AFAIK the DB and application servers are both in eu-west-2 (London), so I can't throw Amazon's egregious prices for cross-region data transfer into the equation

Do you have any indexes on those database tables at the moment as if the database is having to do full table scans it may as well chuck all the data at the other server (although the transfer times even over fibre will add to processing times).

[NickFitz]

Do you have any indexes on those database tables at the moment as if the database is having to do full table scans it may as well chuck all the data at the other server (although the transfer times even over fibre will add to processing times).

It's all heavily indexed - I believe they even go to some trouble to add indexes to things like date fields automatically, just in case whoever owns the data forgets

The meeting was had, and the DB guy said he wasn't too fussed about the load given that we only have maybe a few thousand rows and they have tables that are literally billions of rows long. But he came down very much in favour of using SQL properly, as apart from loading issues, they also have a general policy of doing as much in SQL as possible, they're training people who need it in using SQL, and every single other tool and application they've created for this data warehouse is based around SQL

There was then some kerfuffle about whether the other dev would have to rewrite two weeks' work to use SQL instead of Pandas, until I pointed out that there was no need - his existing code isn't going to stop working because we move to my preferred approach in future

TBH it would take about ten minutes to make the change, as the SQL in question wouldn't be at all complex. But that made everybody happy.

[NickFitz]

Lunch has been a bowl of last night's Scotch broth with an end of sesame bloomer. The soup has developed very nicely in the fridge overnight

So that's one less I have to try and fit into the freezer, which is full to bursting again

[NickFitz]

Rain's here

[ladymuck]

Rain's here

Been and gone here but expected to return.

[vetran]

Apparently that may have been the case fifteen years ago, but he informed us that nowadays it isn't a problem



Departmental requirements are that all data gathered from various sources and used for generating reports sits in this big PostgreSQL server, and all reports are generated therefrom. I've no doubt it could handle the excess load of fetching whole tables given that our data isn't even very big; but I also have no doubt that the DB developers (one of whom is being called in to decide) will see his "SELECT *" without any kind of WHERE or LIMIT and scream in horror

It's all hosted on AWS but AFAIK the DB and application servers are both in eu-west-2 (London), so I can't throw Amazon's egregious prices for cross-region data transfer into the equation

The thing people seem to have a real problem with is that the cloud is someone else's computer connected via your probably limited internet pipe.

Key things are it may be a big computer but they charge per cycle. The price rises will be like boiling lobsters, raise the temperature slowly enough and the Lobster doesn't realise he is for lunch.

Easy way to avoid it is to use so few cycles they forget you are in the bucket.

Also security frequently stops at the server because once its out people export it to excel and send it out without a clue.

And of course you want to do it properly so no one blames you later.