Originally posted by NickFitz
View Post
- Visitors can check out the Forum FAQ by clicking this link. You have to register before you can post: click the REGISTER link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. View our Forum Privacy Policy.
- Want to receive the latest contracting news and advice straight to your inbox? Sign up to the ContractorUK newsletter here. Every sign up will also be entered into a draw to WIN £100 Amazon vouchers!
test please delete
Collapse
This is a sticky topic.
X
X
Collapse
-
Always forgive your enemies; nothing annoys them so much. -
Originally posted by NickFitz View PostFFS. Big barney with the other developer, because he thinks we should fetch the entire contents of a database table into memory, manipulate it in application code, and end up throwing away the 98% of it that we never needed in the first place.
However, i can propose a solution that should suit you both. Switch to an in memory database!
Down with racism. Long live miscegenation!Comment
-
Originally posted by vetran View Post
returning all the data will load the database. SQL server is designed to filter data down to a small subset. How is he dealing with security? if you want to secure it do it in sql once its out of there you have no control.
Originally posted by NotAllThere View PostHe's a moron. I don't know how things are in your technical environment, but the SAP mantra is: do everything you can on the database server, always minimise all data transrers to the application servers. Or, simply, don't be a moron.
However, i can propose a solution that should suit you both. Switch to an in memory database!
It's all hosted on AWS but AFAIK the DB and application servers are both in eu-west-2 (London), so I can't throw Amazon's egregious prices for cross-region data transfer into the equationComment
-
Can I just say that I wish to find anyone and everyone who designed BACS and take them out the back and shoot the lot of them.
Why does every bank have multiple different formats none of which are identical and most contradict one another to the extent the AI has given up...
Last edited by eek; 5 August 2021, 11:03.merely at clientco for the entertainmentComment
-
Originally posted by NickFitz View Post
Apparently that may have been the case fifteen years ago, but he informed us that nowadays it isn't a problem
Departmental requirements are that all data gathered from various sources and used for generating reports sits in this big PostgreSQL server, and all reports are generated therefrom. I've no doubt it could handle the excess load of fetching whole tables given that our data isn't even very big; but I also have no doubt that the DB developers (one of whom is being called in to decide) will see his "SELECT *" without any kind of WHERE or LIMIT and scream in horror
It's all hosted on AWS but AFAIK the DB and application servers are both in eu-west-2 (London), so I can't throw Amazon's egregious prices for cross-region data transfer into the equationmerely at clientco for the entertainmentComment
-
Originally posted by eek View Post
Do you have any indexes on those database tables at the moment as if the database is having to do full table scans it may as well chuck all the data at the other server (although the transfer times even over fibre will add to processing times).
The meeting was had, and the DB guy said he wasn't too fussed about the load given that we only have maybe a few thousand rows and they have tables that are literally billions of rows long. But he came down very much in favour of using SQL properly, as apart from loading issues, they also have a general policy of doing as much in SQL as possible, they're training people who need it in using SQL, and every single other tool and application they've created for this data warehouse is based around SQL
There was then some kerfuffle about whether the other dev would have to rewrite two weeks' work to use SQL instead of Pandas, until I pointed out that there was no need - his existing code isn't going to stop working because we move to my preferred approach in future
TBH it would take about ten minutes to make the change, as the SQL in question wouldn't be at all complex. But that made everybody happy.Comment
-
Lunch has been a bowl of last night's Scotch broth with an end of sesame bloomer. The soup has developed very nicely in the fridge overnight
So that's one less I have to try and fit into the freezer, which is full to bursting againComment
-
-
Originally posted by NickFitz View PostRain's hereComment
-
Originally posted by NickFitz View Post
Apparently that may have been the case fifteen years ago, but he informed us that nowadays it isn't a problem
Departmental requirements are that all data gathered from various sources and used for generating reports sits in this big PostgreSQL server, and all reports are generated therefrom. I've no doubt it could handle the excess load of fetching whole tables given that our data isn't even very big; but I also have no doubt that the DB developers (one of whom is being called in to decide) will see his "SELECT *" without any kind of WHERE or LIMIT and scream in horror
It's all hosted on AWS but AFAIK the DB and application servers are both in eu-west-2 (London), so I can't throw Amazon's egregious prices for cross-region data transfer into the equation
The thing people seem to have a real problem with is that the cloud is someone else's computer connected via your probably limited internet pipe.
Key things are it may be a big computer but they charge per cycle. The price rises will be like boiling lobsters, raise the temperature slowly enough and the Lobster doesn't realise he is for lunch.
Easy way to avoid it is to use so few cycles they forget you are in the bucket.
Also security frequently stops at the server because once its out people export it to excel and send it out without a clue.
And of course you want to do it properly so no one blames you later.
Always forgive your enemies; nothing annoys them so much.Comment
Partners
- Home
- News & Features
- First Timers
- IR35 / S660 / BN66
- Employee Benefit Trusts
- Agency Workers Regulations
- MSC Legislation
- Limited Companies
- Dividends
- Umbrella Company
- VAT / Flat Rate VAT
- Job News & Guides
- Money News & Guides
- Guide to Contracts
- Successful Contracting
- Contracting Overseas
- Contractor Calculators
- MVL
- Contractor Expenses
Advertisers
Contractor Services
CUK News
- Streamline Your Retirement with iSIPP: A Solution for Contractor Pensions Sep 1 09:13
- Making the most of pension lump sums: overview for contractors Sep 1 08:36
- Umbrella company tribunal cases are opening up; are your wages subject to unlawful deductions, too? Aug 31 08:38
- Contractors, relabelling 'labour' as 'services' to appear 'fully contracted out' won't dupe IR35 inspectors Aug 31 08:30
- How often does HMRC check tax returns? Aug 30 08:27
- Work-life balance as an IT contractor: 5 top tips from a tech recruiter Aug 30 08:20
- Autumn Statement 2023 tipped to prioritise mental health, in a boost for UK workplaces Aug 29 08:33
- Final reminder for contractors to respond to the umbrella consultation (closing today) Aug 29 08:09
- Top 5 most in demand cyber security contract roles Aug 25 08:38
- Changes to the right to request flexible working are incoming, but how will contractors be affected? Today 08:25
Comment