• Visitors can check out the Forum FAQ by clicking this link. You have to register before you can post: click the REGISTER link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. View our Forum Privacy Policy.

Reply to: test please delete

Collapse

You are not logged in or you do not have permission to access this page. This could be due to one of several reasons:

  • You are not logged in. If you are already registered, fill in the form below to log in, or follow the "Sign Up" link to register a new account.
  • You may not have sufficient privileges to access this page. Are you trying to edit someone else's post, access administrative features or some other privileged system?
  • If you are trying to post, the administrator may have disabled your account, or it may be awaiting activation.

Previously on "test please delete"

Collapse

  • NickFitz
    replied
    Some more of Russians Among Us read, and then I watched some paramedic thing on My5

    Early night now with the usual vague plan of getting the walk and inwards trundling done first thing tomorrow. Looks like I'll probably get wet, but I live for pleasure alone

    Goodnight all

    Leave a comment:


  • NickFitz
    replied
    Tea has been chicken balti with rice and naan. Very nice

    The rain keeps stopping, then coming down at the rate of a monthly average a minute, then stopping again. Currently stopped

    Leave a comment:


  • xoggoth
    replied
    Used Dbase in a contract once. Use MySQLi on small business website but nowt very complex.

    Leave a comment:


  • DoctorStrangelove
    replied
    Haven't used a database since dBase II was a thing.

    I think the 5.25" diskettes with it on have rotted.

    Leave a comment:


  • ladymuck
    replied
    Showers off and on still.

    Dinner was chicken casserole and chips from the freezer, as I need to make room ahead of shopping being delivered tomorrow.

    I like SQL it feels intuitive. I only know basic stuff but I at least know to do a bit better than SELECT * with no WHERE clause. I suppose if I applied myself I'd probably do quite well with it.

    Leave a comment:


  • NickFitz
    replied
    Torrential rain out there now

    It's quite enjoyable to watch from in here

    Leave a comment:


  • NickFitz
    replied
    And the recycling bin got a trundling as I was heading out

    Leave a comment:


  • NickFitz
    replied
    Walk briskly walked, green ring closed

    I was persistently drizzled on for the second mile

    Still, it wasn't cold so it was quite refreshing if anything

    Leave a comment:


  • NickFitz
    replied
    Rain's stopped, but apparently it'll be back again in forty minutes or so. Better get my walk started!

    Leave a comment:


  • NickFitz
    replied
    Originally posted by vetran View Post


    The thing people seem to have a real problem with is that the cloud is someone else's computer connected via your probably limited internet pipe.

    Key things are it may be a big computer but they charge per cycle. The price rises will be like boiling lobsters, raise the temperature slowly enough and the Lobster doesn't realise he is for lunch.

    Easy way to avoid it is to use so few cycles they forget you are in the bucket.

    Also security frequently stops at the server because once its out people export it to excel and send it out without a clue.

    And of course you want to do it properly so no one blames you later.


    All of this stuff is only accessible within the VPN anyway, and any cost compared to the department's overall AWS bill is going to be negligible. The thing he really had a problem with is that he doesn't like writing SQL, I think.

    Meanwhile, one of the department's data analysts was asked for demonstration purposes to copy-and-paste my SQL query, and they had a visualisation of the data up on one of their internal platforms within minutes. For the other stuff that grabs whole tables and builds the data with Pandas: not so much

    Leave a comment:


  • vetran
    replied
    Originally posted by NickFitz View Post

    Apparently that may have been the case fifteen years ago, but he informed us that nowadays it isn't a problem



    Departmental requirements are that all data gathered from various sources and used for generating reports sits in this big PostgreSQL server, and all reports are generated therefrom. I've no doubt it could handle the excess load of fetching whole tables given that our data isn't even very big; but I also have no doubt that the DB developers (one of whom is being called in to decide) will see his "SELECT *" without any kind of WHERE or LIMIT and scream in horror

    It's all hosted on AWS but AFAIK the DB and application servers are both in eu-west-2 (London), so I can't throw Amazon's egregious prices for cross-region data transfer into the equation

    The thing people seem to have a real problem with is that the cloud is someone else's computer connected via your probably limited internet pipe.

    Key things are it may be a big computer but they charge per cycle. The price rises will be like boiling lobsters, raise the temperature slowly enough and the Lobster doesn't realise he is for lunch.

    Easy way to avoid it is to use so few cycles they forget you are in the bucket.

    Also security frequently stops at the server because once its out people export it to excel and send it out without a clue.

    And of course you want to do it properly so no one blames you later.



    Leave a comment:


  • ladymuck
    replied
    Originally posted by NickFitz View Post
    Rain's here
    Been and gone here but expected to return.

    Leave a comment:


  • NickFitz
    replied
    Rain's here

    Leave a comment:


  • NickFitz
    replied
    Lunch has been a bowl of last night's Scotch broth with an end of sesame bloomer. The soup has developed very nicely in the fridge overnight

    So that's one less I have to try and fit into the freezer, which is full to bursting again

    Leave a comment:


  • NickFitz
    replied
    Originally posted by eek View Post

    Do you have any indexes on those database tables at the moment as if the database is having to do full table scans it may as well chuck all the data at the other server (although the transfer times even over fibre will add to processing times).
    It's all heavily indexed - I believe they even go to some trouble to add indexes to things like date fields automatically, just in case whoever owns the data forgets

    The meeting was had, and the DB guy said he wasn't too fussed about the load given that we only have maybe a few thousand rows and they have tables that are literally billions of rows long. But he came down very much in favour of using SQL properly, as apart from loading issues, they also have a general policy of doing as much in SQL as possible, they're training people who need it in using SQL, and every single other tool and application they've created for this data warehouse is based around SQL

    There was then some kerfuffle about whether the other dev would have to rewrite two weeks' work to use SQL instead of Pandas, until I pointed out that there was no need - his existing code isn't going to stop working because we move to my preferred approach in future

    TBH it would take about ten minutes to make the change, as the SQL in question wouldn't be at all complex. But that made everybody happy.

    Leave a comment:

Working...
X