+ Reply to Thread
Page 1 of 2 1 2 LastLast
Posts 1 to 10 of 11
  1. #1

    Potty-trained

    suityou01's Avatar
    Join Date
    Jan 2009
    Location
    Skid row
    Posts
    20,965
    Thanks (Given)
    177
    Thanks (Received)
    208
    Likes (Given)
    742
    Likes (Received)
    713

    Default Caching shared application data in dot net

    Hi,

    Any frameworks anyone can suggest for this?

    Say my app runs this

    Code:
    SELECT BigFatMahoosiveBlob FROM BigFatTable WHERE Predicate = xyz
    After it's done the heavy lifting over the wire from the DB, I want to cache this on my Application server (In memory) and then when a query with exactly the same predicate comes along, my app can go "tadaaaaa" without having to do a fresh DB fetch.

    Usual expiration stuff etc.

    So, what does the panel recommend?

    For bonus points, is there anything where the DB can ping back a message to say you're data is stale now?
    Embrace your inner weird.

  2. #2

    Godlike


    Join Date
    Sep 2014
    Location
    Non-Event Horizon
    Posts
    8,676
    Thanks (Given)
    332
    Thanks (Received)
    586
    Likes (Given)
    2208
    Likes (Received)
    2628

    Default

    Have you relocated to Bangalore?

    Man up, do your job or hand your pass in you fraud.
    The greatest trick the devil ever pulled was convincing the world that he didn't exist

  3. #3

    More time posting than coding


    Join Date
    Dec 2007
    Posts
    259
    Thanks (Given)
    0
    Thanks (Received)
    0
    Likes (Given)
    0
    Likes (Received)
    1

    Default

    I'd have to search MSDN because it's been so long but I think you're looking for the SqlDependency class assuming you're using SQL server which I believe will provide some mechanism of cache invalidation when data becomes stale. Perhaps there is something similar available for different database servers.

    Quote Originally Posted by LondonManc View Post
    Have you relocated to Bangalore?

    Man up, do your job or hand your pass in you fraud.

  4. #4

    Double Godlike!


    Join Date
    Dec 2012
    Posts
    10,224
    Thanks (Given)
    728
    Thanks (Received)
    710
    Likes (Given)
    4820
    Likes (Received)
    2796

    Default

    Quote Originally Posted by suityou01 View Post
    Hi,

    Any frameworks anyone can suggest for this?

    Say my app runs this

    Code:
    SELECT BigFatMahoosiveBlob FROM BigFatTable WHERE Predicate = xyz
    After it's done the heavy lifting over the wire from the DB, I want to cache this on my Application server (In memory) and then when a query with exactly the same predicate comes along, my app can go "tadaaaaa" without having to do a fresh DB fetch.

    Usual expiration stuff etc.

    So, what does the panel recommend?

    For bonus points, is there anything where the DB can ping back a message to say you're data is stale now?
    Know your limits

    Put you requirements down.

    LM and I will be back in a few days with timescales, resource requirements and a quote

  5. #5

    Faqqed Off

    TheFaQQer's Avatar
    Join Date
    Oct 2006
    Posts
    35,660
    Thanks (Given)
    361
    Thanks (Received)
    1207
    Likes (Given)
    3490
    Likes (Received)
    3009

    Default

    Quote Originally Posted by MrMarkyMark View Post
    Put you requirements down.
    Optimistic
    Best Forum Advisor, 2014
    Work in the public sector? You can read my FAQ here
    Click here to get 15% off your first year's IPSE membership

  6. #6

    Richer than sasguru

    DimPrawn's Avatar
    Join Date
    Jul 2005
    Location
    Brexit Britain
    Posts
    32,549
    Thanks (Given)
    265
    Thanks (Received)
    705
    Likes (Given)
    4166
    Likes (Received)
    3504

    Default

    Quote Originally Posted by suityou01 View Post
    Hi,

    Any frameworks anyone can suggest for this?

    Say my app runs this

    Code:
    SELECT BigFatMahoosiveBlob FROM BigFatTable WHERE Predicate = xyz
    After it's done the heavy lifting over the wire from the DB, I want to cache this on my Application server (In memory) and then when a query with exactly the same predicate comes along, my app can go "tadaaaaa" without having to do a fresh DB fetch.

    Usual expiration stuff etc.

    So, what does the panel recommend?

    For bonus points, is there anything where the DB can ping back a message to say you're data is stale now?
    I for one think this is an inneresting technical question.

    Is the data going to be shared across different processes and or servers or loaded into one process?

    How much memory will these large objects occupy?

    In terms of stale data, one pattern is a write through cache. All access to the DB / persistence store is through the write through cache. This way the cache always knows when data has been invalidated. You can even have a lazy writer process persist the data from the cache in the background (write behind).

    There are many open source and cloud solutions. Memcached, AWS Elasticache, Azure Redis Cache for example.

    Patiently waiting for the much publicised and feared Brexit Doom.....

  7. #7

    Contractor Among Contractors

    woohoo's Avatar
    Join Date
    Nov 2007
    Location
    Manchester
    Posts
    1,918
    Thanks (Given)
    86
    Thanks (Received)
    118
    Likes (Given)
    579
    Likes (Received)
    436

    Default

    .net has a built in memorycache, key, value. You could hash the SQL criteria and use that as the key.

    In the past when changing config data through an admin site, I've called a web service to invalidate cache which would force the next query to fetch the data from the db.
    Last edited by woohoo; 28th March 2017 at 21:11.

  8. #8

    Richer than sasguru

    DimPrawn's Avatar
    Join Date
    Jul 2005
    Location
    Brexit Britain
    Posts
    32,549
    Thanks (Given)
    265
    Thanks (Received)
    705
    Likes (Given)
    4166
    Likes (Received)
    3504

    Default

    Quote Originally Posted by woohoo View Post
    .net has a built in memorycache, key, value. You could hash the SQL criteria and use that as the key.
    You can, that's why I asked if the cache needs to be distributed and maintained across multiple processes and machines and how large the data is.

    If it's 4TB of data to cache, good luck holding that in one .Net process.
    Patiently waiting for the much publicised and feared Brexit Doom.....

  9. #9

    Fingers like lightning


    Join Date
    Mar 2013
    Location
    Northeast or where the money is
    Posts
    700
    Thanks (Given)
    7
    Thanks (Received)
    29
    Likes (Given)
    81
    Likes (Received)
    120

    Default

    ... but it works on his 3 rows in Development

  10. #10

    Potty-trained

    suityou01's Avatar
    Join Date
    Jan 2009
    Location
    Skid row
    Posts
    20,965
    Thanks (Given)
    177
    Thanks (Received)
    208
    Likes (Given)
    742
    Likes (Received)
    713

    Default

    Quote Originally Posted by DimPrawn View Post
    I for one think this is an inneresting technical question.

    Is the data going to be shared across different processes and or servers or loaded into one process?

    How much memory will these large objects occupy?

    In terms of stale data, one pattern is a write through cache. All access to the DB / persistence store is through the write through cache. This way the cache always knows when data has been invalidated. You can even have a lazy writer process persist the data from the cache in the background (write behind).

    There are many open source and cloud solutions. Memcached, AWS Elasticache, Azure Redis Cache for example.

    Thanks DP

    The data would be shared across processes and servers. The data is really only likely to be a few 100,000 rows of normal text on some fairly wide tables so not really as bad as I was making out with the blobs, but I was exaggerating to see if some super duper frameworks would come to the fore.

    This isn't cloud but on premise.

    The write through cache is exactly the kind of thing I was thinking of, but the first rule is don't cut code if there is something already available.

    I will look at the Open Source solutions you mention.
    Embrace your inner weird.

+ Reply to Thread
Page 1 of 2 1 2 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts

Content Relevant URLs by vBSEO 3.6.0 ©2011, Crawlability, Inc.