Improve consistency of caching and data storage #118
ian
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Provide a common wrapper for both hot (spots, alerts) and cold (semi-static and other lookups) data caches.
Hot data should use TTLCache and a separate thread to page to disk every X minutes and on shutdown, rather than writing to disk on every change like diskcache currently does.
Solar conditions is more like a cold data store as it doesn't expire... But maybe the forecast bits should do rather than being managed with a separate cleanup thread. This makes the solar conditions data store a bit of a complex special case.
Replace the way storing in-memory dicts of data currently works, rather than request caching on every use we should have a separate thread that handles querying the semi-static URLs and storing in the cold data store every e.g. 30 days and on startup if the data is old/missing. There's no need to expire this data, just to try and replace it on a cron job-like basis.
Clublog / cty.dat stuff should move to the same way of working with the cold data cache with no expiry and a thread that fetches the data from source every so often. Might need to move away from pyhamtools to allow this.
With QRZ etc lookups we should do something similar, replace our in-memory dicts with the cold data store, use the data from cache until it expires, but then re-query the server the next time it's needed after that - we don't want a thread getting QRZ data with random credentials, they have to come from the user making the request.
There should be no need for requests cache library, we should be controlling whether a request happens at all based on our own data store validity rather than just making web requests and letting requests cache handle whether it happens or not.
But for that to happen we do need a way of saying "QRZ had no data, stop asking", maybe by storing an empty map of data for the callsign in a callaign-data structure.
Maybe if we are caching WWFF etc data in a nicer way, we should actually do the same for POTA etc as well instead of a live lookup. I think there are downloadable data files for all programmes? Set up a whole set of new providers similar to spotproviders etc to power this, report status etc.
Redis backendto Improve consistency of caching and data storageFinished SIG lookups. Now got a weird problem where infer_missing is super slow. Also app doesn't shut down properly, need to try and stop it then py-spy threads to see what's still running.
Callsign lookup, Clublog/CTY are still to do, as is improving the URL data cache mechanism.
Performance problem is resolved I think.
Callsign lookup, Clublog/CTY still to do.
I think we also need to change our caching approach so that everything is an in-memory cache with timed snapshot storing to diskcache. Currently we are hitting performance penalties with stuff like dxcc_data where we have to iterate through all dxccs finding a matching callsign regex, which is being slowed down because every entry needs unpickling every time.
OK, callsign regex stuff sorted.
Next up Clublog/CTY if we can.
For callsign lookup, I think it's flawed to generate a single callsign-lookup-data object for a single callsign, because for example if the user didn't provide QRZ credentials, that will be generated without QRZ data. If the user later provides credentials but the old data object TTL hasn't expired, how do we know we can now re-fetch better data? Maybe we do need to keep the current structure where QRZ and HamQTH data are cached separately and merged if available on request. But these can still be our own (newly defined) objects rather than caching the server response.
Callsign lookup: