[Tutor] looking but not finding

[o1bigtenor]

Thank you for your patience mr Alan.

What I'm finding is that the computing world uses terms in a different way than
does the sensor world than does the storage world than does the rest of us.
That seriously complicates things. (Details in the following.)

On Tue, Jul 11, 2023 at 8:15 PM Alan Gauld via Tutor <tutor at python.org> wrote:
>
> On 11/07/2023 22:08, o1bigtenor wrote:
>
> >>> - fluid system is on a load cell system
> >>
> >> I googled load call sysem and it seems to be a hardware based system,
> > The load cell system does NOT have to be proprietary.
> >
> > Your response is headed in quite a different direction than my question.
>
> Sorry, but it sounded like you were having trouble *recording* the data.
> It was not obvious that you had managed to read the data and were
> interested in storing it. Quite a different problem I agree.
>
> >>> but I want to record that information (as well as other information around it).
> >>>
> >>> I've tried looking for data log software
>
> There are many, many different types of storage available
> from structured files(shelves, JSON, YAML, XML, etc to
> databases both SQL and NoSQL based.

This is where I start getting 'lost'. I have no idea which of the three
listed formats will work well with what I'm doing - - - dunno even how to
find out either!
>
> Much depends on what you want to do with the data and
> how regular it is. If its regular a SQL database may be
> the best option - SQLite comes with python and is
> lightweight and easily portable.

The first point with the data is to store it. At the time of its collection
the data will also be used to trigger other events so that will be (I think
that is the proper place) be done in the capturing/organizing/supervising
program (in python).
>
> If the data is irregular then a NoSQL database like
> Mongo may be better. But if you only need the storage
> for persistence then a structured file would be simpler.

Went looking at MongoDB - - - honestly cannot tell what makes data
'irregular' - - - - perhaps you might advise - - - I cannot find a reasonable
definition - - - -yet!

I am going to try to delineate at least some of what I'm thinking.

(Each point has data associated with it.)
1. reset weighing system
2. data is captured from the identification sub-system
3. data is captured from the calendar system (date/time)
4. (this one is not in initial plans but possibly in later
      test vacuum in system for level)
5. get weighing system value
6. zero value before starting measurement
7. poll sensor every 0.5 sec using step $8 if there is a change from #6
8. (assuming change in sensor value - - store data
9. when difference between values shows that the value change is less
    than 250 g/min continue for another 20 seconds
10. after #9 is fulfilled issue 'final weight'
11. 2 seconds (may change) after #10 open valve  #1
12. after 30 seconds (this value may change) close valve #1
13. after valve #1 closes over valve #2 connecting system to wash/rinse system
14. 30 sec after valve #2 is opened valve #3 is opened
      (This may be opened to the rinse system at the same time as valve #2 is
       design not yet complete - - - needs fabbing and testing!!!)
15. 5 seconds after rinse is complete valves connecting to the rinse system
     (and its drain) are closed.
16. see #1

Each weighing subsystem will have its own control. There will be multiple
such subsystems (initially at least 3 more likely 12 - - - further
(future) - - - up to
a couple hundred all operating at the same time).
>
> > What I specifically need assistance on is software that enables one to log
> > data. What I find mountainous reams of information is on syslog - - -
> > useful - - - - but when it comes to logging data - - - - well - - - quite NOT.
>
> The logging module and its cousins are intended for
> logging program progress/output/errors etc not for
> storing data. You could use them for that but its
> not their designed purpose.
>
> >>> stuff. Tried looking to see if there were python libraries that could
> >>> be manipulated into service
>
> It sounds like you either want to persist data between
> program runs or to store data for processing/analysis.
> The former is probably best met with a structured file
> format such as JSON. (Or even good old CSV!)
>
> If you want to slice/dice and perform calculations on
> the data, especially if it's a large volume, you may be
> better with a proper database - it's what they are
> designed for after all.
>
> If the data is regular go with SQL if irregular go NoSQL.

Now if I only knew what regular and/or irregular data was.
Have been considering using Postgresql as a storage engine.
AFAIK it has the horsepower to deal with serious large amounts of data.
This weighing routine will be running for up to 3.5 minutes. That doesn't
sound like much - - - some 400 'chunks' (meaning all the data that comprises
what is being stored) of data. (Then there may be a repeat of this function
every 4.5 to 6 minutes for say up to 180 minutes - - - multiple times (varying
up to 3x) per day. Its not serious huge data amounts but its definitely non-
trivial.
>
> > Hardware should be immaterial.
> > Sensors are available for a plethora of measurables.
> > I want to grab a value from sensor A and deposit that value in a
> > data log over there.
>
> OK, but that can be as simple as writing a string to a file.
> Or it can mean a complex SQL query into a database. We don't
> know what your data looks like or what you intend to do
> with it, or how much data there is to store. (We have a
> clue in the data capture rate of 2 samples per second
> (but from how many sensors?) but we don't know the length
> of a measurement period - minutes, hours, days, "infinite"?

Sorry - - - previous - - -
>
> If there are several sensor types are the sensor data similar
> for each? Or do you need a separate data format(table or file?)
> for each sensor type 9or even each sensor!) And if that's
> the case do you need to combine those data later for analysis,
> based on which keys?

This is part of what I hope I was able to answer in my list (1-16).
>
> Also will concurrent reading/writing be required? ie Will
> there be reporting/analysis being performed at the same
> time as the sensors are being read and their data stored?
> What about locking issues?

Here you have opened another pandora's box for me.
Dunno much about any of this.
Concurrent - - - yes, there are multiple sensors per station and
multiple stations and hopefully they're all working (problems
with the system if they're not!)

There is a small amount of analysis being done as the data is stored
but that's to drive the system (ie at point x you stop this, at point
y you do this, point y does this for z time then a happens etc - - - I
do not consider that heavy duty analysis. That happens on a different
system (storage happens first to the local governing system. After
a round of data (one cycle) has completed that data is shipped in a burst
to the long term storage system. (there may be a cascade of storage systems
but I haven't got that far yet!) This system is quite capable of handling a LOT
of information - - - or that's the goal anyway. Maybe One system will store the
info and another will do analysis and then return that analysis to the storing
system for accumulation - - - also not decided.)

Locking - - - how, when, where and why - - - the why I think I understand ( to
make sure data doesn't get corrupted) but the rest - - - would love some
guidance on.
>
> There are a myriad factors to consider when choosing
> an appropriate storage mechanism.
>
> > I understand how to read the sensors
>
> That was the bit that was not apparent in your original
> message, and usually the hardest bit by far!
>
> > ... problems here would go to a much more
> > hardware slanted list - - - - not software.
>
> We have had several hardware interface discussions on
> the tutor list. Often involving USB or RS232 serial
> links and such. The list caters for experienced programmers
> new to Python as well as new programmers.

Hmmmmmmmmmmm - - - Modbus, Probus and RS-485 are all
being considered - - - likely using a tcp/ip backbone.
>
> > The Python info https://docs.python.org/3/howto/logging.html  is quite
> > slanted toward syslog although it does not state that.
>
> I agree, I don't think that's the best route for
> what I believe you want. Logging is quite a well defined
> operation in software engineering so I guess the authors
> of the docs just assumed folks would think in terms
> of system logs.

Can understand that but think that those functions might also be used
for what I'm trying to do - - - but I'm really not sure how to.
>
> > someone just might have produced something like pyplot for producing
> > graphs or scipy which gets close to what I want in its statistics section
>
> But this is a completely different issue again. This is
> moving into how you analyse and report on the data
> you have stored. That's a completely different toolset.

snip
The two mentioned items were examples of code (and/or analysis)  - - - - not
functions I wished to use.
>

> >  but I haven't been able to find anything when
> > I include the term data log (logger/logging etc).
>
> I suspect what you are trying to do is not what
> most programmers would think of as logging.
> It sounds more like data storage.
>
> > What I'm storing - - -
> > 1. item identification
> > 2. time/date (routine is called every 0.5 seconds at this point in the planning)
> > 3. value (from the weighing sensor system)
> >
> > above is data most likely shipping over tcp/ip.
>
> See above. That looks simple and regular enough that
> a simple formatted text string is all that is needed.
> But for potential future complexity(*) I'd suggest
> that a CSV or JSON file would be better for persistent
> storage. A SQL database could handle it easily
> too if you need more complex processing of the data
> post storage, or concurrent access.
>
> (*)From my experience of  building telemetry applications
> it isn't long before people want geographical location,
> and error detection/correction keys etc. The amount
> of data per event tends to grow over time.
>
Oh yes - - - - I'm quite greedy with trying to get as much data
as I can. There are a lot of uses for such and if its easy to put
together then there are less excuses for the not using.

I don't think I'll I'll be starting with csv  - - - - really need to figure out
what I should be using for what you termed 'structured files'.
Do you have any suggestions as to where I find information that
will help me figure out a good or a great way to do that part of things?
(Choosing yaml/xml/???? for starters.)

Thanking you for your patience and assistance.

Regards