[Tutor] looking but not finding

[Alan Gauld]

On 11/07/2023 22:08, o1bigtenor wrote:

>>> - fluid system is on a load cell system
>>
>> I googled load call sysem and it seems to be a hardware based system,
> The load cell system does NOT have to be proprietary.
> 
> Your response is headed in quite a different direction than my question.

Sorry, but it sounded like you were having trouble *recording* the data.
It was not obvious that you had managed to read the data and were
interested in storing it. Quite a different problem I agree.

>>> but I want to record that information (as well as other information around it).
>>>
>>> I've tried looking for data log software

There are many, many different types of storage available
from structured files(shelves, JSON, YAML, XML, etc to
databases both SQL and NoSQL based.

Much depends on what you want to do with the data and
how regular it is. If its regular a SQL database may be
the best option - SQLite comes with python and is
lightweight and easily portable.

If the data is irregular then a NoSQL database like
Mongo may be better. But if you only need the storage
for persistence then a structured file would be simpler.

> What I specifically need assistance on is software that enables one to log
> data. What I find mountainous reams of information is on syslog - - -
> useful - - - - but when it comes to logging data - - - - well - - - quite NOT.

The logging module and its cousins are intended for
logging program progress/output/errors etc not for
storing data. You could use them for that but its
not their designed purpose.

>>> stuff. Tried looking to see if there were python libraries that could
>>> be manipulated into service

It sounds like you either want to persist data between
program runs or to store data for processing/analysis.
The former is probably best met with a structured file
format such as JSON. (Or even good old CSV!)

If you want to slice/dice and perform calculations on
the data, especially if it's a large volume, you may be
better with a proper database - it's what they are
designed for after all.

If the data is regular go with SQL if irregular go NoSQL.

> Hardware should be immaterial.
> Sensors are available for a plethora of measurables.
> I want to grab a value from sensor A and deposit that value in a
> data log over there.

OK, but that can be as simple as writing a string to a file.
Or it can mean a complex SQL query into a database. We don't
know what your data looks like or what you intend to do
with it, or how much data there is to store. (We have a
clue in the data capture rate of 2 samples per second
(but from how many sensors?) but we don't know the length
of a measurement period - minutes, hours, days, "infinite"?

If there are several sensor types are the sensor data similar
for each? Or do you need a separate data format(table or file?)
for each sensor type 9or even each sensor!) And if that's
the case do you need to combine those data later for analysis,
based on which keys?

Also will concurrent reading/writing be required? ie Will
there be reporting/analysis being performed at the same
time as the sensors are being read and their data stored?
What about locking issues?

There are a myriad factors to consider when choosing
an appropriate storage mechanism.

> I understand how to read the sensors

That was the bit that was not apparent in your original
message, and usually the hardest bit by far!

> ... problems here would go to a much more
> hardware slanted list - - - - not software.

We have had several hardware interface discussions on
the tutor list. Often involving USB or RS232 serial
links and such. The list caters for experienced programmers
new to Python as well as new programmers.

> The Python info https://docs.python.org/3/howto/logging.html  is quite
> slanted toward syslog although it does not state that. 

I agree, I don't think that's the best route for
what I believe you want. Logging is quite a well defined
operation in software engineering so I guess the authors
of the docs just assumed folks would think in terms
of system logs.

> someone just might have produced something like pyplot for producing
> graphs or scipy which gets close to what I want in its statistics section

But this is a completely different issue again. This is
moving into how you analyse and report on the data
you have stored. That's a completely different toolset.

You essentially have at least 3 separate problems:
1) get the raw data from the sensor
2) format it and store it
3) analyze it and report the results

It sounds like you have cracked number 1.
I've discussed some options for 2 but maybe
you have cracked that too?

For number 3 you've already mentioned the obvious sources
with Pandas and RPy being perhaps the most obvious candidates
for analysis and pyplot (or maybe gnuplot) for generating
graphical output. (But there are many other options from
using Excel to building SVG graphics in a web page)

> (for the optimisation routines for a PID controller I think I would find a
> way to call a fortran routine that I've found that is scary good instead of
> using Python as in scipy)

I'm not sure where the optimisation fits in your
workflow - is that before or after the
analysis/reporting bit? Perhaps based on the
analysis results?

Calling external routines is certainly possible,
either directly from Python (have a look at ctypes
as one possibility) or via command line utilities.

>  but I haven't been able to find anything when
> I include the term data log (logger/logging etc).

I suspect what you are trying to do is not what
most programmers would think of as logging.
It sounds more like data storage.

> What I'm storing - - -
> 1. item identification
> 2. time/date (routine is called every 0.5 seconds at this point in the planning)
> 3. value (from the weighing sensor system)
> 
> above is data most likely shipping over tcp/ip.

See above. That looks simple and regular enough that
a simple formatted text string is all that is needed.
But for potential future complexity(*) I'd suggest
that a CSV or JSON file would be better for persistent
storage. A SQL database could handle it easily
too if you need more complex processing of the data
post storage, or concurrent access.

(*)From my experience of  building telemetry applications
it isn't long before people want geographical location,
and error detection/correction keys etc. The amount
of data per event tends to grow over time.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos