[Tutor] looking but not finding

[o1bigtenor]

Some interesting ideas mr Mats

On Wed, Jul 12, 2023 at 1:53 PM Mats Wichmann <mats at wichmann.us> wrote:
>
> On 7/12/23 09:26, o1bigtenor wrote:
> > Thank you for your patience mr Alan.
> >
> > What I'm finding is that the computing world uses terms in a different way than
> > does the sensor world than does the storage world than does the rest of us.
> > That seriously complicates things.
>
> I'm guessing you have no idea how bad the jargon problem is... and it
> certainly doesn't help when terms are "overloaded" - given different
> special meanings by different groups.

I have some idea - - - general engineering sorta started that and anything to do
with engineering in any way shape or form (plus the medical/dental fields) seems
to glory in obfuscation - - - seems to make far too many think that
they know something
where in fact this kind of usage  - - - - well - - - - I'll stop - - -
'cause I'm a supposed to
be at least a little polite!
(Will admit to having great fun showing 30 year practitioners with the
highest level
of education how they really need to get back to hands on - - - -
instead of paper
manipulation - - - - except that never works with a bureaucrat - -  :-(  ! )
>
> Logging:  what you've found has to do with event logging - where a
> program wants to record things that have happened in the running of the
> program itself.  What you're after is what some call data logging.   You
> can even look for projects that self-declare being related to that term
> like this:

Well - - - - sorta - - - - I want to do both.
I need to record the results from the sensors because some of that information
is used in the running of the program but almost all of the sensor data is going
to be stored as well. SDo this isn't a clean store only nor is it where the data
isn't used within the management program - - - its both!
>
> https://github.com/topics/data-logging

In the first example listed I find the using of the python 'logging' libraries.
>
>  >> There are many, many different types of storage available
>  >> from structured files(shelves, JSON, YAML, XML, etc to
>  >> databases both SQL and NoSQL based.
>
>  > This is where I start getting 'lost'. I have no idea which of the three
>  > listed formats will work well with what I'm doing - - - dunno even how to
>  > find out either!
>
> The storage format is something you might consider an implementation
> detail, that you don't have to worry about yet. The first steps are the
> ones you're taking: to flesh out what your requirements are, and move to
> a conceptual design. When you think you have a vision of what it will
> look like, you can see if some particular existing library provides an
> easy to use implementation of one of these that would fit your needs,
> with some room to group.  If you design it right, the piece that drives
> that could also drive a different thing, if you found you chose poorly,
> or outgrew it.
>
> JSON, YAML and XML are text-based formats, with some markup to help both
> you as a human reader and in some cases, computer code, to decode it. If
> there's not a ton of data, and/or there's not a lot of interrelationship
> between different types of data (suggesting a "relational" database),
> one of these will probably be fine. XML is a lot chattier, and the XML
> libraries in Python are quite fiddly to use so recommend you cross XML
> off your candidate list unless there are really good reasons to needs
> its level of structure.  You can even use comma-separated-values (like a
> spreadsheet program would dump put), which is quite inflexible - one
> line is one record, and every field has to be present in every record,
> you don't have the option of extra or omitted fields, or it just breaks.
>   Pretty sure each one of those has an eloquent wikipedia page, e.g.
>
> https://en.wikipedia.org/wiki/JSON#Syntax

With some digging it seems that yaml is pretty much a superset of of
json and with your notes I'm leaning very much toward using yaml.
>
> If you want to use a database, rather than a text file, Alan laid out
> some choices. For reasonably small amounts of data, sqlite is usually
> fine. It's perhaps closer to the text-based interchange formats in that
> there's no separate database program that you communicate with, data is
> read from and written to a disk files in-process, but the interaction
> with that data is through SQL statements so it's also like the "big
> databases" - mysql/mariadb, SQL Server, Postgres.  Then the "noSQL"
> databases (that's not a terribly precise term) are the ones that have
> less structure, to aid in scalability - a problem you're not likely to
> have from your description so won't say more here.
>
Its looking quite like postgresql is going to be the storage apparatus.

Spent some time looking to see the difference between regular and irregular
data - - - another set of terms used commonly and just not defined!
(Or at least not well or perhaps only hard to find!)

Thanking you for your assistance.