[Tutor] How to refactor a simple, straightforward script into a "proper" program?

[DL Neil]

Echo: Best New Year wishes to you and yours.


Surprise: I run a series? OK, yes it must be true.
On-line it is usually an extension of discussions, I or A.N.Other has 
provoked over-the-tea-cups/during a tea/coffee/meal break, which has 
become something of a Friday tradition amongst one of the teams with 
which I often work. It is a mentoring program(me) - a training 
opportunity for 'master' level programmers and a learning-opportunity 
for 'apprentices', and often, 'journeymen'.

(all due apologies...)


On 2/01/20 7:30 AM, boB Stepp wrote:
> In a November thread I started, "How to type annotate complex
> dictionary containing lambdas?"
> (https://mail.python.org/pipermail/tutor/2019-November/115726.html), I
> justifiably received multiple critiques on what an awful dictionary I
> had created.  Eventually I responded that the program that that
> dictionary was from (and other dictionaries like it) came from using
> what originally was a very simple script which over time became rather
> complex and gnarly from using it to try out different things I was
> learning about over time.  At the time I started the aforementioned
> thread my learning topic of the day was type annotations, and, of
> course, I used my go to "simple" (Hah!) script to play around with
> that concept.  So perhaps in line with D. L. Neil's "Friday Finking"
> series on the main Python list, I would like to start a discussion on
> how to properly transform a simple Python script into a
> battle-hardened program suitable for release to the general public.
...

> So I am curious as to how you pros would approach the refactoring
> process.  I think that such a demonstration would surely be
> educational for myself, and hopefully others.  But I do realize that
> everyone does their thing voluntarily and my thoughts might not align
> with yours.  So I am *hoping* a few experts go along with this, but,
> regardless, in typical boB fashion I will plow away and post my future
> efforts to transform version1.py into something more robust.  Then
> those that wish to can critique version2.py leading ultimately to
> version-who-knows-what.py ...
> 
> Anyway, it seems like a useful New Year's exercise that might actually
> be helpful.
> 
> HAPPY NEW YEAR TO YOU AND YOURS!!!


Why did I cut-out the code and the suggested improvements? Because the 
reason why many of us are reluctant to "maintain" or "improve" 
someone-else's code is an anxiety that if we fail to understand every 
nuance of thinking that went into the original author's work, we will 
introduce errors!

Indeed, any number of programming 'management' books (or chapters in 
'advanced' texts) will recount how programming (dept) productivity 
declines with the age of a (any) system. The reason is not merely the 
lethargy or reluctance devs have for maintenance work, but the reality 
of the system's growing complexity requiring every 'change' to be the 
culmination of an every-growing effort to investigate any 'corner' of 
the system which may contribute input or require output from the code in 
question! Even looking at this single routine, not only will the 
code-complexity creep upwards (as described, above), but 'just a little 
change' and 'quick fix' type approaches lead to such ghastly artifacts 
becoming common-place!


Accordingly, should we pay any regard to 'the rest', before completing:

> 6)  There are no tests, which makes it more difficult to grow/maintain
> the program in the future.


I've just re-read an interesting observation of the collection of tests 
one might build-up, as a system grows - a challenging question to ponder:

- if the code and all other documentation of a program/system were 
'lost', could you?should you, be able to reconstruct the whole from the 
test-set alone?


I am currently working on code which aims to re-use other classes. There 
are three different data-structures which may be used as input, from 
basic strings, through PSL class, to a locally-coded data-object; the 
consideration then becomes: were all the inputs provided or do 
(different for the various sub-classes) defaults apply; then there are 
all the different real-world combinations which the data represents, eg 
does the transaction currently 'exist' or is it hypothetical at this 
point; and they're only the first three considerations which generate 
permutations or combinations!

During the task, I have found that the new code has failed because I 
didn't adequately code around my own tests (TDD's intent), but also two 
further conditions:

1 my assumptions about how the underlying classes work were 
incomplete/faulty - and thus my 'new code' was inadequate, and

2 that when certain things didn't work, it was a fault in the provided 
classes - apparently a combination of factors they did not anticipate, 
and in at least one case, had no test to reveal (but my tests of my code 
(unintentionally) did).


Whilst the latter was frustrating (which is one of the points I'm 
making), the advantage of being given not just the code (classes, etc) 
for re-use, but the tests to verify that code (and consequently the 
superstructure I was building 'above' it) have been an enormous 
advantage - had I needed to 'read the code' with no further help, 
perhaps I would have thrown it out and 're-invented the wheel'!

(which decision, strangely enough, has justified many an (expensive!) 
system-rewrite project!)
-- 
Regards =dn