<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><div style="box-sizing:inherit;color:rgb(85,85,85);font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;font-size:14px"><span style="box-sizing:inherit"><span style="box-sizing:inherit;color:rgba(0,0,0,0.9)"><span style="box-sizing:inherit;font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue","Fira Sans",Ubuntu,Oxygen,"Oxygen Sans",Cantarell,"Droid Sans","Apple Color Emoji","Segoe UI Emoji","Segoe UI Emoji","Segoe UI Symbol","Lucida Grande",Helvetica,Arial,sans-serif"><span style="box-sizing:inherit;font-size:18px">Hello,</span></span></span></span></div><div style="box-sizing:inherit;color:rgb(85,85,85);font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;font-size:14px"><span style="box-sizing:inherit"><span style="box-sizing:inherit;color:rgba(0,0,0,0.9)"><span style="box-sizing:inherit;font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue","Fira Sans",Ubuntu,Oxygen,"Oxygen Sans",Cantarell,"Droid Sans","Apple Color Emoji","Segoe UI Emoji","Segoe UI Emoji","Segoe UI Symbol","Lucida Grande",Helvetica,Arial,sans-serif"><span style="box-sizing:inherit;font-size:18px">At my new role, I started to use papermill (admittedly with reservation and skepticism) to orchestrate the execution of parameterized jupyter notebooks used to automate my group's data processing needs.  The result?  I am now totally blown away and thrilled with papermill (<a href="https://github.com/nteract/papermill">https://github.com/nteract/papermill</a>).  At my previous role, I had used jupyter notebook to prototype a data process to automatically create invoices.  But to productionalize (sadly, didn't know papermill existed at the time), I had to port my jupyter notebook code to using a data pipelining framework called Luigi (<a href="https://github.com/spotify/luigi">https://github.com/spotify/luigi</a>).  </span></span></span></span><br style="box-sizing:inherit"></div><div style="box-sizing:inherit;color:rgb(85,85,85);font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;font-size:14px"><br style="box-sizing:inherit"></div><div style="box-sizing:inherit;color:rgb(85,85,85);font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;font-size:14px"><span style="box-sizing:inherit"><span style="box-sizing:inherit;color:rgba(0,0,0,0.9)"><span style="box-sizing:inherit;font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue","Fira Sans",Ubuntu,Oxygen,"Oxygen Sans",Cantarell,"Droid Sans","Apple Color Emoji","Segoe UI Emoji","Segoe UI Emoji","Segoe UI Symbol","Lucida Grande",Helvetica,Arial,sans-serif"><span style="box-sizing:inherit;font-size:18px">Now with papermill, I see no need for me to port my jupyter notebooks to something completely different and thus incur more technical debt from having to learn yet another framework.  With papermill, I define which jupyter notebooks run and what they depend on (dependency management).  Furthermore, when a failure occurs, I can fix the bug, then run only the notebooks that didn't run (failure recovery) instead of re-running everything.   So with these features afforded to me in papermill, the beauty is I can keep developing in jupyter notebooks as data pipeline scripts and use them for production.  Also, with papermill,  those that don't come from a software engineering or computer science background, can perform practical "data engineering" without having to learn sophisticated data engineer tools like Apache Airflow or Spotify Luigi, since papermill is far easier to learn, use, and get started than those heavyweight frameworks.  Furthermore, a well annotated jupyter notebook is so much easier to debug and explain what it is doing to a team member.  If you've avoided jupyter notebooks because of the tediousness of version controlling them or not being able to work with them using your favorite text editor or IDE, there is now jupytext (<a href="https://github.com/mwouts/jupytext">https://github.com/mwouts/jupytext</a>) which allows you to edit notebooks as plain text.  I think with all of these things in place, there is little to not like about using papermill to execute notebooks as workflows.  However, since I am just a data analyst or "citizen developer", I would like to hear opinions/thoughts from seasoned IT professionals or data engineers who have worked with jupyter notebooks.  I</span></span></span></span><span style="color:rgba(0,0,0,0.9);font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue","Fira Sans",Ubuntu,Oxygen,"Oxygen Sans",Cantarell,"Droid Sans","Apple Color Emoji","Segoe UI Emoji","Segoe UI Emoji","Segoe UI Symbol","Lucida Grande",Helvetica,Arial,sans-serif;font-size:18px"> know of a couple limitations with papermill, it does not come with a means for scheduling tasks and it doesn't come with a fancy GUI or visualization tool.  But would like to see what others come up with.</span></div></div><div style="box-sizing:inherit;color:rgb(85,85,85);font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;font-size:14px"><span style="color:rgba(0,0,0,0.9);font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue","Fira Sans",Ubuntu,Oxygen,"Oxygen Sans",Cantarell,"Droid Sans","Apple Color Emoji","Segoe UI Emoji","Segoe UI Emoji","Segoe UI Symbol","Lucida Grande",Helvetica,Arial,sans-serif;font-size:18px"><br></span></div><div style="box-sizing:inherit"><span style="color:rgba(0,0,0,0.9);font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue","Fira Sans",Ubuntu,Oxygen,"Oxygen Sans",Cantarell,"Droid Sans","Apple Color Emoji","Segoe UI Emoji","Segoe UI Emoji","Segoe UI Symbol","Lucida Grande",Helvetica,Arial,sans-serif;font-size:18px">I was inspired to give papermill a spin from Netflix's blog article: </span><font color="rgba(0, 0, 0, 0.901960784313726)" face="-apple-system, system-ui, BlinkMacSystemFont, Segoe UI, Roboto, Helvetica Neue, Fira Sans, Ubuntu, Oxygen, Oxygen Sans, Cantarell, Droid Sans, Apple Color Emoji, Segoe UI Emoji, Segoe UI Emoji, Segoe UI Symbol, Lucida Grande, Helvetica, Arial, sans-serif"><span style="font-size:18px"><a href="https://medium.com/netflix-techblog/notebook-innovation-591ee3221233">https://medium.com/netflix-techblog/notebook-innovation-591ee3221233</a></span></font><br></div><div style="box-sizing:inherit"><font color="rgba(0, 0, 0, 0.901960784313726)" face="-apple-system, system-ui, BlinkMacSystemFont, Segoe UI, Roboto, Helvetica Neue, Fira Sans, Ubuntu, Oxygen, Oxygen Sans, Cantarell, Droid Sans, Apple Color Emoji, Segoe UI Emoji, Segoe UI Emoji, Segoe UI Symbol, Lucida Grande, Helvetica, Arial, sans-serif"><span style="font-size:18px"><br></span></font></div><div style="box-sizing:inherit"><span style="color:rgba(0,0,0,0.9);font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue","Fira Sans",Ubuntu,Oxygen,"Oxygen Sans",Cantarell,"Droid Sans","Apple Color Emoji","Segoe UI Emoji","Segoe UI Emoji","Segoe UI Symbol","Lucida Grande",Helvetica,Arial,sans-serif;font-size:18px">I figured if papermill/jupyter notebooks were a good fit for Netflix, I explored if they could be a good fit for my uses at work.  So far, I am pleased with the initial results.</span><br></div><div style="box-sizing:inherit;color:rgb(85,85,85);font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;font-size:14px"><span style="box-sizing:inherit"><span style="box-sizing:inherit;color:rgba(0,0,0,0.9)"><span style="box-sizing:inherit;font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue","Fira Sans",Ubuntu,Oxygen,"Oxygen Sans",Cantarell,"Droid Sans","Apple Color Emoji","Segoe UI Emoji","Segoe UI Emoji","Segoe UI Symbol","Lucida Grande",Helvetica,Arial,sans-serif"><span style="box-sizing:inherit;font-size:18px"><span style="box-sizing:inherit"><br></span></span></span></span></span></div><div style="box-sizing:inherit;color:rgb(85,85,85);font-family:Arial,"Helvetica Neue",Helvetica,sans-serif;font-size:14px"><span style="box-sizing:inherit"><span style="box-sizing:inherit;color:rgba(0,0,0,0.9)"><span style="box-sizing:inherit;font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue","Fira Sans",Ubuntu,Oxygen,"Oxygen Sans",Cantarell,"Droid Sans","Apple Color Emoji","Segoe UI Emoji","Segoe UI Emoji","Segoe UI Symbol","Lucida Grande",Helvetica,Arial,sans-serif"><span style="box-sizing:inherit;font-size:18px"><span style="box-sizing:inherit">- Daniel</span></span></span></span></span></div></div></div></div></div></div>