Getting to the “Plateau of Productivity” with Python


As we close out the year, I wanted to take a step back and write a post that will
motivate people to learn python and apply it to their daily jobs. Based on
some comments I’ve received (and my own personal observations), some people
struggle to get started on this journey. They see the potential value of using python in their jobs but are not
sure where to start and can not find the time to take the first steps.
Closely related to this challenge is finding the perseverance to make it through
the inevitable barriers you will encounter. My goal in this article is to provide some
items to keep in mind so that you can be successful in your endeavors to learn python and apply it to your job.
If you take the time (definitely no easy task) to develop your python skills, you
can reap many benefits – outside of the obvious ones you may have started out seeking.

Knowledge is Power

Many people have probably heard of the hype cycle.

Technology Hype Cycle

This graph is generally used to illustrate the hype associated with new technology
in the market. Think of ideas such as 3D Printing, Nano-technology or anything “cloud.”
All these technologies (and many more) lie somewhere on this continuum. I contend
that this hype cycle can apply to your own personal journey as you endeavor to
learn a new technology.

Ok, so maybe it can apply to your own technical journey with python (or some
other technology), but why should you care? For people of a certain age, do
you remember the old GI Joe cartoon and the motto at the end?

GI Joe

I am not sure if knowing is truly half the battle but I do know that more knowledge
about what you are getting yourself into will help you prepare for the challenges
you will face as you try to learn something new.

Beware the Hype

For this article, we will apply the stages of the hype curve to a hypothetical person
who has never typed a line of python code in their life.

Technology Trigger
For some reason, this person has realized that they need to automate
some of the tedious processes they do on a daily basis. After some google searching, they
learn that python might really be useful.
Peak of Inflated Expectations
Now that they know python is a thing, they start
researching it. They become excited about all the big companies using python, the
rich library of 3rd party tools, the promise of time savings and the joys of coding.
They immediately start to see all the problems they could solve with python.
Their excitement builds and this is where they believe so many things are possible!
Trough of disillusionment

As this person starts to delve into the python world and apply it to their problem,
it’s not quite as easy as they think. They have a lot to figure out.

  • How do they install python?
  • Which version should they use?
  • How do they even decide how to tackle their problem?
  • How do they get the data in a easy to use format?
  • Where do they find the time to spend on this “side” project?
  • Probably 100’s of other challenges both big and small.

At the end of the day, I think a lot of people dabble with python, get stuck here
and never move forward.

Slope of Enlightenment
However, if they persevere, tackle the right problems and have the right goals; they
start to climb out of the trough and see how python can help. Slowly but surely, they climb
out of this valley and start to apply their newfound knowledge to more and more problems.
This is where they start to see the light at the end of the tunnel.
Got That Goin For Me
Plateau of Productivity
At the end of the day, the new pythonista reaches a level where they are more productive
with their python tools than they were before. They know which tools to use and when to use them.
They start to understand their business problems better and deliver higher quality
results with less time and energy. In this model, it’s called a “plateau” but
I would contend that someone can continue to grow here for years and years.
No one ever “arrives.” There are always options to learn and grow.

If we understand the hype cycle correctly, then it is important to find ways to
to push through the Trough of Disillusionment. I believe that if a person new
to python or new to the tools discussed in this blog keeps this in mind, they will be
more successful in their journey. The rest of this article will talk through some things
to keep in mind in order to make it through this process.

Take The Easy First Step

I think the most important item to consider is picking a good starting point for
your first python project. Because python is so powerful and you may be climbing towards
the Peak of Inflated Expectations, there is a tendency to try to tackle a really
challenging problem. Do not start off trying to apply the latest machine learning
algorithms to your company’s customer purchase patterns! Rather, take a simple
process that you know really well and try to automate it.

Starting with a well-known and simple problem has some advantages:

  • Since you know the problem space, you can focus on learning python.
  • By comparing your python results with your known results, you can ensure you
    are delivering the right results.
  • Everyone needs to figure out their coding environment and best practices.
    Take the time to get your environment set up correctly so it’s maintainable in the long term.
    Review this best practices article for some ideas to keep in mind.

Give Yourself Permission to be Inefficient

The other important item is to give yourself permission to spend time on
the problem. Let’s say you choose to automate a fairly simple file manipulation
process that takes you 10 minutes a day. As you start automating it, you will
probably take several hours to get the project where it needs to be. That is ok!
Remember, you may find yourself in the Trough of Disillusionment. Do not stop,
keep going!

How many times have you heard people say something like “I have too much work to do”
or “I’m too busy to improve this process?” We have all done it. It is easy to keep doing the same old thing
and not break out of your comfort zone.
If you want to improve a process, you need to realize that the short term time
investment will pay off in the end. It is ok to take more time as you start this process.

Keep an Eye Out For Unexpected Wins

As I have climbed the Slope of Enlightenment with my own projects, one of the most
surprising items I have found is that many times the benefits I thought I was going
to get were not as important as the benefits I did receive. In other words,
automation will yield more improvements than just time savings.

Also, I will be the first to admit, that there are times when automation may
not be the right thing to do. Not surprisingly, xkcd has a comic about this as well.

Technology Hype Cycle

So, if sometimes the automation does not work out or it is more work than expected,
what are some of the unexpected wins you might get?

Offloading the Task to Someone Else

Highly manual tasks fall on someone’s shoulders because “they have always done it.”
It may be difficult to hand the task off to a coworker
because there is no documentation (or no time to make the documentation). I have
found though that once you automate a process, it is much easier to hand it off
to someone else. By handing it off, you free up your time. That’s one way you can start
climbing the Slope of Enlightenment.

Understanding Your Data Better

When you start automating your processes, you may be forced to deal with some
of the ugliness of your data. I don’t think there are very many people that started
on a data automation project that did not need some level of data
cleaning. You will frequently find that there are opportunities to improve
upstream systems to filter out the data before it gets to you. Maybe you can help
out others that don’t even know there are problems with their data.

Additionally, as you start to manipulate your data and build solutions, you get
a much better feel of what data is available to you and how easy (or difficult)
it is to bring multiple data sources together. Overall, I believe that understanding
your data at the detailed level can help you understand your business better. If nothing
else, it will make you much wiser on your next automation project as you move
into the Plateau of Productivity.

Increasing Accuracy

Almost any time you automate a process, you will build increased accuracy
or quality in your output. The interesting side effect is that when the data is
correct and your output is correct, it is relied upon more and more.
Sometimes, people know that reports are “wrong” but choose to suffer
the bad results instead of working through the issue and resolving it. Once you
iron out those small problems, people will start to appreciate your solution and
incorporate it in their daily processes.

Increasing the Frequency of the Task

Let’s say you have to run a manual Excel report once a week and it takes you 1 hour
to pull it together and distribute it. This hour is not a huge investment but it
is painful enough that you don’t want to run it every day. Once
you automate it, you are much more willing to run it as frequently as the business needs.

Minor Changes are Easy

A highly manual process is not very robust. Because it is not very robust, it is
difficult to make minor changes to it. I have had many experiences where automation
supports incremental changes that were not possible before. Sometimes it is as
simple as a minor column name change or breaking reports into multiple sections.
Sometimes it is a more complex solution that is difficult to implement in Excel
but is trivial in python. Either way, there is more flexibility with a robust
code-based solution (stored in version control) than a manual process stored in
someone’s head.

Fixing Issues Can Be Easier

A lot of times identifying and fixing bugs are much easier with an automated solution.
To illustrate this item, here’s a real life issue I recently experienced. As part
of a process to create ~100 Excel files, a user could not find one of the files.
I looked through the list of files and realized immediately what the problem was.

The code looked something like this:

default_file_name = "{}-report-{:%m-%d-%Y}.xlsx".format(LName,

Obviously this is going to cause a problem if I have someone named John Smith
and Fred Smith. The solution was simple- add their first name into the naming convention.

default_file_name = "{}-{}-report-{:%m-%d-%Y}.xlsx".format(FName, LName,

Granted this is a somewhat simplistic example but it highlights the point that it is easier
to troubleshoot this issue than to figure out if I just forgot a name when creating
these files by hand. I also know it is fixed for good and I do not need to
remember another “manual workaround.”

Major Improvements May Not Require Major Re-work

The python environment will give you options that were not available to your manual process.
If you want to integrate with other services or slice and dice multiple files,
you’ll have a full python environment available to you. As you build your skills,
you will find that there is less need to manually do things. At this point, you are
truly moving up the slope of enlightenment.

Do Not Underestimate the Basic Knowledge You Will Learn

Sometimes when you are struggling to figure out some python syntax or get
a bug resolved, you may not realize how much you really have learned. The big
trick to make it through the Trough of Disillusionment is to recognize
how far you have come and how much knowledge you have gained. Even if
the automation project is a failure and does not work, you will be smarter for the effort.

To illustrate my point, here are several things that I learned over the past year
while working on various projects. Most of these items are related
to data analysis topics that I never knew about or understood but are are applicable
more broadly:

Even if I never used any of the scripts I developed, I do have this knowledge and
can use it to be more effective in my daily work.

Closing Out

This article was a bit of a departure from my normal topics. However, I think it
is important to help people think about technology in the context of the bigger picture.
I truly believe that many people could learn python and apply it to their
jobs to make it better. I think the concept of going through the stages of the hype
cycle might frame the process and help set realistic expectations. My hope is
that readers will bookmark this post and refer back to it when they get stuck.
As always, if you have thoughts on this article, feel free to comment below.

Success Kid

Source From:
Original article title: “Getting to the “Plateau of Productivity” with Python”.
This full article can be read at Getting to the “Plateau of Productivity” with Python.


Random Article You May Like

Leave a Reply

Your email address will not be published. Required fields are marked *