Code Anatomy - Writing high performance Python code
Writing short and efficient Python code is not always easy or straightforward. However, it's often that we see a piece of code and we don't realize the thought process behind the way it was written. We will be taking a look at the difference snippet, which returns the difference between two iterables, in order to understand its structure.
Based on the description of the snippet's functionality, we can naively write it like this:
def difference(a, b): return [item for item in a if item not in b]
The above implementation may work well enough, but doesn't account for duplicates in
b, making the code take more time than necessary in cases with many duplicates in the second list. To solve this issue, we can make use of the
set() method, which will only keep the unique values in the list:
def difference(a, b): return [item for item in a if item not in set(b)]
This version, while it seems like an improvement, may actually be slower than the previous one. If you look closely, you will see that
set() is called for every
a causing the result of
set(b) to be evaluated every time. Here's an example where we wrap
set() with another method to better showcase the problem:
def difference(a, b): return [item for item in a if item not in make_set(b)] def make_set(itr): print('Making set...') return set(itr) print(difference([1, 2, 3], [1, 2, 4])) # Making set... # Making set... # Making set... # 
The solution to this issue is to call
set() once before the list comprehension and store the result to speed up the process:
def difference(a, b): _b = set(b) return [item for item in a if item not in _b]
Another option worth mentioning when analyzing performance for this snippet is the use of a list comprehension versus using something like
list(). Implementing the same code using the latter option would result in something like this:
def difference(a, b): _b = set(b) return list(filter(lambda item: item not in _b, a))
timeit to analyze the performance of the last two code examples, it's pretty clear that using list comprehension can be up to ten times faster than the alternative, as it's a native language feature that works very similar to a simple
for loop without the overhead of the extra function calls. This explains why we prefer it, apart from readability.
Understand Python's named tuples and start using them in your projects today.
Learn how Python's lists and tuples are different and level up your code today.
Learn the difference between Python’s built-in list sorting methods and when one is preferred over the other.