Python Exceptions are Exceptional (and so much more than error handling!)
Exception handling in Python not only provides a way to respond to errors, it allows you to elegantly structure your code and solve problems, reduces code, and sometimes speeds up your script. Being Pythonic means knowing how to use exceptions to their full extent. I've learned about exceptions from many different sources, each having a particular insight, but I haven't found one write-up that covers it all, so this entry an attempt at that.
This post is dedicated to exception handling; I'll do a future post on creating and raising exceptions.
A bit of exceptional history
Traditional programming languages tended to treat errors as a catastrophic situation. An error usually meant that the script couldn't possibly continue, and would probably instantly crash. Therefore, the job of the programmer was to do everything in their power to avoid errors.
That meant that the programmer needed to check for everything that could go wrong before calling a function: Does the file exist? Is the network up? Etc. And if something unexpected went wrong, then the script crashed anyway.
The process of checking for every possible error condition ahead of time is known as Look Before You Leap, or LBYL.
Enter exception handling
Exception handling provided a more robust mechanism for dealing with errors. Rather than instantly crashing, the error would result in an exception being raised (or thrown, as many languages describe it), allowing the programmer to catch the exception and choose how to respond to an error situation.
However, most programmers have retained that original fear of something going wrong, and are still trained to do everything they can to avoid an exception from being raised. Exceptions are still thought of as just a way to respond to catastrophic errors.
Exceptions separate return values from errors
A huge benefit of exceptions in any language is that they solve the age-old problem of combining return values and error codes. Before exceptions, if an error occurred that didn't crash the script, then the function needed a way to indicate the error.
Typically, this meant that usually a function would return a number or string or whatever if everything had worked properly, or it would return 0 if an error occurred.
Unfortunately, for some functions, 0 could be a valid return value, so in that case they had to find a different value. Perhaps -1 would indicate an error. But what if -1 was a valid return value? Then maybe 999,999 would indicate an error...so the programmer was stuck reading through the documentation for each function to determine what return value indicated an error.
Exceptions solved this, because if nothing went wrong, then a valid value was returned. If something did go wrong, and exception was thrown, which was handled separately from the return value.
Basics of exception handling in Python
Suppose you are going to open a file to read data. If the file you are expecting doesn't exist, an exception is raised and your script crashes:
myfile = open('input.txt')
Exception handling allows you to decide what happens when an error occurs, by putting the potentially problematic code in a try block.
The try block
In Python we don't use the throw and catch terminology -- instead you try an operation ("There is no try, only do", said a wise yet non-Pythonic alien) and if something goes wrong an exception is raised and you can choose to handle it.
Here is an example of simple exception handling, with a try block and an except block:
try:
print('Trying to open the input file...')
myfile = open('input.txt')
except:
print('Something went wrong! Oh well, continuing on and hoping for the best...')
print('Continuing...')
You can have as much code as you like in a try block. If an exception is raised by any of the code in that block, Python will then look for an exception handler for the exception that was raised. An exception handling block starts with except and specifies which exceptions it handles. In this case, because the except block doesn't specify any particular exceptions, it matches any exception that is raised.
The example above is not particularly robust exception handling (probably we couldn't really continue on at this point), but illustrates the point that you have complete control over what happens when an exception is raised, including the ability to simply ignore it.
Ignoring an exception with pass
It turns out there are many cases where you don't actually care if an exception occurs, but you don't want your script to crash. For example, if you call os.mkdir() to create a directory, it raises an exception if the directory already exists:
import os
os.mkdir('/tmp')
Since you were trying to create the directory, the fact that it already exists is not something you'd consider an error, and you just want to move silently on. You can do this using pass, which is a keyword that means "this code block intentionally left blank":
try:
os.mkdir('/tmp')
except:
pass
print("Continuing...")
In this case, the exception occurred, it was ignored, and you move on.
The except block
When you do want to do something in response to an exception, you usually need to get information about what exception occurred. Let's say you are getting a value from a dictionary and there is no entry for the key you specify:
zoo = {'penguins': 5, 'lions': 12, 'zebras': 1}
rhino_count = zoo['rhinos']
print(f'There are {rhino_count} rhinos in our zoo!')
Once you realize that it's possible for this code to raise a KeyError exception, you can create a specific exception handler for that case by specifying the exception name as part of the except statement:
zoo = {'penguins': 5, 'lions': 12, 'zebras': 1}
try:
rhino_count = zoo['rhinos']
print(f'There are {rhino_count} rhinos in our zoo!')
except KeyError:
print('There are NO rhinos in our zoo!')
The except KeyError: statement means this exception handler will only be called if it's an exception of type KeyError. Any other exception type will still crash the script. For example, say you do have rhinos in the zoo, but mis-typed the rhino_count variable name in print():
zoo = {'penguins': 5, 'lions': 12, 'zebras': 1, 'rhinos': 2}
try:
rhino_count = zoo['rhinos']
print(f'There are {rhoni_count} rhinos in our zoo!') # Error: typo in variable name
except KeyError:
print('There are NO rhinos in our zoo!')
Since the exception was of type NameError, your exception handler didn't match it and the script crashed. You could do a specific NameError handler, but you also have the option of handling any exception that comes along without having to know what it will be ahead of time, by creating an exception handler for type Exception:
zoo = {'penguins': 5, 'lions': 12, 'zebras': 1, 'rhinos': 2}
try:
rhino_count = zoo['rhinos']
print(f'There are {rhoni_count} rhinos in our zoo!')
except KeyError:
print('There are NO rhinos in our zoo!')
except Exception as e:
print(f'Got exception of type {type(e)}: {e}')
All exceptions must be subclasses of Exception, so a handler for Exception matches any exception. For this reason, you should always put this handler last, since Python will stop checking the exception handlers once one matches. The except Exception as e: statement assigns the exception object with information about what went wrong to the specified variable name. I'm not a fan of one-letter variable names, but it's very common to assign exceptions to the variable name e so I'm doing so here.
It's up to you how you want to respond to an unexpected exception. You might log information about it to a file, or give the user a message, then you can either continue on or you can choose to let the script crash. To do the latter, simply re-raise the exception like so:
zoo = {'penguins': 5, 'lions': 12, 'zebras': 1, 'rhinos': 2}
try:
rhino_count = zoo['rhinos']
print(f'There are {rhoni_count} rhinos in our zoo!')
except KeyError:
print('There are NO rhinos in our zoo!')
except Exception as e:
print(f'Got exception of type {type(e)}: {e}')
print("Not sure what happened, so it's not safe to continue -- crashing the script!")
raise e
In this case, you print a message to the user and then crash the script using raise e. The raise statement is like returning a value from a function, except it instead raises the specified exception (within an exception handler you can raise without specifying an exception and it will automatically raise the same exception that was received, but I recommend the clarity of specifying the exception object).
If you want to handle multiple exceptions in the same handler, you can use a list to specify the exceptions like so:
try:
animals = {}
num = animals['rhino']
except (KeyError, ValueError) as e:
print(f'{type(e)}: {e}')
As you can see, you are in complete control of what happens when an exception is raised. But the story doesn't end there. Now we get to the two most misunderstood features of Python exception handling, else and finally blocks.
The else block
An else block is an optional feature of exception handling that allows you to have a block of code that is only run if no exceptions occurred. Why would you use this? Let's say you are handling exceptions rather than letting the script crash. Here is a situation you can get into if an exception occurs:
zoo = {'penguins': 5, 'lions': 12, 'zebras': 1}
try:
rhino_count = zoo['rhinos'] # Error: There is no 'rhinos' key
except:
print("Error!")
print(f"# of rhinos: {rhino_count}") # Error: Attempting to use a variable that doesn't exist
print("Continuing...")
In this case, an exception occurred, which kept rhino_count from getting created, but the exception handling kept the script from crashing. I then tried to reference rhino_count, which caused another exception that did crash the script. A classic way of handling this is to create the variable ahead of time and set it to False or zero, then check the value before using it, like so:
zoo = {'penguins': 5, 'lions': 12, 'zebras': 1}
rhino_count = False # Initializing the variable to a known state.
try:
rhino_count = zoo['rhinos'] # Error: There is no 'rhinos' key
except:
print("Error!")
if rhino_count is not False:
print(f"# of rhinos: {rhino_count}")
print("Continuing...")
This approach works, but adds unnecessary code and complexity. The else blocks achieves the same with less code, because if you reach the else block, you know everything is as expected, so you don't have to initialize or check values before using:
zoo = {'penguins': 5, 'lions': 12, 'zebras': 1}
try:
rhino_count = zoo['rhinos']
except:
print("Error!")
else:
print(f"No exceptions occurred, # of rhinos: {rhino_count}")
print("Continuing...")
Here's what happens when there are no exceptions:
zoo = {'penguins': 5, 'lions': 12, 'zebras': 1, 'rhinos': 2}
try:
rhino_count = zoo['rhinos']
except:
print("Error!")
else:
print(f"No exceptions occurred, # of rhinos: {rhino_count}")
print("Continuing...")
The benefit of an else block is that you can have code that relies on everything going as expected, and can skip that code if something does go wrong. It's completely optional, but is helpful for organizing your code more elegantly.
The finally block
A finally block contains code that will run before the exception handling is exited no matter what happens. That is, whether or not an exception occurs, the code in the finally block will be run. I will emphasize this multiple times because people often take a bit to absorb what this means.
The finally block is usually used for cleanup that needs to occur whether or not an exception occurred.
I'll start with the aspect of exception handling that I learned most recently: You can use a finally block without needing to provide an exception handler. Here is what happens when no exception occurs:
big_data = 'a' * 100000
try:
print(len(big_data))
finally:
print("Cleaning up!")
del(big_data)
print("Continuing...")
In this case the try block was executed, there were no exceptions, and the finally block was called before moving on.
If you don't have an exception handler, then any exception will crash the script, but the finally block still gets called before that happens:
big_data = 'a' * 100000
try:
print(len(bug_data)) # Error: Typo in variable name
finally:
print("Cleaning up!")
del(big_data)
print("Continuing...")
If you do have an exception handler, then that code will be executed and then the finally block will be executed:
big_data = 'a' * 100000
try:
print(len(bug_data)) # Error: Typo in variable name
except:
print("Error!")
finally:
print("Cleaning up!")
del(big_data)
print("Continuing...")
The finally block will be called even if the script is quit -- it is called no matter what happens.
At this point in classes, a student inevitably asks, "Will the finally block be called if..."
Instructor: (Deep breath) "Yes. NO MATTER WHAT HAPPENS."
Got that?
Using exceptions in the finally block
I felt all good about my knowledge of exceptions, then I ran into a problem that seemed to render the whole thing pointless. If I got an exception while opening a file, and I was using the finally block to make sure the file was closed, I ran into this:
try:
myfile = open('bogus_path.txt')
except:
pass
finally:
myfile.close()
Because the file was never opened, my attempt to close it failed and my finally block triggered an exception and crashed the script.
This flummoxed me for a while and gave me a dim view of exceptions. Now, presumably there's some convenient way to test whether a variable is defined or not, but before I got to that point I had a revelation: This is what exception handling is all about!
If it might raise an exception but you don't care if it does, just use an empty exception handling within the finally block, like so:
try:
myfile = open('bogus_path.txt')
except:
pass
finally:
# myfile may or may not exist, so use an exception handler.
try:
myfile.close()
except:
pass
print("Continuing...")
Now if the file was opened it gets closed, if it wasn't we just move on. This is a perfect expression of the way exception handling solves many problems elegantly.
By knowing what the else and finally blocks do, you've become an elite Python programmer who knows something even many experienced folks don't.
Being Pythonic: Using exceptions to improve your code
That covers the features of exception handling. Now let's talk about the philosophy.
I covered LBYL (Look Before You Leap) above. Now I want to talk about the best part of Python exception handling, the thing that really makes your code Pythonic:
EAFP. It's Easier to Ask Forgiveness than Permission.
Just do it, then handle any errors
When you think about it, because so much code in the world follows an LBYL model, our logic tends to be inverted. First we do a whole bunch of error checking, and only if everything is good do we then actually do the thing we want to do. This buries the actual logic and makes the less interesting part of the code the most prominent. Here's a typical example:
import os
import json
def load_project_data(path):
if os.path.exists(path):
data_dict = json.load(open(path))
else:
print(f"{path} doesn't exist.")
return
if 'project name' in data_dict:
print(f"Project loaded: {data_dict['project name']}")
else:
print("Project data badly formatted, missing project name")
return
return data_dict
data = load_project_data('data.json')
The key to EAFP is to stop worrying about getting exceptions. Start with what you actually want to do, and handle errors after if need be. While we could handle specific exceptions, often you can get away with a generic exception handler:
import os
import json
def load_project_data(path):
try:
data_dict = json.load(open(path))
print(f"Project loaded: {data_dict['project name']}")
return data_dict
except Exception as e:
print(f"Error while loading {path}: {e}")
data = load_project_data('bogus_data.json')
Less code, and the logic is up front.
Even better and more Pythonic, let the person calling your function deal with any exceptions:
import os
import json
def load_project_data(path):
data_dict = json.load(open(path))
print(f"Project loaded: {data_dict['project name']}")
return data_dict
try:
data = load_project_data('bogus_data.json')
except Exception as e:
print(f"Failed to load data: {e}")
else:
print(data)
Exceptions as a different kind of return value
Exceptions aren't just for errors! They can be a normal part of structuring code.
As discussed above, exceptions allow for separating errors from return values. They also allow for returning a different kind of value when there is no error.
Python strings demonstrate an example of the old and new way of doing this. Strings have a find() method that returns the index where the substring was found:
loc = 'Monty Python'.find('P')
print(loc)
If the substring isn't found, find() returns the special value -1:
loc = 'Monty Python'.find('a')
print(loc)
This means you have to look up what -1 means and then check for that specific value, which once again pushes the error checking to the forefront as well as not being the most intuitive code:
loc = 'Monty Python'.find('a')
if loc != -1:
print(loc)
else:
print("Substring not found")
Strings also have an index method that takes the Pythonic approach and simply raises an exception if the substring is not found:
loc = 'Monty Python'.index('a')
print(loc)
It might seem extreme to raise an exception just because a substring wasn't found, but that's the point: Exceptions are not always catastrophic, they can be a normal part of control flow:
try:
loc = 'Monty Python'.index('a')
print(loc)
except:
print('Substring not found')
Solving race conditions in error handling
Sometimes EAFP is the only way.
As mentioned above, os.mkdir() creates a directory, but raises an exception if the directory already exists.
Any guesses what's wrong with this attempt to use LBYL to avoid an exception if the path already exists?
import os
if not os.path.exists('/tmp'):
os.mkdir('/tmp')
Seems straightforward -- we check to see if the path already exists, and if it doesn't we then proceed to call os.mkdir().
Add a + to your grade if you spotted the problem: A tiny bit of time passes between the call to os.path.exists() and the subsequent call to os.mkdir(). In that tiny bit of time, someone (maybe even another copy of your script) might have created the /tmp path, in which case you'll get an exception even though you tried to avoid it with the existence check.
This is called a race condition, and it's a terrible bug to have to find. Because 99.9999% of the time everything is going to work fine, but 0.00001% of the time it will fail, and you'll tear your hair out trying to figure out why it never happens on your system but does happen for some of your users.
The only absolutely reliable solution to this race condition is the EAFP approach: Just try making the directory, and if it fails with an OSError exception, then you know the directory already exists at that instant and can move on:
import os
try:
os.mkdir('/tmp')
except OSError:
pass
Be careful about this one weird thing
I love Python exception handling, and I'm using the techniques discussed above more and more in my code as I fully realize the benefits.
But I have found one little problem popping up with some regularity: If I'm using exception handling to ignore the exception I expect, and then I'm adding some new code to the try block, sometimes the code won't work and I can't figure out why. Then it turns out that an exception was being raised in the new code but was being ignored, so I couldn't see the problem. For example:
import os
try:
# Original code
os.mkdir('/tmp')
# New code that fails because the file doesn't exist but I don't see the exception.
myfile = open('/tmp/stuff.txt')
except:
pass
To deal with this, when adding new code I've gotten into the habit of printing out what exception occurred, so I can see problems:
import os
try:
# Original code
os.mkdir('/tmp')
# New code that fails because the file doesn't exist but I don't see the exception.
myfile = open('/tmp/stuff.txt')
except Exception as e:
print(f'DEBUG: Got exception of type: {type(e)}: {e}')
pass
References
My knowledge of the Pythonic approach to exceptions has come from a variety of sources. Here are a couple that stand out (if you know of other good discussions I should link to, please ping me!):