Kaushik's Blog

Tric-key stuff: Modifying keys while iterating over a dict

  1. You can't change keys when iterating over a dictionary...
  2. ...but you kinda can?

You can't change keys when iterating over a dictionary...

I wanted to remove some fields from a dictionary that cannot be printed as plaintext. Keys ending with "Msg" or "data" contain serialized data that I don't want to print as plaintext.

# remove_binary_keys.py

def remove_binary_data_keys(d):
    for k in d.keys():
        if k.endswith("Msg") or k == "data":
            d.pop(k)
    print(f"Removed keys ending with 'Msg' or 'data'. Dict is now: {d}")

msg_dict = {
    "id": 123,
    "data": b"abcd",
    "otherMsg": b"bcde",
}

remove_binary_data_keys(msg_dict)
print(msg_dict)

This threw a RuntimeError!

Traceback (most recent call last):
  File "/home/kaushik/blog/remove_binary_keys.py", line 29, in <module>
    remove_binary_data_keys(msg_dict)
  File "/home/kaushik/blog/remove_binary_keys.py", line 6, in remove_binary_data_keys
    for k in d.keys():
RuntimeError: dictionary changed size during iteration

The problem here is that d.keys() is a view into the dictionary object. Changing the object changes what is viewed, which changes the limits of the iteration, which is messy. Python says, "No, that's naughty".

The workaround is to create a list from the keys and iterate over that instead. The list() constructor makes a copy of the keys, which is less optimal than using a view, but is a concise way to solve the problem.

def remove_binary_data_keys(d):
    for k in list(d.keys()):
        if k.endswith("Msg") or k == "data":
            d.pop(k)

Output:

Removed keys ending with 'Msg' or 'data'. Dict is now: {'id': 123}
{'id': 123}

Neat stuff.

...but you kinda can?

Diving into this further, there's actually some strange legacy Python behaviour here.

Consider this snippet:

# change_keys_during_iteration.py

d = {"a": 1, "b": 2}

for k in d.keys():
    del d[k]            # Remove the key
    d[f"new_{k}"] = 0   # Add a new key

print(d)

This is similar to what we just did, right? Modifying the keys of a dictionary while iterating over a view of the keys.

As expected, this raises a RuntimeError...

$ docker run --rm -v /home/kaushik/blog/change_keys_during_iteration.py:/opt/file.py python:3.8-alpine python /opt/file.py
Traceback (most recent call last):
  File "/opt/file.py", line 4, in <module>
    for k in d.keys():
RuntimeError: dictionary keys changed during iteration

...so long as you're using Python 3.8 or later.

$ docker run --rm -v /home/kaushik/blog/change_keys_during_iteration.py:/opt/file.py python:3.7-alpine python /opt/file.py
{'new_new_a': 0, 'new_new_b': 0}

$ docker run --rm -v /home/kaushik/blog/change_keys_during_iteration.py:/opt/file.py python:3.6-alpine python /opt/file.py
{'new_new_a': 0, 'new_new_b': 0}

Apparently, Python pre-3.8 would merely check, at the end of each iteration, if the size of the dictionary remains the same. And since this code just replaces a key with a different key, it maintains the size each iteration.

Another reminder to always use the most recent Python version you can!

#python