RegEx - Regular Expressions in Python

Regular expressions (also called regex, short for regular expressions) are a language that detects patterns. You write a pattern that will be applied to strings. It is an efficient way to find patterns and extract information from text. In a pattern, each symbol represents a type of information and, in this article, we will see the main symbols and their meanings with the help of several examples.

Python has the re module for regular expressions. The documentation is very detailed and there is also this text with a more step-by-step introduction to the module.

The module has three methods that are widely used and that will be our focus here: match, search and findall.

Match

#

match looks for the given pattern from the start of the string. The signature is re.match(pattern, string, flags=0). The return value is None when the pattern is not found. When it is found, it returns a Match object, which we will discuss throughout the article.

Let’s start by looking for the pattern “Fran” in the string “Francisco Bustamante”, the author of the site:

>>> import re

>>> re.match('Fran', 'Francisco Bustamante')
<re.Match object; span=(0, 4), match='Fran'>

>>> re.match('Fran', 'Bustamante, Francisco') 

# note that the last one returns None because it does not match at the start

span=(0, 4) means that the pattern was found between indices 0 and 4. Remember that counting in Python sequences starts at zero and that the last index is exclusive.

Search

#

search looks for the given pattern along the string. The signature is re.search(pattern, string, flags=0). The return value is None when the pattern is not found. When it is found, it returns a Match object, which we will discuss throughout the article. It is important to stress that it returns the first location where the pattern was found.

>>> re.search('an', 'Francisco Bustamante')
<re.Match object; span=(2, 4), match='an'>

The pattern an appears in the string more than once, but search returns in span only the indices of the first location.

Findall

#

findall looks for the given pattern along the string. The signature is re.findall(pattern, string, flags=0). The return value is an empty list when the pattern is not found. When it is found, it returns a list with each occurrence. It is important to stress that it returns all the occurrences found.

>>> re.findall('an', 'Francisco Bustamante')
['an', 'an']

>>> re.findall('an', 'Chico')
[]

The power of findall will be better explored with some tools that we will learn throughout the article.

Flags

#

The signature of the methods presented has a flags argument. With flags we can modify some aspects of how regular expressions work. See the difference in behavior in the following two examples:

>>> re.match('fran', 'Francisco Bustamante')

>>> re.match('fran', 'Francisco Bustamante', re.IGNORECASE)
<re.Match object; span=(0, 4), match='Fran'>

In the first case, the return value was None, so the interpreter just showed an empty line. That is because the pattern was given with a lowercase first letter and, in the string, it is uppercase. The re.IGNORECASE flag, as the name suggests, disregards the difference between uppercase and lowercase.

Below is a table with the main flags of the re module and their meanings. The letters after each name are the abbreviations that can be used in place of the full name.

Some new words appeared in this table. Don’t worry, they will be explained throughout the article. One of them is metacharacter, our next topic.

The power of regular expressions comes from metacharacters, characters that represent a set of specific characters, general patterns.

The dot metacharacter (.)

#

The . metacharacter represents any character, except a line break. Let’s use it with the match method seen earlier:

>>> re.match('.', 'Francisco Bustamante') 
<re.Match object; span=(0, 1), match='F'>

>>> re.match('.', '42')
<re.Match object; span=(0, 1), match='4'>

>>> re.match('.', ' Francisco Bustamante')
<re.Match object; span=(0, 1), match=' '>

Note in the last example that there was a space at the start of the string and that space was recognized by the metacharacter.

By definition, the . metacharacter should match control characters except \n, which indicates a line break. Let’s see:

>>> re.match('.', '\t\t')  # \t represents TAB
<re.Match object; span=(0, 1), match='\t'>

>>> re.match('.', '\n')

>>> print(re.match('.', '\n'))
None

This behavior of ignoring \n can be modified by one of the flags we saw earlier, DOTALL:

>>> re.match('.', '\n', re.DOTALL)                                              
<re.Match object; span=(0, 1), match='\n'>

We saw earlier that search looks for the pattern along the string and returns the first position where it finds it. Let’s see the behavior with .:

>>> re.search('.', ' Francisco Bustamante')
<re.Match object; span=(0, 1), match=' '>

>>> re.search('.', 'Francisco Bustamante')
<re.Match object; span=(0, 1), match='F'>

>>> re.search('.', '\nFrancisco Bustamante')
<re.Match object; span=(1, 2), match='F'>

>>> re.search('.', '\nFrancisco Bustamante', re.DOTALL)
<re.Match object; span=(0, 1), match='\n'>

The first two examples return the first position, as expected. In the third example, the control character \n is ignored, returning the first character right after it, the letter F. This behavior is modified by the DOTALL flag in the last example.

We saw earlier that findall looks for the given pattern along the string, returning a list with all the occurrences. Let’s combine it with .:

>>> re.findall('.', 'Ciência\nProgramada')                                      
['C', 'i', 'ê', 'n', 'c', 'i', 'a', 'P', 'r', 'o', 'g', 'r', 'a', 'm', 'a', 'd', 'a']

The control character \n was ignored and all the others were returned as a list.

Anchors

#

The ^ symbol represents the start of a string and the $ symbol represents the end of a string. Let’s test with the findall method:

>>> re.findall('^.', 'Ciência\nProgramada\nPython')
['C']

>>> re.findall('^.', 'Ciência\nProgramada\nPython', re.MULTILINE)
['C', 'P', 'P']

The pattern ^. means looking for any character other than a line break at the start of the string. So, in the first case, it returns the letter C. But note that the given string has line breaks. So we can pass the re.MULTILINE flag so that the pattern is searched for in each line. That is why the second example returns a list with the first letter of the string and also the first letter after each \n control character.

We can apply the same logic to the pattern .$, which searches at the end of the string:

>>> re.findall('.$', 'Ciência\nProgramada\nPython')
['n']

>>> re.findall('.$', 'Ciência\nProgramada\nPython', re.MULTILINE)
['a', 'a', 'n']

Some edge cases we can analyze are when we have a string with only one character, or an empty one, or one with a line break:

>>> re.match('^.$', 'a') 
<re.Match object; span=(0, 1), match='a'>

>>> re.match('^$', '')  # the start equals the end, empty string
<re.Match object; span=(0, 0), match=''>

>>> re.findall('^$', '\n', re.MULTILINE)
['', '']

The . metacharacter is very broad; we usually want to be a bit more specific.

Character classes

#

When the pattern has square brackets, they declare a character class. Each character between the brackets will be searched for in the text of the string. Let’s look for lowercase vowels in the string Ciência Programada:

>>> re.findall('[aeiou]', 'Ciência Programada')
['i', 'i', 'a', 'o', 'a', 'a', 'a']

Note that ê was not put in the list, after all it is different from e.

The ^ symbol inside a character class means negation. So, if we are looking for everything except lowercase vowels:

>>> re.findall('[^aeiou]', 'Ciência Programada')
['C', 'ê', 'n', 'c', ' ', 'P', 'r', 'g', 'r', 'm', 'd']

It is also possible to define ranges of values. Searching from “a” to “f”:

>>> re.findall('[a-f]', 'Ciência Programada') 
['c', 'a', 'a', 'a', 'd', 'a']

We can define more than one range. Searching from “a” to “f” and from “A” to “Z”:

>>> re.findall('[a-fA-Z]', 'Ciência Programada') 
['C', 'c', 'a', 'P', 'a', 'a', 'd', 'a']

A very common pattern is to search for all letters and digits and for “_”, since they tend to be the characters most accepted in online form fields such as e-mail, for example:

>>> re.findall('[a-zA-Z0-9_]', 'Ciência Programada')
['C', 'i', 'n', 'c', 'i', 'a', 'P', 'r', 'o', 'g', 'r', 'a', 'm', 'a', 'd', 'a']

It is such a special sequence that there is a shortcut, \w:

>>> re.findall('\w', 'Ciência Programada')
['C', 'i', 'ê', 'n', 'c', 'i', 'a', 'P', 'r', 'o', 'g', 'r', 'a', 'm', 'a', 'd', 'a']

Note two small differences between this result and the previous example. First, square brackets are not needed to use \w. Second, the character ê appears in the result. That is, here Unicode characters are recognized. If you really want only the equivalent of the class [a-zA-Z0-9_], use the re.ASCII flag:

>>> re.findall('\w', 'Ciência Programada', re.ASCII)
['C', 'i', 'n', 'c', 'i', 'a', 'P', 'r', 'o', 'g', 'r', 'a', 'm', 'a', 'd', 'a']

The main special sequences defined by default are:

• \d equivalent to any Unicode digit, which includes [0-9]
• \D equivalent to the negation of \d
• \s equivalent to Unicode whitespace characters, which includes [ \t\n\r\f\v]
• \S equivalent to the negation of \s
• \w equivalent to characters that can be used in text in general, which includes [a-zA-Z0-9_]
• \W equivalent to the negation of \w

When the re.ASCII flag is used, each case above is restricted to the bracket representation shown. In Unicode they are broader.

Raw strings

#

All special sequences use the \ symbol. This is problematic in Python, because strings accept control characters in the language. Literal strings evaluate the character after the backslash to check whether it is a control character or not. Let’s see an example:

>>> print('1\n2')
1
2

When we want to indicate that it should not be treated as a control character, we use another backslash to signal escape:

>>> print('1\\n2')  # escape of the second backslash
1\n2

This behavior related to backslashes can be very problematic in some contexts. For example, in science LaTeX is widely used to produce documents. And LaTeX environments are delimited with commands that use \. For example, \begin{equation}...\end{equation} delimits an environment for a mathematical equation. The \section command indicates the start of a section in a document. Let’s see how the Python interpreter recognizes the command followed by a line break:

>>> '\section\n' # \s has no meaning in Python
'\\section\n'
>>> text = '\section\n'  # LaTeX
>>> print(text)
\section

Note that, since \s has no meaning for the Python interpreter, it automatically adds a backslash to signal that this character should be ignored. When we use print, the text of the command appears normally, followed by a blank line.

We can verify that control characters are counted as a single character by checking the length of the string stored in the variable text:

>>> len(text)
9

There are 8 characters in “\section” and the ninth is the control character \n.

In theory, there should be a match when searching for \\section in the string stored in text:

>>> print(re.match('\\section', text))  # \s is a special sequence in regex
None

But, as we saw in the previous section, \s has a meaning in the context of the re module. So we need to indicate that the backslashes must be escaped:

>>> print(re.match('\\\\section', text))
<re.Match object; span=(0, 8), match='\\section'>

Welcome to what we call the backslash plague! But calm down, there is a way to avoid all those backslashes. In Python, there are what we call raw strings. In this type of string, denoted by an r before the quotes, backslashes are not treated in any special way. Notice the difference:

>>> len('\n')
1

>>> len(r'\n')
2

In the raw string, we have two characters, “" and “n”, whereas in the normal (literal) string there is only one character, which represents a line break.

Back to our example, we just need to use a raw string in the pattern:

>>> print(re.match(r'\\section', text)) # raw string, meaning there is no control character
<re.Match object; span=(0, 8), match='\\section'>

So, a very important tip: use raw strings whenever there is a backslash that should not be interpreted as a special sequence.

Using the pipe

#

The pipe | means or, indicating alternatives in the regular expression. See examples:

>>> re.search('a|b', 'abc')
<re.Match object; span=(0, 1), match='a'>

>>> re.search('a|b', 'bcd')
<re.Match object; span=(0, 1), match='b'>

>>> re.search('a|b', 'cde')

Note in the first example that, even though “b” exists in the string, since “a” was found first only the position of “a” was returned. With findall, both are returned:

>>> re.findall('b|a', 'abc')
['a', 'b']

Repetitions

#

Searching for repetitions is one of the most common reasons for using regular expressions. Let’s look at the various possible forms.

Specific quantities

#

Let’s check the behavior of searching for the pattern \d{4}, which looks for any digit four times.

>>> re.match(r'\d{4}', '1234')                 # string with 4 digits
<re.Match object; span=(0, 4), match='1234'>

>>> re.match(r'\d{4}', '123')                  # string with 3 digits

>>> re.match(r'\d{4}', '12345')                # string with 5 digits
<re.Match object; span=(0, 4), match='1234'>

In the examples, it is clear that, when there are fewer than four digits, the return value is None. In the other cases, it returns the first four digits. The same happens with the search method:

>>> re.search(r'\d{4}', 'abc123def12345')
<re.Match object; span=(9, 13), match='1234'>

The first segment 123 was ignored because it has fewer than 4 digits. The second segment, which has 5 digits, was considered and returned the first four digits.

Minimum and maximum quantity

#

Using a comma indicates that the value being searched for is the minimum. That is, if there are more digits the behavior will be greedy and will return the digits beyond the minimum. If there are fewer digits than the minimum, the return value will be None.

>>> re.match(r'\d{2,}', '12')
<re.Match object; span=(0, 2), match='12'>

>>> re.match(r'\d{2,}', '12345')  # greedy
<re.Match object; span=(0, 5), match='12345'>

>>> re.match(r'\d{2,}', '1')

Using the ? symbol after the braces turns the behavior into lazy, so this symbol is a repetition modifier.

>>> re.match(r'\d{2,}?', '12345')  # lazy, minimum possible
<re.Match object; span=(0, 2), match='12'>

A value after the comma indicates the maximum. The previous explanations remain valid.

>>> re.match(r'\d{2,4}', '12345')
<re.Match object; span=(0, 4), match='1234'>

>>> re.match(r'\d{2,4}', '123')
<re.Match object; span=(0, 3), match='123'>

>>> re.match(r'\d{2,4}', '12')
<re.Match object; span=(0, 2), match='12'>

>>> re.match(r'\d{2,4}', '1')

>>> re.match(r'\d{2,4}?', '12345')  # from greedy to lazy
<re.Match object; span=(0, 2), match='12'>

0 or 1 occurrence, optional element

#

Searching for an optional element in the string is, in fact, a special case of minimum and maximum quantity, with a minimum of zero and a maximum of one.

>>> re.match(r'\d{0,1}', '12345')
<re.Match object; span=(0, 1), match='1'>

>>> re.match(r'\d{,1}', '12345')  # 0 can be omitted
<re.Match object; span=(0, 1), match='1'>

The ? symbol after a regular expression has the same effect as {,1}. So:

>>> re.match(r'\d?', '12345') 
<re.Match object; span=(0, 1), match='1'>

But we have already seen that the same symbol turns a greedy search into a lazy one. So the following expression returns an empty string, because the minimum is no (0) occurrences:

>>> re.match(r'\d??', '12345') 
<re.Match object; span=(0, 0), match=''>

Let’s go step by step. The first question mark is a repetition modifier of the regular expression immediately before it, looking for 0 or 1 occurrence of \d. The second question mark is a modifier of the repetition operator, turning it lazy.

0 or more times

#

Another special case of minimum and maximum. It also has a special symbol, *. Look at the examples:

>>> re.match(r'\d{0,}', '12345')
<re.Match object; span=(0, 5), match='12345'>

>>> re.match(r'\d{,}', '12345')  # 0 can be omitted
<re.Match object; span=(0, 5), match='12345'>

>>> re.match(r'\d*', '12345')  # symbol
<re.Match object; span=(0, 5), match='12345'>

>>> re.match(r'\d*?', '12345')  # lazy
<re.Match object; span=(0, 0), match=''>

>>> re.match(r'\d*', 'abc')
<re.Match object; span=(0, 0), match=''>

In the last example, the match is an empty string, because the minimum is no occurrences.

1 or more times

#

Another special case of minimum and maximum. It also has a special symbol, +. Look at the examples:

>>> re.match(r'\d{1,}', '12345')
<re.Match object; span=(0, 5), match='12345'>

>>> re.match(r'\d+', '12345')  # + requires at least one occurrence, being greedy
<re.Match object; span=(0, 5), match='12345'>

>>> re.match(r'\d+?', '12345')  # turns it lazy
<re.Match object; span=(0, 1), match='1'>

>>> re.match(r'\d+', 'abc')

In the last example, it returns None, because the minimum is one occurrence and there are no digits in the string.

Understanding the importance of controlling repetitions

#

In case everything has seemed too abstract so far, let’s start putting in some situations closer to the real world. Consider the string below, from which we would like to extract all the attribute values.

>>> text = 'name="Francisco" site="Ciência Programada"'

Since the values are between double quotes, at first we might imagine the pattern r'".+"', since . would grab the characters, with the + modifier indicating one or more times. But look at the result:

>>> re.findall(r'".+"', text)
['"Francisco" site="Ciência Programada"']

It did not work because we matched anything between quotes with at least one occurrence. The search extends from the first double quote to the last. In fact, we want what is between each pair of quotes. So we must turn the search lazy:

>>> re.findall(r'".+?"', text)
['"Francisco"', '"Ciência Programada"']

Now we have what we wanted. But it is still not a good way to get the values. Look at what would happen if empty fields were passed:

>>> text = 'name="" site=""'

>>> re.findall(r'".+?"', text)
['"" site="']

It returns the wrong result because + requires at least one occurrence. In fact, we want 0 or more occurrences, which we can solve using *:

>>> re.findall(r'".*?"', text)
['""', '""']

Understanding the Match Object

#

In several examples we saw that the result is a Match object. Let’s understand this object a bit better. There are four important methods for the level of this article:

>>> m = re.match(r'\d+', '12345')

>>> type(m)
re.Match

>>> m.group()  # returns the string that was matched
'12345'

>>> m.start()  # start position of the match
0

>>> m.end()    # end position of the match
5

>>> m.span()   # tuple with the start and end position of the match
(0, 5)

The group method will be important for the next section.

Capture groups

#

Sometimes the string being analyzed has several fields from which we want to extract data. We can then define capture groups. Groups are delimited by parentheses and we can repeat the content of a group with a repetition qualifier, such as *, +, ?, or {m,n}. For example, (ab)* will match zero or more repetitions of ab.

Consider the HTML tag below, from which we want to extract the name (input) and the values of type, id and name. We can create a pattern, the pattern variable in the code, following the format of the string. In each group we want to grab one or more occurrences of characters, lazily, to avoid the problems seen in the section on controlling repetitions:

>>> html = '<input type="text" id="id_cpf" name="cpf">'

>>> pattern = r'<(.+?) type="(.+?)" id="(.+?)" name="(.+?)"'
# any character before a space for the tag name, likewise between quotes for the other groups

>>> m = re.match(pattern, html)  # storing the match result in a variable

>>> m
<re.Match object; span=(0, 41), match='<input type="text" id="id_cpf" name="cpf"'>

The variable m stores the result, which is a Match object. By default, it shows the complete match, but we can explore more details of the object. The groups method shows all the extracted groups and group lets us return specific groups:

>>> m.groups()  # capture groups
('input', 'text', 'id_cpf', 'cpf')

>>> m.group(0)  # entire match
'<input type="text" id="id_cpf" name="cpf"'

>>> m.group(1)  # first group
'input'

>>> m.group(2, 1, 3)  # specific groups
('text', 'input', 'id_cpf')

Now suppose that the order of the HTML tag attributes can change:

>>> html1 = '<input type="text" id="id_cpf" name="cpf">'

>>> html2 = '<input id="id_cpf" name="cpf" type="text">'

Clearly, our previous pattern will not work, since it depended heavily on the order of the attributes. We have already seen the | metacharacter, which indicates an alternative, something that will be useful here because we want to grab each attribute regardless of order.

Another concept we will use is the non-capturing group, represented by (?:...), replacing the ellipsis with a regular expression. To understand, consider the following examples:

>>> m = re.match('name="(.+?)"', 'name="Francisco"')                         

>>> m.groups()
('Francisco',)

>>> m = re.match('name="(?:.+?)"', 'name="Francisco"')

>>> m.groups()
()

In the first case, we used the pattern in the group to get the value “Francisco” and in the second we excluded that group.

We can use this pattern alternately to get each group of the HTML tag regardless of order:

>>> pattern = r'<(.+?) (?:(?:type="(.+?)"|id="(.+?)"|name="(.+?)") ?)*'
# the outer group indicates the presence or not of a space

>>> m = re.match(pattern, html1)

>>> m
<re.Match object; span=(0, 41), match='<input type="text" id="id_cpf" name="cpf"'>

>>> m.groups()
('input', 'text', 'id_cpf', 'cpf')

>>> m = re.match(pattern, html2)

>>> m
<re.Match object; span=(0, 41), match='<input id="id_cpf" name="cpf" type="text"'>

>>> m.groups()
('input', 'text', 'id_cpf', 'cpf')

Named groups

#

Finally, when there are several groups, it is useful to name them. Python has a specific way to indicate the name of each group: ?P<name>.

>>> pattern = r'<(?P<tag>.+?) (?:(?:type="(?P<type>.+?)"|id="(?P<id>.+?)"|name="(?P<name>.+?)") ?)*'

>>> m = re.match(pattern, html1)

>>> m.groups()
('input', 'text', 'id_cpf', 'cpf')

When named groups are used, there is a very useful method, groupdict, which returns the group name and the value as a dictionary:

>>> m.groupdict() 
{'tag': 'input', 'type': 'text', 'id': 'id_cpf', 'name': 'cpf'}

>>> m = re.match(pattern, html2)

>>> m.groups()
('input', 'text', 'id_cpf', 'cpf')

>>> m.groupdict()
{'tag': 'input', 'type': 'text', 'id': 'id_cpf', 'name': 'cpf'}

Conclusion and an IMPORTANT note

#

The important note is: just because something is possible to solve with regular expressions does not mean it should be solved with regular expressions.

In Python specifically, string manipulations involving substitutions are usually more effective with string methods such as replace. The documentation itself makes this clear.

Likewise, even though I used the HTML tag case as an example, there are many online discussions about more efficient ways to extract information from HTML and why regex is generally not the best option. See here, here and here. Get ready for heated discussions. And use good judgment to know when using it is acceptable and when another tool should be used.

Regular expressions are useful, but they can be very hard to read and debug. Try to keep them as short and as specific as possible. It is a much broader subject than what was presented here; I tried to include what I consider a good start and, obviously, what is within my knowledge. It is certainly a very valuable tool to know.

See you next time!