Diving Deeper in Vim Regular Expressions

This is the second article of a series about regular expressions in Vim:

We’ve seen, in the first article, general metacharacters we can use in our Vim regexes, as well as equivalent Perl-style regexes we can use with other tools (like GNU grep). This time, we’ll dive deeper in Vim’s regex engine by looking at more metacharacters we can use in our favorite editor.

As a reminder, I don’t advise writing 32908309 regexes in your codebases, but instead using regexes with Vim (or other tools) for one-off tasks. A codebase is often a system in constant evolution, and regexes can ramp up in complexity fairly quickly.

We’ll see, in this article:

• How to do some substitutions in Vim using Perl-style regexes.
• What is the “level of magic” of a regex in Vim, and what “level of magic” to choose.
• Some characters class shorthands relying on Vim options.
• What are lookaround assertions in Vim and Perl-style regex engines, and how to use them.
• How to match a pattern on multiple lines.
• How to match a pattern in a visual selection.

I recommend you to actively follow along by trying the different regexes directly in Vim. There are also many exercises throughout the article; you can try to solve them using this example file. Only trying will help you understand and memorize the different metacharacters, even if you don’t try very hard!

Are you ready to dive into the depths of Vim’s regular expressions?

Vim and Perl Regex Engines

We’ve seen, in the first article of this series, some differences between Vim’s regex engine and a Perl-style regex engine (using the PCRE implementation of GNU grep). We can also look at Vim’s help to see the differences between both engines at a glance; to do so, simply run :help perl-patterns.

That’s not all: we can also use Perl’s regex engine directly in Vim, thanks to the :perldo command. It can be useful if you really don’t want to deal with the quirks of Vim’s regexes in your substitutions.

For example, if you want to substitute “regex” with “replacement”, you can run:

:perldo s/regex/replacement/g

You’ll need to have Perl installed on your system for this command to work. If you use Neovim, you’ll also need to install the Perl provider. You can run :checkhealth provider to make sure the provider is indeed available.

It’s a Kind of Magic

The first article of this series left an elephant in the room : the “level of magic” for Vim regexes. Let’s address now this oddity.

I was recommending prefixing our Vim regexes with \v (making the regex “very magic”) to be able to use every metacharacter available. What does this “very magic” means? What mechanism is used here?

First, a little reminder: some characters can act as metacharacter in a regex pattern. For example, the character full stop (or dot) . can be interpreted by our regex engine as a literal character; in that case, it will match a dot in your text. It can also be interpreted as a metacharacter; in that case, the full stop means “any character”.

Depending on the “level of magic” of the regex, we might need to escape some literal characters to use them as metacharacters, or to escape some metacharacters to use them as literal characters. To escape characters, you simply need to add a backslash in front of it (i.e. \.).

As a rule, I always try to avoid escaping any character (literal or metacharacters) in my regexes. Backslashes make the pattern more difficult to read and understand. As a result, I often change the “level of magic” of my regexes depending on what I need: do I want to match more literal characters, or do I want to use mostly metacharacters?

From there, we can ask ourselves many questions:

1. What “levels of magic” forces us to escape characters to use their metacharacter counterparts?
2. What “levels of magic” forces us to escape metacharacters to use their literal character counterparts?
3. Do we have to escape all characters (or metacharacters), or only some of them?
4. How do we change this level of magic for a specific regex?

We can already answer the last question: we need to add a prefix to a regex in order to change its “level of magic”. The following table tries to answer the other questions:

By default, if we don’t add any of these prefixes to our regexes, they’re considered “magic”. That’s a shame, because I think that “magic” and “nomagic” are quite confusing: for the first, you need to escape some (but not all) metacharacters to match the literal ones, and for the second it’s the contrary. It’s not consistent, because not all metacharacters (or literal characters) need to be escaped; only some of them, and you need to learn them by heart.

As a result, I try to avoid “magic” and “nomagic” as much as possible; I always try to add the prefix \v (for “very magic”) or \V (for “very nomagic”) to my regexes.

For “very magic” patterns, most characters are considered as metacharacters by the regex engine, without the need to escape anything. It’s useful when you need to use many metacharacters in your regex. The downside: you’ll need to escape every literal character you want to match.

If you want to mostly match literal characters instead of relying on metacharacters, you can use “very nomagic”. This time, most characters will be interpreted as literal characters by the regex engine, without the need to escape anything. The downside: you have to escape any metacharacter you want to use.

To drive the point home, here are some equivalent regexes using different “level of magic”: “magic”, “very magic”, and “very nomagic”. The first regex tries to match a year with four digits, and the second one tries to match a literal string { Mouseless }.

Let’s solidify our understanding with an exercise: using Vim’s search in our example file, how would you match all HTML attributes rel, their values, and the quotes surrounding them? Would you use a “very magic” or a “very nomagic” pattern?

/\vrel\=".*"

/\Vrel="\.\*"

/\vrel\=".{-}"

/\Vrel="\.\{-\}"

Here’s another exercise: how would you match any text surrounded by literal stars *? Would you use “very magic” or “very nomagic”?

/\v\*.+\*

/\V*\.\+*

Character Classe Shorthands and Vim Options

We’ve already seen some shorthands for character classes in the last article. There are more available, specific to Vim; the characters they include depend on the value of some Vim options:

If you look at the value of these options, you might see weird ranges of number. For example, if I run :set isfname?, I’ll get @,48-57,/,.,-,_,+,,,#,$,%,~,=. The range 48-57 means that the ASCII characters from 48 to 57 are included (the digits 0 to 9). The characters separated by commas , are also included in the character class.

You can learn more about Vim options in this other article I wrote. Another tip: you can have access to a fancy ASCII table by running man ascii in your terminal.

I’ll finish this short section with a word of caution: these options are sometimes used by other commands or plugins, and changing their values can have unforeseen consequences. For example, the keystroke gf use the option isfname under the hood.

Vim Lookaround Assertions

What if we want to match a pattern only if it’s before (or after) another pattern? It’s where lookaround assertions enter the chat.

There are two sorts of these assertions:

1. Positive lookaround: the pattern you want to match needs to be before (or after) another pattern.
2. Negative lookaround: the pattern you want to match shouldn’t be before (or after) another pattern.

We’ll first look at the handiest metacharacters we can use for positive lookaround assertions; then, we’ll talk about other metacharacters for both positive and negative lookaround.

Start and End of the Matched Pattern

We can use two different atoms to mark the beginning or the end of the pattern we want to match:

Here’s an exercise: using Vim’s search and our example file (as always), how would you match all non-empty strings surrounded by double quotes " and beginning by the string application\? We don’t want to match application\ however, just what’s after.

For example, for the attribute type="application/ld+json", we want to match ld+json.

/\v"application\/\zs.{-1,}"

Lookahead and Lookbehind

There are other atoms we can use in Vim to look around (ahead or behind) the pattern we want to match. Here’s the complete list (assuming, as always, that you’re in “very magic” mode):

The two strings <pattern> and <match> are just placeholders here, you should replace them by what you want.

It’s time for another exercise! Still using the example file, Vim’s search, and the lookaround and lookbehind assertion described above, how would you match every pattern tag followed by the pattern small (the string small prefixed with a space)? Another constraint: the match should not be after the pattern article:.

For example, section:tag small should match, or even tag small, but not article:tag small.

/\v(article:)@<!tag(\ssmall)@=

Equivalent Perl-style Metacharacters

Lookahead and lookbehind assertions are not unique to Vim: you can also use them in Perl-style regexes, like PCRE.

Here’s a table summarizing the equivalent syntaxes:

As you can see, the two syntaxes have some similarities; learning one can help to memorize the other.

Let’s practice with this exercise: using GNU grep with the PCRE engine (using the option -P), how would you solve the previous exercise above?

grep -P '(?!article:)tag(?=\ssmall)' example.html

/\v(article:)@<!tag(\ssmall)@=

:perldo s/(?!article:)tag(?=\ssmall)/another-thing/

The PCRE metacharacter \K can also be used for lookbehind assertions. In fact, you have to use it if you want have an assertion which is not fixed in length.

For example, let’s say that you only want to output the content of your <h1> tag. You could try something like this:

grep -Po '(?<=<h1.*>).*(?=</h1>)' example.html

It doesn’t work, warning you that lookbehind assertion is not fixed length. It’s because you use the quantifier * in the assertion itself. But the following works:

grep -Po '<h1.*>\K.*(?=</h1>)' example.html

Regexes Matching On Multiple Lines

Until now, every regex we’ve written are limited to line. We can match the same pattern on different lines, but we can’t yet create a pattern which match multiple lines.

There are other metacharacters we can use in Vim for that purpose. Many of them are similar to the metachacters operating on a single line, except that they’re prefixed with \_. It means that these metacharacters will also match end-of-lines:

We can also prefix character class shorthands with \_. For example, if you want the character class \s (for whitespaces) to match multiple lines (that is, it includes also end-of-lines), you can use \_s.

Let’s try to solve some exercises, shall we? Using Vim’s search in our example file, how would you select the opening and closing HTML tag <head>, as well as every line in between?

/\v\<head\>\_.*\<\/head\>

Another exercise: how would you match HTML link tags a and their possible attributes, but only when they don’t have any inner HTML? That is, there is nothing except possible spaces, tabs, or newlines between the opening tag and the closing one.

/\v\<a.*\>\_s+\<\/a\>

If you’re not tired of exercises yet, here’s another one: what about matching list tags li? More precisely, we want to match the opening tag (and its possible attributes), the closing tag, and everything else in between. Let’s add more constraints: we don’t want anything else on the line of the opening tag, and we don’t want to match list tags which are only on one line.

For example, the following is a valid match:

<li class="nav-opened" role="presentation">
    <a href="https://thevaluable.dev/tags/fundamentals">Fundamentals</a>
</li>

The following is not a valid match, because the opening tag is not alone on its line; there’s also the string “Some content on the same line”.

<li class="nav-opened" role="other">Some content on the same line
    <a href="https://thevaluable.dev/tags/others">Others</a>
</li>

/\v\<li [^<]+\>\_$\_.{-}\<\/li\>

A last exercise: how would you match the first opening HTML link tag a and its possible attributes in our example file?

/\v%^\_.{-}\zs\<a.{-}\>

Only Matching the Visual Selection

What about matching a pattern, but only if it’s visually selected? Here’s what you need:

Let’s look at a real-life example: when I try to rename a file in my shell (Zsh), I often end up with this kind of command:

mv my-file-name.jpg my-file-name.jpg

I simply type mv in the shell, and then I use Zsh completion to get the filename I want to rename. I have it two times to modify the second filename, instead of typing it entirely. I then use a keystroke to directly edit this command in Vim.

The goal here is to rename the second string my-file-name.jpg to my_file_name.jpg, replacing the hyphen - with underscores _. I could do that manually, but I think using a substitution is easier and less prone to errors.

Here’s what I tried at the beginning:

1. Switch to VISUAL mode.
2. Select the second my-file-name.jpg.
3. Run the following:

:'<,'>s/\V-/_/g

Too bad: it doesn’t work. This substitution will replace every hyphen on the line because of the g flag. It doesn’t matter what’s selected: the substitution command will always operate on the whole line (if no other range is specified). If I don’t use the g flag, it will only substitute the first hyphen.

It’s where %V can be useful:

:'<,'>s/\v%V-/_/g

It works as expected: the pattern only match what I’ve selected in VISUAL mode, thanks to the atom %V.

Here’s the last exercise: on line 74 of the example file, how would you substitute nav-and-other with nav_and_other, without modifying any other attribute in the h3 tag?

:'<,'>s/\v%V-/_/g

:'<,'>s/\%V-/_/g

Ready for Vim’s Regexes?

We’ve seen, throughout this series, the most important metacharacters we can use in our Vim regexes, from the basics to the more complicated ones. Now, instead of writing a script to parse some plain text files, you can directly open them in Vim and use some good regexes! You’ll also be able to use tools using a Perl-style regex engine; it’s not that different, in principle, than the Vim regexes!

So, what did we see in this article?

• We can make some substitutions using Perl-style regexes directly in Vim, thanks to the Ex command :perldo. Handy if you’re allergic to Vim’s regex engine.
• A Vim regex can have different levels of magic: “very magic” (using the prefix \v) gives you all metacharacters without the need to escape anything, “very nomagic” gives you all literal characters without the need to escape anything.
• There are a couple of Vim character class shorthands which depends on the value of some Vim options.
• We can try to match a pattern if another pattern is just before or after. To do so, we can use \zs (before the start of the pattern for a lookbehind assertion) or \ze (after the end of the pattern for a lookahead assertion).
• Lookaround assertions are not exclusive to Vim: we can use some similar syntax with a Perl-style regex engine, like PCRE.
• Vim introduced, in Vim 6, the possibility to match a pattern on multiple lines. Most of these new atoms begins with \_.
• We can try to match our regexes in a visual selection only, using the atom %V (\%V if we’re not in “very magic” mode).

I’m curious: what are the most useful regexes you’re often using in Vim? Don’t hesitate to share with the community in the comment section just below! You know, sharing is caring.