Improve heredoc end detection for embedded languages

[reese]

Motivation

Related to¹ #3919

Embedded heredoc patterns (e.g., <<C, <<SQL, <<HTML) could fail to terminate properly when the heredoc content contained characters that start begin/end patterns in the embedded language grammar. This caused syntax highlighting to "leak" into subsequent Ruby code.

When a heredoc with embedded language highlighting contained certain characters (like ? which starts C's ternary operator pattern, or ( which starts a group), the embedded grammar's begin/end pattern would start but never find its closing match. Because the inner heredoc pattern used end to detect the terminator, and end is only checked after nested patterns are processed, the heredoc terminator was never recognized.

Example that failed:

  <<C                                                                                                                                 
  (#{' ' unless foo.empty?})                                                                                                          
  C                                                                                                                                   
                                                                                                                                      
  def bar = "test"  # Incorrectly highlighted as C, not Ruby

Implementation

Changed all 15 embedded heredoc language patterns from using end to while for the inner pattern:

• end behavior: Check terminator after processing nested patterns (gets blocked if nested pattern is stuck open)
• while behavior: Check terminator before processing nested patterns on each line (exits immediately when terminator is found)

The while condition is evaluated at the start of each line before any nested patterns run, so when it fails (line matches terminator), the entire block closes, including any nested patterns that might be open. This prevents the inner syntax highlighting from leaking beyond the closing terminator when it's busted.

Automated Tests

See the added automated test.

Manual Tests

I ran a local VSCode instance with this change to check that tokens after the closing heredoc delimiter were correctly identified as Ruby again.

Before:

After:

Footnotes

1. note that I think there's more to do to close that issue out, see my comment on interpolation highlighting -- this is just a stopgap to prevent things being broken after the heredoc ends ↩