File.dirname: add a spec for Shift JIS handling #1330

byroot · 2026-01-19T05:51:31Z

While trying to speedup various File.* methods, I realized they were way slower and complicated than they should for no apparent reason. However after asking Nobu he explained that Shift JIS encoded text can contain 0x5C (ASCII backslash) as the second byte of a two byte character sequence.

Since on Windows 0x5C is File::ALT_SEPARATOR, this can easily break naive path related algorithms searching for directory separators.

cc @eregon what do you think of that spec? I think it would have helped me figure things out way sooner, so would have been valuable. If you agree I can generalize it to more "path handling" methods.

core/file/dirname_spec.rb

While trying to speedup various `File.*` methods, I realized they were way slower and complicated than they should for no apparent reason. However after asking Nobu he explained that Shift JIS encoded text can contain `0x5C` (ASCII backslash) as the second byte of a two byte character sequence. Since on Windows `0x5C` is `File::ALT_SEPARATOR`, this can easily break naive path related algorithms searching for directory separators.

core/file/dirname_spec.rb

eregon · 2026-01-20T12:18:22Z

Thanks! Nasty edge case indeed 😅

byroot · 2026-01-20T12:32:36Z

Yep. I can add it to more File.* methods if you'd like.

eregon · 2026-01-20T13:04:46Z

Yep. I can add it to more File.* methods if you'd like.

Yeah that'd be great for File methods which "split" paths.
Potentially could reduce to just one example per method on Windows, since that's the tricky one, on non-Windows I think ShiftJIS isn't any special since it's split on / and not on \.

The extra ASCII-compatible/or-not examples are good to have too.

byroot · 2026-01-20T13:16:12Z

Yeah that'd be great for File methods which "split" paths.

Ok, I'll add some as I go when I optimize these methods (like: ruby/ruby#15898, ruby/ruby#15902, ruby/ruby#15912)

Potentially could reduce to just one example per method on Windows, since that's the tricky one, on non-Windows I think ShiftJIS isn't any special since it's split on / and not on \.

Indeed. But since the same spec should pass on both I figured it was simpler not to restrict it.

eregon · 2026-01-20T13:21:10Z

Ok, I'll add some as I go when I optimize these methods (like: ruby/ruby#15898, ruby/ruby#15902, ruby/ruby#15912)

Thanks. Feel free to just add them directly as part of the ruby/ruby PRs for convenience. I think they don't need extra reviews given it would be similar to this PR.

byroot force-pushed the shiftjs-windows-path-operations branch 2 times, most recently from e8c8ec8 to 7cfe29b Compare January 19, 2026 05:53

trinistr reviewed Jan 19, 2026

View reviewed changes