Furlo's Playground

Táto stránka je iba na čítanie. Môžete si prehliadnuť zdrojový kód, ale nemôžete ho meniť. Opýtajte sa správcu, ak si myslíte, že niečo nie je v poriadku.
====== Unicode NFC normalisation for Rclone on macOS ======

Apple devices create all filenames in Unicode //Decomposed Normalisation Form// (NFD), while every other major OS uses //Composed Normalisation Form// (NFC). This makes **//you//**, as a Mac user, the bad guy, because **//it is you who is incompatible with the rest of the world//**.

In a nutshell, the problem is this: **//Whenever you create files with diacritics they will be copied to other devices with filenames stored as decomposed strings//**. This is a non-standard for these OS'es, and you never know [[https://medium.com/@sthadewald/the-utf-8-hell-of-mac-osx-feef5ea42407|what problems that will cause]].

This article presents my way of solving the problem by **configuring Rclone to create all files in NFC (composed form) instead of NFD (decomposed form)** – which is not at all that straightforward is it would seem.



===== Direct way to solve the problem =====
TL;DR: If you just want to solve the problem without actually delving into the problem and its technical details, simply follow the steps below. Otherwise, head over to [[#Technical background]].

**Prereqs:** download [[https://github.com/rclone/rclone/releases|Rclone]] and [[https://github.com/macfuse/macfuse|macFUSE]].((I have no idea whether this process works at all with [[https://github.com/macos-fuse-t/fuse-t|FUSE-T]], the kext-less implementation of FUSE.)) **Important note: the latest version of macFUSE with which this whole workaround works is [[https://github.com/macfuse/macfuse/releases/tag/macfuse-4.9.1|4.9.1]].** Version [[https://github.com/macfuse/macfuse/releases/tag/macfuse-4.10.0|4.10.0]]+ changes the way macFUSE handles encoding at it no longer uses iconv.

  - **Download the [[https://github.com/fumiyas/libiconv-utf8mac|custom-made iconv library]]** – this is actually Apple's own version of the library (which you have on your macOS), but with ''iconv'' base updated to the latest version and with support for [[http://zuga.net/articles/text-what-is-a-unicode-surrogate-pair/|surrogate pairs]] (this is actually important, because for example all emojis are surrogate pairs of characters).
  - **Tweak this library yourself, because it is not //fully// compatible with Apple's version**. Specifically, open the file ''./include/iconv.h.in'' and comment out these 6 code blocks (I list the lines below already commented):
    - lines 69–71:<code C>
//#ifndef LIBICONV_PLUG
//#define iconv_open libiconv_open
//#endif</code>
    - lines 79–81:<code C>
//#ifndef LIBICONV_PLUG
//#define iconv libiconv
//#endif</code>
    - lines 85–87:<code C>
//#ifndef LIBICONV_PLUG
//#define iconv_close libiconv_close
//#endif</code>
    - line 129:<code C>
//#define iconv_open_into libiconv_open_into</code>
    - line 134:<code C>
//#define iconvctl libiconvctl</code>
    - line 214:<code C>
//#define iconvlist libiconvlist</code>
  - **Compile and install the double-tweaked library** using the following commands:<code bash>
make -f Makefile.utf8mac autogen
./configure --prefix=/usr/local
make
make install
</code> These are [[https://github.com/fumiyas/libiconv-utf8mac?tab=readme-ov-file#building-libiconv-program-and-libraries-from-the-git-repository|stated here]] by author of the tweaked version and are similar to the [[https://www.gnu.org/software/libiconv/|building commands]] of the original GNU ''libiconv''. However, I added the ''--prefix=/usr/local'' parameter to the ''configure'' command (present in the GNU version, but not in the tweaked version), since I wasn't sure where the library would put itself without it.
    * Note that for the first building step, you need ''autoconf'' & ''automake'' (install it e.g. through MacPorts by running ''sudo port install autoconf automake'')
  - Now you have two Apple-style ''libiconv'' binaries on your system: the original (and old) Apple one in ''/usr/bin/iconv'' and the tweaked (and updated) one in ''/usr/local/bin/iconv''.
    - First, **test that the new binary itself works with the following conversions:**<code bash>
echo 🙂 | /usr/bin/iconv -f utf-8 -t utf-8-mac
> �
echo 🙂 | /usr/local/bin/iconv -f utf-8 -t utf-8-mac
> 🙂
</code>
    - Second, **test that the new dynamic library has a proper symbol table.** If everything went right, typing the following command:<code bash>
nm -gU /usr/local/lib/libiconv.2.dylib
</code> should output something similar to left column, and **not** to the right column (the order of lines might wary, the symbol names are important): <columns 100% 50% 50%>**<color green>Correct</color> (Apple-style) symbol table**<code bash>
00000000000e3290 D __libiconv_version
0000000000002ce0 T _iconv
0000000000003430 T _iconv_canonicalize
0000000000002d10 T _iconv_close
00000000000016b0 T _iconv_open
0000000000002d20 T _iconv_open_into
0000000000003160 T _iconvctl
0000000000003270 T _iconvlist
0000000000015eb0 T _libiconv_set_relocation_prefix
0000000000015dd0 T _locale_charset
</code><newcolumn>**<color red>Incorrect</color> (GNU-style) symbol table**<code bash>
00000000000e3290 D __libiconv_version
0000000000002ce0 T _libiconv
0000000000003430 T _iconv_canonicalize
0000000000002d10 T _libiconv_close
00000000000016b0 T _libiconv_open
0000000000002d20 T _libiconv_open_into
0000000000003160 T _libiconvctl
0000000000003270 T _libiconvlist
0000000000015eb0 T _libiconv_set_relocation_prefix
0000000000015dd0 T _locale_charset
</code></columns>
  - Now, you need to **update the Fuse library to search for the dynamically-loaded ''libiconv'' library on the new place**:
    - First, check that Fuse actually looks for the library under ''/usr/lib/'':<code bash>
otool -L /usr/local/lib/libfuse.2.dylib
/usr/local/lib/libfuse.2.dylib:
  /usr/local/lib/libfuse.2.dylib (compatibility version 12.0.0, current version 12.9.0)
  /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
  … [other libraries]
</code>
    - Now, change the path by using ''install_name_tool'':<code bash>
sudo install_name_tool -change /usr/lib/libiconv.2.dylib /usr/local/lib/libiconv.2.dylib /usr/local/lib/libfuse.2.dylib
</code>
    - Finally, check that the change is successful:<code bash>
otool -L /usr/local/lib/libfuse.2.dylib
/usr/local/lib/libfuse.2.dylib:
  /usr/local/lib/libfuse.2.dylib (compatibility version 12.0.0, current version 12.9.0)
  /usr/local/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
  … [other libraries]
</code>
  - Done! From now on, whenever you create new file or update the name of existing one, Rclone will produce its filename in Unicode composed form.


==== Using macFUSE supplied by MacPorts ====
If you prefer to use macFUSE version built from [[https://ports.macports.org/port/macfuse/details/|MacPorts archive]] (which I recommend – because with that you can also use MacPorts [[https://ports.macports.org/port/rclone/details/|versions of Rclone]] already unsupported by your macOS system), you will need to do two more steps to have Rclone find and use macFUSE:

  - Link macFUSE filesystem, as suggested by MacPorts itself after the installation:<code Bash>
sudo ln -fsn /opt/local/Library/Filesystems/macfuse.fs /Library/Filesystems/macfuse.fs</code>
  - Link also macFUSE library, as suggested by [[https://github.com/rclone/rclone/issues/6807#issuecomment-1506312510|user on GitHub]]:<code Bash>
sudo ln -s /opt/local/lib/libfuse.2.dylib /usr/local/lib/libfuse.2.dylib</code>

That's it, now everything should work correctly – without the need to install stand-alone macFUSE!



===== Technical background =====

==== Historical aspect and the situation now ====

Historically, the way of encoding filenames((//filenames// – that is, file- or folder-names. I will use the term "file" to mean any [[https://en.wikipedia.org/wiki/Inode|inode]], whether it is a file or a directory.)) on Mac OS X started to differ from other operating systems when Apple switched from HFS to [[https://en.wikipedia.org/wiki/HFS_Plus#Design|HFS+]] file system in 1998. [[https://web.archive.org/web/20140812023313/http://developer.apple.com/library/ios/documentation/MacOSX/Conceptual/BPInternational/Articles/FileEncodings.html|HFS+ uses Unicode 3.2]] to encode filenames and Unicode allows four different ways of storing these filenames on disk – so called [[https://www.unicode.org/reports/tr15/#Norm_Forms|"normalisation forms"]].

But while HFS+ [[https://developer.apple.com/library/archive/technotes/tn/tn1150.html#CanonicalDecomposition|enforces canonical decomposed normalisation form (NFD)]] for filenames, virtually all other common filesystems treat filenames simply as sequences of bytes – this is true both [[http://nedbatchelder.com/blog/201106/filenames_with_accents.html#comment_12842|for Linux distros]] (ext*, ReiserFS) and [[https://stackoverflow.com/questions/7041013/unicode-normalization-in-windows|for Windows]] filesystems (FAT with [[https://en.wikipedia.org/wiki/Long_filename|Long filenames support]], NTFS).

However, **standard filename-manipulation libraries on both Linux and Windows normalize filenames to composed normalisation form (NFC)**, so while it is technically possible to create decomposed filenames on them, this does not usually happen unless the user specifically wants to do that and bypasses standard OS routines. The same is true for all web technologies (CSS, HTML, XML), where W3C [[https://www.w3.org/TR/charmod-norm/#normalizationChoice|specifically recommends]] to always use composed form.((NFC, to be more specific.))

When Apple [[https://eclecticlight.co/2024/03/27/happy-birthday-apfs-7-years-old-today/|switched from HFS+ to APFS file system]] in its devices back in 2017, the situation //could// have changed, since APFS adopts the same approach as other file systems and [[https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html#//apple_ref/doc/uid/TP40016999-CH6-DontLinkElementID_3|does not normalize filenames]] – instead, it treats them as [[https://mjtsai.com/blog/2017/03/24/apfss-bag-of-bytes-filenames/|"bag of bytes"]]. However, just as on Linux distros and Windows, standard filename-manipulation libraries on macOS //**do**// normalize filenames – but to NFD, so exactly in opposition to these other OS's. So the current situation with Apple devices is that **while the APFS file system itself is normalisation-agnostic, the underlying macOS routines standardly create decomposed filenames**.((As with Windows/Linux distros, it is possible to bypass standard macOS routines, and thus to create composed filenames under macOS – for example under Terminal – but this is not common.))


==== Problems raised by the difference in normalisation forms ====

This incompatibility of unicode normalisation forms between Apple and other devices creates three different problems:
  - First, **when other users on different OS's are renaming the files you created (or renamed), they need to press <kbd>Backspace</kbd> twice when they want to remove a letter with diacritics (e.g. <kbd>á</kbd> or <kbd>ü</kbd>)**. Renaming a filename with //a lot of// diacritics can become pretty lengthy process. And note, this applies also to //**you**// when you are accessing these clouds from a web client (i.e. your browser), since the recommended standard for web technologies is NFC, as [[https://www.w3.org/TR/charmod-norm/#normalizationChoice|proposed by W3C]].
  - Second, and for the same reason, the **files encoded in NFD are often impossible to find on various web-based projects**, since many libraries do not properly work with NFD, as it is //de facto// a non-standard encoding. In short, anything you upload from Apple device to web will likely cause some kind of problems.
  - Third, **composed and decomposed sequences of characters do not look exactly the same**. I do not know if this is related to particular fonts or whether it is a rendering issue of a browser, but the fact is that composed and decomposed characters with combining marks simply look different - and the decomposed ones always look worse.

===== Solution, part one: using Rclone with iconv module =====

Rclone has a [[https://rclone.org/commands/rclone_mount/#options|special switch]] ''-o'' (''%%--option%%'') which will forward its parameters to the underlying [[https://github.com/osxfuse/osxfuse|macFUSE]] or [[https://github.com/macos-fuse-t/fuse-t|FUSE-T]] system providing the mounting functionality of the remote system. This way, it is possible to [[https://man7.org/linux/man-pages/man8/mount.fuse3.8.html#FUSE_MODULES_(STACKING)|order Fuse to load iconv module]] and have it automatically converting all filenames to NFC when they are moved to remote cloud.

This is achieved by **converting the filenames to ''UTF-8'' from special ''UTF-8-MAC'' encoding**, which is basically [[https://apple.stackexchange.com/questions/346453/what-the-difference-and-usage-of-encodings-utf-8-and-utf-8-mac-in-iconv|UTF-8 in decomposed form]], but with some specifics of [[https://developer.apple.com/library/archive/qa/qa1173/_index.html|macOS's version of UTF-8 decomposition]]. Thus, the straightforward way to solve the problem when mounting the system with Rclone //should be// to use the following command:
<code bash>$ rclone mount […] -o modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC</code>

This is what the [[https://forum.rclone.org/t/possible-special-character-encoding-issue-on-macos/39048/16|the creator of Rclone himself recommends]] and it //kind of// works, but **with problems**.


==== Problem: Missing files when using iconv in Rclone ====

The first problem you will encounter immediately after adding the ''%%-o modules=iconv%%'' part is that **some files start to be missing in the Rclone listing**. This is because **macOS uses its own "Apple-tweaked" implementation of ''iconv'', which is: (1) very old; (2) non-standard; and (3) it cannot convert significant parts of Unicode characters** – for example, emoji 😢. The moment you use the ''%%-o modules=iconv%%'' parameter in Rclone mount command, you bring into play this broken iconv implementation – and with it the aforementioned problems. All three of these problems will be crucial in our attempt to deal with the problem.

=== Third problem: Apple's version of iconv is bugged and incomplete ===
The **third point** is actually causing the missing files – **macOS's broken version of iconv simply cannot convert some parts of Unicode** (most notably surrogate pairs used e.g. in emoji) and thus [[https://github.com/RsyncProject/rsync/issues/77|any program using it will be missing files]]. As was [[https://fumiyas.github.io/2019/12/06/macos-libiconv-bug.utf8mac-curse.html|suggested elsewhere]],(([[https://fumiyas-github-io.translate.goog/2019/12/06/macos-libiconv-bug.utf8mac-curse.html?_x_tr_sl=auto&_x_tr_tl=en&_x_tr_hl=sk&_x_tr_pto=wapp|translation here]])) you can test this simply by trying to convert any emoji between ''UTF-8'' and ''UTF-8-MAC'' encoding (:
<code bash>
$ echo 😀 | iconv -f UTF-8-MAC -t UTF-8
iconv: (stdin):1:0: cannot convert
$ echo 😀 | iconv -f UTF-8 -t UTF-8-MAC
�
</code>

=== First problem: Apple's version of iconv is very old ===
The **first point** is hard to solve because the executable itself resides in ''/usr/bin/iconv'' (the library within ''/usr/lib/libiconv.dylib''), which is under SIP, so you cannot normally do anything with it, and the only way to update it is to actually update the whole macOS.

However, updating the macOS won't help you that much, since **//the standard library versions Apple provides are almost always very obsolete//**.((The [[https://opensource.apple.com/releases/|Apple Open Source site]] provides listings of all of the open source software included in each release, together with their versions (these sometimes have some weird Apple-specific versioning, but when you go to the respective GitHub page, you can usually dig out the actual software version there).)) In the case of ''iconv'', versions supplied with different macOS'es are these:
^                    macOS                             ^^              iconv                    ^^^
^ version              ^ release date ^ version (Apple) ^ version (library)        ^ release date((See [[https://ftp.gnu.org/pub/gnu/libiconv/|libiconv archive]] with downloads for each version.)) ^
| macOS Sequoia 15     | 2024-09-16 | [[https://github.com/apple-oss-distributions/libiconv/tree/libiconv-107|libiconv-107]]/[[https://github.com/apple-oss-distributions/libiconv/tree/libiconv-109|109]]((libiconv-109 since macOS Sequoia 15.2))      | FreeBSD libiconv 1.11[?] | 2009-03-03   |
| macOS Sonoma 14      | 2023-09-26 | [[https://github.com/apple-oss-distributions/libiconv/tree/libiconv-102|libiconv-102]]      | FreeBSD libiconv 1.11[?] | 2009-03-03   |
| macOS Ventura 13     | 2022-10-24 | [[https://github.com/apple-oss-distributions/libiconv/tree/libiconv-64|libiconv-64]]       | GNU libiconv 1.11        | 2006-07-19   |
| macOS Monterey 12    | 2021-10-25 | [[https://github.com/apple-oss-distributions/libiconv/tree/libiconv-61|libiconv-61]]       | GNU libiconv 1.11        | 2006-07-19   |
| macOS Big Sur 11     | 2020-11-17 | [[https://github.com/apple-oss-distributions/libiconv/tree/libiconv-59|libiconv-59]]       | GNU libiconv 1.11        | 2006-07-19   |
| macOS Catalina 10.15 | 2019-10-07 | [[https://github.com/apple-oss-distributions/libiconv/tree/libiconv-59|libiconv-59]]       | GNU libiconv 1.11        | 2006-07-19   |
| macOS Mojave 10.14   | 2018-09-24 | [[https://github.com/apple-oss-distributions/libiconv/tree/libiconv-51.200.6|libiconv-51.200.6]] | GNU libiconv 1.11        | 2006-07-19   |

So in a nutshell, despite what the internal Apple versioning says, all macOS'es still use libiconv 1.11 released back in 2006.

=== Second problem: Apple's version of iconv is non-standard ===
Lastly, the **second point** means that you cannot simply use standard [[https://www.gnu.org/software/libiconv/|GNU libiconv distribution]] of the library, because **standard version of the library differs from Apple's native one** in two respects.

First, it does not have the special ''UTF-8-MAC'' encoding. This effectively means that we cannot use the standard ''iconv'' library for the UTF-8 ↔ UTF-8-MAC conversion which we wanted from the very beginning, so dropping the Apple version of the library defeats the purpose.

Second, the functions which the ''libiconv.2.dylib'' library exports [[https://mw.gl/posts/libiconv_mac/|are named differently]], so they are incompatible with Apple's native version of the library. This can be seen by running the ''nm -L'' tool (see [[https://keith.github.io/xcode-man-pages/nm.1.html|documentation]] – [[https://web.archive.org/web/20140503095937/https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/nm.1.html|archived here]]).


===== Solution, part two: updating iconv with UTF-8-MAC support =====

All this boils down to a solution summed up above in [[#Direct way to solve the problem]]:

  - **We need an updated iconv library with UTF-8-MAC encoding**, but since there are no updates to Apple library and standard library does not have UTF-8-MAC encoding, a hybrid library is needed.
  - **We need the other apps and executables to actually use our updated library**, but since the system folders (''/usr/bin/*'' and ''/usr/lib/*'') are protected by SIP, we need to manually point these apps to the proper library version.

As for the first point, there is a [[https://github.com/fumiyas/libiconv-utf8mac|patched iconv library]] by Fumiyas which combines Apple's iconv library supporting UTF-8-MAC encoding with the latest iconv library. Installing it allows you not only to convert between real UTF-8 and UTF-8-MAC encodings, but it also supports surrogate pairs. You have to build it yourself or download built version [[https://github.com/noloader/libiconv-utf8mac|from another user on GitHub]].

As for the second point, this is solved by [[/blog/odborny/2025-01-09-tweaking_apps_executables_and_dynamic_libraries_with_install_name_tool_and_otool|manually changing the iconv library path]] in the executable with the install_name_tool.



===== Further reading =====

==== Tools for manual conversion of filenames between NFC/NFD ====
  * [[https://gist.github.com/JamesChevalier/8448512|convmv]]
  * [[https://en.bandisoft.com/bandinamer/|BandiNamer]]
  * [[https://github.com/cr0sh/jaso|jaso]]

==== Tools for analysing texts and filenames ====
  * [[https://eclecticlight.co/2021/05/08/explainer-unicode-normalization-and-apfs/|Apfelstrudel]]
  * [[https://eclecticlight.co/text-utilities-nalaprop-dystextia-and-others/|Other tools from EclecticLight]]
  * [[https://github.com/brunerd/clui?tab=readme-ov-file|Clui – Command Line Unicode Information tool]]
  * [[https://russellcottrell.com/greek/utilities/SurrogatePairCalculator.htm|Surrogate Pair Calculator]]


~~socialite~~
{{tag>draft}}


===== Comments =====

~~DISQUS~~