rust_icu: low-level rust language bindings for the ICU library

See: The latest version of this file is available at

rust_icu: low-level rust language bindings for the ICU library

Item Description
ICU 63, 68..70
Source https://github.com/google/rust_icu
README https://github.com/google/rust_icu/blob/main/README.md
Coverage View report
Docs https://docs.rs/crate/rust_icu

This is a library of low level native rust language bindings for the International Components for Unicode (ICU) library for C (a.k.a. ICU4C).

If you just want quick instructions on how to download and install, see the quickstart guide

See the ICU project home page for details about the ICU library. The library source can be viewed on Github.

The latest version of this file is available at https://github.com/google/rust_icu.

This is not an officially supported Google product.

Why wrap ICU (vs. doing anything else)?

  • The rust language Internationalisation page confirms that ICU support in rust is spotty, so having a functional wrapper helps advance the state of the art.

  • Projects such as Fuchsia already depend on ICU, and having rust bindings allows for an easy way to use Unicode algorithms without taking on more dependencies.

  • Cooperation on the interface with projects such as the ICU4X could allow seamless transition to an all-rust implementation in the future.

Structure of the repository

The repository is organized as a cargo workspace of rust crates. Each crate corresponds to the respective header in the ICU4C library's C API. Please consult the coverage report for details about function coverage in the headers.

Crate Description
rust_icu Top-level crate. Include this if you just want to have all the functionality available for use.
rust_icu_common Commonly used low-level wrappings of the bindings.
rust_icu_intl Implements ECMA 402 recommendation APIs.
rust_icu_sys Low-level bindings code
rust_icu_ubrk Support for text boundary analysis. Implements ubrk.h C API header from the ICU library.
rust_icu_ucal ICU Calendar. Implements ucal.h C API header from the ICU library.
rust_icu_ucol Collation support. Implements ucol.h C API header from the ICU library.
rust_icu_udat ICU date and time. Implements udat.h C API header from the ICU library.
rust_icu_udata ICU binary data. Implements udata.h C API header from the ICU library.
rust_icu_uenum ICU enumerations. Implements uenum.h C API header from the ICU library. Mainly UEnumeration and friends.
rust_icu_uformattable Locale-sensitive list formatting support. Implements uformattable.h C API header from the ICU library. Since 0.3.1.
rust_icu_ulistformatter Locale-sensitive list formatting support. Implements ulistformatter.h C API header from the ICU library.
rust_icu_uloc Locale support. Implements uloc.h C API header from the ICU library.
rust_icu_umsg MessageFormat support. Implements umsg.h C API header from the ICU library.
rust_icu_unorm2 Unicode normalization support. Implements unorm2.h C API header from the ICU library.
rust_icu_unum Number formatting support. Implements unum.h C API header from the ICU library.
rust_icu_unumberformatter Number formatting support (modern). Implements unumberformatter.h C API header from the ICU library.
rust_icu_upluralrules Locale-sensitive plural rules support. Implements upluralrules.h C API header from the ICU library.
rust_icu_ustring ICU strings. Implements ustring.h C API header from the ICU library.
rust_icu_utext Text operations. Implements utext.h C API header from the ICU library.
rust_icu_utrans Transliteration support. Implements utrans.h C API header from the ICU library.

Limitations

The generated rust language binding methods of today limit the availability of language bindings to the available C API. The ICU library's C API (sometimes referred to as ICU4C in the documentation) is distinct from the ICU C++ API.

The bindings offered by this library have somewhat limited applicability, which means it may sometimes not work for you out of the box. If you come across such a case, feel free to file a bug for us to fix. Pull requests are welcome.

The limitations we know of today are as follows:

  • There isn't a guaranted feature parity. Some algorithms that are implemented in C++ don't have a C equivalent, and vice-versa. This is usually not a problem if you are using the library from C++, since you are free to choose whichever API surface works for you. But it is an issue for rust bindings, since we can only use the C API at the moment.

  • A C++ implementation of a new algorithm is not necessarily always reflected in the C API, leading to feature disparity between the C and C++ API surfaces. See for example this bug as an illustration.

  • While using icu_config feature will likely allow you some freedom to auto-generate bindings for your own library version, we still need to keep a list of explicitly supported ICU versions to ensure that the wrappers are stable.

Compatibility

The compatibility guarantee is as follows:

  1. Automated tests are executed for last three major ICU library versions in all feature combinations of interest.
  2. Automated tests are executed for the ICU library version in use by the docs.rs system (so the documentation could be built).
rust_icu version ICU 63.x ICU 67.1 ICU 68.1 ICU 69.1 ICU 70.1 ICU 71.1
0.5
1.0
2.0

If you must use a 0.x relase, please use the release 0.5. Otherwise, we recommend using the newest release that supports your use case.

Features

The rust_icu library is intended to be compiled with cargo, with one of several features enabled. Compilation with cargo allows us to do some library detection in a custom build.rs file in the rust_icu_sys library and adapt the build process to your build environment. However, since not every development environment will use the same settings, we opted to offer certain features (below) as configuration options.

While our intention is to keep the list of features below up to date with the actual list in Cargo.toml, the list may periodically go out of date.

To use any of the features, you will need to activate the feature in all the rust_icu_* crates that you intend to use. Failing to do this will result in confusing compilation end result.

Feature Default? Description
use-bindgen Yes If set, cargo will run bindgen to generate bindings based on the installed ICU library. The program icu-config must be in $PATH for this to work. In the future there may be other approaches for auto-detecting libraries, such as via pkg-config.
renaming Yes If set, ICU bindings are generated with version numbers appended. This is called "renaming" in ICU, and is normally needed only when linking against specific ICU version is required, for example to work around having to link different ICU versions. See the ICU documentation for a discussion of renaming. This feature MUST be used when bindgen is NOT used.
icu_config Yes If set, the binary icu-config will be used to configure the library. Turn this feature off if you do not want build.rs to try to autodetect the build environment. You will want to skip this feature if your build environment configures ICU in a different way. This feature is only meaningful when bindgen feature is used; otherwise it has no effect.
icu_version_in_env No If set, ICU bindings are made for the ICU version specified in the environment variable RUST_ICU_MAJOR_VERSION_NUMBER, which is made available to cargo at build time. See section below for details on how to use this feature. This feature is only meaningful when bindgen feature is NOT used; otherwise it has no effect.

Prerequisites

Required

  • rust_icu source code

    Clone with git:

  • rustup

    Install from https://rustup.rs. Used to set toolchain defaults. This will install cargo as well.

  • Clang

    You must have Clang installed to access the right headers.

  • The ICU library development environmnet

    You will need access to the ICU libraries for the rust_icu bindings to link against. Download and installation of ICU is out of scope of this document. Please read through the ICU introduction to learn how to build and install.

    Sometimes, the ICU library will be preinstalled on your system, or you can pull the library in from your package management program. However, this library won't necessarily be the one that you need to link into the program you are developing. In short, it is your responsibility to have a developer version of ICU handy somewhere on your system.

    We have a quickstart install that may get you well on the way in case your environment happens to be configured very similarly to ours and you want to build ICU from source.

Optional

  • GNU Make, if you want to use the make-based build and test.

    Installing GNU Make is beyond the scope of this file. Please refer to your OS instructions for installation.

  • docker, if you decide to use docker-based build and test.

    Installing docker is beyond the scope of this file, please see the docker installation instructions for details. As installing docker is intrusive to the host machine, your company may have internal documentation on how to install docker properly.

  • icu-config utility, if icu_config feature is used.

    You need to install the ICU library on your system, such that the binary icu-config is somewhere in your $PATH. The build script will use it to discover the library settings and generate correct link scripts. If you use the feature but icu-config is not found,

  • bindgen utility, if bindgen feature is used.

    bindgen user guide for instructions on how to install it.

  • rustfmt utility, if bindgen feature is used.

    See https://github.com/rust-lang/rustfmt for instructions on how to install.

Testing

There are a few options to run the test for rust_icu.

Cargo

Building and testing using cargo is the canonical way of building and testing rust code.

In the case of the rust_icu library you may find that your system's default ICU development package is ancient, in which case you will need to build your own ICU4C library (see below for that). That will make it necessary to pass in PKG_CONFIG_PATH and LD_LIBRARY_PATH environment variables to help the bulid code locate and use the library you built, instead of the system default.

The following tests should all build and pass. Note that because the libraries needed are in a custom location, we need to set LD_LIBRARY_PATH when running the tests, as well as PKG_CONFIG_PATH.

If you find that you are able to use your system's default ICU installation, you can safely omit the two libraries.

If you think that the above approach is too much of a hassle, consider trying out the Docker-based approach.

GNU Make

If you happen to like the GNU way of doing things, you may appreciate the GNU Make approach.

The easiest way is to use GNU Make and run:

You may want to use this method if you are working on rust_icu, have your development environment all set up and would like a shorthand to run the tests.

Docker-based

See optional dependencies section above.

To run a hermetic build and test of the rust_icu source code, issue the following command:

This will run docker-based build and test of the source code on your local machine. This is a good way to test that your code works with a specific reference version of ICU.

Prior art

There is plenty of prior art that has been considered:

  • https://github.com/servo/rust-icu
  • https://github.com/open-i18n/unic
  • https://github.com/fullcontact/icu-sys
  • https://github.com/rust-locale
  • https://github.com/unicode-rs

The current state of things is that I'd like to do a few experiments on my own first, then see if the work can be folded into any of the above efforts.

See also:

  • https://github.com/rust-lang/rfcs/issues/797
  • https://unicode-rs.github.io
  • https://github.com/i18n-concept/rust-discuss

Assumptions

There are a few competing approaches for ICU bindings. However, it seems, at least based on information available in rust's RFC repos, that the work on ICU support in rust is still ongoing.

These are the assumptions made in the making of this library:

  • We need a complete, reusable and painless ICU low-level library for rust.

    This, for example, means that we must rely on an external ICU library, and not lug the library itself with the binding code. Such modularity allows the end user of the library to use an ICU library of their choice, and incorporate it in their respective systems.

  • No ICU algorithms will be reimplemented as part of the work on this library.

    An ICU reimplementation will likely take thousands of engineer years to complete. For an API that is as subtle and complex as ICU, I think that it is probably a better return on investment to maintain a single central implementation.

    Also, the existence of this library doesn't prevent reimplementation. If someone else wants to try their hand at reimplementing ICU, that's fine too.

  • This library should serve as a low-level basis for a rust implementation.

    A low level ICU API may not be an appropriate seam for the end users. A rust-ful API should be layered on top of these bindings. It will probably be a good idea to subdivide that functionality into crates, to match the expectations of rust developers.

    I'll gladly reuse the logical subdivision already made in some of the above mentioned projects.

  • I'd like to explore ways to combine with existing implementations to build a complete ICU support for rust.

    Hopefully it will be possible to combine the good parts of all the rust bindings available today into a unified rust library. I am always available to discuss options.

    The only reason I started a separate effort instead of contributing to any of the projects listed in the "Prior Art" section is that I wanted to try what a generated library would look like in rust.

Additional instructions

Quickstart guide

Before you begin, please ensure the following prerequisites are met:

  • You have docker installed and it runs on your system.
  • You have GNU Make.
  • You have git.
  • You have plenty of disk space. The docker images for the build environment are a bit large, so a few GiB are needed to fit all of them.
  • You have an Internet connection.

From there, the following sequence of commands will check out, build and test the rust_icu source code.

You can now make changes to the code and tests. You can re-run the compile and test cycle by running make docker-test.

ICU installation instructions

These instructions follow the "out-of-tree" build instructions from the ICU repository.

Assumptions

The instructions below are not self-contained. They assume that:

  • you have your system set up such that you can follow the ICU build instructions effectively. This requires some upfront time investment.
  • you can build ICU from source, and your project has access to ICU source.
  • your setup is Linux, with some very specific settings that worked for me. You may be able to adapt them to work on yours.

Compilation

If the compilation finishes with success, the directory $HOME/local/bin will have the file icu-config which is necessary to discover the library configuration.

You can also do a

to run the unit tests.

If you add $HOME/local/bin to $PATH, or move icu-config to a directory that is listed in your $PATH you should be all set to compile rust_icu.

ICU rebuilding instructions

If you change the configuration of the ICU library with an intention to rebuild the library from source you should probably add an intervening make clean command.

Since the ICU build is not hermetic, this ensures there are no remnants of the old compilation process sitting around in the build directory. You need to do this for example if you upgrade the major version of the ICU library. If you forget to do so, you may see unexpected errors while compiling ICU, or while linking or running your programs.

Compiling for a set version of ICU

Assumptions

  • You have selected the feature set [renaming,icu_version_in_env]o

OR:

  • You have manually verified that the compatibility matrix has a "Yes" for the ICU version and feature set you want to use.

The following is a tested example.

The following would be an as of yet untested example of compiling rust_icu against a preexisting ICU version 66.

Adding support for a new version of ICU.

In general, as long as icu-config approach is supported, it should be possible to generate the library wrappers for newer versions of the ICU library, assuming that the underlying C APIs do not diverge too much.

An approach that yielded easy support for ICU 65.1 consisted of the following steps. Below, $RUST_ICU_SOURCE_DIR is the directory where you extracted the ICU source code.

  • Download the new ICU version from source to $RUST_ICU_SOURCE_DIR.
  • Build the ICU library following for example the compilation steps above with the new version.
  • Get the file lib.rs from the output directory $RUST_ICU_SOURCE_DIR/target/debug/build/rust_icu_sys-..., rename it to lib_66.rs (if working with ICU version 66, otherwise append the version you are using).
  • Save the file to the directory $RUST_ICU_SOURCE_DIR/rust_icu_sys/bindgen, this is the directory that contains the pre-generated sources.

These files lib_XX.rs may need to be generated again if build.rs is changed to include more features.

Adding more bindings

When adding more ICU wrappers, make sure to do the following:

  • Check rust_icu_sys/build.rs and rust_icu_sys/bindgen/run_bindgen.sh to add appropriate lines into BINDGEN_SOURCE_MODULES, then BINDGEN_ALLOWLIST_FUNCTIONS and BINDGEN_ALLOWLIST_TYPES.

Testing with a specific feature set turned on

Here's an example of running a docker test on ICU 67, with features icu_version_in_env and renaming turned on instead of the default. Note that the parameters are mostly passed into the container that runs docker-test via environment variables.

Some clarification:

  • The environment variable RUST_ICU_MAJOR_VERSION_NUMBER is used for the feature icu_version_in_env to instruct cargo to use the file rust_icu_sys/bindgen/lib_67.rs as a prebuilt bindgen source file instead of trying to generate one on the fly.
  • The environment variable DOCKER_TEST_CARGO_TEST_ARGS is used to pass the command line arguments to the cargo test which is used in the docker container. The environment is passed in verbatim to cargo test without quoting, so separate words in the environment end up being separate args to cargo test.
  • The environment variable DOCKER_TEST_ENV is the base name of the Docker container used to run the test in. The container rust_icu_testenv-67 is a container image that contains preinstalled environment with a compiled version of ICU 67.

Refreshing static bindgen files

Requires docker.

Run make static-bindgen periodically, to refresh the statically generated bindgen files (named lib_XX.rs, where XX is an ICU version, e.g. 67) in the directory rust_icu_sys/bindgen which are used when bindgen features are turned off.

Invoking this make target will modify the local checkout with the newer versions of the files lib_XX.rs. Make a pull request and check them in.

For more information on why this is needed, see the bindgen README.md.

Issues

Collection of the latest Issues

kaj

kaj

8

Trying to build rust_icu_ustring on FreeBSD (after getting around #235), I get a bunch of errors like this:

I get eleven such errors, all for different function names u_\w+_70. Checking the actual library, (with nm --dynamic --defined-only (libfile) it contains the symbols like u_strFromUTF8 but no symbols ending with _70 (or any other _number). The same command on linux (Fedora 35) shows u_strFromUTF8_69 (but no unversioned symbols).

The version number itself is correct, libicuuc.so on my FreeBSD host is a symlink to libicuuc.so.70.1. But the symbols in it are not versioned.

In rust_icu_sys/bindgen/macros.rs, it seems that versioning can be disabled by not using the renaming or the icu_version_in_env feature, but apparently at least one or those gets defined in my build even though it should not. Or should the cfg

https://github.com/google/rust_icu/blob/fc64773b3894089c9e9bbc28a75af842d62aa8b4/rust_icu_sys/bindgen/macros.rs#L32-L39

be changed to just

?

kaj

kaj

2

I get the following error when compiling rust_icu_sys on FreeBSD:

The error message mentions icu-config (which I do have on my $PATH), but the command it tried to execute is pkg-config, which is not available on FreeBSD. Should the be a different implementation of struct ICUConfig for FreeBSD (and possibly other systems that use icu-config instead of pkg-config) ? Is there such an implemention that just failed to be selected in my compilation (but if so, I didn't find it in the source) ?

izderadicka

izderadicka

3

I want to use rust_icu_col in building fully statically linked binary. I'm building on alpine with RUSTFLAGS="-C target-feature=+crt-static -C link-self-contained=yes"

Problem here is that I'm getting this error when rust_icu_sys is compiled:

I do have static version of libclang - e.g libclang.a on system, but build is still preferring dynamic version.

In my other projects I' m using feature static which will enforce println!("cargo:rustc-link-lib=static=ssl"); (here should be clang) for dependant C libraries.

However I'm not sure how to do it for indirect dependencies - e.g. you depend on C library and I depend on your crate.

I guess same will be for icu C library, but just hit clang first.

nkovacs

nkovacs

5

According to https://unicode-org.github.io/icu/userguide/icu/design.html#thread-safe-const-apis, using the methods that take a const this pointer is thread safe.

In my case I'd like to use UCollator across multiple threads, which should be safe since ucol_strcoll takes a const pointer, but I can't because it's not Sync+Send. I've looked at the source of rust_icu_ucol, and all the methods that take a non-const pointer in C are &mut self in Rust, except for set_attribute, which is generated by the generalized_fallible_setter macro. If generalized_fallible_setter is fixed to generate &mut self, would it be safe to mark UCollator (and other service objects, assuming the same is true for their methods) as Sync (edit: and Send)?

Manishearth

Manishearth

4

In ICU4X we're moving towards no_std support, and this means that by default the Error trait is not implemented. It's possible to explicitly enable the "std" feature and have it be enabled, however it seems like it's unnecessary to require Error on the error associated types since none of Errors functionality is actually used — perhaps we should just require Debug+Display, or maybe even just one or none of those traits. In general we should require the traits we need.

filmil

filmil

1

For some reason this started not when I upgraded to ICU 69.1 but a few runs later.
Disabling the specific brittle test for now.

EvanCarroll

EvanCarroll

1

I'm not even sure what I'm reading.. Is this not made for users at all because it's low level?

Where is the simple "hello world" example, with instructions on how to get to/from xliff? This crate is the first thing that comes up when you search for icu messageformat rust.

filmil

filmil

enhancement
0

The work is expanding on the initial proposal and has as goal to expand the coverage of the ECMA 402 implementation to all functionality currently exposed.

The initial status is given below. The goal is to check all boxes in the matrix.

ECMA 402 API Trait Definition rust_icu API rust_icu adaptor icu4x API icu4x adaptor
Intl.Collator
Intl.DateTimeFormat
Intl.DisplayNames
Intl.NumberFormat
Intl.Locale
Intl.ListFormat
Intl.PluralRules
Intl.RelativeTimeFormat
filmil

filmil

bug
2

@kpozin noticed that the magic flags --c-kinds=fp used by make cov to extract function signatures from Unicode source doesn't exist in ctags versions other than ctags-exuberant. (yay!)

We should, perhaps, make a hermetic version of make cov that doesn't make this assumption.

filmil

filmil

enhancement
2

IIUC it is not possible to tell cargo to bake in the absolute path of a non-default library when compiling.

For example, one would do this in case of gcc: https://stackoverflow.com/questions/8835108/how-to-specify-non-default-shared-library-path-in-gcc-linux-getting-error-whil (i.e. set -rpath=... to the correct value).

See here: https://github.com/rust-lang/cargo/issues/5077

I tried to modify the resulting binaries with patchelf but that somehow didn't work either, but I didn't really dig deep enough to understand why. Filing this bug so I don't forget.

sffc

sffc

enhancement
14

Split from #6

The code currently calls u_strToUTF8. It would be better to use Rust standard library functions like std::str::encode_utf16. You would avoid having to go across the FFI boundary when performing the conversion.

sffc

sffc

enhancement
0

Splitting off my comments about date parsing from #2.

ECMA-402 does not and probably will not add support for parsing. The bottom line is that parsing of localized strings is a hard problem that is best solved by simply building your application to avoid having to do it. Do you really need to support parsing in the wrapper?

sffc

sffc

help wanted
2

My question here is, where and how do you use the Text that you make? In ICU, a UText is primarily useful for interfacing with APIs like the BreakIterator and regex engine. It doesn't do a whole lot on its own, except maybe providing some more UTF-8 to UTF-16 conversions, but you already have that functionality in rust_icu_ustring.

sffc

sffc

enhancement
1

Nothing wrong here from a functionality point of view, but it raises questions about what is the best way to represent strings when interfacing with ICU. Many ICU APIs want UTF-16 strings but also accept UTF-8 strings. I've been toying with the idea of making a Rust version of UnicodeString that you can toggle between UTF-8 and UTF-16 at compile time. For now, I think the safest thing is for users to keep their strings in Rust-standard UTF-8, and only do the UTF-16 conversion at the API boundary. Even if you have to round-trip a few times between 8 and 16, I strongly suspect that this won't be a performance bottleneck.

sffc

sffc

enhancement
1

Maybe you want to use @zbraniecki's Rust Locale class. :smiley:

It looks like all this does is basically call uloc_canonicalize. IIRC, Zibi's class performs partial but not full canonicalization yet. Once Zibi's class supports that, maybe you can just remove this wrapper altogether.

sffc

sffc

enhancement
4

ECMA-402 does not and probably will not add support for parsing. The bottom line is that parsing of localized strings is a hard problem that is best solved by simply building your application to avoid having to do it. Do you really need to support parsing in the wrapper?

ECMA-402 also does and probably will not support patterns. ICU has long recommended that people don't use patterns; they should use skeletons instead. Do you really need to support patterns?

Speaking of skeletons, you don't have an API for skeletons right now. Could you remove your pattern API and replace it with a skeleton API?

sffc

sffc

enhancement
2

ECMA-402 does not expose the Calendar class, in large part because we are working toward a better representation for dates and times in Temporal. I know pure ICU does not currently support any other good types for inputting to your DateFormat, but perhaps this is a place where you want to consider a wrapper with a more friendly API that hides the UCalendar under the hood.

Information - Updated Apr 21, 2022

Stars: 87
Forks: 19
Issues: 26

Repositories & Extras

Rust library for Self Organising Maps (SOM)

Add rusticsom as a dependency in Cargo

Rust library for Self Organising Maps (SOM)

Rust library for parsing configuration files

The 'option' can be any string with no whitespace

Rust library for parsing configuration files

Rust library for the Pimoroni Four Letter pHAT

This library aims to port ht16k33 (or rather a fork, as of right now) so credit goes to ht16k33-diet

Rust library for the Pimoroni Four Letter pHAT

Rust library for emulating 32-bit RISC-V

This library can execute instructions against any memory and register file that implements

Rust library for emulating 32-bit RISC-V

Rust library for connecting to the IPFS HTTP API using Hyper/Actix

You can use actix-web as a backend instead of hyper

Rust library for connecting to the IPFS HTTP API using Hyper/Actix

Rust library to manipulate file system access control lists (ACL) on macOS, Linux, and FreeBSD

This module provides two high level functions, getfacl and setfacl

Rust library to manipulate file system access control lists (ACL) on macOS, Linux, and FreeBSD

Rust library translation (rust-src/rust-std/stdlib/rustlib translation)

This is the place to translate Having a documentation in your native language is essential if you don't speak English, and still enjoyable even if...

Rust library translation (rust-src/rust-std/stdlib/rustlib translation)

Rust library for using Infrared hardware decoders (For example a Vishay TSOP* decoder),

enabling remote control support for embedded project

Rust library for using Infrared hardware decoders (For example a Vishay TSOP* decoder),

Rust library for interaction with the OriginTrail Decentralized Knowledge Graph

open up an issue on this repository and let us know

Rust library for interaction with the OriginTrail Decentralized Knowledge Graph

Rust library for parsing COLLADA files

Notice: This library is built around files exported from Blender 2

Rust library for parsing COLLADA files

Rust library for low-level abstraction of MIPS32 processors

This project is licensed under the terms of the MIT license

Rust library for low-level abstraction of MIPS32 processors
Facebook Instagram Twitter GitHub Dribbble
Privacy