Previously: https://lemmyrs.org/post/175672

I originally had sources and data of the site public, hoping they would be interesting to study, aid in bug reporting, bring contributions, and make site's algorithms transparent.

Instead, I got knee-jerk reactions about lines of code taken out of context. I got angry dogpiles from the Rust community, including rust-lang org members. I don't need to endure such mudslinging. Therefore, the sources are no longer available.

As of right now bitcoin crate is not deprecated, instead libs.rs responds with error 502.

If you have a workspace with dependencies you probably noticed that sometimes cargo seemingly unnecessary recompile external dependencies as you switch between different members of your workspace.

This is caused by something called feature unification ([1]). Since features by design should be additive only cargo tries to avoid redundant work by using a superset of all required features. Problem comes when there are multiple crates in the workspace require external dependencies with different set of features. When you are working with the workspace as a whole - unified features include all the dependencies, when you target a single crate - unified features will include only features of that crate's dependencies.

What's worse - if you are using nix with crate2nix to manage dependencies - you'll get no feature unification at all and every dependency with each unique combination of features is considered a separate dependency so the same crate can be compiled (and linked in) multiple times - I've seen 20+ copies of a single crate.

Unless there are special requirements it is better to make sure that all the external dependencies have exact same set of features enabled across the workspace. One option is to do it by hand manually editing Cargo.toml files for individual dependencies or with inherited workspace dependencies. As with anything manual - it would be error prone.

That's where cargo-hackerman comes in. It can check if there are feature unification issues in the workspace so you can run it in CI if you want to do it manually or it can apply the smallest possible hack by itself (and remove it later).

This is not a new crate, we've been using it in production with a large workspace for over a year now.

In addition to feature unification it can help with some other compilation time things giving easy answers to questions like :

• are there any duplicate dependencies in my workspace?
• why is this dependency or feature on dependency is required?
• what are all the dependencies of this crate?
• where is the repository of this crate?

With updated bpaf documentation on https://crates.io/crates/cargo-hackerman should be always up to date and cli - a bit more user friendly

• https://doc.rust-lang.org/cargo/reference/features.html#feature-unification
• https://crates.io/crates/cargo-hackerman
• https://github.com/pacak/hackerman/

What do you think about this kind of indication for conflicting or otherwise invalid arguments?

With command line arguments being 1D and line length valid up to hundreds of kilobytes only inline indication seems to work.

Would you change anything?

One of the digits of your credit card number is not like the rest of them: it's purpose is to check for value correctness, this way any website or form knows what if checksum matches - the value was typed in correctly. Or you are lucky with a typo because it's not a very good checksum - it was designed to be easy to calculate on a mechanical device and been sticking around mostly for historical reasons.

To check if a number is valid according to Luhn checksum you would go from right to left, double every second digit, add all the digits together, if result ends with 0 - checksum matches, if it ends with some other value - it does not.

For example let's check the number 1594: write down the number as a bunch of digits

1 5 9 4

double every second digit from the right

2 5 18 4

add all the digits

2 + 5 + 1 + 8 + 4 = 20

ends with 0, so checksum is valid

Three key optimizations help to calculate it fast:

• You can split longer sums into short ones
• You can skip second splitting into digits by doing multiplication as usual and adding number of digits above 5 to the total
• SWAR can to perform a bunch of operations on individual digits at once

To illustrate first optimization let's calculate 1 5 9 4 sum in two parts

1 5 => 2 5 => 2 + 5 = 7
9 4 => 18 4 => 1 + 8 + 4 = 13
7 + 13 = 20 - checksum is valid

to illustrate the second one

// digits themselves
1 5 9 4 => 2 5 18 4 => 2 + 5 + 18 + 4 = 29
// correction
0 0 1 0 => 0 + 0 + 1 + 0 = 1
total: 29 + 1 = 30

Result is off by 10, but for checksum validity purposes it's good enough.

Last trick comes from doing SWAR and skipping extracting digits in the first place.

User input comes as an ASCII string "1594", which we split into chunks of size 8 to fit into 64 bit register and pad with "0" from the right:

"1594" => "00001594"
// subtract  0x3030303030303030 ("00000000")
// to get decimal values
"00001594" - "00000000" => [0, 0, 0, 0, 1, 5, 9, 4]

At this point it is easy to check if string consists of only decimal digits just by adding 0x4646464646464646 and looking for overflows and underflows in both results - can be done for both at once

Multiplying every other digit by 2 and adding them together is done with a regular multiplication by 0x0201020102010201, and math will do the rest:

    A   B
  * 1   2
----------
   2A  2B
 A  B

Top byte of the lower half of the result will contain 2A + B for 2 digit case or the full result for all 8 digits in the actual code. Because all the digits are below 10 - there's no overflow from earlier digits.

Whole code looks like this:

fn fold10_swar(mask1: u64, mask2: u64, raw: &[u8]) -> Option<u64> {
    let mut sum = 0;

    for c in raw.rchunks(8) {
        let mut buf = [b'0'; 8];
        copy_from_small_slice(&mut buf, c);

        let mut v = u64::from_le_bytes(buf);
        // try to overflow value up
        let a = v.wrapping_add(0x4646464646464646);
        // and down
        v = v.wrapping_sub(0x3030303030303030);
        // if either direction overflows - there are non 0..9 digits so
        // checksum can't possibly be valid, otherwise all the values are digits
        if (a | v) & 0x8080808080808080 == 0 {
            // Calculate number of digits above 5 located at positions that would double
            // Doubling them would result to values above 9 which will necessitate subtracting
            // 9. But in mod 10 arithmetic -9 and +1 is the same so simply adding
            // count of such digits is enough
            sum += u64::from((mask2.wrapping_sub(v) & 0x8080808080808080).count_ones());
            sum += v.wrapping_mul(mask1) >> 56;
        } else {
            return None;
        }
    }
    Some(sum)
}

And good news - luhn3 crate can check the correctness or calculate a digit to use as a checksum for you and it works somewhat fast - on a 5 year old processor when compiled with target-cpu=native it can verify a 16 digit credit card checksum in about 3 nanoseconds - in 12 or so CPU cycles, benchmarks included.