Search should not be case sensitive

[datarza]

If I enter something is search box, should be displayed everything without restriction.

Example: for `ПлА`, `пла` or `ПЛА` should be displayed everything, what contains `пла` in any case

image_2023-07-05_153113201.png (41.16 KiB) Viewed 2258 times

image_2023-07-05_153214852.png (55.33 KiB) Viewed 2258 times

PS: not only for train overview windows, for all searches

[Rseding91]

Thanks for the report however we don't attempt to do case translation for non-English character sets as the complexity is too much for us to maintain.

[datarza]

to do case translation for non-English character sets as the complexity is too much for us to maintain

What?

images.jpg (4.96 KiB) Viewed 2219 times

Could you try to find another person for the answers who is more experienced?

[waitofaiorur]

Uh... He's one of the main developers. If he says it's too complex it probably is.
Internationalization is troublesome at the best of times.

[datarza]

Internationalization is troublesome at the best of times.

I don't believe you.

[lm-]

Internationalization is troublesome at the best of times.

I don't believe you.

Believe it.

It is hard. With english char, to make characters lower or upper, all you have to do is unset or set specific bit. And that's it. It's easy. It's efficient. Uses almost no memory to do.

Now for the unicode characters... not so trivial, because upper/lower spacing is not symmetrical in unicode and thus you need a table with mapping. This table is big. And you need a hash map which is a lot heavier that just flipping a bit. Or ordinary array, they will be quick but will waste a lot of ram for nothing, since there will have to be empty fields. And these gaps will be small enough so that OS won't be able to optimize memory usage of it.

So yeah, case insensitive for anything other than plain ASCII *IS* a *LOT* harder. Whole internationalization is a mess already. First computers were basically designed with only ascii characters in mind.

[KeepResearchinSpoons]

the stance of "call me a manager here!" is not something that would get you anywhere :>

but that aside; I've looked the thread back in the day, yet decided it was not worthy of any input when I glanced over the other posts of the author.
nowadays I still believe this thread is not of any interest per itself.

however the underlying issue is indeed is.
Thus I would like to cover both bases, hehehe.


First of all,
this one thread can be easily killed away as a yet another duplicate of this old one
since this thread per se does not REALLY introduce anything new or worthy, putting it under-the-carpet might be for the best overall.
the "original" duplicate will never know that yet another one has bitten the same dust; neither will the original get more attention; but it is the duplicate all the same.


Secondly,
the issue of unicode collation is largely misunderstood in general. Even here we can see both parties either over-value or under-value it. (and both being right, though)

cpp does not natively support unicode

I would like to remind all the newbs that factorio is written down to the CPP (c++) classes as a way of packing the "logic".
As a side effect of such decision, we get the woefully lacking "native" support of unicode.
Simply because cpp as a language kinda predates the era where unicode took over the world.

std::string might just not be aware that characters could be multi-byte sequences;

and even if it was, declaring a full unicode support is kinda tricky in general (even emojies support is "fun" when you start to look into country flags, for example; look up TW flag history, if you want some IRL drama);

basically, in the cpp land there is no simple-and-natural way out with unicode. Might have changed with the latest cpp revisions, not sure; but they are probably kinda tricky to follow either way.

total complete collation is crazy hard

collation kinda means you can compare strings A and B to decide if they are "the same".
flip-the-bit option proposed above is an example of lowercase-it-all approach (uppercase it all, for the fans of Microsoft, haha).

Simple example of dates collation: m/d/y vs d.m.y; could be solved with a perfect "isodate" yyyy-mm-dd form.
Harder example of numbers collation: 1.00 and 1,00 is probably what looks the same for you. but 1_00 and 1,00 can also mean the same as 100; could only be solved with the help of the "context" of what 1,00 *probably* means by-default.

Same kind of stuff exists for languages.
In English you get kinda close to what "canonical form" of a grammar could be. Programming then dances around it even more, using a subset with a number of strict rules (such as with JSON) that make the "context-free parsing" part of it easy.
And that largely influences what ppl consider languages to be in the first place.

Yet, with, say, Russian, all the words are mutable. Some have to, some depend on your taste preference. You can say change the "copper-plate" depending on how or where you want to use it. ("plateys", "coppah" etc is kinda allowed, if you are willing to) You can even say "iron-gears" in a few different popular ways, still meaning "gears" for any native speaker (since the same "root", think "gearz"). Maybe not something one would consider for collation, but even the "plurals formula" is fun; declension; wiki is what you can look at for some more fun.

But CJK ("glyphy langs") can get you further than that.
There, you can write the stuff that sounds the same, means the same ...yet is written not the same way.
And there is more than one way to do that!
Not just "capital" letters but the "sets" of same-meaning glyphs.
And don't get me started on how they "mutate", compact or extend.
As such, imagine the collation that allows you to, say, have:
LDS -> Low density structure, or even GC -> (eh) Electronic-Circuit (yep, no Green in ANY form here, haha).

the ability to have the latter WOULD have been nice. Especially if user could control the mapping.
But I suppose that would have some fun effects on determination and overall system complexity, hehe.
Yet, for the CJK languages, having such an extensible vocab would be a killer feature in more ways than one.

And who knows, maybe there is even a goto-way to collate all that stuff already in the cpp world; but sorry, I do not speak cpp fluently ;>

Well, even Hebrew lets you "omit" the vowels as you go. German has ẞ/ß (for "ss", see eszett; wiki).
And most languages there are have some sort of irregular yet popular and widely used stuff.
And there is a whole another issue of motors-vs-engines like collation, often seen with players reading the non-english locale try to communicate in English.

so in short, making a perfect collation even for a known subset of world grammar is tough. Even for any "reasonable" limit on what is "the same".
thus having a take of "latin only, and the rest we do not care" is "valid" and "understandable".
Definitely not something of "call me a manager here!", I would say.

With that said, let's imagine we were trying to actually solve this issue at hand.
(since, well, it is VERY popular one, kinda a FAQ item at this point)

apparently for cyrillics it is kinda easy

Most "duplicates" of the matter that I've seen so far refer to Cyrillic family.
Skipping the strong-and-complex lib-that-does-it-all option, it is kinda hard to grasp why it just "does not work" for `К==к` and `М==м` as it does for `K==k` with `M==m`.

In fact, looking at the unicode Cyrillic blocks (wiki) does give a suggestion that it would be as easy as adding a Titillium style font.
Well, the font was added only with v1.1.38, @@2021-08-18 (post-release), but now we have it :>

Nicely enough, unicode blocks are mostly linear, which in turn allows for these kinds of "hacks" that do the "flip-a-bit" approach with a range-check. Without any sizeable hash-maps, dicts or the like. (still, a few more checks)
Please look into it if you have the time to, as it introduces pretty much a crapton chunk of value for quite a reasonable buck. Even though not directly.

There are some not-so-trivial details as when the "lowecase bit" bleeds out to the first byte of the pair that represent the codepoint; but overall it is on the "easily manageable by one person" scale.

Would be nice to have this issue addressed in some form of a best-effort approach.
But let's speak on workarounds for mods as well.

scripting scripting if you please take me across the raging seas

Naturally, both Python and JS have a native way around with `"Привет Мир!".lower()` or `"Привет Мир!".toLowerCase()`.
And Lua does not really. Because Lua is Lua. Nice and *simple*.

Having a game-engine implementation bindings to a "proper" lowercase in a form of `scripts.lowercase(str)` would be fun and neat, right.
Having it overload the string.lower might even be more fun (and *probably* won't cause too much problems, as far as I can imagine).

If we are talking static content (searching over pre-defined strings) one can probably even build-and-bake a radix tree right away, but well, to hell with it. Let's speak "real scale" effort where ppl do not even wrap the critical routines with pcalls. Really, what can possibly go wrong. Just mark it as incompatible if any problems arrive and that's it. Hehehe. (yet, less complex often also means easier to reason about! which is ok)

Then, say, "static" content allows us to have a "canonical" form of a string stored alongside for indexing purposes. These can be made with either Python or JS with quite a small scripting effort. As long as the canonical representation is uniform, it lifts up half the trouble (at the cost of memory usage). And works for even more complex languages where the "canonical" form could be manually tweaked to whichever is the norm.

But since we are in the land of naive contraptions,
here's a workaround that you can do for your own script on Cyrillic (with the naive hashmap-ing approach). A proper "lowercase" of sorts, baked with your own alphabet-like collation of choice.
All it really allows for is to have the case-insensitive search over the strings that have some Cyrillic inside in the Lua land. Dynamically.
Maybe not much, but, well, it does mean a lot for some.
Would prefer if it "worked" right away somehow, but this workaround is also an option; if you are aware of the issue, that is.

afterall, this issue still needs an "escalation", mostly for awareness reasons though.
the collation hack also linked above probably solves the issues of all the players (or most) that are being repeatedly referred to this old post (as for duplicates) each time they re-discover it.
"the fix" does not introduce much value of itself, but saves a day for all the newbs perceiving the "limited" search utility as "permanently broken since long before" in the current state of things.

To my personal opinion, going with a modding-interface-request for this specific sort of collation for Lua might be easier, since this one seems to be eternally perma-stuck in "duplicates". But well, all things have a natural chance to change for the better sooner or later with enough love and time to it.

Have fun and may the gearz force be with u! :>

[datarza]

lm-: was you frozen in the 1980s and melted just now? - https://www.google.com/search?q=app+loc ... or+dummies

Incredible nonsense, many times moaned feature for seven years, seven string of code to implement, stubbornness of Factorio staff is amazing

[jamiechi1]

I agree with @datarza on this one. I can't imagine the coding on this to be that difficult.

[coppercoil]

I agree with @datarza on this one. I can't imagine the coding on this to be that difficult.

You are not familiar enough with Unicode. Some players are so incompetent, so they don't even know what to google for.

On the other hand, maybe devs could implement lowercase for some of most common and relatively simple languages.

[Koub]

[Koub] Moderator-me speaking. The request was clear, I'm sure the devs have understood what's requested. There's no need fighting publicly on this forum over whether the request is easy or hard to implement. And certainly not calling each other names.
So keep calm, constructive, and polite, or I will enforce calmness myself.