You are currently working on UAT 

  • State Suggested Answer
  • Replies 13 replies
  • Answers 3 answers
  • Subscribers 429 subscribers
  • Views 5767 views
  • Users 0 members are here
Options
Related

Broken segmentation in recent Studio updates (newer than 2015 SR3 and 2017 SR1 onwards)

Evzen Polenka

I'm creating this separate thread for paragraph based segmentation issues I mentioned elsewhere:

https://community.sdl.com/solutions/language/translationproductivity/f/90/p/13299/46835#46835
(additional info in https://community.sdl.com/product-groups/translationproductivity/f/90/p/13299/48428#48428)

I'm attaching test files here:
5383.testfiles.zip

ZIP contains:
IAD_Pseudo-HTML.sdlftsettings - file type definition
test_en-US_de-DE.sdltm - testing TM with the additional Line break segmentation rule
test_English.txt.html - source file to test with
test_English.txt - original client's source format for reference only (it's very weird hybrid of plain text and HTML content with some custom non HTML-compliant tags and some "Excel formulas-like code"... normally it contains more of that 'garbage', but I deleted it as it's not relevant now)

Now, already in the "working" version I see some weirdness in the LineBreak rule - the only rule which works is this weird one:
.?[\r\n][\r\n]
The logical one - .?[\r\n]+ - does NOT work for unknown reason :-O
But at least something works...

In Studio 2015 SR3 (12.3.5262.0) and 2017 CU5 (14.0.5889.5) I get this - all properly segmented as expected...

In Studio 2015 SR3 CU10 (12.3.5281.10) and 2017 SR1 CU7 (14.1.6329.7) I get this - segments broken at totally weird places in a middle of a word... but at the same time NOT broken at the line break (sometimes :-O)
It must have something to do with the inline locked content, since plaintext lines further down the test file are segmented correctly.

And more weirdly, I am not able to find regex which would make it work in these new Studio updates!
Neither the basic .?[\r\n]+ regex, nor the [\w\p{P}][\r\n]+ regex according this knowledge base article work - the basic regex produces exactly the same weirdness as above, the regex from the KB article gets better, but fails on lines with trailing spaces (and adding \s* anywhere in the regex yields again exactly the same weird result as above):

So, my conclusion is that there is (still) something broken in the segmentation rules.
Rate translation
×
Suggest better translation
×
Moderator UI
×
  • Get AI Suggestion
Parents
  • 0

    Thanks Evzen, this is being logged with development for resolution. I’ll get you the bug number when it’s done.

    Paul Filkin | RWS

    Design your own training!
    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub

  • 0 in reply to Paul Filkin

     , do you have some news regarding this? At least something like "yep, it's confirmed to be a bug"...
    I'm not really asking for ETA ;-), but would welcome some info too. The thing is that we are being "pushed" to move to GroupShare Cloud 2017, but that requires at least 2017 SR1 to work properly... but without properly working segmentation it's absolutely out of question to update from good-working CU5 :-\

    EDIT: And any info about possible update for 2015 will be welcome as well.
  • 0 in reply to Evzen Polenka
    Hi ,

    Apologies for not getting back to you sooner on this. Patrik confirmed this as a bug and that he would try to address it in the next release of the component. I can't find the bug number at the moment but will come back to you on that one.

    Paul Filkin | RWS

    Design your own training!
    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub

  • 0 in reply to Evzen Polenka
    Hi ,

    Apologies again for not following up on this. I just found the bug report and can confirm the conclusion from development. First of all it is not seen as a bug so there will be no "fix" through an update.

    The problem seems to be the use of the regex. If you don't use this:

    .?[\r\n][\r\n]

    And work with groups instead of lists then it works:

    .?(\r\n)(\r\n)

    I note the KB has not been changed so I'll report this too. In theory the original regex should work as it's valid for this case, and it did work in 2015, but the current status of this one has been set as no fix by the development team so I think we need to at least quote both options.

    If you want to discuss this further it may be helpful to have the bug number which is CRQ-8233

    Paul Filkin | RWS

    Design your own training!
    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub

  • 0 in reply to Paul Filkin
    HUH, that can't be true... you must be kidding :-(
    Looks like someone completely missed the point :(
    The core problem is _NOT_ the "use of the regex"... The core problem is the obviously broken segmentation.
    The regex problem is secondary. And the issue in the regex is that there is absolutely NO REASON why such weird construct would be needed at all! It should work with the simplest
    .[\r\n]+
    regex! The fact that such weird construct is required - and it does NOT matter if its lists or groups - to make it work IS CLEAR SIGN OF A BUG.

    This is really frustrating... :-\
  • 0 in reply to Paul Filkin

    Paul said:
    In theory the original regex should work as it's valid for this case, and it did work in 2015, but the current status of this one has been set as no fix by the development team

    This is what bugs me to the extreme - so it doesn't matter what's right or wrong... some developer simply says "I don't bother fixing it" and that's it... who cares about correct functionality... he's the mighty developer and he has the last word, no matter how wrong it may be :(
  • 0 in reply to Evzen Polenka

    Hi Evzen,

    Welcome to the world of homo economicus codicus. The budget isn't big enough to solve everything and your case is unlikely to affect anyone else. So you have been given a workaround.

    My irritation with Dragon and Excel (for some reason Studio invokes Excel when it does not need to do so, thereby screwing up Dragon every once in a while) is at an even lower level. There is apparently no workaround, so I get an ignore, which does not surprise me.

    You should be happy to get a workaround. You are lucky.

    I imagine that a lot of effort is being spent on getting fragment matching and lookahead working. And given that the bug list is so long, these new bells and whistles probably only got a budget in the first place because the competition was offering something similar.

    Can't fall behind the competition.

    But now SDL has to throw even more money at these things to get them to actually work, so they won't have to advise us to turn off all the new features any more (bad marketing).

    Of course, these new bells and whistles will be great once they work, so I am looking forward to the marketing hype finally becoming reality.

    But in the meantime, unless your bug has the potential to affect a good chunk of the user base, I wouldn't hold my breath.

    Best regards,
    Bruce Campbell
    ASAP Language Services
  • 0 in reply to Bruce Campbell
    The point here is that the bug DOES affect enough people... because it's about "segmentation after line break" - which is an elementary and fundamental feature - not working as it should. So it's not just "my bug", it's the silent majority's bug.

    As I mentioned elsewhere, creating "manager's new toys" can come ONLY after the foundation is solid and works as it should (not "as it was designed", since the design was apparently BAD at many places... but "as it should").
    Because then even creating the bells and whistles is much easier... since the underlying foundation WORKS as it should.

    For example, how come that after so many years "get default project template configured in Studio" via API does not return ACTUAL default template, but "Default.sdltpl" instead?!?! This is TERRIBLE ommission (the developer AND tester AND their manager should get some financial penalty) and terrible BUG!
    How can someone even try to build something on top of such BUGGY foundation?!
  • 0 in reply to Paul Filkin

    Paul said:
    The problem seems to be the use of the regex. If you don't use this:

    .?[\r\n][\r\n]

    And work with groups instead of lists then it works:

    .?(\r\n)(\r\n)

    Paul, looking closer at your "workaround" I just have to PROTEST loudly against it.
    The regex you guys suggest clearly means something COMPLETELY DIFFERENT!

    It means "sequence of CR LF CR LF characters", which is fundamentally different from "at least one of the CR or LF character"!

    While the first means "exactly two consecutive DOS/Windows linebreaks", the proper one (I mean the PROPER one, NOT the one mentioned as weird workaround which actualyl should NOT work!!!) means "one or more either Linux, or Mac, or DOS/Windows linebreaks"

    Who gave the "won't fix" conclusion and how much experience with regexes that person has?!?!
  • 0 in reply to Evzen Polenka

    Hi ,

    since we touched the segmentation area in the other thread, I gave a try the Studio 2017 SR1 CU9 and the "solution" of the broken segmentation problem provided by development...

    ...and, unsurprisingly, the wannabe-workaround DOES NOT WORK. Has the developer even bothered to CHECK it?

    Of course it cannot work, since it sets the segment break to TWO CONSECUTIVE linebreaks... i.e. SINGLE linebreak does NOT trigger segmentation.

    Okay, so I tried with single (\r\n) group... but that produces totally crippled output (segment breaks in a middle of word in lines with internal tags, while single linebreaks still completelly ignored... and fullstops also ignored - see the "You have not selected..." sentence in segment 7):

    So the question still stands: When do we get the segmentation finally fixed?!
  • 0 in reply to Evzen Polenka

    Hi Evzen,

    I think you are right.

    If something like this is not considered serious enough to even make it onto the bug list, that gives you an idea of how serious the "real" bugs are that actually make it onto the list.

    And knowing there is such a long list of "real" bugs doesn't exactly inspire confidence in Studio.

    The question in my mind is whether SDL will ever get out of the hole they have gotten themselves into, so that they can start to think about rewriting sections of the decades-old Studio code. I wouldn't hold my breath.

    Instead, let's just hope they can get rid of some of the backlog of "real" bugs in the next few years, so that what they consider "insignificant" bugs like this one can be allocated some resources.

    Best regards,
    Bruce Campbell
    ASAP Language Services
Reply
  • 0 in reply to Evzen Polenka

    Hi Evzen,

    I think you are right.

    If something like this is not considered serious enough to even make it onto the bug list, that gives you an idea of how serious the "real" bugs are that actually make it onto the list.

    And knowing there is such a long list of "real" bugs doesn't exactly inspire confidence in Studio.

    The question in my mind is whether SDL will ever get out of the hole they have gotten themselves into, so that they can start to think about rewriting sections of the decades-old Studio code. I wouldn't hold my breath.

    Instead, let's just hope they can get rid of some of the backlog of "real" bugs in the next few years, so that what they consider "insignificant" bugs like this one can be allocated some resources.

    Best regards,
    Bruce Campbell
    ASAP Language Services
Children
No Data