You are currently working on UAT 

  • State Suggested Answer
  • Replies 13 replies
  • Answers 3 answers
  • Subscribers 429 subscribers
  • Views 5756 views
  • Users 0 members are here
Options
Related

Broken segmentation in recent Studio updates (newer than 2015 SR3 and 2017 SR1 onwards)

Evzen Polenka

I'm creating this separate thread for paragraph based segmentation issues I mentioned elsewhere:

https://community.sdl.com/solutions/language/translationproductivity/f/90/p/13299/46835#46835
(additional info in https://community.sdl.com/product-groups/translationproductivity/f/90/p/13299/48428#48428)

I'm attaching test files here:
5383.testfiles.zip

ZIP contains:
IAD_Pseudo-HTML.sdlftsettings - file type definition
test_en-US_de-DE.sdltm - testing TM with the additional Line break segmentation rule
test_English.txt.html - source file to test with
test_English.txt - original client's source format for reference only (it's very weird hybrid of plain text and HTML content with some custom non HTML-compliant tags and some "Excel formulas-like code"... normally it contains more of that 'garbage', but I deleted it as it's not relevant now)

Now, already in the "working" version I see some weirdness in the LineBreak rule - the only rule which works is this weird one:
.?[\r\n][\r\n]
The logical one - .?[\r\n]+ - does NOT work for unknown reason :-O
But at least something works...

In Studio 2015 SR3 (12.3.5262.0) and 2017 CU5 (14.0.5889.5) I get this - all properly segmented as expected...

In Studio 2015 SR3 CU10 (12.3.5281.10) and 2017 SR1 CU7 (14.1.6329.7) I get this - segments broken at totally weird places in a middle of a word... but at the same time NOT broken at the line break (sometimes :-O)
It must have something to do with the inline locked content, since plaintext lines further down the test file are segmented correctly.

And more weirdly, I am not able to find regex which would make it work in these new Studio updates!
Neither the basic .?[\r\n]+ regex, nor the [\w\p{P}][\r\n]+ regex according this knowledge base article work - the basic regex produces exactly the same weirdness as above, the regex from the KB article gets better, but fails on lines with trailing spaces (and adding \s* anywhere in the regex yields again exactly the same weird result as above):

So, my conclusion is that there is (still) something broken in the segmentation rules.
Rate translation
×
Suggest better translation
×
Moderator UI
×
  • Get AI Suggestion
  • 0 in reply to Evzen Polenka

    Hi Evzen,

    I think you are right.

    If something like this is not considered serious enough to even make it onto the bug list, that gives you an idea of how serious the "real" bugs are that actually make it onto the list.

    And knowing there is such a long list of "real" bugs doesn't exactly inspire confidence in Studio.

    The question in my mind is whether SDL will ever get out of the hole they have gotten themselves into, so that they can start to think about rewriting sections of the decades-old Studio code. I wouldn't hold my breath.

    Instead, let's just hope they can get rid of some of the backlog of "real" bugs in the next few years, so that what they consider "insignificant" bugs like this one can be allocated some resources.

    Best regards,
    Bruce Campbell
    ASAP Language Services
  • 0
    Hi , ,

    There has been a major refactoring of the segmentation engine in which we have incuded your files for testing as well as some other customer issues related to strange segmentation behaviour. This work was not trivial which is why it didn't make it into a patch, and we had reached the point where each attempt to fix the problems was making more work elsewhere. I can't give you any details on the release vehicle for the fixes but I wanted to make you aware that we have in fact done them and they are being tested. Once we have a Beta available I'll come back to you and see whether you are interested to participate in the testing to confirm that things are working as required.

    Regards

    Paul

    Paul Filkin | RWS

    Design your own training!
    You've done the courses and still need to go a little further, or still not clear? 
    Tell us what you need in our Community Solutions Hub

  • 0

    Seeing the following lines in Studio 2017 CU13

    • Fixed the way whitespace is accounted for when splitting text (marking enders) (CRQ-9468).
    • Fixed handling of the text separated by CRLF characters in certain circumstances (CRQ-9467).
    • Fixed segmentation for line breaks resulting in incorrect segments for certain files (CRQ-8233).

    I did a quick test to see if it fixes the problem reported in this thread... and yes, it seems to be ALMOST(?) fixed.

    Even using the simplest appropriate regex for linebreak [\r\n]+ gives the desired result (with the current custom parser settings):

    However... if I further finetune the file type settings and change the "NOT" rule segmentation hint to "Include with text" (to get the complete sentence in segments 1 and 6), I still get some broken segmentation:

    Perhaps it has something to do with the combination of the "Finish" line having trailing whitespaces and the next line starting with tag...