Sarah D 

The message seems to be saying "you have TUs in your file that are not nl-BE".  The implications being that your project in Extract is nl-BE.  So if you change all your codes to something other than nl-BE you'll make things worse, not better.

Perhaps you can confirm the languages of your Extract project, and then confirm the languages in your original TMX?

Thank you for your reply Paul.

I'm not sure what you mean? The target language I used for the extract project was nl-BE, given that the tmx target language was also nl-BE.

 Sarah D 

ok -so read what you have written again.  I can only go on what you're telling us.

Sarah D said:

"Warning! There were translation units (tu) found in file (name of my TMX file) which did not have a translation unit variant (tuv) with target language "Dutch (Belgium)"

It says it found TUs in your file that did not have nl-BE language codes.

You then said this:

Sarah D said:

I was able to change all target tuvs from nl-BE to nl, but that did not help. Changing them to nl-NL did not help either.

Why?  If the TMX contained TUs that were not nl-BE then you have made certain there aren't any at all.

Or of course I'm completely misunderstanding what you have written here?

Maybe you can share your TMX so I can try it?

I realise it's contradictory, but it was the solution that was suggested in the only Q&A I found, to the same problem I had, just with a different target language.

I would be grateful if you could give it a try. What would be a safe way to share my tmx to you privately?

Sarah D 

Sure... pfilkin at sdl dot com

Thank you! I sent you the export yesterday by e-mail.

Sarah D 

Thanks for the file... found some time to look at it this evening.  I get the same error you mentioned, but the application still extracts the terminology:

So, I split the file and kept splitting it until I found the problem.  It relates to the segment contents rather than the language codes so this was a misleading error message.  The segment (<seg>) for both the fr-BE and nl-BE languages contains placeholders (<ph>, <bpt>, and <ept>) but no actual translatable content in one single <tu>. The placeholders refer to certain tag formats or structures that were present in the original document.  If you're extracting terminology, the actual terms or sentences should be inside the <seg> tags and since this file you provided has a single TU with only placeholders and no content Extract seems to get a bit upset!

This is the offending translation unit variant:

      <tuv xml:lang="fr-BE">
        <seg><bpt i="1" type="110" x="1" /><ph x="2" type="114" /><ept i="1" /></seg>
      </tuv>
      <tuv xml:lang="nl-BE">
        <seg><bpt i="1" type="110" x="1" /><ph x="2" type="114" /><ept i="1" /></seg>
      </tuv>

If you search for that and delete the entire translation unit and then try again the error is not there.

Sarah D 

Just to close this off... you can use this regex to quickly find segments like this... also segments with no content at all... and that should save some time trying to find the offending segments:

<seg>(?:\s*<(bpt|ept|ph)[^>]*\/?>\s*)*<\/seg>

Hi Paul

Thank you for your detective work!

You are right, the extraction works despite the bug on the small file (207 kb) with 161 TUs.

Hoewever it doesn't work on our actual database of +265.000 KB and +200.000 TUs I'm actually trying to extract from.

The regex you provided works, but only to find just one faulty TU at a time.

Do you know of a regex that could erase the faulty TU's in our big extraction file from <tu to </tu>?

Hi Paul

Thank you for your detective work!

You are right, the extraction works despite the bug on the small file (207 kb) with 161 TUs.

Hoewever it doesn't work on our actual database of +265.000 KB and +200.000 TUs I'm actually trying to extract from.

The regex you provided works, but only to find just one faulty TU at a time.

Do you know of a regex that could erase the faulty TU's in our big extraction file from <tu to </tu>?

Sarah D 

Sarah D said:

Do you know of a regex that could erase the faulty TU's in our big extraction file from <tu to </tu>?

Do I know of one... no ;-) Can I create one... yes.  But it may not work for all files as it could be dependent on the specific format of your TMX files.  So based on the sample you sent me this seems to work.

Search for this:

<tu[^>]*>(?:(?!<\/tu>).)*(<seg>(?:\s*<(?:bpt|ept|ph)[^>]*\/?>\s*)*<\/seg>|<seg>\s*<\/seg>)(?:(?!<\/tu>).)*<\/tu>

Replace with nothing!

Hm.. We're not quite there yet ...

I tried the string (in Notepad++ I have to add, search mode Regular expression), and 0 results were found.

That shouldn't be possible as it includes the tu of the sample I sent you.

Do you have any more ideas?

Sarah D 

It's often troublesome to use something without fully understanding it!  Make sure you check the "dot matches newline" checkbox... seems to work just fine:

That may be, but I'm determined to find a solution, so willing to learn trying to understand!

I'm getting this error message now:

Sarah D 

And can you paste the regular expression you used in here so we can see it exactly.  Clearly there is nothing wrong with it for Notepad++ or I would have had the same message.  This is what I used:

<tu[^>]*>(?:(?!<\/tu>).)*(<seg>(?:\s*<(?:bpt|ept|ph)[^>]*\/?>\s*)*<\/seg>|<seg>\s*<\/seg>)(?:(?!<\/tu>).)*<\/tu>

Sarah D 

I don't really use Notepad++ but I deliberately broke my expression to see what that little note at the end of the message does and it seems to tell you what's wrong with it if you hover over it:

Maybe share that... or use it to fix your own problem?

Sarah D 

Maybe your Notepad++ version is too old. Latest version is 8.5.6 and fixes are implemented every few days.

Try upgrading from https://notepad-plus-plus.org/downloads/.

Thank you for that suggestion Jesús.

I installed version 8.5.6 only yesterday.

This is the message I'm getting:

Sarah D 

ok - so you must be running this on your larger file?

I think the error is probably due to the complexity of the regular expression, which is taking too long to evaluate.  Regular expressions that have lots of nested repetitions or non-greedy matches can be highly inefficient, especially on large texts.  You may fare better with another editor... I use EditPadPro which handles far larger files than notepad++ can work with, but even with this I don't know how well it would work with your specific file.  You'd need to test and see.

I also don't think I know how to simplify this expression with regex alone as you'd have to start getting too specific and this would remove the benefit of the regex in the first place, so you may be better off using another method to clean up such large files.  Maybe Powershell or Python for example which can make use of some built in libraries that make it easier to manipulate XML.