Richmond User Group - kcura-media.s3.amazonaws.com€¦ · © kCura LLC. All rights reserved....

© kCura LLC. All rights reserved.

Richmond User Group

A Deep Dive on Analytics Regular Expressions


• Review of Regular Expressions (RegEx)

• Structured Analytics Text Overview

• RegEx Use Case Scenarios

• RegEx: dtSearch vs. Structured Analytics

• RegEx Validation Tool: How to Access

Agenda – Beginners Regular Expressions for dtSearch


Review of Regular ExpressionsRegEx Examples & Common Metacharacters


\w+@\w+\.\w+

[a-z]{2} [0-9]{5}

[1-9][0-9]{2}-[0-9]{4}

\d{1,2}/\d{1,2}/\d{2,4}

\w+([-+.']\w+)*@\w+([-.]\w+)*\.\w+([-.]\w+)*

Examples of Regular Expressions


• dtSearch (and Data Grid’s Lucene search)– Search for words that have a particular pattern– Each pattern matches one whole “word”; multiple consecutive words in separate expressions;

everything lowercase– E.g. find part numbers with a particular pattern

• Structured analytics– Remove (i.e. filter out) content using a single pattern designated in Structured Analytics Set– Can span words, or even multiple lines; case sensitive by default– E.g. strip Bates stamps before running email threading

• Conceptual analytics– Remove (i.e. filter out) content using one or more patterns attached to the Analytics Profile– Can span words, or even multiple lines; case sensitive by default– E.g. strip foreign language headers before indexing

Where do Regular Expressions live in Relativity?


\d Whole number 0-9

\w “word” character (alphanumeric plus underscore)

\W “non-word” character (symbols or whitespace, basically)

. Any character (including \w and \W)

[a-z] Any character a-z

[0-9] Any whole number 0-9

(abc) Capture group abc

| Alternation (i.e. “or”)

{n} Preceding character / group occurs n times exactly

{n,m} Preceding character / group occurs at least n times, and at most m times

? Character/group preceding ? occurs 0 or 1 time only

* Character/group preceding * matches 0 or more times

+ Character/group preceding + matches 1 or more times

Common Metacharacters


• You can activate RegEx in dtSearch by prepending your search string with "## and ending it with ". There is nothing extra to install.

• Thus, a dtSearch RegEx might look like:

"##[a-z]{4}[0-9]{4}"

• "##" signals to dtSearch to treat the encapsulated string as a RegEx. When using RegEx in Relativity Analytics, you should not incorporate "##".

Ensure that you have straight quotes "" and not curly quotes “”. Curly quotes

will cause the RegEx to break.

Never use capital letters in your RegEx as this will return 0 results because dtSearch

normalizes all characters to lowercase.

Getting started—quick tips for dtSearch RegEx


• You can activate RegEx in Data Grid by encapsulating your search string with / (slash) characters. There is nothing extra to install.

• Thus, a Data Grid RegEx might look like:

/[a-z]{4}-[0-9]{4}/

• “/” signals to Lucene (Data Grid’s keyword engine) to treat the encapsulated string as a RegEx.

Unlike in dtSearch, in Data Grid, regulareexpressions are not case sensitive;

therefore, your results are the same whether you use capital or lower-case letters.

Note on Data Grid RegEx


Structured Analytics Text OverviewExtracted Text Configuration


What text looks like in the Extracted Text Viewer:

What text looks like in the Structured Analytics pipeline à one long string:

Goal: Your RegEx filter needs to match for how the text appears in the Structured Analytics pipeline, not the Extracted Text Viewer.

Structured Analytics Text Overview


Exercises Overview


• 8 Exercises, 1 email threading set in each exercise

• Each email threading set consists of 3 duplicate emails

• Each email contains extraneous text such as Bates numbers

• Email threading analysis has been run on each set with the extraneous text, which has caused overly inclusive results

• We will build Regular Expressions to filter out the extraneous text

• We will rerun a full analysis on each email threading set with the appropriate RegEx filter attached

Exercises Overview


The following exercises are based on actual events. Some names and details have been changed to protect

the privacy of individuals.


Bates NumbersExercise #1


Unique production Bates numbers on the face of each email are causing all emails to be incorrectly identified as inclusive and non-duplicate spare. Examples of text to filter out:

RKCDC-012275 RKCDC 012275 RkCdc012275

1) Click on shortcut #1 – Validate to access the RegEx validation tool. Preview the extracted text in the SA pipeline.a) To view the text in the extracted text viewer, open the saved search, #1 Email Threading.

2) Construct a RegEx and insert into the filter RegEx 01. Refer to the RegEx Analytics Workshop application.3) Run Full Analysis on the SA set, #1 Email Threading. 4) Switch to the RegEx validation tool to determine if your RegEx filtered out the appropriate text. Click F5 to refresh to

see the RegEx filter in action.

RegEx to Apply:

(?i)(rkcdc[\s-]?[0-9]{6})

Exercise #1: Bates Numbers


Confidentiality FootersExercise #2


Confidentiality footers scattered throughout each email are causing the emails to be incorrectly identified as inclusive and non-duplicate spare. Example of text to filter out:

CONFIDENTIALITY NOTE: This message and any attached documents contain information which may be confidential, subject to privilege or exempt from disclosure under applicable law. If you are not the intended recipient of this transmission, you are hereby notified that any distribution, disclosure, printing, copying, storage, modification or the taking of any action in reliance upon this transmission is strictly prohibited.


2) Construct a RegEx and insert into the filter RegEx 02. Refer to the RegEx Analytics Workshop application.3) Run Full Analysis on the SA set, #2 Email Threading. 4) Switch to the RegEx validation tool to determine if your RegEx filtered out the appropriate text. Click F5 to refresh to see the RegEx filter in

action.

RegEx to Apply: (?i)(confidentiality note: this message.*?strictly prohibited)

Exercise #2: Confidentiality Footers


Bates Numbers & Confidentiality FootersExercise #3


Two sets of Bates numbers, in addition to confidentiality footers, on the face of each email are causing all emails to be incorrectly identified as inclusive and non-duplicate spare. Examples of text to filter out:


2) Construct a RegEx and insert into the filter RegEx 03. Refer to the RegEx Analytics Workshop application.3) Run Full Analysis on the SA set, #3 Email Threading. 4) Switch to the RegEx validation tool to determine if your RegEx filtered out the appropriate text. Click F5 to refresh to see the RegEx filter in

action.

RegEx to Apply:

(?i)(acme[0-9]{6})|(rkcdc[\s-]?[0-9]{6})|(confidentiality note: this message.*?strictly prohibited)

Exercise #3: Bates Numbers & Confidentiality Footers


URLs & Image TagsExercise #4


URLs and image tags applied to emails by servers are causing them to be incorrectly identified as inclusive and non-duplicate spare. Examples of text to filter out:

<https://urlcheck.verifyme.com/v2/url?u=http-A__www.acme.com&d=BQMFAg&c=Qbuvbn0JdC8PHg&r=9Rm8&s&e> <http://defensepoint.check.com/v2/url?u=http-Z__www.acme.com&d=gvLXh0dAZvjikJgxLnjN5bbynKJ1x9Rm8&s&e> {Description: cid:5BBE39F7-447B-4557-9DDF-2549607EAF24} {Description: cid:[email protected]} {cid:[email protected]}


2) Construct a RegEx and insert into the filter RegEx 04. Refer to the RegEx Analytics Workshop application.3) Run Full Analysis on the SA set, #4 Email Threading. 4) Switch to the RegEx validation tool to determine if your RegEx filtered out the appropriate text. Click F5 to refresh to see the

RegEx filter in action.

RegEx to Apply: (?i)<https?://.*?>|\{description:.*?\}|\{cid:.*?\}

Exercise #4: URLs & Image Tags


Greedy Match ExampleExercise 5: Beware the .*


The combination of .* is useful to match across chunks of text as we have been doing. Let’s see what happens if you do not apply ? to the end of .*, thus making .* a greedy match (a RegEx that will strip out too much text). Then rerun the SA set using a non-greedy RegEx. Example of text to filter out:

{GET RID OF ALL TEXT BETWEEN BRACKETS, including the brackets}


2) Insert the Greedy Match RegEx into the filter RegEx 05. Refer to the RegEx Analytics Workshop application.3) Run Full Analysis of the SA set, #5 Email Threading.4) Switch to the RegEx validation tool to determine if your RegEx filtered out the appropriate text. Click F5 to refresh to

see the RegEx filter in action.5) Rerun the SA set with Non-Greedy Match to see how the text should have been filtered.

Greedy Match RegEx: \{.*\}

Non-Greedy Match RegEx: \{.*?\}

Exercise #5: Greedy Match Example – Beware!


Top Header Extraneous TextExercise #6


Extraneous text at the top of each email, right before the very first instance of the email header From:, is causing all emails to be incorrectly identified as inclusive and non-duplicate spare. Unfortunately this “junk” text does not accord with any pattern that you can discern. Examples of text to filter out:

II XYZ IV @#$* ( II 123 II 456 II %& ABC456 II %& ABC IV @#$* ( II 123 II

Issue to Consider: You can construct a RegEx to match for any string of text that contains letters, numbers, and symbols (use \S). But if you do not limit the RegEx to the beginning of each email, you will filter out valid text.


2) Construct a RegEx and insert into the filter RegEx 06. Refer to the RegEx Analytics Workshop application.3) Run Full Analysis on the SA set, #6 Email Threading. 4) Switch to the RegEx validation tool to determine if your RegEx filtered out the appropriate text. Click F5 to refresh to see the

RegEx filter in action.5) Rerun the SA set without the ^ anchor to see what happens to the text.

RegEx to Apply: ^(\S.*?(\r\n))

Exercise #6: Top Header Extraneous Text


Bottom Footer Extraneous TextExercise #7


Similar to Exercise #6, but now extraneous footers attached to the bottom of each email are causing all emails to be incorrectly identified as inclusive and non-duplicate spare. Each chunk of footer text will always be preceded by the phrase, “Message IDs”. Examples of text to filter out:

Message IDs:Received: from ACME.ACME-R-US.localX-MS-Exchange-Organization-AuthAs: Anonymous X-MS-Exchange-Organization-AuthAs: XYZ




RegEx to Apply: (?i)(message ids:.*)

Exercise #7: Bottom Footer Extraneous Text


Conceptual Analytics Cleanup: Foreign Language Headers

Exercise #8


You are building a conceptual Analytics index on a set of Danish emails. But the Analytics engine can only filter out English email headers and footers. We will focus on headers in this scenario. To prevent the Danish headers from clouding your Analytics index, you can construct a RegEx filter to strip out each header line. Then link the RegEx filter to your Analytics profile. Example of Danish headers:

Fra: Mary Jones <[email protected]>Til: Alle ACME <[email protected]>CC: John Smith <[email protected]>Sendt: 2016/07/12 08:43:05




RegEx to Apply: (?i)(Fra:.*?\r\n)|(Til:.*?\r\n)|(CC:.*?\r\n)|(Sendt:.*?\r\n)

Exercise #8: Foreign Language Email Headers


Regex Validation ToolHow to access the tool on your

Analytics server


1. This tool was created by kCura developers to assist users in validating their RegEx filters, as well as verifying many other operations.

2. There is no separate installation involved. The tool is a user interface that is included with your install of Relativity Analytics.

What Is the RegEx Validation Tool?


1. Run a Full Analysis of your Structured Analytics Set.

2. After the set has completed, take note of the workspace ArtifactID and the ArtifactID of the structured analytics set. This is how the structured analytics set is identified on the backend.

3. Open the REST UI on the Analytics Server. The URL will be similar to: https://localhost:8443/nexus/r1/.

4. You will need to log in with the REST credentials. Contact your Infrastructure Manager for assistance.

5. Navigate to the structured analytics set which will be listed under "staging." This is an example URL:

https://localhost:8443/nexus/r1/staging/1015598_1248495

This is the breakdown of the URL:

https://AnalyticServer:8443/nexus/r1/staging/CaseArtifactID_StructuredAnaSetArtificatID

6. You are now viewing the structured analytics set page. From here, you can look at the text in any item. You can navigate to alist of all documents by clicking on "Manage staging area item data." If you have the document ID that you want to evaluate, youcan also navigate straight to it by pasting the ID to the end of the URL, as follows:

https://localhost:8443/nexus/r1/staging/1015598_1248495/item/EN000017

How to Access the Structured Analytics RegEx Validation Tool


7. To validate a RegEx filter, look at the document’s text. You can do this by clicking on the "text" hyperlink towards the bottom of the screen. This will take you to a URL as follows:

https://localhost:8443/nexus/r1/staging/1015598_1248495/item/EN000017/text/text

You will now be able to see the extracted text of the document as it appears in the Analytics pipeline, as shown here:

How to Access the Structured Analytics RegEx Validation Tool – cont.


RegEx: dtSearch vs. Structured AnalyticsKey differences


• Lower case only

– Capital letters will never match

• Spaces seen as word breaks

– \s will never match

• Need to encapsulate string with "##" so that it can be recognized as RegEx

– E.g. "##[0-9]{3}[a-z]{10}"

• Case sensitive

– Use (?i) to make case insensitive

• Any spaces are normalized to single space

– \s will match

• No need to add special syntax for string to be recognized as RegEx

RegEx dtSearch RegEx Structured Analytics


Wrap-up


• To succeed with RegEx, start small and build

• Take it piece by piece, character by character

• Mind your backslashes!

• Be safe

– Always consider the false positives and worse case scenarios (especially when filtering)

– Be as restrictive as possible (don’t use \w when you could use \d)

• With great power comes great responsibility

Final thoughts on regular expressions


RegEx Documentation

Using Regular Expressions with Structured Analytics

This documentation page contains an overview of how to use Regular Expressions with structured analytics, as well as descriptions of common RegEx metacharacters.

How to access the RegEx Validation Tool on your Analytics server

Link to Recipe

This recipe contains step by step instructions, with screen shots, of how to access the Structured Analytics RegEx Validation Tool.

RegExr

regexr.com

RegExr is useful for testing whether your regular expression is valid. It is a very simple, lightweight RegEx tester. RegExr also contains sections with helpful hints and explanations.

Regular Expression User Guide

www.zytrax.com/tech/web/regex.htm

This is an in-depth guide to regular expressions.

Resources


• https://xkcd.com/208/

• https://www.explainxkcd.com/wiki/index.php/208:_Regular_Expressions

• https://store.xkcd.com/collections/apparel/products/i-know-regular-expressions

RegEx Humor


Questions?


Thank You!

Richmond User Group - kcura-media.s3.amazonaws.com€¦ · © kCura LLC. All rights reserved....

Documents

Transcript of Richmond User Group - kcura-media.s3.amazonaws.com€¦ · © kCura LLC. All rights reserved....