Building a Lightweight Discovery Interface for Chinese Patents, Presented by Eric Pugh at...

BUILDING A LIGHTWEIGHT DISCOVERY INTERFACE FOR CHINESE PATENTS

ERIC PUGH | [email protected] | @dep4b

mailto:[email protected]

Who am I?

• Principal of OpenSource Connections - Solr/Lucene Search Consultancy http://bit.ly/OSCCommercialSummary

• Member of Apache Software Foundation

• SOLR-284 UpdateRichDocuments (July 07)

• Fascinated by the art of software development

http://bit.ly/OSCCommercialSummary

Co-AuthorN

ext Edition May!

Congrats to Trey and Tim!(Tim is here somewhere)

Agilista

Selected Customers

Telling some stories

Telling some storieswar ^

Risks

• Cloud new at USPTO

• Discovery is tenuous concept

• Conflicting User Goals

• Fixed Budget: trade scope for budget/quality

• First USPTO application in “the cloud”

• Simple, and discoverable

• Expresses our philosophy of “Cloud meets Ocean”

!

• Check it out at http://gpsn.uspto.gov

http://gpsn.uspto.gov


➡How to inject “Discovery” into your app

• The Cloud to the Rescue (sorta!)

• Parsers and Parsers and Parsers

• Don’t be Afraid to Share!

Flow of understanding

Data UnderstandingInformation

Building “Discovery”

UX DataTension

Building “Discovery”

Engine

UX DataTension

Grok data at gut level

Look for outliers

!

!

User Interviews

Surveys

Card Sorting

Scenarios/Personas

!

UX

Data

brainstormMockups

Proof of concept

!

!

Where to spend time?

UX

Engine

Data

40%

!

20%

!

40%

!

Where to spend time?

UX

Engine

Data

40%

!

20%

!

40%

!

40%

!

40%

!

20%

We spent

!

!


• How to inject “Discovery” into your app

➡The Cloud to the Rescue (sorta!)



Boy meets Girl Story

Boy meets Girl Story

Metadata

Ingest Pipeline

Discovery UX

Content Files

How we built it

EmberJS Single Page Search App

HTML

XML

JSON

Server Dashboard

GPSN UI (Bootsrap CSS)

BrowsersMobile/

Tablet

Third Party Application

Servers

S3 BucketSolr

Lessons Learned

Don’t Move Files

• Copying 5 TB data up to S3 was very painful.

• We used S3Funnel which is “rsync like”

• We bought more network bandwidth for our office

Never underestimate

the bandwidth of a station wagon

full of tapes hurtling down the highway.

–Andrew Tanenbaum, 1981

Data Size

0

250000

500000

750000

1000000

1985 1987 1989 1991 1993 1995 1997 1999 2001 2003 2005 2007 2009 2011

Patent Count

277871

Think about Data Volume• Started with older dataset, and tasks like TIFF -> PNG

conversion became progressively harder. Map/Reduce nice, need more visibility into progress..

• Should have sharded our Search Index from the beginning just to make indexing faster and cheaper process (500 gb index!)

• 8 shards dropped time from 12 hours to 2 hours. Merging took 5!

• We had too many steps in our pipeline

Building a Patents IndexM

achi

ne C

ount

0

75

150

225

300

5 days 3 days 30 Minutes

1 5

300

Key scaling concept behind GPSN:

!

Cloud meets Ocean

More prosaically…

Database

Server

Server

Server

Client

Client

Client

$

$

$

$

More prosaically…

Database

Server

Server

Server

Client

Client

Client

$

$

$

$$

More prosaically…

Database Server

Client

Client

Client

$

$

$

$$

More prosaically…

Database Server

Client

ClientClient

Client

$

$

$

$$

Client

More prosaically…

Database Server

Client

ClientClient

Client

$

$

$

$ $$

Client

$




➡Parsers and Parsers and Parsers


Why so many pipelines?Morphlines

Tika as a pipeline?

Lot’s of File Types

• Sometimes in ZIP archives, sometimes not!

• multiple XML formats as well as CSV and EDI

• Purplebook, Yellowbook, Redbook,Greenbook, Questel, SIPO…

Tika as a pipeline!

• Auto detects content type

• Metadata structure has all the key/value needed for Solr

• Allows us to scale up with Behemoth project (and others!).

Lots of files!HHHHHT APS1 ISSUE - 760106!PATN!WKU 039302717!SRC 5!APN 5328756!APT 1!ART 353!APD 19741216!TTL Golf glove!ISD 19760106!NCL 4!ECL 1

<PatentGrant>! <BibliographicData>! <GrantIdentification>! <DocumentKindCode>B1</DocumentKindCode>! <GrantNumber>06644224</GrantNumber>! <CountryCode>US</CountryCode>! <IssueDateText>2003-11-11</IssueDateText>

Detector to pick Filepublic class GreenbookDetector implements Detector { ! private static Pattern pattern = Pattern.compile("PATN"); @Override public MediaType detect(InputStream stream, Metadata metadata) throws IOException { ! MediaType type = MediaType.OCTET_STREAM; InputStream lookahead = new LookaheadInputStream(stream, 1024); String extract = org.apache.commons.io.IOUtils.toString(lookahead, "UTF-‐8"); ! Matcher matcher = pattern.matcher(extract); ! if (matcher.find()) { type = GreenbookParser.MEDIA_TYPE; } ! lookahead.close(); return type; } }





➡Don’t be Afraid to Share!

Your BigData solution isn’t perfect

• Allow users to export data

• Most business users want to work in Excel! Accept it!

• Allow other applications to build on top of it.

GPSN has• Lots of easy “Print to

PDF” options.

• Data stored in S3 as:

• individual patent files

• chunky downloads.

• Filtering to expand or select specific data sets.

• Permalinks: simple, very sharable URLs.

• Underlying Solr service is exposed to public via proxy. You can query Solr yourself.

• Need advance querying? Use Lucene syntax in search bar.

One more thought...

Measuring the impact of our algorithms

changes is just getting harder with Big Data.

www.quepid.com

Quepid: Give your Queries some Love

Project SolrPanl

http://www.quepid.com

www.quepid.com

Quepid: Give your Queries some Love

Project SolrPanl

We

needbeta users!

http://www.quepid.com

Thank you! !

Questions?

• [email protected]

• @dep4b

• www.opensourceconnections.com

• slideshare.com/o19s

Nervous about speaking up? Ask

me later!

mailto:[email protected]

http://www.opensourceconnections.com

Building a Lightweight Discovery Interface for Chinese Patents, Presented by Eric Pugh at...

Technology

Transcript of Building a Lightweight Discovery Interface for Chinese Patents, Presented by Eric Pugh at...