Post on 10-May-2015
ELASTICSEARCH AT
THE NORWEGIAN TAX SERVICES
Oslo Enterprise Search Meetup
Teknologihuset
Magnus Haug - magnus.haug@bekk.no Jørgen Braseth - jorgen.braseth@bekk.no
18.02.2014
AGENDA
MAG & Search
Examples of use
How we did it
Tools
Conclusion
MAG & SEARCH
Background
MAG
Replacing 19 existing systems
Data for tax returns
~ 200M documents each year
At least 10 years data retention
«We want a Google search»
OUR ALTERNATIVES
Designed for horizontal scaling
Lucene
HTTP or Binary
FINDING REPORTED INFORMATION
Example of use
{CICS}
IDENTIFICATION
Example of use
HOW WE DO IT
Architecture
Document store
ATOM Indexing process
Elasticsearch
ATOMATOM
HOW WE DO: INDEXING
HOW WE DO: COMBINING DATA SOURCES
Document store
ATOM Indexing process
Elasticsearch
ATOMATOM
Person infoCompany info
HOW WE DO: SECURING DATA
Document Index
Read unfiltered Write
Search API Indexing process
Read filtered
HOW WE «DO»: ROBUSTNESS
Rack 1
ES Node 1
ES Node 2
Rack 2
ES Node 3
ES Node 4
Rack 3
ES Node 5
HOW WE DO: REINDEXING
Document store
ATOM Indexing process
Elasticsearch
ATOMATOM
Person infoCompany info
ATOM
HOW WE DO: HOT SWAPPING INDICES
Read Write
Search API Indexing process
New indexOld index
TOOLSEveryday maintenance
Head
Big Desk
Elasticsearch HQ
Kibana
Marvel
Marvel
IMPRESSIONS
What do we think?
PERFORMANCE
100 million docs
Two ES-nodes
< 1000 ms
http://www.sxc.hu/photo/1430721
DEVELOPER FRIENDLY
Easy setup
Easy testing
Easy debugging
cURL!
INTEGRATION TESTING
@RunWith( ElasticsearchRunner.class )
public class MinSystemTest {
@ElasticsearchNode( name = "iris-test", clusterName = "iris-test" )
private static Node node;
@ElasticsearchClient( nodeName = "iris-test" )
private static Client client;
...
SearchRequestBuilder searchRequestBuilder =
client.prepareSearch( INDEKSNAVN ).setQuery( QueryBuilders.matchAllQuery() );
return client.search( searchRequestBuilder.request() ).actionGet();
...
UNUSED FEATURES
Percolate
Highlighting
Suggestions
Rivers
REST-API
FUTURE PLANS FOR ELASTICSEARCH AT MAG
Upgrade to v1.0
Used as a cache
As a service in our PaaS
ARE WE HAPPY?
?