NJIN 2.0Kate Silhol, Lead Software Engineer

NJIN 2.0 Design

Background

■ XML Message Router (XMR) installed 12

years ago

– Advanced Technology Systems product

– First foray into XML, web services

■ XMR converted to Nlets Justice Information

Network (NJIN)

– Majority of code re-written, now considered in-

house product

Overview

■ Current NJIN is approaching capacity

– Scalability limitations inherent to the 12 year old design

– Volume has increased by 209% in last 10 years

– Manageable but recurring system issues (slowness)

– Low risk mitigating changes made to sustain current NJIN

■ Design work for new system commenced end of 2015

Design Overview and Principles

■ Distributed architecture

■ Three primary components which guide

each transaction through the system

■ Specific processes located in distinct

modules – true SOA

High Level Design

Input/Output Layer

Validation and ESB Initiation Layer (VEIL)

Enterprise Service Bus

Modules

SBG Customized

PMB

Completely

separate code;

“black boxes”

mostly based on

existing code

Initial validation,

broker to the ESB

Input-Output Layer

■ Provides communication between the NJIN

2.0 and the users

■ Leverages SBG’s Plexus Message Broker

Code

■ Communication between Input-Output Layer

and the VEIL is via queues (one queue for

each user/line)

Enterprise Service Bus

■ Backbone of the NJIN 2.0

■ Provides communication between the

system components

■ Dynamic and static metadata will dictate

the actions taken for each transaction

ESB Selection ProcessProduct Open Source/COTS Pros Cons

WSO2

Open SourceCostGreat reputation

Java-based open source –significant ramp-up for Nlets staff

NServiceBus

Open Source .NET based platform

Performance may not be adequateFunctionality specifics limit outside contractor helpUnclear roadmap

Neuron ESBCOTS .NET based, local

Prior experience with toolPoor reputation

MuleSoftCOTS (Open Source also available but inadequate)

Good reputationFunctionality appears to be adequate

Company is focusing on cloud-oriented architecture.Requires on-going support costs

TIBCO

COTS

Reputation/functionality/performance – best in classSupport availabilityAbility to involve contractors

Cost

Home Grown N/A Ability to control functionality Time to implement

Processing Queues

■ Between the ESB implementations and

connected layers and modules,

communication will be via queueing

■ Message durability, prioritization,

guaranteed delivery, etc.

Pipelines

■ The VEIL will apply a Pipeline to the transaction before sending to the ESB

– Set of instructions based upon configurable rules

■ Different Pipelines based on message type, destination, etc.

■ Pipeline may be updated along the way – error conditions, etc.

■ Provide for better configurability and enhanced diagnostics

Sample Catalog of Pipeline Components

Rules ValidationRules Validation

Parsing

Logging

Conversion

Sub Route

Message Header Change

Alerting (PAL/ Graham Scrape )

Federation

Image Resizing

Image Stripping

Special Instructions

Send Message

Caveating

Sample Pipeline Configuration

Rules Validation

Parsing

Conversion

Image Resizing

Send Message

Transactional Objects

■ Each message traversing the NJIN 2.0 will

be wrapped in a Transactional Object

■ Metadata about the transaction and its

processing that will follow the message

through the process

■ Pipeline will be maintained within the

Transactional Object

<NletsTransactionalWrapper>

<NletsTransactionalObject>

<OriginalUniqueID>111111111111</OriginalUniqueID>

<UniqueID>111111111120</UniqueID>

<IPAddress>10.10.10.10</IPAddress>

<InboundHeaderData>

<MessageKey>DR</MessageKey>

<OriginatingORI>AZMVD0000</OriginatingORI>

<DestinationORI>AZNLETS23</DestinationORI>

<ControlField>CONTROLFLD</ControlField>

</InboundHeaderData>

<OutboundHeaderData/>

<MessagePriority>3</MessagePriority>

<InputHeaderFormat>Text</InputHeaderFormat>

<InputBodyFormat>Text</InputBodyFormat>

<OriginalMessage>DR.AZMVD0000.AZNLETS23.CONTROLFLD.TXTDRIVER STATUS

DATA</OriginalMessage>

<ErrorCondition/>

<PrimaryIndicator>true</PrimaryIndicator>

</NletsTransactionalObject>

<TransactionalPipeline>

<PipelineStep>

<StartDateTime/>

<EndDateTime/>

<Service>GetDestinationInfo</Service>

</PipelineStep>

<PipelineStep>

<StartDateTime/>

<EndDateTime/>

<Service>Validation</Service>

</PipelineStep>

</TransactionalPipeline>

<OutboundMessage/>

</NletsTransactionalWrapper>

Enhancements to NJIN

■ Better performance

■ Scalability

■ More detailed and usable logging

■ Improved diagnostics

■ Easier configurability

■ Correlation of inbound message with outbound

■ Spill queues

– Side by side queues allowing new traffic to flow alongside queued traffic following a state outage

■ Schema validation as an option (test system)

■ Statistics

■ Dashboard/console improvements

Monitoring Improvements

Monitoring Possibilities

0

200

400

600

800

1000

1200

1400

1600

0:0

0

1:0

0

2:0

0

3:0

0

4:0

0

5:0

0

6:0

0

7:0

0

8:0

0

9:0

0

10

:00

11

:00

12

:00

13

:00

14

:00

15

:00

16

:00

17

:00

18

:00

19

:00

20

:00

21

:00

22

:00

23

:00

Overall Message Volume

SUM

0

100

200

300

400

500

600

700

800

900

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Volume by Message Key

Message Key AM CR

DQ DR FQ

FR RQ RR

0

500

1000

1500

2000

2500

0:0

0

1:0

0

2:0

0

3:0

0

4:0

0

5:0

0

6:0

0

7:0

0

8:0

0

9:0

0

10

:00

11

:00

12

:00

13

:00

14

:00

15

:00

16

:00

17

:00

18

:00

19

:00

20

:00

21

:00

22

:00

23

:00

0

200

400

600

800

1000

1200

1400

1600

0:0

0

1:0

0

2:0

0

3:0

0

4:0

0

5:0

0

6:0

0

7:0

0

8:0

0

9:0

0

10

:00

11

:00

12

:00

13

:00

14

:00

15

:00

16

:00

17

:00

18

:00

19

:00

20

:00

21

:00

22

:00

23

:00

Overall Message Volume

Total Traffic Volume (Past Hour)

Monitoring Possibilities

Future Possibilities

■ Analytics

■ Proactive scaling and load balancing

■ Proactive alerting functionality

■ Reuse of modules

■ Executive Dashboard

NJIN 2.0 Implementation Plan

Addition of QA/Dev System

■ Current test system doing double duty – we

need a true test system!

■ First implementation of NJIN 2.0 will be in

new QA/Development system

■ No external users – sandbox

■ Then followed by implementations in test

and production

Production

NJIN

Test

NJIN

Production

NJIN 2.0

Test

NJIN 2.0

Development

NJIN 2.0

Phased Approach

■ Same phased approach will be used on the

Test System and Production System

■ Incremental process to minimize risk

■ All steps may be iterated for each individual

state/user in a category

■ All phases have clear roll-back plans

Nlets Users

NJIN 1.0PMB

Nlets Users

Current NJIN

Nlets Users

NJIN 1.0

Input Layer

Output Layer

NJIN 2.0PMB

Nlets Users

Current NJIN

Nlets Users

NJIN 1.0

Input Layer

Output Layer

NJIN 2.0PMB

Nlets Users

Native MQ (PMB)

users sending via

new Input Layer

Nlets Users

NJIN 1.0

Input Layer

Output Layer

NJIN 2.0PMB

Nlets Users

Native MQ (PMB)

users sending and

receiving via new

Input and Output

Layers

Nlets Users

NJIN 1.0

Input Layer

Output Layer

NJIN 2.0PMB

Nlets Users

All users sending and

receiving via new

Input and Output

Layers

Nlets Users

NJIN 1.0

Input Layer

Output Layer

NJIN 2.0PMB

Nlets Users

NJIN 2.0 processes in

parallel (no messages

outbound) to allow

testing and review of

logs

Logging

Nlets Users

NJIN 1.0

Input Layer

Output Layer

NJIN 2.0PMB

Nlets Users

Parallel processing

turned off to prepare

for cutover

Logging

Nlets Users

NJIN 1.0

Input Layer

Output Layer

NJIN 2.0PMB

Nlets Users

Logging

NJIN 2.0 now

performs all

processing

Status

■ Finalizing ESB selection and procurement

process

■ Development work to begin immediately

– Input-Output layers and modules first

■ Timeline: Plan to implement in

approximately one year