International Directory Network (IDN)Scalability, Security and Interoperability

WGISS, 2006

Tom NorthcuttSystems Administrator: GCMD

September 13, 2006

I. Scalability, Interoperability

0

1000000

2000000

3000000

4000000

5000000

6000000

7000000

8000000

Jan-03

Mar-03

May-03

Jul-03

Sep-03

Nov-03

Jan-04

Mar-04

May-04

Jul-04

Sep-04

Nov-04

Jan-05

Mar-05

May-05

Jul-05

Sep-05

Nov-05

Jan-06

Mar-06

month

#hits

# GCMD/IDN Web Page Hits Since January 2003

Introduction of the new web page

Cache opened to InternetSearch robots

FreetextFreetext (Lucene) Database Index Layer

SpatialSpatial Database Index Layer

Controlled VocabularyControlled Vocabulary Database Layer joinsdelegates

ControllerController

User performs query1)

2)

3)

Middleware Search/Retrieval Component(Integration of spatial, freetext, and controlled queries)

4)

Set of unique IDs Result Result ProcessorProcessor

Multi-Layer Search ComponentMulti-Layer Search Component

5)

Title set information, brief summary, dataset links, etc.

6) Returns to user

SearchSearchResultsResults

Cache7) Refines search

Scalability: Core GCMD/IDN Architecture

Complexity:“it is hard to make things look easy.”

– These are complex queries, with very fast search results. – Another example: data resolution refinement

● difficult to implement● Makes it easier for the user

Scalability

● Conventional clustering approach– Load balancing– High availability– (source: redhat.com)

Scalability: GCMD/IDN ImplementationStateful, Web Proxy Based Clusters

http://gcmd.nasa.gov/Keywords.do?...&lbnode=2

http://gcmd.nasa.gov/DocumentBuilder/...

AcceleratedCaching

Scalability: Extensibility of Stateful Web Proxy Clusters

http://gcmd.nasa.gov/OAI-script? ...

http://gcmd.nasa.gov/ontology.wsdl

http://gcmd.nasa.gov/soap/httphttp://gcmd.nasa.gov/xml-rpc

http://gcmd.nasa.gov/ajax/some.jsp

Harvester

SOAP

XML-RPC

AJAX

Scalability: Stateful Web Proxy Clusters

How we implemented this architecture:– Modified version of Squid proxy server– Custom perl scripts to implement state and redirection– Dynamic query caching done on the server end so each

refinement uses cached results

Scalability:Advantages of Web Proxy Clusters for CEOS Partners● Accelerated Caching● Load balance nodes● Stateful architecture ● Open source

● Multiple uses:– Web services– Browse imagery– Metadata search– Data access and retrieval

Scalability:

Google Map

Scalability:Google Map

● Utility:– Google map is a form of spreading the load– Utilize third party resources for map generation– Google’s resources are distributed globally

II. Security

Security:Transparent Bridge Filters

BRIDGE

Firewall Network monitoringPort remappingIntrusion detection

BRIDGE

Internal FirewallingNetwork monitoring

BRIDGEProject SegregationNetwork protection

Security:Transparent Bridge AdvantagesApplicability for CEOS Partners

● Applicable to heterogeneous environments– Unobtrusive

● No changes needed on servers or network controllers● May assist with GRID network security requirements

– Ultra secure: invisible at the IP level ● Implements emerging security policies

– Two factor authentication– Efficient encryption, authentication– Port knocking capabilities

● Open source– Non-proprietary– Universal

Conclusion

● IDN continues to grow in popularity– Users – Earth science partnerships

● The system continues to develop – Scalability– Security– Usability– Interoperability

End