Post on 12-Jan-2016
Copyright © 1998-2014 Curt Hill
Foundational Software Systems
A Brief Overview
Introduction
• Certain classes of software need to be understood by all technical people
• Operating Systems• Servers• Databases and Data Warehouses• Enterprise Resource Planning• Customer Relationship
ManagementCopyright © 1998-2014 Curt Hill
Outroduction
• What this presentation is not about:
• Virtualization• Cloud computing • Service Oriented Architecture• Software as a Service• These are important topics to be
considered at another time
Copyright © 1998-2014 Curt Hill
OS
• Foundation of the foundation• Every general purpose computer
has one– Many special purpose computers as
well
• Applications are why we buy the computer, but the operating system is required to run the applications
Copyright © 1998-2014 Curt Hill
What does an OS do?
• Execute application programs• Provide services to these programs• Establish and manage a file system• Utility functions• Many functions that are used by
most programs will eventually be incorporated into the OS
Copyright © 1998-2014 Curt Hill
Execute application programs
• Interpret commands– Interpret typed commands or mouse
actions
• Load and execute programs• Provide services to the running
programs– The exact description of these
services distinguishes different OSs
Copyright © 1998-2014 Curt Hill
Provide services to the program
• Memory management– Memory allocation – Virtual or fixed memory
• I/O management– Manage file activity– Often the most visible and important
• Task Scheduler– Inter-process communication– Managing priorities
Copyright © 1998-2014 Curt Hill
Establish and manage a file system
• Determine how things are stored on disk– Do we have a directory or just search?– Are things stored in a contiguous fashion
or scattered?– How are things named?– How are things accessed?
• Interrupt processing• Utility functions
– Prepare a disk for use– Copy/move/delete files– Show contents of a file or disk
Copyright © 1998-2014 Curt Hill
How are files accessed?
• Access to files is a program service– Program says that it wants access to a file
of some filename– System finds it– Gets it ready– Gives it to the program one chunk at a
time
• Several things should be transparent to program
• Buffering• Devices
Copyright © 1998-2014 Curt Hill
Utility functions• Protection
– Files from deletion by someone other than owner
– Memory of one program from another• Hardware manipulation
– Mount disks and tapes• Miscellaneous
– Error detection– Multitasking– Accounting
Copyright © 1998-2014 Curt Hill
Pieces of the operating system
• CPU management• Memory management• Process management• I/O management• User management• File management• Others
Copyright © 1998-2014 Curt Hill
OS Layers
• Most OSs are onion shaped– For historical and maintenance reasons– Historically the innermost was all that
was available
• Each layer then communicates with adjacent layers through well defined paths– Well defined in terms of this particular
OS
• Layers: Kernel, Service, Command
Copyright © 1998-2014 Curt Hill
Kernel is the innermost layer• Low level activities only
– Load and/or execute a program– Device allocation and access– Process management– Low level file access
• This is the lowest level interface to the hardware and the only one that touches the hardware directly
• In DOS there is the BIOS which amounts to a kernel
• The kernel is usually always resident in the memory
• Basic process management is usually here, since we need to make each program we load a process
Copyright © 1998-2014 Curt Hill
Service layer• Set of routines that provide services
– Use the kernel– This is where the traditional file manipulation
commands are executed, such as opening and closing files, reading and writing records
– Entire file copying is usually done at the next higher level
• The service layer represents the services that are available to a program
• The service layer is also designed to generalize the interface between devices– A CRT, disk and tape are wildly different, yet we
would like their interface to look similar
Copyright © 1998-2014 Curt Hill
Service Layer
• In the assembly language interface each of the service calls becomes a macro or subroutine call
• In a HLL it is usually built into the command part of the language or is in a library of some sort
• Some process management is here as well– Inter-process communication
Copyright © 1998-2014 Curt Hill
Command layer• The interface that allows the user to
interface with the system directly• This includes command interpretation• This is Command.com in DOS or
program manager in Windows• Many utility functions are placed in this
layer– The DOS Format is a command in this layer
• It performs an essential function, but is its own program
Copyright © 1998-2014 Curt Hill
Servers• Client server computing is the basis of
the Internet – Among other things
• Requires two computers– A client – A server
• Communication medium• The illusion is that each client
computer thinks it is has exclusive access to the server
Copyright © 1998-2014 Curt Hill
The Paradigm
• Server provides one service• Waits for requests for that service• Client initiates a request• Server replies to the request• Information may flow in both
directions• Many clients may make requests
with just the one server
Copyright © 1998-2014 Curt Hill
Why Client - Server?
• Specialization– The client handles the user interface
using local computational ability– The server manages the data and
communication
• Sharing– One server may support many clients– Services provided are effectively shared– Data may be maintained in only one
location
Copyright © 1998-2014 Curt Hill
Clients
• An application program that runs on the local computer
• Able to use the local computer for doing work
• When invoked by a user, it initiates communication with a server
• Handles all the user interface issues
• Only used for a single session
Copyright © 1998-2014 Curt Hill
Example• The ubiquitous client is the web
browser• Several functions:• Communicate with server• Display HTML/XML• Interpret client-side languages, such as
JavaScript• The trend is for the web browser to be
the universal interface to every kind of server on the web
Copyright © 1998-2014 Curt Hill
Servers• Specialized program that typically
provides only one service• May handle multiple client requests
simultaneously (or pseudo simultaneously)
• Interacts with users in many sessions– Both simultaneously and over long time
periods
• Waits for a request and then produces the reply
Copyright © 1998-2014 Curt Hill
Example
• The web server• Waits for requests using Hyper Text
Transmission Protocol• Produces results in HTML/XML• May handle many clients
simultaneously
Copyright © 1998-2014 Curt Hill
Connection Diagrammed
Copyright © 1998-2014 Curt Hill
Server
Client
RequestReply
Client Client
Request
ReplyReplyRequest
Interaction Diagrammed
• The server is not waiting if it is processing other requests
• The client may also perform other processing while waiting for the reply
Copyright © 1998-2014 Curt Hill
Client
ServerWait Wait
Wait
Request Reply
Time
Processing
Multiple tiers• The classic client-server is a two tier
system• The client is one tier and the server
another• Three or more tiers are also possible• The middle server functions as a
– Server to the client– Client to other servers– Known as middleware
• Often used to distribute load
Copyright © 1998-2014 Curt Hill
A Three Tier Approach
Copyright © 1998-2014 Curt Hill
Server
Application Server
Client ClientClientClient
ServerServer Server
Examples of Multi-Tier
• Authentication server– Maintains a connection to verify user– Funnels all requests to appropriate
server
• Shopping cart– Records purchases– The product lookup/purchase handled
by other servers
• Distributed Database Management System
Copyright © 1998-2014 Curt Hill
Horizontal Distribution• Multi-Tier is a vertical distribution• A Horizontal distribution may also be
effective• A single server owns the web address:
www.x.com• It then distributes all further requests
to a rotation of identical web servers– www1.x.com– www2.x.com– …– wwwN.x.com
Copyright © 1998-2014 Curt Hill
Horizontal Distribution
Copyright © 1998-2014 Curt Hill
www.x www1.x wwwN.xwww2.x …
Client ClientClientClientClientClient
FirstRequest Subsequent
Request
Database
• The web server may be the most common server type, but the database server is the workhorse
• It is the foundation for many other important classes of programs
Copyright © 1998-2014 Curt Hill
Historical Perspective• Actions we might want on any file of
fixed records:– Create– Update (add, remove, change records)– Sort– Generate any of several reports
• Each of these would be a program for an overworked programming staff– Typically a COBOL program
Copyright © 1998-2014 Curt Hill
Database
• The key– All these programs are about the
same– All that changes is the underlying file
• The solution– Describe the file in a general way
• This is the metadata – data that describes data
– Generate a program that handles the file based on the description
Copyright © 1998-2014 Curt Hill
How to describe a file• A file is a collection of records• Each record is a collection of fields
– Typically only one type of record in a file
• Each field is described by a:– Name– Type
• For example numeric, string, boolean etc.
– Length• Booleans have a predefined length, others
require specification
Copyright © 1998-2014 Curt Hill
Database Advantages 1
• Data independence– Application program no longer need
some or all of the files– Do not know or care how data is
stored, aka abstraction – Simplifies application development
• Efficient access– The DBMS employs sophisticated
access techniques seldom used by normal programmers
Copyright © 1998-2014 Curt Hill
Database Advantages 2
• Integrity constraints– The DBMS may check data in a way
seldom done in normal file processing– Eg. Account validity
• Security– A DBMS may enforce requirements on
who can access the data and in what way
Copyright © 1998-2014 Curt Hill
Database Advantages 3
• Administration– Minimize redundancy– Manage sharing of the data– Optimize for the enterprise, not a
small group– Easier to backup the data
• Concurrent access– Manages the simultaneous update
problem
Copyright © 1998-2014 Curt Hill
Functions of a Database
• Data storage and retrieval• User-accessible catalog• Transaction support• Concurrency support• Recovery services• Authorization services• Support for data communication• Integrity services• Data independence services
Copyright © 1998-2014 Curt Hill
Data storage and retrieval
• While hiding the file structure, users may: – Add– Delete– Update in place
• Most relational databases support a common language– Structured Query Language
Copyright © 1998-2014 Curt Hill
SQL
• A language where a question is asked of the database and it provides an answer:– Pronounced Sequel or spelled out– Usually state what is wanted rather
than how to get it
• The Data Manipulation Language usually includes both query and update facilities
Copyright © 1998-2014 Curt Hill
Transaction Support
• The unit of update is a transaction– All or nothing– No partial updates are allowed
• The DBMS must guarantee the integrity of a transaction, even in the face of:– Program errors– Incorrect input– System crashes
Copyright © 1998-2014 Curt Hill
Transactions: ACID• Atomic
– Transaction perceived to be indivisible• Consistent
– Transforms database from one consistent state to another
• Isolated– Understandable without regards to any
other agents
• Durable– Once committed permanence is
guaranteed even with system crashes
Copyright © 1998-2014 Curt Hill
Types of database• A large organization often have
two distinct types of databases• A transactional or operational
database• An analytical database or data
warehouse• NoSQL• These have separate uses
Copyright © 1998-2014 Curt Hill
Database uses• The database is about transactions
– Maintaining current knowledge
• The data warehouse is about decision support– Spotting trends
• NoSQL is much harder to classify, any of these– Massive data– Unique organizations– Distributed – Among many others
Copyright © 1998-2014 Curt Hill
Characteristics
• Operational database have:– Strict performance requirements– Predictable workloads– Small units of work– High utilization
• Data warehouses are contrary in every respect
Copyright © 1998-2014 Curt Hill
Warehouses
• A data warehouse– Subject oriented, integrated, non-
volatile, time-variant collection of data in support of management decsions
• Data warehouses support– OLAP (OnLine Analytical Processing)– DSS/EIS (Decision Support Systems or
Executive Information Systems)– Data mining
Copyright © 1998-2014 Curt Hill
What is Data Mining?• Exploratory data analysis based on a
data warehouse– Knowledge Discovery in Databases (KDD)
• Data Mining extracts previously unknown and potentially useful information– Rules, constraints, correlations, patterns,
signatures and irregularities
• The goal is to automate the methods for finding these in the data
Copyright © 1998-2014 Curt Hill
Goals of Data Mining
• Prediction of future behaviors– Seasonal or non-seasonal trends– How will consumers respond to
discounts?– Allows the enterprise to be ready
• Identification of item, event or activity– Intruders may be identified by the
files they access or programs they use
Copyright © 1998-2014 Curt Hill
Goals Again
• Classification of categories of users or products– Shoppers may be categorized as:
• Discount seeking• Rush• Regular• Attached to certain brand names
– The store may be made more friendly to such
• Optimize the use of time, space, materials and money
Copyright © 1998-2014 Curt Hill
Knowledge Discovery
• There are several types of discoverable knowledge:
• Association Rules• Classification hierarchies• Sequential patterns• Time series patterns• Clustering
Copyright © 1998-2014 Curt Hill
Databases• The two types of databases are the
transactional and analytical– The day to day transactional feeds the
analytical
• We do not want knowledge discovery, data mining and the like to occur on the transactional database– Could seriously impede performance
• Either might be used in the next two applications– Transactional is more likely
Copyright © 1998-2014 Curt Hill
Some Definitions• Data
– Small measurable pieces of information– Foundation of knowledge
• Information– Organized data– Putting data into meaningful patterns
• Knowledge – A higher aggregation of information and
data that enables interpretation– Ability to use information– Sets of rules and relationships
Copyright © 1998-2014 Curt Hill
Example• Any numbers or text is data
– Such as: 324 3.83
• It becomes information when we know its context– Such as these are the number of items
in stock and how much we paid per unit
• Knowledge is when we take this information and combine it with processes and rules– Computing our price for selling
something made out of these
Copyright © 1998-2014 Curt Hill
Data Mart
• A data warehouse restricted to a single subject– Such as a group or line of business
within a larger organization
• A dependent data mart obtains its data from the central organizational data warehouse
• An independent data mart obtains data from other sources– No overlap with the data warehouse
Copyright © 1998-2014 Curt Hill
Knowledge Warehouse
• An architectural integration of knowledge management, decision support, artificial intelligence and data warehousing
• Built on top of a data warehouse• Must also encode the rules and
processes of the business– These rules used to only be in the
minds of employees– Next captured on documents
Copyright © 1998-2014 Curt Hill
ERP• Enterprise Resource Planning• The key idea behind database
technology is that the process of manipulating files is basically the same– The only thing that changes is the file
format
• The key to ERP is that many business processes are very similar– Payroll is just payroll
• Took off because of the Euro and Y2K threat
Copyright © 1998-2014 Curt Hill
ERP Areas
• Financial Accounting• Management Accounting• Human Resources• Manufacturing• Supply Chain Management• Project Management• Customer Relationship Management• These were all separate programs at
one time
Copyright © 1998-2014 Curt Hill
Configuration
• Most systems need to be customized by programmers modifying the code
• Most ERPs may be configured to customize to a particular situation
• This configuration is usually much easier than code modification
• The program typically handles best practices, but the configuration can take into account the idiosyncrasies of this business
Copyright © 1998-2014 Curt Hill
Simple Example
• The form of a product number drastically varies between companies– Usually groups of letters and digits
• Configuring the system to store and validate the type used by a particular company needs to be a simple configuration
Copyright © 1998-2014 Curt Hill
CRM• Customer Relationship Management• The notion of a call center• A past, present or future customer calls• A representative, with no previous
knowledge of this customer, has to deal with them
• CRM puts all the information necessary to help in front of the representative, regardless of the purpose of the call
Copyright © 1998-2014 Curt Hill
Types of calls
• Sales– Availability, delivery, costs and
discounts
• Support after the sale– Warranty, replacements, returns
• Information and marketing– Request for information
• Appointments for sales force• Analysis of customer or products
Copyright © 1998-2014 Curt Hill
Implementation
• The leading CRMs, like ERPs are highly configurable
• Also like ERPs they are backed by a transactional database
• Both generate large quantities of SQL on the fly
• ERPs are starting to absorb CRMs
Copyright © 1998-2014 Curt Hill
Finally
• The type or size of businesses that would be architected will likely have all of these
• They will also have serious customizations to these products
• Many other specialized applications also exist– EG. EA tools
Copyright © 1998-2014 Curt Hill