Operating system security (a brief)

IBM Global Business Services

OS Security March-2007 copy 2007 IBM Corporation

OS Security


copy 2007 IBM Corporation2 March-2007OS Security

Objectives

What is OS Security

OS security breakdown

Security in different OS environments



OS security is important

1048708 Fundamental basis of most systems

1048708 Control hardwaresoftware resources

Introduction



Road Map OS security basics

Security For User Accounts

File Systems

Networking

Architecture

Authentication

Unix Authentication

PAM

Windows Authentication

GINA Access Control Impersonation Logging And Auditing API Memory Protection Buffer Overflow SAP On Windows

SAP User Security Best Practices On

SAP-Windows Environments

Best Practices On SAP-UnixLinux Environments



OS security basics

Security is typically achieved based on

1048708 separation and controlled sharing Separation applies to (everything)

1048708 Internal resources typically process memory and

OS data structures 1048708 User resources typically files

1048708 System resources from normal users

Sharing with access control protection

Contd



OS security basics

Separation and controlled sharing require 1048708 Memory protection

1048708 Subjects (users and processes) identification and authentication

1048708 Objects (files and other resources) identification

1048708 Access control for all



Accounts

User identification and authentication 1048708 Based on account identifier and credentials

Accounts hold user rights and privileges 1048708 For access control

Accounts may belong to groups 1048708 Group has associated rights and privileges

1048708 Group-based access control



UNIX accounts

Each user has an account 1048708 On a computer or an NIS(+) domain 1048708 Non-human users are for system processes

Account has name and password 1048708 Authentication based on hashed password 1048708 OS supports password strength aging policies 1048708 Add-on supports for other mechanisms such as Kerberos skey etc

available

A user may belong to many groups 1048708 Has the groupsrsquo rights 1048708 But effectively only 1 group at a time



Windows accounts

Each user has an account 1048708 On a computer andor an Active Directory domain

1048708 Non-human accounts are for system processes

Account typically has name and password 1048708 Authentication based on Kerberos or hashed password (for NT compatibility

only)

1048708 OS supports password strength aging policies

1048708 Certificates and smartcards are also supported (in 2000XP but not commonly used yet)

A user may belong to many groups 1048708 Has the union of the groupsrsquo rights at any time



Networking

Most systems allow users network access

OS tools and services enable these access 1048708 Their own security issues

Required integrated network access are explained later 1048708 Integrated domain authentication

1048708 Network file shares



UNIX networking

Traditionally set of r- commands 1048708 rlogin rsh rcp etc and corresponding servers

1048708 Host address based authentication

1048708 Implicit trust on ports lower than 1024

1048708 Send passwords in clear-text if required

1048708 Very insecure should not be used anymore

The ubiquitous telnet ftp 1048708 Clear-text passwords in basic setup

More secure tools available 1048708 SSH Kerberized telnet ftp

Integrated NFS NIS(+) explained later



Windows networking

Essentially similar tools 1048708 telnet ftp with clear-text passwords

1048708 SSH and augmented versions of telnet ftp more secure

Integrated networking explained later 1048708 Server Message Block (SMB) based

integrated domain authentication file shares access



File systems

File systems security governs 1048708 Access control to files based on subjects 1048708 Security of files sharing 1048708 Files encryption (if any)

Files include 1048708 Data program and 1048708 Other file-based resources eg system caches named

pipes



UNIX file systems

Basically one system with native UNIX format

Access controls using permission bits 1048708 read write execute permissions

1048708 owner group or others

1048708 Eg ndashrwxr-x---

1048708 Coarse-grained

Files sharing using Network File System (NFS) 1048708 Machine access to shares is based on IP address

1048708 User access to shares based on permission bits

1048708 Add-on support for Kerberos auth available

No support for files encryption



Windows file systems

FAT (for backward compatibility) 1048708 FAT supports no access control

NTFS (NT File System) 1048708 Access control based on user IDs and file permissions

1048708 Basic permissions are Read Write Execute Delete Change Permissions Take Ownership

1048708 Standard permissions are basic ones combined

1048708 Different permissions to a file can be granted to individual usersgroups using ACL

1048708 More fine-grained flexible than UNIX

Contd




Files sharing using Common Internet File System (CIFS) 1048708 Shares are managed in directory (in common with domain management

ndash more later)

1048708 Machine access to shares is based on computer account in domain and inter-domain trust

1048708 User access to shares is based on share passwords or standard ACLs

1048708 NT systems use hashed password SMB auth

1048708 Windows 2000XP use Kerberos authentication

Encrypting File System (EFS) 1048708 Files encryption using random secret keys which are in turn encrypted

with EFS public keys



UNIX security Architecture

Basic UNIX based on monolithic kernel

Fundamental OS security based on 1048708 User id and password

1048708 Group id

1048708 Process id

1048708 File permission bits

1048708 Process memory protection



Windows security Architecture

Windows (NT2000XP) have layered components on top of a kernel

Security Reference Monitor (SRM) 1048708 Part of the kernel 1048708 Handles core of access control checks

Protected security services include 1048708 Win logon process 1048708 Local Security Authority (LSA) and policy database 1048708 Security Account Manager (SAM) and database 1048708 These services perform user authentication and non-core part of

access control

Contd




Security identifiers (SID) 1048708 Represent uniquely each user or group

Access control entry (ACE) 1048708 Contains permissions to an object explicitly denied or granted to a

subject (SID)

Access control list (ACL) 1048708 List of ACErsquos for an object

Security descriptor of an object 1048708 Contains is owner SID primary group SID its ACL the applicable

system ACL

Access token for a logged on user 1048708 Contains the userrsquos SID primary group SID etc



UNIX security Authentication

Username and clear-text password 1048708 For single computer or NIS(+) domain

1048708 System stores (modified DES) hashed passwords 1048708 etcpasswd readable by everyone or

1048708 etcshadow readable only by root or

1048708 NIS(+) database

1048708 Passwords are hashed before matching

1048708 Logged on users are identified by numeric IDs

1048708 Passwords are open to dictionary attacks

Integration of Kerberos and others methods 1048708 Pluggable Auth Module (PAM) for Solaris Linux

1048708 Security Integration Architecture (SIA) for HPUX



Pluggable Authentication Module (PAM)

Login Telnet Ftp

PAM API

PAM Framework

PAM

ConfigurationPAM SPI

UNIX Kerberos Smart Cards



Windows security Authentication

NT uses NTLM authentication 1048708 NT (MD4) and LM (DES-based) hashed password

1048708 Domains integration relies on sending hashed passwords through insecure SMB protocols

1048708 Inter-domain trusts are one-way non-transitive

Windows 2000XP in domains use Kerberos 1048708 NTLM supported for backward compatibility

1048708 Domains are managed by Active Directory

1048708 Integrated Kerberos auth as domain controllers are KDCs

1048708 Enable hierarchical organization and delegation

1048708 Inter-domain trusts are two-way transitive thereby simplifying trust management

Logged on users run processes with their access tokens basis for access control impersonation



Graphical Identification And Authentication(GINA)

Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA



UNIX security Access control

Only discretionary access control (DAC) 1048708 Based on file permissions and UID GID PID

1048708 File has permission bits UID (owner) GID

1048708 File permission bits are r w e and s (later)

1048708 A process has real and effective UID and GID

1048708 Kernel matches these IDs to control a processrsquos access to a file

1048708 Super-user (root) has all access to everything

1048708 Some variants such as Solaris 25 or newer have

ACL systems for more fine-grained controls

Some experimental systems (eg SE Linux) have Mandatory Access Control (MAC)



Windows security Access control

Discretionary access control 1048708 Based on subject SIDs and object ACLs 1048708 Each object has an ACL

1048708 Null ACL or empty means no restrictions or no access

1048708 Each process has an access token with its owner SID group SIDs 1048708 Access control checks are matching of access tokens against ACLs 1048708 Administrators group can access everything 1048708 SRM performs core matching

Less so discretionary access control 1048708 Some system-wide policies applying to subjects regardless of individual

objectrsquos ACL



UNIX security Logging and auditing

Flexible and comprehensive ldquosyslogrdquo 1048708 Logging daemon can store locally or on remote server

1048708 System processes store relevant information through logging APIs

1048708 System administrators can configure what to log and where to store logs

1048708 However auditing tools are not natively available in the basic OS



Windows security Logging amp auditing

The LSA and SRM create logs through the system event logger

The LSA logs mostly logon events based on its audit policy

The SRM logs access check events based on the system access control list (SACL)

1048708 Each object has an SACL

Logs are stored locally



UNIX security Impersonation

Static privileges are often too restricted

Impersonation allows dynamic changes in a user or processrsquos security privileges

Programs run with its owner or group ID instead of user who runs them if

1048708 Set-UID (suid) bit set or

1048708 Set-GID (sgid) bit set

Flaws in these programs can be extremely dangerous

User can impersonate other users by 1048708 Running ldquosurdquo to have an impersonated shell

1048708 Running ldquosudordquo to impersonate for a command



Windows security Impersonation

No equivalence of UNIX suid sgid or ldquosurdquo ldquosudordquo programs

But processes frequently programmatically impersonate others 1048708 A thread takes on access token of another subject

1048708 This access token may be exact copy or variant of a primary access token

1048708 Thread gets security privileges of the impersonated subject

Impersonation is application-controlled as opposed to administrator-controlled in UNIX



OS security buffer overflow

Example code

int auth_user()

char name[32]

printf(ldquoEnter username ldquo)

gets(name)

do authentication

User enters more than 32 characters

Variable name gets the first 32 characters

The rest goes on the program stack

May override program pointer

Program then jumps to unexpected code



OS security memory protection

Standard process memory protection 1048708 Process memory is accessed through page table

1048708 No process can normally access anotherrsquos memory

1048708 Historically for safety but critical for security

Buffer overflow 1048708 Arguments and program pointer on the stack

1048708 Writing beyond the buffer for an argument may overwrite the program pointer

1048708 Careful selection of argument data may get program to execute malicious code

1048708 Compilers andor operating system can help prevent this from happening



UNIX security APIs

Basic OS supports few security APIs 1048708 Essentially user password and process management

APIs

Modern variants support more 1048708 Eg PAM APIs

Add-on services are relatively common 1048708 Kerberos APIs GSSAPI OpenSSL



Windows security APIs

Windows support 1048708 Essential user password process management APIs

1048708 Graphical Identification and Authentication (GINA) APIs fairly similar to PAM SIA

1048708 Security Services Providers Interface (SSPI) similar to GSSAPI

1048708 CryptoAPI supports encryption smartcards



SAP And Windows Security



Protecting the Operating System Users Used in an SAP System

User type User Function and Rights Security Measures

Windows users Administrator The local superuser who has unlimited access to all local resources

Change the user name and hide its password Create other users for administrative tasks and limit their rights to those tasks for which they are used

Guest A local guest account who has guest access to all local resources


SAP system users ltsapsidgtadm The SAP system administrator who has unlimited access to all local resources related to SAP systems

bull Change its password regularly bull Restrict its access rights to instance-specific

resources for the SAP system only

SAPServiceltSAPSIDgt

A special user who runs the Windows services related to SAP systems

Cancel the userrsquos right to Log on locally bull Restrict its access rights to instance-specific

and database-specific resources only



1 Data Relevant to the SAP System

2 Database Files

3 Protection for Dynamically-Created Files

4 Protecting Shared Memory

5 Defining Start and Stop Permissions

6 Secure Using Windows Trusted Domains

An Windows Environment For SAP Security Should Encompass

Security Of



An UNIXLinux Environment For SAP Security Should Encompass

Protecting Specific Properties Files and Services

SUIDSGID programs

Password file (passwd)

BSD services rlogin and remshrsh

Services such as Network Information System (NIS) or Network File System (NFS)

Protected SAP System Directory Structures Under UNIXLINUX

Security Of



Objectives

What is OS Security

OS security breakdown

Security in different OS environments






Introduction





File Systems

Networking

Architecture

Authentication

Unix Authentication

PAM








OS security basics







Contd



OS security basics







Accounts







UNIX accounts



available




Windows accounts




only)






Networking







UNIX networking











Windows networking







File systems



pipes



UNIX file systems



















Contd





ndash more later)












1048708 Group id

1048708 Process id









access control

Contd






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of






Introduction





File Systems

Networking

Architecture

Authentication

Unix Authentication

PAM








OS security basics







Contd



OS security basics







Accounts







UNIX accounts



available




Windows accounts




only)






Networking







UNIX networking











Windows networking







File systems



pipes



UNIX file systems



















Contd





ndash more later)












1048708 Group id

1048708 Process id









access control

Contd






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of





File Systems

Networking

Architecture

Authentication

Unix Authentication

PAM








OS security basics







Contd



OS security basics







Accounts







UNIX accounts



available




Windows accounts




only)






Networking







UNIX networking











Windows networking







File systems



pipes



UNIX file systems



















Contd





ndash more later)












1048708 Group id

1048708 Process id









access control

Contd






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of



OS security basics







Contd



OS security basics







Accounts







UNIX accounts



available




Windows accounts




only)






Networking







UNIX networking











Windows networking







File systems



pipes



UNIX file systems



















Contd





ndash more later)












1048708 Group id

1048708 Process id









access control

Contd






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of



OS security basics







Accounts







UNIX accounts



available




Windows accounts




only)






Networking







UNIX networking











Windows networking







File systems



pipes



UNIX file systems



















Contd





ndash more later)












1048708 Group id

1048708 Process id









access control

Contd






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of



Accounts







UNIX accounts



available




Windows accounts




only)






Networking







UNIX networking











Windows networking







File systems



pipes



UNIX file systems



















Contd





ndash more later)












1048708 Group id

1048708 Process id









access control

Contd






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of



UNIX accounts



available




Windows accounts




only)






Networking







UNIX networking











Windows networking







File systems



pipes



UNIX file systems



















Contd





ndash more later)












1048708 Group id

1048708 Process id









access control

Contd






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of



Windows accounts




only)






Networking







UNIX networking











Windows networking







File systems



pipes



UNIX file systems



















Contd





ndash more later)












1048708 Group id

1048708 Process id









access control

Contd






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of



Networking







UNIX networking











Windows networking







File systems



pipes



UNIX file systems



















Contd





ndash more later)












1048708 Group id

1048708 Process id









access control

Contd






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of



UNIX networking











Windows networking







File systems



pipes



UNIX file systems



















Contd





ndash more later)












1048708 Group id

1048708 Process id









access control

Contd






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of



Windows networking







File systems



pipes



UNIX file systems



















Contd





ndash more later)












1048708 Group id

1048708 Process id









access control

Contd






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of



File systems



pipes



UNIX file systems



















Contd





ndash more later)












1048708 Group id

1048708 Process id









access control

Contd






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of



UNIX file systems



















Contd





ndash more later)












1048708 Group id

1048708 Process id









access control

Contd






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of










Contd





ndash more later)












1048708 Group id

1048708 Process id









access control

Contd






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of





ndash more later)












1048708 Group id

1048708 Process id









access control

Contd






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of






1048708 Group id

1048708 Process id









access control

Contd






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of







access control

Contd






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of






subject (SID)



system ACL

















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of
















Login Telnet Ftp

PAM API

PAM Framework

PAM


















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of
















Win Logon

GINA

LSA

Shell

Registry

Win Logon Shell

My GINA Registry

GINA LSA

LSA




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of




















objectrsquos ACL






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of






































Example code

int auth_user()

char name[32]


gets(name)

do authentication


















UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of













UNIX security APIs


APIs































2 Database Files






Security Of





SUIDSGID programs





Security Of





























2 Database Files






Security Of





SUIDSGID programs





Security Of





SUIDSGID programs





Security Of

Operating system security (a brief)

Technology

Transcript of Operating system security (a brief)