Practical Applications in Security Lessons learned from recent security incidents in the Computer...

Practical Applications in SecurityLessons learned from recent security incidents

in the Computer Science Department

Chris SchenkMark Dehus

Michael Oberg

CSCI 6268 December 6, 2005

December 06, 2005 2

Outline

History and Background of CS Attacks (Chris) Fall 2004 Compromise

Fall 2005 Compromise

Details on the Recent Compromise (Mark) Detection

Recovery

Mitigation

Computational Science Center Perspective (Michael) Systems Inspection

Security Tools Used By The CSC

December 06, 2005 3

You know it’s a bad day when…

December 06, 2005 4

It is possible…

Pre-attack on the CSEL, Fall 2004 Request for user login name change Password file distribution from central source

Rdist + hosts.allow

*poof*

Password file corruption 600 accounts lost Empty root password Restore from single nightly backup copy (luckily!)

December 06, 2005 5

That someone may be…

Reload of machine ‘duos’ to support ldap (with help from Dirk Grunwald)

All machines on external network accessible via ssh toucan, cardinal, woodpecker, etc

ssh woodpecker: Remote host key has changed! Not initially thought of as a threat (woops) Host ID in base64 form, not in hex (strange)

Two days later, logged into ‘duos’ ‘who’ command (some.company.ro) The Romanians are upon us

December 06, 2005 6

Doing something nasty!!

‘freeshell.org’ email account shutdown Contact ITS to stop mail forwarding Attempt to email sysadmin at freeshell.org His response:

Chris -

Thanks for writing. Your colorado.edu account may been compromised as well. Please change your password. There is currently an investigation being done on you by the FBI and you may be contacted by a special agent Matthew Rowald concerning some efnet IRC activity. I'm not even sure if you will get this, but I certainly wasn't going to email it to your SDF account. I only saw connections from comcast and colorado.edu, which leeds me to believe that the individual (or in the FBI's case, you) are using your colorado.edu acct.

Can you please change your passwords? Also, you might want to look into your ssh versions used at colorado.edu.

December 06, 2005 7

Response

Almost all machines compromised (~90%)

Entire network bogged down with flooding traffic from compromised machines ‘ls’ command taking 10 seconds to complete Single 100Mbit copper line to connect all CSEL machines to

the world Very unhappy databases class (some were glad homework

couldn’t get done, others were not)

Contacts from a few places around the world about attempted dictionary attacks from CSEL machines.

December 06, 2005 8

Remediation

Bad Mojo No logs saved Unsure of how many accounts were logged (anyone using

ssh that week) All account passwords expired to force password changes

(didn’t completely work)

Good Mojo Reload of machines with Fedora Core 2 Replacing RedHat 9 (1 ½ years after support dropped for OS) All machines moved to internal subnet No machines externally accessible except server csel.cs

No further incidences seen after reload and NATing of machines

December 06, 2005 9

John the Ripper

Password cracking utility

Ran on password file after remediation

22 account passwords discovered after 2 hours 6 found immediately (one dead guy)

2800 accounts leftover in the lab Only 300 active users for the year

December 06, 2005 10

Fall 2005 Compromise

Dirk Grunwald’s machines compromised

Local user with full sudo access logged in from Amsterdam

Sshd immediately replaced on all machines

Took one full day to figure out something was wrong


CSEL Second Compromise Overview

CSEL Background Attack Timeline Detection

Lockups and Errors Log Auditing

Recovery Data migration Server Recovery Workstation Recovery

Mitigation Locking Down OS Locking Down Dameons Automated Break-in detection

csel.cs.colorado.edu

Primary public server

Used for public ssh shell access

Hosts the CSEL website & student websites

Has e-mail capabilities

duos.cs.colorado.edu

Duos is the LDAP & File Server

This means it contains authentication data and home directories

It also manages 'Self-Account Creation' from PLUS

It has access to Uniquid (Unique User ID system)

lesc.cs.colorado.edu

Lesc is used as a backup mirror of duos

At the time Lesc was being used to phase out duos

Important information was on it.. such asRecent LDAP Directory Dump (Contained password

hashes)Workstation Image (The one I used that same day)Recent Backup of Home directories

Csel.cs.colorado.edu Locked up!

.. We were oblivious ..

CSEL Attack Timeline (What we saw)

Friday, August 19th csel.cs.colorado.edu locked up at 10:30am Computer did not want to reboot (Hanging on Shorewall) Shorewall was disabled and computer was booted with basic iptables Shrugged off as bad luck

Sunday, August 21st Chris is concerned and checks sshd on servers by executing..

$ ps aux | grep sshdroot 2092 0.0 0.0 4668 264 ? Ss Aug21 0:00 /usr/sbin/sshd

Everything looks OK! Monday, August 22nd

Check of log files shows a compromised account logged in on Friday!! Tuesday, August 23rd

We determined that all the attacker was able to do was crash csel.cs

Lesc.cs.colorado.edu

Finding out what happened (Log Auditing)

First sign.. csel.cs.colorado.edu was locked up for no reason Question: Who & How - > /var/log/messages & /var/log/secure Two ways to find answers

Find by Hand Automated Log Analysis

This question was answered by a mix of both.. but mostly by hand (I love Grep!)

Dec 5 15:43:30 csel OTRS-PM-10[12240]: [Notice][Kernel::System::PostMaster::FollowUp::Run] FollowUp Article to Ticket [2005120510000011] created (TicketID=128, ArticleID=410).

Dec 5 15:50:03 csel crond(pam_unix)[5839]: session closed for user otrsDec 5 15:50:05 csel crond(pam_unix)[5837]: session closed for user rootDec 5 15:50:41 csel unix_chkpwd[5906]: password check failed for user (buschk)Dec 5 15:50:41 csel sshd(pam_unix)[5904]: authentication failure; logname= uid=0 euid=0 tty=ssh

ruser= rhost=camras.colorado.edu user=buschkDec 5 15:50:41 csel sshd[5904]: pam_ldap: error trying to bind as user

"uid=buschk,ou=People,dc=csel,dc=cs,dc=colorado,dc=edu" (Invalid credentials)Dec 5 15:50:50 csel unix_chkpwd[5909]: password check failed for user (trumpler)Dec 5 15:50:50 csel sshd(pam_unix)[5907]: authentication failure; logname= uid=0 euid=0 tty=ssh

ruser= rhost=camras.colorado.edu user=trumplerDec 5 15:50:50 csel sshd(pam_unix)[5910]: session opened for user trumpler by (uid=0)Dec 5 15:50:51 csel automount[5937]: mount(nfs): nfs: mount failure duos:/home/nsa on /home/nsa

CSEL Second Compromise (What the logs told us)

Friday, August 19th Attempted logins using previously collected usernames started

at 5AM Csel.cs & Lesc.cs were attacked simultaneously Attacker was io.physics.tamu.edu (obviously compromised)

Payback.. Buffs beat Texas A&M 40 – 21! That'll teach um!

35 Usernames were attempted, 4 were successful When io discovered successful login it alerted attacker Attacker then used login information from 81.181.251.2 81.181.251.2 Belongs to RIPE.net (Large ISP) in Amsterdam

Lesc Damage

Attackers got into Lesc using root password

SSHD was replaced with chroot jailed trojan

Contents of /root rhel4-workstation-image.tgz ldap-data-backup.ldif

All attacker did was replace SSHD!

Duos.cs.colorado.edu

Hackers did not bother with Duos

We got lucky.. Would have been very easy to gain root access

PHP was setuid root due to Uniquid requirements

Who cares? Duos has access to Mailing Addresses!

$ cat 0wnDuos.php#!/usr/local/sbin/php<?phpexec(“passwd root”);?>$ ./0wnDuos.phpChanging password for user rootNew password: i0wnU$ suPassword: i0wnu# rm / -rf ( or worse )

Recovery

Copied important data to another computer

Reloaded all three servers

Reloaded all workstations

Easy but time consuming (30 hours)

Mitigation (General)

Don't use weak passwords and change them often!

SSH Restrictions and Rate Limiting on incoming connections

Use nosuid, nodev, and noexec to restrict files

Monitor log files frequently with automatic analyzer Logcheck Logwatch

Considering using RSA/DSA keys instead of passwords

Mitigation (duos.cs.colorado.edu)

Self-Account Creation scripts still need setuid So-So Solution.. C setuid wrapper designed to call uniquid

scripts only Better solution.. rewrite all scripts in perl Prefect solution.. fix uniquid so it does not need setuid

Passwords are no longer allowed.

Fine grained sudo controls and su restrictions

SSH restrictions

Blocked Internet Traffic & Remote Logins

Installed Intrusion Detection Software

Mitigation (csel.cs.colorado.edu)

Passwords are allowed but restrictions are put on pam_cracklib Still can’t restrict passwords from SAC.. Yet

Fine grained sudo controls and su restrictions

SSH Restrictions

Rate Limiting on incoming connections


Put restrictions on file system


Mitigation (lesc.cs.colorado.edu)

Removed from Internet completely

Applied same restrictions as Duos



Outline

Computational Science Center Perspective on the

Compromise Systems Inspection immediately following notification of the

compromise

Security Tools Used By The CSC


Computational Science Center

100 systems under Dr. Henry Tufo

http://csc.cs.colorado.edu/

Three clusters, plus several additional machines: 65 node Intel based Computational Cluster (hemisphere) 28 node PowerPC based Computational Cluster (occam) 3 node Storage Cluster 1 x86_64 Big RAM (8GB) machine Management Server License Server Log / Security Server

8 externally accessible, remainder are on a private LAN

http://csc.cs.colorado.edu/


CSC Systems Inspection

This work done by Sean McCreary, Matthew Woitaszek and myself

Received notification of compromise of unknown number of systems in the CS department

Manually audited logs to determine if anyone successfully logged in to our servers, found this:

Aug 19 06:04:26 hemisphere sshd[4212]: Accepted password for grunwald from 165.91.181.6 port 39256 ssh2





Hacker Successfully logged into our system, now what? Treat all machines as if they are hacked, full inspection If any machines are found to be compromised, remove from

network and do full recovery

Management server Only Administrator accounts Persistent connections to other main servers under ‘GNU

screen’, a terminal multiplexer All Administrators use password protected RSA keys on

known client machines to connect to CSC machines Trojan SSHD can’t capture administrator passwords

Administrators use ‘sudo’ exclusively



Attempted logins from all 32 compromised accounts Only user ‘grunwald’ was successful First, make sure ‘grunwald’ is not logged in and lock

the account: `sudo mkpasswd grunwald` Change shell to /sbin/nologin

Examine the following looking for any suspicious files: /home/grunwald /tmp /var/tmp

Examine the process table: `ps auxw`



Manually Run Tripwire, Osiris, Samhain These are “Filesystem Integrity Scanners” Example Tripwire Output (Osiris is similar):

Host name: hemisphereHost IP address: 128.138.207.210Policy file used: /etc/tripwire/tw.polConfiguration file used: /etc/tripwire/tw.cfgDatabase file used: /var/lib/tripwire/hemisphere.twdCommand line used: /usr/sbin/tripwire --check …-------------------------------------------------------------------------------Rule Name: Critical configuration filesSeverity Level: 100-------------------------------------------------------------------------------Modified:"/etc/crontab""/etc/fstab""/etc/motd"…



These Filesystem Integrity Scanners verify that the system binaries and configuration files are not compromised Attacker would have to modify the Filesystem Integrity

Scanner database to prevent notification Could also disable the cron entry

If you monitor systems, be sure to watch for missing emails

Tripwire and Osiris use cron entries to run daily checks, which send out a single summary email

Samhain has a client-server architecture, can send out alerts within minutes of finding an issue We highly recommend Samhain for new installations



"Hemisphere was the only system to survive the attacks”

Why? Luck We do what we can to limit vulnerabilities but focus most of

our attention on early detection and the ability to recover

Have taken the time to focus on security Is not easy, or quick So what have we done?


CSC Security Tools and Policies

In addition to the Filesystem/Configuration Integrity

Scanning and Reporting:

System Hardening

Foreign login detection and reporting (custom scripts)

“Rootkit” scanning and reporting (custom wrapper scripts)

We have and follow Incident Response Procedures



Centralized Log Server

We were in the process of building this server when the CS

compromise happened

All syslog, Samhain, chkrootkit, rkhunter logs to this server

Extremely secure configuration:

Only three Administrators have accounts, use different

passwords from any other system

Root cannot login

EAL4+ Common Criteria Installation

IBM RPM/script which hardens SuSE Enterprise Linux 9



System Hardening Common Service Audit

Remove extraneous installed packages Only core, necessary services running

Configuration of required software PAM: ssh, sudo, su

Host based firewalling Shorewall

User Policies We run John-the-Ripper Password cracking utility Users are created with account aging



chkrootkit - a tool to scan for rootkits We have a wrapper to consolidate the output from

the many servers into a single email: Lack of scalability in security tools is an issue!

chkrootkit-0.45 found the following warnings, vulnerabilities, and infections on hosts:node0[01-64],hemispheretransputeroccam,blade0[01-27]loaner,store01,store02,store03

Errors occured on the follwing nodes that prevented chkrootkit from completing:NONE

The following hosts were not scanned:node008blade003blade016

All logs generated for this run are located at: /home/admin/logs/chkrootkit/20051206/Quick Stats:

warnings = 0vulnerabilities = 0infections = 0



Foreign Login Detection custom script

Detects login activity from new machines Short summary describes new logins

Administrator follow-up Look for accesses from outside of known activity System administrators contact Advisor/PI to verify student’s

location (this gets their attention)

We have caught several compromised passwords this way

(Source: CSC Group Slide by Matthew Woitaszek)



User training: Don’t type your Hemisphere password in on a remote machine, it might have a trojan SSH! Connect directly from your client!

--------------------------------------Host report times

hemisphere 2005-12-01 00:02:15

--------------------------------------First logins for each user from a host

oberg 2005-11-30 14:05:50 hemisphere ec240-175-dhcp.colorado.edu 2005-11-30 15:47:43 hemisphere ec240-250-dhcp.colorado.edu

sliu 2005-11-30 23:44:33 hemisphere ae203-053.resnet.stonybrook.edu 2005-11-30 23:46:22 hemisphere ae203-053.resnet.stonybrook.edu

Why is sliu, usually at a desk in ECCS122, connecting from a dorm in Long Island at 1:45AM Eastern Time?

(Source: CSC Group Slide by Matthew Woitaszek)

Foreign Login Detection Example



Incident Response Procedure

Secure email: Commercial PGP, GPG for the rest

Cell phones Upon any notification of compromise,

immediately disconnect all machines from the internet, and conduct onsite analysis

Start to download offsite backups for restore

Questions?

Practical Applications in Security Lessons learned from recent security incidents in the Computer...

Documents

Transcript of Practical Applications in Security Lessons learned from recent security incidents in the Computer...