Post on 20-Dec-2015
Practical Applications in SecurityLessons learned from recent security incidents
in the Computer Science Department
Chris SchenkMark Dehus
Michael Oberg
CSCI 6268 December 6, 2005
December 06, 2005 2
Outline
History and Background of CS Attacks (Chris) Fall 2004 Compromise
Fall 2005 Compromise
Details on the Recent Compromise (Mark) Detection
Recovery
Mitigation
Computational Science Center Perspective (Michael) Systems Inspection
Security Tools Used By The CSC
December 06, 2005 4
It is possible…
Pre-attack on the CSEL, Fall 2004 Request for user login name change Password file distribution from central source
Rdist + hosts.allow
*poof*
Password file corruption 600 accounts lost Empty root password Restore from single nightly backup copy (luckily!)
December 06, 2005 5
That someone may be…
Reload of machine ‘duos’ to support ldap (with help from Dirk Grunwald)
All machines on external network accessible via ssh toucan, cardinal, woodpecker, etc
ssh woodpecker: Remote host key has changed! Not initially thought of as a threat (woops) Host ID in base64 form, not in hex (strange)
Two days later, logged into ‘duos’ ‘who’ command (some.company.ro) The Romanians are upon us
December 06, 2005 6
Doing something nasty!!
‘freeshell.org’ email account shutdown Contact ITS to stop mail forwarding Attempt to email sysadmin at freeshell.org His response:
Chris -
Thanks for writing. Your colorado.edu account may been compromised as well. Please change your password. There is currently an investigation being done on you by the FBI and you may be contacted by a special agent Matthew Rowald concerning some efnet IRC activity. I'm not even sure if you will get this, but I certainly wasn't going to email it to your SDF account. I only saw connections from comcast and colorado.edu, which leeds me to believe that the individual (or in the FBI's case, you) are using your colorado.edu acct.
Can you please change your passwords? Also, you might want to look into your ssh versions used at colorado.edu.
December 06, 2005 7
Response
Almost all machines compromised (~90%)
Entire network bogged down with flooding traffic from compromised machines ‘ls’ command taking 10 seconds to complete Single 100Mbit copper line to connect all CSEL machines to
the world Very unhappy databases class (some were glad homework
couldn’t get done, others were not)
Contacts from a few places around the world about attempted dictionary attacks from CSEL machines.
December 06, 2005 8
Remediation
Bad Mojo No logs saved Unsure of how many accounts were logged (anyone using
ssh that week) All account passwords expired to force password changes
(didn’t completely work)
Good Mojo Reload of machines with Fedora Core 2 Replacing RedHat 9 (1 ½ years after support dropped for OS) All machines moved to internal subnet No machines externally accessible except server csel.cs
No further incidences seen after reload and NATing of machines
December 06, 2005 9
John the Ripper
Password cracking utility
Ran on password file after remediation
22 account passwords discovered after 2 hours 6 found immediately (one dead guy)
2800 accounts leftover in the lab Only 300 active users for the year
December 06, 2005 10
Fall 2005 Compromise
Dirk Grunwald’s machines compromised
Local user with full sudo access logged in from Amsterdam
Sshd immediately replaced on all machines
Took one full day to figure out something was wrong
December 06, 2005 11
CSEL Second Compromise Overview
CSEL Background Attack Timeline Detection
Lockups and Errors Log Auditing
Recovery Data migration Server Recovery Workstation Recovery
Mitigation Locking Down OS Locking Down Dameons Automated Break-in detection
csel.cs.colorado.edu
Primary public server
Used for public ssh shell access
Hosts the CSEL website & student websites
Has e-mail capabilities
duos.cs.colorado.edu
Duos is the LDAP & File Server
This means it contains authentication data and home directories
It also manages 'Self-Account Creation' from PLUS
It has access to Uniquid (Unique User ID system)
lesc.cs.colorado.edu
Lesc is used as a backup mirror of duos
At the time Lesc was being used to phase out duos
Important information was on it.. such asRecent LDAP Directory Dump (Contained password
hashes)Workstation Image (The one I used that same day)Recent Backup of Home directories
CSEL Attack Timeline (What we saw)
Friday, August 19th csel.cs.colorado.edu locked up at 10:30am Computer did not want to reboot (Hanging on Shorewall) Shorewall was disabled and computer was booted with basic iptables Shrugged off as bad luck
Sunday, August 21st Chris is concerned and checks sshd on servers by executing..
$ ps aux | grep sshdroot 2092 0.0 0.0 4668 264 ? Ss Aug21 0:00 /usr/sbin/sshd
Everything looks OK! Monday, August 22nd
Check of log files shows a compromised account logged in on Friday!! Tuesday, August 23rd
We determined that all the attacker was able to do was crash csel.cs
Finding out what happened (Log Auditing)
First sign.. csel.cs.colorado.edu was locked up for no reason Question: Who & How - > /var/log/messages & /var/log/secure Two ways to find answers
Find by Hand Automated Log Analysis
This question was answered by a mix of both.. but mostly by hand (I love Grep!)
Dec 5 15:43:30 csel OTRS-PM-10[12240]: [Notice][Kernel::System::PostMaster::FollowUp::Run] FollowUp Article to Ticket [2005120510000011] created (TicketID=128, ArticleID=410).
Dec 5 15:50:03 csel crond(pam_unix)[5839]: session closed for user otrsDec 5 15:50:05 csel crond(pam_unix)[5837]: session closed for user rootDec 5 15:50:41 csel unix_chkpwd[5906]: password check failed for user (buschk)Dec 5 15:50:41 csel sshd(pam_unix)[5904]: authentication failure; logname= uid=0 euid=0 tty=ssh
ruser= rhost=camras.colorado.edu user=buschkDec 5 15:50:41 csel sshd[5904]: pam_ldap: error trying to bind as user
"uid=buschk,ou=People,dc=csel,dc=cs,dc=colorado,dc=edu" (Invalid credentials)Dec 5 15:50:50 csel unix_chkpwd[5909]: password check failed for user (trumpler)Dec 5 15:50:50 csel sshd(pam_unix)[5907]: authentication failure; logname= uid=0 euid=0 tty=ssh
ruser= rhost=camras.colorado.edu user=trumplerDec 5 15:50:50 csel sshd(pam_unix)[5910]: session opened for user trumpler by (uid=0)Dec 5 15:50:51 csel automount[5937]: mount(nfs): nfs: mount failure duos:/home/nsa on /home/nsa
CSEL Second Compromise (What the logs told us)
Friday, August 19th Attempted logins using previously collected usernames started
at 5AM Csel.cs & Lesc.cs were attacked simultaneously Attacker was io.physics.tamu.edu (obviously compromised)
Payback.. Buffs beat Texas A&M 40 – 21! That'll teach um!
35 Usernames were attempted, 4 were successful When io discovered successful login it alerted attacker Attacker then used login information from 81.181.251.2 81.181.251.2 Belongs to RIPE.net (Large ISP) in Amsterdam
Lesc Damage
Attackers got into Lesc using root password
SSHD was replaced with chroot jailed trojan
Contents of /root rhel4-workstation-image.tgz ldap-data-backup.ldif
All attacker did was replace SSHD!
Duos.cs.colorado.edu
Hackers did not bother with Duos
We got lucky.. Would have been very easy to gain root access
PHP was setuid root due to Uniquid requirements
Who cares? Duos has access to Mailing Addresses!
$ cat 0wnDuos.php#!/usr/local/sbin/php<?phpexec(“passwd root”);?>$ ./0wnDuos.phpChanging password for user rootNew password: i0wnU$ suPassword: i0wnu# rm / -rf ( or worse )
Recovery
Copied important data to another computer
Reloaded all three servers
Reloaded all workstations
Easy but time consuming (30 hours)
Mitigation (General)
Don't use weak passwords and change them often!
SSH Restrictions and Rate Limiting on incoming connections
Use nosuid, nodev, and noexec to restrict files
Monitor log files frequently with automatic analyzer Logcheck Logwatch
Considering using RSA/DSA keys instead of passwords
Mitigation (duos.cs.colorado.edu)
Self-Account Creation scripts still need setuid So-So Solution.. C setuid wrapper designed to call uniquid
scripts only Better solution.. rewrite all scripts in perl Prefect solution.. fix uniquid so it does not need setuid
Passwords are no longer allowed.
Fine grained sudo controls and su restrictions
SSH restrictions
Blocked Internet Traffic & Remote Logins
Installed Intrusion Detection Software
Mitigation (csel.cs.colorado.edu)
Passwords are allowed but restrictions are put on pam_cracklib Still can’t restrict passwords from SAC.. Yet
Fine grained sudo controls and su restrictions
SSH Restrictions
Rate Limiting on incoming connections
Installed Intrusion Detection Software
Put restrictions on file system
December 06, 2005 26
Mitigation (lesc.cs.colorado.edu)
Removed from Internet completely
Applied same restrictions as Duos
Installed Intrusion Detection Software
December 06, 2005 27
Outline
Computational Science Center Perspective on the
Compromise Systems Inspection immediately following notification of the
compromise
Security Tools Used By The CSC
December 06, 2005 28
Computational Science Center
100 systems under Dr. Henry Tufo
http://csc.cs.colorado.edu/
Three clusters, plus several additional machines: 65 node Intel based Computational Cluster (hemisphere) 28 node PowerPC based Computational Cluster (occam) 3 node Storage Cluster 1 x86_64 Big RAM (8GB) machine Management Server License Server Log / Security Server
8 externally accessible, remainder are on a private LAN
December 06, 2005 29
CSC Systems Inspection
This work done by Sean McCreary, Matthew Woitaszek and myself
Received notification of compromise of unknown number of systems in the CS department
Manually audited logs to determine if anyone successfully logged in to our servers, found this:
Aug 19 06:04:26 hemisphere sshd[4212]: Accepted password for grunwald from 165.91.181.6 port 39256 ssh2
Aug 19 06:11:32 hemisphere sshd[6929]: Accepted password for grunwald from 81.181.251.2 port 2412 ssh2
Aug 19 06:36:27 hemisphere sshd[7861]: Accepted password for grunwald from 81.181.251.2 port 2441 ssh2
December 06, 2005 30
CSC Systems Inspection
Hacker Successfully logged into our system, now what? Treat all machines as if they are hacked, full inspection If any machines are found to be compromised, remove from
network and do full recovery
Management server Only Administrator accounts Persistent connections to other main servers under ‘GNU
screen’, a terminal multiplexer All Administrators use password protected RSA keys on
known client machines to connect to CSC machines Trojan SSHD can’t capture administrator passwords
Administrators use ‘sudo’ exclusively
December 06, 2005 31
CSC Systems Inspection
Attempted logins from all 32 compromised accounts Only user ‘grunwald’ was successful First, make sure ‘grunwald’ is not logged in and lock
the account: `sudo mkpasswd grunwald` Change shell to /sbin/nologin
Examine the following looking for any suspicious files: /home/grunwald /tmp /var/tmp
Examine the process table: `ps auxw`
December 06, 2005 32
CSC Systems Inspection
Manually Run Tripwire, Osiris, Samhain These are “Filesystem Integrity Scanners” Example Tripwire Output (Osiris is similar):
Host name: hemisphereHost IP address: 128.138.207.210Policy file used: /etc/tripwire/tw.polConfiguration file used: /etc/tripwire/tw.cfgDatabase file used: /var/lib/tripwire/hemisphere.twdCommand line used: /usr/sbin/tripwire --check …-------------------------------------------------------------------------------Rule Name: Critical configuration filesSeverity Level: 100-------------------------------------------------------------------------------Modified:"/etc/crontab""/etc/fstab""/etc/motd"…
December 06, 2005 33
CSC Systems Inspection
These Filesystem Integrity Scanners verify that the system binaries and configuration files are not compromised Attacker would have to modify the Filesystem Integrity
Scanner database to prevent notification Could also disable the cron entry
If you monitor systems, be sure to watch for missing emails
Tripwire and Osiris use cron entries to run daily checks, which send out a single summary email
Samhain has a client-server architecture, can send out alerts within minutes of finding an issue We highly recommend Samhain for new installations
December 06, 2005 34
CSC Systems Inspection
"Hemisphere was the only system to survive the attacks”
Why? Luck We do what we can to limit vulnerabilities but focus most of
our attention on early detection and the ability to recover
Have taken the time to focus on security Is not easy, or quick So what have we done?
December 06, 2005 35
CSC Security Tools and Policies
In addition to the Filesystem/Configuration Integrity
Scanning and Reporting:
System Hardening
Foreign login detection and reporting (custom scripts)
“Rootkit” scanning and reporting (custom wrapper scripts)
We have and follow Incident Response Procedures
December 06, 2005 36
CSC Security Tools and Policies
Centralized Log Server
We were in the process of building this server when the CS
compromise happened
All syslog, Samhain, chkrootkit, rkhunter logs to this server
Extremely secure configuration:
Only three Administrators have accounts, use different
passwords from any other system
Root cannot login
EAL4+ Common Criteria Installation
IBM RPM/script which hardens SuSE Enterprise Linux 9
December 06, 2005 37
CSC Security Tools and Policies
System Hardening Common Service Audit
Remove extraneous installed packages Only core, necessary services running
Configuration of required software PAM: ssh, sudo, su
Host based firewalling Shorewall
User Policies We run John-the-Ripper Password cracking utility Users are created with account aging
December 06, 2005 38
CSC Security Tools and Policies
chkrootkit - a tool to scan for rootkits We have a wrapper to consolidate the output from
the many servers into a single email: Lack of scalability in security tools is an issue!
chkrootkit-0.45 found the following warnings, vulnerabilities, and infections on hosts:node0[01-64],hemispheretransputeroccam,blade0[01-27]loaner,store01,store02,store03
Errors occured on the follwing nodes that prevented chkrootkit from completing:NONE
The following hosts were not scanned:node008blade003blade016
All logs generated for this run are located at: /home/admin/logs/chkrootkit/20051206/Quick Stats:
warnings = 0vulnerabilities = 0infections = 0
December 06, 2005 39
CSC Security Tools and Policies
Foreign Login Detection custom script
Detects login activity from new machines Short summary describes new logins
Administrator follow-up Look for accesses from outside of known activity System administrators contact Advisor/PI to verify student’s
location (this gets their attention)
We have caught several compromised passwords this way
(Source: CSC Group Slide by Matthew Woitaszek)
December 06, 2005 40
CSC Security Tools and Policies
User training: Don’t type your Hemisphere password in on a remote machine, it might have a trojan SSH! Connect directly from your client!
--------------------------------------Host report times
hemisphere 2005-12-01 00:02:15
--------------------------------------First logins for each user from a host
oberg 2005-11-30 14:05:50 hemisphere ec240-175-dhcp.colorado.edu 2005-11-30 15:47:43 hemisphere ec240-250-dhcp.colorado.edu
sliu 2005-11-30 23:44:33 hemisphere ae203-053.resnet.stonybrook.edu 2005-11-30 23:46:22 hemisphere ae203-053.resnet.stonybrook.edu
Why is sliu, usually at a desk in ECCS122, connecting from a dorm in Long Island at 1:45AM Eastern Time?
(Source: CSC Group Slide by Matthew Woitaszek)
Foreign Login Detection Example
December 06, 2005 41
CSC Security Tools and Policies
Incident Response Procedure
Secure email: Commercial PGP, GPG for the rest
Cell phones Upon any notification of compromise,
immediately disconnect all machines from the internet, and conduct onsite analysis
Start to download offsite backups for restore