PowerPoint Presentation

Welcome to the Website Emergency Room: Find and Pinpoint Problems When Everything Falls Apart

Julia Kulla-MaderJackson River@juliakm

1

Its 6am. Your website is down. People care.

The scenarios described today have been slightly changed to protect the innocent. Lets imagine that you are the IT manager for a small nonprofit. You woke up to a flood of text messages from Pingdom alerting you that your site is down and has been since about 2 am. Even worse, the organizations annual conference kicks off in a few hours. 2

You are the webmaster and it is your job to get the site back up.

While it would be great to be able to call someone else to fix this, it is clearly your responsibility. Part of your job is making sure that the website stays up.3

The usual solution (restarting the server) isnt working.

You have some tech experience. Youve tried to log into the server and couldnt. So, you restarted the server. The site came back up for a couple of minutes and then once again crashed. You tried restarting two more times with the same result a couple of minutes of uptime and then a crash.4

You need a way to diagnose and troubleshoot difficult website problems.

Restarting isnt working. Your executive director will be coming online any minute. How can you troubleshoot the problem?

Photo Credit: http://www.flickr.com/photos/24480842@N03/3040615449/">pablo lizardo5

Look for clues to avert a website crisis.

The goal today is to step back and look for clues like a good detective. This is often incredibly hard to do when it feels like everything is falling apart. But for today, pretend that you are Sherlock Holmes or Olivia Benson. 6

1. Check for obvious causes like a service outage

In any crime show, they always start with the most obvious cause first. The first suspect almost always ends up being ruled out, but we still need to start with that person. I once worked for an organization where the website suddenly went down and no one knew why. It turned out that the server was in someones basement and their cat had sat on the off switch. As you gather clues you will begin to deduce the root cause of the issue, by developing a working theory, or theories.

7

Is it down for everyone?

Is it down for everyone or just me? (Website example) The best case scenario is that your website is just down for you. For example, since you are at a conference, perhaps there is something off with the wireless network. A good way to check whether this is true is to visit downforeveryoneorjustme. On this site, you can enter a website to check if it is down. Sometimes a site can be down just for your local network or for your router. 8

Is the ISP down?

Another, less positive option, is that your Internet service provider is down. Almost always, ISPs will have a website (off of their servers) that you can check to see if the site is down. Or, Twitter is also always a good option. 9

Is DNS down?

Another, less likely, option is that your Domain Host is down while your ISP is up. This has happened with Network Solutions and AT&T in the past few years. If this happens, you dont have a lot of options. Your site will be accessible via IP but the actual domain will not be accessible.

Source: http://media.infospesial.net/image/p/2013/08/tips-jika-dns-google-down_a9729.jpg10

Did someone do something by accident? (User error)

Another unique possibility is that someone did something by accident. For example, we recently had a problem on a client site at work where the client accidentally managed to disable the Drupal panel powering the homepage, which led to a blank page. 11

Are all databases online? (Example of deleting a DB)

Sometimes your databases will go down. Sometimes they will get accidentally deleted. You can easily check and see if the database is up via phpMiyAdmin or another tool. 12

Is Apache on?

Another easy to spot issue is that Apache, or Nginx or another web server powering your site, is just offline. Ive experienced this a couple of times. A server will have been automatically restarted and for one reason or another, Apache will simply have never been turned on. 13

Is the site offline?

Another possibility is that your site is in the equivalent of maintenance mode. For example, in Drupal, you can take your site offline so that users cant enter content. You need to remember to bring it back afterwards, which is sometimes easy to forget in the heat of the moment. When I was IT manager, we once had site visitors calling in because they couldnt see anything. We could see everything fine because we were logged in. Little did we know that the site was in maintenance mode. 14

Can you reproduce the issue?

vs

Provided a real problem exists. The next important thing to do is to try and reproduce. 15

Browser?

Different browsers will behave in different ways. Okay, that was polite. The real thing is that a lot of the time IE will not display in standard ways. When you are checking for browsers, check IE from the version your site supports. Then, check the other browsers. 16

Operating system?

Is this issue only for Linux? Android? You need to identify what type of computer or phone the person is using. The next step is often to figure out how many people who visit your site use this type of device or operating system. 17

Mobile?

If this is a mobile issue, its good to identify quickly. As before, one of the first things you should do is check how many people are using this device. For example, we had a client where someone called in reporting a major problem with the donation form. It was for a very old version of Blackberry. 18

2. Check with the beat cops (tools)

Weve checked for obvious causes, now its time to check with all the tools that track information about your site. 19

Are there any server red flags?

There are a number of ways that a server can tell you it is unhappy. You do not necessarily have to be a systems administrator to read these tea leaves. You can just know when something is awry. 20

High memory usage

What your server is used for will determine how much memory and storage you need. Memory determines how fast you can retrieve information on your server. So, how fast Apache can return a page or how many applications (or for example websites) you can run on one server at the same time.

When memory is running out on your server, everything will slow down and applications like PHP powering your site may start to crash. Anyone who has every had a computer with subpar memory knows what it feels like when your machine runs out memory it slows to a crawl. 21

Full disk

Disk space or storage is the size of your hard drive. You want to avoid having your server fill to capacity without any warning. You can mitigate this with per-user storage limits. For example, everyone except for the super admin may be limited to 1 GB. When your server is full, maintenance services like database backups will often stop running. This is an early sign. Later on, your whole server may crash again and again, leading to hard drive corruption. This is something you want to avoid. 22

High CPU

High CPU typically means that you have a service running at 100 percent processing capacity. Depending on how many cores your server has, this could be 100 percent of the processor power of your machine or less. For example, if you have a dual core machine, it will have 50 percent of processing capacity in use. High CPU use over 50 percent can point to a performance problem. For example, high httpd use can crash Apache.

CPU typically also corresponds to power use. More CPUs cost more money. TODO: Verify above with Ben/Matt. 23

Are all of your security patches up-to-date?

Another scenario is that your code is running fine until some time of security vulnerability is exposed. You need to make sure that everything is up to date on your server. 24

Open Source Code (Drupal/Wordpress)

If you are using a CMS, you also need to make sure that your core code and contributed plugins and modules are up-to-date. The benefit and drawback of Open Source is that you have thousands of people using the same code as you. This is great because problems are found and fixed by people who dont necessarily work for you. It can be problematic because hackers can find security holes and exploit them. To really take advantage of Open Source, you need to be frequently updating your code. 25

What does Google say?

Google offers a number of great resources for debugging. 26

Google Webmaster tools says the site is hacked

Anyone who has had Google Webmaster tools identify their site has hacked likely remembers the experience. You start getting emails from Google that indicate your site will be removed from their search engine and all of your visitors see the scary message This site may be hacked.

This site may be hacked: This warning means that Google detected some suspicious links or pages in your site that are not malware related in a way that would infect your users, but they still should not be there.

Visiting this site may harm your computer: Youll see this message if Google believes that a site can download malicious software onto your computer27

10 percent drop or increase in traffic

If you see a big increase or decrease in traffic, particularly to one page, it may indicate a problem. It could be that one page is inaccessible or that spammers are targeting one page in particular. We had a huge increase in traffic and a crashing site at a previous job and it was because there were links to Justin Timberlake naked everywhere that were hidden28

Odd search queries View the site as a bot

In the Timberlake example, we could have also seen that there was a lot of incoming traffic for Justin Timberlake naked as opposed to sustainability, the focus of the site29

3. Interview everyone connected to the event

Like a good detective, the next step is to talk to people if we havent solved the problem yet. Put on your detective hat and lets get started. 30

How do staff web administrators use your site?

First, we want to look at how anyone making changes to the site interacts with the website. 31

Maintain a description of common operations on the site. Then walk through the process.

The key thing to know in advance is how administrators work with your site. Do they typically make lots of changes each day. What content are they editing? 32

Did you change anything recently?

Once weve gone over basic options. Next we want to know what each person on the team as done recently. Example: base64 encoding issue. We ran into a problem recently where someone was dragging and dropping images in Firefox, which was triggering a problem with the SpamSpan module33

How to website visitors outside of your organization use the site?

Once weve made sure that no one internal is the cause of the problem, its time to look outside at who is visiting your site and why

Source: https://www.digitalgov.gov/files/2014/03/Kidsgov-Usability-test-IMG_9987a-600-x-400.jpg34

Look at the most popular parts of your site

Since a typical site has thousands of visitors, interviewing each one may be challenging. Lets try looking at the most popular pages as reported by Google Analytics first. Here in GA we can see that these are the most popular parts of the site. It looks like theres a lot of incoming traffic to this page in particular. If we retrace our steps, it could be that Huffington Post has made this blog post their headline story. That explains the increase.

35

Neighbors: How do the other sites sharing your IP address use the site?

If weve eliminated everything else, it may be time to look at even more external factors. For example, I once had an Oscar predicting blog. Each Oscars, the site would eventually crash due to traffic, taking down every other site hosted on the same server. 36

Reverse IP lookup

You can do a reverse IP lookup to see what other websites may share your IP address. 37

Youll remember that we started out today by looking at what to do when your site is down, people care, and the easy solutions like restarting your server arent working. 38

You need a way to diagnose and troubleshoot difficult website problems.39

Today I hope weve shown you that you can look for clues to systematically avoid a website crisis. In particular, you can start out by looking for obvious causes, then check with the beat cops (tools like server red flags and google), and then talk to everyone connected to the event (people internally and externally)40