Just another bughunt
-
Upload
charles-fulton -
Category
Technology
-
view
1.257 -
download
0
description
Transcript of Just another bughunt
![Page 1: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/1.jpg)
Just another bughunt?
Tools to improve your site without nuking it from orbit#DPA11Ken Newquist (@knewquist) | Charles Fulton (@mackensen)
![Page 2: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/2.jpg)
Who we are
Ken NewquistDirector, Web Applications Development Lafayette College
Charles FultonSenior Web Applications DeveloperLafayette College
#DPA11
![Page 3: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/3.jpg)
Rebuild or Fix?
● Your website’s problems may seem intractable
● The temptation to nuke the bugs and start fresh is strong
● We’ve found tools that identify the problems so we can surgically eliminate them○ (and find a few issues we didn’t know about in the
process)#DPA11
![Page 4: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/4.jpg)
#DPA11Tools
![Page 5: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/5.jpg)
● Crawls web presence● Reports broken links and common
misspellings● Shows changes over time● Pretty graphs!
Siteimprove
#DPA11
![Page 6: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/6.jpg)
Pretty graph!#DPA11
![Page 7: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/7.jpg)
Splunk
● Log aggregation● Real-time monitoring● Rich analysis● More pretty graphs!
#DPA11
![Page 8: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/8.jpg)
Another pretty graph!#DPA11
![Page 9: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/9.jpg)
Nagios
● Real-time monitoring● Defines a base-line of system performance● Does not detect presence of dinosaurs
#DPA11
![Page 10: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/10.jpg)
Dinosaurs! #DPA11
![Page 11: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/11.jpg)
OSSEC
● Log-based intrusion detection system● Define states of acceptable behavior● No pretty graphs
#DPA11
![Page 12: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/12.jpg)
Not a pretty graph :/#DPA11
![Page 13: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/13.jpg)
● Define expected behavior with OSSEC & Nagios
● Test expectations with Siteimprove & Splunk
● Here be monsters
Discovering your web presence
#DPA11
![Page 14: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/14.jpg)
Investigations #DPA11
![Page 15: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/15.jpg)
The Lost Thumbnails
● Site: Moodle● Tools: Splunk, OSSEC● Outcome: Improved
Apache configuration
#DPA11
![Page 16: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/16.jpg)
Sky falling!
● Splunk reported ~400 500 internal server errors within a few minutes
● Also showed concentrated bursts of 404 errors when viewing resources
● Concern within department that sky was falling
#DPA11
![Page 17: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/17.jpg)
Sky not falling!
● System ran out of memory generating thumbnails from massive images; threw 500s
● Preview of missing images generated the 404s
#DPA11
![Page 18: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/18.jpg)
Outcomes
● Memory limits were not reasonable● Users do not report catastrophic errors
#DPA11
![Page 19: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/19.jpg)
Comments
● Site: WordPress● Tools: Splunk, OSSEC● Outcome: WordPress
core fixes
#DPA11
![Page 20: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/20.jpg)
What Lies Beneath
● 500 errors are reserved for server issues● WordPress has notions of its own
○ Double-submitted comment? 500 error○ Missing a required field? 500 error○ Blank comment? 500 error
● OSSEC would ban all of these for bad behavior
#DPA11
![Page 21: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/21.jpg)
https://github.com/bigcompany/know-your-http#DPA11
![Page 22: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/22.jpg)
Outcomes
● Learned reasonable mistakes can yield unreasonable error codes
● Hacked core to return 200s and 400s instead
● Core is discussing what to do○ https://core.trac.wordpress.org/ticket/11286
#DPA11
![Page 23: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/23.jpg)
Revenge of the Base Theme
● Site: WordPress● Tools: Siteimprove● Outcome: WordPress
theme fix; Apache configuration change
#DPA11
![Page 24: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/24.jpg)
March 10: the day the links broke#DPA11
![Page 25: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/25.jpg)
Nothing to see here … oh wait--
● Developer dismissed initial reports of login issues as user error
● Then Siteimprove said we had 1,800 new broken links
● A two-character change in RHEL defaults for httpd.conf broke WordPress
#DPA11
![Page 26: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/26.jpg)
Lessons
● Small changes have vast consequences● Documentation is doubleplusgood
#DPA11
![Page 27: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/27.jpg)
The Incredible Shrinking Provost
● Site: Drupal● Tools: Splunk● Outcome: Cleaned data in
ERP system
#DPA11
![Page 28: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/28.jpg)
Who’s the fairest of them all?
● The directory passes the search query via a GET parameter
● Splunk told us our associate provost, “Jane Doe”, was most-searched by an order of magnitude
#DPA11
![Page 29: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/29.jpg)
...we searched for “Jane Doe”...
...and the search returned...
...NOTHING!
#DPA11
![Page 30: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/30.jpg)
Lessons
● “Jane A. B. Doe !== Jane Doe”● Data lies
#DPA11
![Page 31: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/31.jpg)
Dumpster fire#DPA11
![Page 32: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/32.jpg)
The Virtual Tour
● Site: Custom app● Tools: Splunk● Outcome: Fixed PHP
bugs
#DPA11
![Page 33: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/33.jpg)
Pretty graphs!● 238,908 errors...in three days● (We didn’t expect that)
#DPA11
![Page 34: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/34.jpg)
Fixed it!
#DPA11
![Page 35: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/35.jpg)
Outcomes
● No one cares that we fixed the Virtual Tour ○ (we feel better though)
#DPA11
![Page 36: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/36.jpg)
Mr. Foo and Mr. Bar
● Site: WordPress● Tools: Splunk● Outcome: Disproved long-
standing alleged bug
#DPA11
![Page 37: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/37.jpg)
I swear I wasn’t there!
● Various reports over the years alleging that WordPress improperly reported another user was editing a post
● Much speculation and theorizing in absence of facts
#DPA11
![Page 38: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/38.jpg)
Outcomes
● People are wrong on the Internet
#DPA11
![Page 39: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/39.jpg)
The Cache That Wouldn’t Die● Site: WordPress● Tools: Nagios● Outcome: Database
size reduced by two-thirds
#DPA11
![Page 40: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/40.jpg)
Doom at 11….
● Nagios had concerns
● MySQL ran out of disk space
● Size of WordPress DB tripled in two weeks
#DPA11
![Page 41: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/41.jpg)
Pretty terminal dumps?
SELECT option_name FROM wp_190_options WHERE option_name LIKE "displayed_gallery%";...| displayed_gallery_rendering_ffffb5e48845fbb7b3347244f8aa06d4 || displayed_gallery_rendering_ffffd6d9f2ab40195295c70f775b0ee8 || displayed_gallery_rendering_ffffe1416b8d969e25ec7a6094282bbe || displayed_gallery_rendering_ffffe8e4a0c399605f434bd51be2d9d7 |+--------------------------------------------------------------+722141 rows in set (2.28 sec)
#DPA11
![Page 42: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/42.jpg)
…Salvation at Noon
● The Google Mini found something terrible lurking in club websites
● NextGEN Gallery bug caused near-endless crawl by the mini
● Code bug meant the cache never expired
#DPA11
![Page 43: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/43.jpg)
Outcomes
● NextGEN Gallery has stability issues● Listen to Nagios● It’s turtles all the way down
#DPA11
![Page 44: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/44.jpg)
Attack of the Python Script● Site: WordPress● Tools: Nagios, Splunk● Outcome: Quickly
identified source of massive load event
#DPA11
![Page 45: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/45.jpg)
Traffic Jam!
● Load on a server spiked at 800%
● Seemed bad● Nagios had more
concerns
#DPA11
![Page 46: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/46.jpg)
Hello there!
● Splunk real-time monitoring revealed top client IPs
● We’re very popular with a misconfigured IIS Server in Oregon and its “Python-urllib/3.4” script
#DPA11
![Page 47: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/47.jpg)
Outcomes
● Banned the IP on the proxy
● Began developing rate-limiting rules for OSSEC
#DPA11
![Page 48: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/48.jpg)
Alternatives #DPA11
![Page 49: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/49.jpg)
Bughunting on the cheap
W3C Link Checker● Reports on broken links to a specified depth● http://validator.w3.org/checklinkGoogle Webmaster Tools● Details on broken links and server errors● https://www.google.com/webmasters/tools/
#DPA11
![Page 50: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/50.jpg)
More options● Bureau of Internet Accessibility
○ Cheaper than Siteimprove○ Broken link and accessibility reports○ http://www.boia.org
● Google Analytics○ Identify high-traffic broken pages○ http://google.com/analytics
● vim | grep○ Eyeballing your logs can’t hurt
#DPA11
![Page 51: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/51.jpg)
Conclusions #DPA11
![Page 52: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/52.jpg)
Did we really fix all those errors?
Or is logging broken?#DPA11
![Page 53: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/53.jpg)
● Data are free● Bugs are hard to find● Reports are expensive● Good reports make finding bugs easy● You can improve your site without rebuilding
it from scratch● You will find more bugs than you can fix
Takeaways
#DPA11
![Page 54: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/54.jpg)
#DPA11
![Page 55: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/55.jpg)
Anatomy of a Redirect
● Tool: Splunk● Forthcoming from
Lafayette College● WordPress tries to be
helpful!
#DPA11
![Page 56: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/56.jpg)
Join the discussion at https://core.trac.wordpress.org/ticket/16557!
#DPA11
![Page 57: Just another bughunt](https://reader034.fdocuments.in/reader034/viewer/2022052507/5581b1afd8b42a1d1a8b4710/html5/thumbnails/57.jpg)
Ken Newquist ● [email protected]● @knewquistCharles Fulton ● [email protected]● @mackensen
Questions?
#DPA11