Stacktrace Berlin RC.2

38
Bits of Experience DB Scaling, Cloud Hosting, Code Quality

description

Slides of my talk at Stacktrace meetup on 2014-04-03 in Berlin at Hub:raum.

Transcript of Stacktrace Berlin RC.2

  • 1. Bits of Experience DB Scaling, Cloud Hosting, Code Quality
  • 2. Hi! Nice to meet you.
  • 3. I want to offer some insights Everything's different, there's not one set of rules for everyone.
  • 4. Small Dev team, big system 10 Applications, Up to 5 stages, 36 machines.
  • 5. Our main tool is Python Also lots of SQL; More JavaScript than we'd like; Even some Perl.
  • 6. A few Tera Bytes of data Most of it in PostgreSQL. We like PostgreSQL a lot.
  • 7. Agenda Things I'll dive into a little.
  • 8. 1. Scaling Data What we do, why it works.
  • 9. 2. To cloud or not to cloud We've mostly run on bare metal, but how much longer?
  • 10. 3. Complexity and Code Quality A few thoughts.
  • 11. There's no universal right or wrong I just want to provide an additional perspective.
  • 12. We're storing advertising data Lots of data, but not quite Big Data.
  • 13. A SaaS business Isolated customers, varying amounts of data.
  • 14. All relational, structured data We aggregate and filter data a lot. SQL is great for that.
  • 15. Starting small One database, relational model by the book.
  • 16. Trying built-in PG partitioning It's useful, but very limited.
  • 17. Maybe use schemas? Then we'd lose other schema use cases.
  • 18. Partition using databases Each customer his own db. Scales very well.
  • 19. Very good tool support DBs can be easily created, copied, dumped, restored, dropped.
  • 20. Isolation also benefits security Nothing to rely on, but helpful nonetheless.
  • 21. There are downsides, of course Overhead, connection management, no quick global queries.
  • 22. For us, it's great Your mileage may vary.
  • 23. Dedicated Hardware Good performance for little money.
  • 24. Roots precede AWS Why move a working infrastructure?
  • 25. B2B: growth better predictable Elasticity requirements are limited.
  • 26. We're missing: Proper Failover Automatic failover of all components is not that trivial.
  • 27. For a while: mixed setup Frontend on AWS. Async Backend on our HW. IPSEC in between.
  • 28. Now: Something different Software development is hard.
  • 29. Start-up vs. Enterprise How different can it be?
  • 30. When is a start-up a start-up? And when not?
  • 31. Lots of uncertainty Building, testing, learning.
  • 32. Building good systems In a startup that's not always easy.
  • 33. Time is always a constraint In small companies as in big.
  • 34. Clean code is not what sells Unfortunately.
  • 35. Fast now, right later? Technical debt will cost you big time
  • 36. Small but steady Little things add up
  • 37. Delete code, drop features Code is a liability.
  • 38. Thanks for listening Time for questions.