Prior to joining REA (known for realestate.com.au - Australians 2nd highest traffic website), I worked for Rackspace as a Level 3 support engineer whilst on a working holiday in London. I took it upon myself to up-skill on MySQL as most customer calls were related to site performance of a LAMP configuration and almost always the problems were MySQL database related. Poor indexing, poor data structures poor everything!
Taking the web hosting experience and particularly the MySQL experience to REA was a great fit as there were many faults with the architecture design and configuration. Then along Marcus Barczak and others and the leadership of James Wilson we executed an operational uplift of infrastructure operations to build a really rockin' foundation to ensure high availability and performance of our key sites.
At this point of time, some 20 months ago, we had a siloed development and operations team and for the most part things worked out. Our development organsation went through a transformation three years ago moving from Waterfall delivery to Agile. Though while the developers really progressed with Agile, stories ended at code complete - not 'production'. Theres still was the 'throw it over the wall and it will make it in production' mentality/strategy/whatever.
It wasnt until a key initiative with an aggressive delivery timeline that pushed us into what some now call 'devops'. The project was 'Real Time Email Alerts' (or as the project team called it 'Fully Sick Email Alerts' of FSEA for short - theres in fact still references to FSEA at REA). Working directly with developers? Are you serious? Those guys!?. The team on the project was three developers, a technical lead, senior operations [me] and a QA. The highhanded collaboration of developers and operations really lead to 'if I write this, the code passes tests and manual verification, it can make in into production'.
This is where REA was at 20 months ago and over the next year we progressed into having type types of operations - embedded operations in (almost) all development teams and infrastructure operations. However during that year long period, it didnt feel all 'peaches and cream' - it kinda sucked! Sure, there was a reduced turn around time than prior, though most of the tasks of 'in development team operations' was deployment. Deploy deploy deploy boo! Boring! Whats worse was there was the small but apparent silo between 'embedded operations' and 'infrastructure operations'. How do you keep everyone on the same page?
Over the past year we really set out to take the next step and address the shortcomings of our development and operations model while trying to improve our code-complete-to-production timeline and really set down the automation path. Delivery manager Tomas Varsavsky allocating time for key people to address and automate some of our key repetitive tasks that were too big to piecemeal away at, and James Wilson and others allocated resourcing to set the foundation of building a continuous delivery framework for new initiatives.
This is where we are today. Our implicit mantra is
1. All new initiatives should have a build pipeline to production in an automated way.
2. Automate all existing deployments and out-source them to developers themselves.
3. Spend the recouped time by providing value with
a) pairing and consulting with developers on how to build operationally sound code
b) continuously improve existing systems
c) outsource the boring repetative work through automation
I'd love to talk about the REA journey at DevOpsDays Rome 2012!
PlanetMySQL Voting:
Vote UP /
Vote DOWN
DIGITAL JUICE
No comments:
Post a Comment
Thank's!