YAGNI vs. Agile @ Scale

From AgileOpenNorthwest

Agile @ Scale

In my current job at AboutUs I work on several different web applications, one of which is a high traffic website backed by large amounts of data. In general we're a very agile team, practicing and continually refining a process which combines elements of Extreme Programming and Kanban.

We have a recurring experience.

When we are working on some of the newer, smaller projects we feel very agile. Change happens quickly and cheaply. Stakeholders are wowed by our progress. It is easy to change directions when it's deemed to be appropriate.

Contrast this with our experience working at scale. It's not unusual for us to have cards which involve doing updating 10 or 15 or 150 million records. Did you know that 1 million seconds is 11.5 days? In these contexts we often feel unagile. It takes long periods to make progress on a job. Perhaps an error is discovered and we have to start over. Perhaps it took a week of work to even get the feedback of error. Hours become days. Days become weeks, and our once happy stakeholders don't see rapid user facing progress.

• So how to be agile at scale?
• How to keep change cheap when a (similar) change has to be made 150 million times?
• How to get rapid feedback when a job might take 11 hours to fail?
• How to work iteratively and without Big Design Up Front, when a solution that works for 100 records may not easily scale to 100 million?

Session Notes

Main Take-aways for me:

• Know when to scale. Build to your requirements. Scale when necessary (right away sometimes)
• Identify "Canaries" that warn you of upcoming scaling issues.
• You can't always predict where the pain points will be, but the more experience you have with the system/domain the better you can guess.
• If a task is too big, try to break it into smaller chunks
• It's important to inform your stakeholders about where you predict scaling issues. Make your performance expectations clear.
• Draw a pretty graph which predicts "$0 revenue by {some date}". This is when the servers will catch fire.
• Scaling teams is also difficult and important.
• Scaling is similar to refactoring. If you're consistently alleviating pain points that come up, it will likely be easier to scale your system.
• Always try to benchmark new tools/technologies you're considering using. Fail early. Don't waste a year of development because is sounded cool in some blog post. Make sure it can perform to your requirements.
• Some cloud technologies have promise for being able to transparently scale to "Oprah traffic" then back down when the wave has passed.

Image:AgileAtScale1of3.jpg Image:AgileAtScale2of3.jpg.jpg Image:AgileAtScale3of3.jpg.jpg