This is the second in a series of blogs where Data Scientists Anna Godwin and Cory Everington discuss five Analytics Best Practices that are key to building a data-driven culture and delivering value from analytics. In this installment, Anna discusses the benefits of using Agile Data Science as a framework for managing data science projects.
If you’ve worked on any project you’ve likely experienced unexpected changes over the project lifetime. It is those surprises that can impact your ability to complete the project as it was initially designed, with changes in the team, the deliverable, and/or the timeline. Being flexible as an individual and as a team throughout a project enables you to quickly respond to changes. But projects benefit from being structured. What is the best framework for managing the right balance for data science projects?
For our project work Elder Research employs the Agile Data Science methodology. It combines best practices for Agile software development with best practices for data mining, as embodied by the Cross Industry Standard Process for Data Mining (CRISP-DM).
Agile Data Science provides the following benefits; It:
- Provides flexible team organization
- Generates early buy-in from stakeholders
- Adapts to unforeseen circumstances
Agile Data Science Brings Organization to the Project Team
While it is possible to use an agile methodology when working alone, the approach is designed to help organize the work for a team. Any time team members are working on different aspects of a project there can naturally be confusion, duplication of efforts, or work on tasks that are not focused on project deliverables. Agile Data Science helps mitigate these problems and create a shared vision through sprint planning, daily stand-ups, team transparency, and roadblock awareness.
Two main components of agile methodology govern project work:
Sprint Planning is a process where project work is broken down into small chunks of work that can be completed in the span of a few hours or days and assigned to team members. It is often done on a two-week cycle to allow team members to iterate quickly and provide flexibility in how the work will be completed. We will go into more detail on how to plan sprints in our upcoming blog post about Task Management.
Daily Stand-ups facilitate communication and project success by creating a forum for each team member to give a short (2-3 minute) update about their progress on assigned tasks and on any roadblocks they encountered that could impact the schedule. Daily stand-ups are designed to be less than 15 minutes. They are not a forum for detailed discussions, instead, they provide an opportunity for team members to understand what others are working on and for the project lead to adjust course as necessary.
Agile Data Science Engages Stakeholders
The iterative nature of the agile methodology requires frequent check-ins with project stakeholders. Sharing results early and often allows the project team to focus on building a quick solution that provides value and gains end-user adoption. A focus on deployment early in the process is core to the Elder Research approach. Whether the deliverable is a complete software solution or an analytic algorithm, it is important to gain early buy-in from stakeholders— and that happens by getting their feedback early in the process. A solution is only successful if implemented by the end-user, so early feedback from key users is crucial for shaping a solution’s design. And, a strong side benefit is that the key users become invested in the solution.
Agile Data Science Quickly Adapts to Change
Changes come from all directions in project work. For example, a team member moves to another project or a new member joins the team. Being able to quickly get new team members on board is a core agile strength. Team stand-ups and sprint planning provide frequent project updates and introductory assignments that new members can quickly complete until thorough knowledge transfer occurs.
Changes to the group of stakeholders may occur as well. As the stakeholders change, so may the vision for the analytics project. When it feels like the project target is constantly moving, the adaptive nature of an agile approach works in your favor. The iterations that take place during an agile project lifecycle provide a key opportunity to either stay the course or to pivot in a different direction. Having multiple “Yes” or “No” decision points with stakeholders drives the difference between project completion and successful project completion.
A final type of unexpected change may come in the form of a technical hurdle. For example, the initial technology stack chosen for the project may not be working well. Again, the quick iterations of Agile Data Science allow for a “fail fast” reality where technical limitations come to light much sooner in the project lifecycle than otherwise. This early insight allows the team to quickly discover an alternative that can provide a better solution for the project during the project timeline.
Find an Approach That Works for You
The keyword in Agile Data Science is — you guessed it — agile. We highlighted the benefits of adopting a flexible structure for your project work, and at the end of the day, each project team should adopt a structure that works best for them. Be flexible in how you set up your agile approach so you spend less time on project management and focus more on completing the technical project work to create actionable solutions that deliver real value to end-users.
