Seth’s Big List of Publications

 

A to Z Testing in Production: TiP Methodologies, Techniques, and Examples

  • Presentation to STPCon Spring 2012
  • March 28, 2012

Testing in production (TiP) is a set of software testing methodologies that utilizes real users and production environments in a way that both leverages the diversity of production, while mitigating risks to end users. By leveraging the diversity of production we are able to exercise code paths and use cases that we were unable to achieve in our test lab, or did not anticipate in our test planning.
This session introduces participants to TiP and gives testers methodologies they can use to TiP like Controlled Test Flights, Synthetic Test in Production, Load/Capacity Test in Production, Data Mining, Destructive Testing and more. These are illustrated with examples from Microsoft, Netflix, Amazon and others. Along the way we will cover techniques that enable TiP like Exposure Control, Test Data Handling, Production Test Hooks, and Crowd Sourcing.

  • Test labs try to approximate the production environment – TiP is the production environment.
  • Test plans try to anticipate user behavior and workflows – TiP is user behavior.

Encore presentation STPCON Fall 2012 – TBD

The future of software testing Part two– TestOps

As the software world moves to services, we can benefit from moving our signal for quality from a set of test results, to instead use the big data signal constantly emitted by these services.  This is TestOps.

Quality in the Cloud: The new Role of TestOps

  • Online Article for Software Test Professionals
  • March 2012

TWiST #85 – Testing in Production, Part I

  • This Week in Software Testing Podcast – March 2012

The future of software testing Part one – Testing in production

  • The Testing Planet, November 2011

Leaping into The Cloud: Rewards, Risks, and Mitigations

  • Presentation to Better Software Conference East
  • November 9, 2011

The cloud has rapidly gone from “that thing I should know something about” to the “centerpiece of our corporate IT five-year strategy.” However, cloud computing is still in its infancy. Sure, the marketing materials presented by cloud providers tout huge cost savings and service level improvements—but they gloss over the many risks such as data loss, security leaks, gaps in availability, and application migration costs. Ken Johnston and Seth Eliot share new research on the successful migrations of corporate IT and web-based companies to the cloud. Ken and Seth lay out the risks to consider and explore the rewards the cloud has to offer when companies employ sound architecture and design approaches. Discover the foibles of poor architecture and design, and how to mitigate these challenges through a novel Test Oriented Architecture (TOA) approach. Take back insights from industry leaders—Microsoft, Amazon, Facebook, and Netflix—that have jumped into the cloud so that your organization does not slam to the ground when it takes the leap.

Encore presentation Better Software Conference West – June 13

How the Cloud Changes Software Production

  • Better Software Magazine
  • March 31, 2011

 

Ditch the Requirements – Focus on the Customer Instead

  • Software Test Professionals
  • February 1, 2011

 

Testing in Production, Your Key to Engaging Customers

  • Presentation to STP Con – Software Test Professionals Conference 2011
  • March 23, 2011

Seth Eliot will show you how to use Testing in Production (TiP) to align your software development to your customers’ needs and discover those unarticulated needs that drive emotional attachment and market share. Seth will demonstrate the tools you can use to TiP and get direct, actionable feedback from actual users. Feature lists do not drive customer attachment, meeting key needs does. Seth maintains that getting prototypes and product in front of real users is crucial to uncover features that meet these key needs and quantify how much of an impact they will have. Understanding this impact is important since evidence shows that more than half of the ideas that we think will improve the user experience actually fail to do so—and some actually make it worse. Techniques like Online Experimentation and Exposure Control enable you to find what works and what doesn’t. Production however can be a dangerous place to test, so these techniques must also limit any potential negative impact on users. Seth shows several examples from software leaders like Microsoft, Amazon.com, and Google to show how Testing in Production with real users will enable you to realize better software quality.

 

Tracking Users’ Clicks and Submits: Tradeoffs between User Experience and Data Loss

  • Microsoft
  • September 1, 2010

 

Testing with Real Users: User Interaction and Beyond, with Online Experimentation

  • Presentation to the Better Software Conference 2010
  • June 9, 2010

Evidence shows than more than half of the ideas that we think will improve the user experience actually fail to do so—and some actually make it worse. Instead of guessing, why not measure what your real users like and don’t like? Controlled, online experiments (A/B tests being the simplest version) are a proven way to make data-driven decisions about what works and what doesn’t. Seth Eliot shares numerous examples of online experimentation within Microsoft to test new user interfaces with their customers. Seth shows how special frameworks, such as Microsoft’s ExP (Experimentation Platform), can also move testing into the high-value realm of testing-in-production. In addition to new features and designs, Microsoft tests the impact of new code in production. By employing online experimentation, you can control how and when new, potentially dangerous code is exposed to users. Exposure control enables you to reap the benefits of testing in production while limiting the potential negative impact on your customers and users

 

Method for Metallographically Revealing Intermetallic Formation at Galfan/Steel Interfaces

  • Materials Characterization, Volume 30, Issue 4, Pages 295-297
  • June 1, 1993