Building a Resilient Store

The cornerstone of any feature flag system is its configuration store. The implementation choices made here significantly impact your system’s behavior under various conditions, particularly during failures or high-load scenarios. These choices become especially important when implementing circuit breaking and fallbacks and performance optimizations.

class ConfigurationStore {
  private readonly cache: LocalCache<FeatureConfig>
  private readonly store: RemoteStore<FeatureConfig>
 
  async getFeatureConfig(feature: string): Promise<FeatureConfig> {
    try {
      // Try to get fresh config first
      const config = await this.store.fetch(feature)
      await this.cache.set(feature, config)
      return config
    } catch (error) {
      // On failure, try cache
      const cached = await this.cache.get(feature)
      if (cached && !this.cache.isStale(cached)) {
        return cached
      }
 
      // Last resort: secure defaults
      return {
        name: feature,
        enabled: this.getSecureDefault(feature),
        rules: [],
        lastUpdated: Date.now(),
        source: 'default',
      }
    }
  }
 
  private getSecureDefault(feature: string): boolean {
    // This should return the most secure default value for the feature
  }
}

This implementation demonstrates a store-first strategy with fallbacks:

1. Fresh Configuration: Attempt to get the most up-to-date values first
2. Local Cache: Handle temporary outages and reduce load on the primary store
3. Secure Defaults: Ensure safe operation when all else fails

Change Propagation Strategies

Configuration changes in distributed systems present a fundamental challenge: balancing consistency with performance. The propagation strategy you choose directly impacts both system reliability and user experience. The key considerations are:

1. Consistency requirements: How quickly must changes propagate?
2. Resource constraints: How much load can your infrastructure handle?
3. Failure modes: What happens when updates fail?

You may opt for a single implementation, or an implementation that supports multiple propagation strategies:

class StrategyBasedConfigStore extends ConfigurationStore {
  async getFeatureConfig(feature: string): Promise<FeatureConfig> {
    const strategy = this.getStrategyForFeature(feature)
 
    switch (strategy) {
      case 'ALWAYS_FRESH':
        return this.getFreshConfig(feature)
      case 'CACHE_FIRST':
        return this.getCacheFirstConfig(feature)
      case 'CACHED_WITH_BACKGROUND_REFRESH':
        this.refreshInBackground(feature)
        return this.getCachedConfig(feature)
      default:
        return super.getFeatureConfig(feature)
    }
  }
}

For example:

• Payment processing flags might use ALWAYS_FRESH with real-time updates
• UI customization flags might use CACHE_FIRST with polling
• A/B test flags might use CACHED_WITH_BACKGROUND_REFRESH with a message queue

For systems that can tolerate eventual consistency, polling with jitter provides an efficient baseline solution:

class PollingConfigSync {
  private readonly interval: number
  private lastSync: number = 0
 
  async startPolling(): Promise<void> {
    while (true) {
      try {
        await this.syncConfigurations()
        await this.sleep(this.calculateNextInterval())
      } catch (error) {
        await this.handleSyncError(error)
      }
    }
  }
 
  private calculateNextInterval(): number {
    const baseInterval = this.interval
    const jitter = Math.random() * (baseInterval * 0.1)
    return baseInterval + jitter
  }
}

The jitter prevents synchronization of polling cycles across instances, protecting against load spikes (the “thundering herd” effect). For stronger consistency requirements, WebSocket or Server-Sent Events connections enable immediate updates.

Most production deployments benefit from a hybrid approach, using different propagation methods based on the consistency requirements of each flag type:

Version Control and Conflict Resolution

Version control in distributed systems centers on a fundamental challenge: managing concurrent changes reliably. If you’re thinking “how hard can it be?” — well, distributed systems have a way of humbling us all. These challenges are closely related to our discussion of race conditions and change propagation strategies.

Each configuration change requires:

• A strictly increasing version number
• A precise timestamp
• Author identification
• A reference to the previous version

The version number acts as a simple conflict detector — if a service tries to apply an update based on an old version, we can reject it immediately. Timestamps help with auditing and debugging, while author information and previous version references create a clear chain of modifications.

When conflicts occur, the system needs clear resolution rules. The simplest approach is “last write wins,” but this can lead to lost updates. A better strategy is to detect conflicts early and force explicit resolution, especially for critical flags. This might mean requiring manual intervention or applying predefined merge rules based on the flag’s type and importance.

History and Rollbacks

Feature flag changes require robust audit and rollback capabilities. When issues occur, you need both a clear record of changes and a safe path back to a known good state.

Core requirements:

1. Atomic transactions for state changes
2. Complete change metadata
3. Dependency tracking
4. Impact analysis

For example:

interface FeatureChange {
  id: string
  feature: string
  oldState: boolean
  newState: boolean
  metadata: {
    author: string
    reason: string
    timestamp: number
    environment: string
  }
}
 
class FeatureHistory {
  constructor(
    private store: FeatureStore,
    private history: ChangeHistory
  ) {}
 
  async updateFeature(feature: string, enabled: boolean, metadata: ChangeMetadata): Promise<void> {
    const currentState = await this.store.get(feature)
    const change = this.createChange(feature, currentState?.enabled ?? false, enabled, metadata)
 
    await this.store.transaction(async (tx) => {
      await this.history.record(change, tx)
      await this.store.save(feature, { enabled }, tx)
    })
  }
 
  private createChange(
    feature: string,
    oldState: boolean,
    newState: boolean,
    metadata: ChangeMetadata
  ): FeatureChange {
    return {
      id: crypto.randomUUID(),
      feature,
      oldState,
      newState,
      metadata: {
        ...metadata,
        timestamp: Date.now(),
      },
    }
  }
}

Before executing a rollback, assess its potential impact. Performing a dry run helps identify affected systems and dependencies:

class RollbackPreview {
  async previewRollback(feature: string): Promise<RollbackImpact> {
    const [history, dependencies] = await Promise.all([
      this.history.getChanges(feature, { limit: 2 }),
      this.dependencies.findDependents(feature),
    ])
 
    if (history.length < 2) {
      throw new Error('Insufficient history for rollback')
    }
 
    const lastChange = history[0]
    // Analyze which systems would be affected by reverting to the old state
    // This includes checking service dependencies, active user sessions,
    // and any downstream systems that consume this flag
    const impactedSystems = await this.getImpactedSystems(feature, lastChange.oldState)
 
    return {
      feature,
      currentState: lastChange.newState,
      proposedState: lastChange.oldState,
      timestamp: lastChange.metadata.timestamp,
      dependentFeatures: dependencies,
      impactedSystems,
      // Evaluate potential risks based on:
      // - Time of day/week for the rollback
      // - Number of active users affected
      // - Criticality of impacted systems
      // - Recent deployments or changes in dependent systems
      risks: this.assessRollbackRisks(dependencies, impactedSystems),
    }
  }
}

To understand why thorough impact analysis is crucial, let’s look at how feature flags typically depend on each other in a real-world application:

Looking at this dependency graph, we can quickly spot some specific rollback scenarios that might cause issues:

1. Cascading Rollbacks: If we need to roll back the Payment API v2 flag due to an issue, we can’t do so without also considering both the New Checkout UI and the currently-disabled Express Checkout features. Additionally, we need to ensure Payment API v1 is still operational and can handle the traffic, as it’s the fallback system.

2. Conflict Resolution During Rollback: In our current state, Performance Mode is active, which is why Express Checkout is disabled. If we need to roll back Performance Mode, we would need to decide whether to automatically re-enable Express Checkout. This isn’t just a technical decision—it requires understanding the business context of why Express Checkout was disabled and whether it’s safe to re-enable.

3. Legacy System Interactions: The migration from Payment API v1 to v2 presents a particularly challenging rollback scenario. If we’ve migrated customers to v2 and need to roll back, we need to ensure v1 and the Legacy Payment System can still handle current transaction volumes and that all transaction data is compatible with the older system. This might also require maintaining both versions temporarily during the transition.

These scenarios demonstrate why the RollbackPreview implementation above needs to be thorough in its impact analysis. A rollback strategy needs to consider not just the technical dependencies but also the business implications of each change. The assessRollbackRisks function must account for all these factors to provide accurate guidance for rollback decisions.

Access Control and Audit

The first line of defense is knowing who can change what. Building upon our discussion of RBAC and entitlements , effective access control goes beyond simple permission checks. You need to think about:

• Who should be able to create new flags?
• Which environments can they modify?
• Do certain flags need extra approval?
• How do you handle emergency changes?

Here’s a basic implementation that captures these concerns:

class SecureFeatureManager {
  async updateFeature(
    feature: string,
    update: FeatureUpdate,
    context: SecurityContext
  ): Promise<void> {
    // Throws if user lacks required permissions
    await this.verifyAccess(feature, context)
 
    if (this.isHighRiskChange(update)) {
      await this.requestApproval(feature, update)
    }
 
    await this.store.transaction(async (tx) => {
      await this.auditLog.record(feature, update, context)
      await this.store.update(feature, update)
    })
  }
}

Lifecycle Management

Feature flags are like code — they have a lifecycle. As we saw in the fundamentals article , flags can silently persist long after they should have been retired. There are codebases with hundreds of “temporary” flags that were anything but temporary.

The solution requires both tooling and process. While tools can help track and manage flags, they must be coupled with clear ownership and processes. Every flag needs:

Organizational Requirements:

• A designated owner who understands its purpose
• Clear documentation of affected systems
• Defined criteria for retirement
• Regular reviews and audits

Technical Implementation:

• Automated tracking of flag usage
• Dependency management systems
• Audit logging capabilities
• Expiration date enforcement

The principle of least privilege applies here too. Development teams should be able to experiment freely with flags in lower environments, but production changes need more oversight. This doesn’t mean creating bureaucracy — it means having clear processes supported by appropriate tooling.

Technical Monitoring

Feature flag checks might seem lightweight individually, but their performance impact compounds quickly. Even a seemingly fast 50ms check becomes significant when multiplied across your application. For example, if your homepage makes 30 flag checks per load across millions of requests, that’s a substantial performance cost that needs careful management. This is why we emphasize the importance of multi-level caching and resource optimization .

class FeatureFlagMetrics {
  async checkFeature(feature: string, context: Context): Promise<boolean> {
    const startTime = performance.now()
    try {
      const result = await this.evaluator.evaluate(feature, context)
      this.recordMetrics('success', feature, performance.now() - startTime)
      return result
    } catch (error) {
      this.recordMetrics('error', feature, performance.now() - startTime)
      throw error
    }
  }
}

Your caching strategy becomes crucial here. Keep an eye on cache hit rates — they tell you how effectively your system is performing. A sudden drop in cache hits might mean something’s wrong with your cache configuration, or that your cache size needs adjusting.

Set up alerts for things that need immediate attention:

• Latency spikes in flag evaluation
• Unusual error rates
• Sudden changes in evaluation patterns
• Cache performance issues
• Unexpected flag state changes

Business Impact Analysis

Technical metrics tell you how your system is running, but business metrics tell you if your features are actually working. When you’re rolling out a new feature, you need to watch both.

The key is comparing metrics between your control group and test group. You might find surprising results — like a feature that improves one metric but hurts another. For instance, a new UI might increase engagement time but decrease conversion rates. These insights help you make informed decisions about your rollouts.

You’ll want to track both technical and business metrics together:

interface FeatureMetrics {
  technical: {
    evaluationLatency: Histogram // Distribution of flag evaluation times
    errorRate: Rate // How often flag checks fail
    cacheHitRate: Rate // Effectiveness of your caching
    evaluationVolume: Counter // How often the flag is checked
  }
  business: {
    conversionRate: Rate // How often users complete the desired action
    averageOrderValue: Histogram // Average value of orders placed
    userEngagement: Counter // How often users interact with the feature
    customMetrics: Record<string, Metric> // Custom metrics for your specific use case
  }
}
 
class MetricsAggregator {
  async collectMetrics(feature: string, timeRange: TimeRange): Promise<FeatureMetrics> {
    const [technical, business] = await Promise.all([
      this.getTechnicalMetrics(feature, timeRange),
      this.getBusinessMetrics(feature, timeRange),
    ])
 
    // Look for relationships between technical and business metrics
    // For example:
    // - Do higher latencies correlate with lower conversion rates?
    // - Does cache performance affect user engagement?
    // - Are error spikes impacting business metrics?
    const correlations = await this.analyzeCorrelations({
      technical,
      business,
      significanceThreshold: 0.05, // Only report strong correlations
    })
 
    return {
      technical,
      business,
      correlations,
    }
  }
}

Multi-Level Caching Architecture

Caching isn’t just an optimization — it’s a necessity. Think of your feature flags like your phone’s contacts: you keep your favorites just a tap away, your regular contacts on your phone, and your complete history safely backed up to the cloud. The same goes for feature flags — keep your most-used ones in memory, common ones in a shared cache, and the complete set in reliable storage.

Each layer serves a specific purpose:

1. Local cache gives you lightning-fast access to frequently used flags
2. Distributed cache helps your application instances share data efficiently
3. Primary store keeps the source of truth
4. Background refreshes keep everything in sync without blocking requests

Resource Optimization

As your system grows, you’ll find that it’s not just about caching the flags themselves — it’s about being smart with how you handle the context around them. Every bit of data you pass around has a cost, whether it’s memory usage or processing time. This becomes particularly important when implementing technical monitoring and managing performance at scale.

Consider implementing context validation and normalization:

class ContextOptimizer {
  normalizeContext(context: RawContext): NormalizedContext {
    return {
      userId: this.normalizeUserId(context.userId),
      attributes: this.filterRelevantAttributes(context.attributes),
      environment: this.validateEnvironment(context.environment),
      // Only include necessary fields
      ...this.getOptionalFields(context),
    }
  }
 
  private filterRelevantAttributes(attributes: Record<string, any>): Record<string, any> {
    // Keep only the attributes that matter for your active flags
    const relevantKeys = this.getRelevantAttributeKeys()
    return Object.fromEntries(
      Object.entries(attributes)
        .filter(([key]) => relevantKeys.has(key))
        .map(([key, value]) => [key, this.normalizeValue(value)])
    )
  }
}

Circuit Breaking and Fallbacks

Think of circuit breaking like a modern electrical breaker panel — while traditional breakers need manual resets, newer ones can automatically retest and restore power when safe. Similarly, our feature flag circuit breaker automatically monitors for issues and can self-heal when conditions improve. This is a critical part of our error handling strategy and ties directly into our monitoring system .

When your feature flag service shows signs of trouble (like increased error rates or latency), the circuit breaker temporarily routes requests to fallback mechanisms, protecting your system while periodically testing if normal operation can resume.

Here’s a basic implementation example including recovery:

class ResilientFeatureManager {
  private readonly circuitBreaker = {
    failures: 0,
    lastFailure: 0,
    threshold: 5,
    resetTimeout: 60000, // 1 minute
 
    recordFailure() {
      this.failures++
      this.lastFailure = Date.now()
    },
 
    isOpen(): boolean {
      // Reset after timeout
      if (Date.now() - this.lastFailure > this.resetTimeout) {
        this.failures = 0
        return false
      }
      return this.failures >= this.threshold
    },
 
    reset() {
      this.failures = 0
      this.lastFailure = 0
    }
  }
 
  async isEnabled(feature: string, context: Context): Promise<boolean> {
    // If we've seen too many recent failures, don't even try
    if (this.circuitBreaker.isOpen()) {
      return this.getFallbackValue(feature)
    }
 
    try {
      const result = await this.evaluateFlag(feature, context)
      // Success! Reset the circuit breaker
      this.circuitBreaker.reset()
      await this.cache.set(feature, result)
      return result
    } catch (error) {
      // Record the failure and use fallback
      this.circuitBreaker.recordFailure()
      return this.getFallbackValue(feature)
    }
  }
 
  private getFallbackValue(feature: string): boolean {
    // Use cached value if available, otherwise fall back to secure default
    const cached = this.cache.getSync(feature)
    return cached?.enabled ?? this.defaults.getSecureDefault(feature)
  }
}

Recovery and Resilience

Handling immediate failures is just the start. Your system needs to be able to recover gracefully and get back to a healthy state. A few key strategies help here:

1. Smart Retries: When a request fails, don’t immediately retry at full speed. Start with a short delay and gradually increase it (exponential backoff). This gives overloaded systems time to recover.

2. State Reconciliation: Even with careful error handling, your caches can get out of sync. Run periodic checks to verify cache contents against your source of truth, quietly fixing any discrepancies.

3. Health Checks: Don’t wait for users to tell you something’s wrong. Actively monitor your system’s vital signs — response times, error rates, resource usage. Often, you can spot problems developing before they affect users.

Learning from Failures

Every failure is a chance to make your system more robust. When things go wrong, capture the full context:

• What was happening when it failed?
• What were the system conditions?
• Which fallback mechanisms kicked in?
• Did they work as expected?

This information is gold for preventing similar issues in the future. Pay special attention to failures during high-traffic periods — they often reveal the weak points in your architecture.

Graceful Degradation

Not every failure is a complete outage. Sometimes parts of your system become slow or unreliable. Your feature flag system should adapt:

• If complex targeting rules are slow, fall back to simpler ones
• If real-time updates aren’t working, switch to polling
• If a cache is struggling, temporarily bypass it

The goal isn’t to maintain perfect functionality — it’s to keep your core features working reliably, even if some advanced capabilities are temporarily unavailable.