Crisis Management Framework:

Immediate Actions (First Hour):

1. Assess Impact:

Questions to Answer:
- How many records are affected?
- Which departments are impacted?
- Is data actively being corrupted?
- Can we stop the issue from spreading?
- What's the business criticality?

2. Contain the Problem:

// Example: Disable the problematic trigger immediately
// Create a custom setting to control trigger execution
Trigger_Control__c control = Trigger_Control__c.getInstance();
control.AccountTrigger_Disabled__c = true;
update control;

// In trigger:
if(!Trigger_Control__c.getInstance().AccountTrigger_Disabled__c) {
    // Execute trigger logic
}

3. Communicate Immediately:

Email Template:

Subject: [URGENT] Production Issue Identified - Account Data Processing

Team,

Issue Identified: [Time]
Impact: Account data calculations are producing incorrect results
Affected: Sales, Finance, Operations teams
Current Status: Issue contained, investigating root cause
Expected Resolution: [Estimate based on initial assessment]

Actions Taken:
1. Disabled problematic automation
2. Isolated affected records
3. Initiated root cause analysis

Next Update: Within 2 hours

We'll keep you informed of progress.

[Your Name]
Technical Lead

Root Cause Analysis (Hours 2-4):

1. Investigate Thoroughly:

• Review recent deployments
• Check debug logs for errors
• Analyze data patterns
• Interview developers about recent changes
• Review code changes in version control

2. Document Findings:

Root Cause Analysis Template:

Issue: Incorrect opportunity amounts being calculated
Root Cause: Recent trigger modification didn't account for multi-currency
Discovery: Debug logs showed currency conversion errors
Contributing Factors:
- Insufficient test coverage (only tested USD)
- Code review missed edge case
- No multi-currency test data in sandbox

Impact Assessment:
- 1,247 opportunities affected
- $4.2M in incorrect forecasting
- 3 departments relying on this data

Resolution and Recovery (Hours 4-24):

1. Develop Fix:

// Implement proper fix with comprehensive testing
public class OpportunityAmountCalculator {
    public static void calculateAmounts(List<Opportunity> opps) {
        // Get exchange rates for all currencies
        Map<String, Decimal> exchangeRates = getExchangeRates(opps);
        
        for(Opportunity opp : opps) {
            Decimal rate = exchangeRates.get(opp.CurrencyIsoCode);
            if(rate != null) {
                opp.Amount_USD__c = opp.Amount / rate;
            }
        }
    }
    
    private static Map<String, Decimal> getExchangeRates(
        List<Opportunity> opps
    ) {
        Set<String> currencies = new Set<String>();
        for(Opportunity opp : opps) {
            currencies.add(opp.CurrencyIsoCode);
        }
        
        Map<String, Decimal> rates = new Map<String, Decimal>();
        for(Exchange_Rate__c rate : [
            SELECT Currency__c, Rate__c 
            FROM Exchange_Rate__c 
            WHERE Currency__c IN :currencies
        ]) {
            rates.put(rate.Currency__c, rate.Rate__c);
        }
        
        return rates;
    }
}

@IsTest
private class OpportunityAmountCalculatorTest {
    @IsTest
    static void testMultiCurrencyCalculation() {
        // Test EUR
        Opportunity eurOpp = new Opportunity(
            Name = 'EUR Test',
            Amount = 100,
            CurrencyIsoCode = 'EUR',
            CloseDate = Date.today(),
            StageName = 'Prospecting'
        );
        
        Test.startTest();
        insert eurOpp;
        Test.stopTest();
        
        eurOpp = [SELECT Amount_USD__c FROM Opportunity WHERE Id = :eurOpp.Id];
        System.assertNotEquals(null, eurOpp.Amount_USD__c);
    }
}

2. Data Correction Strategy:

// Batch job to correct affected records
global class DataCorrectionBatch implements Database.Batchable<SObject> {
    global Database.QueryLocator start(Database.BatchableContext bc) {
        return Database.getQueryLocator([
            SELECT Id, Amount, CurrencyIsoCode, Amount_USD__c
            FROM Opportunity
            WHERE LastModifiedDate = LAST_N_DAYS:7
            AND Amount_USD__c != NULL
        ]);
    }
    
    global void execute(Database.BatchableContext bc, List<Opportunity> scope) {
        OpportunityAmountCalculator.calculateAmounts(scope);
        update scope;
    }
    
    global void finish(Database.BatchableContext bc) {
        // Send completion notification
        notifyStakeholders();
    }
}

Post-Incident (Days 1-7):

1. Conduct Retrospective:

Retrospective Framework:

What Went Wrong:
- Insufficient test coverage for edge cases
- Code review process didn't catch the issue
- Deployment happened without adequate testing

What Went Right:
- Issue detected within 4 hours
- Communication was prompt and clear
- Fix deployed successfully

Action Items:
1. Implement mandatory multi-currency test cases
2. Add checklist to code review process
3. Create monitoring alerts for data anomalies
4. Schedule training on testing best practices
5. Update deployment checklist

Owner assignments and deadlines for each action item

2. Implement Preventive Measures:

• Enhanced test coverage requirements
• Additional validation in deployment pipeline
• Monitoring and alerting improvements
• Updated runbooks for similar issues