# Pipeline Troubleshooting

Common Azure DevOps pipeline issues and solutions for Forge projects.

---

## 📋 Overview

This guide covers common pipeline failures in Azure DevOps and how to diagnose and resolve them.

!!! tip "Looking for what a pipeline does?"
    This guide is about **fixing failures**. For what each pipeline deploys, what triggers it,
    and which pipeline to run for a given change, see the
    [Pipelines reference](../../reference/pipelines.md).

---

## 🏗️ Pipeline Architecture

### Forge Pipeline Structure

A repository contains only the files its templates added. The full set for an application
repository, with the condition under which each is present:

```text
.azdo/
├── azure-pipelines-api.yml             # API deploy            — API
├── azure-pipelines-api-pr.yml          # API PR validation     — API
├── azure-pipelines-auth.yml            # Auth deploy (manual)  — API
├── azure-pipelines-auth-pr.yml         # Auth PR validation    — API
├── azure-pipelines-web.yml             # Web deploy            — front end
├── azure-pipelines-web-pr.yml          # Web PR validation     — front end
├── azure-pipelines-sub.yml             # Subscription deploy   — event subscription
├── azure-pipelines-sub-pr.yml          # Subscription PR       — event subscription
├── azure-pipelines-pr-slot-cleanup.yml # PR slot teardown      — API or standalone web
├── azure-docs.yml                      # Docs publish (manual) — docs site
├── azure-docs-pr.yml                   # Docs PR validation    — docs site
├── dependabot-scan.yml                 # Daily scan (cron)     — API services
└── vars/
    ├── base.yml                        # Shared variables      — always
    ├── api.yml                         # API variables         — API
    ├── auth.yml                        # Auth variables        — API
    ├── web.yml                         # Web variables         — front end
    ├── sub.yml                         # Subscription variables— event subscription
    └── docs.yml                        # Docs variables        — docs site
```

Event service and package repositories use a different layout — a single `azure-pipelines.yml`
and `azure-pipelines-pr.yml` pair. See the
[Pipelines reference](../../reference/pipelines.md) for the full inventory by repository type.

### Pipeline Templates

Forge uses centralized templates from `SAIF/pipeline-templates`, referenced as
`refs/heads/releases/v3`:

| Template                      | Purpose                          |
| ----------------------------- | -------------------------------- |
| `azure-dotnet-api-v3.yml`     | .NET API build and deploy        |
| `azure-dotnet-api-pr-v3.yml`  | .NET API PR validation           |
| `azure-react-web-v3.yml`      | React web app build and deploy   |
| `azure-react-web-pr-v3.yml`   | React web PR validation          |
| `azure-auth.yml`              | Auth configuration deployment    |
| `azure-auth-pr.yml`           | Auth PR validation               |
| `azure-dotnet-sub-v2.yml`     | Event subscription deploy        |
| `azure-dotnet-sub-pr-v2.yml`  | Event subscription PR validation |
| `azure-docs-v2.yml`           | Documentation site publish       |
| `azure-docs-pr-v2.yml`        | Documentation PR validation      |
| `azure-pr-slot-cleanup.yml`   | PR deployment slot teardown      |

---

## ❌ Common Pipeline Failures

### 1. Template Reference Errors

**Error:** `Template reference not found` or `Unable to find template`

**Cause:** Missing or incorrect `ref` for templates repository

**Solution:**

```yaml
# Correct template reference
resources:
  repositories:
    - repository: templates
      type: git
      name: SAIF/pipeline-templates
      ref: refs/heads/releases/v3  # Must specify version
```

!!! warning "Known issue: event-service PR pipeline template not found"
    Repositories scaffolded from `saif-event-service` on Forge 3.8.2 or earlier may
    reference `azure-event-service-pr-v2.yml@templates`, which does not exist in
    `SAIF/pipeline-templates`. The Forge template source has been corrected for future
    scaffolds, but existing repositories keep their generated file. In the `extends:` block of
    `.azdo/azure-pipelines-pr.yml`, change
    `template: azure-event-service-pr-v2.yml@templates` to
    `template: azure-event-service-pr.yml@templates`.

### 2. Variable Group Access

**Error:** `Variable group 'X' is not authorized for use`

**Cause:** Pipeline not authorized to access variable group

**Solution:**

1. Go to Azure DevOps → Library → Variable Groups
2. Click on the variable group
3. Go to "Pipeline permissions"
4. Authorize the pipeline

### 3. Agent Pool Issues

**Error:** `No agent pool found` or `No hosted parallelism`

**Cause:** Agent pool not available or parallelism quota exhausted

**Solution:**

```yaml
# Use hosted agent
pool:
  vmImage: 'ubuntu-latest'

# Or specific agent pool
pool:
  name: 'SAIF-AgentPool'
```

### 4. .NET SDK Not Found

**Error:** `SDK 'Microsoft.NET.Sdk' not found`

**Cause:** .NET SDK version not installed on agent

**Solution:**

```yaml
# Ensure UseDotNet task runs first
- task: UseDotNet@2
  inputs:
    version: '10.x'
    includePreviewVersions: true
```

### 5. Node.js Version Mismatch

**Error:** `node: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_X.XX' not found`

**Cause:** Node.js version incompatible with agent OS

**Solution:**

```yaml
# Use UseNode task
- task: UseNode@1
  inputs:
    version: '22.x'
```

### 6. Docker Build Failures

**Error:** `Cannot connect to Docker daemon`

**Cause:** Docker service not running or insufficient permissions

**Solution:**

1. Verify agent has Docker installed
2. Check service account has Docker permissions
3. Use hosted agent with Docker pre-installed

### 7. Terraform State Lock

**Error:** `Error locking state` or `state is locked by another process`

**Cause:** Previous pipeline run crashed without releasing lock

**Solution:**

```bash
# Force unlock (with caution)
terraform force-unlock <lock-id>

# Or wait for lock timeout
```

### 8. Artifact Publishing Failures

**Error:** `Failed to publish artifact`

**Cause:** Artifact path doesn't exist or permissions issue

**Solution:**

```yaml
# Ensure build produces artifacts
- task: PublishBuildArtifacts@1
  inputs:
    pathToPublish: '$(Build.ArtifactStagingDirectory)'
    artifactName: 'drop'
```

---

## 🔍 Diagnostic Approaches

### 1. Enable Verbose Logging

```yaml
variables:
  System.Debug: true
```

### 2. Check Pipeline Logs

Look for these key sections:

- **Initialize job** - Agent setup issues
- **Checkout** - Repository access issues
- **Build tasks** - Compilation errors
- **Test tasks** - Test failures
- **Deploy tasks** - Deployment errors

### 3. Run Locally

Reproduce the issue locally:

```powershell
# Simulate pipeline environment
$env:BUILD_BUILDNUMBER = "1.0.0"
$env:BUILD_SOURCESDIRECTORY = (Get-Location).Path

# Run the same commands
dotnet build
dotnet test
```

### 4. Check Terraform State

```powershell
# List workspaces
terraform workspace list

# Check state
terraform state list
terraform show
```

---

## 🔎 Reading Build Logs from the CLI

Use `saif pipeline logs <build>` to stream build log output straight into your terminal —
no browser round-trip needed. The `<build>` argument is located in one of two modes, and the
CLI auto-detects which:

| Mode           | You pass…                          | Context comes from…                          | Use when |
| -------------- | ---------------------------------- | -------------------------------------------- | -------- |
| **Remote**     | A full build **URL**               | The URL itself (self-contained)              | Someone pasted a build link from Teams, an alert, or the browser |
| **Contextual** | A bare build **id**                | `--repo` → Flags → the cwd repo clone → prompt | You're investigating a build for your own service |

### Remote mode (by URL)

Paste the build URL exactly as it appears in the browser. It already carries the host,
collection, project, and build id, so **context flags are not valid** in this mode (passing
`--host`, `--collection`, `--project`, or `--repo` with a URL is a usage error):

```powershell
saif pipeline logs "https://dev.azure.com/SAIFCorporation/Customer/_build/results?buildId=162774"
```

### Contextual mode (by id)

A bare build id is only meaningful relative to a repo/pipeline context. Run it from inside a
repository clone and the host/collection are inferred automatically from the git remote:

```powershell
# Inside a clone — context inferred from the git remote
saif pipeline logs 162774

# Outside a clone — name the repo, or supply coordinates explicitly
saif pipeline logs 162774 --repo my-service
saif pipeline logs 162774 --host dev.azure.com --collection SAIFCorporation
```

### Filtering and output

Narrow the output to the part of the build you care about. Filters are case-insensitive and
combinable:

```powershell
# Only the failed steps (the analog of gh's --log-failed)
saif pipeline logs 162774 --status failed

# Scope to a stage, job, or step
saif pipeline logs 162774 --stage Build --step Compile

# Last 50 lines, or raw structured output for scripting
saif pipeline logs 162774 --tail 50
saif pipeline logs 162774 --format json
```

> `saif pipeline monitor <build-id>` shares the same context resolution, so a bare build id
> is likewise inferred from the cwd repo clone when you run it from inside one.

---

## 📊 Stage-Specific Issues

### Build Stage

| Issue         | Symptom                          | Resolution                                  |
| ------------- | -------------------------------- | ------------------------------------------- |
| Restore fails | `NU1101: Unable to find package` | Check nuget.config, verify feed access      |
| Build fails   | `CSxxxx` error                   | Fix code issue, check package versions      |
| Test fails    | `XUnit test failed`              | Review test output, check test dependencies |

### Deploy Stage

| Issue               | Symptom                   | Resolution                         |
| ------------------- | ------------------------- | ---------------------------------- |
| Slot swap fails     | `Slot busy`               | Retry or check App Service status  |
| Config update fails | `KeyVault access denied`  | Check managed identity permissions |
| Health check fails  | `503 Service Unavailable` | Check app startup, review logs     |

### Auth Stage

| Issue                 | Symptom                   | Resolution                          |
| --------------------- | ------------------------- | ----------------------------------- |
| Okta API error        | `401 Unauthorized`        | Rotate Okta API credentials         |
| Entra ID error        | `Insufficient privileges` | Check service principal permissions |
| Role assignment fails | `Principal not found`     | Verify user/group exists            |

---

## 🌍 Environment-Specific Issues

### Development (DEV)

- More permissive, may have different variable values
- Uses non-prod Okta tenant
- Terraform workspaces suffixed with `-dev`

### UAT

- Mirrors production configuration
- May have restricted access
- Requires approval gates

### Production (PROD)

- Strict approval requirements
- Uses production Okta tenant
- Blue-green deployment slots
- Extended health check timeouts

---

## 🔄 Recovery Patterns

### Failed Deployment Rollback

```yaml
# Pipeline includes rollback logic
- task: AzureFunctionApp@1
  inputs:
    deployToSlotOrASE: true
    slotName: 'staging'
    # If health check fails, no slot swap occurs
```

### Terraform State Recovery

```powershell
# Import existing resource
terraform import azurerm_storage_account.main /subscriptions/.../storageAccounts/xxx

# Remove orphaned state
terraform state rm azurerm_storage_account.old
```

### Retry Failed Stage

1. Go to pipeline run
2. Click on failed stage
3. Click "Rerun failed jobs"

---

## 📝 Pipeline Variables Reference

### Built-in Variables

| Variable                            | Description             |
| ----------------------------------- | ----------------------- |
| `$(Build.BuildNumber)`              | Pipeline build number   |
| `$(Build.SourceBranch)`             | Git branch              |
| `$(Build.Repository.Name)`          | Repository name         |
| `$(System.DefaultWorkingDirectory)` | Agent working directory |

### Forge Variables

| Variable          | Description              |
| ----------------- | ------------------------ |
| `$(ProjectId)`    | Forge project identifier |
| `$(Environment)`  | Deployment environment   |
| `$(IsProduction)` | Boolean for prod checks  |

---

## 🛡️ Prevention Strategies

### 1. Pin Versions

```yaml
# Pin .NET version
- task: UseDotNet@2
  inputs:
    version: '10.0.x'

# Pin Node version
- task: UseNode@1
  inputs:
    version: '22.x'
```

### 2. Use Lock Files

- `packages.lock.json` for NuGet
- `package-lock.json` for npm
- `.terraform.lock.hcl` for Terraform

### 3. Validate Before Deploy

```yaml
# Add validation stage
- stage: Validate
  jobs:
    - job: ValidateTerraform
      steps:
        - script: terraform validate
```

### 4. Health Checks

```yaml
# Configure deployment health checks
- task: AzureWebApp@1
  inputs:
    healthCheckPath: '/health'
    healthCheckTimeout: '300'
```

---

## 📚 Related Resources

- [Aspire Publish](aspire-publish.md) - Generate pipeline YAML from your AppHost instead of maintaining it by hand
- [Pipelines Reference](../../reference/pipelines.md) - What each pipeline does, its triggers, and PR path filters
- [Aspire Troubleshooting](aspire-troubleshooting.md) - Debug Aspire startup issues
- [Azure DevOps Pipeline Documentation](https://learn.microsoft.com/en-us/azure/devops/pipelines/)
- [Azure DevOps Services Reference](../../reference/tools/azure-devops-services.md)
- [Forge Pipeline Templates](https://dev.azure.com/SAIFCorporation/SAIF/_git/pipeline-templates)