Dibs On Stuff

Here's part of an email we received a week ago:

"We use Dibs (with Slack and the API) to coordinate access to pulumi/terraform stacks beyond the per-execution locks those tools provide. We use long TTLs (48h) on those locks to avoid people losing them by accident."
They're talking about how Terraform provides a locking mechanism to help avoid inconsistent infrastructure states. But there's another sort of locking that they're alluding to that's important. For a lot of companies it often takes the form of a conversation in Slack and it happens in tech teams all the time.

"I'm just deploying XYZ, nobody mess with it until I'm done."
This message grants someone exclusive use of a stack for the next few hours while they get the damn thing working. I'm calling it quiet social locking.

As far as CI/CD and Terraform are concerned, the stack is available. As far as the engineer is concerned, it's still very much in use, and the more engineers and shared environments you have, the more this becomes a coordination problem.

Traditionally these conversations happen in Slack or Discord, so Dibs On Stuff works mainly as a Slack and Discord app, but if you have the sheer gumption, it's also controllable via API...

This is an example of a CI/CD workflow we see in action that works with Terraform and the Dibs API to unite the two locking paradigms.

1. Alice creates a pull request that makes changes in the staging/usw2/ec2 terraform folder.
2. She pushes her PR up, the CI/CD service takes over, does some tests, runs a terraform plan and then...
3. CI/CD makes a call to the Dibs On Stuff API stating that Alice is attempting to claim staging/usw2/ec2. If the API says it's available, the deploy proceeds. If not, she is placed in the Dibs queue and the pipeline waits.
4. When the queue becomes available, Alice takes poll position, a terraform apply runs and she begins testing.
5. When Alice is done, she kicks off the final stages of the CI/CD pipeline, which merges the code, performs any clean-up and releases her hold on the Dibs queue.
6. The rest of the team are (optionally) notified by Dibs, and the next person in line gets their deploy started.

A python script that exits 0 if you're the leader of the queue, or exits 2 if you're in line. Wrap it in a loop in your CI/CD pipeline. Or use the GitHub Actions instead.

#!/usr/bin/env python3
import os
import sys
import requests


def env(name, default=None):
    value = os.environ.get(name, default)
    if value is None:
        print(f"Missing environment var: {name}")
        sys.exit(1)
    return value


payload = {
    "conch": f"{env('SLACK_TEAM')}/{env('DIBS_QUEUE')}",
    "name": env('DIBS_USER'),
    "email": f"{env('DIBS_USER')}@{env('SLACK_TEAM')}",
    "duration": env('DIBS_DURATION'),
    "slack_team_id": env('SLACK_TEAM_ID')
}

response = requests.post(
    f"{env('DIBS_API_URL')}/seize",
    headers={"X-Api-Key": env('DIBS_API_KEY')},
    json=payload
)

result = response.json()
code = result.get("code", "")

if code in ("created", "alreadybearer"):
    print(f"Claimed {env('DIBS_QUEUE')} {code} {result.get('key', '')}")
    sys.exit(0)

elif code in ("added", "alreadyexists"):
    print(f"Waiting for {env('DIBS_QUEUE')} {code} {result.get('key', '')}")
    sys.exit(2)

else:
    print(f"Error: {response.text}")
    sys.exit(1)

This code can poll the /seize endpoint of the Dibs API. It'll either claim the item, or add the person into the queue (the queue participants can also be changed in Slack at any time using chat commands).

When it comes time for Alice to release their hold on the queue and for the conch to pass to the next person, the CI/CD platform can call the /release endpoint using a script like this:

#!/usr/bin/env python3
import os
import sys
import requests


def env(name, default=None):
    value = os.environ.get(name, default)
    if value is None:
        print(f"Missing environment var: {name}")
        sys.exit(1)
    return value


payload = {
    "conch": f"{env('SLACK_TEAM')}/{env('DIBS_QUEUE')}",
    "slack_team_id": env('SLACK_TEAM_ID'),
    "key": env('DIBS_KEY', '')
}

if env("DIBS_FORCE", False):
    del payload["key"]
    payload["force"] = True

response = requests.post(
    f"{env('DIBS_API_URL')}/release",
    headers={"X-Api-Key": env('DIBS_API_KEY')},
    json=payload
)

result = response.json()
code = result.get("code", "")

if code in ("released", "releasedgone", "releasedgonenobearer"):
    print(f"Released {env('DIBS_QUEUE')} {code}")
    sys.exit(0)

elif code in ("notreleased", "noconch"):
    print(f"Not queued in {env('DIBS_QUEUE')} {code}")
    sys.exit(0)

else:
    print(f"Error: {response.text}")
    sys.exit(1)

If you're interested in the rest of the email:

...a recurring issue is that we forget to release the locks when we are done. Then the next person who needs it has to chase down the original conch holder.
So we made a new API endpoint that lets you chase up any stragglers who might be holding onto queues for too long. Thus /nudge was born!

Sometimes adding a technical solution to a people-problem leads to over-engineering, but sometimes there's a problem standing right at your door, and Dibs On Stuff might just be the key ... for that door ... that turns the quiet social locking upside down and makes it click.

(ok it may be unclear as to whether or not Dibs has just let "the problem" into the house, look it's not a perfect metaphor - The End)