One of the nice features of Azure DevOps Repos is the ability to control Pull Requests with PR Policies. Policies such as minimum number of reviewers, PR build checks, comment response and Work Item linking are great, however those looking to implement custom checks (like Description keywords or tags in Titles) are often in need of further validation options.

We can use a PR server and Pull Request Status validation policies to enforce compliance with custom rules we manage and control outside of Azure DevOps itself.

Setup

Setup a new Node Package

$ nvm use 10.22.1
Now using node v10.22.1 (npm v6.14.6)

Next we npm init to make this a node app.

$ npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.

See `npm help init` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg>` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
package name: (azdoprstatus)
version: (1.0.0)
description: PRStatus
entry point: (index.js) app.js
test command:
git repository:
keywords: AzDO
author: Isaac Johnson
license: (ISC) MIT
About to write to /home/builder/Workspaces/AzDOPRStatus/package.json:

{
  "name": "azdoprstatus",
  "version": "1.0.0",
  "description": "PRStatus",
  "main": "app.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [
    "AzDO"
  ],
  "author": "Isaac Johnson",
  "license": "MIT"
}


Is this OK? (yes) yes

Install Express

$ npm install --save express
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN azdoprstatus@1.0.0 No repository field.
+ express@4.17.1
added 50 packages from 37 contributors and audited 50 packages in 1.774s
found 0 vulnerabilities
   ╭─────────────────────────────────────────────────────────────────╮
   │                                                                 │
   │      New patch version of npm available! 6.14.6 → 6.14.10       │
   │   Changelog: https://github.com/npm/cli/releases/tag/v6.14.10   │
   │                Run npm install -g npm to update!                │
   │                                                                 │
   ╰─────────────────────────────────────────────────────────────────╯

Create an app.js file with contents

const express = require('express')
const app = express()
app.get('/', function (req, res) {
res.send('Hello World!')
})
app.listen(3000, function () {
console.log('Example app listening on port 3000!')
})
e.g.
$ cat > app.js <<EOF
> const express = require('express')
> const app = express()
>
> app.get('/', function (req, res) {
> res.send('Hello World!')
> })
app.lis>
> app.listen(3000, function () {
ole.log(> console.log('Example app listening on port 3000!')
> })
> EOF

Test it


Add a line to note POST received

$ cat >> app.js <<EOF
>
> app.post('/', function (req, res) {
>     res.send('Received the POST')
> })
> EOF

Test it


Exposing a public ingress

Download and install ngrok

Unzip then copy locally for WSL

builder@DESKTOP-JBA79RT:~$ cp /mnt/c/Users/isaac/Downloads/ngrok-stable-linux-amd64/ngrok /usr/local/bin/
cp: cannot create regular file '/usr/local/bin/ngrok': Permission denied
builder@DESKTOP-JBA79RT:~$ sudo cp /mnt/c/Users/isaac/Downloads/ngrok-stable-linux-amd64/ngrok /usr/local/bin/
[sudo] password for builder:
builder@DESKTOP-JBA79RT:~$ sudo chmod u+x /usr/local/bin/ngrok

Verify it works

Check our version

$ ngrok --version
ngrok version 2.3.35

Get the token from ngrok:

save the token

$ ngrok authtoken *************
Authtoken saved to configuration file: /home/builder/.ngrok2/ngrok.yml

Then launch ngrok http 3000

Now we can curl -X POST <ngrok URL> to test

Setting up Service Hooks in AzDO

First, I had the strangest thing happen in that the new project I had created of which I was owner in an organization of which i was owner would not let me set a Service Hook.  It claimed I did not have sufficient permission.

If this happens to you, ensure you have “Edit Project-level information” set.  Mine was set to “Allow (inherited)”, but clearly it wasn’t allowing me.  I changed to “Allow” (explicit) and that fixed things.

Create a subscription in a Project

Choose webhook

We will want to use “Pull request creaeted"

Enter in your URL you got from ngrok then click test

Testing should show it triggered both in Azure DevOps and in the ngrok window

And in that Test Notification window, we can click on the “Response” tab to see it got 200 status back

Click Finish

And we should see it's now live:

Allowing PR Writebacks

Next we need to actually change this PR server to send status back to Azure DevOps

After you’ve turned off ngrok and npm server, go ahead and add azure-devops-node-api and body-parser packages

$ npm install --save azure-devops-node-api body-parser
npm WARN azdoprstatus@1.0.0 No repository field.
+ body-parser@1.19.0
+ azure-devops-node-api@10.2.0
added 5 packages from 9 contributors, updated 1 package and audited 56 packages in 0.976s
1 package is looking for funding
  run `npm fund` for details
found 0 vulnerabilities

We’ll do the rest of the edits in VS Code.

builder@DESKTOP-JBA79RT:~/Workspaces/AzDOPRStatus$ code .

the app.js file:

const vsts = require("azure-devops-node-api")
const bodyParser = require('body-parser')
 
app.use(bodyParser.json())
 
const collectionURL = process.env.COLLECTIONURL;
const token = process.env.TOKEN;
 
var authHandler = vsts.getPersonalAccessTokenHandler(token);
var connection = new vsts.WebApi(collectionURL, authHandler);
 
var vstsGit = null;
connection.getGitApi().then((api) => { vstsGit = api; })
 
app.get('/', function (req, res) {
    res.send('Hello World!')
})
 
app.listen(3000, function () {
    console.log('Example app listening on port 3000!')
})
 
app.post("/", function (req, res) {
 
    // Get the details about the PR from the service hook payload
    var repoId = req.body.resource.repository.id
    var pullRequestId = req.body.resource.pullRequestId
    var title = req.body.resource.title
 
    console.log("repoID: " + repoId )
    console.log("pullRequestId: " + pullRequestId )
    console.log("title: " + title )
 
    // Build the status object that we want to post.
    // Assume that the PR is ready for review...
    var prStatus = {
        "state": "succeeded",
        "description": "wip-checker",
        "targetUrl": "https://visualstudio.microsoft.com",
        "context": {
            "name": "wip-checker",
            "genre": "continuous-integration"
        }
    }
 
    // Check the title to see if there is "WIP" in the title.
    if (title.includes("WIP")) {
 
        console.log("We got a winner!")
        // If so, change the status to pending and change the description.
        prStatus.state = "pending"
        prStatus.description = "wip-checker"
    }
 
    // Post the status to the PR
 
    vstsGit.createPullRequestStatus(prStatus, repoId, pullRequestId).then( result => {
        console.log(result)
    });
 
    res.send("Received the POST")
 
})

To test, we need to set our CollectionURL and Token (PAT).  If you need a PAT for this project, you can find/create them in the “Personal Access Tokens” of your account

We can now launch the app

And if needbe, restart ngrok

Since i did restart ngrok, I’ll need to go to the service hooks to update the URL

Change to the URL.

Note, if you test, you may see an error:

$ node app.js
Example app listening on port 3000!
repoID: 4bc14d40-c903-45e2-872e-0462c7748079
pullRequestId: 1
title: my first pull request
TypeError: vstsGit.createPullRequestStatus is not a function
    at /home/builder/Workspaces/AzDOPRStatus/app.js:57:13
    at Layer.handle [as handle_request] (/home/builder/Workspaces/AzDOPRStatus/node_modules/express/lib/router/layer.js:95:5)

Because the example status isn’t a real PR nor a real Repo ID

Testing

First, lets make an edit on the README.md

Then click commit and save it to a new branch and PR

The PR

Quick note: if you are testing changes and it fails a lot, the service connection can become disabled.  Go back to service connections to re-enable if you suspect it’s not being triggered due to too many failures:

We can see it set to “Running” (pending) if the title has [WIP]

And changing to remove “[WIP]” then updates that check (provided you added a ‘PR updated’ webhook event, see next section).

A note…

I had a lot of troubles getting the demo code to work.. It seems the latest azure CLI node package either doesnt work for Node 10.x (maybe there are some node 12 assumptions) or its got a bug on PR update:


(node:9399) UnhandledPromiseRejectionWarning: Error: TF400813: The user '' is not authorized to access this resource.
    at RestClient.<anonymous> (/home/builder/Workspaces/AzDOPRStatus/node_modules/typed-rest-client/RestClient.js:202:31)
    at Generator.next (<anonymous>)
    at fulfilled (/home/builder/Workspaces/AzDOPRStatus/node_modules/typed-rest-client/RestClient.js:6:58)
    at process._tickCallback (internal/process/next_tick.js:68:7)
(node:9399) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:9399) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

In working it out, i found the older version (9.0.1) works well with node 10.22.1 which i use:

 "azure-devops-node-api": "^9.0.1",
    "body-parser": "^1.19.0",
    "express": "^4.17.1"

And from package-lock.json:

  "azure-devops-node-api": {
      "version": "9.0.1",
      "resolved": "https://registry.npmjs.org/azure-devops-node-api/-/azure-devops-node-api-9.0.1.tgz",
      "integrity": "sha512-0veE4EWHObJxzwgHlydG65BjNMuLPkR1nzcQ2K51PIho1/F4llpKt3pelC30Vbex5zA9iVgQ9YZGlkkvOBSksw==",
      "requires": {
        "tunnel": "0.0.4",
        "typed-rest-client": "1.2.0",
        "underscore": "1.8.3"
      }
    },

Another example.

Take a look at this “Checklist checker”: https://github.com/jagdish7908/Check

Download and install the npm packages

$ git clone https://github.com/jagdish7908/Check.git
Cloning into 'Check'...
remote: Enumerating objects: 723, done.
remote: Total 723 (delta 0), reused 0 (delta 0), pack-reused 723
Receiving objects: 100% (723/723), 1.14 MiB | 5.10 MiB/s, done.
Resolving deltas: 100% (169/169), done.
builder@DESKTOP-JBA79RT:~/Workspaces$ cd Check/
builder@DESKTOP-JBA79RT:~/Workspaces/Check$ nvm use 10.22.1
Now using node v10.22.1 (npm v6.14.6)
$ npm install
added 54 packages from 45 contributors and audited 54 packages in 0.859s
found 0 vulnerabilities

Set our env vars and run:

builder@DESKTOP-JBA79RT:~/Workspaces/Check$ export COLLECTIONURL="https://dev.azure.com/princessking"
builder@DESKTOP-JBA79RT:~/Workspaces/Check$ export TOKEN=******************************
builder@DESKTOP-JBA79RT:~/Workspaces/Check$ export PORT=3000
builder@DESKTOP-JBA79RT:~/Workspaces/Check$ node app.js
Checklist checker listening on port 3000

Now when we create a PR, we can see the lack of a checklist fails the PR:

$ node app.js
Checklist checker listening on port 3000
{ id: 1,
  state: 3,
  description: 'Checklist',
  context:
   { name: 'checklist-checker', genre: 'continuous-integration' },
  creationDate: 2021-01-07T17:03:48.772Z,
  updatedDate: 2021-01-07T17:03:48.772Z,
  createdBy:
   { displayName: 'Isaac Johnson',
     url:
…..

However, a small issue.. When we edit...

It still shows failed. That’s because we need to update our Service Hook for _more_ than just the created event

Add a webhook for updated as well (to same URL)

Now when we edit, we see it re-evaluate to good:

Note: Updated events are not inclusive of created, so you would need both rules for PRs.

Enforcement

You’ll notice that the PR update happens asynchronously after the PR is created. This means one could “sneak” in a PR before the wip-checker indicated it was satisfactory.  To avoid this, we can use a PR policy to enforce compliance.

We can set it for main any repo in our project (Project policies):

To be satisfied by our hook, we need to match the genre/name

This requires wip-checker to indicate “succeeded” as a blocking gate on PRs for main.

Now when I create a PR in that project and indicate WIP:

I will see immediately that “wip-checker” was not run and it cannot merge yet:

In a moment, the webhook is invoked and we see the “pending” status (animated arrows):

If I edit the title and save:

In a moment the webhook runs again and the check is satisfied

Summary

In this demo we dug into a couple examples of nodejs based PR service connection servers. Using the vsts API we served up an http service and proved we could decorate PRs with it.  We then extended this to leverage PR policies to force compliance to external PR validations.

These patterns make it easy to extend the Pull Request capabilities to add custom checks to ensure compliance.  Next steps would be to containerize the service and expose it via an HTTP ingress.  We would also want to add some form of token to ensure the webhook access is restricted.