Feedback Forms to AzDO: Adding Email

Published: Jun 24, 2021 by Isaac Johnson

In our last post we created a feedback form that updated Azure DevOps with a new work item (feature) based on content from a static web form.

Next we would like to expand on that to include sending emails. We have a few options from Azure Logic Apps to Sendgrid to AWS SES. Let’s start with SES on this since we will be well within the free tier.

Setting up SES

If you have not already, verify a domain in SES. It was easy to do with one already hosted in Route53

Next. we need to verify an email address

which sends an email

Email and confirmation

Once verified, we can see that as a result

In testing, I was required to validate both my from and to addresses. I did some digging and initially we are fixed to a “sandbox” account, albeit that is shown oddly under metrics.

$ cat destination.json 
{
   "ToAddresses": ["isaac.johnson@gmail.com"],
   "CcAddresses": [],
   "BccAddresses": []
}

$ cat message.json 
{
   "Subject": {
       "Data": "Test email sent using SES",
       "Charset": "UTF-8"
   },
   "Body": {
       "Text": {
           "Data": "You are Great! Mostly....",
           "Charset": "UTF-8"
       },
       "Html": {
           "Data": "<H1>You are Great!</H1><p>Mostly...</p>",
           "Charset": "UTF-8"
       }
   }
}


$ aws ses send-email --from isaac@freshbrewed.science --destination file://destination.json --message file://message.json

An error occurred (MessageRejected) when calling the SendEmail operation: Email address is not verified. The following identities failed the check in region EU-CENTRAL-1: isaac@freshbrewed.science, isaac.johnson@gmail.com

For this test, I went ahead and verified the destination address

Once verified, that worked

$ aws ses send-email --from isaac@freshbrewed.science --destination file://destination.json --message file://message.json
{
    "MessageId": "0100017a3de1238c-1c58e528-50af-429b-b7dc-f521d3a70726-000000"
}

Result:

Secure SMTP

The other way we could go about it is with secure SMTP.

Go to SES and get your secure SMTP settings:

To use Secure SMTP, i tend to use s-nail:

With ubuntu/debian one can use

sudo apt-get update
sudo apt-get install -y s-nail || true

With mac, just use the brew formulae (gripe: Big Sur made homebrew update a lot so be patient)

brew install s-nail

On a container with ubuntu we can use snail :

root@my-shell:/# echo "TEsting" | s-nail -s "Welcome" -M "text/html" -S smtp=email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user=AKAAAAAAAAAAAAAAA -S smtp-auth-password=BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA -c isaac.johnson@gmail.com -r isaac@freshbrewed.science isaac.johnson@gmail.com

Or on the Mac

$ echo "TEsting" | s-nail -s "Welcome" -M "text/html" -S smtp=smtp://email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user=AAAAAAAAAAAAAAAAAAAAAA -S smtp-auth-password=BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA -c isaac.johnson@gmail.com -r isaac@freshbrewed.science isaac.johnson@gmail.com
s-nail: Warning: variable superseded or obsoleted: smtp
s-nail: Warning: variable superseded or obsoleted: ssl-verify
s-nail: Warning: variable superseded or obsoleted: smtp-auth-user
s-nail: Warning: variable superseded or obsoleted: smtp-auth-password
s-nail: Obsoletion warning: please do not use *smtp*, instead assign a smtp:// URL to *mta*!
s-nail: Obsoletion warning: Use of old-style credentials, which will vanish in v15!
s-nail: Please read the manual section "On URL syntax and credential lookup"

Both send emails

Updating the Pipeline to send mails

Lastly we can update our form processor to send emails.

We do this by adding the SMTP Credentials to our Group Vars library

Then we can add to the pipeline like this

- script: |
   sudo apt-get update
   sudo apt-get install -y s-nail || true
  
   set -x
   export WIID=`cat azresp.json | jq -r '.id' | tr -d '\n'`
   cat rawSummary |sed ':a;N;$!ba;s/\n/ /g' | sed 's/"/\\"/g' > emailSummary
  
   echo "<h1>New Feature Requested</h1><p>user $ has requested "`cat emailSummary`"</p><p>https://dev.azure.com/princessking/ghost-blog/_workitems/edit/$WIID/</p><br/><br/>Kind Regards,<br/>Isaac Johnson" | s-nail -s "Blog: Feature $WIID Requested" -M "text/html" -S smtp=email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user=$(SMTPAUTHUSER) -S smtp-auth-password=$(SMPTAUTHPASS) -c isaac.johnson@gmail.com -r isaac@freshbrewed.science isaac.johnson@gmail.com
 displayName: 'Email Me'

Moving out of SES sandbox

When you are ready, go to the “Sending Statistics” section in SES to move yourself out of sandbox.

We use “Edit your account details” to put in a request for Production use

Until we get approved, expect to see errors going to unverified emails:

2021-06-24T12:39:47.7129853Z + echo '<h1>New Feature Requested</h1><p>user isaac.johnson@mycompany.com has requested "`cat inputSummary`"</p><p>https://dev.azure.com/princessking/ghost-blog/_workitems/edit/$WIID/</p><br/><br/>Kind Regards,<br/>Isaac Johnson'
2021-06-24T12:39:47.7132219Z + s-nail -s 'Blog: Feature 109 Requested' -M text/html -S smtp=email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user= ***-S smtp-auth-password=*** -c isaac.johnson@gmail.com -r isaac@freshbrewed.science isaac.johnson@gmail.com
2021-06-24T12:39:48.4526656Z + echo '<h1>New Feature Requested</h1><p>user isaac.johnson@mycompany.com has requested "`cat inputSummary`"</p><p>https://dev.azure.com/princessking/ghost-blog/_workitems/edit/$WIID/</p><br/><br/>Kind Regards,<br/>Isaac Johnson'
2021-06-24T12:39:48.4529918Z + s-nail -s 'Blog: Feature 109 Requested' -M text/html -S smtp=email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user= ***-S smtp-auth-password=*** -c isaac.johnson@mycompany.com -r isaac@freshbrewed.science isaac.johnson@gmail.com
2021-06-24T12:39:49.0259121Z s-nail: SMTP server: 554 Message rejected: Email address is not verified. The following identities failed the check in region US-EAST-1: isaac.johnson@mycompany.com
2021-06-24T12:39:49.0583946Z /home/vsts/dead.letter 13/574
2021-06-24T12:39:49.0584784Z s-nail: ... message not sent

Once approved, you can see you are now in the Production System:

Getting approved actually required a few back and forth exchanges. I am guessing they are keen to stop spammers from abusing SES. So be thorough and verbose in your request verbiage.

The pipeline step added with the CC requestor commented out (for now)

- script: |
   sudo apt-get update
   sudo apt-get install -y s-nail || true
  
   set -x
   export WIID=`cat azresp.json | jq -r '.id' | tr -d '\n'`
   cat rawSummary |sed ':a;N;$!ba;s/\n/ /g' | sed 's/"/\\"/g' > emailSummary
  
   echo "<h1>New Feature Requested</h1><p>user $ has requested "`cat emailSummary`"</p><p>https://dev.azure.com/princessking/ghost-blog/_workitems/edit/$WIID/</p><br/><br/>Kind Regards,<br/>Isaac Johnson" | s-nail -s "Blog: Feature $WIID Requested" -M "text/html" -S smtp=email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user=$(SMTPAUTHUSER) -S smtp-auth-password=$(SMPTAUTHPASS) -c isaac.johnson@gmail.com -r isaac@freshbrewed.science isaac.johnson@gmail.com
   # may not work
   # echo "<h1>New Feature Requested</h1><p>user $ has requested "`cat emailSummary`"</p><p>https://dev.azure.com/princessking/ghost-blog/_workitems/edit/$WIID/</p><br/><br/>Kind Regards,<br/>Isaac Johnson" | s-nail -s "Blog: Feature $WIID Requested" -M "text/html" -S smtp=email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user=$(SMTPAUTHUSER) -S smtp-auth-password=$(SMPTAUTHPASS) -c $ -r isaac@freshbrewed.science isaac.johnson@gmail.com
 displayName: 'Email Me'

Verification

we can request a new feature from the webform

which triggers the pipeline

and we of course get an email now

Azure DevOps Dashboards

One more change would be to add some reports we can see on the Dashboard of our project.

From there we can add some Query results to the Project Dashboard

Now when we add a feature:

We see a nice results page in our Dashboard:

As well as email

and update to slack

Production SES Enabled

with production SES enabled, let’s circle back and CC users if they ask for it

- script: |
   sudo apt-get update
   sudo apt-get install -y s-nail || true
  
   set -x
   export WIID=`cat azresp.json | jq -r '.id' | tr -d '\n'`
   cat rawSummary |sed ':a;N;$!ba;s/\n/ /g' | sed 's/"/\\"/g' > emailSummary
  
   export USERTLD=`echo "$" | sed 's/^.*@//'`
 
   if [["$USERTLD" == "dontemailme.com"]]; then
      # do not CC user
      echo "<h1>New Feature Requested</h1><p>user $ has requested "`cat emailSummary`"</p><p>https://dev.azure.com/princessking/ghost-blog/_workitems/edit/$WIID/</p><br/><br/>Kind Regards,<br/>Isaac Johnson" | s-nail -s "Blog: Feature $WIID Requested" -M "text/html" -S smtp=email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user=$(SMTPAUTHUSER) -S smtp-auth-password=$(SMPTAUTHPASS) -c isaac.johnson@gmail.com -r isaac@freshbrewed.science isaac.johnson@gmail.com
   else
      # may not work
      echo "<h1>New Feature Requested</h1><p>user $ has requested "`cat emailSummary`"</p><p>https://dev.azure.com/princessking/ghost-blog/_workitems/edit/$WIID/</p><br/><br/>Kind Regards,<br/>Isaac Johnson" | s-nail -s "Blog: Feature $WIID Requested" -M "text/html" -S smtp=email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user=$(SMTPAUTHUSER) -S smtp-auth-password=$(SMPTAUTHPASS) -c $ -r isaac@freshbrewed.science isaac.johnson@gmail.com
   fi
 displayName: 'Email Me'

I’ll try using my corporate email

Which was delivered to my corp email

Azure Logic Apps

Let’s do the same thing but this time with Azure Logic Apps.

Go to the portal and create a Logic App. I tend to use Consumption plan.

Next we give it a name and Resource Group

Then create it

Once created we need to design a basic emailer. I generally start with a webhook trigger

If you click “use a sample payload” it can be easy to create that first step

For instance, we could use a payload such as:

{
    "requestorEmail": "anonymous@dontemailme.com",
    "emailSummary": "make a feature",
    "workitemID": "12345",
    "workitemURL": "https://dev.azure.com/princessking/ghost-blog/_workitems/edit/12345/",
    "desitinationEmail": "isaac.johnson@gmail.com"
}

which when we click done creates

{
    "type": "object",
    "properties": {
        "requestorEmail": {
            "type": "string"
        },
        "emailSummary": {
            "type": "string"
        },
        "workitemID": {
            "type": "string"
        },
        "workitemURL": {
            "type": "string"
        },
        "desitinationEmail": {
            "type": "string"
        }
    }
}

Next largely depends on if you are using sendgrid or an o365 connection.

Sendgrid

Sendgrid will need an API key

Here we can use the dynamic content and fill in the fields

Before we go further, recall that we want to CC the user if it’s not a “dontemailme” email address.

We can add a condition to check on this:

When done we should have a condition that forks logic with the only difference being that if they specified a real email address, we will CC it

Once we save the Logic App, we can go to the first step to get a URL for our Logic App

Validation

We can use curl in our shell to test

$ cat valuesLA.json 
{
    "requestorEmail": "anonymous@dontemailme.com",
    "emailSummary": "make a feature",
    "workitemID": "12345",
    "workitemURL": "https://dev.azure.com/princessking/ghost-blog/_workitems/edit/12345/",
    "desitinationEmail": "isaac.johnson@gmail.com"
}

$ curl -v -X POST -d @valuesLA.json -H "Content-type: application/json" "https://prod-34.eastus.logic.azure.com:443/workflows/851f197339fa4f09b1517ebf712efdcc/triggers/manual/paths/invoke?api-version=2016-10-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=uSUo-SNU_TZqorFctkDEVaiA0fCo92HFS-XLAR3N5V8"
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 137.135.106.54...
* TCP_NODELAY set
* Connected to prod-34.eastus.logic.azure.com (137.135.106.54) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=eastus.logic.azure.com
* start date: Apr 14 21:57:31 2021 GMT
* expire date: Apr 9 21:57:31 2022 GMT
* subjectAltName: host "prod-34.eastus.logic.azure.com" matched cert's "*.eastus.logic.azure.com"
* issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure TLS Issuing CA 02
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7faa8180d600)
> POST /workflows/851f197339fa4f09b1517ebf712efdcc/triggers/manual/paths/invoke?api-version=2016-10-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=uSUo-SNU_TZqorFctkDEVaiA0fCo92HFS-XLAR3N5V8 HTTP/2
> Host: prod-34.eastus.logic.azure.com
> User-Agent: curl/7.64.1
> Accept: */*
> Content-type: application/json
> Content-Length: 255
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
* We are completely uploaded and fine
< HTTP/2 202 
< cache-control: no-cache
< pragma: no-cache
< expires: -1
< x-ms-workflow-run-id: 08585766400244707146123513754CU79
< x-ms-correlation-id: 6e044971-7197-4420-b651-be08fae9b7ec
< x-ms-client-tracking-id: 08585766400244707146123513754CU79
< x-ms-trigger-history-name: 08585766400244707146123513754CU79
< x-ms-execution-location: eastus
< x-ms-workflow-system-id: /locations/eastus/scaleunits/prod-34/workflows/851f197339fa4f09b1517ebf712efdcc
< x-ms-workflow-id: 851f197339fa4f09b1517ebf712efdcc
< x-ms-workflow-version: 08585766402382171437
< x-ms-workflow-name: FeatureRequestedEmail
< x-ms-tracking-id: 6e044971-7197-4420-b651-be08fae9b7ec
< x-ms-ratelimit-burst-remaining-workflow-writes: 4999
< x-ms-ratelimit-remaining-workflow-download-contentsize: 357913600
< x-ms-ratelimit-remaining-workflow-upload-contentsize: 357913345
< x-ms-ratelimit-time-remaining-directapirequests: 33333257
< x-ms-request-id: eastus:6e044971-7197-4420-b651-be08fae9b7ec
< strict-transport-security: max-age=31536000; includeSubDomains
< date: Tue, 29 Jun 2021 11:41:00 GMT
< content-length: 0
< 
* Connection #0 to host prod-34.eastus.logic.azure.com left intact
* Closing connection 0

_ Note : if you get an error like_

< 
* Connection #0 to host prod-34.eastus.logic.azure.com left intact
{"error":{"code":"DirectApiAuthorizationRequired","message":"The request must be authenticated only by Shared Access scheme."}}* Closing connection 0

Chances are you forgot to put the URL in double quotes (i do this all the time)

I almost immediately got an email:

Office 365 / Outlook

If you instead want to use O365 instead of Sendgrid, that is just as easy.

We use the “Office 365 Outlook” task

These steps fill out just the same as Sendgrid albeit you cannot specify the “from” email address (Outlook determines this for you)

While i’ll avoid displaying my corp email address, you can see O365 spewed in the standard boilerplate legal footer

Adding Logic App Invokation to Pipeline

To add to our azure pipeline is even easier than the SES step as we can assume the Azure Logic App will do the logic on CCing

...snip...
   cat >valuesLA.json <<EOLA
   {
       "requestorEmail": "$",
       "emailSummary": "`cat emailSummary`",
       "workitemID": "$WIID",
       "workitemURL": "https://dev.azure.com/princessking/ghost-blog/_workitems/edit/$WIID/",
       "desitinationEmail": "isaac.johnson@gmail.com"
   }
   EOLA
 
   curl -v -X POST -d @valuesLA.json -H "Content-type: application/json" "https://prod-34.eastus.logic.azure.com:443/workflows/851f197339fa4f09b1517ebf712efdcc/triggers/manual/paths/invoke?api-version=2016-10-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=uSUo-SNU_TZqorFctkDEVaiA0fCo92HFS-XLAR3N5V8"
  
 displayName: 'Email Me'

Validation

Create a feedback ticket

which then on submit confirms

and we see the feedback form pipeline kick in

and completes

And i can see the Logic App results

O365 :

Sendgrid :

and AWS SES :

In the Azure portal we can see the runs of this app:

And because I signed in as myself, I, interestingly enough, see the emails in my Sent box of Outlook:

I have found Logic Apps can add up costwise over time so I generally only use them when I want to leverage the Microsoft Graph to pull in user details from Exchange.

For instance, pulling in the users First and Last name when i just have their userid (which i pulled from git or AzDO)

Note, you can regenerate the SIG (key) for the logic app from the portal, say, you posted a blog with the real URL

Which now if using the posted URL would error

< 
* Connection #0 to host prod-34.eastus.logic.azure.com left intact
{"error":{"code":"AuthorizationFailed","message":"The authentication credentials are not valid."}}* Closing connection 0

Lastly for cleanup, i commented out the CURL line in my pipeline yaml (as i’m fine with SES).

Summary

It was easy to extend our pipeline to send emails out using secure SMTP. We showed using the AWS command line client and secure SMTP via s-nail. Additionally we covered how to apply for production SES access and update our Azure DevOps Dashboard for improved visibility.

We then looked into Azure Logic Apps using both Sendgrid and O365 to send form email and implemented conditional logic in the Logic App itself. Our final tests included creating a form driven feature request that emailed us three ways: SES, Azure Logic App to Sendgrid and Azure Logic App to Office 365.

While we have thoroughly covered multiple ways to send out emails from pipelines for user feature request ingestion, we might want to start to think next about how we might use SES to ingest emails. What if the requestor has more details or feedback? Could we automate a system that processes emails with SES and Dapr in a cluster? We’ll tackle that next week as we dive into SES, SNS, SQS, Dapr and Mime parsing.