In our last post we created a feedback form that updated Azure DevOps with a new work item (feature) based on content from a static web form.
Next we would like to expand on that to include sending emails. We have a few options from Azure Logic Apps to Sendgrid to AWS SES. Let's start with SES on this since we will be well within the free tier.
Setting up SES
If you have not already, verify a domain in SES. It was easy to do with one already hosted in Route53
Next. we need to verify an email address
which sends an email
Email and confirmation
Once verified, we can see that as a result
In testing, I was required to validate both my from and to addresses. I did some digging and initially we are fixed to a "sandbox" account, albeit that is shown oddly under metrics.
$ cat destination.json
{
"ToAddresses": ["isaac.johnson@gmail.com"],
"CcAddresses": [],
"BccAddresses": []
}
$ cat message.json
{
"Subject": {
"Data": "Test email sent using SES",
"Charset": "UTF-8"
},
"Body": {
"Text": {
"Data": "You are Great! Mostly....",
"Charset": "UTF-8"
},
"Html": {
"Data": "<H1>You are Great!</H1><p>Mostly...</p>",
"Charset": "UTF-8"
}
}
}
$ aws ses send-email --from isaac@freshbrewed.science --destination file://destination.json --message file://message.json
An error occurred (MessageRejected) when calling the SendEmail operation: Email address is not verified. The following identities failed the check in region EU-CENTRAL-1: isaac@freshbrewed.science, isaac.johnson@gmail.com
For this test, I went ahead and verified the destination address
Once verified, that worked
$ aws ses send-email --from isaac@freshbrewed.science --destination file://destination.json --message file://message.json
{
"MessageId": "0100017a3de1238c-1c58e528-50af-429b-b7dc-f521d3a70726-000000"
}
Result:
Secure SMTP
The other way we could go about it is with secure SMTP.
Go to SES and get your secure SMTP settings:
To use Secure SMTP, i tend to use s-nail:
With ubuntu/debian one can use
sudo apt-get update
sudo apt-get install -y s-nail || true
With mac, just use the brew formulae (gripe: Big Sur made homebrew update a lot so be patient)
brew install s-nailOn a container with ubuntu we can use snail :
root@my-shell:/# echo "TEsting" | s-nail -s "Welcome" -M "text/html" -S smtp=email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user=AKAAAAAAAAAAAAAAA -S smtp-auth-password=BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA -c isaac.johnson@gmail.com -r isaac@freshbrewed.science isaac.johnson@gmail.comOr on the Mac
$ echo "TEsting" | s-nail -s "Welcome" -M "text/html" -S smtp=smtp://email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user=AAAAAAAAAAAAAAAAAAAAAA -S smtp-auth-password=BAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA -c isaac.johnson@gmail.com -r isaac@freshbrewed.science isaac.johnson@gmail.com
s-nail: Warning: variable superseded or obsoleted: smtp
s-nail: Warning: variable superseded or obsoleted: ssl-verify
s-nail: Warning: variable superseded or obsoleted: smtp-auth-user
s-nail: Warning: variable superseded or obsoleted: smtp-auth-password
s-nail: Obsoletion warning: please do not use *smtp*, instead assign a smtp:// URL to *mta*!
s-nail: Obsoletion warning: Use of old-style credentials, which will vanish in v15!
s-nail: Please read the manual section "On URL syntax and credential lookup"
Both send emails
Updating the Pipeline to send mails
Lastly we can update our form processor to send emails.
We do this by adding the SMTP Credentials to our Group Vars library
Then we can add to the pipeline like this
- script: |
sudo apt-get update
sudo apt-get install -y s-nail || true
set -x
export WIID=`cat azresp.json | jq -r '.id' | tr -d '\n'`
cat rawSummary |sed ':a;N;$!ba;s/\n/ /g' | sed 's/"/\\"/g' > emailSummary
echo "<h1>New Feature Requested</h1><p>user ${{ parameters.feedbackForm.userId }} has requested "`cat emailSummary`"</p><p>https://dev.azure.com/princessking/ghost-blog/_workitems/edit/$WIID/</p><br/><br/>Kind Regards,<br/>Isaac Johnson" | s-nail -s "Blog: Feature $WIID Requested" -M "text/html" -S smtp=email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user=$(SMTPAUTHUSER) -S smtp-auth-password=$(SMPTAUTHPASS) -c isaac.johnson@gmail.com -r isaac@freshbrewed.science isaac.johnson@gmail.com
displayName: 'Email Me'Moving out of SES sandbox
When you are ready, go to the "Sending Statistics" section in SES to move yourself out of sandbox.
We use "Edit your account details" to put in a request for Production use
Until we get approved, expect to see errors going to unverified emails:
2021-06-24T12:39:47.7129853Z + echo '<h1>New Feature Requested</h1><p>user isaac.johnson@mycompany.com has requested "`cat inputSummary`"</p><p>https://dev.azure.com/princessking/ghost-blog/_workitems/edit/$WIID/</p><br/><br/>Kind Regards,<br/>Isaac Johnson'
2021-06-24T12:39:47.7132219Z + s-nail -s 'Blog: Feature 109 Requested' -M text/html -S smtp=email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user=*** -S smtp-auth-password=*** -c isaac.johnson@gmail.com -r isaac@freshbrewed.science isaac.johnson@gmail.com
2021-06-24T12:39:48.4526656Z + echo '<h1>New Feature Requested</h1><p>user isaac.johnson@mycompany.com has requested "`cat inputSummary`"</p><p>https://dev.azure.com/princessking/ghost-blog/_workitems/edit/$WIID/</p><br/><br/>Kind Regards,<br/>Isaac Johnson'
2021-06-24T12:39:48.4529918Z + s-nail -s 'Blog: Feature 109 Requested' -M text/html -S smtp=email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user=*** -S smtp-auth-password=*** -c isaac.johnson@mycompany.com -r isaac@freshbrewed.science isaac.johnson@gmail.com
2021-06-24T12:39:49.0259121Z s-nail: SMTP server: 554 Message rejected: Email address is not verified. The following identities failed the check in region US-EAST-1: isaac.johnson@mycompany.com
2021-06-24T12:39:49.0583946Z /home/vsts/dead.letter 13/574
2021-06-24T12:39:49.0584784Z s-nail: ... message not sent
Once approved, you can see you are now in the Production System:
Getting approved actually required a few back and forth exchanges. I am guessing they are keen to stop spammers from abusing SES. So be thorough and verbose in your request verbiage.
The pipeline step added with the CC requestor commented out (for now)
- script: |
sudo apt-get update
sudo apt-get install -y s-nail || true
set -x
export WIID=`cat azresp.json | jq -r '.id' | tr -d '\n'`
cat rawSummary |sed ':a;N;$!ba;s/\n/ /g' | sed 's/"/\\"/g' > emailSummary
echo "<h1>New Feature Requested</h1><p>user ${{ parameters.feedbackForm.userId }} has requested "`cat emailSummary`"</p><p>https://dev.azure.com/princessking/ghost-blog/_workitems/edit/$WIID/</p><br/><br/>Kind Regards,<br/>Isaac Johnson" | s-nail -s "Blog: Feature $WIID Requested" -M "text/html" -S smtp=email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user=$(SMTPAUTHUSER) -S smtp-auth-password=$(SMPTAUTHPASS) -c isaac.johnson@gmail.com -r isaac@freshbrewed.science isaac.johnson@gmail.com
# may not work
# echo "<h1>New Feature Requested</h1><p>user ${{ parameters.feedbackForm.userId }} has requested "`cat emailSummary`"</p><p>https://dev.azure.com/princessking/ghost-blog/_workitems/edit/$WIID/</p><br/><br/>Kind Regards,<br/>Isaac Johnson" | s-nail -s "Blog: Feature $WIID Requested" -M "text/html" -S smtp=email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user=$(SMTPAUTHUSER) -S smtp-auth-password=$(SMPTAUTHPASS) -c ${{ parameters.feedbackForm.userId }} -r isaac@freshbrewed.science isaac.johnson@gmail.com
displayName: 'Email Me'Verification
we can request a new feature from the webform
which triggers the pipeline
and we of course get an email now
Azure DevOps Dashboards
One more change would be to add some reports we can see on the Dashboard of our project.
From there we can add some Query results to the Project Dashboard
Now when we add a feature:
We see a nice results page in our Dashboard:
As well as email
and update to slack
Production SES Enabled
with production SES enabled, let's circle back and CC users if they ask for it
- script: |
sudo apt-get update
sudo apt-get install -y s-nail || true
set -x
export WIID=`cat azresp.json | jq -r '.id' | tr -d '\n'`
cat rawSummary |sed ':a;N;$!ba;s/\n/ /g' | sed 's/"/\\"/g' > emailSummary
export USERTLD=`echo "${{ parameters.feedbackForm.userId }}" | sed 's/^.*@//'`
if [[ "$USERTLD" == "dontemailme.com" ]]; then
# do not CC user
echo "<h1>New Feature Requested</h1><p>user ${{ parameters.feedbackForm.userId }} has requested "`cat emailSummary`"</p><p>https://dev.azure.com/princessking/ghost-blog/_workitems/edit/$WIID/</p><br/><br/>Kind Regards,<br/>Isaac Johnson" | s-nail -s "Blog: Feature $WIID Requested" -M "text/html" -S smtp=email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user=$(SMTPAUTHUSER) -S smtp-auth-password=$(SMPTAUTHPASS) -c isaac.johnson@gmail.com -r isaac@freshbrewed.science isaac.johnson@gmail.com
else
# may not work
echo "<h1>New Feature Requested</h1><p>user ${{ parameters.feedbackForm.userId }} has requested "`cat emailSummary`"</p><p>https://dev.azure.com/princessking/ghost-blog/_workitems/edit/$WIID/</p><br/><br/>Kind Regards,<br/>Isaac Johnson" | s-nail -s "Blog: Feature $WIID Requested" -M "text/html" -S smtp=email-smtp.us-east-1.amazonaws.com:587 -S smtp-use-starttls -S ssl-verify=ignore -S smtp-auth=login -S smtp-auth-user=$(SMTPAUTHUSER) -S smtp-auth-password=$(SMPTAUTHPASS) -c ${{ parameters.feedbackForm.userId }} -r isaac@freshbrewed.science isaac.johnson@gmail.com
fi
displayName: 'Email Me'I'll try using my corporate email
Which was delivered to my corp email
Azure Logic Apps
Let's do the same thing but this time with Azure Logic Apps.
Go to the portal and create a Logic App. I tend to use Consumption plan.
Next we give it a name and Resource Group
Then create it
Once created we need to design a basic emailer. I generally start with a webhook trigger
If you click "use a sample payload" it can be easy to create that first step
For instance, we could use a payload such as:
{
"requestorEmail": "anonymous@dontemailme.com",
"emailSummary": "make a feature",
"workitemID": "12345",
"workitemURL": "https://dev.azure.com/princessking/ghost-blog/_workitems/edit/12345/",
"desitinationEmail": "isaac.johnson@gmail.com"
}
which when we click done creates
{
"type": "object",
"properties": {
"requestorEmail": {
"type": "string"
},
"emailSummary": {
"type": "string"
},
"workitemID": {
"type": "string"
},
"workitemURL": {
"type": "string"
},
"desitinationEmail": {
"type": "string"
}
}
}
Next largely depends on if you are using sendgrid or an o365 connection.
Sendgrid
Sendgrid will need an API key
Here we can use the dynamic content and fill in the fields
Before we go further, recall that we want to CC the user if it's not a "dontemailme" email address.
We can add a condition to check on this:
When done we should have a condition that forks logic with the only difference being that if they specified a real email address, we will CC it
Once we save the Logic App, we can go to the first step to get a URL for our Logic App
Validation
We can use curl in our shell to test
$ cat valuesLA.json
{
"requestorEmail": "anonymous@dontemailme.com",
"emailSummary": "make a feature",
"workitemID": "12345",
"workitemURL": "https://dev.azure.com/princessking/ghost-blog/_workitems/edit/12345/",
"desitinationEmail": "isaac.johnson@gmail.com"
}
$ curl -v -X POST -d @valuesLA.json -H "Content-type: application/json" "https://prod-34.eastus.logic.azure.com:443/workflows/851f197339fa4f09b1517ebf712efdcc/triggers/manual/paths/invoke?api-version=2016-10-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=uSUo-SNU_TZqorFctkDEVaiA0fCo92HFS-XLAR3N5V8"
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying 137.135.106.54...
* TCP_NODELAY set
* Connected to prod-34.eastus.logic.azure.com (137.135.106.54) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=eastus.logic.azure.com
* start date: Apr 14 21:57:31 2021 GMT
* expire date: Apr 9 21:57:31 2022 GMT
* subjectAltName: host "prod-34.eastus.logic.azure.com" matched cert's "*.eastus.logic.azure.com"
* issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure TLS Issuing CA 02
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7faa8180d600)
> POST /workflows/851f197339fa4f09b1517ebf712efdcc/triggers/manual/paths/invoke?api-version=2016-10-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=uSUo-SNU_TZqorFctkDEVaiA0fCo92HFS-XLAR3N5V8 HTTP/2
> Host: prod-34.eastus.logic.azure.com
> User-Agent: curl/7.64.1
> Accept: */*
> Content-type: application/json
> Content-Length: 255
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
* We are completely uploaded and fine
< HTTP/2 202
< cache-control: no-cache
< pragma: no-cache
< expires: -1
< x-ms-workflow-run-id: 08585766400244707146123513754CU79
< x-ms-correlation-id: 6e044971-7197-4420-b651-be08fae9b7ec
< x-ms-client-tracking-id: 08585766400244707146123513754CU79
< x-ms-trigger-history-name: 08585766400244707146123513754CU79
< x-ms-execution-location: eastus
< x-ms-workflow-system-id: /locations/eastus/scaleunits/prod-34/workflows/851f197339fa4f09b1517ebf712efdcc
< x-ms-workflow-id: 851f197339fa4f09b1517ebf712efdcc
< x-ms-workflow-version: 08585766402382171437
< x-ms-workflow-name: FeatureRequestedEmail
< x-ms-tracking-id: 6e044971-7197-4420-b651-be08fae9b7ec
< x-ms-ratelimit-burst-remaining-workflow-writes: 4999
< x-ms-ratelimit-remaining-workflow-download-contentsize: 357913600
< x-ms-ratelimit-remaining-workflow-upload-contentsize: 357913345
< x-ms-ratelimit-time-remaining-directapirequests: 33333257
< x-ms-request-id: eastus:6e044971-7197-4420-b651-be08fae9b7ec
< strict-transport-security: max-age=31536000; includeSubDomains
< date: Tue, 29 Jun 2021 11:41:00 GMT
< content-length: 0
<
* Connection #0 to host prod-34.eastus.logic.azure.com left intact
* Closing connection 0
Note: if you get an error like
<
* Connection #0 to host prod-34.eastus.logic.azure.com left intact
{"error":{"code":"DirectApiAuthorizationRequired","message":"The request must be authenticated only by Shared Access scheme."}}* Closing connection 0
Chances are you forgot to put the URL in double quotes (i do this all the time)
I almost immediately got an email:
Office 365 / Outlook
If you instead want to use O365 instead of Sendgrid, that is just as easy.
We use the "Office 365 Outlook" task
These steps fill out just the same as Sendgrid albeit you cannot specify the "from" email address (Outlook determines this for you)
While i'll avoid displaying my corp email address, you can see O365 spewed in the standard boilerplate legal footer
Adding Logic App Invokation to Pipeline
To add to our azure pipeline is even easier than the SES step as we can assume the Azure Logic App will do the logic on CCing
...snip...
cat >valuesLA.json <<EOLA
{
"requestorEmail": "${{ parameters.feedbackForm.userId }}",
"emailSummary": "`cat emailSummary`",
"workitemID": "$WIID",
"workitemURL": "https://dev.azure.com/princessking/ghost-blog/_workitems/edit/$WIID/",
"desitinationEmail": "isaac.johnson@gmail.com"
}
EOLA
curl -v -X POST -d @valuesLA.json -H "Content-type: application/json" "https://prod-34.eastus.logic.azure.com:443/workflows/851f197339fa4f09b1517ebf712efdcc/triggers/manual/paths/invoke?api-version=2016-10-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=uSUo-SNU_TZqorFctkDEVaiA0fCo92HFS-XLAR3N5V8"
displayName: 'Email Me'Validation
Create a feedback ticket
which then on submit confirms
and we see the feedback form pipeline kick in
and completes
And i can see the Logic App results
O365:
Sendgrid:
and AWS SES:
In the Azure portal we can see the runs of this app:
And because I signed in as myself, I, interestingly enough, see the emails in my Sent box of Outlook:
I have found Logic Apps can add up costwise over time so I generally only use them when I want to leverage the Microsoft Graph to pull in user details from Exchange.
For instance, pulling in the users First and Last name when i just have their userid (which i pulled from git or AzDO)
Note, you can regenerate the SIG (key) for the logic app from the portal, say, you posted a blog with the real URL
Which now if using the posted URL would error
<
* Connection #0 to host prod-34.eastus.logic.azure.com left intact
{"error":{"code":"AuthorizationFailed","message":"The authentication credentials are not valid."}}* Closing connection 0
Lastly for cleanup, i commented out the CURL line in my pipeline yaml (as i'm fine with SES).
Summary
It was easy to extend our pipeline to send emails out using secure SMTP. We showed using the AWS command line client and secure SMTP via s-nail. Additionally we covered how to apply for production SES access and update our Azure DevOps Dashboard for improved visibility.
We then looked into Azure Logic Apps using both Sendgrid and O365 to send form email and implemented conditional logic in the Logic App itself. Our final tests included creating a form driven feature request that emailed us three ways: SES, Azure Logic App to Sendgrid and Azure Logic App to Office 365.
While we have thoroughly covered multiple ways to send out emails from pipelines for user feature request ingestion, we might want to start to think next about how we might use SES to ingest emails. What if the requestor has more details or feedback? Could we automate a system that processes emails with SES and Dapr in a cluster? We'll tackle that next week as we dive into SES, SNS, SQS, Dapr and Mime parsing.
