Setting up a WebJob to run HDInsight jobs


Managing an HDInsight cluster, or running an HDInsight job, from an Azure WebJob, requires you to set up a certificate to access the HDInsight cluster. This post shows how to upload the certificate to the Azure management portal, and how to configure our WebJob.

To generate the certificate file we need, you can follow the steps for the .pfx certificate file generation here.

Uploading the certificate file

The .pfx file should be uploaded in the “Configure” section in your Web App. To do so, follow these steps:

  1. Open the Azure management portal (
  2. Go to the Web Apps section on the left menu.
    Web Apps
  3. Select your Web App from the list.
  4. Go to the Configure tab.
    Web App configure
  5. Scroll to the “certificates” area.
    If your Web App is in the Free or Shared tier, you will need to scale it.
    Scale to Basic or Standard
    To do so, follow these steps:

    1. Click on “Scale your app now” (or go to the “Scale” tab).
    2. Select “Basic” or “Standard”.
      Scale web app
    3. Click Save on the bottom bar.
    4. Go back to the “certificates” section in the “Configure” tab.
  6. Click “upload a certificate”.
    Upload a certificate
  7. Browse for the .pfx certificate file and enter the password.
    Upload a certificate
  8. Click OK and wait for the certificate to be available, it will appear in the list when it’s ready.

Adding the app settings

In the “Configure” tab, scroll to the “app settings” section:

App settings

Add the setting WEBSITE_LOAD_CERTIFICATES, with the thumbprint of your certificate as the value.


Save the changes:

Save configuration

You can put multiple certificate thumbprints separated by comma, or use * to load all certificates.

This will install the certificate/s in the Personal certificate store of the ApplicationPool Identity of the worker process.

Using the certificate from your code

Here is a helper function to find a certificate from the store:

public X509Certificate2 FindCertificate(StoreLocation storeLocation, StoreName storeName, X509FindType findType, object searchCriteria)
X509Store certificateStore = new X509Store(storeName, storeLocation);
X509Certificate2Collection certificates = certificateStore.Certificates;
X509Certificate2Collection matchingCertificates = certificates.Find(findType, searchCriteria, false);
if (matchingCertificates != null && matchingCertificates.Count > 0)
return matchingCertificates[0];
throw new ArgumentException("Unable to find a matching certificate in the certificate store. Please modify the search criteria.");

To get the certificate we configured in the previous steps, we’ll need to call the function with these parameters:

  • storeLocation: StoreLocation.CurrentUser
  • storeName: StoreName.My
  • findType: X509FindType.FindByThumbprint
  • searchCriteria: your certificate thumbprint

And that’s it!!!

Happy coding!



Leave a comment

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s