I have specified parameters in some pipelines in Azure Synapse Analytics, like this:
The value of the variables is different in Production and Pre-Production’s environments. During the release in Azure DevOps I want default values of the parameters to stay as they are in Pre-Production but be updated in Production’s environment.
In Azure DevOps > Pipelines > Releases > Deployment process > Agent jobs > Synpase deployment task for workspace > Override Parameters, I have added needed parameters together with $workspace parameter, it looks like:
Questions:
- If I specify parameters, which have to be overridden this way they are not being updated after the release is finished. How do I have to specify the parameters (from the pipelines in Azure Synapse Analytics) then?
- Is there a safer way how to specify storage key inside the pipeline and how to update it during the release (instead of specifying it so explicitly)?
An update after the comment:
TemplateForWorkspace.json
{
"$schema": "http://schema.management.azure.com/..",
"contentVersion": "1.0.0.0",
"parameters": {
"workspaceName": {
.......
}
"resources": {
.......
"name": "PL_example",
"type": "Microsoft.Synapse/workspaces/pipelines",
"apiVersion": ".",
"properties": {
"description": ".",
"activities": [
{
"name": "Notebook",
"type": "SynapseNotebook",
"dependsOn": [..],
"policy": {..},
"userProperties": [],
"typeProperties": {
"notebook": {
"referenceName": "Notebook",
"type": "NotebookReference"
},
"parameters": {
"url": {
"value": {
"value": "@pipeline().parameters.URL",
"type": "Expression"
}
......
"parameters": {
"URL": {
"type": "string",
"defaultValue": "jdbc:sqlserver://..."
},
}
In the end of the script it is stated as default value, which has to be updated after the release with the value in override parameters. but it stays the same value.
I think that after the release the parameter should be added to the beginning of TemplateForWorkspace.json script in the section parameters, but it does not, how do I do it? Am I doing something wrong?
Taking into account that I have added correct values (for the production, which should be updated after the release for the necessary parameters in following locations:
Azure DevOps>Pipelines>Releases>Deployment process>Tasks>Agent jobs>Synpase deployment task for workspace>Override ParametersAzure DevOps>Pipelines>Releases>Deployment process>Variables>Pipeline variables
3
To better align this post with the topic, it is recommended to include the azure-synapse tag, as the primary focus is on how to create custom parameters in the workspace template for parameterizing Synapse Pipeline features and settings that are not covered by the default deployment parameters template.
For more detailed guidance, please refer to this resource on Azure Synapse Analytics CI/CD with Custom Parameters – Made Easy! – Microsoft Community Hub.
In general, to expose Synapse pipeline parameters in the collaboration branch (main), you need to create a template-parameters-definition.json file that specifies the Synapse pipeline parameters. Once published, the new Synapse pipeline parameters should appear in the TemplateForWorkspace.json template within the publish branch (workspace_publish).
Here is an example of the template-parameters-definition.json file:
{
"Microsoft.Synapse/workspaces/pipelines": {
"properties": {
"parameters": {
"URL": {
"defaultValue": "="
},
"BlobStorageKey": {
"defaultValue": "="
},
"Server": {
"defaultValue": "="
},
"TempFolder": {
"defaultValue": "="
}
}
}
}
}
In the release pipeline, ensure to replace the ARM template parameters with the corresponding pipeline variables’ values, rather than directly using the Synapse pipeline parameter names. These pipeline variables can also be configured as secret variables.
Images illustrating these steps are provided below.
4