Introduction

At this point, you should already be familiar with the basics of Biml and BimlScript code. Both give you the ability to create new objects from scratch. For example, you might create a package to stage each of the source tables in your solution, according to some metadata and business logic that you’ve specified. However, Biml and BimlScript alone do not provide the capability to modify existing object, whether those be created through Biml code or imported from existing projects. This is where BimlScript transformers come in.

BimlScript Transformers are exactly like regular BimlScript files with 2 key differences:

1. Transformers include a <#@ target … #> directive that provides additional information about what types of existing objects to transform and how to perform the transformation.
2. The Biml portion of the Transformer does not start with the <Biml> tag but will instead use a tag compatible with the targeted object type from the previous bullet.

The Target Directive

Directives are special tags that are included, usually at the top, of your Biml file. They provide special handling instructions for the BimlScript. The target directive is used to provide all information the compiler needs to properly execute the transformer. It offers 3 property settings to developers to customize the behavior of the transformer

Target Types

The most important aspect of the target directive is the target type. It specifies the target objects that are eligible for modification by the given transformer. Target types can either be specified using the name of their underlying .NET type or by using their Biml friendly names. Both of these are covered in the language documentation, but in general, Mist intelliprompt is the fastest way to find the right label for the type you wish to target.

Exact Matches

By default, the transformer will support the objects of the target type and any objects of a type that derives/inherits from the target type. This is usually the desired behavior, but sometimes you will have logic that is specially tailored to an individual type (perhaps with special handling for derived types in a separate transformer.) In these cases, you can force the transformer to only work with the exact type specified in the target type by setting exactMatch = "True".

Merge Modes

RootAdd

The RootAdd merge mode is used to add a new root item whenever an object of a certain target type is encountered. While this merge mode is extensively used within the compiler itself for code generation, it will be seldom used by Biml developers. The most common scenario for RootAdd transformers would be to replace a foreach loop that iterates over one object type to produce another in live script.
Consider the following live script that creates an empty package for each table in the project:

<Biml xmlns="http://schemas.varigence.com/biml.xsd">
    <Packages>
    <# foreach (var table in RootNode.Tables) { #>
        <Package Name="<#=table.Name#>" ConstraintMode="Parallel" />
    <# } #>
    </Packages>
</Biml>

This live script can be replaced by the following transformer that targets table objects with a RootAdd transformer that adds the same empty package:

<#@ target type="Table" mergemode="RootAdd" #>
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
    <Packages>
        <Package Name="<#=TargetNode.Name#>" ConstraintMode="Parallel" />
    </Packages>
</Biml> 

LocalMerge

The LocalMerge merge mode is probably the most useful and commonly used merge mode for Biml developers. In most cases, you don’t want to completely rewrite an object – requiring all of the basic properties to be re-specified to make an otherwise minor change. Instead, it would be useful to leave all of the object properties exactly as they were before the transformer was run, and specify only those things that you’d like to see changed. This is the purpose of the LocalMerge merge mode. It merges changes into an existing object. A very simple LocalMerge transformer that changes the data type of table columns is listed below:

<#@ target type="TableColumn" mergemode="LocalMerge" #>
<Column DataType="Int64" />

Unfortunately, it’s not terribly useful to blindly modify the types of all columns in all tables. It would be useful to perhaps only change our ID columns to Int64. So we would like to add a check that the column name ends with ID and the data type is presently Int32 before we run the transformation. The following listing adds this logic. If the check is satisfied, then the column is transformed. Otherwise, the column is left unchanged. This is a very powerful pattern that you will use extensively in your transformers.

<#@ target type="TableColumn" mergemode="LocalMerge" #>
<# if (TargetNode.Name.EndsWith("ID") && TargetNode.DataType == System.Data.DbType.Int32) { #>
<Column DataType="Int64" />
<# } #>

LocalMerge transformers are not limited to changing property values. They can also add new items to child collections of the target object. This is useful for a variety of activities, such as the addition of project variables, table columns, and bookkeeping control flow tasks. Below is a LocalMerge transformer that adds logging variables to target objects of type Package:

<#@ target type="Package" mergemode="LocalMerge" #>
<Package>
    <Variables>
        <Variable Name="LogStart" DataType="DateTime">1/1/1900</Variable>
        <Variable Name="LogLabel" DataType="String"></Variable>
    </Variables>
</Package>

LocalMergeAndTypeReplace

The LocalMergeAndTypeReplace merge mode is most useful for developers who are implementing extensions to the Biml language. Within some frameworks, there are common patterns comprised of many standard operations that would ideally be combined into a single logical operation. This is normally the role of the LocalMergeAndTypeReplace merge mode. The developer creates a new language tag for an abstract type – let’s say a dataflow component.

This merge mode will then convert the abstract type into the specified component, and copy over as many of the properties from the original object to the new object as possible. It does this by examining the .NET inheritance hierarchies of the two object types, locating the nearest common ancestor type, and then copying all of the property values from the nearest common ancestor to the newly specified object. After doing so, all of the property overrides specified in the Transformer are applied to the new object. Listed below is a very simple LocalMergeAndTypeReplace transformer that can be used to transform Primary Keys into the equivalent Clustered Indexes in a Biml table:

<#@ target type="TablePrimaryKey" mergemode="LocalMergeAndTypeReplace" #>

<Index Unique="true" Clustered="true">
    <Columns>
        <# foreach (var columnReference in TargetNode.Columns) { #>
        <Column ColumnName="<#=columnReference.Column.ScopedName#>" SortOrder="<#=columnReference.SortOrder#>" />
        <# } #>
    </Columns>
</Index>

LocalReplace

The LocalReplace merge mode is used when you want to completely remove the original object and replace it with an entirely new object or collection of objects. The major downside of this approach is that it requires you to re-specify every property value in the newly created object, even simple things like Name. For this reason, LocalReplace is avoided unless there is no other reasonable alternative.

Working with Collections

In an earlier example, it was shown how to add objects to child collections of the transformer target, as follows:

<#@ target type="Package" mergemode="LocalMerge" #>
<Package>
    <Variables>
        <Variable Name="LogStart" DataType="DateTime" />
        <Variable Name="LogLabel" DataType="String" />
    </Variables>
</Package>

It is important to note that there might be other objects already present in the child collection. In most cases, this is of no direct consequence, provided that you perform duplicate checking before adding new items, if necessary. In some cases, you need more fine-grained control over how your objects are added to a list of existing objects. For example, if you have a Package with a Linear ConstraintMode, and you would like to transform it so that an ExecuteSQL task is added to both the beginning and end of the task list to perform logging startup and shutdown, then you must have the ability to both add to the beginning and end of the child collection. The addToHeadOfCollections directive property is supplied for this purpose. In the above example, you can use 2 transformers to achieve the desired effect:

<#@ template addtoheadofcollections="True" #>
<#@ target type="Package" mergemode="LocalMerge" #>
<Package>
    <Tasks>
        <ExecuteSQL Name="StartLog" ConnectionName="Log">
        <DirectInput>EXEC sp_startlog</DirectInput>
        </ExecuteSQL>
    </Tasks>
</Package>

And

<#@ target type="Package" mergemode="LocalMerge" #>
<Package>
    <Tasks>
    <ExecuteSQL Name="EndLog" ConnectionName="Log">
        <DirectInput>EXEC sp_endlog</DirectInput>
        </ExecuteSQL>
    </Tasks>
</Package>

Transformer Biml Code

As you’ve seen in the examples in the previous section, the Biml code in a Transformer BimlScript should use a root node that is compatible with the type of the target object. Intelliprompt in Mist will help you by showing the list of eligible root tags in the autocomplete dropdown list. Be certain to correctly specify the target type and the target merge mode in order to get the correct list of eligible tags. There is an additional advanced scenario of which to be aware. In some cases, you want to replace a single object with multiple objects. The following example allows the developer to add a single column as a metadata placeholder to a table and automatically expand it to several columns.

<#@ target type="TableColumn" mergemode="LocalReplace" #>
<# if (TargetNode.Name == "MetadataPlaceholder") { #>
<Columns>
    <Column Name="MetadataColumn1" DataType="Int32" />
    <Column Name="MetadataColumn2" DataType="String" Length="8" />
</Columns>
<# } #>

As another example, the following code listing shows a complicated transformer that replaces a Lookup dataflow component with several components that implement a “late-arriving dimension” pattern for the lookup:

<#@ target type="Lookup" mergemode="LocalReplace" #>

<Transformations>
    <#=TargetNode.EmitAllXml()#>
    <DerivedColumns Name="_<#=TargetNode.Name#>_DerivedColumnsDefaultValue">
        <InputPath OutputPathName="<#=TargetNode.Name#>.NoMatch" />
        <Columns>
           <Column Name="<#=TargetNode.Annotations["LA"].Text#>" DataType="Int32">0</Column>
        </Columns>
    </DerivedColumns>
    <OleDbCommand Name="_<#=TargetNode.Name#>_OleDBCommandInsertPlaceholderRow" ConnectionName="DataWarehouse" ValidateExternalMetadata="False">
        <DirectInput>EXEC pInsertCustomer ?, ? OUTPUT</DirectInput>
        <Parameters>
            <Parameter SourceColumn="Customer" TargetColumn="@CustomerName" />
            <Parameter SourceColumn="SalesAmount" TargetColumn="@CustomerID" />
        </Parameters>
    </OleDbCommand>
    <UnionAll Name="_<#=TargetNode.Name#>_UnionAllMatchAndNoMatch">
        <InputPaths>
            <InputPath OutputPathName="<#=TargetNode.Name#>.Match" />
            <InputPath OutputPathName="_<#=TargetNode.Name#>_OleDBCommandInsertPlaceholderRow.Output" />
        </InputPaths>
    </UnionAll>
</Transformations>

In each of the above cases, the root node of the transformer Biml is not the tag for a specific node, but rather the name of the collection property of the parent of the target node that contains the target node. This is the core language mechanism that allows you to replace the single target node with a collection of items.

Executing Transformers

Interactive Execution in Mist

Transformer BimlScripts are normally authored in the BimlScript Library node of the Logical View in Mist. If you want to execute a Transformer BimlScript across a large number of target objects, either:

1. Open the Transformer BimlScript in the designer, and then click the Execute” button on the “BimlScript” ribbon.
2. Right click the BimlScript Transformer file in the logical view, and then click “Execute BimlScript.”

Both of the above options will produce a dialog box which allows you to select which objects of the target type you would like to transform. If you instead want to transform a single object, simply right-click that object in its visual designer, select the Execute Transformer sub-menu from the context menu, and then click on the desired transformer to execute.

Transformer Frameworks

Instead of manually/interactively executing transformers, you might instead want the compiler to automatically execute the transformer on every build. In order to do this, you can create a Transformer Framework, which is just an XML file that specifies a list of transformers to be executed in a particular order. For example:

<FrameworkSettings RootItemName="RootNode" TargetItemName="TargetNode" xmlns="http://schemas.varigence.com/FlowModel.xsd">
    <TransformerBimlScripts>
        <TransformerBimlScript TransformerBimlScriptFilePath="LateArrivingDimension.biml" />
        <TransformerBimlScript TransformerBimlScriptFilePath="ScriptProject.biml" />
    </TransformerBimlScripts>
</FrameworkSettings>

The key to the transformer framework is to list the desired transformers as TransformerBimlScript elements. Frameworks support more advanced features that will be covered separately, since they apply primarily to developers who are developing frameworks for use by third parties. Note that the framework approach to transformers is ideal for building many small transformers that incrementally build on each other, rather than attempting to fit all of the logic for a desired transformation within a single transformer. If the logic for a transformer is challenging, try splitting it into two or more transformers – perhaps even targeting different object types. You’ll likely find this the quickest way to simplify your framework logic.

Finally, add the following command line option to MSBuild: /p:TransformationScriptSettings=

In Mist, you can do the same by right-clicking the project in Logical View, selecting Properties, and then adding the line above to the “Command Line Options” text box.

In a Biml language extension

Biml actually permits end-users to develop plugins that add entirely new language elements. Normally the code generation for those new language elements is defined using transformer frameworks. Biml language extensions are a big topic and will be covered in a separate article.

Replacing core code emission

As the power of transformers is becoming more clear, it may not surprise you that much of the core emission logic of the Biml compiler is actually written using transformers. That's right. Biml is used heavily to write important parts of the Biml compiler! This also means that if an end-user has a sufficiently complex framework or if they want to fundamentally change Biml code generation (for example, to support an entirely different platform), this can be done by instructing the Biml compiler to use your framework instead of its default framework. This allows you to override EVERY aspect of code generation. This is obviously a big topic, so it will also be covered in more depth in a follow-up article.

Advanced Transformers Using the .NET Object Model

While it is strongly recommended that developers use declarative transformers as have been described thus far, it is also possible to directly access the .NET object model underlying the target object for modification. This approach is most often useful in advanced scenarios where workflows need to be largely rewritten. As a simple example of using the object model directly, consider the following declarative transformer, which adds two variables and forces the constraint mode to be parallel.:

<#@ target type="Package" mergemode="LocalMerge" #>
<Package Constraint="Parallel">
    <Variables>
        <Variable Name="LogStart" DataType="DateTime">1/1/1900</Variable>
        <Variable Name="LogLabel" DataType="String"></Variable>
    </Variables>
</Package>

This could be equivalently written using the .NET object model as follows. Note that in this case, the merge mode is basically ignored, since the change logic is handled explicitly within .NET code.

<#@ target type="Package" mergemode="LocalMerge" #>
<#
TargetNode.ConstraintMode = ContainerConstraintMode.Parallel;

TargetNode.Variables.Add(new AstVariableNode(TargetNode) { Name = "LogStart", DataType = TypeCode.DateTime });

TargetNode.Variables.Add(new AstVariableNode(TargetNode) { Name = "LogLabel", DataType = TypeCode.String});
#>