In this section:
Performance
   Summary of Tests
   Testing performance in isolation (non-BizTalk)
   Performance Test Results
   Measuring Memory Usage in BizTalk
   BizTalk Memory Test Results
   Byte Arrays
   Analysing the performance results
Maintainability
   External XSLT
   Serializable Classes
   Why is it so difficult to edit code in the Script functoid?
   Documentation

Any large BizTalk project will likely have had the inevitable conversations about performance and maintainability: will it be fast/sustainable enough, and will the tech support team (or whoever looks after the code once the developers have finished) be able to maintain it?

In this post I want to look at the performance of the Mapper, and also look at how maintainable maps are generally.

In order to do this, I want look at the different options you have for executing XSLT with the Mapper, and compare this to the most common non-Mapper mechanism for performing transformation: using serializable classes.


In BizTalk, your options for transformations using the Mapper are:

• Built-in functoids
• Inline XSLT script
• Inline C# / VB.NET / JScript.NET
• Classes/methods in external assemblies
• Custom functoids
• External XSLT file

Normally you'd use a combination of these in a complex BizTalk solution.

So how do you decide which to use?
Which gives you the best performance?
Which option(s) are the easiest to maintain?

The answer is: it depends!

In this section I'm going to try and give you some hard data you can use to try and answer these questions. In the next post, I'll try and answer the questions.

Performance
Performance is a very subjective subject. You could spend weeks getting your maps to execute in under 10ms, but if your performance requirement is "anything less than 100ms" then why bother?

What's more important is that performance is "good enough" and is sustainable.

Sustainable performance is the key - its one thing to be able to perform a single transformation in less than 10ms, but what about performing 40 simultaneous transformations continuously for over an hour?
And what about the memory footprint?

When we talk about performance, we also need to look at memory/resource usage: If you have a very fast system that for each transformation uses 1MB memory, then at a certain point under sustained load your memory usage will start to affect performance as memory is paged out to disk (assuming that garbage collection can't keep up with the number of transformations you're performing).

In order to find out the performance and memory footprints of the various options, I put together some tests.

I ran two suites of tests:

• Using a stand-alone test harness with the XslTransform class to perform transformation in isolation (this gives the comparative performance)
• Using a BizTalk application to measure memory usage (this gives the comparative memory usage)

Summary of Tests
I simplified the six options above into four separate XSLT tests, and then added two tests involving serializable classes for comparison:

1. Standard Map
   A map using default built-in functoids, with a mix of functoids that emit XSLT and functoids that emit C#
2. Map using external XSLT
   A map which used an external XSLT file consisting of pure XSLT i.e. no external assemblies or inline script
3. Map using inline C#
   A map using inline C# code in a Script functoid
4. Map using external assembly
   A map using an external assembly via a Script functoid
5. Transformation using Classes
   De-serializing the input message into a class, transforming to a new class, and serializing the class back into the output message
6. Transformation using classes and serializers
   Same as 5. above but using the sgen.exe tool to pre-generate a Serialization class

My test scenario was: Transforming a message containing 20 employee records into a new message which contained separate Manager and Staff records:
i.e. this:

becomes this:


Each of the maps performed the transformation in the same way, as much as was possible.

Testing performance in isolation (non-BizTalk)
For the XSLT tests, I tested using both the XslTransform class (used by BizTalk) and the newer XslCompiledTransform class (for comparison).

For both of these class tests I used static and non-static instances of the classes:

• In the static test a transform class was created once and then re-used for each iteration
• In the non-static test a new transform class was created for each iteration

I ran each test twice: once with 20 iterations, and once with 500 iterations.

I also measured the amount of memory in use before and after each test - from this I calculated a rough "memory used" for each test (this is before garbage collection kicked in).

For each test I show:

1. Total
   This is the total time (in ms) that the test took to run
2. Average
   This is the average time for each iteration
3. Average without first
   This is the average time without the first iteration
4. Average setup
   This is the average time it took to create the transform object in each iteration (will be 0 for the static tests as creation of transform is performed once and not included in the timings)
5. Average transform
   This is the average time it took to perform the transformation
6. Memory Used
   This is the amount of memory used to perform the entire test

Performance Test Results
The results of the 500 iteration tests are:

	Standard Map	Map with External Assembly	Map with Inline Script	Map with External XSLT	Class Transform	Class Transform with Serializer
Iterations	500	500	500	500	500	500
XslTransform - NonStatic
Total (ms)	82402	78815	79507	1622	1749	728
Average (ms)	164	157	159	3	3	1
Average without first (ms)	164	157	158	3	2	1
Average setup (ms)	151	151	152	1	1	0
Average transform (ms)	12	5	5	1	1	1
Memory Used	504MB	452MB	456MB	62MB	90MB	86MB
XslTransform - Static
Total (ms)	4392	2234	1564	720	769	735
Average (ms)	8	4	3	1	1	1
Average without first (ms)	8	4	3	1	1	1
Average setup (ms)	0	0	0	0	0	0
Average transform (ms)	8	4	3	1	1	1
Memory Used	64MB	40MB	40MB	30MB	88MB	88MB
XslCompiledTransform - NonStatic
Total (ms)	101214	88181	89608	16325
Average (ms)	202	176	179	32
Average without first (ms)	202	176	179	32
Average setup (ms)	156	145	147	3
Average transform (ms)	45	30	30	28
Memory Used	192MB	141MB	145MB	82MB
XslCompiledTransform - Static
Total (ms)	95	131	70	55
Average (ms)	0	0	0	0
Average without first (ms)	0	0	0	0
Average setup (ms)	0	0	0	0
Average transform (ms)	0	0	0	0
Memory Used	14MB	15MB	13MB	13MB

(the lowest result on each row is highlighted in green)

And for comparison, the results from the 20 iteration tests:

	Standard Map	Map with External Assembly	Map with Inline Script	Map with External XSLT	Class Transform	Class Transform with Serializer
Iterations	20	20	20	20	20	20
XslTransform - NonStatic
Total (ms)	3820	3931	3718	78	1213	82
Average (ms)	191	196	185	3	60	4
Average without first (ms)	171	194	184	3	4	1
Average setup (ms)	175	189	178	1	56	1
Average transform (ms)	15	7	6	1	4	2
Memory Used	20MB	18MB	17MB	2MB	4MB	3MB
XslTransform - Static
Total (ms)	169	100	66	29	122	83
Average (ms)	8	5	3	1	6	4
Average without first (ms)	8	4	3	1	2	1
Average setup (ms)	0	0	0	0	2	1
Average transform (ms)	8	4	3	1	3	2
Memory Used	4MB	1MB	1MB	1MB	3MB	3MB
XslCompiledTransform - NonStatic
Total (ms)	4628	3758	5094	1190
Average (ms)	231	187	254	59
Average without first (ms)	224	183	254	50
Average setup (ms)	180	154	211	11
Average transform (ms)	50	32	42	48
Memory Used	8MB	6MB	6MB	3MB
XslCompiledTransform - Static
Total (ms)	46	39	67	81
Average (ms)	2	1	3	4
Average without first (ms)	0	0	1	1
Average setup (ms)	0	0	0	0
Average transform (ms)	2	1	3	4
Memory Used	642KB	647KB	583KB	542KB

(the lowest result on each row is highlighted in green)

Measuring Memory Usage in BizTalk
Although the Memory Used amount from the performance tests was useful, I wanted to know exactly how much memory BizTalk used for performing transformations – and what objects were in memory.
In order to measure this I used SciTech Software's .NET Memory Profiler.
This tool attaches to the BizTalk service (BTSNTSvc.exe) and can create a snapshot of all the object instances currently in use, including how many there are and how much memory they're using.

I created a BizTalk application which contained a separate map for each of the tests above, and created orchestrations to execute the maps (and to call the C# code to perform the transform using the classes).

I performed a memory snapshot before and after running the orchestrations, and restarted the BizTalk service between each test.

I ran each test twice: once with a single message, and once with 50 messages.

For each test I show:

1. Byte[] Instances
   This is the count of Byte arrays in use (the relevance of Byte arrays is explained below)
2. Byte[] Instances Size (MB)
   The is the total size of all Byte arrays (i.e. the size of all the data they contain)
3. Total Instances
   The is the count of all .NET objects in use
4. Total Instances Size(MB)
   This is the total size of all .NET objects in use

Note that any sizes measured are after garbage collection has occurred i.e. these are objects which are still classed as being in-use.

BizTalk Memory Test Results
The results I measured were:
Single Message:

Test - 1 Iteration	Byte[] Instances	Byte[] Instances Size (MB)	Total Instances	Total Instances Size (MB)
Standard	5,652	0.63	17,277	1.78
External XSLT	5,752	0.59	19,038	2.11
Inline Script	5,763	0.63	20,522	2.23
Referenced Assembly	5,757	1.25	19,897	2.42
Class	30	0.03	7,476	0.74

(the lowest result in each column is highlighted in green)

50 Messages:

Test - 50 Iterations	Byte[] Instances	Byte[] Instances Size (MB)	Total Instances	Total Instances Size (MB)
Standard	146	1.63	18,354	3.24
External XSLT	126	0.80	14,359	1.87
Inline Script	5,780	1.89	22,590	3.13
Referenced Assembly	5,757	1.36	20,100	3.00
Class	146	0.85	14,202	1.81

(the lowest result in each column is highlighted in green)

Byte Arrays
I can let Microsoft explain this in their own words (this is taken from a knowledge base article here):
The System.Policy.Security.Evidence object is often used in transforms and can consume a lot of memory. Whenever a map contains a scripting functoid that uses inline C# (or any other inline language), the assembly is created in memory. The System.Policy.Security.Evidence object uses the object of the actual calling assembly. This situation creates a rooted object that is not deleted until the BizTalk service is restarted.
 
Most of the default BizTalk functoids are implemented as inline script. These items can cause System.Byte[] objects to collect in memory. To minimize memory consumption, we recommend that you put any map that uses these functoids into a small assembly. Then, reference that assembly. Use the chart below to determine which functoids use inline script and which functoids do not use inline script.

In the second column, “Yes” means that this functoid is implemented as inline script, and that it will cause System.Byte[] objects to collect in memory. “No” means that this functoid is not implemented as inline script, and that it will not cause System.Byte[] objects to collect in memory.

Functoids	Inline script?
All String Functoids	Yes
All Mathematical Functoids	Yes
All Logical Functoids except IsNil	Yes
Logical IsNil Functoid	No
All Date/Time Functoids	Yes
All Conversion Functoids	Yes
All Scientific Functoids	Yes
All Cumulative Functoids	Yes
All Database Functoids	No
All Advanced Functoids (apart from Script functoids using Inline C#/VB/Jscript)	No

Assemblies created by compiling inline script in an XSLT are temporary assemblies and are loaded into the appropriate AppDomain – they will remain in memory until the AppDomain is unloaded i.e. the BizTalk Host Instance is restarted.

So if you keep all your maps, orchestrations, schemas, etc in one big assembly, then that assembly will stay loaded in memory until the BizTalk Host Instance that loaded it is restarted.
Solution? Keep your maps (especially maps using inline C#) in a separate project/assembly - and try and keep that assembly as small as possible!

Analysing the performance results
The results shouldn't really come as any surprise.
What they say in a nutshell is: pure XSLT is much faster than XSLT which uses inline script or referenced assemblies. How much faster? Well, my tests show an average 5500% speed increase over using the default functoids (i.e. 55 times faster)!
Additionally, using pure XSLT uses 1/8 the memory.
Of course, your mileage will vary.

What's interesting though is how fast using serializable classes is i.e. de-serializing a message into a class, performing operations to create a new class, and then serializing the new class into a message.
When used with a pre-generated serialization assembly, this mechanism chases closely behind using pure XSLT (and actually beat it in one of the tests).

Maintainability
One of BizTalk's trump cards is the BizTalk Mapper: you can create maps which can be easily maintained – what's more, because the Mapper is a visual tool you can see at a glance how the mapping works.
At least, that's the theory.
If you have a relatively simple map which uses no script functoids, and has less then, say, 50 connections then Yes, I'd say this was true: it's easy to see what the map does, and probably easy to maintain it.

But if you have a map with 1000 connections, or with a whole smattering of Script functoids (or a bird's nest of Logical functoids) then No, I don't think it's easy to see what the map does or to maintain it.

At what point do you have to admit defeat with a Map and say that it's got as bit too complex? or that the next developer to come along will have problems maintaining it?
In that case, would you be better off using external XSLT or serializable classes?

External XSLT
One of the main complaints I hear about using external XSLT with the Mapper is that it's difficult to maintain. This can be true – if your editing tool is notepad! But there are great tools around for maintaining XSLT – have a look at Altova's MapForce for an example of one.
The other complaint is that XSLT is difficult, or hard to learn. Well, so is C# if you've never used it before.
Go buy a book on it, or do a course!

Truth be told, if you work for a company which uses XSLT for other projects, then you're more likely to find support for using it as external XSLT in BizTalk.
Some companies have teams in IT which do nothing but create XSLT.

My point here is that although there's a myth that external XSLT is hard to maintain, it's a lot easier to maintain than a complicated map. And if you use the right tool, you can get a graphical view of what it does as well.

Serializable Classes
In my experience, it's very very common for developers to create a whole slew of transform and utility classes for handling transformations.
Sometimes this is because it's the best way to do it.
Other times it's because they simply didn't know how to achieve something in the Mapper.
One of the best features of BizTalk (the ability to call out to C# classes/methods from an orchestration or map) can also be its worst: Just because you *can* create C# code, doesn't mean that you *should*.

Maintaining poorly written C# code is a nightmare.
So if you're using serializable classes to perform transformations make sure they're well written and well documented – but most importantly: understand *why* you're using them over XSLT.

Why is it so difficult to edit code in the Script functoid?
Ever wondered why you can't resize the Script functoid code window? Or do a Ctrl-A to select all the code in it? It turns out there is actually a reason for it...

Scott Woodgate (former Product Manager for BizTalk) had this to say...
The article is correct when it points out we discourage the use of .NET objects directly inside the map. While this is possible, we encourage good developer design which is encapsulating code in external assemblies. This turns out to be much better because you have a single assembly with code shared across multiple maps that can be versioned once and managed more easily

Unfortunately, this restriction also means that it's difficult to use inline XSLT, which is a shame.

Documentation
You can't get away from the fact that code that is easy to maintain is either self-documenting, or is accompanied by excellent documentation.
Regardless if you use maps, external XSLT, or serializable classes you should really document your transforms: explain what they do, how they do it, why they do it – and most importantly, give some context: explain why you chose to do it that particular way.
A developer who has to maintain your code in 2 years time might not have your background of development and political issues to understand why you made your choices.

The next post is going to look at the performance/maintainability of the different transformation options and attempt to help you to decide when to use which option.