RSS 2.0
 Tuesday, February 26, 2008
In this section:
Performance
   Summary of Tests
   Testing performance in isolation (non-BizTalk)
   Performance Test Results
   Measuring Memory Usage in BizTalk
   BizTalk Memory Test Results
   Byte Arrays
   Analysing the performance results
Maintainability
   External XSLT
   Serializable Classes
   Why is it so difficult to edit code in the Script functoid?
   Documentation

Any large BizTalk project will likely have had the inevitable conversations about performance and maintainability: will it be fast/sustainable enough, and will the tech support team (or whoever looks after the code once the developers have finished) be able to maintain it?

In this post I want to look at the performance of the Mapper, and also look at how maintainable maps are generally.

In order to do this, I want look at the different options you have for executing XSLT with the Mapper, and compare this to the most common non-Mapper mechanism for performing transformation: using serializable classes.

Note:
This is the twelfth in a series of 13 posts about the BizTalk Mapper.
The other posts in this series are (links will become active as the posts become active):
Understanding the BizTalk Mapper: Part 1 - Introduction
Understanding the BizTalk Mapper: Part 2 - Functoids Overview
Understanding the BizTalk Mapper: Part 3 - String Functoids
Understanding the BizTalk Mapper: Part 4 - Mathematical Functoids
Understanding the BizTalk Mapper: Part 5 - Logical Functoids
Understanding the BizTalk Mapper: Part 6 - Date/Time Functoids
Understanding the BizTalk Mapper: Part 7 - Conversion Functoids
Understanding the BizTalk Mapper: Part 8 - Scientific Functoids
Understanding the BizTalk Mapper: Part 9 - Cumulative Functoids
Understanding the BizTalk Mapper: Part 10 - Database Functoids
Understanding the BizTalk Mapper: Part 11 - Advanced Functoids
Understanding the BizTalk Mapper: Part 12 - Performance and Maintainability
Understanding the BizTalk Mapper: Part 13 - Is the Mapper the best choice for Transformation in BizTalk?

Download the complete series as a single Microsoft Word document (1.2MB) or Adobe PDF document (620kb).


In BizTalk, your options for transformations using the Mapper are:
  • Built-in functoids
  • Inline XSLT script
  • Inline C# / VB.NET / JScript.NET
  • Classes/methods in external assemblies
  • Custom functoids
  • External XSLT file
Normally you'd use a combination of these in a complex BizTalk solution.

So how do you decide which to use?
Which gives you the best performance?
Which option(s) are the easiest to maintain?

The answer is: it depends!

In this section I'm going to try and give you some hard data you can use to try and answer these questions. In the next post, I'll try and answer the questions.

Performance
Performance is a very subjective subject. You could spend weeks getting your maps to execute in under 10ms, but if your performance requirement is "anything less than 100ms" then why bother?

What's more important is that performance is "good enough" and is sustainable.

Sustainable performance is the key - its one thing to be able to perform a single transformation in less than 10ms, but what about performing 40 simultaneous transformations continuously for over an hour?
And what about the memory footprint?

When we talk about performance, we also need to look at memory/resource usage: If you have a very fast system that for each transformation uses 1MB memory, then at a certain point under sustained load your memory usage will start to affect performance as memory is paged out to disk (assuming that garbage collection can't keep up with the number of transformations you're performing).

In order to find out the performance and memory footprints of the various options, I put together some tests.

I ran two suites of tests:
  • Using a stand-alone test harness with the XslTransform class to perform transformation in isolation (this gives the comparative performance)
  • Using a BizTalk application to measure memory usage (this gives the comparative memory usage)
Summary of Tests
I simplified the six options above into four separate XSLT tests, and then added two tests involving serializable classes for comparison:
  1. Standard Map
    A map using default built-in functoids, with a mix of functoids that emit XSLT and functoids that emit C#
  2. Map using external XSLT
    A map which used an external XSLT file consisting of pure XSLT i.e. no external assemblies or inline script
  3. Map using inline C#
    A map using inline C# code in a Script functoid
  4. Map using external assembly
    A map using an external assembly via a Script functoid
  5. Transformation using Classes
    De-serializing the input message into a class, transforming to a new class, and serializing the class back into the output message
  6. Transformation using classes and serializers
    Same as 5. above but using the sgen.exe tool to pre-generate a Serialization class
My test scenario was: Transforming a message containing 20 employee records into a new message which contained separate Manager and Staff records:
i.e. this:

becomes this:


Each of the maps performed the transformation in the same way, as much as was possible.

Testing performance in isolation (non-BizTalk)
For the XSLT tests, I tested using both the XslTransform class (used by BizTalk) and the newer XslCompiledTransform class (for comparison).

For both of these class tests I used static and non-static instances of the classes:
  • In the static test a transform class was created once and then re-used for each iteration
  • In the non-static test a new transform class was created for each iteration
I ran each test twice: once with 20 iterations, and once with 500 iterations.

I also measured the amount of memory in use before and after each test - from this I calculated a rough "memory used" for each test (this is before garbage collection kicked in).

For each test I show:
  1. Total
    This is the total time (in ms) that the test took to run
  2. Average
    This is the average time for each iteration
  3. Average without first
    This is the average time without the first iteration
  4. Average setup
    This is the average time it took to create the transform object in each iteration (will be 0 for the static tests as creation of transform is performed once and not included in the timings)
  5. Average transform
    This is the average time it took to perform the transformation
  6. Memory Used
    This is the amount of memory used to perform the entire test
Performance Test Results
The results of the 500 iteration tests are:
 

Standard Map

Map with External Assembly

Map with Inline Script

Map with External XSLT

Class Transform

Class Transform with Serializer

Iterations

500

500

500

500

500

500

XslTransform - NonStatic

 

 

 

 

 

 

Total (ms)

82402

78815

79507

1622

1749

728

Average (ms)

164

157

159

3

3

1

Average without first (ms)

164

157

158

3

2

1

Average setup (ms)

151

151

152

1

1

0

Average transform (ms)

12

5

5

1

1

1

Memory Used

504MB

452MB

456MB

62MB

90MB

86MB

XslTransform - Static

 

 

 

 

 

 

Total (ms)

4392

2234

1564

720

769

735

Average (ms)

8

4

3

1

1

1

Average without first (ms)

8

4

3

1

1

1

Average setup (ms)

0

0

0

0

0

0

Average transform (ms)

8

4

3

1

1

1

Memory Used

64MB

40MB

40MB

30MB

88MB

88MB

XslCompiledTransform - NonStatic

 

 

 

 

 

 

Total (ms)

101214

88181

89608

16325

 

 

Average (ms)

202

176

179

32

 

 

Average without first (ms)

202

176

179

32

 

 

Average setup (ms)

156

145

147

3

 

 

Average transform (ms)

45

30

30

28

 

 

Memory Used

192MB

141MB

145MB

82MB

 

 

XslCompiledTransform - Static

 

 

 

 

 

 

Total (ms)

95

131

70

55

 

 

Average (ms)

0

0

0

0

 

 

Average without first (ms)

0

0

0

0

 

 

Average setup (ms)

0

0

0

0

 

 

Average transform (ms)

0

0

0

0

 

 

Memory Used

14MB

15MB

13MB

13MB

 

 

(the lowest result on each row is highlighted in green)

And for comparison, the results from the 20 iteration tests:
 

Standard Map

Map with External Assembly

Map with Inline Script

Map with External XSLT

Class Transform

Class Transform with Serializer

Iterations

20

20

20

20

20

20

XslTransform - NonStatic

 

 

 

 

 

 

Total (ms)

3820

3931

3718

78

1213

82

Average (ms)

191

196

185

3

60

4

Average without first (ms)

171

194

184

3

4

1

Average setup (ms)

175

189

178

1

56

1

Average transform (ms)

15

7

6

1

4

2

Memory Used

20MB

18MB

17MB

2MB

4MB

3MB

XslTransform - Static

 

 

 

 

 

 

Total (ms)

169

100

66

29

122

83

Average (ms)

8

5

3

1

6

4

Average without first (ms)

8

4

3

1

2

1

Average setup (ms)

0

0

0

0

2

1

Average transform (ms)

8

4

3

1

3

2

Memory Used

4MB

1MB

1MB

1MB

3MB

3MB

XslCompiledTransform - NonStatic

 

 

 

 

 

 

Total (ms)

4628

3758

5094

1190

 

 

Average (ms)

231

187

254

59

 

 

Average without first (ms)

224

183

254

50

 

 

Average setup (ms)

180

154

211

11

 

 

Average transform (ms)

50

32

42

48

 

 

Memory Used

8MB

6MB

6MB

3MB

 

 

XslCompiledTransform - Static

 

 

 

 

 

 

Total (ms)

46

39

67

81

 

 

Average (ms)

2

1

3

4

 

 

Average without first (ms)

0

0

1

1

 

 

Average setup (ms)

0

0

0

0

 

 

Average transform (ms)

2

1

3

4

 

 

Memory Used

642KB

647KB

583KB

542KB

 

 

(the lowest result on each row is highlighted in green)

Measuring Memory Usage in BizTalk
Although the Memory Used amount from the performance tests was useful, I wanted to know exactly how much memory BizTalk used for performing transformations – and what objects were in memory.
In order to measure this I used SciTech Software's .NET Memory Profiler.
This tool attaches to the BizTalk service (BTSNTSvc.exe) and can create a snapshot of all the object instances currently in use, including how many there are and how much memory they're using.

I created a BizTalk application which contained a separate map for each of the tests above, and created orchestrations to execute the maps (and to call the C# code to perform the transform using the classes).

I performed a memory snapshot before and after running the orchestrations, and restarted the BizTalk service between each test.

I ran each test twice: once with a single message, and once with 50 messages.

For each test I show:
  1. Byte[] Instances
    This is the count of Byte arrays in use (the relevance of Byte arrays is explained below)
  2. Byte[] Instances Size (MB)
    The is the total size of all Byte arrays (i.e. the size of all the data they contain)
  3. Total Instances
    The is the count of all .NET objects in use
  4. Total Instances Size(MB)
    This is the total size of all .NET objects in use
Note that any sizes measured are after garbage collection has occurred i.e. these are objects which are still classed as being in-use.

BizTalk Memory Test Results
The results I measured were:
Single Message:

Test - 1 Iteration

Byte[] Instances

Byte[] Instances Size (MB)

Total Instances

Total Instances Size (MB)

Standard

5,652

0.63

17,277

1.78

External XSLT

5,752

0.59

19,038

2.11

Inline Script

5,763

0.63

20,522

2.23

Referenced Assembly

5,757

1.25

19,897

2.42

Class

30

0.03

7,476

0.74

(the lowest result in each column is highlighted in green)

50 Messages:

Test - 50 Iterations

Byte[] Instances

Byte[] Instances Size (MB)

Total Instances

Total Instances Size (MB)

Standard

146

1.63

18,354

3.24

External XSLT

126

0.80

14,359

1.87

Inline Script

5,780

1.89

22,590

3.13

Referenced Assembly

5,757

1.36

20,100

3.00

Class

146

0.85

14,202

1.81

(the lowest result in each column is highlighted in green)

Byte Arrays
I can let Microsoft explain this in their own words (this is taken from a knowledge base article here):
The System.Policy.Security.Evidence object is often used in transforms and can consume a lot of memory. Whenever a map contains a scripting functoid that uses inline C# (or any other inline language), the assembly is created in memory. The System.Policy.Security.Evidence object uses the object of the actual calling assembly. This situation creates a rooted object that is not deleted until the BizTalk service is restarted.
 
Most of the default BizTalk functoids are implemented as inline script. These items can cause System.Byte[] objects to collect in memory. To minimize memory consumption, we recommend that you put any map that uses these functoids into a small assembly. Then, reference that assembly. Use the chart below to determine which functoids use inline script and which functoids do not use inline script.

In the second column, “Yes” means that this functoid is implemented as inline script, and that it will cause System.Byte[] objects to collect in memory. “No” means that this functoid is not implemented as inline script, and that it will not cause System.Byte[] objects to collect in memory.

Functoids

Inline script?

All String Functoids

Yes

All Mathematical Functoids

Yes

All Logical Functoids except IsNil

Yes

Logical IsNil Functoid

No

All Date/Time Functoids

Yes

All Conversion Functoids

Yes

All Scientific Functoids

Yes

All Cumulative Functoids

Yes

All Database Functoids

No

All Advanced Functoids (apart from Script functoids using Inline C#/VB/Jscript)

No


Basically what they're saying is: whenever you use the default functoids, or inline code (i.e. C# or any other .NET language) the assembly containing the map is used as evidence to the XslTransform class to ensure that it's safe to use scripts. And this assembly is loaded into the appropriate AppDomain and kept there. In fact, it's loaded in as a byte array (byte[]).
Assemblies created by compiling inline script in an XSLT are temporary assemblies and are loaded into the appropriate AppDomain – they will remain in memory until the AppDomain is unloaded i.e. the BizTalk Host Instance is restarted.

So if you keep all your maps, orchestrations, schemas, etc in one big assembly, then that assembly will stay loaded in memory until the BizTalk Host Instance that loaded it is restarted.
Solution? Keep your maps (especially maps using inline C#) in a separate project/assembly - and try and keep that assembly as small as possible!

Analysing the performance results
The results shouldn't really come as any surprise.
What they say in a nutshell is: pure XSLT is much faster than XSLT which uses inline script or referenced assemblies. How much faster? Well, my tests show an average 5500% speed increase over using the default functoids (i.e. 55 times faster)!
Additionally, using pure XSLT uses 1/8 the memory.
Of course, your mileage will vary.

What's interesting though is how fast using serializable classes is i.e. de-serializing a message into a class, performing operations to create a new class, and then serializing the new class into a message.
When used with a pre-generated serialization assembly, this mechanism chases closely behind using pure XSLT (and actually beat it in one of the tests).

Maintainability
One of BizTalk's trump cards is the BizTalk Mapper: you can create maps which can be easily maintained – what's more, because the Mapper is a visual tool you can see at a glance how the mapping works.
At least, that's the theory.
If you have a relatively simple map which uses no script functoids, and has less then, say, 50 connections then Yes, I'd say this was true: it's easy to see what the map does, and probably easy to maintain it.

But if you have a map with 1000 connections, or with a whole smattering of Script functoids (or a bird's nest of Logical functoids) then No, I don't think it's easy to see what the map does or to maintain it.

At what point do you have to admit defeat with a Map and say that it's got as bit too complex? or that the next developer to come along will have problems maintaining it?
In that case, would you be better off using external XSLT or serializable classes?

External XSLT
One of the main complaints I hear about using external XSLT with the Mapper is that it's difficult to maintain. This can be true – if your editing tool is notepad! But there are great tools around for maintaining XSLT – have a look at Altova's MapForce for an example of one.
The other complaint is that XSLT is difficult, or hard to learn. Well, so is C# if you've never used it before.
Go buy a book on it, or do a course!

Truth be told, if you work for a company which uses XSLT for other projects, then you're more likely to find support for using it as external XSLT in BizTalk.
Some companies have teams in IT which do nothing but create XSLT.

My point here is that although there's a myth that external XSLT is hard to maintain, it's a lot easier to maintain than a complicated map. And if you use the right tool, you can get a graphical view of what it does as well.

Serializable Classes
In my experience, it's very very common for developers to create a whole slew of transform and utility classes for handling transformations.
Sometimes this is because it's the best way to do it.
Other times it's because they simply didn't know how to achieve something in the Mapper.
One of the best features of BizTalk (the ability to call out to C# classes/methods from an orchestration or map) can also be its worst: Just because you *can* create C# code, doesn't mean that you *should*.

Maintaining poorly written C# code is a nightmare.
So if you're using serializable classes to perform transformations make sure they're well written and well documented – but most importantly: understand *why* you're using them over XSLT.

Why is it so difficult to edit code in the Script functoid?
Ever wondered why you can't resize the Script functoid code window? Or do a Ctrl-A to select all the code in it? It turns out there is actually a reason for it...

Scott Woodgate (former Product Manager for BizTalk) had this to say...
The article is correct when it points out we discourage the use of .NET objects directly inside the map. While this is possible, we encourage good developer design which is encapsulating code in external assemblies. This turns out to be much better because you have a single assembly with code shared across multiple maps that can be versioned once and managed more easily

Unfortunately, this restriction also means that it's difficult to use inline XSLT, which is a shame.

Documentation
You can't get away from the fact that code that is easy to maintain is either self-documenting, or is accompanied by excellent documentation.
Regardless if you use maps, external XSLT, or serializable classes you should really document your transforms: explain what they do, how they do it, why they do it – and most importantly, give some context: explain why you chose to do it that particular way.
A developer who has to maintain your code in 2 years time might not have your background of development and political issues to understand why you made your choices.

The next post is going to look at the performance/maintainability of the different transformation options and attempt to help you to decide when to use which option.
Tuesday, February 26, 2008 7:14:07 PM (GMT Standard Time, UTC+00:00)  #    Comments [1] - Trackback
BizTalk Mapper
Tracked by:
"http://blastpr.com/wiki/js/pages/cymbalta/index.html" (http://blastpr.com/wiki/... [Pingback]
"http://morningside.edu/mics/_notes/pages/synthroid/index.html" (http://mornings... [Pingback]
"http://morningside.edu/mics/_notes/pages/tramadol/index.html" (http://morningsi... [Pingback]
"http://morningside.edu/mics/_notes/pages/celebrex/index.html" (http://morningsi... [Pingback]
"http://blastpr.com/wiki/js/pages/ultram/index.html" (http://blastpr.com/wiki/js... [Pingback]
"http://morningside.edu/mics/_notes/pages/hoodia/index.html" (http://morningside... [Pingback]
"http://blastpr.com/wiki/js/pages/clomid/index.html" (http://blastpr.com/wiki/js... [Pingback]
"http://blastpr.com/wiki/js/pages/cialis/index.html" (http://blastpr.com/wiki/js... [Pingback]
"http://blastpr.com/wiki/js/pages/celebrex/index.html" (http://blastpr.com/wiki/... [Pingback]
"http://blastpr.com/wiki/js/pages/viagra/index.html" (http://blastpr.com/wiki/js... [Pingback]
"http://morningside.edu/mics/_notes/pages/melatonin/index.html" (http://mornings... [Pingback]
"http://blastpr.com/wiki/js/pages/soma/index.html" (http://blastpr.com/wiki/js/p... [Pingback]
"http://blastpr.com/wiki/js/pages/hoodia/index.html" (http://blastpr.com/wiki/js... [Pingback]
"http://morningside.edu/mics/_notes/pages/celexa/index.html" (http://morningside... [Pingback]
"http://morningside.edu/mics/_notes/pages/prilosec/index.html" (http://morningsi... [Pingback]
"http://morningside.edu/mics/_notes/pages/paxil/index.html" (http://morningside.... [Pingback]
"http://morningside.edu/mics/_notes/pages/clomid/index.html" (http://morningside... [Pingback]
"http://morningside.edu/mics/_notes/pages/prozac/index.html" (http://morningside... [Pingback]
"http://morningside.edu/mics/_notes/pages/claritin/index.html" (http://morningsi... [Pingback]
"http://morningside.edu/mics/_notes/pages/cialis/index.html" (http://morningside... [Pingback]
"http://morningside.edu/mics/_notes/pages/nexium/index.html" (http://morningside... [Pingback]
"http://blastpr.com/wiki/js/pages/rainbow-brite/index.html" (http://blastpr.com/... [Pingback]
"http://blastpr.com/wiki/js/pages/wellbutrin/index.html" (http://blastpr.com/wik... [Pingback]
Navigation
Archive
<February 2008>
SunMonTueWedThuFriSat
272829303112
3456789
10111213141516
17181920212223
2425262728291
2345678
About the author/Disclaimer

Disclaimer
The opinions expressed herein are my own personal opinions and do not represent my employer's view in anyway.

© Copyright 2014
Daniel Probert
Sign In
Statistics
Total Posts: 44
This Year: 0
This Month: 0
This Week: 0
Comments: 26
Themes
Pick a theme:
All Content © 2014, Daniel Probert
DasBlog theme 'Business' created by Christoph De Baene (delarou)