In this section:
Performance
   Summary of Tests
   Testing performance in isolation (non-BizTalk)
   Performance Test Results
   Measuring Memory Usage in BizTalk
   BizTalk Memory Test Results
   Byte Arrays
   Analysing the performance results
Maintainability
   External XSLT
   Serialisable Classes
   Why is it so difficult to edit code in the Script functoid?
   Documentation

Any large BizTalk project will likely have had the inevitable conversations about performance and maintainability: will it be fast/sustainable enough, and will the tech support team (or whoever looks after the code once the developers have finished) be able to maintain it?

In this post I want to look at the performance of the Mapper, and also look at how maintainable maps are generally.

In order to do this, I want look at the different options you have for executing XSLT with the Mapper, and compare this to the most common non-Mapper mechanism for performing transformation: using serialisable classes.

Interestingly, all of the advanced functoids emit XSLT. No C# in sight at all.
The reason for this is that the functoids in this category all perform operations best suited to trees of data i.e. XML.
The only way to do this in C# would be to load the data into a DOM (i.e. XmlDocument) or XmlReader, or treat the XML as string data and search for tokens.

Note: this category was the one that actually started this series – I felt that if you knew the XSLT emitted by these functoids it would help understand when to use them, and what you can achieve with them.

Functoids covered in this category:

Assert	Record Count
Index	Scripting
Iteration	Table Looping
Looping	Table Extractor
Mass Copy	Value Mapping
Nil Value	Value Mapping (Flattening)

This category contains both Database and Cross Referencing Functoids – but they all connect to a database to retrieve/update data.
Unlike all other default functoids, these functoids all call classes/methods in external assemblies – no inline C# is emitted at all. Because of this, this is the only category that emits an ExtensionObjects file listing the strong names of the external assemblies used.

Note: in this category I show some of the source code from the external assemblies as well.

Functoids covered in this category:

Database Lookup	Get Common Value
Error Return	Remove Application ID
Format Message	Set Common ID
Get Application ID	Value Extractor
Get Application Value	Common Code
Get Common ID

Of the functoids in this category, only Cumulative Sum has a counterpart in XSLT v1.0 – all the others can be performed in XSLT v2.0, but not XSLT v1.0.

Functoids covered in this category:

Cumulative Average	Cumulative Minimum
Cumulative Concatenate	Cumulative Sum
Cumulative Maximum	Common Code

Yet another category which has no direct support in XSLT v1.0 or XSLT v2.0!
However, given the strong support for scientific functions in .NET, it's easy to call out to .NET classes, which is exactly what every single one of the functoids in this category does.

Having said that: have you ever used one of these functoids in a map? Care to share a real world example?
I'd be interested to find out how often they are used.

Functoids covered in this category:

10^n	Natural Logarithm
Arc Tangent	Sine
Base-Specified Logarithm	Tangent
Common Logarithm	X^Y
Cosine	Common Code
Natural Exponential Function

Surprisingly, neither XSLT v1.0 nor XSLT v2.0 have any built-in conversion support (well, not for the scenarios represented in this category anyway).
It is possible to download XSLT libraries which can do this sort of conversion (as mentioned in the notes below each functoid), but the XSLT is not pretty, and I'm not convinced about performance.
So C# is generally your only option here.

Functoids covered in this category:

ASCII to Character	Octal
Character to ASCII	Common Code
Hexadecimal

XSLT v1.0 has no support for Date/Time values, whilst XSLT v2.0 has full support.
Therefore it's not surprising that your only option is to use C#'s rich support for Date/Time values.
And this is why all of the functoids in this category emit inline C#.

Functoids covered in this category:

Add Days	Time
Date	Common Code
Date and Time

Whenever I've looked at the XSLT generated by a map I've always been confused by the amount of inline C# generated by these functoids.
After the String Functoids I'd say that these are the next most widely used and yet all but one of them has an XSLT v1.0 equivalent!
The code emitted for "Logical Equal" always makes me laugh – 12 lines of C# code can be replaced by... (wait for it)... one "=" symbol!

Functoids covered in this category:

Equal	Logical Existence
Greater Than	Logical NOT
Greater Than or Equal To	Logical Numeric
IsNil	Logical OR
Less Than	Logical String
Less Than or Equal To	Not Equal
Logical Date	Common Code

Mathematics is not a strong point of XSLT. XSLT v1.0 has very poor mathematic support, whilst XSLT v2.0 has better support, but only by a small amount. Therefore most of the functoids in this category can only be implemented in C#.

So if you want to perform a complicated mathematical function (i.e. anything more than addition or subtraction!) you’re better off using one of these functoids, or an external assembly.

Once again, inline C# isn’t the fastest, but given a choice between a slow function and no function, you might not have a choice.

Functoids covered in this category:

Absolute Value	Modulo
Addition	Multiplication
Division	Round
Integer	Square Root
Maximum Value	Subtraction
Minimum Value	Common Code

The String Functoids are probably the most frequently used in maps (in my experience), mainly because they're the most familiar to a procedural programmer (i.e. a C# or VB programmer). However because they all emit inline C#, they perform the slowest so if you want your maps to run faster you're better off using the corresponding XSLT, or implementing the functionality you require in a separate assembly.

Functoids covered in this category:

Lowercase	String Left Trim
Size	String Right
String Concatenate	String Right Trim
String Extract	Uppercase
String Find	Common Code
String Left

This whole series of posts started because I wanted to show what XSLT was emitted when using the default functoids provided by Microsoft.
Specifically, I wanted to show the XSLT emitted by the Advanced Functoids. Understanding this XSLT can help in understanding how to use the functoids.
For some reason (as seems to happen with me) the post expanded into a whole series on the Mapper... every time I explain one thing, I seem to want to explain all the things that the first thing is based on... oops.

Anyway, suffice to say that the next 9 posts will cover the code emitted by all of the default functoids provided with BizTalk 2004 / 2006 / 2006 R2.

One thing to realise is that the majority of the default functoids emit inline C# code – which is odd as quite a lot of the functionality can be performed using pure XSLT.

So for each functoid I've shown:

1. Whether XSLT or C# is emitted
2. Whether an XSLT equivalent exists
3. The XSLT or C# emitted by the functoid
4. Where C# is emitted, the equivalent XSLT to achieve the same functionality (in both XSLT v1.0 and v2.0)

In this post:
Introduction
BizTalk Mapper 101
History
BizTalk Mapper in BTS 2004 / 2006 / 2006R2
What happens when a map is compiled
What happens when a map is executed
XslTransform vs XslCompiledTransform
XSLT 1.0 vs XSLT 2.0

Introduction
The BizTalk Mapper is an integral part of the BTS toolkit and, depending on your inclination and experience, you will probably either love, hate it, or not care about it.
At its core, the BizTalk Mapper is simply a visual tool for specifying XSLT (eXtensible Stylesheet Language Transformations).
This XSLT is used to transform one stream of XML data into another stream of XML data. And since a "stream of XML data" = "message in BizTalk", this means transforming one message into another message.

[Aside: you might think that the Mapper allows you to specify multiple source or destination messages, but this is actually a trick - BizTalk creates a special schema with one part per input/output message - there's still only one input/output message]

How well the BizTalk Mapper performs its task depends on the complexity of the transformation you're attempting - and the skill/experience of the developer using the BizTalk Mapper!

For those of you who are unaware, XPath is a powerful query language for XML (ignoring XQuery for now).

In my experience I've worked with a lot of BizTalk developers who have come from a C# (and usually VB before that) background, and so haven't always had the background in XML, XSLT, XSD, and XPath needed to fully appreciate BizTalk.

Because let's face it: before you were working on BizTalk, how often did you need to deal directly with Xml? When writing ASP.NET web services, the .NET framework abstracts away the need to understand XML, leaving you with classes which are handily serialised/de-serialised across the wire for you.

Note: I have come across a similar pattern in the J2EE world – ask many Java developers about an Xml document they received as a request in a J2EE Web Service, and they'll only know about Beans, as the framework looks after the de-serialization from Xml into a Bean for them).

However, BizTalk is a different beastie: it works predominantly with data streams containing Xml.
Therefore the ability to know how to query this Xml without having to resort to C# code is very very important.

BizTalk 2004 and 2006/2006R2 support XPath 1.0 - as at this time, there is no support for XPath 2.0 in BizTalk.
You can get a good idea of the functions available in XPath 1.0 here (which is a useful summary of the W3 Recommendation)

(as a comparison, here's what XPath 2.0 gives you - here's hoping we get this in BizTalk sometime soon...!)

This post follows on from yesterday's post: Creating BizUnit Test Cases for comparing Xml Files

If you are using the XmlValidationStep/XmlValidationStepEx BizUnit steps and using XPath validation, then it can be a pain to write all these XPathValidation statements by hand.

So I wrote a utility to generate the XPath statements for you.
In fact, it generates the entire BizUnit Test Case for validating an Xml file.
Once it has been generated, all you have to do is[...]

BizUnit is a great tool for performing end-to-end testing of BizTalk applications (and can even be used for testing non-BizTalk applications like Web services too).
Although you're not strictly performing unit-testing of your BizTalk artefacts per-se, with the right setup you can get very close to unit testing.

For example if you want to test an orchestration in isolation you can:

1. Hook your orchestration up to file send/receive ports
   or
2. Write a façade orchestration which calls your orchestration, and is itself hooked up to file send/receive ports or
   or
3. Write your own orchestration hosting engine

I don't know many people who'd be crazy enough to do 3) (although I do know one…) but BizUnit is of great help for 1) and 2)

In its simplest form you can use BizUnit to[...]

Updates:
03/10/2011
The source code and MSIs for this tool (inc v1.01) are now available on CodePlex here: http://dansharpxmlviewer.codeplex.com

22/01/2008
I'll release v1.1 this week - only has minor bug fixes (one of them related to problems with opening files from the recently used file menu).
Also I'm planning to release the source code this weekend.

---

So here's a little utility that I've been using for myself for a while, but thought I'd release in case it's of use to anyone else.

There are three tasks I perform quite regularly when doing XML or BizTalk development:

1. Building and testing the XPath needed to retrieve a value from a given request/response document
2. Validating a document against a schema
3. Generating BizUnit tests cases (from an Xml instance doc, creating XPath statements to validate for all the data in the document)

This tool does all three of these things:


XmlViewer main form


Generating a BizUnit Test Case

Although there are good tools on the market for doing 1. and 2. (e.g. Altova XmlSpy and Stylus Studio) most of them require installation and are not free.
And none of them generate BizUnit Test Cases!
I wanted a single EXE which would do the most common tasks for me.

Also note that Visual Studio will give you the XPath for an element/attribute in an Xsd, but not an Xml document.
And although you can validate a document in Visual Studio, it's a hassle each time you want to change the Xml document used (right click schema, choose properties, change Input Instance Filename, cick OK, right click schema, choose Validate Instance).

Having a scratch pad to play with XPath queries has been very useful, and is the original reason I wrote this.
But generating BizUnit Test Cases has so far been the most useful... if you use the REG file provided, you can even right-click an Xml file to generate a BizUnit TestCase for it.

There is a command line version supplied as well, so if you need to generate a lot of BizUnit Test Cases, you can do so in a batch file/script.

The utility is written as a single EXE, with no additional files needed (other than .NET 2.0) and it doesn't need to be installed.
I have a copy on my USB key and I run it from there.

Caveat: I use this tool every day, so most bugs have (hopefully) been eradicated.
So my apologies if you come across something I haven't tested yet - please let me know if you do.

Files available to download:
Note: I will be making the source code for this utility available, but need to tidy it up a bit first.
Will add it here when it's ready - probably in a week or so.

MSI Installer (contains all of the below files)
    DanSharp.XmlViewer.Setup.msi (1.8MB)

Executable (Windows version)
    XmlViewer.exe (80kb)

Executable (Command line version)
    XmlViewer.com (80kb)

Registry File (.reg) for adding right-click support to Xml files
    GenerateBizUnitTestCase.reg (2kb)

Documentation
    DanSharp XmlViewer Documentation v1.0.pdf (860kb PDF)
    DanSharp XmlViewer Documentation v1.0.doc (600kb Word 97 Doc)

Source Code
   (Coming soon)

If you come across bugs or have any comments, then leave a comment here, or email me at:
xmlviewerATprobertsolutions.com

This is something that bit me recently:
The current version of BizUnit (v.2.2.1003.0)  uses the .NET 1.0 XmlValidatingReader to do schema validation in both the XmlValidationStep and XmlValidationStepEx steps.

.NET 2.0 deprecated the XmlValidatingReader class, proposing that you use the XmlReader class and the new XmlReaderSettings class instead.
Microsoft also introduced a slew of bug fixes to the validation logic[...]

Thought this might be of interest to a few BizTalk developers out there.

It's quite common practice to use Distinguished Fields in orchestrations, to get/set the value of an element or attribute.

However, if the element you're trying to set/get doesn't exist, then this poses a few problems.

Setting a Distinguished Field
If the element/attribute doesn't exist in the target message, then you'll get an exception, no questions asked – there's no way around this.
Under the covers, the SetDistinguishedField method is called to set the value – and it doesn't check if the element/attribute exists first.
The same thing happens if you use the xpath() function to set a value, and the element/attribute doesn't exist – BizTalk isn't about to modify your message and add the element/attribute for you[...]

(or how to do asynchronous processing of a synchronous request/response message)
[Note: this post follows on from my previous post How to validate Xml Documents against schemas in BizTalk. For example, if you validate a message in a pipeline and validation fails, how do you send a response back to a waiting web service client.]

For a while now, people have been struggling with how to asynchronously process request/response messages - basically, how to get away from having a request/response port in an orchestration bound to a request/response  receive location.

For example, Yossi Dohan blogged on this a while back, and came up with a solution involving multiple orchestrations:
http://www.sabratech.co.uk/blogs/yossidahan/2006/06/sync-to-async-conversion.html

The problem revolves around the fact that BizTalk will always demote the EpmRRCorrelationToken context property when your message leaves the orchestration if you try and manually set it yourself.

I got asked a question the other day: How would you validate an incoming message against a schema if the message was the request part of a request-response pair and you wanted to return a response if the request wasn't valid?

In the example given, an orchestration had been exposed as a web service, and the requirement was to validate the incoming message. If the message did not validate they wanted to return a response message with an error message in it.

I gave two of the ways I would do it, but that wasn't what they were expecting: they were expecting the simplest (and computationally slowest) way of doing it. And I realised that many people use this mechanism as they don't know there's any other way.

Why do I say this? I'll explain as I give my solutions.
First of all: The solution that was expected was to use an orchestration to do the validation - as the person explained to me, that was the only way to get the response message back to the same "connection" i.e. have it go back out as a response to the matching request.
As you'll see this is not true.
In this post I'll cover the ways to do validation.
In the next post, I'll cover how you correlate the response back to the client who is waiting for a response.

Let me say one thing: BizTalk is not magic. There is no magic (thanks Nakor). There's simply some COM+ applications, some .NET assemblies, instances of a Windows Service, some database tables... and a lot of unmanaged code.
What gets confusing[...]