Understanding the BizTalk Mapper: Part 4 - Mathematical Functoids#
Mathematics is not a strong point of XSLT. XSLT v1.0 has very poor mathematic support, whilst XSLT v2.0 has better support, but only by a small amount. Therefore most of the functoids in this category can only be implemented in C#.

So if you want to perform a complicated mathematical function (i.e. anything more than addition or subtraction!) you’re better off using one of these functoids, or an external assembly.

Once again, inline C# isn’t the fastest, but given a choice between a slow function and no function, you might not have a choice.

Functoids covered in this category:
Absolute Value Modulo
Addition Multiplication
Division Round
Integer Square Root
Maximum Value Subtraction
Minimum Value Common Code

Monday, February 11, 2008 3:13:46 PM (GMT Standard Time, UTC+00:00) #    Comments [0]  |  Trackback

 

Understanding the BizTalk Mapper: Part 3 - String Functoids#
The String Functoids are probably the most frequently used in maps (in my experience), mainly because they're the most familiar to a procedural programmer (i.e. a C# or VB programmer). However because they all emit inline C#, they perform the slowest so if you want your maps to run faster you're better off using the corresponding XSLT, or implementing the functionality you require in a separate assembly.

Functoids covered in this category:
Lowercase String Left Trim
Size String Right
String Concatenate String Right Trim
String Extract Uppercase
String Find Common Code
String Left  
Saturday, February 09, 2008 12:28:03 PM (GMT Standard Time, UTC+00:00) #    Comments [0]  |  Trackback

 

Understanding the BizTalk Mapper: Part 2 - Functoids Overview#
This whole series of posts started because I wanted to show what XSLT was emitted when using the default functoids provided by Microsoft.
Specifically, I wanted to show the XSLT emitted by the Advanced Functoids. Understanding this XSLT can help in understanding how to use the functoids.
For some reason (as seems to happen with me) the post expanded into a whole series on the Mapper... every time I explain one thing, I seem to want to explain all the things that the first thing is based on... oops.

Anyway, suffice to say that the next 9 posts will cover the code emitted by all of the default functoids provided with BizTalk 2004 / 2006 / 2006 R2.

One thing to realise is that the majority of the default functoids emit inline C# code – which is odd as quite a lot of the functionality can be performed using pure XSLT.

So for each functoid I've shown:
  1. Whether XSLT or C# is emitted
  2. Whether an XSLT equivalent exists
  3. The XSLT or C# emitted by the functoid
  4. Where C# is emitted, the equivalent XSLT to achieve the same functionality (in both XSLT v1.0 and v2.0)
Friday, February 08, 2008 12:03:15 PM (GMT Standard Time, UTC+00:00) #    Comments [0]  |  Trackback

 

Understanding the BizTalk Mapper: Part 1 - Introduction#
In this post:
Introduction
BizTalk Mapper 101
History
BizTalk Mapper in BTS 2004 / 2006 / 2006R2
What happens when a map is compiled
What happens when a map is executed
XslTransform vs XslCompiledTransform
XSLT 1.0 vs XSLT 2.0

Introduction
The BizTalk Mapper is an integral part of the BTS toolkit and, depending on your inclination and experience, you will probably either love, hate it, or not care about it.
At its core, the BizTalk Mapper is simply a visual tool for specifying XSLT (eXtensible Stylesheet Language Transformations).
This XSLT is used to transform one stream of XML data into another stream of XML data. And since a "stream of XML data" = "message in BizTalk", this means transforming one message into another message.

[Aside: you might think that the Mapper allows you to specify multiple source or destination messages, but this is actually a trick - BizTalk creates a special schema with one part per input/output message - there's still only one input/output message]

How well the BizTalk Mapper performs its task depends on the complexity of the transformation you're attempting - and the skill/experience of the developer using the BizTalk Mapper!

Thursday, February 07, 2008 11:38:44 AM (GMT Standard Time, UTC+00:00) #    Comments [0]  |  Trackback

 

XPath – the hidden language of BizTalk?#

For those of you who are unaware, XPath is a powerful query language for XML (ignoring XQuery for now).

In my experience I've worked with a lot of BizTalk developers who have come from a C# (and usually VB before that) background, and so haven't always had the background in XML, XSLT, XSD, and XPath needed to fully appreciate BizTalk.

Because let's face it: before you were working on BizTalk, how often did you need to deal directly with Xml? When writing ASP.NET web services, the .NET framework abstracts away the need to understand XML, leaving you with classes which are handily serialised/de-serialised across the wire for you.

Note: I have come across a similar pattern in the J2EE world – ask many Java developers about an Xml document they received as a request in a J2EE Web Service, and they'll only know about Beans, as the framework looks after the de-serialization from Xml into a Bean for them).

However, BizTalk is a different beastie: it works predominantly with data streams containing Xml.
Therefore the ability to know how to query this Xml without having to resort to C# code is very very important.

BizTalk 2004 and 2006/2006R2 support XPath 1.0 - as at this time, there is no support for XPath 2.0 in BizTalk.
You can get a good idea of the functions available in XPath 1.0 here (which is a useful summary of the W3 Recommendation)

(as a comparison, here's what XPath 2.0 gives you - here's hoping we get this in BizTalk sometime soon...!)

Friday, November 30, 2007 4:53:03 PM (GMT Standard Time, UTC+00:00) #    Comments [0]  |  Trackback

 

Auto-generating BizUnit Test Cases#

This post follows on from yesterday's post: Creating BizUnit Test Cases for comparing Xml Files

If you are using the XmlValidationStep/XmlValidationStepEx BizUnit steps and using XPath validation, then it can be a pain to write all these XPathValidation statements by hand.

So I wrote a utility to generate the XPath statements for you.
In fact, it generates the entire BizUnit Test Case for validating an Xml file.
Once it has been generated, all you have to do is[...]

Tuesday, November 27, 2007 5:59:18 PM (GMT Standard Time, UTC+00:00) #    Comments [1]  |  Trackback

 

Creating BizUnit Test Cases for comparing Xml Files#

BizUnit is a great tool for performing end-to-end testing of BizTalk applications (and can even be used for testing non-BizTalk applications like Web services too).
Although you're not strictly performing unit-testing of your BizTalk artefacts per-se, with the right setup you can get very close to unit testing.

For example if you want to test an orchestration in isolation you can:

  1. Hook your orchestration up to file send/receive ports
    or
  2. Write a façade orchestration which calls your orchestration, and is itself hooked up to file send/receive ports or
    or
  3. Write your own orchestration hosting engine

I don't know many people who'd be crazy enough to do 3) (although I do know one…) but BizUnit is of great help for 1) and 2)

In its simplest form you can use BizUnit to[...]

Monday, November 26, 2007 5:51:27 PM (GMT Standard Time, UTC+00:00) #    Comments [0]  |  Trackback

 

DanSharp XmlViewer#

Updates:
19/04/2018
The source code and MSIs for this tool (inc v1.1) have moved to GitHub here: https://github.com/probertdaniel/dansharpxmlviewer

03/10/2011
The source code and MSIs for this tool (inc v1.1) are now available on CodePlex here: http://dansharpxmlviewer.codeplex.com

22/01/2008
I'll release v1.1 this week - only has minor bug fixes (one of them related to problems with opening files from the recently used file menu).
Also I'm planning to release the source code this weekend.


So here's a little utility that I've been using for myself for a while, but thought I'd release in case it's of use to anyone else.

There are three tasks I perform quite regularly when doing XML or BizTalk development:

  1. Building and testing the XPath needed to retrieve a value from a given request/response document
  2. Validating a document against a schema
  3. Generating BizUnit tests cases (from an Xml instance doc, creating XPath statements to validate for all the data in the document)

This tool does all three of these things:


XmlViewer main form


Generating a BizUnit Test Case

Although there are good tools on the market for doing 1. and 2. (e.g. Altova XmlSpy and Stylus Studio) most of them require installation and are not free.
And none of them generate BizUnit Test Cases!
I wanted a single EXE which would do the most common tasks for me.

Also note that Visual Studio will give you the XPath for an element/attribute in an Xsd, but not an Xml document.
And although you can validate a document in Visual Studio, it's a hassle each time you want to change the Xml document used (right click schema, choose properties, change Input Instance Filename, cick OK, right click schema, choose Validate Instance).

Having a scratch pad to play with XPath queries has been very useful, and is the original reason I wrote this.
But generating BizUnit Test Cases has so far been the most useful... if you use the REG file provided, you can even right-click an Xml file to generate a BizUnit TestCase for it.

There is a command line version supplied as well, so if you need to generate a lot of BizUnit Test Cases, you can do so in a batch file/script.

The utility is written as a single EXE, with no additional files needed (other than .NET 2.0) and it doesn't need to be installed.
I have a copy on my USB key and I run it from there.

Caveat: I use this tool every day, so most bugs have (hopefully) been eradicated.
So my apologies if you come across something I haven't tested yet - please let me know if you do.

Files available to download:
Note: I will be making the source code for this utility available, but need to tidy it up a bit first.
Will add it here when it's ready - probably in a week or so.

MSI Installer (contains all of the below files)
    DanSharp.XmlViewer.Setup.msi (1.8MB)

Executable (Windows version)
    XmlViewer.exe (80kb)

Executable (Command line version)
    XmlViewer.com (80kb)

Registry File (.reg) for adding right-click support to Xml files
    GenerateBizUnitTestCase.reg (2kb)

Documentation
    DanSharp XmlViewer Documentation v1.0.pdf (860kb PDF)
    DanSharp XmlViewer Documentation v1.0.doc (600kb Word 97 Doc)

Source Code
   (Coming soon)

If you come across bugs or have any comments, then leave a comment here, or email me at:
xmlviewerATprobertsolutions.com

Sunday, November 25, 2007 5:45:14 PM (GMT Standard Time, UTC+00:00) #    Comments [1]  |  Trackback

 

BizUnit: XmlValidationStep using .NET 2.0 Schema Validation#
This is something that bit me recently:
The current version of BizUnit (v.2.2.1003.0)  uses the .NET 1.0 XmlValidatingReader to do schema validation in both the XmlValidationStep and XmlValidationStepEx steps.

.NET 2.0 deprecated the XmlValidatingReader class, proposing that you use the XmlReader class and the new XmlReaderSettings class instead.
Microsoft also introduced a slew of bug fixes to the validation logic[...]
Saturday, November 24, 2007 6:43:10 PM (GMT Standard Time, UTC+00:00) #    Comments [0]  |  Trackback

 

Distinguished Fields and Optional Elements#

Thought this might be of interest to a few BizTalk developers out there.

It's quite common practice to use Distinguished Fields in orchestrations, to get/set the value of an element or attribute.

However, if the element you're trying to set/get doesn't exist, then this poses a few problems.

Setting a Distinguished Field
If the element/attribute doesn't exist in the target message, then you'll get an exception, no questions asked – there's no way around this.
Under the covers, the SetDistinguishedField method is called to set the value – and it doesn't check if the element/attribute exists first.
The same thing happens if you use the xpath() function to set a value, and the element/attribute doesn't exist – BizTalk isn't about to modify your message and add the element/attribute for you[...]

Thursday, November 15, 2007 3:09:42 PM (GMT Standard Time, UTC+00:00) #    Comments [0]  |  Trackback

 

Manually sending a response back to a waiting Web Service client#

(or how to do asynchronous processing of a synchronous request/response message)
[Note: this post follows on from my previous post How to validate Xml Documents against schemas in BizTalk. For example, if you validate a message in a pipeline and validation fails, how do you send a response back to a waiting web service client.]

For a while now, people have been struggling with how to asynchronously process request/response messages - basically, how to get away from having a request/response port in an orchestration bound to a request/response  receive location.

For example, Yossi Dohan blogged on this a while back, and came up with a solution involving multiple orchestrations:
http://www.sabratech.co.uk/blogs/yossidahan/2006/06/sync-to-async-conversion.html

The problem revolves around the fact that BizTalk will always demote the EpmRRCorrelationToken context property when your message leaves the orchestration if you try and manually set it yourself.

Sunday, September 09, 2007 11:41:48 AM (GMT Daylight Time, UTC+01:00) #    Comments [1]  |  Trackback

 

How to validate Xml Documents against schemas in BizTalk#
I got asked a question the other day: How would you validate an incoming message against a schema if the message was the request part of a request-response pair and you wanted to return a response if the request wasn't valid?

In the example given, an orchestration had been exposed as a web service, and the requirement was to validate the incoming message. If the message did not validate they wanted to return a response message with an error message in it.

I gave two of the ways I would do it, but that wasn't what they were expecting: they were expecting the simplest (and computationally slowest) way of doing it. And I realised that many people use this mechanism as they don't know there's any other way.

Why do I say this? I'll explain as I give my solutions.
First of all: The solution that was expected was to use an orchestration to do the validation - as the person explained to me, that was the only way to get the response message back to the same "connection" i.e. have it go back out as a response to the matching request.
As you'll see this is not true.
In this post I'll cover the ways to do validation.
In the next post, I'll cover how you correlate the response back to the client who is waiting for a response.

Let me say one thing: BizTalk is not magic. There is no magic (thanks Nakor). There's simply some COM+ applications, some .NET assemblies, instances of a Windows Service, some database tables... and a lot of unmanaged code.
What gets confusing[...]
Saturday, September 01, 2007 11:16:55 AM (GMT Daylight Time, UTC+01:00) #    Comments [0]  |  Trackback

 

Sending HTML emails with embedded images in BizTalk#
Ever wanted to send emails from BizTalk (using the SMTP adapter) which have images embedded in them? I did recently.

Sending HTML emails (or text emails) in BizTalk without embedded images has been done to death – all you need is the RawString message type.

But how do you use the SMTP adapter to send emails with embedded images?
I didn't want to cheat and write a helper class.

I first looked at using the MIME/SMIME encoder pipeline component.
By default the MIME encoder uses a mime Content-Type of multipart/mixed.

So if you have a multipart message with an HTML body part and a single image as a second part, this is what the MIME encoder generates[...]
Tuesday, June 12, 2007 10:01:16 AM (GMT Daylight Time, UTC+01:00) #    Comments [0]  |  Trackback

 

Validating Schemas in WebMethods using Attributes#
One of the most annoying things in BizTalk is when an error is generated in a Receive Pipeline indicating that a message has been suspended because of the content of the message doesn't validate against the given schema (for the moment, we'll gloss over the fact that in an ideal situation this should only ever happen during testing).

Using BizTalk 2006's new Failed Message Routing feature, it's fairly simple to write something which can deal with this situation (using either an Orchestration or SendPort subscribing to the Error Report).
Returning a message back to a Web Service caller indicating that there is something wrong with their message, is non-trivial – it's possible, but requires some in-depth knowledge of how instance subscriptions work.

However, there is an easier way: validating the message at the endpoint (i.e. inside the Web Service itself).
Aaron Skonnard and Dan Sullivan wrote an article for MSDN magazine in July 2003 on how to implement attribute based schema validation for Web Method parameters.

It's a great idea: decorate a Web Method with an attribute, and when the web method is called then any messages received will be validated against their schemas – if they don't validate, then a SOAP Fault is returned back to the caller[...]
Sunday, April 15, 2007 5:44:54 PM (GMT Daylight Time, UTC+01:00) #    Comments [1]  |  Trackback

 

BizTalk 2006 R2 Public Beta released#
So now everyone gets to play...!
Announcement is here.
Download it from here (you'll need to register for Microsoft Connect and/or sign in - but it's free to register).
If you download everything, it's a 558MB download.

This the Beta2 version.

So download it, and go play with all the WCF goodness...

What's new in R2?
I'll have a play with this new Beta and blog about my findings soon.


Tuesday, April 03, 2007 8:25:48 AM (GMT Daylight Time, UTC+01:00) #    Comments [0]  |  Trackback

 

Finding out the last shape executed in a dehydrated orchestration#
You know, I'm really sad that Lee has left the BizTalk team.
'Cause he's one of the few that can come up with gems like this:

"As a "parting gift", I managed to convince the team to let me add a small enhancement to persistence as long as I did not push changes to the UI... As part of default dehydration now, the orchestration engine also persists, in the clear, the name of the shape on which it was blocked"

How is this useful? Well if you don't know, you can't have had to debug too many orchestrations... ;-)
If you have an orchestration which gets "blocked" i.e. gets dehydrated whilst waiting for an operation to finish, or a message to arrive (or any one of the things which can cause a dehydration event to occur) now you can see the name of the shape at which dehydration occurred.
Previous to this, you would need to log the names of all the shapes as they executed, or fire up orchestration debugger to see what the last shape executed was.

So now the race is on to develop a GUI for this...
Pity GotDotNet is no more, as that would have been the perfect place for it.

Also, note that this applies to BizTalk 2006 R2 only, so unless you're on the TAP, you'll have to wait for the Beta or final release to use this.


Sunday, April 01, 2007 9:23:23 AM (GMT Daylight Time, UTC+01:00) #    Comments [0]  |  Trackback

 

BizTalk and Certificate Revocation Lists (CRLs) - Part II#
(This is the third in a series of three posts about CRLs - the first was Web Services and the 15 Second Delay, and the second was BizTalk and Certificate Revocation Lists (CRLs) - Part I).

Note: A lot of the information in this post comes from a great MSDN article located here.

Caveat: My client uses 64-bit servers (AMD Opterons), running 64-bit versions of Windows 2003 R2 and BizTalk 2006. IIS is running in 32-bit compatibility mode (as we use Sharepoint). I haven't yet worked out if the CRL problem occurs on 32-bit servers - I definitely haven't noticed the problem on our 32-bit servers as of yet.

For 2 months, my BizTalk application was working fine. The system passed performance testing, and was deployed on the Live servers in preparation for final connectivity testing.

Then one Monday, last week, the test team complained that they were experiencing sporadic timeouts. On the same day, I was doing some testing on an unrelated BizTalk application on a separate server... and I noticed that I would occasionally get request-response latency approaching 70 secs...

Given that the same day I'd noticed I no longer had access to iTunes Radio from that morning (bah!), I assumed that changes had been made to our proxy sever or firewall. I fired up TCP View on the server I was working on, and there was our old friend SYN_SENT: something was blocking access to the CRL again. I spoke to the Tech Support team and discovered that no changes had been made to the proxy server. Leaving them to check for changes to our firewall and security policies, I decided to do some research into why this delay exists (if the call is blocked) and if there was a way around it. Here's what I discovered (refer to this article for a more in-depth explanation of Certificates and CRLs):

  • Any given Digital Certificate contains a property called the CRL Distribution Point which is a collection of URIs.
  • When a certificate is validated, a CRL retrieval attempt is made using each URI in the list. Retrieval stops with the first URI to return a valid CRL
  • When a valid CRL is obtained, it is stored in the Certificate Store for the Local Machine (under Certificates (Local Computer)/Intermediate Certification Authorities/Certificate Revocation Lists)
  • A CRL is a certificate in its own right and as such, it contains an expiry/update date called the Next Update date
  • If the CRL already exists in the Certificate Store and is still valid then this CRL is used; otherwise an attempt is made to download an updated CRL
  • URI schemas valid for CRLs include http://, ldap://, and file:// - it is the Publisher of the certificate who decides upon the contents of the CRL Distribution Point
  • In large corporations, it is common to use Active Directory (AD) as the provider of CRLs: AD can download the required CRLs and either publish them to a master location, or distribute them to servers that need them

One thing I was curious about was this 15 second delay which kept popping up.

The Xceed Software post I had read had made reference to there being a 15 second delay hard-coded into the WinVerifyTrust API call.

Looking through the documentation for WinVerifyTrust I noticed two things:

  1. Microsoft recommend the use of CertGetCertificateChain for validating a certificate (instead of WinVerifyTrust)
  2. That WinVerifyTrust enumerated a registry key (HKLM\SOFTWARE\Microsoft\Cryptography\Providers\Trust\) to find out what API call to use to verify the trust of the given object

I'm not about to trace what WinVerifyTrust does to actually check the CRL, but I'd suspect that it ends up delegating to either CertGetCertificateChain or CertVerifyRevocation (and I'd bet that internally, CertGetCertificateChain calls CertVerifyRevocation to verify the CRL for a given certificate).

Suffice to say that CertGetCertificateChain will build a chain of certificates starting from the given certificate, and building the chain all the way up to the root CA, and will optionally check the revocation status for each certificate in the chain; whilst CertVerifyRevocation will verify the revocation status for a single certificate.

And both of them take, as one of their parameters, a struct called CERT_REVOCATION_PARA.
The format of that structure is:

typedef struct _CERT_REVOCATION_PARA {
  DWORD cbSize;
  PCCERT_CONTEXT pIssuerCert;
  DWORD cCertStore;
  HCERTSTORE* rgCertStore;
  HCERTSTORE hCrlStore;
  LPFILETIME pftTimeToUse;
  DWORD dwUrlRetrievalTimeout;
  BOOL fCheckFreshnessTime;
  DWORD dwFreshnessTime;
  LPFILETIME pftCurrentTime;
  PCERT_REVOCATION_CRL_INFO pCrlInfo;
  LPFILETIME pftCacheResync;
  PCERT_REVOCATION_CHAIN_PARA pChainPara;
} CERT_REVOCATION_PARA,
 *PCERT_REVOCATION_PARA;

Heh, look, there's a member called dwUrlRetrievalTimeout.

Wonder if that's relevant??? ;-)

The documentation has this to say:

This member contains the time-out limit, in milliseconds. If zero, the revocation handler's default time-out is used.

And what's the revocation handler's default time-out?

Well, Microsoft doesn't specify this directly... but I notice in a related knowledge base post, that a value of 15000 milliseconds is used… i.e. 15 seconds!

So that's as far as we can go with that – unless IIS includes an option to configure this timeout, then we can't change it (and they do, sort of).

Whilst researching this post, I noticed that one solution that is frequently touted is to modify the following registry key:

HKCU\Software\Microsoft\Windows\CurrentVersion\WinTrust\Trust Providers\Software Publishing\State

But that's not much use, as that's for the Current User (hence the HKCU). Great if I was using my own local user account for the application pools, bad if I'm using a non-interactive user account (which we are). Plus I'm not sure this would work for IIS… maybe I'll try it at some stage.

(Note: looks like Microsoft are aware of this issue, because in Windows Vista/Longhorn there's now a Group Policy setting which lets you set this default timeout for non-interactive processes i.e. IIS App Pools!!)

So what's the solution in this case?

Well, unless the technical support guys can work out what they changed to block CRL access (I suspect they turned on authentication on the proxy), we have four choices:

  1. Use Active Directory to store and publish CRLs (which we should have been doing from the start IMO)
    This is Microsoft's preferred way of doing it for large customers.
    More information on configuring CRLs with AD can be found here.
  2. Manually download the required CRL and install it
    This is my preferred solution for this particular issue, and is detailed below.
  3. Disable CRL checking for the server
    This is an interesting one. I'm not convinced that this can be done - there are a few posts about how to do this, including one on how to do it for IIS here.
    However, this seems to be related to certificate exchange for HTTP request/responses, as opposed to certificate validation for signed code, which is a whole different thing. Plus, turing off certificate checking is a rather large security hole as you don't know if a given certificate is still valid. 
  4. Change the default CRL timeout period for CAPI
    I noticed in the Knowledge Base article for an update to IIS 5.0 that new registry keys had been added, including allowing a value called ChainUrlRetrievalTimeoutMilliseconds to be set.

    Then when browsing through the PKI documentation, I noticed a reference to the same registry keys, plus a note saying "this setting was first introduced with MS04-011" (the IIS 5.0 update linked to above).

    So it looks like it is possible to set the default timeout.

    I haven't tried this, so can't verify that it works, but to me it's not the correct solution: the CRL should be available, either from AD or the URL, or by installing it manually – setting the timeout to a lower value seems to be just ignoring the problem, plus creates a potential security hole as you can't be sure that the certificate used to sign code is valid anymore.

Manually downloading and installing a CRL
Needless to say, I thought I'd have a go manually downloading the CRL and installing it – and it worked a treat. Problem solved (at least until the next CRL update is needed, which is August 2007). Still, gives us a breather to get it properly sorted.

Finding the URL to the certificate is easy: look in the certificate details for the CRL Distribution Point, and copy the URL from there. In this case, it's the Microsoft Code Signing Public Certification Authority CRL: http://crl.microsoft.com/pki/crl/products/CodeSignPCA.crl

You can put this URL in a web browser, and download the certificate.

(Note: if you're doing this in Windows Server 2003, you'll need to add crl.microsoft.com to your list of Trusted Sites, otherwise you won't be able to download the CRL file)

Once you have the file, you can install it following the instructions here:


And lo and behold, the problem was fixed.
At least, it is fixed until August 30th 2007 when the CodeSignPCA.crl expires... ;-)
But by then, I'm sure we'll have found a permanent fix!

Saturday, March 24, 2007 8:41:17 PM (GMT Standard Time, UTC+00:00) #    Comments [0]  |  Trackback

 

BizTalk and Certificate Revocation Lists (CRLs) - Part I#

(This post follows on from last week's post Web Services and the 15 Second Delay).


Having accounted for 15 secs of the 45 secs latency I'd been observing in my BizTalk application, I started looking at where the additional 30 secs was coming from.

 

When I looked at my log files, I noticed the entire process from start to end (including calling external Web Services) was now taking just over three seconds. However, when invoking the BizTalk Web Service from a test client, the measured latency was still over 30 seconds. Something wasn't right…

 

I put logging code in the BizTalk Web Service around the ServerProxy.Invoke() call (the method call the submits the received SOAP message to BizTalk).

In my BizTalk application, the BizTalk Web Service is used to submit a message to the MessageBox, via the SOAP Adapter and a static request-response Receive Port using the PassThru pipeline. There is an Orchestration which subscribes to messages coming from this receive port, processes them, and then returns a response back to the receive port (and therefore the Web Service). My logging code logged the time at which the orchestration received a request message, and the time at which it sent a response message.

 

What I saw in my log files was that there was a 15 sec delay between calling ServerProxy.Invoke(), and the orchestration logging that it had received a request, and then another 15 sec delay between the orchestration sending a response, and the return from ServerProxy.Invoke() in the WebService.

Interesting.

 

Our BizTalk Load Test environment is composed of a single BizTalk 2006 Server, a single SQL Server, and SAN disk storage. Given that everything that happens between ServerProxy.Invoke() and an orchestration instance being started is non-user code (i.e. not my code!!), the problem was most likely environmental i.e. network, disk, database, security or "other".

 

So my first step was to attach SQL Server Profiler and log all the database calls going between the BizTalk Server and SQL Server.

If you haven't used Profiler before, be aware that it captures a huge amount of data – going through its logs can be very very painful.

However, in this case we're only interested in calls that have originated from BizTalk – specifically, calls which have originated from one of the Service Accounts that BizTalk runs under. In this case, the service account that the Isolated host instance runs under, and the account that my Orchestration Host's host instance runs under.

 

Why these two? The SOAP adapter runs under the Isolated host, which means that it'll be this Isolated host instance that will be putting the request into the MessageBox. And the Orchestration Host's host instance will be the one which picks up the request from the MessageBox, and then writes the response back the MessageBox.

For the Orchestration Host's host instance, finding out the service account name is easy – just look in the Services console, find the service named something like BizTalk Service BizTalk Group : <host name>, and then note the value in the Log On As column.

 

For the Isolated Host's host instance, we need to check the identity account for the Application Pool under which the BizTalk Web Service is running (if you're using IIS 6 – if you're using IIS 5, you'd use the service account for IIS itself, usually the ASPNET account).

 

So all we have to do is look for entries in the Profile log file which have a LoginName property equal to one of these two account names. Fortunately, you can setup a filter in Profiler quite easily – either before you run the trace (so that only events you're interested in are captured) or after.

 

I chose to capture all events and then filtered on entries which had an EntryClass of RPC:Completed, and a LoginName value of the Isolated host account name.

(Note: because I'm only interested in the SQL statements being executed by BizTalk to call Stored Procedures, I only need to see RPC events – none of the other events are relevant here as they pertain to Audit events, Keep-Alive requests, or Transaction statements).

 

Running through the filtered log means I only have to trawl through 300 odd lines, rather than 10,000…!

What I'm looking for is the time at which the Isolated host submitted the SOAP request to the MessageBox, so I'm looking for a call to Stored Procedure (SPROC) called bts_InsertMessage:

 

 

(Note: You'll notice in the above image that just before bts_InsertMessage is called, there are calls to bts_InsertPredicate_<hostname> and bts_InsertProperty followed by a call to bts_FindSubscription.

InsertPredicate and InsertProperty populate the MessageProps table, and FindSubscription checks whether the combination of these predicates and properties match a current subscription – if so, bts_InsertMessage is called; if not, a "subscription not found" exception is raised).

 

So now I know at what time the SOAP request was placed in the MessageBox: 17 seconds after ServerProxy.Invoke() was called. Now I need to check what time the message was picked up by my Orchestration Host's host instance.

To do this, I filter the Profiler log on the service account name used by the host instance.

However, this time I'm looking for a call to bts_DeQueueMessages_<hostname> which did some work – that is, a call which has a higher duration than the others:

 

 

However, this doesn't clear up the mystery – this just shows that the message was picked up by the Orchestration Host's host instance 4 secs after being inserted by the Isolated host instance.

 

And then something hits me: what if this related to the problems I was having earlier, with the Web Service and the CRL?

What if "17 seconds" is really "15 seconds plus some BizTalk processing time"?

 

I started up TCP View, passed a test message through – and sure enough, there was a CRL call with a SYN_SENT state.

Only this time, the TCP connection was to http://crl.microsoft.com/.

What are the chances that the BizTalk assemblies are digitally signed? Specifically Microsoft.BizTalk.WebServices.ServerProxy.dll, the assembly used by BizTalk Web Services.

Let's have a look:

 

 

And if we drill down into the Certificate Details, and look at the CRL Distribution point we see:

 


 

So there we are – because the code is hosted in IIS, and is digitally signed, IIS will attempt to validate the certificate, which involves attempting to download the CRL for the certificate.

 

Unlike my Web Service, I don't think I'd have much luck in convincing Microsoft to *not* digitally sign the BizTalk assemblies… ;-)

 

So, we set the default proxy on the BizTalk server (using the proxycfg command line tool discussed in the previous post).

 

And sure enough, the delay in the BizTalk Web Service went away.

The 45 secs latency dropped to 2 secs… and then dropped to 0.5 secs once I did some low-latency tuning.

 

So there we go. Now the only reason my client has this issue is because they don't want to have separate BizTalk servers which just run the BizTalk Web Services (i.e. dedicated BizTalk Servers, most likely in a DMZ, which don't do anything but put received SOAP messages in the MessageBox – all the other BizTalk servers would be behind the firewalls and in the secure portion of the network – this is Microsoft's recommended way of doing it, but is costlier in terms of licensing).

 

We decided we could live with the Load Test server having access to the internet via a proxy server, so applied this change to all our BizTalk servers (both Live and Test).

Saturday, March 17, 2007 10:22:34 PM (GMT Standard Time, UTC+00:00) #    Comments [0]  |  Trackback

 

Web Services and the 15 Second Delay#
Background: The large(ish) BizTalk system I've been working on for the last few months consumes a few external Web Services. Whilst doing some early performance tuning, I noticed that I was getting pretty serious request-response latency from the primary BizTalk Web Service... Like over 45 secs, instead of the expected 0.5 – 2 secs...

Fortunately, I routinely put tracing/audit calls in my BizTalk code which includes timing info - and I always log the times around orchestration entry/exit... So I was quite surprised to see that a request to one of the external Web Services was taking 15secs to respond... especially as the web service was hosted (temporarily) on the same server...

Although I didn't have the source code, with the help of Lutz Roeder's trusty .NET Reflector I was poking around the innards within minutes… and couldn't find anything that would cause a delay of 15 secs.

What's more, after further testing I found that the problem only occurred the first time after IIS was restarted. After the first time, it would return within 2 secs (still not very good, but much better). One thing I noticed about the Web Service was that it made use of some code in an unmanaged assembly (old school C++ DLL)...

On a hunch, I wrote a simple test Web Service which simply referenced some functions in the DLL (using the DllImport attribute)... and bang, a 15 sec delay before the Web Service constructor was called.

I also tried the same trick in a console app... and there was no delay.

By this time I was fairly stumped. This issue only happened on our Load Test server: it didn't happen on the Dev or Test servers. And it seemed to only happen to code hosted under IIS.

I figured it was time to dig a bit deeper and see what IIS was doing: starting up Process Monitor (to monitor file and registry access) and TCP View (to monitor network activity), I called my test Web Service.

Filtering the Process Monitor logs to the w3wp.exe process (the IIS Worker Process for Application Pools) I noticed that w3wp was reading a lot of security and cryptography keys from the registry:

Hmm. Interesting.

Then I looked at the TCP View log. And I noticed something very interesting coming from the w3wp process: a connection was being attempted to http://crl.versign.com and was being blocked.

How do I know it was being blocked? Well, the TCP View log showed a SYN_SENT, which appeared and was visible for 15 secs before disappearing:

Now if you're not familiar with how TCP works, then you might miss the significance of this: When a TCP client attempts to contact a TCP server, the first thing performed is a handshake. This involves the client sending a SYN, the server responding with a SYN-ACK, and the client sending a SYN-ACK-ACK.

In the TCP View tool, the rightmost column indicates the state of the local TCP connection end-point. You'll see that in this case, the state is SYN_SENT: this state indicates that the local TCP client is waiting for a SYN-ACK response from the SYN it sent.

Why would this be? Well, many reasons, but mostly because the address can't be reached e.g. a proxy server is required, or a firewall rule is blocking the call.

And what's the relevance of crl.verisign.com? A CRL (Certificate Revocation List) is used to indicate which Certificates (issued by a Certificate Authority) are no longer valid.

And this is only done when you are using an assembly which has been digitally signed.

(Note: Microsoft became really really serious about CRL's back in 2001 when VeriSign accidentally released two Microsoft certificates to a third party – which meant that the third party could sign code that appeared to have come from Microsoft).

So what does this have to do with my Web Service and a 15 second delay?

Looking through the assemblies in the Web Service, I noticed that the unmanaged assembly was digitally signed. Aha! So when the unmanaged assembly was referenced in a Web Service, IIS would hand off control to the Windows Cryptography API (CAPI) and ask it to verify the certificate. And this involves retrieving a CRL for the certificate.

After doing some more poking around, I found this blog post (from one of the developers at Xceed Software, a company specializing in .NET components).

The thing that drew my attention was this comment:

Some people behind a very strict firewall, ignoring outgoing connections instead of rejecting them, had a timeout while loading our assemblies. That timeout (at least 15 seconds) is hardcoded in WinVerifyTrust!

Eureka! That's exactly my problem!

And it explained why it was only happening on our Load Test server: Our Dev and Test servers have access to the internet, whilst the Load Test server does not: it has no proxy server configured.

To test this theory, I configured the Load Test server to use a proxy server. If you haven't done this before, you can't do it via Internet Explorer settings – that only configures a proxy for application which go via wininet.dll i.e. the Internet Explorer API, which uses the proxy server settings in the Internet Explorer/Internet Options/Connections/LAN Settings panel.

CAPI, however, uses winhttp.dll (Microsoft Windows HTTP Services). And to set the proxy server for that, you need to use the proxycfg command line tool which sets proxy server settings for winhttp.

If you run proxycfg –u, this will import your Internet Explorer proxy server settings:

So I did this, restarted IIS and presto! The problem went away.

However, giving the Load Test server access to the Internet is a fairly large security hole (at least it is in our network, where the Load test server is behind both the firewall and DMZ). 

So, we contacted the vendor who supplied the unmanaged DLL, and managed to get them to release a new version which wasn't digitally signed.

Which solved the problem of the latency in the Web Service (which accounted for 15 secs of the 45 secs latency).

Solving the remaining 30 secs latency is the subject of next week's post.

Saturday, March 10, 2007 12:01:13 PM (GMT Standard Time, UTC+00:00) #    Comments [0]  |  Trackback

 

All content © 2020, Daniel Probert
On this page
This site
Calendar
<February 2008>
SunMonTueWedThuFriSat
272829303112
3456789
10111213141516
17181920212223
2425262728291
2345678
Archives
Sitemap
Blogroll OPML
Disclaimer

Powered by: newtelligence dasBlog 2.3.12105.0

The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.

Send mail to the author(s) E-mail

Theme design by Jelle Druyts


Pick a theme: