Hey Friends, This is my very old very neglected blog. I thought I would put a quick update here for continuity sake. This blog was started 2006 when I was working as a specialist BizTalk guy for Microsoft Australia. My company https://www.cloudster.com.au (Cloudster Pty Ltd) still has a small team doing BizTalk related work but I myself have transitioned to a focus on Integration-as-a-service work in the ecommerce and logistics industry. My company https://www.cloudsterconnect.co is where you can find out more about my I-a-a-S work. All our work there is a culmination of everything I learnt doing BizTalk projects and InvoiceSmash. But taking the technology and packaging and democratising access. Its all good fun. The hot new thing for me these last few months is our project IDAConnect (https://www.idaconnect.com) which is a connector for ecommerce customers using Shopify (WooCommerce etc) who are selling on market places such as The Iconic. Mark

I have no idea how or where or from whom I picked up this turn of phrase. It might even have been one of my own sparks of originality. But we at Burch Technologies now refer to any situation where someone suggests connecting a DEV environment to UAT environment or doing UAT testing in a PROD environment (or any other suggestion to mix and match environments) as “crossing the streams”. 

This video clip should explain why we use this phrase and why we feel it is a bad bad bad thing to do.

https://www.youtube.com/watch?v=jyaLZHiJJnE

Ah a classic moment of comedy from the 80s…but you know what’s not funny…crossing the streams between environments. Nothing good ever comes from it…and lots of bad things happen when you do. Crossing the streams bad is how bad it is when you cross the streams.

It violates so many principles of good IT management that almost none of the reasons ever given actually justify the costs, risks and complexities that follow if you do.

- It muddies the waters when it comes to deployment procedures from DEV to UAT, or UAT to PROD. If part of the solution is already deployed to part of the UAT environment its so easy for people forget the little things like that patch that was done last Thursday morning gets forgotten and now there is a regression in PROD even though it was fixed last Thursday and now the project manager thinks the IT team are bad at their jobs and why is it so hard to get software right in enterprise IT…. In other words, the outcome of every software development project needs to be nice, clean software with nice, clean, repeatable deployment procedures. If the environments are ‘mix and matched’ then the possibility of clean repeatable deployment procedures is gone…never to seen again. 

- Security across environments should ALWAYS be kept completely isolated. The PROD security regime should be identical to the UAT security regime except for the credentials themselves (of course).  It follows then that anyone who suggests crossing the streams is implying that a UAT environment is given access to a PROD environment and that person is insane. Just say no. Don’t do it.

- The moment you cross the streams you have to think carefully about data syncing issues and data reset issues. So you crossed from UAT to PROD during the UAT phase…that's cool…I guess (/sarcasm).  What is the data reset procedure in the downstream system so that we can re test things after going to PROD. Or worse…as a BizTalk developer…the last thing I want to see in DEV is real life customer data…”oh wow, there’s someone I know…what are the chances…” kind of thing.

- So you’re building a new system and the plan is to use one environment for UAT testing and the promote it into PROD at go live…ok, what happens right after production go live when we need to UAT test a small business requirement change?  What’s that you say, there’s no UAT environment now…well your small business requirement change is now 6 weeks away from PROD release while we build a new UAT environment – Btw I hope that the guy who built PROD before he quit did good documentation because otherwise how will be sure PROD and UAT are the same...  No, build a proper UAT and PROD environment at the outset so the solutions can be promoted through each environment in a hygienic and repeatable way.

- So you were up late last night fixing some crappy bug on a PROD  system. Good job, you fixed it in prod, NOW FIX IT IN UAT THIS MORNING AS WELL THANKS, BEFORE YOU GET DISTRACTED BY THAT BUTTERFLY. You don’t want continental drift to separate UAT from PROD so everyone has to be militant about keeping UAT and PROD identical in all respects.

Don’t cross the streams folks…just don’t do it.

 

Putting this here in case anyone else runs into this nasty bug…

Was configuring Service Bus for Windows Server (The on premise version of Azure Service Bus) for a client and was getting a bunch of errors during the configuration phase.

“An object or column name is missing or empty. For SELECT INTO statements, verify each column has a name. For other statements, look for empty alias names. Aliases defined as "" or [] are not allowed. Change the alias to a valid name.”

Basically a meaningless random SQL error. My first reaction was that it was because we had a Availability Group SQL listener in front of the main SQL node…but that was a red herring.

What it actually was was we were using an Admin Group name like “Service Bus Admins – UAT”. It was correct as per AD and passed validation in the configuration wizard but blew up at the creating database stage of the configuration process.

I ask the Active Directory Admin to change the name to “ServiceBusAdmins_UAT” and it worked like a charm.

It was either the spaces or the hyphen that SQL didn’t like.

 

Yesterday Nevatech Sentinet 3.0 was released.

What is Sentinet?

“Sentinet 3.0, a comprehensive SOA and APIs management platform for Microsoft on-premises and Windows Azure cloud, adds advanced support for RESTfull and SOAP services”

 

Go here the announcement from Nevatech:   Click here

We’re resellers for this product so call us if you are interested in trialling it or require a demo. http://www.burch.com.au/contact

If you’re like me and been around the Microsoft ecosystem for a while you will have heard the term “unsupported” or “not supported” from time to time. I believe that this term is too overloaded to be useful and we need to use more nuanced terminology….

Next time you hear a sentence like “We wanted to do X but it was unsupported by Microsoft” make sure you ask the follow on question which is  “why is it unsupported?” The answer to the second question completely changes the outcomes and options available to you.

  You want to… Support
Statement
reason/clause meaning/outcome
1 connect BizTalk Server to Exchange Server using an old sock and sticky tape its unsupported because it doesn’t work stop trying that its technically impossible.
2 install BizTalk 2009 on Windows Server 2008 R2 its unsupported because BizTalk 2009 was never tested by Microsoft on Win2008 R2
(Win2008 was the latest OS supported)
there’s is nothing technically stopping this from working  because the compatibility between the two OSes is very good. Proceed at your own risk.
3 connect to SAP from BizTalk using a third party connectivity library its unsupported Microsoft doesn’t  know anything about that third party thing you could probably get the third party thing working, but Microsoft isn’t responsible if there are problems with it.
Proceed at your own risk.
4 run BizTalk production in a cloud VM using SPLA licencing its unsupported Microsoft licencing doesn’t allow it. nothing technical stopping you – good luck with your software audit.
5 flim flam the interwidget using Microsoft BizTalk and a cloud powered kite and key. its unsupported you wanna what? That’s kind of out of left field – not sure its supported by Microsoft  nobody knows what flim flamming the interwidget means so they’re trying to stop you before you hurt yourself.

So you see above that the 5 statements are factually correct (ie not supported) but the reason is actually just as important as the support statement so always ask the follow on question….

 

At Burch Technologies we’re pretty excited to announce that we are now a reseller for Nevatech Sentinet in Australia and New Zealand.

As BizTalk/ESB/SOA and integration specialists Sentinet is a very nice compliment to the work we do for our clients. In fact I would go so far as to say that Nevatech Sentinet is the piece that’s been missing from the Microsoft SOA technology stack for about the last 3 years. As a technologist I was pretty interested to get my hands on the Sentinet software and take a look. I was pretty happy when my feature wishlist was pretty much covered off completely.

This is what Nevatech themselves say about their product…

“Nevatech Sentinet™ is a flexible, lightweight and scalable
API management platform that promotes integration through the use of SOA standards. It is designed to connect, mediate, and manage interactions between services across
the enterprise or in the cloud.”

At Burch Technologies we embrace technology that makes the work that we do easier and enables our clients to achieve  SOA capabilities quicker, and this definitely fits the bill.

So Microsoft announced that the CTP for the next release of BizTalk is available internally to Microsoft and others with special access.

CTP from 18 July 2012

TAP program begins 24 July 2012.

Based on past timelines for these releases we can predict that the public beta will be out in about Oct 2012 and RTM will be Jan-Feb 2013. They’ve made their promises to be Windows 8 release wave + 6 months – so it kind of lines up. And the BizTalk team have surprised everyone in the past by releasing a month earlier than anticipated -meaning that their ability to set and meet milestones is pretty darn good now.

I’m just prognosticating here, don’t hold me to any of this.

The name of the release is currently BizTalk 2010 R2, but my bet is that it’ll be renamed to BizTalk 2013 before RTM. I called it… :-)

Below is a copy and paste of the release email from Microsoft. It lists out the improvements…and its a pretty good looking list…

Improved productivity with new Microsoft Platform support

Customers can now leverage the latest and greatest platforms, such as Windows Server 2012 RC, SQL Server 2012, Visual Studio 2012 RC. All new BizTalk projects will target .Net Framework 4.5 RC by default. The CTP also provides support for latest LOB versions enabling customers to use BizTalk for integrating their applications with the latest versions of SAP, Oracle and SQL Server. The new adapters provide a seamless experience to enable hybrid connectivity, all done via configuration. The CTP provides native support for ACS authentication and is extensible for other authentication mechanisms.

· Platform support

o Windows Server 2012 RC, Windows Server 2008 R2

o SQL Server 2012, SQL Server 2008 R2

o Visual Studio 2012 RC

o Office 2010

o Support for latest LOB versions

§ Support for SQL Server 2012

§ Support for SAP 7.2

§ Support for Oracle DB 11.2

§ Support for Oracle EBS 12.1 …

· Adapters

o WCF-WebHttp adapter, to consume REST service or expose REST service

o SB-Messaging, for sending/pulling data from Service Bus Queues/Topics

o WCF-NetTCPRelay, for hosting relays or sending data to NetTCPRelay end points

o WCF-BasicHttpRelay, for hosting relays or sending data to BasicHttpRelay end points

Better B2B with schema updates

EDI standards evolve and one of the key investments made in this new BizTalk CTP is to ensure that we support the latest B2B standards natively. This enables you to transact messages based on the latest versions of EDI protocol.

· B2B enhancements to support latest standards natively

o Support for X12 5040, 5050, 6020, 6030

o Support for EDIFACT D06A, D06B, D07A, D07B, D08A, D08B, D09A, D09B, D10A, D10B

o HL7 2.5.1

We are working on further schema updates such as HL7 2.6, these will be enabled in the BizTalk 2010 R2 Beta.

Improved Performance

The CTP provides performance improvement for certain key scenarios. In case of two way MLLP adapter scenarios where ordered delivery is set, the tests have revealed up-to 5X performance improvement so far in our environments. We have also made enhancements in our engine to improve the performance in ordered send port scenarios.

Building hybrid applications

Today, there is an increase in the adoption of hybrid application scenarios where some components of an application run in the cloud and some other components/LOB applications remain on-premise. It then becomes important to integrate between these components and leverage the richness of both worlds. In this CTP release, we enable hybrid connectivity by providing first class support for integrating with Azure Service Bus Queues/Topics/Relays. We are introducing the following adapters

· SB-Messaging, for sending/pulling data from Service Bus Queues/Topics

· WCF-NetTCPRelay, for hosting relays or sending data to NetTCPRelay end points

· WCF-BasicHttpRelay, for hosting relays or sending data to BasicHttpRelay end points

Integrating with Azure Service Bus entities is now just a few configurations away!

Integration with RESTful services

One of the other prevalent trends in the market today is the proliferation of RESTful services. Almost all new services, as well a lot of services created previously, have a REST interface exposed. For example, all services in Windows Azure, data market place, Salesforce, etc. have support for REST services. With this CTP release, we are making it really easy for you to integrate RESTful services with BizTalk Server using the new WCF-WebHttp adapter. All the REST operations like GET, PUT, POST and DELETE are now supported natively. It gets better. We received community feedback during and post TechEd conference that there should be a way to expose REST services as well from BizTalk. We listened to your feedback. Along with consuming REST servicesm we are also really excited to announce that you now have an early preview to exposing REST services from BizTalk Server as well in this CTP.

BizTalk Server in Azure Virtual Machine role

All the above enhancements are available right away for you to preview with BizTalk Server in Azure Virtual Machine role. Setting up a new BizTalk Server environment usually involves long lead time to procure hardware, get the dependencies in place, set up the server, etc. This means long lead times before you can get started with your new BizTalk Server environment. We are now leveraging the power of the cloud and the richness of Windows Azure to provide an experience where you can get up and running with your BizTalk Server environment in matter of minutes and move your existing applications to the cloud without making any changes. Furthermore, the CTP provide improvements to the BizTalk multi machine configuration and now you can do this using some basic configuration settings with the click of a button in a single machine, without having to go and configure BizTalk Server Group in each of the individual nodes.

 

Our speciality at Burch Technologies over the last few years has been tackling the hardest integration scenarios out in Enterprise IT land. We often get called in as the ‘project turn around’ guys, after somebody else has screwed something up. We’ve turned around several BizTalk/Integration projects for several of our clients in Sydney over the last half year.

There are basically 3 ways to screw up an Enterprise Application Integration (EAI) project.

1. Not getting the High Level Architecture right.  What I mean by this is, EAI/ESB/SOA has its own patterns, its own ‘paradigm’ and its own traps and pitfalls. For instance, if your way of thinking about integration is moving files (flat or xml) around then you’ll be completely oblivious to the advantages of using Line of Business Adapters or database (WCF SQL, Oracle) adapters for transactional, real time, event driven processing. Where ever possible, we avoid moving files around – ask me why - because the disadvantages add up quickly. And this is 12 years of BizTalk/integration experience talking here. Another way of not getting the high level architecture right is to focus on object types (customer, invoice, whatever) and not the messaging sequence required to get that customer object updated in the ERP correctly FOR THAT SYSTEM – If only it was as easy as just calling an update operation. I could go on.

Another gotchya I’ve seen is falling in love with an ESB (Enterprise Service Bus) type of architecture and not understanding the trade offs and costs associated with it. BizTalk folks get polarised by this discussion – needlessly in most cases.

2. Not getting the code/implementation right. Some of the worst BizTalk implementations I’ve seen have been done by “BizTalk Contractors”.  This is someone who might have a fair bit of experience on BizTalk but only in one particular area. For example I know of a situation where a “BizTalk Contractor” who knew B2B and EDI well but not Line of Business systems told a client that the BizTalk needed to receive an XML message from a SQL Server database (Which is not true, the database adapters in BizTalk can interoperate natively) and so the client needlessly went down the path of custom coding stored procedures to return XML. One of the advantages of using Consultants over contractors is that as Consultants we move across more client projects, more types of systems and see more types of architectures and patterns than others. And beware of the general consultant who thinks she knows integration because they did a 3 month project. You don’t get a Dynamics CRM consultant to implement a SharePoint site. So don’t get a .Net/WCF consultant to implement BizTalk – call us instead Smile.

3. The third major way of not getting projects right is not getting the infrastructure and non-functional requirements right. BizTalk infrastructure requires a careful design. Putting in the installation disk and clicking next 20 times is a recipe for disaster. Additionally, there are simple things we do at Burch Technologies to make our systems easy to manage which payoff in the operator satisfaction/total cost of ownership department. For example when BizTalk is polling for data out of a LOB system the naive implementation is to SELECT * FROM Customers WHERE Status = ‘FOO’. We don’t do that. We wrap that in a storedprocedure, do SELECT TOP PARAMX * FROM CUSTOMERS WHERE Status = ‘FOO’. Then in the BizTalk receive location we can tune the throughput of that process by playing with the polling interval and the number of records (PARAMX) processed in each interval. Its nice, and it stops flood (eg end of month) scenarios from hosing BizTalk and other down stream systems. Naive implementations will never account for this type of non functional niceness. Even better is to not poll for data at all – again, ask me how Smile.

Get those three high level components right and success is much easier to attain.

As with anything, there is a continuum of expertise. Starting with naive through to wisened and battle scarred. Make sure you get the right level of expertise to match the problem space – that's my (self serving) advice.

My other  advice is beware of the person or product brochure that tells you integration is easy. At Burch Technologies, we make it LOOK EASY for our clients, but we’re under no illusions about how difficult EAI is.

My next blog posts will be about how cloud technologies are making integration easier. Easier being a relative term.

 

I was recently compiling a list of software systems Burch Technologies Pty Ltd (aka Burch Consulting) has integrated to over the last few years. It’s a pretty impressive list, even if I do say so myself.

I’d thought I share the list on my blog here in an effort to get some Google/Bing attention.  Some of these systems were pretty straightforward since they were webservice’s based. But some had no real integration capabilities at all, namely Pronto (Australian ERP system) and BasePlan (another Australian ERP system (I detect a theme here)). Those are the most interesting/challenging projects because you start from scratch…

Software System

Integration method

Complexity (1-10) 10 = Hardest

· Dynamics Ax

BizTalk adapter

6

· Dynamics CRM

Webservices

4

· SAP

BizTalk adapter

8

· JD Edwards

BizTalk adapter/webservices

9

· Oracle E-Business Suite

BizTalk adapter

7

· Oracle Databases

BizTalk adapter

4

· SQL Server

BizTalk adapter

4

· Seibel

Webservices

3

· B2B (various)

NA

 

· EDI (various)

NA

 

· Objective

Webservices

7

· Pronto

Custom Integration Implementation*

10

· Dynamics GP

BizTalk adapter

5

· Chris21

Custom Integration Implementation*

8

· Baseplan

Custom Integration Implementation*

10

· BasWare

Webservices

3

· Maximo

Webservices

7

· HP Trim

Webservices

10

· SharePoint

Webservices

3

· Intelledox

Webservices

6

· CargoWise EdiEnterprise

Webservices

5

· Onyx

Webservices

4

If you have one of these systems and you need some help integrating to them contact us….

Mark