From a developer’s point of view, you are right in saying that these “visual” environments are really hard to work with. SharePoint 2010 Workflows, which I use, throws out every best practice around creating good enterprise software – no automated testing, no code reuse, unreadable software… Anything more complex than an out-of-the-box template can be painful to maintain, as you are experiencing.

But from the business’ point of view, workflows have massive benefits – at least, in theory. To quote from this white paper, the Efficiency, Accountability, Control and Ease of use of an automated workflow provide huge productivity gains. Imagine how much more inefficient a simple approval or on-boarding process would be without this automation. Also, the very act of defining a workflow is valuable to an organization that is trying to get control over their business processes.

The current state of workflow software is not the business’ fault. They just want to make their lives easier, and workflows are great for that. I would mostly blame us, the IT department:

1. For not being more transparent with the business about how complex and fragile the system really is. We hide all the complexity.
2. For not being able to “scratch our itch” with intuitive, simple workflow solutions. This is probably more of a rant against large enterprise packages like SharePoint and SAP, but they are better than the custom solutions out there

There is only one real benefit, yet its huge:
Separation of Concerns.

So, instead of process orchestration logic being embedded in our system, it becomes and external configuration. A map, basicly. You can change it (much more) independently, you can have multiple processes, multiple versions of processes, multiple versions of multiple processes running at the same time, and that’s all out of the box in any decent solution.

Historically, the concept of SoC have won many times – starting from the Unix principle “do one thing, but do it good”, and being applied again and again – like having dedicated server components like ESB, different persistance systems, caching, load balancing, monitoring, like splitting CSS from HTML, etc.

Your business process and its flow rules are often orthogonal to your data, UI “screens” or users “hierarchy”. So, it makes perfect sense to develop and change it separately from the other aspects of the system. That was the premise on which BPM has appeared in early 1990s.

Since then, many of tools and languages were created to support this concept, with most well known being BPMN – a graphical language for creating “flowcharts” that directly map to processes. While people complain that its large and unwieldy (having over 100 symbols in vocabulary), and advocating modern approaches like S-BPM (has only 5 base symbols), current industry practice is to stick to BPMN or its derivatives, subsets or siblings.

You do not look pleased with BPMN:

Programmers here appear to be doing a significant amount of work in the lines and boxes layer, which to me just looks like a really crappy, really basic visual programming language.

But its not that bad ) There is theory behind it. And version 2.0 took some good insight from 1.0 shortcomings.

If you’re going to go that far, why not just use some sort of scripting language?

Imperative paradigm and scripting languages are not always the best answer. As you’ve probably seen in declarative languages (like HTML, CSS, SQL, Drools or internal ones of Nginx, Graddle and Maven, Puppet etc) the resulting code can be much smaller and cleaner, than a version written in “decent language, like Java or C++“.

As for your other point:

as far as I can tell, are visual programming languages at this point, complete with conditionals and loops.

have you looked into the Events and Triggers ? BPMN is a language and you have to learn it before using, or at least get familiar with it.

Under the hood, BPMN is XML, so you can edit it by hand, or generate. And you can version control them, because XML is text-based. However, just having an XML that can be translated into flowcharts, does not sound like its goona help you, and that’s correct – writing your own parser or editor for it is a tough and expensive task with questionable benefits.

Luckily, there are already tools at the market which do exaclty that.

Activiti is free, and quite popular among both developers and busines owners, because of its initial price (zero), availability of information and humbleness. The last point is really unique, as Activi only focuses on managing your business processes, without trying to tie you in with whole-package solutions. Also, its open – so you only need to know Java and REST to get it up and running. The drawback is that client side, integration and application/businesss logic and the whole architecture are left to developer, so if your develpment team is weak – prepare for the fail. Total cost of ownership can be surprisingly high for a free tool 😉

On the other side of spectrum is Pega (Pega PRPC), the reigning king of BPM systems (according to Gartner and Forester), which looks surprisingly good for its age. This and-a-kitchen-sink behemoth is even capable of CRM, OCR and (if i’m not mistaken) speech recognition capabilites, predictive analytics, business rules management and WYSIWYG UI editor. It comes with a serious a price tag, though. Not only does it cost a fortune, but all the development is being done within a web-app, which means you have to use browser, which is IE8 (plus some plugins, plus ugly hacks, like using Excel for editing data tables). Also, Java, Javascript or HTML/CSS editing is also being done withing web browser – so say goodbye to unit tests, IDE code highlighting, refactoring and all your programming toys you’ve leaned to love.

Good side of it ? you can implement complex system WITHIN WEEKS [PDF, see page 22]. And yes, result is not guaranteed.

IBM have somewhat recently (accoring to the enterprise time pace) have bought Lombardi, and is now offering very competitive solution (but then you will have to buy everything ibm, you’know). Appian is young vendor who has interesting insights and positive feedback, but the way they are written (two extra DSL languages in addition to visual one) just doesn’t appeal to me.

There are other players, and their solutions. Most of them are plain horrible. Like – your eyes, brain and heart would literally bleed, when you simply look at them. So, trust your guts and don’t make your developers and user hate you.

Closing note:

BPM system is the same for for processes, what Photoshop is for images. Don’t be afraid that its visual. Don’t make it do the job not suited for it (remember web sites created entirely in Photoshop, which were next to impossible to edit?). Keep it simple and don’t make bugs 😉

Years ago, before most of us were born, software developers had to write their own code to store data. If you needed to save program state, well, that was seen as part of the function of the code, so many software developers ended up writing code to organize data and save and read it and so forth.

And then someone realized that this was something that happened a lot — that, the logic to save, organize, read and search data was actually a component that was so commonly used that it should be it’s own component. And we got databases.

In the last 10 to 15 years, IT departments have been realizing that the logic to choreograph and/or orchestrate business processes is so common that it should also be it’s own component, which has lead to all sorts of different workflow tools.

There are 3 primary benefits of a workflow tool:

1. Time needed to make and deploy changes: you can develop and change the logic of a workflow without the same technical risks that you have with changing a piece of code.
2. Transparency: the business logic in a BPM based system is immediately available to and readable by the business analyst, whereas only developers will be able to read the business logic in “code-based” systems.
3. Reuse of technical components: BPM tools are often used in conjunction with systems that have a Service Oriented Architecture. By separating the business logic from the technical components — especially for enterprise systems which must implement hundreds or thousands of different business processes — you’re able to reuse the technical components while spending relatively little time on developing the business logic (with a workflow tool).

However, one of the most common problem that I run into with workflow and BPM tool usage, is that developers still try to “bury” business logic in non-transparent code.

What I see, all the time, is developers still trying to add the business logic in the most technical ways possible, instead of the most transparent way possible. This is natural, because developers are, by their very nature, much more comfortable with code than with a workflow tool. Furthermore, the more logic you keep in a technical way, the more you’ll be needed as a developer. Unfortunately, this is the worst thing a developer can do to a BPM system because he or she is undercutting the primary benefits of using BPM.

Lastly, most BPM tools aren’t far enough that business analysts can develop the workflows themselves: however, that’s what the goal is. Ideally, business analysts would develop the workflows/business process diagrams and developers would only work on the technical components that are called by the workflow engine.

Below statements are my personal experience using workflow tools, specifically Oracle BPM Suite(10.3G & 11G). First to specify, your question is focusing on workflow tools which are enabling modeling executable process models, these tools are part of Business Process Management Systems(BPMS). This specific process modeling is definitely developing and you can refer to it as a visual programming language.

The main benefit is the agility of understanding and changing process logic

With business process models you cover a visual explaination of the process logic and at the same time an executable integrative component. This allows faster on-and offboardings of programmers, because less documentation on the transitions, conditionals(Gateways or Business Rules) and process flow in general has to be documented, since its part of the development.

Additionally you have attached reporting/monitoring capabilities, what you would have to develop individually per each application, which is covered by most BPMS.

Moreover in most BPM development environments service adapters are prebuilt(e.g. JMS, Web Services, JDBCs etc.), allowing to develop middleware solutions faster in a step by step interactive integration, reducing human errors in coding.

Following workflow platform does achieve a lot of benefits mentioned above – Platform based approach for automation of workflows

The Value

The value proposition is that workflows can be created or altered quickly and easily to suit the changing nature of a business. An important part of realising this value proposition, is that business processes themselves are units of resource in the system.

Workflows as units of resource, means that a business process is defined as a single ‘unit’. To understand what I mean by this, consider any computer program written for a business. It implements a business process for sure, but the logic for that process is likely to be spread around the source code to a certain extent. A workflow tool should allow the process to be defined in one well contained place. It could be in one single config file, or extracted from one visual diagram, or it could mean that the workflow is in a single module of code that can be plugged in, perhaps even swapped or configured on the fly.

Why the Value may not be realised

This value proposition can be undermined by the difficulties of covering non-vanilla cases, integrating with UI technology which changes very rapidly, bad practices such as using the workflow tool as a wrapper only and putting the real logic in code anyway.

Poor design of the tools themselves may be a limiting factor. An example would be a tool that requires all parameters passed between processes to be in a Java Map, with restrictions on what values you can actually put on the map, instead of just plain old method parameters (I’m thinking of one of the more popular tools in particular that does this).

I think it is probably fair to say that IBM as a big player with a large technology eco-system, fairs better than the others. If they also control the UI technology, and database and SOA technology that is used in conjunction with the workflow tool, they stand a better chance of coming up with an eco-system that integrates together nicely, and creates an opportunity to really make use of this idea.

It is definitely true that the effort of writing the interfacing between a workflow tool and the other parts of a system can completely negate the whole value proposition. It is always worth considering if there is a better way of doing things.

Programming is Workflow

The truth is that programming (in imperative languages at least) IS WORKFLOW already. You may have code that implements workflow which is to do with handling the system technology; accessing files and running SQL queries and so on. You may have code that implements business workflow; setting the state of a document and passing it to a reviewer for example.

Recognizing this and designing your code to cleanly split these separate concerns is hard to pull off completely, but you can certainly get a long way with doing just that.

I agree with you, sometimes we end up using these tools because someone else decided it was a good idea, and they are just too complex and making our job harder. I don’t think that is always the case, it needs some careful consideration to decide if its worth it or not.

It is not very clear which tool you are using. I guess you may be referring to the general set of tools called Business Process Modeling tools. There is a good reason for using such tools. Any quality business define its functions in terms of processes and analyst as well as business experts can draw such processes comfortably (until you tie them with standards…). You can create such processes at the conceptual level without any knowledge of web programming and if you have the right tool, you may be able to convert the process into a working application too (experienced people must come on board for this magic to go into production of course). So the idea is good.

The objective of the visual tools is not just documentation of the processes. The use of the tools is meant to help professional re-engineer processes and, in times, run simulations to discover points where the processes are less efficient than desired so that plans can be put in place to remove the bottlenecks.

There is a standard way that many companies use today called BPMN 2.0 (Business Process Modeling Notation). I do recommend that you get into understanding this notation if your tool is using it.

The Business Process Management Body of Knowledge is a famous resource, that you may want to consider as well.

The above covers the business side. The technical side requires SOA and BPEL, I am not sure I can provide advice on those here and now though.

In simple example by history.

Stone age

At first you had small menhir companies where people just where told what to do, and they did it. Sometimes things wend wrong, and Person X or Y got blamed (never sure who really did it).

Next internet and Email was invented.

People now wrote others what to do, and those people often had problems with their Email, read it wrongly, or just simply deleted an Email without ever reading it; so often things where not don blaming it on bad Email

Workflow evolved out of administration
By standardizing the actions, finally people could see at what phase who halted a process, and at the same time, got a digital overview of what actually had been done.
This worked nice till people wanted to change standard processes, or till some unknown X Y person caused an improper database request resulting in database corruption, sending production back to stone age..