ceronio.net

My vision/pipe dream: Building an application server…

For a long time, I have dreamed of building an application server for business software, one which simplifies many aspects of the software development process.

The big Idea

On a high level, the idea is to create a database-connected application server for business applications.

The application server hosts not only the business applications, but also the development environment in which the business applications are written, as well as the monitoring, software logistics and other related tools to manage the application server; all these being applications themselves. (The software logistics capabilities will support promoting software from one environment to another, e.g. from Dev to QA and from QA to Production).

(With regards to hosting the development environment, there is a conceptual similarity to a Smalltalk environment, if that helps to provide an idea of what I’m aiming for).

For creating the business applications, a domain-specific language (DSL) will be created. The development environment (meaning the editors and associated tooling) and other system-related applications, will also be created in the same language. A debugger should also exist (and by implication the language should be debuggable).

The language itself is not intended to be a general-purpose language, but should be geared toward building general-purpose applications, both business and technical, and needs to expose the system-level services (e.g. transaction handling, database integration, etc.), while also providing type checking and safety.

The DSL is compiled to bytecode to run on a VM and the bytecode resides in the database.

Any software on the system is instantly debuggable. Because the tools are part of the system, you don’t need to connect a separate system to start debugging.

The software that is built on the system is part of a global repository. All libraries/programs built inside the system are visible to all other libraries/programs in the system without the need for “importing” something.

The system also comprises a global metadata repository, or dictionary, that provides all globally defined types, data structures, data elements and database tables that were defined via the development environment, and makes them visible to all code written in the system.

The system is a multi-user system. The user management is part of the base system. Logged in users are able to launch applications based on their profiles. Both developers and business users inhabit the same space.

Technical Challenges

Building such a system requires a VM. The JVM seems to be a good choice. I have personally tried to avoid Java most of my career, but am sold on the idea that, as some have stated (more or less) it is the most sophisticated and reliable piece of software machinery around.

Scala seems like an excellent fit to write the system runtime, DSL compiler (and perhaps the debugger) and to serve as a compilation target.

The system must be able to arbitrarily load applications (in the form of compiled JVM class files, or bundles thereof) and unload them again cleanly. I understand this is technically challenging in the JVM, but not impossible. (OSGi may be the answer).

Debugging the DSL looks like quite a challenge. My limited (and perhaps naive) knowledge of program compilation in general, and compiling to a target language, suggests inserting some code before each compiled DSL statement to facilitate step-through debugging and being able to switch on debugging in any running process.

It requires a front-end. A browser-based front-end where the GUI runs as a standalone application in the browser is probably the way to go. (From the user’s perspective, it’s just a website, but the GUI application is conceptually a standalone piece of software that simply renders the screen based on information received from the server and sends user input to the server; there is zero business logic on the client side). Here, ScalaJS seems like it could be a good fit.

There are a great deal of pieces required to make the puzzle work. The above list mentions a few of the ones I can think of right now.

No more CI/CD?

The general approach to developing server-hosted applications is built around an “outside-in” model, where developers create local environments on their machines, that must reflect a live system and satisfy all the dependencies. They build and test their applications locally, then deploy them to “real” systems.

When multiple developers are involved, this becomes very convoluted process that can be fraught with errors, for which many smart solutions have been built, like continuous integration and continuous delivery (CI/CD), to solve the software logistics problems like conflicts between developers’ changes.

To me, these represent the same types of workarounds that have been built in the form of “programming patterns” to solve problems brought about by the language choices of their day, mostly centered around Java. Ironically, it is mostly for systems built in Java that these software logistical solutions exist today.

By contrast, having a system that is “extensible from the inside”, so to speak, with a global repository, and which can be shared and worked on by many developers simultaneously, can eliminate the need for CI/CD and other complex DevOps practices.

An extensible Platform

A system as proposed here is not limited to building business applications as described above. The system itself, utilizing the tools and language, can be used as a base on which to build further application frameworks which benefit from the ease of development and transparency (e.g. in the case of system monitoring) that the base provides.

A framework can be anything like an ORM, a system for defining business entities, a web framework etc. There is no limit on the possibilities. But having it as part of the platform means that developers have a unified, single point of entry for developing, testing, running and monitoring apps which lowers your overall cost of ownership because it prevents you from having to invest in many auxiliary processes and services, while being able to focus on building business solutions.

It’s been done before…

This idea is not new, by any means.

The inspiration for this idea comes from SAP’s ABAP application server, which is used to power almost all Fortune 500 companies, and has the largest market share for ERP (double its nearest competitor, Oracle).

I have been working with ABAP for almost all of my career, since 1997 and I believe that ABAP (the platform, not just the language, as many people think it is), more than vigorous marketing or head-spinning is SAP’s secret weapon; a winning formula for producing and running business software.

ABAP has been around for decades and carries many warts from its early days, but SAP still heavily invests in its continued development and modernization, not only to support their existing business software, but also to build new business software and technical solutions.

An ABAP system shares some conceptual similarities to a Smalltalk system (as alluded to above) in that the capability for extending the system resides in the system itself. The implications are huge. It not only greatly simplifies the development workflow, to facilitate the building of business applications, but whole new frameworks can be built on top of it, and many such have been developed over the years.

OK, so now the secret has been laid bare. But who is going to invest into copying the secret formula?