IEP-103: Code Implementation - Apache Ignite (2023)

I WENT	IEP-103
author's	Andrei N. Gura
Sponsor	Andrei N. Gura
Made	23 maja 2023 r
Country	SKETCH

To perform calculations on a cluster, the product must provide a flexible mechanism for deploying code at runtime. Since every application evolves over time, code implementation should be able to run smoothly and without cluster downtime or node reboots. It's also a common approach to use versioning and an isolated execution context to run code in a cluster.

Must be able to deploy/undo code to run using CLI and REST API. This proposal does not require a public Java API for code deployment because code deployment is generally related to development activities, not the application development process.
The deployment/deployment process must be reliable in the context of a distributed system. When a deployment unit is deployed in a cluster, there are no nodes that cannot access that deployment unit or request that deployment unit from another node. If the deployment unit is not deployed, there are no nodes with access to that deployment unit.
Due to network unreliability, checksum authentication should be used when transferring deployment units over the network.
Deploying code requires no downtime and must be done at run time.
The deployment unit must have an appropriate human and machine identification mechanism.
Management of the deployment unit should provide states that reflect the lifecycle of the deployment unit and avoid issues with code execution concurrency and isolation.
It should be possible to obtain a list of deployment units deployed on a cluster or a specific node and the status of each deployment unit.
The Compute API must provide the ability to define the execution context: a list of dependencies (versioned deployment units) for the execution of a given task. Enables isolated code execution.
It should be possible to avoid using an explicit version of the plant unit when defining the execution context. Instead, it should be possible to set the latest version of a specific deployment unit (LATEST notation).
Compute API still only supports parameter types and return values defined inCompute API in Apache Ignite 3: Phase 1 - Running Simple Tasks Remotely: Interoperability.
It should be impossible to undo a deployment unit while work is in progress that requires those deployment units.
It should be impossible to start a new task that requires a specific deployment unit when prompted to deactivate the deployment unit.
It must be impossible to complete the task if the required deployment unit is missing. An error should be returned to the client code.
If the deployment unit is not deployed and there are no jobs running that require that deployment unit, all class loaders and classes loaded by those loaders must eventually be collected by the GC.

deployment unit- a set of artifacts needed to run code on cluster nodes (for example, Java class files, JAR files, configuration files, and other resources). An implementation unit (one or more) defines the context for executing a specific code (for example, executing a task).

Deployment unit ID- random text identifier (e.g. name) used by the user and the Ignite 3 cluster to manipulate the deployment unit (e.g. deploy\uninstall). The ID has to please MavenGroup IDnaming conventions (https://maven.apache.org/guides/mini/guide-naming-conventions.html).

Deployment unit version- a similar versionbig.small.patchFormat (semantic version). The deployment unit ID and deployment unit version uniquely identify a particular deployment unit. In fact, the version is rather redundant information, but it can be useful to the end user (e.g. "last used" scenario or in the development process).

Deployment unit state- defines the state of the deployment unit in the cluster or node. Deployment unit state lets you:

simultaneous deployment of the same deployment units.
prevent code that has scheduled deployment unit declustering from running.
monitor implementation progress.

The base deployment unit directory- the directory of the file system where all deployment units should be placed.

The deployment unit directory- directory under the basic directory of inserts, in which the given insert is placed.

Overview of deployment unit operations

There are two perspectives of deployment unit operations: user perspective and cluster perspective.

From the user's point of view, 3 main operations are allowed on the deployment unit:

Deploy the disk to the cluster
Roll back the deployment (delete) the deployment unit from the cluster.
Get Status of Cluster Deployment Units - A list of all deployment units in the cluster along with their status (defined below).

It would also be useful to be able to get a list of all deployment units on a specific node. The main scenario of this operation is troubleshooting.

From a cluster perspective, a deployment unit can be deployed to the entire cluster, but not to a specific node. Therefore, an additional operation is needed: disk deployment on the target node (on demand).

Command line interface

unit deployment

Deploys a deployment unit with identificationto the agglomeration.

The $ignite3 entity is deployed--version--Pad|

Options:

--version - sets the insert version ().
--Pad - path to the JAR file () or to a folder (), which contains all the necessary class files and resources.

remove the unit

Deletes a deployment unit with identification cluster (and all available nodes).

Undo the $ignite3 unit--version

Options:

--version - sets the insert version ().

A list of disks in the cluster or node

Prints a list of all deployment units or specific deployment units with identificationon the cluster or node and its state.

Unit List $ignite3 [] [--version]] [--node] [--status [UPLOADING [, STARTING [, …]]]]]

folder structure

All deployment units should be placed in the deployment unit's home directory, which is a subdirectory under the Ignite working directory:

/gear

In the base directory, create a directory for each deployment unit. The name of this directory must be the same as the deployment unit ID, and a nested directory must be created for the specified version of the deployment unit.

Example:

- implantaten- foo.example.job- 1.0.0- 1.0.1- foo.example.task- 1.0.0- 2.0.0

Deployment unit operations

A deployment unit is deployed in a cluster if it is loaded on at least a majority of CMG Raft group nodes. This allows the system to order the required deployment unit from these nodes on demand. A matching record exists in the metastore.

A deployment unit is deployed to a node if it is loaded on that particular node (during regular or on-demand deployment). A matching record exists in the metastore.

Implementation unit operations should take into account the fact that some processes may run concurrently, so the design should avoid running concurrent processes to avoid race conditions and issues such as ABA.

To meet all requirements, the following conditions are introduced:

DepOpId- Deployment operation ID. It should avoid concurrency issues. It must be unique. Metastore version number

nodeDURrecord- the status of the deployment unit on the node. abstract structurenodeDURecord{depOpId, state, (node1, node2)}means the state of the implantation unit iscountryw onsnr 1minr 2i is the ID of the deployment operationDepOpId.

clusterDRecord- status of the deployment unit in the cluster. abstract structureclusterDURecord{depOpId, status}means the deployment unit is in a clustercountryi is the ID of the deployment operationDepOpId.

Deployment unit state

The status specifies the status of the deployment unit in the cluster. The following statuses are allowed:

SENDING- The disk is being deployed to the cluster/node.
IMPLANTED- unit is deployed in cluster/node.
OUTDATED- a delete command has been initiated for the device and will be removed from the cluster/node soon.
TO REMOVE- Disk removal from cluster/node is in progress.

Unit Deployment Process

The deployment unit must be loaded on most nodes of the CMG Raft group and must include a leader in the current mandate (Figure 1). This allows the system to detect the leader change and position the units correctly; the leader is always part of the majority during the upload process.

While most nodes are available, there is at least one node with a deployment unit loaded, and that deployment unit can be deployed to another node on demand. Otherwise, the cluster itself cannot function properly.

Figure 1

During the upload process, do the following:

Select the majority containing the available nodes, including the leader.
Supplement the selected set of target nodes with user-supplied nodes if necessary.
AssignDepOpIdmetastore version number.
Create a metastore record for the specified deployment unitclusterDURecord(depOpId, SUBMIT). This operation may fail because another process has already created a record for the same deployment unit in any state.
Start transferring files to target nodes and create a metastorage registry for the specified deployment unit and nodenodeDURecord(depOpId, UPLOADING, nó).
When the deployment unit is loaded into the node,nodeDURecord.statusshould be changed toIMPLANTEDCourage -nodeDURecord(depOpId, DEPLOYED, nó).
Once the deployment unit is loaded into the target nodes,clusterDURecord.statusshould be changed toIMPLANTEDCourage -clusterDURecord(depOpId, ENTERED).

A deployment unit is considered deployed to a cluster only whenclusterDURecord.status == IMPLANTED. This means that at least one node is available that can be the source of an on-demand deployment unit download.

On-demand unit deployment process

If code execution starts on a specific node (due to a Compute API call) and that target node doesn't have the required deployment unit, you can request it from a node that already has the required deployment unit. This means it has the required implant unitclusterDURecord.status == RUNand at least one node is truenodeDURecord.status == RUN. If these conditions are met, the following steps should be taken:

Add a metastore record for the target node and deployment unit wherenodeDURecord.status == LOADINGminodeDURecord.depOpId == clusterDUStatus.depOpId. Node should avoid simultaneous races between different threads. It is reachable without a metastore operation because there is no other node to start the on-demand deployment process.

Notice that I knowclusterDURecord.status == IMPLOYED && nodeDURecord.status == LOADING=>on-demand deployment is in progress for the specified deployment unit and node.
When the deployment unit is loaded on the target node,nodeDURecord.statusshould be changed toIMPLANTEDCourage.

The node is restarted during the disk deployment process

If the target node was restarted during the transfer process, the node should find all deployment units fromnodeDURecord.status == LOADINGfor restart and if nodeclusterDURecord.status == RUN all such deployment units must be removed from the node. The corresponding states should also be removed from the meta store for this node. Deployment units can be reordered if needed, and in that case deployment will start on demand.

Note that if the target node is in the majority, the deployment coordinating process must ensure that another majority node is selected as the target, otherwise the restarted node will repeat the process.

Insert removal process

The deployment unit must be removed from all cluster nodes. To achieve this, you must implement the following:

ChangeclusterDURecord.statusDoOUTDATEDvalue. This operation may fail because another process has already changed state toOUTDATEDzTO REMOVEvalue. You also cannot start the deployment process while the deployment process is still running.

After this step, the deployment unit is not available to run new code. Ongoing code execution can still use this deployment unit.
The target storage event should fire on all target nodes due to the changeclusterDURecord.status.
After the target node receives this event, the system must changenodeDURecord.statusDo OUTDATEDCourage.
The node waits for all running code executions that depend on this deployment unit to complete. After all code executions are completenodeDURecord.statusshould be changed toTO REMOVECourage.

From then on, it is impossible to use the implementation unit for code execution, neither for new tasks nor for old tasks (the latter is impossible due to the invariant that all old tasks are completed).
For every changenodeDURecord.statusDoTO REMOVEvalue, the system can pick up the event from the metastore and check if all nodes have itnodeDURecord.status == DELETE. If the condition is met thenclusterDURecord.statusshould be changed toTO REMOVEAlso.
You can now delete each target node's deployment unit and then its status records.
For each deletionnodeDURrecordrecord from the metastore, the system is able to receive the event from the metastore and verify that there is nonenodeDURrecordrecords for a specific deployment unit. Now the system would do itclusterDRecordregistry for the deployment unit.

Note that if a deployment unit is deleted, no class loader is associated with that deployment unit. Finally, the class loader should be downloaded by the GC and all classes should be removed from the JVM. A critical requirement is to avoid memory leaks associated with loading/unloading multiple classes.

Node restarts when removing disks

If the target node was restarted during the deployment unit deletion process, the node should find all deployment unitsclusterDURecord.status == OUT OF ACTUALzclusterDURecord.status == DELETEto the restarted node and complete the Deployment Unit removal process as described in the previous section.

Validation after node restart

Each betting unit withnodeDURecord.status == RUNmust be verified to avoid that a deployment unit is removed and redeployed with the same ID and version as the deployment unit, but with different content (kind ofProblem z ABA). OnodeDURecord.depOpIdcorrespondclusterDURecord.depOpIdmust be the same. Otherwise, the deployment unit must be removed from the node.

For everyonenodeDURecord(depOpId, SUBMIT)appropriate recording clusterDURecord(depOpId, ENTERED)zclusterDURecord(depOpId, SUBMIT)should exist. Otherwise, the invalid deployment unit must be removed from the node.

For everyonenodeDURecord(depOpId, IMPLOYED)appropriate recordingclusterDURecord(depOpId, ENTERED)zclusterDURecord(depOpId, SUBMIT)should exist. Otherwise, the invalid deployment unit must be removed from the node.

For everyonenodeDURecord(depOpId, OBSOLETO)znodeDURecord(depOpId, DELETE)appropriate recordingclusterDURecord(depOpId, OBSOLETO)zclusterDURecord(depOpId, DELETE)should exist. Otherwise, the invalid deployment unit must be removed from the node.

code execution

Execution context

The user must be able to have different execution contexts that can use different versions of classes with the same name. This context is similar to the web application context in servlet containers, but the context in the Apache Ignite Compute API is defined by a list (in predefined order) of deployment units that the task depends on.

Use Case 1: Legacy to Specific

For example, an Apache Ignite cluster is shared between two teams that nevertheless share a common set of shared libraries and logic-related parent libraries. For these use cases, there are 3 deployment units:

payment: 1.0.0
time1:1.0.0
team2:1.0.0

In this casepayment: 1.0.0deployment unit contains most of the application logic while deployment unitstime1:1.0.0miteam2:1.0.0 they contain many specific units and often reuse classes from the parent implementation unit. The general approach here is to look for classes in the parent libraries and then in the specific libraries. Therefore, each team will define the execution context as follows:

Team 1 -[parent:1.0.0, team1:1.0.0]
Team 2 -[parent:1.0.0, team2:1.0.0]

Each team's business logic is completely isolated, and shared libraries are available to both teams.

Use case 2: replace the parent with a specific one

In this case, the user has an application and wants to replace some of the application's business logic. So you have the following deployment units:

applications: 1.0.0
applications: 1.0.1

In this caseapplications: 1.0.1replaces any logicapplications: 1.0.0and the user specifies the execution context as[application:1.0.1, application:1.0.0].

Ignite class load string

The Chain Ignite class loader is a class loader that resolves classes in dependency definition order (first to last).

Class loader hierarchy

Figure 2

For user-defined classes, loading and running are the responsible class loaders. Class loaders are built in the hierarchy shown above.

Bootstrap, extensions, and system classloaders are provided by the runtime, and each classloader first delegates classloading logic to its parent classloader.

This approach is not suitable for all user use cases. For example, it is impossible to override the behavior of some classes. Apache Ignite 3 provides a more flexible classloader behavior that can be controlled by a user-linked Ignite classloader. In general, the Ignite class loader behaves like the web application class loader in popular web application containers (e.g. Apache Tomcat):

The Ignite class loader does not allow class overrides for the following packages:
- org.java.*
- org.javax.*
- org.apache.ignite.*
If the system tries to find a class from the packages listed above, the Ignite class loader must first pass this logic to the parent class loader (in our hierarchy, it's the system class loader).

The Ignite class loader allows you to replace any non-system non-Ignite class. If the system tries to find a package class not listed in the previous entry, the Ignite class loader should try to find the class in the appropriate implementation unit and pass this logic to the parent class loader on failure.

Calculation API changes

`Version`

VersionThe entity should be entered into the public API with the following properties:

name- main version, integer.
kleiner- small version, integer.
adjustment- patch version, integer.

It is also acceptable to use the "pre-release" field (e.g. beta, RC) (seesemantic version), but not required.

final versiona special value must also be available. This value indicates the version number that has the highest value.

The subject must be similar.

`Deployment unit`

Deployment unitThe entity should be entered into the public API with the following properties:

name- deployment unit name, string value.
version- deployment unit version, instanceVersionsubject.

`Run calculations`koppel

all methodsamount to,executeLocalizedmistreamwhat couponClassas a parameter should be removed as it is unobtainableClassinstance to a non-boot, system, or extension class loader job. Instead, the methods that get the job class asRepeatYou should use it.
all methodsamount to,executeLocalizedmistreami need to haveDeployment units as a parameter to define the execution context (class dependencies and load order).

Resolving dependencies and loading classes

normal behavior

When code execution starts due to a call, one of the methodsamount to,executeLocalizedzstreamfollow these steps:

Get the classloader identified by the arrayDeployment unitinstances passed to the called method.
If the classloader doesn't already exist, create one:
1. If the deployment unit is deployed and can be used to run code, use the location to load the classes
2. If the deployment unit is deployed to the cluster but not to the node, start on-demand deployment.
3. If the deployment unit does not exist, bidClass not found exceptionorder“. Insertion unitdoes not exist”message whereis a concatenation of ID and version (e.g.com.example:1.0.0)
1. If the implementation unit exists but cannot be used for code execution, prayClass not found exceptionorder“. Insertion unitcannot use: [clusterStatus=, node state =]”message.

Verify that each deployment unit is deployed:
Verify that the deployment unit can be used to run your code:

Load the task/job class passed as a parameter to the Compute API method. In this pointClass not found exceptioncan be run from the class loader.
Run the necessary code.

You can use a reference counter to detect that some code is running and using a class loader. The counter should be incremented when the task starts and decremented when the code finishes executing.

LAST semantics

The only difference between normal behavior andLASTthe semantics is the need to resolve to the latest version of the available implementation unit before executing any code. This means that two consecutive calls to the same job/task can lead to different classloaders being created, as a new version of the deployment unit can be deployed between these calls.

It is not clear what to do when reconfiguring a CMG as the new CMG may consist of nodes unrelated to the previous CMG.
user experience Probably a good idea to addclearlycommand of operations in deployment units. This command is similarto developbut ignores all conditions and just removes all deployment and state entities from the metastore. This can lead to errors during code execution, but it may also be the only way to remove the deployment unit in case of unexpected problems.
user experience Remove all versions of the deployment unit for each version. This may be useful in the development phase.
user experience Redistribution. It can be useful to have deactivation and atom placement units. But due to the asynchronous nature of the implementation process, it can be difficult to implement it correctly for a good UX.
UX, maintenance, support. Adding the ability to validate deployment units is useful. We could dolistcommand fulfilled--implementflag eg. There are at least two cases of this function:
1. Validation of invariants (no UD in odd status)
2. Validation of DUs implemented in checksums to detect:
  1. file corruption
  2. Consequences of the ABA problem
Choosing between lazy and restless unit placement (per unit placement) can be very useful, including automatic placement of units in a node. It's also a good idea to apply some kind of forced bet on a subset of the nodes (specified by the predicate). Will be designed a bit later.
It seems possible to implement Java lambdas as a sort of temporary implementation unit with a limited lifetime. Will be designed later.

[RFC] Compute API in Apache Ignite 3: Phase 1 - Perform remote tasks easily

Key	Restore	T	Made	updated	Due to	assignees	Reporter	Priority	Priority	Priority	Priority	P	Country	Solution

Landing...

update

Key	Restore	T	Made	updated	Due to	assignees	Reporter	Priority	Priority	Priority	Priority	P	Country	Solution