Eclipse Che

    2019-04-04

    Eclipse Che

    Update 2022/04/01. RedHat 介绍了 DevWorkspaces 3.0 的架构升级；新架构基于 Operator 及 DevWorkspace CRD. 近年的各种 in-house Cloud Development Platform 也差不多都是类似架构，比如 LinkedIn RDev. 19 年蚂蚁筚路蓝缕起步云研发时、啃着 Che6 的代码；如今 Che 的 GitHub 组织也迁移了，Java Stack 在云的滚滚浪潮里一点点在落幕了

    Update 2022/03/01. Eclipse 基金会发布了一个俄乌冲突的声明（#StandWithUkraine），并介绍了 Che 的名称渊源——乌克兰城市 Cherkasy

    Update 2022/01. Devfile 这个工作空间编排 Spec 的子项目，加入了 CNCF: “The Devfile is a CNCF project contributed by Red Hat, AWS and JetBrains.” JetBrains 不知道什么时候凑热闹进来了，但从 Maintainers 里看还没有

    DevWorkspace 3.0

    The core of Workspaces used to be a Java REST web service named che-server, which provisioned pods and other objects to run dev environments

    • Scalability & High Availability:
      • The che-server could not scale horizontally (it wasn’t possible to run 2 instances concurrently)
      • The new ws-engine is a k8s controller (which runs behind the api-server that is designed to scale horizontally)
      • The data is persisted in an etcd key-value store (HA)
    • k8s universal API: workspaces are k8s objects. Useful traits of this architecture include:
      • k8s clients such as kubectl or the OpenShift console can manage them
      • they are secured via k8s RBAC
      • they can be validated and protected by admission webhooks
      • the devfile spec is automatically generated from the API
    • Simpler design:
      • The ws-engine has a single responsibility, which is to manage workspace resources
      • Communication between components happens asynchronously using ConfigMaps and Secrets rather than a REST API
      • Devfiles are IDE agnostic and don’t include plug-in definitions anymore. IDE specific configurations are managed in separate files as extensions.json in VSCode
    • An opportunity for refactoring: The new engine has been written from scratch, and we took the opportunity to address some of the main issues reported:
      • Simpler configuration for admin
      • No hard dependency on RedHat’s SSO
      • Simpler network model and TLS cert management with one unique gateway (Traefik)

    Figure1: Che workspace creation flow

    Figure2: Che workspace arch

    Workspace Server

    • Creates, manages and orchestrates developer environments that are running on a cloud container platform
    • Both the Che server and each workspace have their own embedded RESTful APIs

    Workspace

    • A workspace is usually termed as a local directory with projects and meta-information that the IDE uses to configure projects. In Che, a workspace is the developer environment, which contains Docker containers, Kubernetes pods, and a virtual machine or localhost. Environment variables and storage volumes are part of the workspace. The developer environment also contains projects, project commands, and resource allocation attributes.
    • Che defines the workspace to be the project code files and all of their dependencies necessary to edit, build, run, and debug them. In our world, we treat the IDE and the development runtime as a dependency of the workspace. These items are embedded and included with the workspace, wherever it may run. This compares to classic workspace definitions which may include the project code, but require the developer to bind their IDE and use their laptop localhost ot provide a runtime.
    • A workspace contains one or more runtimes. The default runtime within our workspaces are Docker containers, but these runtimes can be replaced with other types of “machines” that offer different characteristics. The advantage of Docker as runtime type allows users to define the contents of their runtime using Dockerfiles, for which we can then dynamically construct workspace runtimes without the user having to learn a lot of complex Docker syntax.
    • Each workspace is defined with a JSON data model that contains the definition of its projects, its runtimes, its IDE, and other necessary information that allows a Che server to create replicas. Workspaces can also have their internal state snapshot and saved in a registry, so replicas can be created from the original template, or from images that contain modifications made after a user started working with the workspace.

    Project

    • Projects are mounted into the runtimes. A workspace can have 0..n projects, with each project mapping to 0..1 remote version control repositories such as git, subversion. Projects are mounted into the workspace. Each project has a “type”, such as “maven”, which when selected will activate a series of plugins that alter the behavior of the workspace to accommodate that project type.

    Environment

    • The workspace runtime environment is a set of machines where each machine is defined by a recipe. The environment is healthy when all the machines successfully start and the installers execute jobs. The environment is defined by a recipe that can have different types. The environment and infrastructure validate a recipe.

    Machine

    • The runtime environment has a minimum of one machine that can be a Docker-formatted container or a Kubernetes pod. You can create multi-machine environments with as many machines as your project infrastructure requires. Each machine has a configuration and start policy. Machine crashed and start failures are signs of an unhealthy environment. Machines communicate by using the internal network, service:port

    Recipe

    • A workspace environment is defined by a recipe. The recipe can be one of the following: single container image / Dockerfile / Docker Compose file / Kubernetes list of objects with multiple pods and services

    Bootstrapper

    • The bootstrapper starts the installer script after the first process is executed in the machine following the CMD or ENTRYPOINT. The role of the bootstrapper is to start the installer scripts with a set of parameters and a configuration file. The bootstrapper is a small binary compiled from Go code.

    Installer

    • The purpose of the installer is to install software and services, start servers, and activate agents. The workspace agent, executive agent, and terminal servers are important to the IDE and workspace. The language servers, SSH installer, and other servers bring new functionality to a workspace. The bootstrapper executes installer scripts that prepare the environment and checks for dependencies.

    Volume

    • A volume is a fixed amount of storage that is used to persist workspace data. Workspace projects are automatically mounted into a host file system by default. A user can define extra volumes for each machine in the environment. Docker volumes, Kubernetes persistent volumes (PVs), and persistent volumes claims (PVCs) are examples of volumes.

    Web-based IDE

    • Provides a web-based UI for developing code, browsing files, using Git, etc. Eclipse Theia / Jupyter etc.

    Plugins

    // TBD

    CLI

    • Utilities for mounting desktops to workspaces, generating Che workspaces from git repos, and managing the lifecycle of Che server
    • Build Docker CLIs Your Users Will Love
    • The CLI has a number of features including offline installation, cross-platform operation, debugging mode, backup / restore, and version upgrade management. Custom assemblies can inherit from the Che CLI to create their own. For example, Codenvy’s implementation of the CLI keeps the same syntax, but adds additional magic around adding and removing nodes from a cluster – all with the same simple syntax of docker run codeny/cli start
    • We are particularly proud of this achievement and think the community can benefit from thinking about using Docker as not just a runtime execution engine, but also a vehicle to standardize how OS-specific software is installed in a cross-platform way.
    • Docker Launcher: The Docker launcher is a container that launches the Che server within another container
      • The launcher verifies that the appropriate Docker client is configured, configures Che through environments variables, and handles configuration nuances between different forms of Docker such as boot2docker, xhyve, and Docker on Linux.
      • This allows us to offer you a Docker run syntax that is identical for every operating system.

    SPI (Service provider infrastructure)

    • Infrastructure definition: Definition of the physical or virtual infrastructure coupled with networking and file persistence mechanisms
    • Environment configuration: A set of machines that can be represented by one or multiple containers/pods sharing the same network and defined by an environment recipe supported by the infrastructure
    • As part of the SPI creation, the events mechanisms that allows Che’s server to communicate about the workspace’s state to the various clients has also been improved. The events from the workspace’s runtime and each agent are now decoupled. Once the workspace’s agents start, each of them might not be available and ready at the same time – but as soon as one agent is ready for the end-user, the IDE enables it – without waiting for the other agent’s to be ready as well. This ensures a better user experience and a faster loading sequence.

    Eclipse Che Development Process

    • Milestone plans
    • Eclipse Che has generational advances (A.x.x), major releases with features (x.A.x), and bug fix releases (x.x.A).
    • Each release is associated with a milestone target.
    • We develop in three week agile iterations (Sprint) with a milestone release attempted at the end of each sprint.

    Eclipse Cloud Development (ECD)

    ECD is a Top-Level Project (TLP) at Eclipse Foundation that provides open-source implementations of standards, services and frameworks that enable developing for and in the Cloud.

    Focus on:

    • Protocols and Technologies for provisioning cloud development platforms, cloud developer environments, and the interaction of clients with distributed services
    • Frameworks for building, loading, managing, running, and executing distributed tooling extensions with browser-client interfaces
    • Extensions for programming languages, build systems, frameworks, ALM and code assistant tools
    • Tools and Cloud IDEs that orchestrate developer activity and development workflow across distributed services
    • Delivering industry-oriented, ready-to-run technologies

    ECD will establish commonality between its consistent projects by establishing common building blocks, protocols and APIs to facilitate interoperability. The effort features four projects:

    • Orion: Led by IBM, provided an IDE for interpreted languages such as JavaScript (2011 EclipseCon)
    • Che: Led by Codenvy and providing a cloud-based development environment, particularly for compiled languages such as Java
    • Dirigible: Led by SAP and featuring rapid application development for database-style applications
    • Flux: Led by Pivotal, a collaboration development middleware framework which integrates desktop and Web-based tools so changes in a desktop IDE are reflected in the Web-based tools

    Eclipse Dirigible

    Eclipse Dirigible is an open source project that provides IDEaaS as well as runtime containers integration for the running application. The environment itself runs directly in browser and therefore does not require additional downloads and installations. It packs all the needed components, which makes it self-contained and well integrated software bundle that can be deployed on any Java based Web Server.

    Target applications built with Dirigible are atomic yet self-contained cloud based modules that cover end-to-end vertical scenarios. These applications comply with the Dynamic Applications concepts and structure.

    The overall process of building Dynamic Applications lies on well-known and proven principles:

    • In-System Development
      • known from micro-controllers in business software systems. The major benefit is working on a live system where all the changes made by a developer take effect immediately, hence the impact and side-effects can be realized in a very early stage of the development process.
    • Content Centric
      • known from networking, in the context of Dynamic Applications it proclaims that all the artifacts are text-based models and executables scripts stored in a generic repository (along with the related binaries, such as images). This makes the life-cycle management of the application and the transport between IT landscapes (Dev/Test/Prod) simple and straightforward. In addition, a desired effect is the ability to setup the whole system, only by pulling the content from a remote source code repository, such as Git.
    • Scripting Languages
      • programming languages written for a special run-time environment that can interpret (rather than compile) the execution of tasks. Dynamic languages existing nowadays as well as the existing smooth integration in the web servers make possible the rise of the in-system development in the cloud.
    • Shortest turn-around time
      • instant access and instant value became the most important requirement for the developers, that’s why it is the major goal of our tooling.

    In general, a Dynamic Application consists of components, which can be separated in the following categories:

    • Data Structure
      • the artifacts representing the domain model of the application. There is no intermediate adaption layer in this case; hence all the entities represent directly the database artifacts - tables and views.
    • Entity Services
      • domain model entities exposed as web services, following the modern Web 2.0 patterns and scripting language capabilities.
    • Scripting Services
      • general purpose services for utilities, custom modules and extensions.
    • User Interface
      • patterns-based, generated user interfaces, as well as custom forms and reports based on the most popular Ajax frameworks or any other used by the developer.
    • Integration Services
      • in the real world there are always some external services that have to be integrated into your application - for data transfer, triggering external processes, lookup in external sources, etc. For this purpose Dirigible provides capabilities to create simple and dynamic routing services following the Enterprise Integration Patterns guidance.

    Architecture

    • The design-time and the runtime components are well separated. They interact with each other through a dedicated repository where the only linking point is the content itself. At design-time programmers and designers use the Web-based integrated development environment based on the Eclipse Remote Application Platform (RAP). Leveraging this robust and powerful framework the tooling can be easily enhanced using well-known APIs and concepts - SWT, JFaces, OSGi, extension points, etc.
    • The Repository is the container of the project artifacts. It is a generic file-system like content repository built on a relation database (via JDBC).
    • On top are the Dirigible’s containers for services execution depending on the scripting language and purpose - Rhino, jRuby, Groovy, Camel, CXF, etc.
    • The runtime can scale independently by the design time part, and even can be deployed without design time at all (for productive landscapes).
    • Depending on the target cloud platform, Dirigible can be integrated to use the services provided by the underlying platform.

    Building Blocks

    From end user perspective, the IDE is composition of perspective, each consisting of the necessary tools to accomplish certain goal. 5 main perspectives:

    • Workbench: comprises Workspace, Import, Properties, Console and Preview views, plus the editors registered for each file type.
    • Git: present a simplified interface for the most common Git client operations
    • Database: features a database explorer, a console to execute SQL statements and preview results in table format
    • Repository: offers explorer for the repository of the Dirigible instance where the IDE is running. and import/export snapshots
    • Terminal…