This update contains changes to the infrastructure as well as the software. Because of the infrastructure changes, it is necessary to redeploy the CLC Genomics Cloud Engine.
Please also refer to the Cloud Plugin 21.0 latest improvements for associated features and improvements affecting the connection to and use of CLC Genomics Cloud Engine 21.0 from CLC Genomics Workbench 21.x and CLC Genomics Server 21.x software.
- Multiple AWS EC2 instance types are supported. A default set comes preconfigured, but this list can be adjusted in the admin host. Instance types are no longer managed in the Elastic Beanstalk service.
- When workflows containing tools distributed via plugins are run, the relevant plugin version is dynamically deployed to GCE.
- Data stored in Illumina BaseSpace can be imported directly using on-the-fly import in workflows.
- Third party software can be executed from within workflows run on GCE. This is done by including elements in the workflow for containerized external applications that have been made available in Amazon ECR. A local CLC Genomics Server is required for configuring such external applications.
- GCE releases from this point forward are backward compatible with earlier CLC Genomics Server versions within the same major release line.
- Substantially improved memory use (up to 4 times less memory required for each job)
- Executor instances start up faster (up to 3 times faster than previously)
- Executor instances shut down faster
- Events are sent when a batch is complete, not just when a job completes.
- Additional log files from the CLC Genomics Server have been added to CloudWatch.
- Default log retention has been set to 180 days.
- The Spring property: “security.oauth2.resource.id” can be used for OAuth2 configuration.
- Cache buckets can be created in regions and AWS accounts other than those the GCE is deployed in.
- The form filled out when creating a cache bucket using CloudFormation has been simplified.
- The job status description has been improved for situations where a job is rescheduled due to a timeout.
- If a job is submitted for a workflow created in a CLC Workbench version that is not compatible with a version of the CLC Genomics Server available in the GCE instance, a validation error is reported to the user and the job is not submitted.
- Fixed an issue that caused an error when all plugins were deleted.
- The Java version bundled with CLC Genomics Cloud Engine 21.0 is Java 11, where we use the JRE from Amazon Correto.
- Job priority options were replaced by option to specify instance types.
- Several scripts on the admin host have been renamed or moved. The CLC Genomics Cloud Engine administrator manual contains the up-to-date script names. The changes include, but are not limited to:
- Prepare script: admin/prepare.sh → ./prepare.py
- Install GCE: ./create-stack.py → ./create_stack.py
- Configure GCE stack name: /root/setup.sh → /root/config/setup.sh
- Configure authentication: ~/job-manager-config/application-oauth2.yaml → ~/config/application-oauth2.yaml
- Restart license server: admin/tools/restart-licenseserver.sh → ./restart_license_server.py
- The CloudFormation form used when creating an admin host no longer requires the definition of a solution stack.
- The admin host no longer uses the Elastic Beanstalk service.
Frequently Asked Questions
- Fixed an issue that caused job submissions to fail if the workflow .cpw file was located in a bucket with a name that contained a period.
The CLC Genomics Cloud Engine Command Line Tools 21.0.0 is the corresponding client for the CLC Genomics Cloud Engine 21.0.0.
It is not possible to access the CLC Genomics Cloud Engine 21.0.0 with an earlier version of the CLC Genomics Cloud Engine Command Line Tools. Nor is it possible to access an earlier version of the CLC Genomics Cloud Engine with the CLC Genomics Cloud Engine Command Line Tools 21.0.0.
The CLC Genomics Cloud Engine uses CLC Genomics Server 21.0.1.
Workflows to be run on the CLC Genomics Cloud Engine 21.0.0 are designed using a CLC Workbench. Workflow installer files should be made using a CLC Workbench compatible with the CLC Genomics Server run by GCE. For this release, these are:
- CLC Genomics Workbench 21.0.1
- CLC Main Workbench 21.0.1
Workflows can be submitted to run on CLC Genomics Cloud Engine 21.0.0 from compatible versions of CLC Workbenches with the Cloud Plugin installed and from compatible versions of CLC Genomics Server with the Cloud Server Plugin installed.