Skip to main content

Release Notes 1.4.0

Apache Linkis 1.4.0 includes all Project Linkis-1.4.0

Linkis 1.4.0 version, the main new features are as follows: Adapted Hadoop, Hive, Spark The default version is upgraded to 3.x (Hadoop2.7.2-3.3.4, Hive2.3.3-3.1.3, spark2.4.3-3.2 .1 Supplement the specific version information), and support compilation parameters to control the version, so as to reduce the difficulty of transforming and adapting to the non-default base engine version; Hive EC supports running tasks in concurrent mode, which can greatly reduce the use of machine resources and improve the concurrency of hive tasks; ECM service does not kill EC when restarting, providing support for graceful restart; storage of task log result sets, new support for S3 and OSS file system modes; new support for data source services, such as: tidb, starrocks, Gaussdb, etc. ; Service support adapts to postgresql database mode deployment (experimental); Added Impala engine support (experimental); and enhanced Spark ETL functions, supporting Excel, Redis, Mongo, Elasticsearch, etc.;

The main functions are as follows:

  • Upgrade the default versions of hadoop, spark, and hive to 3.x

  • Reduce the compatibility issues of different versions of the base engine

  • Support Hive EC to execute tasks concurrently

  • Support not kill EC when restarting ECM service

  • Support more data sources, such as: tidb, starrocks, Gaussdb, etc.

  • Add postgresql database support (experimental)

  • linkis-storage supports S3 and OSS filesystems (experimental)

  • Added Impala engine connector support (experimental)

  • Enhancements to Spark ETL

  • Version number upgrade rules and submitted code default merge branch modification

abbreviation:

  • ORCHESTRATOR: Linkis Orchestrator
  • COMMON: Linkis Common
  • ENTRANCE: Linkis Entrance
  • EC: Engineconn
  • ECM: EngineConnManager
  • ECP: EngineConnPlugin
  • DMS: Data Source Manager Service
  • MDS: MetaData Manager Service
  • LM: Links Manager
  • PS: Link Public Service
  • PE: Link Public Enhancement
  • RPC: Linkis Common RPC
  • CG: Linkis Computation Governance
  • DEPLOY: Linkis Deployment
  • WEB: Linked Web
  • GATEWAY: Linkis Gateway
  • EP: Engine Plugin

new features​

Enhancement points​

  • [COMMON]LINKIS-4462 code optimization, unified attribute name
  • [COMMON]LINKIS-4425 code optimization, delete useless code
  • [COMMON]LINKIS-4368 code optimization, remove json4s dependency
  • [COMMON]LINKIS-4357 file upload interface optimization
  • [COMMON]LINKIS-4678 Linkis JDBC Driver optimization supports docking different types of engines and tasks
  • [COMMON]LINKIS-4554 Add task link tracking log to facilitate locating problems through unique task ID
  • [ECM]LINKIS-4449 ECM code optimization
  • [EC]LINKIS-4341 Optimize the code logic of CustomerDelimitedJSONSerDe
  • [EC-Openlookeng]LINKIS- Openlookeng EC code conversion to Java
  • [EC-Shell]LINKIS-4473 Shell EC code conversion to Java
  • [EC-Python]LINKIS-4482 Python EC code conversion to Java
  • [EC-Trino]LINKIS-4526 Trino EC code conversion to Java
  • [EC-Presto]LINKIS-4514 Presto EC code conversion to Java
  • [EC-Elasticsearch]LINKIS-4531 Elasticsearch EC code conversion to Java
  • [COMMON]LINKIS-4475 use latest mysql DDL in k8s deployment
  • [EC-Flink]LINKIS-4556 Flink EC adds task interceptor
  • [GATEWAY]LINKIS-4548 Clear all backend caches on user logout
  • [COMMON]LINKIS-4554 Add MDC log format in Linkis to track JobID
  • [CG]LINKIS-4583 When submitting an once task, you can get the result of creating the engine
  • [EC-Spark]LINKIS-4570 Generate Spark sql based on jdbc data source
  • [COMMON]LINKIS-4601 supports integration test Action
  • [EC-Seatunnel]LINKIS-4673 Seatunnel version upgrade to 2.3.1

Repair function​

  • [EC-Hive]LINKIS-4246 The Hive engine version number supports hyphens, such as hive3.1.2-cdh5.12.0
  • [COMMON]LINKIS-4438 fixed nohup startup error
  • [EC]LINKIS-4429 Fix CPU average load calculation bug
  • [PE]LINKIS-4457 fix parameter validation issue configured by admin console
  • [DMS]LINKIS-4500 Fixed type conversion failure between client and data source
  • [COMMON]LINKIS-4480 fixed build default configuration file with jdk17
  • [CG]LINKIS-4663 Fix the problem that engine reuse may throw NPE
  • [LM]LINKIS-4652 fixed the problem that creating engine node throws NPE

Acknowledgments​

The release of Apache Linkis 1.4.0 is inseparable from the contributors of the Linkis community. Thanks to all community contributors, including but not limited to the following Contributors (in no particular order): casionone,MrFengqin,zhangwejun,Zhao,ahaoyao,duhanmin,guoshupei,shixiutao,CharlieYan24,peacewong,GuoPhilipse,aiceflower,waynecookie,jacktao007,chenghuichen,ws00428637,ChengJie1053,dependabot,jackxu2011,s jgllgh,rarexixi,pjfanning,v-kkhuang,binbinCheng, stdnt-xiao, mayinrain.