Skip to main content

Release Notes 1.1.1

Apache Linkis 1.1.1 includes all of Project Linkis-1.1.1.

This release mainly supports the functions of UDF multi-version control and UDF storage to BML; the submission task supports the collection and viewing of Yarn queue resource usage statistics; new support for the data virtualization engine openLooKeng; and known bugs reported by the community are fixed.

The following key features have been added:

  • Support proxy user mode, user A can perform tasks on behalf of user B, one proxy user can proxy multiple users
  • Support UDF multi-version control and UDF storage to BML features
  • Submission of tasks supports the collection of Yarn queue resource usage statistics and the visual view of the management console page
  • Added support for data virtualization engine openLooKeng

abbreviation:

  • EC: Engineconn
  • ECM: EngineConnManager
  • ECP: EngineConnPlugin
  • DMS: Data Source Manager Service
  • MDS: MetaData Manager Service
  • LM: Linkis Manager

New Feature​

  • [Gateway&Entrance][Linkis-1608] Support proxy user mode, user A can perform tasks on behalf of user B, query user B's Related data, a proxy user can proxy multiple users
  • [LM-ResourceManager][Linkis-1616] The resource address configuration of YARN ResourceManager supports high-availability multi-address configuration, the current YARN ResourceManager conversion When the status or stop, the master node will be resolved from the high-availability address list to continue to provide services
  • [EC-openLooKeng][Linkis-1639] Added support for data virtualization engine openLooKeng
  • [UDF][Linkis-1534] Support UDF multi-version control and UDF storage to BML, submit tasks support Yarn queue resources Use statistics collection and management console page visualization
  • [Client][Linkis-1718] The Linkis-cli client supports submitting Once type tasks, which will only run once after the engine process is started Task, automatically destroyed after the task is over
  • [ECP][Linkis-1758] Add engine material refresh interface, support to refresh engine material resources through http interface call

Enhancement​

  • [Gateway][Linkis-1430] For the Token authentication method, the Token acquisition is adjusted from the configuration file to the database table
  • [Entrance][Linkis-1642] Optimize the excel export interface resultsetToExcel: support passing the number of rows of downloaded data
  • [Entrance][Linkis-1733] Add support for more default time variables related to run_date
  • [Entrance][Linkis-1794] Add to limit the data size of a single row in the result set, and optimize the OOM problem caused by large result sets
  • [DMS-Common][Linkis-1757] Support to configure Hive metadata administrator, administrators can obtain hive's metadata through the interface Metadata information for all library tables
  • [Common][Linkis-1799] Optimize the segmentation of service logs: adjust the log history segmentation time from one day to one hour
  • [Common][Linkis-1921] Optimize Jackson's dependency management: manage jackson dependencies uniformly through jackson-bom, and upgrade to Version 2.11.4
  • [ECM][Linkis-1779] Optimize the status monitoring logic of ECM instances, and increase the judgment of heartbeat reporting time. The fix may be due to Eureka performance issues leading to misjudgment issues
  • [ECM][Linkis-1930] ECM resource is not checked when optimizing resource check
  • [Web][Linkis-1596] Optimize the use of the interface for viewing the task log of the management console, and fix the log cannot be used for the running job Timely refresh display issues
  • [Web][Linkis-1650] linkis console - global history page, support to filter historical task data by creator information search

Bugs Fix​

  • [Entrance][Linkis-1623] Fix LogPath and ResultSetPath incorrectly use submitUser as executeUser
  • [Entrance][Linkis-1640] Fix LogReader using singleton InputStream, there is log loss, unable to read the latest persistent log The problem
  • [Entrance][Linkis-2009] Fix the problem of memory leak caused by not closing thread resources in Entrance service
  • [Entrance][Linkis-1901] Replaced the cache in EntranceFactory with Guava Cache, fixed that the user modified the concurrency parameter and it could not take effect The problem
  • [Entrance][Linkis-1986] Fix the abnormal number of lines obtained in the Entrance real-time log, resulting in the duplicated log obtained
  • [ECM][Linkis-1714] Optimize EC by reducing EC Java default memory size and adding retry log for EC application The abnormal problem of "Cannot allocate memory" appears
  • [ECM][Linkis-1806] Optimize the life cycle processing logic of EC, when ECM starts EC due to insufficient queue resources and timeout When the status is Failed, kill the EC process
  • [Common][Linkis-1721] Fixed the issue that hdfsFileSystem was not refreshed when Kerberos authentication failed
  • [UDF][Linkis-1728] Optimize /api/rest_j/v1/udf/all API interface for occasional queries time consuming problem
  • [Config][Linkis-1859] Fix the problem of abnormal primary key duplication in the console parameter configuration saveFullTree interface
  • [Clinet][Linkis-1739] Fix the ujes-client request, the parameter spelling error caused the parameter transmission to fail
  • [Client][Linkis-1783] Fix the problem that the default configuration of the task creator creator parameter does not take effect
  • [Client][Linkis-1821] Fix ujes-client request entity class GetTableStatisticInfoAction parameter is missing
  • [EC][Linkis-1765] Fix the blocking problem that EC triggers tryShutdown when the task is running
  • [LM-AppManager][Linkis-1814] Fix the response information returned by the createEngineConn interface of EngineRestfulApi is incorrect, resulting in NPE in client calls The problem.
  • [Web][Linkis-1972] Remove the dss related interface code left but not used by the management console for historical reasons
  • [EC-Spark][Linkis-1729] Add SparkPreExecutionHook function, compatible with the old package name before Linkis (com.webank.wedatasphere .linkis)
  • [EC-JDBC][Linkis-1851] Fix the jdbc engine, the problem that there are multiple sql statements in one task execution cannot be executed normally
  • [EC-JDBC][Linkis-1961] Fix the problem that the log cannot be printed normally due to the SLF4J dependency problem when the jdbc engine starts
  • [Gateway][Linkis-1898] Fix the problem that the initial domain name cannot be set when the GatewaySSOUtils user successfully logs in to generate a cookie

Others​

  • [License][Linkis-2110] Removed the binary file .mvn/wrapper/maven-wrapper.jar in the source code, and adjusted the LICENSE content related to .mvn/*
  • [License][Linkis-2113] Upgrade py4j-0.10.7-src.zip to py4j-0.10.9.5-src.zip, update the license files of py4j-*.src and adjust it location, from linkis-engineconn-plugins/engineconn-plugins/python /src/main/py4j/LICENSE-py4j-0.10.7-src.txt moved to licenses/LICENSE-py4j-0.10.9.5-src.txt for easy viewing
  • Fixed the issue of using Window's line endings format CTRL in the release source code of shell script:mvnw

Credits​

The release of Apache Linkis 1.1.1 is inseparable from the contributors of the Linkis community. Thanks to all the community contributors, including but not limited to the following Contributors: AbnerHung, Alexkun, barry8023, CCweixiao, Davidhua1996, Fuu3214, Liveipool, casinoone, demonray , husofskyzy, jackxu2011, legendtkl, lizheng920625, maidangdang44, peacewong, seedscoder