AbsaOSS/pramen

Issues

• Pramen pipelines sometimes hang when parallel.tasks > 1

  #302 opened 8 months ago by yruslan
  0

• Use graceful conversion of dates when ingestion from MS SQL Server

  #324 opened 7 months ago by yruslan
  0

• Add basic validation of input tables and columns for possible SQL injections

  #325 opened 7 months ago by yruslan
  0

• Add `Success` prefix to successful email notifications

  #327 opened 7 months ago by yruslan
  0

• Add workflow configuration to the constructor of sources and sinks

  #319 opened 7 months ago by yruslan
  0

• Add support for information columns with special characters in JDBC source

  #317 opened 7 months ago by yruslan
  0

• Make `information.date.sql.format` option obsolete

  #314 opened 7 months ago by yruslan
  0

• When `information.date.sql.format` is not specified for a JDBC source the pipeline fails

  #312 opened 7 months ago by yruslan
  0

• Report config arrays in 1-based way, not 0-based

  #304 opened 7 months ago by yruslan
  0

• Add an option to set the oldest calendar date the pipeline can run for

  #286 opened 7 months ago by yruslan
  0

• Do not mark completed jobs as 'Rerun' if it is running for the first time

  #306 opened 8 months ago by yruslan
  0

• Implement a mode of running multiple pipelines sequentially without re-creating the Spark Session

  #311 opened 8 months ago by yruslan
  0

• Make SqlGenerators.getWhere() accessible on the trait level

  #307 opened 8 months ago by yruslan
  0

• Add support for Spark JDBC sources with SQL inputs

  #283 opened 8 months ago by yruslan
  0

• Make SQL dialects extensible with the ability to load dynamically

  #301 opened 8 months ago by yruslan
  0

• Do not include Delta Lake format libraries in the assembly JAR by default

  #298 opened 8 months ago by yruslan
  0

• The 'preman-runner' component component is obsolete, can be merged with 'pramen-core'

  #291 opened 8 months ago by yruslan
  0

• Allow the ability for multiple transformers to write to the same table

  #294 opened 8 months ago by yruslan
  0

• Add a Spark Catalog Source that takes data from spark.table()

  #288 opened 8 months ago by yruslan
  0

• Pramen considers late data the data that has already been loaded

  #292 opened 8 months ago by yruslan
  0

• When 'trackDays=0', Pramen should never checks source data record count if the table is already loaded for the day

  #279 opened 8 months ago by yruslan
  0

• If a startup hook is not found, the pipeline breaks without a email notification.

  #287 opened 8 months ago by yruslan
  0

• Multithreaded parallel pipeline can deadlock if fatal error has occured

  #284 opened 8 months ago by yruslan
  0

• Add an option to not track late data for a table

  #282 opened 8 months ago by yruslan
  0

• Add varchar length metadata to the data returned by the built-in JDBC source

  #276 opened 8 months ago by yruslan
  0

• Add the ability to customize email notification signature

  #275 opened 9 months ago by yruslan
  0

• Add support for initialization and finalization hooks

  #273 opened 9 months ago by yruslan
  0

• Add support for Spark 3.5.0

  #270 opened 9 months ago by yruslan
  0

• Raw file source fails when copying between file systems

  #266 opened 9 months ago by yruslan
  0

• Use Hive's 'varchar(n)' data type when the maximum length of string fields is known.

  #268 opened 9 months ago by yruslan
  0

• Extend email notification capabilities with lists and files

  #263 opened 9 months ago by yruslan
  0

• Add the ability to store arbitrary metadata for metastore tables.

  #252 opened 9 months ago by yruslan
  2

• Make Spark Application name configurable

  #258 opened 9 months ago by yruslan
  0

• Allow postprocessing as a part of a transformation or ingestion

  #261 opened 9 months ago by yruslan
  0

• Add a File Source that allows loading files based on a date pattern

  #248 opened 9 months ago by yruslan
  0

• Add the ability to continue with dependent jobs even if validation failed for previous jobs.

  #244 opened 9 months ago by yruslan
  0

• A strange 'empty.max' errors are encountered sometime.

  #249 opened 10 months ago by yruslan
  0

• Allow case-insensitive patterns in RawFileSource

  #253 opened 10 months ago by yruslan
  0

• Rename 'Hive' column i notifications to 'Catalog'

  #255 opened 10 months ago by yruslan
  0

• Transient transformers (Scala only)

  #221 opened 10 months ago by yruslan
  0

• Allow transient transformations to run on-demand.

  #245 opened 10 months ago by yruslan
  0

• Do not track schema changes to 'raw' tables

  #237 opened a year ago by yruslan
  0

• Add support for column comments in config transformations

  #239 opened a year ago by yruslan
  0

• Improve error messages for EnceladusSink errors

  #240 opened a year ago by yruslan
  0

• Allow jobs to add info to notifications

  #234 opened a year ago by yruslan
  0

• Allow alternative paths to tables when exposing data via Hive.

  #232 opened a year ago by yruslan
  0

• Pass operation definition config to the transformer constructor instead of the full config

  #229 opened a year ago by yruslan
  0

• Logs for Spark 3.3+ are too verbose

  #227 opened a year ago by yruslan
  0

• Log the list of executor nodes

  #225 opened a year ago by yruslan
  0

• Add support for file-based operations

  #222 opened a year ago by yruslan
  0