BaseConsumer

abstract class BaseConsumer<T, P : WritePersistor<T>> : Runnable

Base definition of a runnable Consumer. Consumers are responsible for persisting data to disk

T -

P -

Base definition of a runnable Consumer. Consumers are responsible for persisting data to disk

BaseConsumer(persistor: P, dataQueue: BlockingQueue<MutableList<T>>, controlQueue: ArrayBlockingQueue<DedupeReport>, deleteIfExists: Boolean)

controlQueue	queue where persistor receives DedupeReport `val controlQueue: ArrayBlockingQueue<DedupeReport>`
dataQueue	queue where persistor receives data to persist `val dataQueue: BlockingQueue<MutableList<T>>`
deleteIfExists	determines whether persistent object (table or file) is dropped before being recreated `val deleteIfExists: Boolean`
persistor	a WritePersistor `val persistor: P`
totalRowsWritten	`var totalRowsWritten: Long`

createTarget	creates the target - either flat file or database table `abstract fun createTarget(deleteIfExists: Boolean, persistor: P): Unit`
getDeduperReportCount	the report metric in the dedupeReport to check `abstract fun getDeduperReportCount(dedupeReport: DedupeReport): Long`
processDeduperReport	consumes the controlQueue for a DedupeReport, indicating the publishing of all data is complete `fun processDeduperReport(): DedupeReport`
processFirstMessage	pulls/processes the first message off of the dataQueue and returns whether or not the message is empty `fun processFirstMessage(): Boolean`
processQueueData	loops over the queue consuming messages until doneFlag is set to true, meaning the last message was empty `fun processQueueData(doneFlag: Boolean): Unit`
run	launches consumer as a runnable `open fun run(): Unit`
unlockCsvFile	removes the lock file from a flat file once persistence to file is complete `fun unlockCsvFile(): Unit`

val logger: Logger!

DeduperDataConsumer	Consumer for processing and persisting target data, IE "deduped" data `class DeduperDataConsumer : BaseConsumer<Map<String, Any>, TargetPersistor>`
DeduperDupeConsumer	Consumer for processing and persisting duplicate data `class DeduperDupeConsumer : BaseConsumer<Pair<String, Pair<MutableList<Long>, Dupe>>, DupePersistor>`
DeduperHashConsumer	Consumer for processing and persisting MD-5 hashes data `class DeduperHashConsumer : BaseConsumer<HashRow, HashPersistor>`