Class DocumentsWriterFlushControl

java.lang.Object
org.apache.lucene.index.DocumentsWriterFlushControl
All Implemented Interfaces:
Closeable, AutoCloseable, Accountable

final class DocumentsWriterFlushControl extends Object implements Accountable, Closeable
This class controls DocumentsWriterPerThread flushing during indexing. It tracks the memory consumption per DocumentsWriterPerThread and uses a configured FlushPolicy to decide if a DocumentsWriterPerThread must flush.

In addition to the FlushPolicy the flush control might set certain DocumentsWriterPerThread as flush pending iff a DocumentsWriterPerThread exceeds the IndexWriterConfig.getRAMPerThreadHardLimitMB() to prevent address space exhaustion.

  • Field Details

    • hardMaxBytesPerDWPT

      private final long hardMaxBytesPerDWPT
    • activeBytes

      private long activeBytes
    • flushBytes

      private volatile long flushBytes
    • numPending

      private volatile int numPending
    • numDocsSinceStalled

      private int numDocsSinceStalled
    • flushDeletes

      private final AtomicBoolean flushDeletes
    • fullFlush

      private boolean fullFlush
    • fullFlushMarkDone

      private boolean fullFlushMarkDone
    • flushQueue

      private final Queue<DocumentsWriterPerThread> flushQueue
    • blockedFlushes

      private final Queue<DocumentsWriterPerThread> blockedFlushes
    • flushingWriters

      private final List<DocumentsWriterPerThread> flushingWriters
    • maxConfiguredRamBuffer

      private double maxConfiguredRamBuffer
    • peakActiveBytes

      private long peakActiveBytes
    • peakFlushBytes

      private long peakFlushBytes
    • peakNetBytes

      private long peakNetBytes
    • peakDelta

      private long peakDelta
    • flushByRAMWasDisabled

      private boolean flushByRAMWasDisabled
    • stallControl

      final DocumentsWriterStallControl stallControl
    • perThreadPool

      private final DocumentsWriterPerThreadPool perThreadPool
    • flushPolicy

      private final FlushPolicy flushPolicy
    • closed

      private boolean closed
    • documentsWriter

      private final DocumentsWriter documentsWriter
    • config

      private final LiveIndexWriterConfig config
    • infoStream

      private final InfoStream infoStream
    • stallStartNS

      private long stallStartNS
  • Constructor Details

  • Method Details

    • activeBytes

      public long activeBytes()
    • getFlushingBytes

      long getFlushingBytes()
    • netBytes

      long netBytes()
    • stallLimitBytes

      private long stallLimitBytes()
    • assertMemory

      private boolean assertMemory()
    • updatePeaks

      private boolean updatePeaks(long delta)
    • ramBufferGranularity

      private long ramBufferGranularity()
      Return the smallest number of bytes that we would like to make sure to not miss from the global RAM accounting.
    • doAfterDocument

    • checkout

      private DocumentsWriterPerThread checkout(DocumentsWriterPerThread perThread, boolean markPending)
    • assertNumDocsSinceStalled

      private boolean assertNumDocsSinceStalled(boolean stalled)
    • doAfterFlush

      void doAfterFlush(DocumentsWriterPerThread dwpt)
    • updateStallState

      private boolean updateStallState()
    • waitForFlush

      public void waitForFlush()
    • setFlushPending

      public void setFlushPending(DocumentsWriterPerThread perThread)
      Sets flush pending state on the given DocumentsWriterPerThread. The DocumentsWriterPerThread must have indexed at least on Document and must not be already pending.
    • doOnAbort

      void doOnAbort(DocumentsWriterPerThread perThread)
    • checkoutAndBlock

      private void checkoutAndBlock(DocumentsWriterPerThread perThread)
      To be called only by the owner of this object's monitor lock
    • checkOutForFlush

      private DocumentsWriterPerThread checkOutForFlush(DocumentsWriterPerThread perThread)
    • addFlushingDWPT

      private void addFlushingDWPT(DocumentsWriterPerThread perThread)
    • toString

      public String toString()
      Overrides:
      toString in class Object
    • nextPendingFlush

      DocumentsWriterPerThread nextPendingFlush()
    • close

      public void close()
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
    • allActiveWriters

      public Iterator<DocumentsWriterPerThread> allActiveWriters()
      Returns an iterator that provides access to all currently active DocumentsWriterPerThreads
    • doOnDelete

      void doOnDelete()
    • getDeleteBytesUsed

      public long getDeleteBytesUsed()
      Returns heap bytes currently consumed by buffered deletes/updates that would be freed if we pushed all deletes. This does not include bytes consumed by already pushed delete/update packets.
    • ramBytesUsed

      public long ramBytesUsed()
      Description copied from interface: Accountable
      Return the memory usage of this object in bytes. Negative values are illegal.
      Specified by:
      ramBytesUsed in interface Accountable
    • numFlushingDWPT

      int numFlushingDWPT()
    • getAndResetApplyAllDeletes

      public boolean getAndResetApplyAllDeletes()
    • getApplyAllDeletes

      public boolean getApplyAllDeletes()
      Check whether deletes need to be applied. This can be used as a pre-flight check before calling getAndResetApplyAllDeletes() to make sure that a single thread applies deletes.
    • setApplyAllDeletes

      public void setApplyAllDeletes()
    • obtainAndLock

      DocumentsWriterPerThread obtainAndLock()
    • markForFullFlush

      long markForFullFlush()
    • assertActiveDeleteQueue

      private boolean assertActiveDeleteQueue(DocumentsWriterDeleteQueue queue)
    • pruneBlockedQueue

      private void pruneBlockedQueue(DocumentsWriterDeleteQueue flushingQueue)
      Prunes the blockedQueue by removing all DWPTs that are associated with the given flush queue.
    • finishFullFlush

      void finishFullFlush()
    • assertBlockedFlushes

      boolean assertBlockedFlushes(DocumentsWriterDeleteQueue flushingQueue)
    • abortFullFlushes

      void abortFullFlushes()
    • abortPendingFlushes

      void abortPendingFlushes()
    • isFullFlush

      boolean isFullFlush()
      Returns true if a full flush is currently running
    • numQueuedFlushes

      int numQueuedFlushes()
      Returns the number of flushes that are already checked out but not yet actively flushing
    • numBlockedFlushes

      int numBlockedFlushes()
      Returns the number of flushes that are checked out but not yet available for flushing. This only applies during a full flush if a DWPT needs flushing but must not be flushed until the full flush has finished.
    • waitIfStalled

      void waitIfStalled()
      This method will block if too many DWPT are currently flushing and no checked out DWPT are available
    • anyStalledThreads

      boolean anyStalledThreads()
      Returns true iff stalled
    • getInfoStream

      public InfoStream getInfoStream()
    • findLargestNonPendingWriter

      DocumentsWriterPerThread findLargestNonPendingWriter()
    • checkoutLargestNonPendingWriter

      final DocumentsWriterPerThread checkoutLargestNonPendingWriter()
      Returns the largest non-pending flushable DWPT or null if there is none.
    • getPeakActiveBytes

      long getPeakActiveBytes()
    • getPeakNetBytes

      long getPeakNetBytes()