head 1.8; access; symbols pkgsrc-2023Q4:1.8.0.18 pkgsrc-2023Q4-base:1.8 pkgsrc-2023Q3:1.8.0.16 pkgsrc-2023Q3-base:1.8 pkgsrc-2023Q2:1.8.0.14 pkgsrc-2023Q2-base:1.8 pkgsrc-2023Q1:1.8.0.12 pkgsrc-2023Q1-base:1.8 pkgsrc-2022Q4:1.8.0.10 pkgsrc-2022Q4-base:1.8 pkgsrc-2022Q3:1.8.0.8 pkgsrc-2022Q3-base:1.8 pkgsrc-2022Q2:1.8.0.6 pkgsrc-2022Q2-base:1.8 pkgsrc-2022Q1:1.8.0.4 pkgsrc-2022Q1-base:1.8 pkgsrc-2021Q4:1.8.0.2 pkgsrc-2021Q4-base:1.8 pkgsrc-2021Q3:1.6.0.28 pkgsrc-2021Q3-base:1.6 pkgsrc-2021Q2:1.6.0.26 pkgsrc-2021Q2-base:1.6 pkgsrc-2021Q1:1.6.0.24 pkgsrc-2021Q1-base:1.6 pkgsrc-2020Q4:1.6.0.22 pkgsrc-2020Q4-base:1.6 pkgsrc-2020Q3:1.6.0.20 pkgsrc-2020Q3-base:1.6 pkgsrc-2020Q2:1.6.0.18 pkgsrc-2020Q2-base:1.6 pkgsrc-2020Q1:1.6.0.14 pkgsrc-2020Q1-base:1.6 pkgsrc-2019Q4:1.6.0.16 pkgsrc-2019Q4-base:1.6 pkgsrc-2019Q3:1.6.0.12 pkgsrc-2019Q3-base:1.6 pkgsrc-2019Q2:1.6.0.10 pkgsrc-2019Q2-base:1.6 pkgsrc-2019Q1:1.6.0.8 pkgsrc-2019Q1-base:1.6 pkgsrc-2018Q4:1.6.0.6 pkgsrc-2018Q4-base:1.6 pkgsrc-2018Q3:1.6.0.4 pkgsrc-2018Q3-base:1.6 pkgsrc-2018Q2:1.6.0.2 pkgsrc-2018Q2-base:1.6 pkgsrc-2018Q1:1.5.0.2 pkgsrc-2018Q1-base:1.5 pkgsrc-2017Q4:1.4.0.2 pkgsrc-2017Q4-base:1.4 pkgsrc-2017Q3:1.3.0.4 pkgsrc-2017Q3-base:1.3 pkgsrc-2017Q2:1.2.0.2 pkgsrc-2017Q2-base:1.2 pkgsrc-2017Q1:1.1.0.2 pkgsrc-2017Q1-base:1.1; locks; strict; comment @# @; 1.8 date 2021.10.26.10.15.08; author nia; state Exp; branches; next 1.7; commitid XMBRkrafc79x2jeD; 1.7 date 2021.10.07.13.40.02; author nia; state Exp; branches; next 1.6; commitid TK9Y8OXA04EYMSbD; 1.6 date 2018.04.05.08.46.37; author fhajny; state Exp; branches; next 1.5; commitid HSK2YcsZmISvlexA; 1.5 date 2018.03.07.11.50.57; author fhajny; state Exp; branches; next 1.4; commitid If6pPgktITiIiwtA; 1.4 date 2017.11.01.13.31.39; author fhajny; state Exp; branches; next 1.3; commitid 06NBxk2unyrmUkdA; 1.3 date 2017.07.04.14.14.46; author fhajny; state Exp; branches; next 1.2; commitid YzeC5jLKYnTfZUXz; 1.2 date 2017.05.10.18.14.06; author fhajny; state Exp; branches; next 1.1; commitid 9qYCS3TY67J36SQz; 1.1 date 2017.02.28.08.17.28; author fhajny; state Exp; branches; next ; commitid JyywhVUn899O4HHz; desc @@ 1.8 log @archivers: Replace RMD160 checksums with BLAKE2s checksums All checksums have been double-checked against existing RMD160 and SHA512 hashes Could not be committed due to merge conflict: devel/py-traitlets/distinfo The following distfiles were unfetchable (note: some may be only fetched conditionally): ./devel/pvs/distinfo pvs-3.2-solaris.tgz ./devel/eclipse/distinfo eclipse-sourceBuild-srcIncluded-3.0.1.zip @ text @$NetBSD: distinfo,v 1.7 2021/10/07 13:40:02 nia Exp $ BLAKE2s (kafka_2.12-1.1.0.tgz) = 002a90d85698a8b3316264e41e223f0e1f32336c7889c065b094dad66d41e679 SHA512 (kafka_2.12-1.1.0.tgz) = 48d1ddc71f5a5b1b25d111f792553be69be62293640a3c6af985203c6ee88c6aa78e01327066bfad3feae6b0b45d71c0cac6ebd2d08843d92269132741a3791b Size (kafka_2.12-1.1.0.tgz) = 50326212 bytes SHA1 (patch-bin_connect-distributed.sh) = 3d530501d50a850dbb52da8c08cb89c7c8407d50 SHA1 (patch-bin_connect-standalone.sh) = 5a00efdf616f761565579741babce3efc6d3a026 SHA1 (patch-bin_kafka-run-class.sh) = 3b8111d2c184327c1277a60f6e554527cb936bab SHA1 (patch-bin_kafka-server-start.sh) = 2e93ef575af6738e4af5555b395a10092e6de933 SHA1 (patch-bin_kafka-server-stop.sh) = 1c690bed0b91ec9de7c551936a04ad5b97d394b1 SHA1 (patch-config_server.properties) = 23c9e1b58a3ddf0a3dc4d5d082d0f90f8a65ba0d @ 1.7 log @devel: Remove SHA1 hashes for distfiles @ text @d1 1 a1 1 $NetBSD: distinfo,v 1.6 2018/04/05 08:46:37 fhajny Exp $ d3 1 a3 1 RMD160 (kafka_2.12-1.1.0.tgz) = 2e92b496a2c4d317d740cfa9d287790ac6d9ef96 @ 1.6 log @devel/kafka: Update to 1.1.0. New Feature - automatic migration of log dirs to new locations - KIP-145 - Expose Record Headers in Kafka Connect - Add the AdminClient in Streams' KafkaClientSupplier - Support dynamic updates of frequently updated broker configs Improvement - KafkaConnect should support regular expression for topics - Move kafka-streams test fixtures into a published package - SSL support for Connect REST API - Grow default heap settings for distributed Connect from 256M to 1G - Enable access to key in ValueTransformer - Add "getAllKeys" API for querying windowed KTable stores - Unify StreamsKafkaClient instances - Revisit Streams DSL JavaDocs - Extend Consumer Group Reset Offset tool for Stream Applications - KIP-175: ConsumerGroupCommand no longer shows output for consumer groups which have not committed offsets - Add a broker metric specifying the number of consumer group rebalances in progress - Use Jackson for serialising to JSON - KafkaShortnamer should allow for case-insensitive matches - Improve Util classes - Gradle 3.0+ is needed on the build - Adding records deletion operation to the new Admin Client API - Kafka metrics templates used in document generation should maintain order of tags - Provide for custom error handling when Kafka Streams fails to produce - Make Repartition Topics Transient - Connect Schema comparison is slow for large schemas - Add a Validator for NonNull configurations and remove redundant null checks on lists - Have State Stores Restore Before Initializing Toplogy - Optimize condition in if statement to reduce the number of comparisons - Removed unnecessary null check - Introduce Incremental FetchRequests to Increase Partition Scalability - SSLTransportLayer should keep reading from socket until either the buffer is full or the socket has no more data - Improve KTable Source state store auto-generated names - Extend consumer offset reset tool to support deletion (KIP-229) - Expose Kafka cluster ID in Connect REST API - Maven artifact for kafka should not depend on log4j - ConsumerGroupCommand should use the new consumer to query the log end offsets. - Change LogSegment.delete to deleteIfExists and harden log recovery - Make ProducerConfig and ConsumerConfig constructors public - Improve synchronization in CachingKeyValueStore methods - Improve Kafka GZip compression performance - Improve JavaDoc of SourceTask#poll() to discourage indefinite blocking - Avoid creating dummy checkpoint files with no state stores - Change log level from ERROR to WARN for not leader for this partition exception - Delay initiating the txn on producers until initializeTopology with EOS turned on Bug - change log4j to slf4j - Use RollingFileAppender by default in log4j.properties - Cached zkVersion not equal to that in zookeeper, broker not recovering. - FileRecords.read doesn't handle size > sizeInBytes when start is not zero - a soft failure in controller may leave a topic partition in an inconsistent state - Cannot truncate to a negative offset (-1) exception at broker startup - automated leader rebalance causes replication downtime for clusters with too many partitions - kafka-run-class has potential to add a leading colon to classpath - QueryableStateIntegrationTest.concurrentAccess is failing occasionally in jenkins builds - FileStreamSource Connector not working for large files (~ 1GB) - KeyValueIterator returns null values - KafkaProducer is not joining its IO thread properly - Kafka connect: error with special characters in connector name - Replace StreamsKafkaClient with AdminClient in Kafka Streams - LogCleaner#cleanSegments should not ignore failures to delete files - Connect Rest API allows creating connectors with an empty name - KIP-212 - KerberosLogin#login should probably be synchronized - Support replicas movement between log directories (KIP-113) - Consumer ListOffsets request can starve group heartbeats - Struct.put() should include the field name if validation fails - Clarify handling of connector name in config - Allow user to specify relative path as log directory - KafkaConsumer should validate topics/TopicPartitions on subscribe/assign - Controller should only update reassignment znode if there is change in the reassignment data - records.lag should use tags for topic and partition rather than using metric name. - KafkaProducer should not wrap InterruptedException in close() with KafkaException - Connect classloader isolation may be broken for JDBC drivers - JsonConverter generates "Mismatching schema" DataException - NoSuchElementException in markErrorMeter during TransactionsBounceTest - Make KafkaFuture.Function java 8 lambda compatible - ThreadCache#sizeBytes() should check overflow - KafkaFuture timeout fails to fire if a narrow race condition is hit - DeleteRecordsRequest to a non-leader - ReplicaFetcherThread should close the ReplicaFetcherBlockingSend earlier on shutdown - NoSuchMethodError when creating ProducerRecord in upgrade system tests - Running tools on Windows fail due to typo in JVM config - Streams metrics tagged incorrectly - ducker-ak: add ipaddress and enum34 dependencies to docker image - Kafka cannot recover after an unclean shutdown on Windows - Scanning plugin.path needs to support relative symlinks - Reconnecting to broker does not exponentially backoff - TaskManager should be type aware - Major performance issue due to excessive logging during leader election - RecordCollectorImpl should not retry sending - Restore and global consumer should not use auto.offset.reset - Global Consumer should handle TimeoutException - Reduce rebalance time by not checking if created topics are available - VerifiableConsumer with --max-messages doesn't exit - Transaction markers are sometimes discarded if txns complete concurrently - Simplify StreamsBuilder#addGlobalStore - JmxReporter can't handle windows style directory paths - CONSUMER-ID and HOST values are concatenated if the CONSUMER-ID is > 50 chars - ClientQuotaManager threads prevent shutdown when encountering an error loading logs - Streams configuration requires consumer. and producer. in order to be read - Timestamp on streams directory contains a colon, which is an illegal character - Add methods in Options classes to keep binary compatibility with 0.11 - RecordQueue.clear() does not clear MinTimestampTracker's maintained list - Selector memory leak with high likelihood of OOM in case of down conversion - GlobalKTable never finishes restoring when consuming transactional messages - Server crash while deleting segments - IllegalArgumentException if 1.0.0 is used for inter.broker.protocol.version or log.message.format.version - Using standby replicas with an in memory state store causes Streams to crash - Issues with protocol version when applying a rolling upgrade to 1.0.0 - Fix system test dependency issues - Kafka Connect requires permission to create internal topics even if they exist - A metric named 'XX' already exists, can't register another one. - Improve sink connector topic regex validation - Flaky Unit test: KStreamKTableJoinIntegrationTest.shouldCountClicksPerRegionWithNonZeroByteCache - Make KafkaStreams.cleanup() clean global state directory - AbstractCoordinator not clearly handles NULL Exception - Request logging throws exception if acks=0 - GlobalKTable missing #queryableStoreName() - KTable state restore fails after rebalance - Make loadClass thread-safe for class loaders of Connect plugins - System Test failed: ConnectRestApiTest - Broken symlink interrupts scanning the plugin path - NetworkClient should not return internal failed api version responses from poll - Transient failure in NetworkClientTest.testConnectionDelayDisconnected - Line numbers on log messages are incorrect - Topic can not be recreated after it is deleted - mBeanName should be removed before returning from JmxReporter#removeAttribute() - Kafka Core should have explicit SLF4J API dependency - StreamsResetter should return non-zero return code on error - kafka-acls regression for comma characters (and maybe other characters as well) - Error deleting log for topic, all log dirs failed. - punctuate with WALL_CLOCK_TIME triggered immediately - Exclude node groups belonging to global stores in InternalTopologyBuilder#makeNodeGroups - Transient failure in kafka.api.AdminClientIntegrationTest.testLogStartOffsetCheckpointkafka.api.AdminClientIntegrationTest.testAlterReplicaLogDirs - NetworkClient.inFlightRequestCount() is not thread safe, causing ConcurrentModificationExceptions when sensors are read - ConcurrentModificationException during streams state restoration - Update KStream JavaDoc with regard to KIP-182 - RocksDB segments not removed when store is closed causes re-initialization to fail - auto commit not work since coordinatorUnknown() is always true. - Fix StateRestoreListener To Use Correct Batch Ending Offset - NullPointerException on KStream-GlobalKTable leftJoin when KeyValueMapper returns null - StreamThread.shutdown doesn't clean up completely when called before StreamThread.start - Update ZooKeeper to 3.4.11, Gradle and other minor updates - output from ensure copartitioning is not used for Cluster metadata, resulting in partitions without tasks working on them - Consumer should not block setting initial positions of unavailable partitions - Non-aggregation KTable generation operator does not construct value getter correctly - AdminClient should handle empty or null topic names better - When enable trace level log in mirror maker, it will throw null pointer exception and the mirror maker will shutdown - Simplify KStreamReduce - Base64URL encoding under JRE 1.7 is broken due to incorrect padding assumption - ChangeLoggingKeyValueBytesStore.all() returns null - Fetcher.retrieveOffsetsByTimes() should add all the topics to the metadata refresh topics set. - LogSemgent.truncateTo() should always resize the index file - Connect: Plugin scan is very slow - Connect: Some per-task-metrics not working - Connect header parser incorrectly parses arrays - Java Producer: Excessive memory usage with compression enabled - New Connect header support doesn't define 'converter.type' property correctly - ZooKeeperClient holds a lock while waiting for responses, blocking shutdown - Transient failure in DynamicBrokerReconfigurationTest.testThreadPoolResize - Broker leaks memory and file descriptors after sudden client disconnects - Delegation token internals should not impact public interfaces - Streams quickstart pom.xml is missing versions for a bunch of plugins - Deadlock while processing Controller Events - Broker doesn't reject Produce request with inconsistent state - LogCleanerManager.doneDeleting() should check the partition state before deleting the in progress partition - KafkaController.brokerInfo not updated on dynamic update - Connect standalone SASL file source and sink test fails without explanation - Connect distributed and standalone worker 'main()' methods should catch and log all exceptions - Consumer bytes-fetched and records-fetched metrics are not aggregated correctly - Coordinator disconnect in heartbeat thread can cause commitSync to block indefinitely - Regression in consumer auto-commit backoff behavior - GroupMetadataManager.loadGroupsAndOffsets decompresses record batch needlessly - log segment deletion could cause a disk to be marked offline incorrectly - Delayed operations may not be completed when there is lock contention - Expression for GlobalKTable is not correct - System tests do not handle ZK chroot properly with SCRAM - Fix config initialization in DynamicBrokerConfig - ReplicaFetcher crashes with "Attempted to complete a transaction which was not started" Test - Add concurrent tests to exercise all paths in group/transaction managers - Add unit tests for ClusterConnectionStates - Only delete reassign_partitions znode after reassignment is complete - KafkaStreamsTest fails in trunk - SelectorTest may fail with ConcurrentModificationException Sub-task - Add capability to create delegation token - Add authentication based on delegation token. - Add capability to renew/expire delegation tokens. - always leave the last surviving member of the ISR in ZK - handle ZK session expiration properly when a new session can't be established - Streams should not re-throw if suspending/closing tasks fails - Use async ZookeeperClient in Controller - Use async ZookeeperClient in SimpleAclAuthorizer - Use async ZookeeperClient for DynamicConfigManager - Use async ZookeeperClient for Admin operations - Trogdor should handle injecting disk faults - Add process stop faults, round trip workload, partitioned produce-consume test - add the notion of max inflight requests to async ZookeeperClient - Add workload generation capabilities to Trogdor - Add ZooKeeperRequestLatencyMs to KafkaZkClient - Use ZookeeperClient in LogManager - Use ZookeeperClient in GroupCoordinator and TransactionCoordinator - Use ZookeeperClient in KafkaApis - Use ZookeeperClient in ReplicaManager and Partition - Tests for KafkaZkClient - Transient failure in kafka.api.SaslScramSslEndToEndAuthorizationTest.testTwoConsumersWithDifferentSaslCredentials - minimize the number of triggers enqueuing PreferredReplicaLeaderElection events - Enable dynamic reconfiguration of SSL keystores - Enable resizing various broker thread pools - Enable reconfiguration of metrics reporters and their custom configs - Enable dynamic reconfiguration of log cleaners - Enable reconfiguration of default topic configs used by brokers - Enable reconfiguration of listeners and security configs - Add ProduceBench to Trogdor - move ZK metrics in KafkaHealthCheck to ZookeeperClient - Add documentation for delegation token authentication mechanism - Document dynamic config update - Extend ConfigCommand to update broker config using new AdminClient - Add test to verify markPartitionsForTruncation after fetcher thread pool resize @ text @d1 1 a1 1 $NetBSD: distinfo,v 1.5 2018/03/07 11:50:57 fhajny Exp $ a2 1 SHA1 (kafka_2.12-1.1.0.tgz) = 057cc111d354d2c20f0125d8e43b44682c69b381 @ 1.5 log @devel/kafka: Update to 1.0.1. Improvement - Kafka metrics templates used in document generation should maintain order of tags - Consolidate MockTime implementations between connect and clients - Fix repeated words words in JavaDoc and comments. - Cache lastEntry in TimeIndex to avoid unnecessary disk access - AbstractIndex should cache index file to avoid unnecessary disk access during resize() - Have State Stores Restore Before Initializing Toplogy - SSLTransportLayer should keep reading from socket until either the buffer is full or the socket has no more data - Improve KTable Source state store auto-generated names Bug - QueryableStateIntegrationTest.concurrentAccess is failing occasionally in jenkins builds - KafkaProducer is not joining its IO thread properly - Kafka connect: error with special characters in connector name - Streams metrics tagged incorrectly - ClassCastException in BigQuery connector - ClientQuotaManager threads prevent shutdown when encountering an error loading logs - Streams configuration requires consumer. and producer. in order to be read - Timestamp on streams directory contains a colon, which is an illegal character - Add methods in Options classes to keep binary compatibility with 0.11 - RecordQueue.clear() does not clear MinTimestampTracker's maintained list - Selector memory leak with high likelihood of OOM in case of down conversion - GlobalKTable never finishes restoring when consuming transactional messages - IllegalArgumentException if 1.0.0 is used for inter.broker.protocol.version or log.message.format.version - Using standby replicas with an in memory state store causes Streams to crash - Issues with protocol version when applying a rolling upgrade to 1.0.0 - A metric named 'XX' already exists, can't register another one. - Flaky Unit test: KStreamKTableJoinIntegrationTest.shouldCountClicksPerRegionWithNonZeroByteCache - AbstractCoordinator not clearly handles NULL Exception - Request logging throws exception if acks=0 - KTable state restore fails after rebalance - Make loadClass thread-safe for class loaders of Connect plugins - Inconsistent protocol type for empty consumer groups - Broken symlink interrupts scanning the plugin path - NetworkClient should not return internal failed api version responses from poll - Topic can not be recreated after it is deleted - mBeanName should be removed before returning from JmxReporter#removeAttribute() - Connect: Struct equals/hashCode method should use Arrays#deep* methods - kafka-acls regression for comma characters (and maybe other characters as well) - punctuate with WALL_CLOCK_TIME triggered immediately - Update KStream JavaDoc with regard to KIP-182 - StackOverflowError in kafka-coordinator-heartbeat-thread - Fix StateRestoreListener To Use Correct Batch Ending Offset - NullPointerException on KStream-GlobalKTable leftJoin when KeyValueMapper returns null - Non-aggregation KTable generation operator does not construct value getter correctly - When enable trace level log in mirror maker, it will throw null pointer exception and the mirror maker will shutdown - Enforce layout of dependencies within a Connect plugin to be deterministic - Connect: Some per-task-metrics not working - Broker leaks memory and file descriptors after sudden client disconnects Test - KafkaStreamsTest fails in trunk - SelectorTest may fail with ConcurrentModificationException @ text @d1 1 a1 1 $NetBSD: distinfo,v 1.4 2017/11/01 13:31:39 fhajny Exp $ d3 7 a9 7 SHA1 (kafka_2.12-1.0.1.tgz) = f24f467be4baa07690214ad6995dc8e62e40b86c RMD160 (kafka_2.12-1.0.1.tgz) = cfbc0cfd056d9cd28be6c9c566905c1cef4282d4 SHA512 (kafka_2.12-1.0.1.tgz) = 935c0df1cf742405c40d9248cfdd1578038b595b59ec5a350543a7fe67b6be26ff6c4426f7c0c072ff4aa006b701502a55fcf7e2ced1fdc64330e3383035078c Size (kafka_2.12-1.0.1.tgz) = 44474706 bytes SHA1 (patch-bin_connect-distributed.sh) = 49e679e9d9d355921054a3723c96e519af871173 SHA1 (patch-bin_connect-standalone.sh) = 9c300a771dbf02f733edd2b8a8cdf9f62c9322b4 SHA1 (patch-bin_kafka-run-class.sh) = afba1d71d7bc40762068e27bda3c77159f7eb37c d11 2 a12 2 SHA1 (patch-bin_kafka-server-stop.sh) = 6121f1a519aff541d018f0e57946a872f4253fdf SHA1 (patch-config_server.properties) = 1bdb5bfded1325a3a5afe070dc3081f6b24185a8 @ 1.4 log @Update devel/kafka to 1.0.0 See upgrade notes: http://kafka.apache.org/100/documentation.html#upgrade New Features - KIP-72 Allow putting a bound on memory consumed by Incoming requests - Expose states of active tasks to public API - Add UnderMinIsrPartitionCount and per-partition UnderMinIsr metrics - KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines - Add ChangeReplicaDirRequest and DescribeReplicaDirRequest (KIP-113) - Add cumulative count attribute for all Kafka rate metrics - Add new metrics to support health checks - Handle SASL authentication failures as non-retriable exceptions in clients Improvement - Remove deprecated and slower scala JSON parser - Add Kafka Connect metrics - Allow setting of default topic configs via StreamsConfig - Improve DumpLogSegments offsets-decoder output format - Improve documentation of configuration listeners=PLAINTEXT://0.0.0.0:9092 - StateRestoreCallback begin and end indication - Update the usage description of clean shutdown file. - Add a '--group' option to the console consumer - Support Java 9 - Details of retention.bytes property at Topic level are not clear on how they impact partition size - Handle disk failure for JBOD (KIP-112) - Improve diagnostics for SASL authentication failures - Remove usage of java.xml.bind.* classes hidden by default in JDK9 - Make server start script work against Java 9 - Only log invalid user configs and overwrite with correct one - Changes to punctuate semantics (KIP-138) - Use per-task converters in Connect - Consumer perf tool should count rebalance time separately - Add ability to batch restore and receive restoration stats. - KIP-162: Enable topic deletion by default - Fix taskClass() method name in Connector and flush() signature in SinkTask - Add checks in integration tests to verify that threads have been shutdown - Add a new Kafka Streams example IoT oriented - KIP-168: Add GlobalTopicCount metric per cluster - Change Source offset commit message to info to match Sink behaviour - Replace -XX:+DisableExplicitGC with -XX:+ExplicitGCInvokesConcurrent in kafka-run-class - Streams should not suspend tasks twice - Replace the deprecated 'ConsumerOffsetChecker' in documentation - Consider removing date formatting from Segments class - Formatting verifiable producer/consumer output in a similar fashion - Streams reset tool should have same console output with or without dry-run - Improve Java doc for LeaderEpochFileCache.endOffsetFor() - Return topic authorization failed if no topic describe access - Explain that `client.id` is just used as a prefix within Streams - Using a logPrefix inside the StreamPartitionAssignor - Add --custom-ducktape flag to ducker-ak - Console Consumer overrides auto.offset.reset property when provided on the command line without warning about it. - Connect REST API should include the connector type when describing a connector - Test DeleteRecordsRequest in AuthorizerIntegrationTest - KafkaConsumer.subscribe() overload that takes just Pattern without ConsumerRebalanceListener - All ducktape services should store their files in subdirectories of /mnt - ShellTest: add tests for attempting to run nonexistent program, error return - Elevate log messages for denials to INFO in SimpleAclAuthorizer class - Refactor Streams to use LogContext - Refactor Producer to use LogContext - Refactor AdminClient to use LogContext - Refactor NetworkClient to use LogContext - Upgrade ducktape version to 0.7.1, and use new kill_java_processes - Implement KafkaPrincipalBuilder interface with support for SASL (KIP-189) - Remove unneeded synchronized keyword in StreamThread - Consistent logging of topic partitions - Add AdminClient.createPartitions() - Support unmapping of mapped/direct buffers in Java 9 - Handle SSL authentication failures as non-retriable exceptions in clients - Avoid call fetchPrevious in FlushListeners - Handle authentication failures from transactional producer and KafkaAdminClient - User StoreListener not available for global stores - Mention the need to close store iterators - Enable custom authentication plugins to return error messages to clients - There is something wrong in Kafka Streams document Bugs - NullPointerException in client on MetadataRequest - The kafka-consumer-perf-test.sh script help information print useless parameters. - Remove ConsumerOffsetChecker, deprecated in 0.9, in 0.11 - Make KStreamTestDriver extending from ExternalResource - Cleanup Kafka Streams builder API - completedReceives can contain closed channels - KIP-163: Offset fetch and commit requests use the same permissions - KafkaProducer.flush hangs when NetworkClient.handleCompletedReceives throws exception - Records from internal repartitioning topics should always use RecordMetadataTimestampExtractor - Augment KStream.print() to allow users pass in extra parameters in the printed string - Calling KafkaProducer.close() from multiple threads may cause spurious error - Kafka batch files does not support path with spaces - creating a window store with one segment throws division by zero error - Replace pattern matching with foreach where the case None is unused - Flaky ResetIntegrationTest - Kafka Streams keeps restoring state after shutdown is initiated during startup - streams task gets stuck after re-balance due to LockException - StreamsResetter doesn't allow custom Consumer properties - Producer buffer pool allocates memory inside a lock. - ProcessorContext.appConfigs() should return parsed/validated values - JmxReporter Fetch metrics for kafka.server should not be created when client quotas are not enabled - Using connect-console-sink/source.properties raises an exception related to "file" property not found - Clients get inconsistent connection states when SASL/SSL connection is marked CONECTED and DISCONNECTED at the same time - LogCleaner stopped due to org.apache.kafka.common.errors.CorruptRecordException - StreamsKafkaClient should not use StreamsConfig.POLL_MS_CONFIG - Created state changelog topics not logged correctly - ListOffsetResponse isn't logged in the right way with trace level enabled - Refactor kafkatest docker support - Rolling upgrade test for streams - bin/kafka-consumer-groups.sh failing to query offsets - kafka-run-class.sh doesn't check for empty $CLASSPATH - Documentation for altering topics - KafkaConsumer: High memory allocation rate when idle - KafkaConsumer offsetsForTimes should include partitions in result even if no offset could be found - Transformations - tranformations for value broken on tombstone events - The LastStableOffsetLag metric should be removed when partition is deleted - SchemaBuilder does not validate input. - KafkaConsumer.commitSync throws IllegalStateException: Attempt to retrieve exception from future which hasn't failed - Instable test QueryableStateIntegrationTest.shouldAllowToQueryAfterThreadDied - With transformations that mutate the topic-partition committing offsets should to refer to the original topic-partition - Support Power platform by updating rocksdb version - SchemaBuilder.type(Schema.Type) should not allow null. - Processor got uncaught exception: NullPointerException - Illegal state in SocketServer; attempt to send with another send in progress - ConsoleConsumer : --new-consumer option as deprecated - Group loading regression causing stale metadata/offsets cache - Streams should not abort transaction when closing zombie task - Improve error handling in SocketServer to reduce illegal states later - System test failure due to timeout starting Jmx tool - KafkaApis.handleWriteTxnMarkerRequest can return UNSUPPORTED_FOR_MESSAGE_FORMAT error on partition emigration - Make --new-consumer option as deprecated in all tools - SerializationException in doSend() masks class cast exception - ducktape kafka service: do not assume Service contains num_nodes - Consumer poll loop over the same record after a CorruptRecordException - Using _DUCKTAPE_OPTIONS has no effect on executing tests - Transient test failure: ResetConsumerGroupOffsetTest.testResetOffsetsToZonedDateTime - Support bulk attributes request on KafkaMbean where some attributes do not exist - LogDirFailureTest system test fails - queryable state window store range scan only returns results from one store - Add logging to distinguish between internally and externally initiated shutdown of Kafka - Sort processor node based on name suffix in TopologyDescription.toString() - Unit test shouldTogglePrepareForBulkLoadDuringRestoreCalls fails - Auto topic creation causes failure with older clusters - Bulk Restore Should Handle Deletes - [streams] 'null' values in state stores - In Jenkins, kafka.api.SaslSslAdminClientIntegrationTest failed with org.apache.kafka.common.errors.TimeoutException - Consumer should ensure consistent ordering between sync and async offset commits - System tests get exception RocksDBException: db has more levels than options.num_levels - Client-ids are not handled consistently by clients and broker - KafkaAdminClient thread should be daemon - Partition.makeLeader() should convert HW to OffsetMetadata before becoming the leader - Delete topic and re-create topic immediate will delete the new topic's timeindex - ShellTest.testRunProgramWithErrorReturn fails on macOS - Synchronization issue on flush - Reassigning a topic's partitions can adversely impact other topics - Move merge() from StreamsBuilder to KStream - Kafka server should halt if IBP < 1.0.0 and there is log directory failure - org.apache.kafka.streams.state.internals.Segments#segments method returns incorrect results when segments were added out of order - WorkerSinkTask failure exception should include cause - StoreChangeLogReader needs to restore partitions that were added post initialization - SocketServer.processNewResponses should not skip a response if exception is thrown - StoreChangelogReader should be resilient to broker-side metadata not available - ChangeLoggingWindowBytesStore needs to retain duplicates when writing to the log - KafkaStreams state transitions not correct - Speedup broker startup after unclean shutdown by reducing unnecessary snapshot files deletion - Reset thread interrupt state in case of InterruptedException - ReassignPartitionsCommand fails if default throttle/timeout used - Mx4jLoader.maybeLoad should only be executed if kafka_mx4jenable is set to true - Remove ZK dependency from Streams reset tool - ReplicaFetcherThread should not die due to replica in offline log directory - Expiring batches with idempotence enabled could cause data loss. - Kafka Connect applications should log info message when starting up - Transient failure in SslSelectorTest.testMuteOnOOM - Controller should read the latest IsrChangeNotification znodes when handling IsrChangeNotification event - CompositeReadOnlyWindowStore range fetch doesn't return all values when fetching with different start and end times - Fix minor typos and errors in the Kafka Streams turotial - Connect classloader isolation may be broken for JDBC drivers - Failure in Connect system test: ConnectRestApiTest - StreamBuilder#table and StreamsBuilder#globalTable should use serdes from Materialized - Producer IllegalStateException due to second deallocate after aborting a batch - NPE in NetworkClient - Producer uses unsupported ProduceRequest version against older brokers - Ineffective check of negative value in CompositeReadOnlyKeyValueStore#approximateNumEntries() - Remove all broker metrics during shutdown - Deadlock due to locking of DelayedProduce and group - RequestChannel.sendReponse records incorrect size for NetworkSend with TRACE logging - kip-120 changed internal streams topic naming - FailOnInvalidTimestamp does not log error - Streams State Restoration never completes when logging is disabled - disableLogging() causes partitions to not be consumed - Rename AlterReplicaDir to AlterReplicaDirs - Replication Fetcher thread for a partition with no data fails to start - Reject JoinGroup request from first member with empty protocol type/protocol list - Kafka Connect: Unsanitized workerID causes exception during startup - Transient failure: MemoryRecordsBuilderTest.convertToV1WithMixedV0AndV2Data - NoSuchElementException in markErrorMeter during TransactionsBounceTest - NPE in RecordAccumulator - Use the idempotent producer in the reassign_partitions_test - KafkaFuture timeout fails to fire if a narrow race condition is hit - System test failure: LogDirFailureTest - Integer overflow in log cleaner cleanable ratio computation - Kafka Request Handler deadlocks and brings down the cluster. - NoSuchMethodError when creating ProducerRecord in upgrade system tests - Running tools on Windows fail due to typo in JVM config - ducker-ak: add ipaddress and enum34 dependencies to docker image - Scanning plugin.path needs to support relative symlinks - replica dir not deleted after topic deletion - Streams quick start crashes Java on Windows - Reconnecting to broker does not exponentially backoff - Major performance issue due to excessive logging during leader election - Silent Data Loss in Kafka011 Transactional Producer - Transaction markers are sometimes discarded if txns complete concurrently - High memory usage on controller during partition reassignment Task - Update Jackson dependencies (from 2.8.5 to 2.9.x) - Enable PowerMock tests when running on Java 9 - Upgrade RocksDB dependency to legally acceptable version Test - Support passing ZK chroot in system tests - Add unit tests for ClusterConnectionStates Sub-task - Replace ControlledShutdown{Request,Response} with org.apache.kafka.common.requests equivalent - Remove temporary mapping to deserialize functions in RequestChannel - Task migration during rebalance callback process could lead the obsoleted task's IllegalStateException - Improve test coverage of StreamsKafkaClient - Improve test coverage of RocksDBStore - Improve test coverage of CompositeReadOnlySessionStore - Improve test coverage of CompositeReadOnlyKeyValueStore - Improve test coverage InMemoryKeyValueLoggedStore - Improve test coverage of CachingKeyValueStore - Improve test coverage UsePreviousTimeOnInvalidTimestamp - Optimize RocksDb config for fast recovery/bulk load - KeyValueStore.put may throw exception unrelated to the current put attempt - Replace zkClient.getChildren method with zkUtils.getChildren - Options for handling corrupt data during deserialization - Improve exception handling for state stores - Distinguish abortable failures in transactional producer - Unexpected state transition Dead to PendingShutdown - Replace zkClient.subscribe*Changes method with an equivalent zkUtils method - Replace zkClient.exists method with zkUtils.pathExists - Replace zkClient.delete* method with an equivalent zkUtils method - Idempotent producer should not require max.in.flight.requests.per.connection=1 - introduce async ZookeeperClient - Check exception handling in all streams tests - Autogenerate Producer sender metrics - Provide a simple way for custom storage engines to use streams wrapped stores (KIP-182) - Add new API methods to KTable - Add new API methods to KGroupedStream - Add new API methods to KGroupedTable - Add Topology and deprecate TopologyBuilder - Add StreamsBuilder and deprecate KStreamBuilder - Refactor KeyValueStore hierarchy so that MeteredKeyValueStore is the outermost store - Refactor WindowStore hierarchy so that Metered Store is the outermost store - Add the archetype project along with "write applications" web docs. - Refactor SessionStore hierarchy - Add the Trogdor fault injection daemon - Add ducktape integration for the Trogdor Fault injection daemon - Tighten up situations where OutOfOrderSequence may be returned - Fix transient unit test failure in trogdor coordinator shutdown - Add Printed class and KStream#print(Printed) - Add Produced class and new to and through overloads to KStream - Add Serialized class and KStream groupBy and groupByKey overloads - Add Joined class and relevant KStream join overloads - Update Docs - Add Consumed class and overloads to StreamBuilder - Upgrade Guide doc changes for KIP-130 - Add groupBy(KeyValueMapper, Serialized) to KTable - Add filter, filterNot, mapValues and Materialized to KTable - Add WindowedKStream interface and implemenation - Add Materialized overloads to StreamBuilder - Transactions system test should check for message order - Create Connect metrics for connectors - Create Connect metrics common to source and sink tasks - Create Connect metrics for source tasks - Create Connect metrics for sink tasks - Create Connect metrics for workers - Create Connect metrics for worker rebalances - Add `hasOffset` and `hasTimestamp` methods to RecordMetadata to indicate whether metadata is available - Add Materialized overloads to WindowedKStream - Add SessionWindowedKStream - Deprecate KTable#to and KTable#through - Improve ProcessorStateManager exception handling - User Callback Exceptions need to be handled properly - Add generated documentation for Connect metrics @ text @d1 1 a1 1 $NetBSD: distinfo,v 1.3 2017/07/04 14:14:46 fhajny Exp $ d3 4 a6 4 SHA1 (kafka_2.12-1.0.0.tgz) = 64bd50ae5b3e99d110226b9b8820df4f5009e306 RMD160 (kafka_2.12-1.0.0.tgz) = 15c62e044b0ad1bbe29638acdb443d097873b30a SHA512 (kafka_2.12-1.0.0.tgz) = 1b647b7f392148aa2b9d47550a1502e50be4b005c70c82dae03065b89484e66400528be40ca2d54f35eb2c0e70f35c88a04777c2b625daa5d5546caab4ed6818 Size (kafka_2.12-1.0.0.tgz) = 44352403 bytes @ 1.3 log @Update devel/kafka to 0.11.0.0. New Feature - KIP-146: Support per-connector/per-task classloaders in Connect - Add Record Headers - Add purgeDataBefore() API in AdminClient - Add KStream.peek(ForeachAction) - Add a tool to Reset Consumer Group Offsets - Add internal leave.group.on.close config to consumer - Add Exactly-Once Semantics to Streams - Implement Transactional Coordinator - Range Scan for Windowed State Stores Improvement - Extend wire protocol to allow CRC32C - Cluster collection returning methods should never return null - Add Prompt to kafka-console-producer - Allow users greater access to register custom streams metrics - Exponential backoff for broker reconnect attempts (KIP-144) - Add JMH module for Benchmarks - Split the ProducerBatch and resend when received RecordTooLargeException - Allow per stream/table timestamp extractor - support throttling on request rate - Replication Quota Tests: Ensure ZK updated before tests start - REST configuration not visible in connector properties config files - TopicCommand --describe shows topics marked for deletion as under-replicated and unavailable (KIP-137) - RocksDB checkpoint files lost on kill -9 - Treat null-key records the same way for joins and aggreations - Per listener security setting overrides (KIP-103) - Kafka does not offer kafka-configs.bat on Windows box - Parametrize streams benchmarks to run at scale - Error message from Struct.validate() should include the name of the offending field. - Add application.id to StreamThread name - Improve Streams Reset Tool console output - Add Float serializer, deserializer, serde - Exploit #peek to implement #print() and other methods - The Kafka build should run findbugs - Inner classes which don't need a reference to the outer class should be static - Fix findbugs warnings in kafka-tools - throw NoOffsetForPartitionException once for all assigned partitions from poll - Stream round-robin scheduler is inneficient - Fix several FindBugs warnings in Clients and Connect - SCRAM secret's should be better protected with Zookeeper ACLs - set internal.leave.group.on.close to false in KafkaStreams - Add listener tag to socket-server-metrics.connection-... metrics - Followups from KIP-101 - We shouldn't pass the underlying exception to RetriableCommitFailedException when an async offset commit fails. - Optionally print out metrics after running the perf tests - ReassignPartitionsCommand should protect against empty replica list assignment - Censor SCRAM config change logging - DumpLogSegments should not open index files with `rw` - Add support for reading PID snapshot files to DumpLogSegments - Improve internal Task APIs - Trunk compatibility tests should test against 0.10.2 - Improve message for Kafka failed startup with non-Kafka data in data.dirs - Controlled shutdown timeout message improvement - reassign-partitions to check if broker of ID exists in cluster - Add option "dry run" to Streams application reset tool - AdminClient: add controller and clusterId methods to DescribeClusterResults - KIP-153: Include only client traffic in BytesOutPerSec metric - Application Reset Tool does not need to seek for internal topics - New Short serializer, deserializer, serde - Change Default unclean.leader.election.enabled from True to False - Sticky Assignor should not cache the calculated assignment (KIP-54 follow-up) - Support ExtendedDeserializer in Kafka Streams - Remove Broker ACL resource type - Request log should log throttle time - Generate javadoc for AdminClient and show configs in documentation - Annoying braces showed on log.error using streams Bug - Implement generation/term per leader to reconcile messages correctly - Hold the produce request with ack > 1 in purgatory until replicas' HW has larger than the produce offset (KIP-101) - Delete consumer offsets from kafka once the topic is deleted - Clean up Controller Object on forced Resignation - ConsumerGroupCommand throws GroupCoordinatorNotAvailableException when describing a non-existent group before the offset topic is created - SASL unit tests dont work with IBM JDK - Follower Broker cannot start if offsets are already out of range - Remove deprecated producer configs. - Kafka default -Xloggc settings should include GC log rotation flags - Streams is creating two ProducerRecords for each send via RecordCollector - Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs - Log should check the return value of dir.mkdirs() - KIP-115: __consumer_offsets wrong number of replicas at startup - Issue with processing order of consumer properties in console consumer - Exit Strategy: using exceptions instead of inline invocation of exit/halt - Documentation still refers to AsyncProducer and SyncProducer - KIP-151: Connect REST API should expose whether each connector is a source or sink - KTable doesn't work with ProcessorTopologyTestDriver in Kafka 0.10.1.0 - rat license check not running in Jenkins - When using ProcessorTopologyTestDriver, the combination of map and .groupByKey does not produce any result - Set more conservative default values on RocksDB for memory usage - Significant startup delays in KStreams app - Kafka should not require SSL trust store password - When the destination brokers are down or misconfigured in config, Streams should fail fast - Connect Producer and Consumer ignore ssl parameters configured for worker - Annotate integration tests and provide gradle build targets to run subsets of tests - Controller send thread can't stop when broker change listener event trigger for dead brokers - the argument of shell in doc wrong and command parsed error - Kafka Streams allows you to provide strings with illegal characters for internal topic names - gradle-generated core eclipse project has incorrect source folder structure - Refresh consumer metadata more frequently for unknown subscribed topics - KIP-154: Connect should create internal topics - Fix Streams window retention policy - KIP-100 api changes break Java 8 lambda expressions in some cases - Python VerifiableConsumer service has thread-safety bug for event_handlers - Avoid unnecessary task movement across threads during rebalance - OffsetValidationTest fails validation with "Current position greater than the total number of consumed records" - StreamsKafkaClient drops security configs - Fix Transient failure in BrokerApiVersionsCommandTest.checkBrokerApiVersionCommandOutput - Implement remaining KIP-66 SMTs - Fix logic for re-checking if internal topic is ready - connect jars are missing LICENSE/NOTICE files - Timed out waiting for consumption in OffsetValidationTest.test_broker_failure - Clean up of state directories can possibly remove stores that are about to be used by another thread - KafkaConsumer#commitSync should clone its input - Fix deadlock issue during MM shutdown - Remove generic type of class ClientState - Memory leak in RecordAccumulator.append - Streams_bounce test failing occassionally - fix join-time-max and sync-time-max MeasurableStat type - Streams Simple Benchmark MB/sec calculation is not correct for Join operations - SimpleBenchmark test fails for streams - Improve NetworkClient trace logging of request details - Connect WorkerSinkTask is missing checks for NO_TIMESTAMP - org.apache.kafka.clients.producer.KafkaProducerTest#testConstructorFailureCloseResource and Similar Tests are Failing on some Systems (127.0.53.53 Collision Warning) - Implement graceful handling for improperly formed compressed message sets - Kafka client Heartbeat thread use all the cpu. - Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py - KIP-128: Blackbox or pass through converter or ByteArrayConverter for connect - Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' not configured per topic. - ProcessorTopologyTestDriver does not forward extracted timestamps to internal topics - Kafka cannot recover after a disk full - Kafka Streams - unable to add state stores when using wildcard topics on the source - Fix some findbugs warnings in Kafka Java client - Streams State transition ASCII diagrams need fixing and polishing - KafkaConsumer: ConsumerConfig gets logged twice. - docker/run_tests.sh should set up /opt/kafka-dev to be the source directory - SchemaBuilder should be more lax about checking that fields are unset if they are being set to the same value - ReplicaFetchThread may fail to create due to existing metric - ZookeeperLeaderElector not respecting zookeeper.set.acl - Fix some findbugs warnings in Kafka Streams - ProcessorTopologyTestDriver does not work when using .through() - Config validation in Connector plugins need to compare against both canonical and simple class names - NetworkClient should only consider a connection to be fail after attempt to connect - Stream thread getting into deadlock state while trying to get rocksdb lock in retryWithBackoff - SessionStore.fetch(key) is a performance bottleneck - Struct SchemaBuilder should not allow duplicate fields. - Querying window store may return unwanted keys - Kafka Secure Migrator tool doesn't secure all the nodes - Kafka console consumer property is ignored - Kafka Connect does not log connector configuration errors - processstreamwithcachedstatestore and other streams benchmarks fail occasionally - kafka.request.logger TRACE regression - Fix findbugs "default character set in use" warnings - Fix findbugs "format string should use %n rather than \n" in tools - Fix findbugs warnings in kafka-core - Make ProduceRequest thread-safe - Utils#delete should correctly handle I/O errors and symlinks - Remove unused code for reading Shell command stdout and add unit test - Add streams tests with brokers failing - Document that stores must not be closed when Processors are closed - Fix findbugs warnings in Kafka-Connect-API - Add a configurable delay to the initial consumer group rebalance - KStreamsTestDriver fails with NPE when KStream.to() sinks are used - Transformation Key/Value type references should be to class name(), not canonicalName() - Batch resetting offsets in Streams' StoreChangelogReader - Kafka Connect: Offset committing times out before expected - Fix an "unread field" findbugs warning in streams examples - Suppress findbugs warnings about machine-generated code in jmh-benchmarks - remove controller concurrent access to non-threadsafe NetworkClient, Selector, and SSLEngine - Delete the kafka to prefix the name of the keystore and truststore file will be more suitable - kafka-connect: fix findbugs issues in connect/runtime - testReprocessingFromScratch unit test failure - Fix findbugs warnings in kafka-clients - Fix remaining findbugs warnings in Kafka Streams - StreamThread should catch InvalidTopicException - JoinIntegrationTest fails occasionally - Fail the build when findbugs fails - SSL Channel not ready but tcp is established and the server is hung will not sending metadata - running multiple kafka streams instances causes one or more instance to get into file contention - Increase number of Streams producer retries from the default of 0 - InFlightRequests#isEmpty() always returns false - Add FindCoordinatorRequest RPC stub and update InitPidRequest for KIP-98 - NullPointerException while using GlobalKTable in KafkaStreams - Chroot check should be done for each ZkUtils instance - Avoid DNS reverse lookup in security-critical TLS code path - Kafka Streams skipped-records-rate sensor producing nonzero values even when FailOnInvalidTimestamp is used as extractor - Kafka Streams stuck rebalancing after exception thrown in rebalance listener - Defer exception to the next pollOnce() if consumer's fetch position has already increased - PartitionRecords.fetchRecords(...) should defer exception to the next call if iterator has already moved across any valid record - two versions of jackson-annotations-xxx.jar in distribution tgz - Update topic expiry time in Metadata every time the topic metadata is requested - some spelling error in code comment - Kafka Streams SessionStore.findSessions javadoc broken - ThreadCacheTest.cacheOverheadsSmallValues fails intermittently - Replica Deletion Regression from KIP-101 - ProducerPerformanceService failing due to parsing error - Remove KafkaController's incrementControllerEpoch method parameter - Transient test failure SocketServerTest.testMetricCollectionAfterShutdown - Controller Health Metrics (KIP-143) - Flaky ResetIntegrationTest - Windows platform does not offer kafka-broker-api-versions.bat - MinTimestampTracker uses confusing variable names - LZ4 decompression is 4-5x slower than Snappy on small batches / messages - SetSchemaMetadata does not replace the schemas in structs correctly - streams task gets stuck after re-balance due to LockException - KafkaConsumer.close should be idempotent - KafkaAdminClientIntegration test should wait until metadata is propagated to all brokers - CachingSessionStore doesn't fetchPrevious correctly. - SASL tests failing with Could not find a 'KafkaServer' or 'sasl_plaintext.KafkaServer' entry in the JAAS configuration - RocksDb might stall in environments with 1 core only - RocksDbStore#openIterators should be synchronized, since it is accessed from multiple threads - Percentilles are calculated incorrectly - CachingSessionStore doesn't use the default keySerde. - RocksDBSessionStore doesn't use default aggSerde. - KafkaConsumer should not skip a corrupted record after throwing an exception. - IllegalStateException in ensureOpenForRecordAppend - Small JavaDoc fix for AdminClient#describeTopics - Cached Session/Window store may return error on iterator.peekNextKey() - NullPointerException (NPE) in SourceNodeRecordDeserializer.deserialize - Reflections logs excessive warnings when scanning classpaths - Recommended values for Connect transformations contain the wrong class name - Kafka broker fails to start if a topic containing dot in its name is marked for delete but hasn't been deleted during previous uptime - Regression in on-disk log size when using Snappy compression with 0.8.2 log message format - GlobalKTable does not checkpoint offsets after restoring state - Tests which delete singleton metrics break subsequent metrics tests - handleFetchRequest should do down conversion after throttling - kakfa-clients consume 100% CPU with manual partition assignment when network connection is lost - Move ACLs, Config, NodeVersions classes into org.apache.kafka.common - Follow-up improvements for consumer offset reset tool (KIP-122) - kafka-console-consumer: `--value-deserializer` is not working but `--property value.deserializer` does - One StopReplicaRequest will caused two Responses - Do not apply exponential backoff if users have overridden reconnect.backoff.ms - PlainSaslServerFactory should allow a null Map in getMechanismNames - FetchRequest doesn't implement toString - Missing logging information in ReplicaFetcher - Stores not queryable after one thread died - Log cleaning can increase message size and cause cleaner to crash with buffer overflow - MemoryRecords.filterTo can return corrupt data if output buffer is not large enough - AdminClient: add close with timeout, fix some timeout bugs - Connection Lose during Kafka Kerberos Renewal process - Replica list in the metadata cache on the broker may have different order from zookeeper - rocksdb.config.setter must be a class instance, not a class name - ListGroup requires Describe on Cluster, but the command-line AclCommand tool does not allow this to be set - There is a Misspell in ResetIntegrationTest - Change message.timestamp.difference.max.ms back to Long.MaxValue - Some socket connections not closed after restart of Kafka Streams - KafkaConsumer occasionally hits IllegalStateException - Modify Unstable annotations in Streams API - baseTimestamp should always have a create timestamp - MirrorMaker not preserving headers - Down-converted uncompressed batches should respect fetch offset - Add EOS integration tests for Streams API - Add EOS system tests for Streams API - Kafka Streams skipped-records-rate sensor produces nonzero values when the timestamps are valid - ConsoleConsumer prints out object addresses rather than what is expected - AdminClient gets "server returned information about unknown correlation ID" when communicating with older brokers - Transactions: Concurrent transactional consumer loses messages when there are broker bounces - Transient test failure: KafkaConsumerTest.testChangingRegexSubscription - Log recovery can fail if topic names contain one of the index suffixes - Transactional Producer allows batches to expire and commits transactions regardless - KafkaAdminClient#timeoutCallsInFlight does not work as expected - Distributed Herder Deadlocks on Shutdown - Add more AdminClient checks to ClientCompatibilityTest - Log cleaner fails due to large offset in segment file - Console consumer offset commit regression - TransactionCoordinator doesn't complete transition to PrepareCommit state - TransactionCoordinator doesn't complete transition to CompleteCommit - ZkUtils.getAllPartitions() may fail if a topic is marked for deletion - Multiple produce request failures causes invalid state transition in TransactionManager - Transient test failure: SaslPlainSslEndToEndAuthorizationTest.testNoProduceWithDescribeAcl - Streams producer `client.id` are not unique for EOS - TimestampConverter's "type" config conflicts with the basic Transformation "type" config - Flaky test TransactionsTest.testReadCommittedConsumerShouldNotSeeUndecidedData - Scripts to startup Connect in system tests have too short a timeout - Update java docs for consumer and producer to be up to date for EOS - Producer fails with NPE if compressed V0 or V1 record is larger than batch size - MemoryRecordsBuilder.hasRoomfor doesn't take into account the headers while computing available space - Controller incorrectly logs rack information when new brokers are added - Connector validate REST endpoint returning duplicate entries in "groups" - Connector config validation REST API endpoint not including fields for transformations - TransactionalProducer sleeps unnecessarily long during back to back transactions - org.apache.kafka logging should go to server.log - The ProducerPerformance tool should support transactions - Connect validation API stops returning recommendations for some fields after the right sequence of requests Task - Remove deprecated APIs for 0.11.0.0 - Drop support for Scala 2.10 (KIP-119) - Remove unstable markers from Connect APIs - KTable materialization and improved semantics Test - Transient failure in ZooKeeperSecurityUpgradeTest.test_zk_security_upgrade with security_protocol = SASL_PLAINTEXT, SSL - Test with two SASL_SSL listeners with different JAAS contexts - Implement KIP-98 transactional methods in the MockProducer - Add cases for concurrent transactional reads and writes in system tests - SyncProducerTest.testReachableServer has become flaky Sub-task - KIP-54: Add rebalance with a minimal number of reassignments to server-defined strategy list - ProducerSendTest.testCloseWithZeroTimeoutFromCallerThread Transient Failure - Transient Failure in kafka.api.PlaintextProducerSendTest.testFlush - Mark the old Scala consumer and related classes as deprecated - Create Java Admin Client - Implement KIP-140 RPCs and APIs for creating, altering, and listing ACLs - Describe/Alter Configs protocol, server and client (KIP-133) - Transient test failure: kafka.api.SslProducerSendTest.testSendNonCompressedMessageWithCreateTime - Unstable test KStreamRepartitionJoinTest.shouldCorrectlyRepartitionOnJoinOperations - Transient test failure: ConsumerBounceTest.testConsumptionWithBrokerFailures - Transient failure in QueryableStateIntegrationTest.queryOnRebalance - Address 2.12 eta-expansion warnings - Update release process so that Scala 2.12 artifacts are published - Run tests on travis-ci using docker - Transient failure in org.apache.kafka.clients.consumer.KafkaConsumerTest.testWakeupWithFetchDataAvailable - QueryableStateIntegrationTest.shouldNotMakeStoreAvailableUntilAllStoresAvailable is occasionally failing on jenkins - Improve test coverage of ProcessorStateManager - Improve test coverage of StreamsPartitionAssignor - Improve test coverage of ProcessorTopology - Improve test coverage AbstractProcessorContext - Improve test coverage of GlobalStreamThread - Improve test coverage StreamTask - Improve test coverage GlobalStateManagerImpl - Improve test coverage KStreamBuilder - Improve test coverage MemoryLRUCache - Improve test coverage of CompositeReadOnlyWindowStore - Improve test coverage KafkaStreams - Improve test coverage TopologyBuilder - Transient test failure: ConsumerCoordinatorTest.testHeartbeatThreadClose - Message format changes for idempotent/transactional producer - Implement idempotent producer - Implement transactional clients - Transient test failure: org.apache.kafka.streams.integration.JoinIntegrationTest.shouldCountClicksPerRegion (again) - Consider disabling record level CRC checks for message format V2 - Failure in kafka.admin.DescribeConsumerGroupTest.testDescribeExistingGroupWithNoMembersWithNewConsumer - Implementation and unit tests for request handler thread utilization quota - Add network handler thread utilization to request quota calculation - Add documentation for request quotas - Transient failure of AdminClientTest.testDeleteRecordsWithException - Add producer per task support - Add request/response classes for transactions (KIP-98) - KeyValueStore.put may throw exception unrelated to the current put attempt - Exactly-once upgrade notes - Update Message Delivery Semantics section to take into account KIP-98 - convert kafka controller to a single-threaded event queue model - Additional validation in validateMessagesAndAssignOffsets - Reconsider default retries for idempotent producer - add controller integration tests - Load only batch header when rebuilding producer ID map - Refactor AdminUtils to use zkUtils methods instad of zkUtils.zkClient - remove preferred replica election state from ControllerContext - Implement transaction index for KIP-98 - shouldInnerLeftJoin unit test fails - TransactionCoordinator - Check inter broker protocol and message format and raise errors if incompatible - TransactionCoordinator - Add ACL check for each request - Change InterBrokerSendThread to use a Queue per broker - WriteTxnMarkers and complete commit/abort on partition immigration - Abort long running transactions - Move coordinatorEpoch from WriteTxnMarkerRequest to TxnMarkerEntry - KafkaProducer's TransactionManager needs a review on synchronization - Refactor TransactionCoordinator in-memory structure and error handling logic - KIP-98 : broker side handling for the TxnOffsetCommitRequest - Add a reference to AdminClient to docs/api.html - TC should not accept empty string transactional id - Transient failure: ControllerIntegrationTest.testPreferredReplicaLeaderElection - Log connection termination during authentication - Transient failure: ControllerIntegrationTest.testControllerMoveIncrementsControllerEpoch - Transient failure: RequestQuotaTest.testResponseThrottleTime - Avoid expensive initialization of producer state when upgrading - Add Integration tests for transactional producer - Autogenerate Consumer Fetcher metrics - LogCleaner should be transaction-aware - Handle topic deletion for ongoing transactions - SaslScramSslEndToEndAuthorizationTest.testNoConsumeWithoutDescribeAclViaSubscribe - TransactinoCoordinator does not bump epoch when aborting open transactions - Consumer GroupCoordinator should continue to materialize committed offsets in offset order even for transactional offset commits - Remove retention time from TxnOffsetCommit RPC - Transaction index recovery does not snapshot properly - Producer should drop queued sends when transaction is aborted - Fix flaky test LogCleanerTest.testCommitMarkerRemoval - move all partition and replica state transition rules into their states - TransactionalId authorization should imply ProducerId authorization - Producer should not send AbortTxn unless transaction has actually begun - TransactionsBounceTest occasionally sees INVALID_TXN_STATE errors - TransactionBounceTest occasionally fails due to partition errors - KafkaConsumer.committed() should get latest committed offsets from the server - Review and improve AdminClient Javadoc for the first release (KIP-117) - Review and potentially tweak AdminClient API for the initial release (KIP-117) - TransactionCoordinator must expire transactionalIds - Protect concurrent access to the cached transaction status - System tests for KIP-98 / transactions - Transactions integration test: Use factory methods to keep track of open producers and consumers and close them all on tearDown - Update clients and server code to make sure that epoch and sequence numbers wrap around - AdminClient should not trigger auto creation of topics - Fix authorization checks in AdminClient - TC should handle UNSUPPORTED_FOR_MESSAGE_FORMAT in WriteTxnMarker response - reset ControllerContext during resignation - Update KIP-98 to reflect changes during implementation. - Update produce/fetch throttle time metrics for any request throttle - Resolve AddPartitions response error code inconsistency - Transactions system test with hard broker bounces fails sporadically - Add additional test cases for batch splitting to ensure idempotent/transactional metadata is preserved - Broker clean bounce test puts the broker into a 'CONCURRENT_TRANSACTIONS' state permanently - Broker returns messages beyond "latest stable offset" to transactional consumer in read_committed mode - StackOverFlow error in transaction coordinator - Producer attempts to send transactional messages before adding partitions to transaction - Fix regression in compressed message iteration affecting magic v0 and v1 - Transactions: Concurrent transactional consumer reads aborted messages - Last Stable Offset not returned in Fetch request - Transactional producer cannot find coordinator when trying to abort transaction after error - Transactional producer aborts batches incorrectly in abortable error state - Producer IllegalStateException: Batch has already been completed - Produce state lost if no snapshot retained - TransactionalMessageCopier should be force killed on test shutdown - UnsupportedOperationException in WriteTxnMarkers handler - Consumer should use last offset from batch to set next fetch offset - read current brokers from zookeeper upon processing broker change @ text @d1 1 a1 1 $NetBSD: distinfo,v 1.2 2017/05/10 18:14:06 fhajny Exp $ d3 4 a6 4 SHA1 (kafka_2.12-0.11.0.0.tgz) = 3dac78b99cd84ff9a8c7b6d12025c391a2b34008 RMD160 (kafka_2.12-0.11.0.0.tgz) = 47372c2f58a72a11a22ccae1645ce85fc482a3b7 SHA512 (kafka_2.12-0.11.0.0.tgz) = 60b44930700ab42e4a0e3945b6eb32db422d2e62388b4fee68aef97aea939fd252ebd0be39b36a4e6149aff9dcbd5b0578095d8a8812bd75111a65299aa5af55 Size (kafka_2.12-0.11.0.0.tgz) = 36644420 bytes @ 1.2 log @Update devel/kafka to 0.10.2.1 Improvement - SCRAM secret's should be better protected with Zookeeper ACLs Bug - Refresh consumer metadata more frequently for unknown subscribed topics - OffsetValidationTest fails validation with "Current position greater than the total number of consumed records" - Failure in kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py - Broker level configuration 'log.segment.bytes' not used when 'segment.bytes' not configured per topic. - Kafka Streams - unable to add state stores when using wildcard topics on the source - Streams State transition ASCII diagrams need fixing and polishing - KafkaConsumer: ConsumerConfig gets logged twice. - docker/run_tests.sh should set up /opt/kafka-dev to be the source directory - Config validation in Connector plugins need to compare against both canonical and simple class names - Stream thread getting into deadlock state while trying to get rocksdb lock in retryWithBackoff - SessionStore.fetch(key) is a performance bottleneck - log.message.timestamp.type=LogAppendTime breaks Kafka based consumers - Querying window store may return unwanted keys - Kafka Secure Migrator tool doesn't secure all the nodes - Kafka Connect does not log connector configuration errors - Make ProduceRequest thread-safe - Add streams tests with brokers failing - Document that stores must not be closed when Processors are closed - remove controller concurrent access to non-threadsafe NetworkClient, Selector, and SSLEngine - testReprocessingFromScratch unit test failure - StreamThread should catch InvalidTopicException - running multiple kafka streams instances causes one or more instance to get into file contention - Increase number of Streams producer retries from the default of 0 - Defer exception to the next pollOnce() if consumer's fetch position has already increased - ThreadCacheTest.cacheOverheadsSmallValues fails intermittently - KafkaConsumer.poll throws IllegalStateException @ text @d1 1 a1 1 $NetBSD: distinfo,v 1.1 2017/02/28 08:17:28 fhajny Exp $ d3 4 a6 4 SHA1 (kafka_2.12-0.10.2.1.tgz) = da6f2b1ae1feb25aac29a2a72e0a271855fcde8c RMD160 (kafka_2.12-0.10.2.1.tgz) = f473c5cc605ab5480485ad7e0e4fda2021d960f0 SHA512 (kafka_2.12-0.10.2.1.tgz) = 48884838980ff182fe1bb089b5f8db91aaa190dad9f9cbf24d762ba8afb897546f66d66ce110eef87ad1541e06564e5ea455e4698a23fbd296d971e033953bbc Size (kafka_2.12-0.10.2.1.tgz) = 34046244 bytes d9 1 a9 1 SHA1 (patch-bin_kafka-run-class.sh) = 3c3202feb125a575346d91465824d9dc8bcfa6e8 @ 1.1 log @Import Kafka 0.10.2.0 as devel/kafka. Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies. @ text @d1 1 a1 1 $NetBSD$ d3 4 a6 4 SHA1 (kafka_2.12-0.10.2.0.tgz) = 655a4b415b0913117ebff88888ea526a3c2a4a06 RMD160 (kafka_2.12-0.10.2.0.tgz) = bbcb13b9e0b16af72efd4a76258d555c07b99d8f SHA512 (kafka_2.12-0.10.2.0.tgz) = 79115d105d68457e437fe539d5f61f404f0a04ca6096299367fc03a3f50f928b9858b6cbf41160ec303de6cdaba8320e12f1219faae467cd08a775c9ac004141 Size (kafka_2.12-0.10.2.0.tgz) = 34021573 bytes @