<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anand Satheesh on Medium]]></title>
        <description><![CDATA[Stories by Anand Satheesh on Medium]]></description>
        <link>https://medium.com/@anands282?source=rss-9950044d7c81------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*R0KOKWAe6WBLgPf1TSB3Ew.png</url>
            <title>Stories by Anand Satheesh on Medium</title>
            <link>https://medium.com/@anands282?source=rss-9950044d7c81------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 03:31:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@anands282/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Handling Large number of Small Files in Big Data Frameworks]]></title>
            <link>https://medium.com/@anands282/handling-large-number-of-small-files-in-big-data-frameworks-5907d0add9a6?source=rss-9950044d7c81------2</link>
            <guid isPermaLink="false">https://medium.com/p/5907d0add9a6</guid>
            <category><![CDATA[etl-pipeline]]></category>
            <category><![CDATA[parquet]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Anand Satheesh]]></dc:creator>
            <pubDate>Tue, 01 Oct 2024 05:12:43 GMT</pubDate>
            <atom:updated>2024-10-01T05:12:43.276Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently I came across an interesting programming challenge, I am ingesting change data capture(CDC) data from a Relational Database into a datalake. The jobs which I run on this data using Apache Spark is taking way too much time to complete and occasionaly gets stuck and fail. The data volume is not too much(of the order of GBs). On closer obervation I saw that the issue could be because the job has to process large number of small files.</p><p>Every X minutes the job pulls the change logs from the source Relational DB and writes as a parquet (append) to data lake path. What this was doing is that, even though the volume of change logs are of the order of MBs, a large number of small files are getting formed under each parquet which was leading to the large number of small files problem. This article details the strategies I used and some other popular tips for tackling this scenario.</p><p>If a Parquet file consists of 100 small files, Spark will likely treat each file as a separate partition by default. Here’s how it works:</p><ul><li><strong>Partitioning Behavior</strong>: When Spark reads data from multiple files (such as 100 small Parquet files), it creates one partition per file. Each partition is a logical chunk of data that can be processed by a single task. Therefore, in your case, with 100 small Parquet files, Spark would create 100 partitions.</li><li><strong>Impact of Small Partitions</strong>: Since each partition would represent one small file, this can lead to inefficient execution due to the following reasons:</li><li><strong>Small Tasks</strong>: A large number of small partitions means many small tasks, which increase scheduling and task management overhead.</li><li><strong>Underutilization</strong>: Small partitions can result in underutilization of the available resources (like CPU and memory).</li></ul><p>Handling small files efficiently in Apache Spark is crucial because they can cause performance bottlenecks due to excessive metadata overhead and unnecessary job scheduling. Here are several strategies to optimize small file handling in Spark:</p><h3>1. Merge Small Files</h3><ul><li><strong>Combine small files into larger ones:</strong> Merging many small files into fewer large ones can help reduce the number of partitions and tasks, decreasing overhead.</li><li><strong>Use Hadoop DistCp</strong> for merging files directly.</li><li>Alternatively, you can use a Spark job to read small files, repartition the DataFrame, and write the output as larger files using formats like <strong>Parquet</strong> or <strong>ORC</strong>.</li></ul><p>Example:</p><pre>df = spark.read.format(&quot;csv&quot;).load(&quot;path/to/small/files&quot;)<br>df.repartition(n).write.format(&quot;parquet&quot;).save(&quot;path/to/output&quot;)</pre><h3>2. Increase HDFS Block Size</h3><ul><li>Increasing the <strong>HDFS block size</strong> ensures that each file occupies a larger portion of a block, reducing the number of small files.</li><li>Set the block size via dfs.blocksize in the Hadoop configuration (e.g., hdfs-site.xml).</li></ul><h3>3. File Compaction</h3><ul><li><strong>Compaction</strong> strategies can be used in certain frameworks (like Hudi or Delta Lake) to automatically merge small files over time.</li><li>In Delta Lake:</li></ul><pre>deltaTable = DeltaTable.forPath(spark, &quot;path/to/table&quot;)<br>deltaTable.optimize().executeCompaction()</pre><h3>4. Coalesce and Repartition</h3><ul><li>Use <strong>coalesce()</strong> for merging partitions into fewer partitions when writing output, which helps in reducing small output files.</li><li><strong>repartition()</strong> can also be used to redistribute data more evenly if needed.</li></ul><p>Example:</p><pre>df.coalesce(1).write.format(&quot;parquet&quot;).save(&quot;output/path&quot;)</pre><h3>5. Broadcast Joins</h3><ul><li>When working with small datasets, use <strong>broadcast joins</strong> to avoid shuffling large amounts of data, as small files can increase the shuffle overhead.</li></ul><p>Example:</p><pre>small_df = spark.read.parquet(&quot;small/file&quot;)<br>large_df = spark.read.parquet(&quot;large/file&quot;)<br>joined_df = large_df.join(broadcast(small_df), &quot;key&quot;)</pre><h3>6. Use Proper File Formats</h3><ul><li>Use efficient file formats like <strong>Parquet</strong> or <strong>ORC</strong> instead of text or CSV formats. These formats support larger blocks, compression, and can reduce the number of small files written to disk.</li></ul><h3>7. Adjust Spark Settings</h3><ul><li><strong>Increase the number of partitions</strong>: If you’re reading from small files, setting spark.sql.files.maxPartitionBytes to a smaller value can prevent underutilization of resources.</li><li><strong>File Source Max Partition Bytes</strong>: Adjust the spark.sql.files.maxPartitionBytes parameter to control how large each file partition is.</li></ul><h3>8. Avoid Over-Partitioning</h3><ul><li>Make sure you’re not creating too many partitions when writing output, as this could lead to small file issues. For example, avoid using a very high number of partitions in repartition() or coalesce() without considering the actual data size.</li></ul><p>By implementing a combination of these strategies, you can mitigate the performance issues caused by small files in Spark and improve overall job efficiency.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5907d0add9a6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Top 20 Hands-On Apache Kafka Interview Questions with Detailed Answers]]></title>
            <link>https://medium.com/@anands282/top-20-hands-on-apache-kafka-interview-questions-with-detailed-answers-999436feaf13?source=rss-9950044d7c81------2</link>
            <guid isPermaLink="false">https://medium.com/p/999436feaf13</guid>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[interview-tips]]></category>
            <category><![CDATA[kafka-connect]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Anand Satheesh]]></dc:creator>
            <pubDate>Fri, 27 Sep 2024 03:46:50 GMT</pubDate>
            <atom:updated>2024-09-27T03:46:50.186Z</atom:updated>
            <content:encoded><![CDATA[<p>Apache Kafka has become a critical tool in modern data architectures due to its ability to handle large-scale, real-time data streams with efficiency and reliability. Originally developed by LinkedIn, Kafka is now widely adopted across industries for its robust messaging, streaming, and event-driven capabilities. It allows businesses to process and analyze real-time data from multiple sources, enabling faster decision-making and improved user experiences.</p><p>Kafka’s importance lies in its unique combination of scalability, fault-tolerance, and high throughput. Whether you’re building event-driven microservices, streaming data pipelines, or real-time analytics platforms, Kafka provides a foundation for managing data flow at enterprise scale. Its distributed architecture ensures that systems can grow seamlessly while maintaining low latency and high availability, making it a go-to solution for companies dealing with massive data streams and the need for reliable, real-time processing.</p><p>Here are top 20 interview questions for testing your knowledge on Apache kafka, these are for an intermediate level experienced folks. These questions are aimed at testing hands-on knowledge of Kafka, from setting up clusters to building producers, consumers, and stream processing applications.</p><h3>1. Kafka Architecture</h3><ul><li>Explain Kafka’s architecture, including producers, consumers, brokers, topics, partitions, and the Zookeeper’s role.</li></ul><p><strong>Answer: </strong>Kafka consists of the following key components:</p><ul><li><strong>Producers</strong>: Send records (messages) to Kafka topics.</li><li><strong>Consumers</strong>: Subscribe to topics and consume records.</li><li><strong>Brokers</strong>: Kafka servers that store topics. Each broker handles partitions of a topic.</li><li><strong>Topics</strong>: Logical categories where records are stored.</li><li><strong>Partitions</strong>: Kafka splits topics into partitions for scalability. Each partition is an ordered sequence of records.</li><li><strong>Zookeeper</strong>: Coordinates Kafka brokers by managing metadata (leader election, cluster configuration, etc.). Kafka uses Zookeeper to manage brokers and leader/follower roles for partitions.</li></ul><h3>2. Partitioning in Kafka</h3><ul><li>How does Kafka distribute messages across partitions? Can you demonstrate custom partitioning logic?</li></ul><p><strong>Answer</strong>: Kafka distributes records across partitions using the <strong>partitioning strategy</strong>. By default, if a partition key is not provided, Kafka uses a <strong>round-robin</strong> approach. However, if a partition key is provided, Kafka hashes the key to determine which partition to send the message to. You can implement <strong>custom partitioning</strong> by overriding the partitioner class and setting the partitioner.class configuration.</p><pre>props.put(&quot;partitioner.class&quot;, &quot;com.custom.kafka.CustomPartitioner&quot;);</pre><h3>3. Kafka Topic Creation</h3><ul><li>How do you create a Kafka topic? What is the purpose of replication factor and partitioning?</li></ul><p><strong>Answer: </strong>You can create Kafka topics manually using the kafka-topics.sh script:</p><pre>kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 2</pre><ul><li><strong>Replication Factor</strong>: The number of copies of each partition. A replication factor of 2 means that each partition is replicated to one other broker for redundancy.</li><li><strong>Partitions</strong>: Defines the number of sub-divisions for a topic to allow parallelism.</li></ul><h3>4. Producer API in Kafka</h3><ul><li>Demonstrate how to configure and use a Kafka producer. What are key settings to optimize for throughput and reliability?</li></ul><p><strong>Answer:</strong> Kafka producers send records to topics. Key configurations include:</p><ul><li>acks=all: Ensures all brokers acknowledge the write, providing stronger guarantees but lower throughput.</li><li>compression.type: Enables compression (e.g., gzip, snappy) to optimize storage and throughput.</li></ul><pre>Properties props = new Properties();<br>props.put(&quot;bootstrap.servers&quot;, &quot;localhost:9092&quot;);<br>props.put(&quot;acks&quot;, &quot;all&quot;);<br>props.put(&quot;key.serializer&quot;, &quot;org.apache.kafka.common.serialization.StringSerializer&quot;);<br>props.put(&quot;value.serializer&quot;, &quot;org.apache.kafka.common.serialization.StringSerializer&quot;);<br><br>Producer&lt;String, String&gt; producer = new KafkaProducer&lt;&gt;(props);<br>producer.send(new ProducerRecord&lt;&gt;(&quot;my-topic&quot;, &quot;key&quot;, &quot;value&quot;));</pre><h3>5. Consumer API in Kafka</h3><ul><li>How do you configure and use a Kafka consumer? How do you handle automatic vs manual offset commits?</li></ul><p><strong>Answer</strong>: Kafka consumers read records from topics. Key settings include:</p><ul><li>enable.auto.commit=false: Disables automatic offset committing for better control.</li><li>auto.offset.reset=earliest: Reads messages from the beginning of the topic if no offset is available.</li></ul><pre>Properties props = new Properties();<br>props.put(&quot;bootstrap.servers&quot;, &quot;localhost:9092&quot;);<br>props.put(&quot;group.id&quot;, &quot;test-group&quot;);<br>props.put(&quot;enable.auto.commit&quot;, &quot;false&quot;);<br>props.put(&quot;key.deserializer&quot;, &quot;org.apache.kafka.common.serialization.StringDeserializer&quot;);<br>props.put(&quot;value.deserializer&quot;, &quot;org.apache.kafka.common.serialization.StringDeserializer&quot;);<br><br>KafkaConsumer&lt;String, String&gt; consumer = new KafkaConsumer&lt;&gt;(props);<br>consumer.subscribe(Collections.singletonList(&quot;my-topic&quot;));<br>while (true) {<br>    ConsumerRecords&lt;String, String&gt; records = consumer.poll(Duration.ofMillis(100));<br>    for (ConsumerRecord&lt;String, String&gt; record : records) {<br>        System.out.printf(&quot;offset = %d, key = %s, value = %s%n&quot;, record.offset(), record.key(), record.value());<br>    }<br>    consumer.commitSync();<br>}</pre><h3>6. Consumer Group Mechanism</h3><ul><li>Explain Kafka consumer groups. How does Kafka ensure message distribution among multiple consumers in a group?</li></ul><p><strong>Answer</strong>: Kafka consumer groups enable multiple consumers to read from the same topic in parallel. Each partition is assigned to only one consumer in the group. If you have more partitions than consumers, consumers may handle multiple partitions.</p><p><strong>Example</strong>: Two consumers in the same group will read different partitions of a topic. Kafka ensures that messages are delivered to only one consumer per group.</p><h3>7. Handling Offsets</h3><ul><li>How do you manually commit offsets in Kafka? When would you choose manual offset management over automatic?</li></ul><p><strong>Answer</strong>: Offsets in Kafka track the last read message for a consumer group. You can manage offsets automatically or manually:</p><ul><li><strong>Manual Offset Commit</strong>: Use consumer.commitSync() after processing a batch of records to manually control when offsets are committed, allowing finer control over failures.</li></ul><p><strong>Scenario</strong>: In case of failure, you can rewind to a specific offset and reprocess messages.</p><h3>8. Kafka Connect</h3><ul><li>Describe the role of Kafka Connect. How do you set up a Kafka Connect cluster and integrate it with an external system (e.g., database or file system)?</li></ul><p><strong>Answer</strong>: Kafka Connect is used to stream data between Kafka and external systems (databases, file systems, etc.). It supports both <strong>source</strong> (pushing data to Kafka) and <strong>sink</strong> (pulling data from Kafka) connectors.</p><ul><li><strong>Example</strong>: Configuring a source connector to pull data from a MySQL database into Kafka.</li></ul><p><strong>Step</strong>: Deploy a JDBC Source Connector:</p><pre>{<br>  &quot;name&quot;: &quot;mysql-source-connector&quot;,<br>  &quot;config&quot;: {<br>    &quot;connector.class&quot;: &quot;io.confluent.connect.jdbc.JdbcSourceConnector&quot;,<br>    &quot;tasks.max&quot;: &quot;1&quot;,<br>    &quot;connection.url&quot;: &quot;jdbc:mysql://localhost:3306/mydb&quot;,<br>    &quot;topic.prefix&quot;: &quot;mysql-&quot;,<br>    &quot;table.whitelist&quot;: &quot;users&quot;<br>  }<br>}</pre><h3>9. Schema Registry and Avro</h3><ul><li>What is the role of the Kafka Schema Registry? How do you serialize and deserialize Avro messages in Kafka?</li></ul><p><strong>Answer</strong>: The Schema Registry provides a centralized location to manage Avro schemas for Kafka messages. Producers serialize messages using Avro, and consumers deserialize them with the same schema.</p><p><strong>Example</strong>: Using Avro with Kafka:</p><pre>props.put(&quot;key.serializer&quot;, &quot;io.confluent.kafka.serializers.KafkaAvroSerializer&quot;);<br>props.put(&quot;value.serializer&quot;, &quot;io.confluent.kafka.serializers.KafkaAvroSerializer&quot;);<br>props.put(&quot;schema.registry.url&quot;, &quot;http://localhost:8081&quot;);</pre><h3>10. Kafka Streams</h3><ul><li>How do you implement a basic Kafka Streams application? How do you handle stateful transformations in Kafka Streams?</li></ul><p><strong>Answer</strong>: Kafka Streams is a stream processing library for real-time analytics on Kafka data. It allows you to process records in real-time.</p><p><strong>Example</strong>: A word count application using Kafka Streams:</p><pre>KStream&lt;String, String&gt; textLines = builder.stream(&quot;input-topic&quot;);<br>KTable&lt;String, Long&gt; wordCounts = textLines<br>    .flatMapValues(text -&gt; Arrays.asList(text.toLowerCase().split(&quot;\\W+&quot;)))<br>    .groupBy((key, word) -&gt; word)<br>    .count();<br>wordCounts.toStream().to(&quot;output-topic&quot;, Produced.with(Serdes.String(), Serdes.Long()));</pre><h3>11. Stateless vs Stateful Processing in Kafka</h3><ul><li>Provide examples of stateless and stateful operations in Kafka Streams.</li></ul><p><strong>Answer</strong>:</p><ul><li><strong>Stateless</strong>: Operations like map(), filter(), or flatMap() process each record independently.</li><li><strong>Stateful</strong>: Operations like aggregate(), join(), or count() maintain state (e.g., counting words over time).</li></ul><p><strong>Example</strong>: Counting words in a stream is a stateful operation since it requires maintaining a running count.</p><h3>12. Exactly Once Semantics (EOS)</h3><ul><li>How does Kafka achieve exactly-once semantics (EOS)? Can you configure a producer and consumer to support EOS?</li></ul><p><strong>Answer</strong>: Kafka provides <strong>Exactly Once Semantics (EOS)</strong> by enabling <strong>idempotent producers</strong> and <strong>transactional consumers</strong>. To enable EOS, configure:</p><ul><li><strong>Producer</strong>: enable.idempotence=true</li><li><strong>Consumer</strong>: Use transactional API for atomic operations across partitions.</li></ul><p><strong>Example</strong>:</p><pre>props.put(&quot;enable.idempotence&quot;, &quot;true&quot;);</pre><h3>13. Kafka Batch Processing</h3><ul><li>How can Kafka be used to handle batch processing? Compare its batch-processing capabilities with its real-time processing model.</li></ul><p><strong>Answer</strong>: Kafka can handle batch processing by buffering records in memory before producing or consuming. Use the linger.ms and batch.size configurations for producers to control the batching of records.</p><h3>14. Compaction in Kafka Topics</h3><ul><li>Explain log compaction in Kafka. How would you enable log compaction for a specific topic?</li></ul><p><strong>Answer</strong>: <strong>Log compaction</strong> in Kafka ensures that only the latest value for each key is retained. This is useful for change data capture (CDC) use cases.</p><p><strong>Enabling Compaction</strong>:</p><pre>kafka-topics.sh --alter --topic my-topic --config cleanup.policy=compact</pre><h3>15. Fault Tolerance and Replication</h3><ul><li>How does Kafka ensure fault tolerance? What happens if a broker fails, and how does Kafka maintain availability?</li></ul><p><strong>Answer</strong>: Kafka ensures fault tolerance by replicating partitions across brokers. If a broker fails, a replica becomes the new leader, and consumers/producers will automatically redirect traffic to the new leader. Data is lost only if all replicas fail.</p><h3>16. Broker Configurations</h3><ul><li>How do you configure a Kafka broker for high throughput? Discuss key configurations like log.retention.ms, log.segment.bytes, etc.</li></ul><p><strong>Answer</strong>: Important broker configurations include:</p><ul><li><strong>log.retention.ms</strong>: Defines how long Kafka retains messages before deletion.</li><li><strong>log.segment.bytes</strong>: Controls the maximum size of a log segment file. Segmenting helps Kafka manage disk space.</li></ul><h3>17. Kafka Monitoring</h3><ul><li>What are the key metrics to monitor in a Kafka cluster? How would you set up monitoring using tools like Prometheus or Grafana?</li></ul><p><strong>Answer</strong>:<br>To ensure the smooth operation of a Kafka cluster, monitoring key metrics is essential. Some of the most critical metrics to track include:</p><ul><li><strong>Consumer Lag</strong>: Indicates the difference between the current offset of the consumer and the last available offset on the topic. High consumer lag can signal that consumers are falling behind.</li></ul><p>Metric: kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec</p><ul><li><strong>Message Throughput</strong>: The number of messages consumed or produced per second.</li></ul><p>Metrics: BytesInPerSec, BytesOutPerSec</p><ul><li><strong>Broker Health</strong>: Monitor broker availability and key metrics like disk space and memory usage. Look for under-replicated partitions and offline partitions, which indicate issues with replication and fault tolerance.</li></ul><p>Metric: kafka.cluster:type=Partition,name=UnderReplicatedPartitions</p><ul><li><strong>Request Latency</strong>: Measures how long it takes for requests (e.g., producing or consuming messages) to be processed. High latency can indicate problems in the Kafka cluster or with its consumers.</li></ul><p>Metric: kafka.network:type=RequestMetrics,name=TotalTimeMs</p><ul><li><strong>Leader Election Rate</strong>: Frequent leader elections could indicate issues in the cluster, such as broker instability.</li></ul><p>Metric: kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs</p><p><strong>Setting up Monitoring with Prometheus and Grafana</strong>:</p><ol><li><strong>Install JMX Exporter</strong>: Kafka exposes metrics via JMX (Java Management Extensions). To integrate with Prometheus, use a <strong>JMX Exporter</strong> to expose JMX metrics on an HTTP endpoint.</li><li>Add the JMX Exporter to your Kafka broker:</li></ol><pre>KAFKA_OPTS=&quot;$KAFKA_OPTS -javaagent:/path/to/jmx_prometheus_javaagent.jar=7071:/path/to/kafka.yml&quot;</pre><p><strong>Prometheus Configuration</strong>: Configure Prometheus to scrape the JMX Exporter metrics.</p><pre>scrape_configs:<br>  - job_name: &#39;kafka&#39;<br>    static_configs:<br>      - targets: [&#39;localhost:7071&#39;]</pre><p><strong>Grafana</strong>: Use Prometheus as a data source in Grafana and visualize Kafka metrics with pre-built Kafka dashboards available in Grafana’s plugin library.</p><h3>18. Kafka Security</h3><ul><li>How do you secure a Kafka cluster? Describe the process for enabling SSL encryption and setting up access control with SASL.</li></ul><p><strong>Answer</strong>:<br>Securing a Kafka cluster is crucial for ensuring data privacy, integrity, and proper access control. Kafka offers several mechanisms for securing a cluster, including SSL for encryption and SASL for authentication.</p><p><strong>Steps to Enable SSL Encryption</strong>:</p><ol><li><strong>Generate SSL Certificates</strong>: Create SSL certificates for each Kafka broker using keytool.</li></ol><pre>keytool -genkey -alias kafka-server -keyalg RSA -keystore kafka.server.keystore.jks -validity 365<br>keytool -certreq -keystore kafka.server.keystore.jks -alias kafka-server -file cert-file</pre><p>2. <strong>Configure Brokers</strong>: Update Kafka broker configurations to enable SSL.</p><pre>listeners=SSL://:9093<br>ssl.keystore.location=/path/to/kafka.server.keystore.jks<br>ssl.keystore.password=yourpassword<br>ssl.key.password=yourkeypassword<br>ssl.truststore.location=/path/to/kafka.server.truststore.jks<br>ssl.truststore.password=yourpassword</pre><p><strong>3. Configure Producers/Consumers</strong>: Configure producers and consumers to use SSL by providing keystore and truststore settings in the client properties.</p><p><strong>Steps to Enable SASL for Authentication</strong>:</p><ol><li><strong>Configure SASL in Brokers</strong>:</li></ol><pre>listeners=SASL_SSL://:9093<br>sasl.mechanism=PLAIN<br>sasl.enabled.mechanisms=PLAIN<br>security.inter.broker.protocol=SASL_SSL</pre><p>2. <strong>Set up JAAS for Authentication</strong>: Configure kafka_server_jaas.conf with SASL credentials:</p><pre>KafkaServer {<br>    org.apache.kafka.common.security.plain.PlainLoginModule required<br>    username=&quot;admin&quot;<br>    password=&quot;admin-secret&quot;;<br>};</pre><p>3. <strong>Client-Side Configuration</strong>: Producers and consumers need to authenticate using the sasl.jaas.config property.</p><pre>security.protocol=SASL_SSL<br>sasl.mechanism=PLAIN<br>sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=&quot;client&quot; password=&quot;client-secret&quot;;</pre><h3>19. Kafka Rebalancing</h3><ul><li>What is rebalancing in Kafka? How can you minimize its impact on performance?</li></ul><p><strong>Answer</strong>:<br><strong>Rebalancing</strong> occurs when the set of active consumers in a consumer group changes, such as when a consumer joins or leaves the group. Kafka reassigns partitions to consumers during a rebalance. This can lead to temporary downtime, as consumers pause to redistribute partitions.</p><p><strong>Minimizing the Impact of Rebalancing</strong>:</p><ol><li><strong>Use Cooperative Sticky Assignor</strong>: By default, Kafka uses the <strong>RangeAssignor</strong>, but you can switch to the <strong>Cooperative Sticky Assignor</strong>, which minimizes disruption by only moving partitions that need reassignment.</li></ol><pre>partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor</pre><p>2. <strong>Increase </strong><strong>session.timeout.ms and </strong><strong>heartbeat.interval.ms</strong>: Tune these parameters to reduce the frequency of rebalances. Increasing the session.timeout.ms and adjusting the heartbeat.interval.ms can help avoid unnecessary rebalances caused by short connection losses.</p><p><strong>3. Optimize </strong><strong>max.poll.interval.ms</strong>: Ensure that your consumers process records within the configured max.poll.interval.ms. If the processing takes longer, Kafka will assume the consumer is down, triggering a rebalance.</p><h3>20. Handling Message Reprocessing</h3><ul><li>If a Kafka consumer fails to process a message, how can you reprocess it? What patterns or strategies can be used to ensure data integrity?</li></ul><p><strong>Answer</strong>:<br>If a Kafka consumer fails to process a message, there are several strategies you can use to ensure that messages are reprocessed while maintaining data integrity:</p><ol><li><strong>Manual Offset Management</strong>:<br>Disable automatic offset commits (enable.auto.commit=false) and manually commit offsets only after successfully processing a message.</li></ol><pre>consumer.commitSync();  // After message is processed successfully</pre><p><strong>2. Seek to Offset</strong>:<br>In case of a failure, you can <strong>seek</strong> to a specific offset and reprocess the messages from that point.</p><pre>consumer.seek(new TopicPartition(&quot;my-topic&quot;, partition), offset);</pre><p><strong>3. Dead Letter Queue (DLQ)</strong>:<br>If processing fails after a few retries, you can send the message to a <strong>Dead Letter Queue</strong> (another Kafka topic) for later analysis or reprocessing.</p><pre>ProducerRecord&lt;String, String&gt; deadLetter = new ProducerRecord&lt;&gt;(&quot;dead-letter-topic&quot;, key, value);<br>producer.send(deadLetter);</pre><p><strong>4. Idempotent Consumers</strong>:<br>If your consumer is idempotent (i.e., processing the same message multiple times results in the same output), you can simply re-read the message without worrying about duplicates.</p><p><strong>5. Retry Mechanism</strong>:<br>Implement a retry mechanism to process failed messages, either with a time delay or through a retry topic.</p><p>By combining these strategies, you can ensure that message processing is resilient to failures and that messages are either successfully processed or safely stored for later reprocessing.</p><p>In conclusion, mastering Apache Kafka requires a deep understanding of its core components, as well as hands-on experience with its real-world applications. From configuring producers and consumers to managing offsets, security, and rebalancing, Kafka offers powerful capabilities for building scalable, fault-tolerant, and real-time data pipelines. By keeping an eye on key metrics and ensuring secure configurations, you can optimize Kafka for high performance and reliability. Whether you’re setting up monitoring systems like Prometheus and Grafana, or handling complex scenarios such as message reprocessing and partition rebalancing, having a practical grasp of these Kafka fundamentals will greatly enhance your ability to design and manage efficient data streaming systems. Understanding these interview questions and their detailed answers can provide a solid foundation for anyone looking to excel in a Kafka-related role.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=999436feaf13" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deriving value from LLMs using RAG]]></title>
            <link>https://medium.com/@anands282/deriving-value-from-llms-using-rag-15fd8b26a8d5?source=rss-9950044d7c81------2</link>
            <guid isPermaLink="false">https://medium.com/p/15fd8b26a8d5</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Anand Satheesh]]></dc:creator>
            <pubDate>Wed, 17 Jul 2024 05:50:59 GMT</pubDate>
            <atom:updated>2024-07-18T07:14:38.424Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Large Language Models (LLMs)</strong> are advanced artificial intelligence models designed to understand, generate, and manipulate human language. These models are trained on vast amounts of text data, enabling them to perform a wide range of language-related tasks. <strong>Retrieval-Augmented Generation (RAG)</strong> is a technique that enhances the capabilities of LLMs by combining retrieval-based and generation-based approaches.</p><p>Here’s how it works:</p><p><strong>Concept</strong>:</p><ul><li>RAG systems first retrieve relevant information from a large corpus or database based on the input query.</li><li>This retrieved information is then used to augment the input, providing the language model with additional context and details.</li></ul><p><strong>Components</strong>:</p><ul><li><strong>Retriever</strong>: A component that searches a vast dataset to find relevant documents or pieces of information. This can be based on traditional information retrieval techniques or modern neural search methods.</li><li><strong>Generator</strong>: An LLM that uses the augmented input (original query plus retrieved information) to generate a coherent and contextually appropriate response.</li></ul><p><strong>Advantages</strong>:</p><ul><li><strong>Improved Accuracy</strong>: By providing relevant context, RAG systems can produce more accurate and informative responses.</li><li><strong>Dynamic Knowledge Access</strong>: They can access up-to-date information that might not be encoded in the model’s parameters, making them suitable for real-time applications.</li><li><strong>Domain-Specific Knowledge</strong>: RAG systems can be tailored to specific domains by using specialized corpora, improving their performance in niche areas.</li></ul><p><strong>Challenges</strong>:</p><ul><li>Ensuring the quality and relevance of retrieved information is critical to the performance of RAG systems.</li><li>Managing and updating the retrieval corpus to keep the information current and accurate is a continuous task.</li></ul><p><strong>Applications</strong>:</p><p>Now here is when RAG and LLM combination gets most interesting. Say you are looking to create a support channel for your application and will have a support team dedicated 24* 7, answering the queries and concerns of the users. Now here RAG and LLM combination can help cut some cost.</p><p>A support channel chat/email dump thats been answering the user queries for a few months, can act as a possible data source for this scenario.</p><p>Also Jira issues,Service now or BMC remedy tickets dump and comments, in addition to any user education documents and FAQ documents that was being used by the support team can also be used as a data set for RAG.</p><p>An AI assistant can then use these datasets to supplement its reponses, custom tailoring answers by finding solutions based on previous responses from support team personal. The advantages here is that the typical queries that can have standard responses will get addressed automatically by the AI assistant. Only the tougher questions that requires a personal touch would need to be looked into by an actual support team member. The number of staff required for support can be reduced and also night shifts and weekend shifts can be handed over completely to the bot with only priority user tickets being addressed by a support team on demand.</p><p>Thanks for reading, will be adding a full fledged project with LLM and RAG systems soon on my feed. Happy coding!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=15fd8b26a8d5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Spark Commonly seen errors in production and their solutions.]]></title>
            <link>https://medium.com/@anands282/apache-spark-commonly-seen-errors-in-production-and-their-solutions-ccdbfbc3d4a3?source=rss-9950044d7c81------2</link>
            <guid isPermaLink="false">https://medium.com/p/ccdbfbc3d4a3</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[best-practices]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[production]]></category>
            <category><![CDATA[error-handling]]></category>
            <dc:creator><![CDATA[Anand Satheesh]]></dc:creator>
            <pubDate>Mon, 01 Jul 2024 06:40:44 GMT</pubDate>
            <atom:updated>2024-07-01T06:40:44.752Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*WZTh9CBgTyPAMDpW1UFoTQ.png" /></figure><p>Apache Spark is a powerful tool for big data processing, it uses distributed data processing in memory to reduce the execution time drastically for big data workloads. But like any technology, it can present challenges, especially in production environments. Along with knowing the fundamentals of Apache Spark, knowing the commonly encountered errors are important for faster issue resolution. This is also an important section in interviews for lateral hires. I always ask the candidates about the errors they have faced in with apache spark in production and dig deep on explaining the solutions for a couple of them. This will give the interviewer an idea on how good we are at understanding the behind the scenes working and how we solve issues in production.Here are some of the most common errors and issues encountered in Apache Spark, along with potential solutions:</p><h3>1. OutOfMemoryError</h3><p>The most common and infamous error with apache spark is the out of memory error. This error occurs when Spark executors or drivers run out of memory during processing. The error would look similar to the one shown below</p><pre>ERROR Executor: Exception in task 7.0 in stage 6.0 (TID 439)<br><br>java.lang.OutOfMemoryError<br>    at java.io.ByteArrayOutputStream.hugeCapacity(Unknown Source)<br>    at java.io.ByteArrayOutputStream.grow(Unknown Source)<br>    at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source)<br>    at java.io.ByteArrayOutputStream.write(Unknown Source)<br>    at java.io.ObjectOutputStream$BlockDataOutputStream.drain(Unknown Source)<br>    at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(Unknown Source)<br>    at java.io.ObjectOutputStream.writeObject0(Unknown Source)<br>    at java.io.ObjectOutputStream.writeObject(Unknown Source)<br>    at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:44)<br>    at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101)<br>    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:239)<br>    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)<br>    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)<br>    at java.lang.Thread.run(Unknown Source)</pre><p><strong>Root Cause:</strong></p><ul><li>The production environment might contain way more volume of data than the staging and qa(preprod) environments.</li><li>A huge increase in daily load due to an upstream error.</li></ul><p><strong>Solution:</strong></p><ul><li><strong>Optimize memory usage:</strong> Increase the executor memory and driver memory settings.</li></ul><pre>spark-submit --executor-memory 4G --driver-memory 2G ...</pre><ul><li><strong>Use DataFrame API:</strong> Switch from RDDs to DataFrames or Datasets to take advantage of Spark’s Catalyst optimizer.</li><li><strong>Persist Data Efficiently:</strong> Use appropriate storage levels when persisting data, such as MEMORY_AND_DISK instead of just MEMORY.</li><li><strong>Use Dynamic Resource allocation: </strong>This feature of spark and enable the executors to scale up or down in accordance with the work load.</li><li><strong>Configure Spark parameters correctly:</strong> Ideally its preffered to have a preprod environment to mimic the production closely in terms of volume of data so that such issues are caught before it reaches prod.</li></ul><h3>2. Spark Shuffle Errors</h3><p>Data shuffling in Apache Spark refers to the redistribution of data across different nodes in the cluster, which can be a costly operation in terms of time and resources. Shuffling is required for certain types of operations that need to rearrange data to complete their tasks. Depending on the specific configuration used, you may get a shuffle FetchFailedException error like the ones shown below:</p><pre><br>org.apache.spark.shuffle.FetchFailedException: Failed to connect to host:port<br><br>ShuffleMapStage has failed the maximum allowable number of times</pre><p><strong>Root Cause:</strong></p><ul><li>Spark shuffle operations (e.g., joins, groupBy) can cause high disk and network I/O, leading to performance degradation or errors.</li><li>Cluster downscaling events, executor loss, or worker decommission. In certain cases, shuffle files from the executor are lost during job run.</li><li>External shuffle service might be down on the worker nodes</li></ul><p><strong>Solution:</strong></p><ul><li><strong>Increase shuffle partitions:</strong> Set the spark.sql.shuffle.partitions parameter to a higher value. This can fix issues caused due to large size partitions failing to get written or read from executors.</li><li><strong>Optimize joins:</strong> Use broadcast joins for smaller datasets by leveraging the broadcast function.</li></ul><pre>spark.conf.set(&quot;spark.sql.shuffle.partitions&quot;, &quot;500&quot;)<br>import org.apache.spark.sql.functions.broadcast<br>val joinedDF = largeDF.join(broadcast(smallDF), &quot;id&quot;)</pre><ul><li><strong>Validate the external shuffle service configurations: </strong>The external shuffle service is a separate daemon process that manages shuffle data for Spark executors. When spark.shuffle.service.enabled is set to true, the external shuffle service takes over the responsibility of serving shuffle files, allowing executors to be safely removed or restarted without losing shuffle data.</li></ul><h3>3. Executor Lost</h3><p>Since apache spark promises to run effectively on commodity hardware, failures and network issues are anticipated and accounted for in the design of the distributed processing system. But occasinally one or more Executors may be lost midway of execution of a job causing the below error in production systems:</p><pre>Lost executor 1 on host: Executor heartbeat timed out after 128083 ms</pre><p><strong>Root Cause:</strong> Executors may be lost due to issues like hardware failures, network problems, or container eviction in a YARN/Kubernetes cluster.</p><p><strong>Solution:</strong></p><ul><li><strong>Increase retry attempts:</strong> Set spark.task.maxFailures to a higher value to allow more retries. Incases where the cluster is facing intermitent issues with network or resouces, more retries provides us better chances for workload completion.</li></ul><pre>spark.conf.set(&quot;spark.task.maxFailures&quot;, &quot;4&quot;)</pre><ul><li><strong>Check cluster stability:</strong> Ensure that the cluster resources are stable and not oversubscribed.</li></ul><h3>4. Skewed Data</h3><p>Skewed data might lead to errors in production much like the one shown below:</p><pre>Stage X contains a task with very large runtime: 1024.0s</pre><p><strong>Root Cause:</strong> Data skew occurs when some partitions have significantly more data than others, leading to unbalanced workloads.</p><p><strong>Solution:</strong></p><ul><li><strong>Salting technique:</strong> Add a random key to distribute skewed keys more evenly.</li></ul><pre>import org.apache.spark.sql.functions._<br>val saltedDF = df.withColumn(&quot;salt&quot;, monotonically_increasing_id() % 10)<br>val joinedDF = saltedDF.join(anotherDF, Seq(&quot;id&quot;, &quot;salt&quot;))</pre><ul><li><strong>Custom partitioning:</strong> Implement custom partitioning to balance the data distribution.</li><li><strong>Understand the data and Variations: </strong>Before running datapipelines in production, try testing the pipelines on preprod systems which closely mimic the production system in quality of data to encounter the skews in data before it reaches production.</li></ul><h3>5. Serialization Errors</h3><p>Serialization errors in Apache Spark occur when Spark cannot serialize user-defined classes or objects, which is essential for sending data across the network between executors. Here are some common scenarios and examples of serialization errors, along with solutions.</p><pre>org.apache.spark.SparkException: Task not serializable</pre><p><strong>Root Cause:</strong></p><ul><li><strong>Non-Serializable Classes:</strong>You have a custom class that is not serializable.</li><li><strong>Anonymous Functions and Inner Classes:</strong>Using an anonymous function or inner class that captures non-serializable outer variables.</li><li><strong>Broadcast Variables: </strong>Broadcast variables used within transformations are not serializable.</li><li><strong>Using </strong><strong>transient Keyword Incorrectly: </strong>Using the transient keyword on non-serializable fields that are later used in transformations.</li></ul><p><strong>Solution:</strong></p><ul><li><strong>Use </strong><strong>Serializable trait:</strong> Ensure all custom classes and objects implement Serializable.</li><li><strong>Check closure:</strong> Verify that all variables used inside transformations are serializable.</li><li><strong>Use </strong><strong>@transient carefully:</strong> Avoid marking fields as transient if they are needed within transformations.</li><li><strong>Test serializability:</strong> Test your classes for serializability outside Spark transformations using SerializationUtils</li></ul><pre>import java.io.{ObjectOutputStream, ByteArrayOutputStream}<br><br>def isSerializable(obj: Any): Boolean = {<br>  try {<br>    val byteArrayOutputStream = new ByteArrayOutputStream()<br>    val objectOutputStream = new ObjectOutputStream(byteArrayOutputStream)<br>    objectOutputStream.writeObject(obj)<br>    objectOutputStream.close()<br>    true<br>  } catch {<br>    case e: Exception =&gt; false<br>  }<br>}<br><br>println(isSerializable(new CustomClass(&quot;example&quot;)))  // Should print true if serializable</pre><h3>6. Long Garbage Collection (GC) Times</h3><p><strong>Root Cause:</strong> High GC times can occur due to inefficient memory usage, leading to performance degradation.</p><pre>WARN TaskSetManager: Lost task X.X in stage Y.Y (TID Z, host, executor Y): ExecutorLostFailure (executor Y exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 128083 ms</pre><p><strong>Solution:</strong> Tune JVM parameters to optimize GC. GC tuning flags for executors can be specified by setting spark.executor.defaultJavaOptions or spark.executor.extraJavaOptions in a job’s configuration. More info on GC tuning can be found at <a href="https://spark.apache.org/docs/latest/tuning.html#garbage-collection-tuning">https://spark.apache.org/docs/latest/tuning.html#garbage-collection-tuning</a></p><pre>spark-submit --conf &quot;spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35&quot; ...</pre><h3>7. Incorrect Results Due to DataFrame API Misuse</h3><p><strong>Root Cause:</strong> Misuse of the DataFrame API can lead to logical errors and incorrect results.</p><pre>WARN QueryExecution: org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to xxx() after end()</pre><p><strong>Solution:</strong></p><ul><li><strong>Understand lazy evaluation:</strong> Remember that Spark transformations are lazy, and actions trigger the execution.</li><li><strong>Debug transformations:</strong> Use explain and debug methods to inspect the execution plan</li></ul><pre>df.explain(true)</pre><h3>8. Slow Job Execution</h3><p><strong>Root Cause:</strong> Poorly optimized Spark jobs can result in slow execution times.</p><pre>WARN TaskSetManager: Stage Y contains a task with very large runtime: 1024.0s</pre><p><strong>Solution:</strong></p><ul><li><strong>Optimize queries:</strong> Rewrite queries to reduce the complexity and number of stages.</li><li><strong>Use caching:</strong> Cache intermediate results to avoid recomputation.</li></ul><pre>val cachedDF = df.cache()</pre><h3>Conclusion</h3><p>By understanding and addressing these common errors, you can ensure smoother operations and better performance in your Apache Spark production environment. Proper tuning, optimization, and debugging techniques are essential for handling large-scale data processing tasks effectively.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ccdbfbc3d4a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark’s Dynamic Resource Allocation, does it really help?]]></title>
            <link>https://medium.com/@anands282/sparks-dynamic-resource-allocation-does-it-really-help-1a81fb085bc9?source=rss-9950044d7c81------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a81fb085bc9</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[Anand Satheesh]]></dc:creator>
            <pubDate>Sun, 30 Jun 2024 17:22:28 GMT</pubDate>
            <atom:updated>2024-06-30T17:22:28.965Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*KqX1VK0HSr4JN_t4lHR1WA.png" /></figure><p>Apache Spark, a powerful distributed computing framework, excels in processing large-scale data workloads efficiently. One of its key features, dynamic resource allocation (DRA), plays a crucial role in optimizing resource utilization within Spark clusters. DRA enables Spark applications to scale resources up or down based on workload demands dynamically, enhancing both performance and cost-efficiency.</p><p>In this article, we delve into the fundamentals of dynamic resource allocation in Apache Spark. We explore how DRA works under the hood, its benefits in managing cluster resources effectively, and practical considerations for configuring and leveraging this feature. Whether you’re new to Spark or looking to optimize resource usage in your Spark deployments, understanding DRA is essential for achieving peak performance and scalability.</p><p>Lets see whats was the limitation with the traditional(Static) resource allocation method which lead to creation of dynamic resource allocation. Usually in the earlier(less than 2.4)versions of spark the resources like your driver memory, executor number or memory etc are hardcoded in the application code itself, or in the spark submit script or added in the spark config files directly. Now what this does is that it puts an upper cap on the amount of resouces with that hardcoded value and in cases where less resources are needed leads to wastage of resources and incases where there is a surge in load it can cause breaking of the pipeline due to low memory or insufficient processing capacity.</p><p>First lets see how to enable DRA in any of the data pipelines, by default this feature is turned off in spark. Simply add these config properites to your spark session builder line to enable DRA.</p><pre>spark = SparkSession.builder \<br>    .appName(&quot;DynamicAllocationExample&quot;) \<br>    .config(&quot;spark.dynamicAllocation.enabled&quot;, &quot;true&quot;) \<br>    .config(&quot;spark.shuffle.service.enabled&quot;, &quot;true&quot;) \<br>    .config(&quot;spark.dynamicAllocation.minExecutors&quot;, &quot;1&quot;) \<br>    .config(&quot;spark.dynamicAllocation.maxExecutors&quot;, &quot;10&quot;) \<br>    .getOrCreate()</pre><p>These are the minimum configurations that are required to enable DRA and as you can see we are setting a minimum and maximum number of executors that the application can spin up depending on the load requirement. DRA offers the following benefits:</p><ul><li><strong>Resource Efficiency:</strong> Allocates resources based on workload needs, reducing waste.</li><li><strong>Cost Savings:</strong> Frees up resources when idle, lowering costs in cloud environments.</li><li><strong>Scalability:</strong> Adapts to varying workloads, improving performance during peak usage.</li><li><strong>Cluster Utilization:</strong> Maximizes the use of available cluster resources, improving overall throughput.</li></ul><p>Here are some real-world scenarios where Spark’s dynamic resource allocation is useful:</p><ol><li><strong>Data Processing Pipelines:</strong> Workloads with varying sizes, such as ETL jobs that change based on daily data volume.</li><li><strong>Ad-Hoc Queries:</strong> Interactive analysis where user query load fluctuates, allowing the system to scale resources as needed.</li><li><strong>Batch Jobs:</strong> Large batch processing tasks where resource demands peak at different stages of the pipeline.</li><li><strong>Streaming Applications:</strong> Streaming workloads with unpredictable input rates, dynamically adjusting resources based on incoming data volume.</li><li><strong>Shared Clusters:</strong> Multiple teams sharing a cluster, allowing resource allocation based on current demand, improving cluster utilization.</li></ol><p>While dynamic resource allocation (DRA) in Spark generally offers significant benefits in resource efficiency and performance optimization, there are scenarios where it might not provide substantial advantages or could potentially cause issues:</p><ol><li><strong>Small and Stable Workloads</strong>: For small, predictable workloads that consistently require a fixed number of resources, the overhead of dynamically allocating and deallocating resources may outweigh any potential benefits. Constantly scaling up and down executors can introduce unnecessary overhead and delay in job execution, potentially impacting overall performance.</li><li><strong>Frequent Executor Scaling</strong>: In environments with high-frequency job submissions or very short-lived tasks, the time taken to scale executors dynamically (both up and down) might be significant.This delay can lead to higher job latencies and reduced overall throughput, as the system spends more time managing executor changes rather than processing data.</li><li><strong>Job Dependencies on Fixed Executor Counts</strong>: Some applications or jobs may have dependencies on a fixed number of executors due to specific configurations, custom resource management, or external dependencies.Dynamic allocation might disrupt these dependencies and cause issues if executors are scaled differently than expected, leading to job failures or performance degradation.</li><li><strong>Resource Fragmentation</strong>: In dynamic environments with frequent scaling events, resource fragmentation can occur where small clusters of resources are left underutilized or idle due to uneven allocation and deallocation patterns. This fragmentation can reduce overall cluster efficiency and utilization rates, negating some of the benefits of dynamic allocation.</li><li><strong>Mixed Workloads and Priorities</strong>:When running multiple workloads with varying priorities and resource requirements on the same cluster, dynamic allocation may not effectively prioritize resources according to workload importance or criticality. Lower-priority jobs or tasks might not receive sufficient resources during peak demand periods if resources are dynamically allocated based on current workload demands rather than predefined priorities.</li><li><strong>Complex Cluster Environments</strong>:In clusters with complex configurations, integration with external services, or specific network/storage dependencies, dynamic allocation may introduce additional complexities or dependencies that are hard to manage. This complexity can lead to operational challenges, increased maintenance overhead, and potential compatibility issues with existing infrastructure.</li></ol><p>In these cases, careful consideration of workload characteristics, cluster environment, and operational requirements is crucial to determine whether dynamic resource allocation in Spark will provide measurable benefits or potential drawbacks. Conducting thorough testing and performance benchmarks specific to your use case can help assess the suitability and impact of dynamic allocation before deployment in production environments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a81fb085bc9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Test-Driven Development (TDD): Explanation, Advantages, and Examples]]></title>
            <link>https://medium.com/@anands282/test-driven-development-tdd-explanation-advantages-and-examples-687a2475ff7f?source=rss-9950044d7c81------2</link>
            <guid isPermaLink="false">https://medium.com/p/687a2475ff7f</guid>
            <category><![CDATA[tddwithpython]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[best-practices]]></category>
            <category><![CDATA[clean-code]]></category>
            <category><![CDATA[tdd]]></category>
            <dc:creator><![CDATA[Anand Satheesh]]></dc:creator>
            <pubDate>Mon, 24 Jun 2024 14:43:42 GMT</pubDate>
            <atom:updated>2024-06-24T14:43:42.948Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction to Test-Driven Development (TDD)</h3><p>Test-Driven Development (TDD) is a software development methodology in which tests are written before the code that needs to be tested. The process involves writing a test, ensuring that it fails (since the code isn’t written yet), then writing the minimal amount of code required to pass the test, and finally refactoring the code while keeping the tests green. This cycle is often referred to as “Red-Green-Refactor.”</p><h3>The TDD Cycle</h3><ol><li><strong>Red</strong>: Write a test that fails. This defines a function or improvement.</li><li><strong>Green</strong>: Write the simplest code to pass the test.</li><li><strong>Refactor</strong>: Improve the code without changing its behavior. Ensure tests still pass.</li></ol><h3>Advantages of TDD</h3><ol><li><strong>Improved Code Quality</strong>: Writing tests first ensures that the code meets the requirements and behaves as expected. This leads to fewer bugs and higher-quality code.</li><li><strong>Better Design</strong>: TDD encourages developers to think about the design and architecture of the code before implementation, leading to cleaner and more modular code.</li><li><strong>Faster Debugging</strong>: Since tests are written for every small piece of functionality, identifying and fixing bugs becomes easier and faster.</li><li><strong>Reduced Fear of Changes</strong>: With a comprehensive suite of tests, developers can refactor and improve code with confidence, knowing that any issues will be caught by the tests.</li><li><strong>Documentation</strong>: Tests serve as a form of documentation that explains what the code is supposed to do, making it easier for new developers to understand the codebase.</li></ol><h3>TDD Workflow Example in Python</h3><p>Let’s walk through an example of creating a simple calculator using TDD.</p><h4>Step 1: Red — Write a Failing Test</h4><p>First, we write a test for a function that adds two numbers.</p><pre>import unittest<br><br>class TestCalculator(unittest.TestCase):<br><br>    def test_add(self):<br>        from calculator import add<br>        self.assertEqual(add(1, 2), 3)<br><br>if __name__ == &#39;__main__&#39;:<br>    unittest.main()</pre><p>Running this test will fail because the add function does not exist yet.</p><h4>Step 2: Green — Write the Minimal Code to Pass the Test</h4><p>Next, we implement the add function in a file named calculator.py.</p><pre># calculator.py<br><br>def add(a, b):<br>    return a + b</pre><p>Now, if we run the test again, it should pass.</p><pre>$ python -m unittest test_calculator.py</pre><h4>Step 3: Refactor — Improve the Code</h4><p>In this simple example, there isn’t much to refactor, but the process of refactoring ensures the code is clean and efficient while all tests pass.</p><h4>Adding More Functionality with TDD</h4><p>Let’s add a subtract function using the same TDD cycle.</p><ol><li><strong>Red</strong>: Write a failing test.</li></ol><pre>class TestCalculator(unittest.TestCase):<br><br>    def test_add(self):<br>        from calculator import add<br>        self.assertEqual(add(1, 2), 3)<br><br>    def test_subtract(self):<br>        from calculator import subtract<br>        self.assertEqual(subtract(5, 3), 2)</pre><p><strong>2. Green</strong>: Implement the minimal code to pass the test.</p><pre># calculator.py<br><br>def add(a, b):<br>    return a + b<br><br>def subtract(a, b):<br>    return a - b</pre><p>3. <strong>Refactor</strong>: In this case, there’s no immediate refactoring needed, but we ensure all tests pass.</p><pre>$ python -m unittest test_calculator.py</pre><h3>Conclusion</h3><p>Test-Driven Development (TDD) is a powerful methodology that promotes writing tests before code, leading to higher quality, more maintainable, and less buggy software. By following the Red-Green-Refactor cycle, developers can ensure their code meets the requirements and behaves as expected from the outset. While TDD requires a disciplined approach and can have a learning curve, the benefits it brings in terms of code quality, design, and maintainability make it a valuable practice for any development team.</p><h3>Additional Considerations</h3><ol><li><strong>Integration with Continuous Integration (CI)</strong>: TDD works well with CI tools like Jenkins, Travis CI, or GitHub Actions, which can automatically run tests on every code change.</li><li><strong>Mocks and Stubs</strong>: When testing, especially with external dependencies, mocks and stubs can be used to simulate those dependencies.</li><li><strong>Edge Cases and Error Handling</strong>: TDD encourages thinking about edge cases and error handling from the beginning, leading to more robust code.</li></ol><p>By incorporating TDD into your development process, you can achieve more reliable and maintainable software, leading to better outcomes for both developers and users.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=687a2475ff7f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Database Dilemmas: Cracking the Code to Your Perfect Database Match]]></title>
            <link>https://medium.com/@anands282/database-dilemmas-cracking-the-code-to-your-perfect-database-match-c4c59f0339bf?source=rss-9950044d7c81------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4c59f0339bf</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[data-modeling]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[nosql]]></category>
            <dc:creator><![CDATA[Anand Satheesh]]></dc:creator>
            <pubDate>Mon, 17 Jun 2024 17:03:09 GMT</pubDate>
            <atom:updated>2025-06-04T06:40:19.669Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*02lOS6HdJNd543x-ksW7FQ.jpeg" /></figure><p>Choosing the right database for your application is a pivotal decision that can significantly influence your projects’s success. With a plethora of options available, each offering unique advantages and suited for different use cases, the process can be daunting. Whether you’re building a small scale application or a large enterprise system, the database you select will impact performance, scalability, maintainability and cost. Having seen my fair share of costly database migrations in projects, which could have been avoided if some due diligence was done while picking the database, I thought it would be useful to pen down the key factors that should influence your decisions on which database to use.</p><p>The most obvious place to start would be choice of relational or non relational database to choose(SQL or NoSQL). There are principles like the CAP Theorem that can help in this choice(more on this later in the article). For the sake of simplicity lets use the <strong>data you want to store</strong> and the <strong>kind of application</strong> you plan to build to help us make the choice.</p><h3>You should go with Relational Databases if:</h3><h4>Your data:</h4><ol><li>is hightly structured(well-defined and organized in a tabular format with rows and columns)</li><li>is highly relational (eg. Customers, Orders, Products)</li><li>is known and the schema is fixed</li><li>needs strong transactional integrity (ACID properties)</li><li>volume is moderate and usually is not categorised as big data</li></ol><h4>or if the application you develop:</h4><ol><li>handles financial data (needs strong transactional integrity)</li><li>is an Enterprise Resource Planning (ERP) System</li><li>is a Customer Relationship Management (CRM) System</li><li>is part of a Healthcare System</li><li>handles transactions in Retail and E-commerce</li></ol><p>The above pointers should give you an idea when to go for the nosql databases, but just for sake of completeness.</p><h3>You should go with NoSQL Databases if:</h3><h4>Your data:</h4><ol><li>is unstructured or semi structured(jsons, xmls, texts, logs, images etc)</li><li>can be represented in various models such as key-value pairs, documents, column families, or graphs.</li><li>does not have a fixed schema and can evolve over time</li><li>consistency is not the top priority, eventually consistent would do</li><li>is of high volume and velocity (big data)</li></ol><h4>or if the application you develop:</h4><ol><li>is like a content management system</li><li>is like E-commerce -product catalog</li><li>is like a social network application</li><li>does realtime data analytics</li><li>does gaming or is internet of things based</li></ol><p>Now that we have the ‘SQL or No SQL’ question answered, its time to figure out which database among the available choices in each category is the best. I have tried my best to give a comprehensive coverage on available choices their strengths and common usecases:</p><p>If you are going for relational Databases(open source first):</p><h4><strong>MySQL:</strong></h4><ul><li><strong>Strengths:</strong> Widely used, extensive documentation, large community, good performance for read-heavy applications.<em>(Open source)</em></li><li><strong>Use Cases:</strong> Web applications, e-commerce platforms, content management systems.</li></ul><h4><strong>PostgreSQL:</strong></h4><ul><li><strong>Strengths: </strong>Advanced SQL compliance, strong performance with complex queries, extensive feature set. <em>(Open source)</em></li><li><strong>Use Cases: </strong>Analytical applications, complex queries, geospatial applications, financial systems.</li></ul><h4><strong>SQLite:</strong></h4><ul><li><strong>Strengths:</strong> Lightweight, self-contained, serverless, zero-configuration.<em>(Open source)</em></li><li><strong>Use Cases:</strong> Embedded applications, mobile apps, small to medium-sized websites.</li></ul><h4>MariaDB</h4><ul><li><strong>Strengths:</strong>A fork of MySQL with additional features and improvements, Compatible with MySQL, making migration straightforward. Advanced storage engines like Aria and ColumnStore.<em>(Open source)</em></li><li><strong>Use Cases:</strong>Web applications, E-commerce platforms, General-purpose database applications</li></ul><p>Proprietary databases, are developed and maintained by companies that charge for their use. These databases typically come with advanced features, professional support, and regular updates. Here are some of the most well-known licensed relational databases:</p><h4><strong>Oracle Database:</strong></h4><ul><li><strong>Strengths: </strong>Comprehensive feature set, strong security, high performance, robust support for large-scale enterprise applications.</li><li><strong>Use Cases:</strong> Enterprise applications, large-scale data warehousing, mission-critical applications.</li></ul><h4><strong>Microsoft SQL Server:</strong></h4><ul><li><strong>Strengths:</strong> Tight integration with Microsoft ecosystem, strong BI and analytics capabilities, robust security features.</li><li><strong>Use Cases:</strong> Enterprise applications, business intelligence, data warehousing, applications within the Microsoft ecosystem.</li></ul><h4>IBM DB2:</h4><ul><li><strong>Strengths:</strong>High performance and scalability for large-scale enterprise applications.Advanced analytics and AI integration, with support for IBM Watson.Robust security features.High availability and disaster recovery options, such as Db2 pureScale.</li><li><strong>Use Cases: </strong>Enterprise resource planning (ERP) systems, Data warehousing and business intelligence, Banking and financial services, Healthcare systems</li></ul><p>If you are going for non relational databases a major point to keep in mind is that it scales by distributing data across multiple servers to handle larger datasets and higher traffic. Lets first see what are the most popular NoSQL databases and their usecases:</p><h3>MongoDB:</h3><ul><li><strong>Strengths: </strong>Flexible document-based data model (JSON-like BSON format), Rich query language with support for complex queries, indexing, and aggregation.Horizontal scalability with sharding and high availability with replication.</li><li><strong>Use Cases:</strong>Content Management Systems (CMS), Real-time Analytics, Internet of Things (IoT)</li></ul><h3>Cassandra:</h3><ul><li><strong>Strengths: </strong>Distributed architecture designed for high availability and scalability.Linear scalability with no single point of failure.Tunable consistency levels for trade-offs between consistency and availability.</li><li><strong>Use Cases: </strong>Time-Series Data, Messaging Systems, High-Write, Low-Latency Applications</li></ul><h3>Redis:</h3><ul><li><strong>Strengths: </strong>In-memory data store with support for various data structures (strings, hashes, lists, sets, sorted sets).Extremely fast read and write operations.Persistence options for durability and data replication.</li><li><strong>Use Cases: </strong>Caching, Session Management,Pub/Sub Messaging</li></ul><h3>Couchbase:</h3><ul><li><strong>Strengths:</strong>High performance with low-latency read and write operations.Flexible data model with support for JSON documents and key-value access.Built-in caching and memory-first architecture.</li><li><strong>Use Cases:</strong>Web Applications, User Profile Management, Content and Metadata Stores, Mobile and Edge Computing Applications</li></ul><h3>HBase:</h3><ul><li><strong>Strengths: </strong>Distributed, scalable, and consistent key-value store.Built on top of Hadoop Distributed File System (HDFS).Optimized for read-heavy workloads and large-scale data processing.</li><li><strong>Use Cases: </strong>Big Data Analytics, Real-time Querying of Massive Datasets, Internet of Things (IoT) Data Storage</li></ul><h3>Elasticsearch:</h3><ul><li><strong>Strengths:</strong>Distributed search and analytics engine.Full-text search capabilities with near real-time indexing.Scalable and highly available, with support for multi-tenancy.</li><li><strong>Use Cases: </strong>Log and Event Data Analysis, Full-Text Search, Real-Time Analytics, Monitoring and Observability</li></ul><p>Lately there is a ton of nosql database options to choose from, something that can assist the process is <strong>CAP Theorem</strong>.CAP Theorem was introduced in 1998, also known as Brewer’s theorem, is a fundamental principle in distributed computing that describes the trade-offs between three desirable properties of a distributed system: Consistency, Availability, and Partition tolerance. According to the CAP theorem, in a distributed system, it is impossible to simultaneously guarantee all three of these properties Consistency, Availability, and Partition tolerance. You can have any two but not all three together.Understanding the CAP theorem helps in designing and choosing appropriate distributed systems architectures based on the desired trade-offs.</p><p><strong>Examples:</strong></p><ul><li><strong>Consistency and Availability (CA):</strong>Traditional relational databases (e.g., Oracle, SQL Server) prioritize consistency and availability but may sacrifice partition tolerance. They typically operate in a single data center with synchronous replication.</li><li><strong>Consistency and Partition tolerance (CP):</strong>Some distributed databases (e.g., HBase, Cassandra) prioritize consistency and partition tolerance, sacrificing availability under network partitions. They use eventual consistency models.</li><li><strong>Availability and Partition tolerance (AP):</strong>Many NoSQL databases (e.g., DynamoDB, Couchbase) prioritize availability and partition tolerance, sacrificing strong consistency guarantees. They provide high availability and can handle network partitions but may have eventual consistency.</li></ul><p>By carefully assessing your requirements, considering factors such as data model, scalability, consistency, and latency, you can narrow down the options and choose a database that best fits your needs. Remember, the most suitable database today may not be the best choice tomorrow as your application evolves, so it’s important to periodically reassess your database choice to ensure it continues to meet your needs. Ultimately, a well-informed decision can pave the way for a robust and efficient application that delivers value to your users.</p><p>You can compare the ranking of the above mentioned databases and more here:<a href="https://db-engines.com/en/ranking">https://db-engines.com/en/ranking</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4c59f0339bf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Flask vs FastAPI for microservices]]></title>
            <link>https://medium.com/@anands282/flask-vs-fastapi-for-microservices-4c81fd77b7fa?source=rss-9950044d7c81------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c81fd77b7fa</guid>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[django]]></category>
            <category><![CDATA[microservice-architecture]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[flask]]></category>
            <dc:creator><![CDATA[Anand Satheesh]]></dc:creator>
            <pubDate>Thu, 13 Jun 2024 03:52:40 GMT</pubDate>
            <atom:updated>2024-06-13T03:52:40.194Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CzogRdlKOfr2oXKVzBWPmw.png" /></figure><p>Recently I came across an interesting design dilemma, to build a few microservices using python, which would be the best framework to use?</p><p>Let’s see the available choice of frameworks in python for building microservices :</p><ul><li>Flask</li><li>FastAPI</li><li>Django</li><li>Tornado</li><li>Sanic</li><li>Falcon</li><li>Pyramid</li><li>aiohttp</li></ul><p>I am pretty sure if you have been working with python in any domain you would have heard of the first three in the list -Flask, Django, FastAPI. Working mostly in data domain for my whole career, I have not worked on full fleged web projects at enterprise level, but I have used Flask and Django for smaller prototypes and hackathon projects. The type of microservices I was trying to build would be used internally in an organization and would have less than a couple of thousand users, though optimal performance is expected the system need not be overengineered to give maximum performance, so I was initially planning to go with Flask.</p><p>Then I started reading about FastAPI and I later changed my mind to use FastAPI framework instead of Flask. I thought it would be a good read to share my findings on comparing the use cases, strengths and weeknesses of each of these frameworks, which can enable someone to take a better decision on which would be the ideal framework to use in your project.</p><p>This analysis will focus on aspects such as request handling speed, concurrency, and ease of use for developers and also cover the pros and cons of each of these frameworks which can help you decide the best fit.</p><h3>Performance Analysis</h3><h4>1. Request Handling Speed</h4><p>FastAPI is designed for high performance, making it faster than Flask in handling requests. This is primarily due to its asynchronous nature and the use of Starlette and Pydantic under the hood.</p><p><strong>Flask:</strong></p><ul><li>Flask is synchronous and blocks each request until it is fully processed.</li><li>The synchronous nature can lead to slower response times when handling multiple concurrent requests.</li></ul><p><strong>FastAPI:</strong></p><ul><li>FastAPI is built on Starlette, which is designed for asynchronous operations.</li><li>It leverages async/await syntax in Python, which allows handling multiple requests concurrently without blocking.</li><li>FastAPI can be several times faster than Flask for I/O-bound operations.</li></ul><h4>2. Concurrency</h4><p>FastAPI’s support for asynchronous programming gives it a significant edge in handling concurrent requests efficiently.</p><p><strong>Flask:</strong></p><ul><li>Flask can handle concurrency using WSGI servers like Gunicorn, but it is not inherently designed for asynchronous operations.</li><li>Concurrency in Flask can be managed, but it requires more effort and careful configuration.</li></ul><p><strong>FastAPI:</strong></p><ul><li>FastAPI natively supports asynchronous request handling.</li><li>It can handle a large number of concurrent connections, making it highly suitable for modern web applications with high traffic.</li></ul><h4>3. Ease of Use and Developer Productivity</h4><p>Both frameworks are easy to use, but they cater to slightly different use cases and developer preferences.</p><p><strong>Flask:</strong></p><ul><li>Flask is simple and minimalistic, making it easy to get started.</li><li>The flexibility of Flask allows developers to build applications their way, but it may require more boilerplate code for complex applications.</li><li>Flask has a larger ecosystem of extensions and community resources.</li></ul><p><strong>FastAPI:</strong></p><ul><li>FastAPI leverages Python type hints, providing better editor support and reducing runtime errors.</li><li>Automatic generation of interactive API documentation (Swagger UI, ReDoc) enhances developer productivity.</li><li>The learning curve for asynchronous programming and type hints might be steeper, but it results in cleaner and more efficient code.</li></ul><h3>Benchmarks</h3><p>Several benchmarks have been conducted by independent developers comparing Flask and FastAPI. Here are some key findings:</p><p><strong>Latency and Throughput:</strong></p><ul><li>FastAPI generally shows lower latency and higher throughput compared to Flask.</li><li>For a simple “Hello World” endpoint, FastAPI can handle more requests per second than Flask.</li></ul><p><strong>CPU and Memory Usage:</strong></p><ul><li>FastAPI tends to use CPU and memory more efficiently due to its asynchronous nature.</li><li>Flask, while efficient for smaller applications, can become resource-intensive under heavy load.</li></ul><h3>Example Benchmark Results</h3><p>A typical benchmark might look like the following:</p><p><strong>Hello World Endpoint:</strong></p><ul><li><strong>Flask:</strong> 1,000 requests per second</li><li><strong>FastAPI:</strong> 5,000 requests per second</li></ul><p><strong>Database Query (I/O-bound):</strong></p><ul><li><strong>Flask:</strong> 500 requests per second</li><li><strong>FastAPI:</strong> 2,500 requests per second</li></ul><p>Now lets see some of the strengths and weekeness of the frameworks:</p><h3>Flask</h3><p><strong>Pros:</strong></p><ul><li><strong>Mature and Well-Documented:</strong> Flask has been around longer, and there is a wealth of documentation and community support available.</li><li><strong>Flexibility:</strong> Flask is very flexible and can be customized to fit almost any need.</li><li><strong>Extensions:</strong> There are numerous extensions available for Flask that can add additional functionality to your application.</li><li><strong>Familiarity:</strong> Many developers are already familiar with Flask, so finding talent or getting up to speed might be easier.</li></ul><p><strong>Cons:</strong></p><ul><li><strong>Performance:</strong> Flask is not as performant as FastAPI, especially for applications that require handling a large number of requests concurrently.</li><li><strong>Asynchronous Support:</strong> While Flask can support asynchronous programming, it is not designed for it out of the box and may require additional configuration.</li></ul><h3>FastAPI</h3><p><strong>Pros:</strong></p><ul><li><strong>Performance:</strong> FastAPI is built on top of Starlette and Pydantic, making it one of the fastest frameworks available. It is designed to handle a high number of requests efficiently.</li><li><strong>Asynchronous Support:</strong> FastAPI has built-in support for asynchronous programming using async/await, making it ideal for modern, high-performance web applications.</li><li><strong>Data Validation:</strong> FastAPI uses Pydantic for data validation, which simplifies the process of handling and validating request data.</li><li><strong>Type Hints:</strong> FastAPI leverages Python type hints to provide better editor support and error checking, which can improve developer productivity.</li><li><strong>Automatic Documentation:</strong> FastAPI automatically generates interactive API documentation using Swagger UI and ReDoc, which can be very useful for development and debugging.</li></ul><p><strong>Cons:</strong></p><ul><li><strong>Less Mature:</strong> FastAPI is relatively newer compared to Flask, so there might be fewer community resources and extensions available.</li><li><strong>Learning Curve:</strong> If you are not familiar with asynchronous programming or type hints, there might be a bit of a learning curve.</li></ul><h3>Conclusion</h3><p>If performance, asynchronous support, and modern features like automatic documentation are critical for your microservices project, <strong>FastAPI</strong> might be the better choice. On the other hand, if you prefer a mature framework with a lot of existing resources and flexibility, <strong>Flask</strong> could be more suitable.</p><p>Considering your experience and the nature of your project could also influence the decision. If you are comfortable with learning new technologies and require high performance, FastAPI might be the way to go. If you prefer stability and a well-established framework, Flask would be a solid choice.Thank you for reading! I appreciate your time and interest in my content. Stay tuned for more insightful articles.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c81fd77b7fa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting up a Kafka pipeline to ingest a stream of stock market data]]></title>
            <link>https://medium.com/@anands282/setting-up-a-kafka-pipeline-to-ingest-a-stream-of-stock-market-data-2ff8ab95d584?source=rss-9950044d7c81------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ff8ab95d584</guid>
            <category><![CDATA[amazon-s3]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[pyspark]]></category>
            <dc:creator><![CDATA[Anand Satheesh]]></dc:creator>
            <pubDate>Thu, 30 May 2024 17:49:50 GMT</pubDate>
            <atom:updated>2024-05-30T17:49:50.314Z</atom:updated>
            <content:encoded><![CDATA[<p>This project covers the following topics:</p><ul><li>Setting up a local kafka and zookeeper docker containers</li><li>Setting up a producer job to imitate a stream of input records, generating and sending data to kafka topic every 3 seconds</li><li>Setting up a consumer job that collects the data from topic and stores in Amazon s3 bucket</li><li>Reading the files from S3 bucket using pyspark for processing</li></ul><p>For setting up Kafka in you local , you would need docker desktop installed. The following docker-compose.yml will create a container of kafka and zookeeper running</p><pre>version: &#39;3&#39;<br><br>services:<br>  zookeeper:<br>    image: wurstmeister/zookeeper:latest<br>    ports:<br>      - &quot;2181:2181&quot;<br>    environment:<br>      ZOOKEEPER_CLIENT_PORT: 2181<br>      ZOOKEEPER_TICK_TIME: 2000<br><br>  kafka:<br>    image: wurstmeister/kafka:latest<br>    ports:<br>      - &quot;9092:9092&quot;<br>    expose:<br>      - &quot;9093&quot;<br>    environment:<br>      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092<br>      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT<br>      KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092<br>      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE<br>      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181</pre><p>Run the following command in the same directory as your docker-compose.yml file to start the Kafka and ZooKeeper containers:</p><pre>docker-compose up -d</pre><p>Verify that the Kafka and ZooKeeper containers are running:</p><pre>docker-compose ps</pre><p>To interact with Kafka, you can use the kafka-console-producer and kafka-console-consumer tools. For example, to start a producer:</p><pre>docker-compose exec kafka /opt/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test</pre><p>And to start a consumer:</p><pre>docker-compose exec kafka /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning</pre><p>Now you have a basic Apache Kafka setup running locally using Docker. You can use this setup for development and testing purposes.</p><p>create a topic called stocks</p><pre>docker-compose exec kafka /opt/kafka/bin/kafka-topics.sh --create --topic stocks --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1</pre><p>Create a new python project in your favorite IDE(I have used PyCharm) and named it StockMarketAnalysis. I have used a csv file from Kaggle called all_stocks_5yr.csv (<a href="https://www.kaggle.com/datasets/camnugent/sandp500">https://www.kaggle.com/datasets/camnugent/sandp500</a> ) download this csv file and add it a data directory within your project. Ideally the producer would fetch data from an api or a source which keeps creating data in real time, here we are mimicing such a source in our producer job by randomly selecting a sample record from the csv and pushing it to the kafka topic , this is the producer code:</p><pre>import pandas as pd<br>from kafka import KafkaProducer<br>from json import dumps<br>import time<br><br>def produce():<br>    producer = KafkaProducer(bootstrap_servers=[&#39;localhost:9092&#39;],<br>                             value_serializer=lambda x: dumps(x).encode(&#39;utf-8&#39;))<br><br>    # Extracting data from our csv file and sending it to producer in loop<br>    df = pd.read_csv(&quot;data/all_stocks_5yr.csv&quot;)<br>    while True:<br>        stock_record = df.sample(1).to_dict(orient=&quot;records&quot;)[0]<br>        producer.send(&#39;stocks&#39;, value=stock_record)<br>        # wait for 30 seconds to send the next<br>        time.sleep(3)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    try:<br>        produce()<br>    except Exception as producer_error:<br>        print(&quot;Error in running producer process. &quot;, producer_error)</pre><p>Once the Producer adds the data to the newly created topic ‘stocks’ in our kafka container, we can use our producer job to fetch the data in realtime and add to a data lake, here I have created an s3 bucket ‘stock-market-project-store’ to act as a data storage layer for the consumer data. This data can directly be processed in the consumer job also but its a nice practise to store the data in a raw zone of the data lake so that it is available for further processing by any other teams, or archival of the as-is data. Below given is the consumer code to fetch the data from kafka topic and add to the s3 bucket.</p><pre>import json<br>from kafka import KafkaConsumer<br>from json import loads<br>from s3fs import S3FileSystem<br><br><br>def consume():<br>    consumer = KafkaConsumer(&#39;stocks&#39;, bootstrap_servers=[&#39;localhost:9092&#39;],<br>                             value_deserializer=lambda x: loads(x.decode(&#39;utf-8&#39;)))<br>    while True:<br>        s3_connection = S3FileSystem()<br>        for i, stock in enumerate(consumer):<br>            with s3_connection.open(f&quot;s3://stock-market-project-store/stock_market_analysis_part{i}.json&quot;) as file:<br>                json.dump(stock.value, file)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    try:<br>        consume()<br>    except Exception as consumer_error:<br>        print(&quot;Error in running consumer process. &quot;, consumer_error)</pre><p>Now that we have the data in our raw zone of our s3 bucket, we can write a simple pyspark pipeline to read the json data into dataframe for further processing.</p><pre>from pyspark.sql import SparkSession<br><br># Create a Spark session<br>spark = SparkSession.builder \<br>    .appName(&quot;Read JSON files from S3&quot;) \<br>    .getOrCreate()<br><br># S3 bucket path where the JSON files are stored<br>bucket_path = &quot;s3://stock-market-project-store/*.json&quot;<br><br># Read JSON files into a DataFrame<br>df = spark.read.json(bucket_path)<br><br>df.printSchema()<br>df.show()<br>spark.stop()</pre><p>Since we have the data in AWS S3, we can also create a glue crawler to crawl and catalog the data so that we can query the data directly using Amazon Athena. Thanks for Reading, Happy coding!!!</p><p>Full code for the project is available here: <a href="https://github.com/anands282/StockMarketAnalysis">https://github.com/anands282/StockMarketAnalysis</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ff8ab95d584" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Practical Guide to Solve LeetCode Problems]]></title>
            <link>https://medium.com/@anands282/a-practical-guide-to-solve-leetcode-problems-8d9930909e0e?source=rss-9950044d7c81------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d9930909e0e</guid>
            <category><![CDATA[leetcode]]></category>
            <category><![CDATA[job-hunting]]></category>
            <category><![CDATA[maang]]></category>
            <dc:creator><![CDATA[Anand Satheesh]]></dc:creator>
            <pubDate>Thu, 30 May 2024 13:51:05 GMT</pubDate>
            <atom:updated>2024-05-30T13:51:46.807Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A11dXFlJMCTFQKpmxnkpSw.png" /></figure><p>If you landed on this article, chances are you are in the 95% (totally made up stats btw) of readers who are trying to make a job switch to a MAANG/Product based Company and are trying to improve your DSA skills by solving LeetCode Problems. If I am right, please continue reading.</p><p>Also I am taking a guess that you already tried to solve or read through a few of those and found it super super difficult to tackle. Lets address the obvious question, does that make you a bad software engineer? I would say Absolutely Not, and here is the reason why. There would be a section of engineers who got trained in a different stream of engineering and joined IT sector as that was the only available option at the time and later you started liking the job and decided to stick to it. And for this catergory of folks DSA(Data Structures and Algorithms) would not have ever crossed paths and hence solving such problems does’nt come naturally to you. For the Computer Engineers, you must have lost touch or forgot the fine details of DSA since it has been years after your college. Whatever be the case, I would say a moderate level of understanding in DSA is needed when you start on this journey. Youtube, udemy, GeeksforGeeks lots of platforms provide courses on this.</p><p>For IT folks who are into tool/UI based roles and do not code regularly, First thing I would suggest is pick a language (Java). You might have expected me to suggest python but for DSA but I would suggest Java. This is again a personal choice as the concepts are going to be fairly same just the implemenation differs.</p><p>Use free youtube resources to learn or revise a particular Data Structure, then once you have done a crash course (a couple of hours) . Go to LeetCode and you can filter questions based on the datastructure you just revised, pick an easy problem. Spend around 5–10 minutes on it trying to understand the question fully, use a pencil and paper to try and visualize the problem draw trees or arrays or stacks and fill in the elements in the question. Without knowledge of any algorithms or implementaion details of any sort, imagine how would you solve this. There is a high chance that either you could not find a good solution or you might not see any solutions at all. This is where AI can come to the rescue, open chatgpt and enter the question and ask it to give a solution and explain it to you.</p><p>The reason I would ask you not to spend more than 10 minutes is because, our attention starts to wander or we would get disheartened and maybe abandon the task altogether. You can give various prompts to explain different areas of the solution and it really does a pretty descent job for the most part. The smart thing to do here is not to take this to your ego, that you are not able to solve this on your own. The key take away from this exercise is that you understand the solution and instead of copy pasting the code try to implement it on your own in the leetcode page. The reason why I suggest this method is because once you start repeating this, you will see that there are only a handful of patterns that you use for all of these problems like <strong>Sliding Window, Two Pointers or Iterators etc.</strong></p><p>Repeat this strategy for 10,20 questions and then you will see that there is some more structure to the way you approach a new problem. Your brain is now conditioned to automatically check for similar problems you solved or which pattern this fits into. In cases where the interviewer is ok with the raw logic or pseudo code you would be able to crack the round easily. Now you add more practice to where you are now, include different datastructures and within a couple of months you have a good number of problems solved and a more practical handson level of understanding of Data Structures and Algorithms.</p><p>Hope this guide is useful for some of you, All the best and keep coding!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d9930909e0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>