What are the ADLS create limits, and how do they affect running batch jobs on a shared interactive cluster?

Azure Data Lake Storage (ADLS) imposes limits on the number of files that can exist in a single namespace (folder) and per account. The default limit is 5 million files per namespace, but this can be reduced by subscription or account settings. When a batch job creates many small files—common with Spark or Flink writes—the file‑count ceiling can be hit quickly, causing write failures or throttling. This limit is independent of compute resources; it is a storage‑side constraint. Running batch jobs on a shared interactive cluster does not directly affect the ADLS file‑count limit, but it can exacerbate the problem if the cluster writes many small files concurrently. Best practices include coalescing or repartitioning data to reduce the number of output files, using a dedicated storage account for heavy write workloads, and monitoring the file‑count metric. By keeping the file count below the threshold, you avoid write errors and maintain cluster performance.