You mention that the constraint for broadcast joins in Spark, and therefore the general size threshold for considering the switch from Kimball to OBT, is 10gb. But the default value for spark.sql.autoBroadcastJoinThreshold is 10mb, not 10gb. Was this a mistake on your part or is the default value for that config too small in your view?
You mention that the constraint for broadcast joins in Spark, and therefore the general size threshold for considering the switch from Kimball to OBT, is 10gb. But the default value for spark.sql.autoBroadcastJoinThreshold is 10mb, not 10gb. Was this a mistake on your part or is the default value for that config too small in your view?
You mention that the constraint for broadcast joins in Spark, and therefore the general size threshold for considering the switch from Kimball to OBT, is 10gb. But the default value for spark.sql.autoBroadcastJoinThreshold is 10mb, not 10gb. Was this a mistake on your part or is the default value for that config too small in your view?
Broadcast join can be bumped up a lot higher than 10 mbs. It struggles after 10 GBs though