chore: update datafusion to 52

[killzoner]

Which issue does this PR close?

Closes #1357

Rationale for this change

Datafusion upgrade changes: https://datafusion.apache.org/library-user-guide/upgrading.html#datafusion-52-0-0

What changes are included in this PR?

• Update to ballista proto to expose new metrics added by datafusion
• Adapt repartition interface to feat: split BatchPartitioner::try_new into hash and round-robin constructors datafusion#19668 changes
• Removal of CoalesceBatchesExec references

Are there any user-facing changes?

No

[milenkovicm]

Looks like test failures should be fixable.
Please update ballista versions to 52 as well

Thanks @killzoner

[killzoner]

Looks like test failures should be fixable. Please update ballista versions to 52 as well

Thanks @killzoner

Hey, i will have a look soon at the failing tests. I need to take a look at FetchPartition inner working, seems like the CSV/parquet files written in the tests are valid ones, but there is something missing.

[milenkovicm]

add file name to written csv&jsons at write time. not sure what changed in datafusion

[andygrove]

@sqlbenchmark run tpch

[sqlbenchmark]

Ballista TPC-H Benchmark Results

PR: #1394 - chore: update ballista crates to 52
PR Commit: 8a63449
Base Commit: 38fca8b (main)
Scale Factor: SF1
Iterations: 3

Query Comparison

Query	Main (ms)	PR (ms)	Change
Q1	743.40	785.50	🔴 +5.7%
Q2	1360.30	1388.80	⚪ +2.1%
Q3	1080.00	1001.70	🟢 -7.2%
Q4	849.70	845.20	⚪ -0.5%
Q5	1791.10	1712.80	⚪ -4.4%
Q6	422.00	388.50	🟢 -7.9%
Q7	1848.70	1803.70	⚪ -2.4%
Q8	2289.80	2310.80	⚪ +0.9%
Q9	1860.10	1913.80	⚪ +2.9%
Q10	1265.50	1400.20	🔴 +10.6%
Q11	1069.10	1083.10	⚪ +1.3%
Q12	856.70	828.90	⚪ -3.2%
Q13	957.30	930.50	⚪ -2.8%
Q14	605.10	627.50	⚪ +3.7%
Q15	967.50	1000.20	⚪ +3.4%
Q16	1307.60	1286.80	⚪ -1.6%
Q17	729.60	722.00	⚪ -1.0%
Q18	1393.20	1230.40	🟢 -11.7%
Q19	779.00	743.60	⚪ -4.5%
Q20	988.10	1070.00	🔴 +8.3%
Q21	1746.50	1726.10	⚪ -1.2%
Q22	898.50	912.90	⚪ +1.6%

Total: Main=25808.80ms, PR=25713.00ms (-0.4%)

---

Automated benchmark run by dfbench

[milenkovicm]

I suggest keeping this PR open for a bit more, as there is datafusion 51.1 in release process

[milenkovicm]

thanks @killzoner
once 51.1 is released we can merge this

[killzoner]

thanks @killzoner once 51.1 is released we can merge this

I'm fixing the conflict and we're good then I think

[milenkovicm]

correction, once 52.1 is released

[milenkovicm]

thanks @killzoner
i've updated df to 52.1 will merge this after CI. If you're interested there is #1395 as follow up

[milenkovicm]

@sqlbenchmark run tpch -s 10 -i 3

[sqlbenchmark]

Ballista TPC-H Benchmark Results

PR: #1394 - update to latest df 52.1
PR Commit: 9e05f66
Base Commit: ac49c18 (main)
Scale Factor: SF10
Iterations: 3

Query Comparison

Query	Main (ms)	PR (ms)	Change
Q1	2554.70	2520.90	⚪ -1.3%
Q2	2004.70	1934.70	⚪ -3.5%
Q3	2037.40	2070.20	⚪ +1.6%
Q4	1269.00	1233.70	⚪ -2.8%
Q5	3937.30	4012.40	⚪ +1.9%
Q6	978.90	930.00	⚪ -5.0%
Q7	5498.50	5504.60	⚪ +0.1%
Q8	5233.40	5045.10	⚪ -3.6%
Q9	6175.50	6197.50	⚪ +0.4%
Q10	2342.90	2389.50	⚪ +2.0%
Q11	1474.90	1541.90	⚪ +4.5%
Q12	1790.20	1776.90	⚪ -0.7%
Q13	2128.20	2114.60	⚪ -0.6%
Q14	1087.00	1066.40	⚪ -1.9%
Q15	1645.30	1554.00	🟢 -5.5%
Q16	1293.60	1409.20	🔴 +8.9%
Q17	4115.30	4045.40	⚪ -1.7%
Q18	6517.70	6453.60	⚪ -1.0%
Q19	1860.30	1806.50	⚪ -2.9%
Q20	1772.30	1778.60	⚪ +0.4%
Q21	6043.40	6080.50	⚪ +0.6%
Q22	1068.80	1089.90	⚪ +2.0%

Total: Main=62829.30ms, PR=62556.10ms (-0.4%)

---

Automated benchmark run by dfbench